SAMI — retrieval-grounded iSAMS assistant

Context

iSAMS is a school information management platform with a large surface area and documentation spread across many places. Practitioners answer the same questions for each other repeatedly. SAMI is a free, community-facing assistant that answers those questions from a curated knowledge base — built and run by EthanCorp, with no commercial relationship to iSAMS.

Why it exists

Two reasons, and it is worth being straight about both. It is genuinely useful to a community I work in. It is also the reference implementation for how EthanCorp builds retrieval-grounded systems: if the approach is sound, it should survive being live, free and open to anyone — including people asking questions it was not designed for.

Constraints

• Free to use, which makes per-answer cost a real design constraint rather than a footnote.
• Answers must be traceable to curated material — a confidently wrong answer about a school system is worse than no answer.
• Open to the public, so it has to behave reasonably on questions outside its knowledge base.
• Run by one person, so operational overhead had to be near zero.

Approach

Retrieval is grounded in a curated knowledge base of 17 in-depth guides, embedded — together with a set of curated Q&A — into more than 8,000 indexed passages in Qdrant. The model answers from retrieved passages rather than from its own general memory, which is the difference between an assistant that cites something and one that improvises plausibly.

The operational layer is the part most RAG demos skip. There is an admin console for content and access, a usage and cost dashboard showing per-model spend, and a feedback review queue. Reviewed corrections go back into the curated knowledge base — that loop, not the model choice, is what makes answers get better over time.

238 automated tests cover the frontend and backend (162 backend, 76 frontend). For a free side project that is arguably excessive; for something being used as evidence of how EthanCorp builds, it is the point.

What I built

• Curated and structured the knowledge base of 17 in-depth guides.
• Built the Next.js frontend with PWA support, and the FastAPI backend.
• Built the retrieval layer over Qdrant, and the answering flow constrained to retrieved passages.
• Built the admin console, the usage and cost dashboard, and the feedback review queue.
• Wrote 238 automated frontend and backend tests, and packaged the whole thing with Docker Compose.

Systems and technologies

Next.js (App Router) on the frontend with PWA support; FastAPI on the backend; PostgreSQL for conversations, feedback and usage; Qdrant as the vector store; Docker Compose for deployment.

How answer quality is checked

Before a prompt or model change ships, a quality gate runs against the live backend: 20 real iSAMS questions with known-good answers, 3 trick questions that have no answer in the knowledge base, and 3 conversations where a trick question comes first and a real one second. Each answer is checked for a refusal where there should be none, and the reverse, then scored by a second model against a rubric for that question.

The latest run, on 25 September 2026, passed 24 of 26 (92%). The two misses were re-read by hand and turned out to be scoring variance, not wrong answers. The set is small and self-written — not yet drawn from real traffic — and it runs by hand because every run costs money. It is a gate before a change, not continuous monitoring.

Known limitations

• No automated backups of the database or the vector store. The vector store can be rebuilt from source; conversation and feedback history since the last manual dump could not.
• Manual deploys. CI runs lint and the test suite on every push; nothing deploys to production automatically.
• Retrieval is a single vector search, with no reranking and no hybrid keyword search.
• The knowledge base is English-only. Questions in other languages are answered in that language, from English guides.
• No service-wide spend cap. Per-user and per-IP limits exist; there is no daily spending breaker or alert across the whole service.
• Single-tenant: one shared knowledge base, not built to isolate multiple organisations.