Ask your documents: AI search that cites its source and keeps private files at home

I ask my assistant "what did I buy at Decathlon?" and a few seconds later it answers with the amount and a link to the scanned receipt. I didn't remember, and the assistant doesn't know it by heart: it looks it up. The receipt came in by email, went through Paperless OCR and ended up split into chunks in a search index that lives on a Raspberry Pi.

It's one of the first things that comes up when a company considers AI: "I want to ask my documents". Contracts, procedures, spec sheets, delivery notes. Here I'll show you how I run it for myself, with my real numbers, and what I'd change to build it for a small business.

What it does#

• It indexes two sources: Paperless documents (scanned, with OCR) and my Obsidian notes.
• It searches two ways at once, by keywords and by meaning, and merges the results.
• It returns chunks with a title and a link to the original document, so the answer can be checked.
• If a document is sensitive (payslips, health, contracts, bank statements), its content never leaves home: a local model answers and the reply reaches me on Telegram.

flowchart LR
  P[Paperless: OCR scans] --> I[Indexer every 30 min]
  O[Notes] --> I
  I --> DB[(SQLite: FTS5 + vectors)]
  Q[Question] --> S[Hybrid search]
  DB --> S
  S -->|normal document| A[Assistant with citations]
  S -->|sensitive document| L[Local model, reply on Telegram]

The parts#

One SQLite file. I didn't want a managed vector database. SQLite ships FTS5 for keyword search (BM25), and the sqlite-vec extension stores the 1024-dimension vectors. It fits on a Raspberry Pi with little RAM to spare: the first index of 167 documents and 1,170 chunks took 44 seconds and used 35 MB of memory. Later runs only process what changed and take a second.

Local embeddings. bge-m3 computes the vectors on my GPU desktop, behind a LiteLLM gateway with no route to the cloud. If the desktop is off, search drops to keywords only and the indexer retries the pending vectors on its next run. It works worse for me, but it still works.

Merging results. Each search returns 30 candidates, and I combine them with Reciprocal Rank Fusion: every document scores by its position in each list. It's a one-line formula and needs no weight tuning.

A read-only MCP server. The assistant gets two tools: search, and ask privately. It can't write, delete or upload anything.

Private stays private#

Privacy works by document type, not by the model's good will. Every Paperless document has a type, and six types are marked sensitive in a config file. When search returns one of them, the assistant only sees the title, type and link, with a "private content" notice. If I want the answer, a local model asks in the background and the reply reaches me on Telegram.

Two details I learned the hard way:

• Fail closed. The indexer's read-only user couldn't see document types, so everything was stored as "no type" and the sensitive rule never matched. I fixed the permission and, more importantly, the rule: if a document's type can't be resolved, it's treated as sensitive.
• Retrieved text is data, not an order. A PDF can contain "ignore your instructions and...". Chunks reach the model delimited and marked as untrusted, and the tools are read-only, so the worst case is a bad answer, not an action.

The evaluation that made me switch a part off#

I also set up a reranker, a model that reorders the top 10 candidates. Before leaving it on, I measured it with 15 real questions, each with its correct document:

The reranker got one more question out of fifteen right in first place. With 15 questions, that's within noise. In exchange it cost almost 3 seconds per query and 1.8 GB of RAM on the desktop. Since hybrid search already puts the right document in the top five, which is what the model reads, I switched it off. I'll measure again when the index holds hundreds of documents.

Without the table, I'd have left it on because "that's what you do".

What broke#

• The assistant answered without searching. MCP tools were hidden behind a tool search and the system prompt didn't name them, so the model improvised. Naming them in the prompt fixed it.
• A 32-bit extension. The first sqlite-vec version I installed shipped a 32-bit library for ARM and wouldn't load on the Pi. The next minor version did.
• The reranker ran out of memory on start. With default settings the container died while loading its weights. I lowered the batch size and gave it a fixed memory limit.

If you build it for a company#

What changes is the size and who controls the data, not the idea:

• Sources. Instead of Paperless, the SharePoint folder or the file server. If they already have Microsoft 365, first check whether Copilot with their licences is enough. Sometimes it is, and then there's nothing to build.
• Where it runs. On a company server or an EU VPS in their name. The model can be local or one in a European region, depending on how sensitive the documents are.
• Permissions. Search has to respect who can see what. It's the hardest part and the one you can't improvise.
• Evaluation before handover. Twenty or thirty real questions from the team, each with its correct document. If you can't measure it, you can't know whether it works.

If your team loses time digging through folders, let's talk and see whether it pays off.