Private RAG: Question Answering Over Internal Documents
Private RAG lets a local model answer questions from your own files and cite them. This walks through each stage on a single server and shows where document permissions have to be enforced.

Take a firm with 8,000 internal documents averaging 6 pages of about 500 words each. That is 8,000 x 6 x 500 = 24 million words, far more than any language model can read in one prompt. Private RAG (retrieval-augmented generation, run entirely on hardware you control) solves this by finding the handful of passages that matter for each question and handing only those to a local model, which answers and cites where each claim came from. The term comes from a 2020 research paper by Lewis and colleagues; the business version is simpler than the paper and fits on one server.
Most of the pipeline is forgiving: a weak chunking choice or a mediocre embedding model costs answer quality and can be fixed later. Document permissions are the exception. Get them wrong and the system answers questions from files the asker was never allowed to open.
The six stages on one machine
A working system has two halves. Ingestion runs in the background and turns files into searchable pieces. Answering runs when someone asks a question.
| Stage | Half | What it does | Typical component |
|---|---|---|---|
| Ingest | Background | Pulls files and their metadata from shares, a DMS or email archives | A scheduled sync script |
| Chunk | Background | Splits text into passages of a few hundred words | Parser plus splitter |
| Embed | Background | Turns each passage into a vector of numbers | A local embedding model |
| Store | Background | Keeps text, vectors, metadata and permissions together | Postgres with pgvector, or a vector database |
| Retrieve | Per question | Finds the passages closest to the question that this user may see | Vector search plus keyword search |
| Answer | Per question | Writes an answer from those passages, with citations | A local LLM |
All six can live on one box. The embedding model is small next to the chat model and can usually share its GPU. The database is ordinary. The heavy part is the language model, and its sizing is the same question as for any local chat deployment.
Ingest and chunk: where answer quality is decided
Most bad answers trace back to ingestion rather than the model. If a scanned PDF has no text layer, the system never sees its contents. If a table is flattened into a stream of numbers, the model cannot tell which figure belongs to which row.
Ingestion should capture, for every file:
- the extracted text, with headings preserved where possible
- the source path or document ID, so citations can link back
- the last-modified date, so changed files get re-processed
- the access list from the source system, which matters later
Chunking then splits each document into passages. Splitting on structure (sections, headings, numbered clauses) usually beats cutting every N characters, because a passage that starts mid-sentence retrieves poorly and reads worse. The trade-offs of size, overlap and table handling are covered in chunking documents for RAG.
Continuing the example: at about 400 words per chunk, 24 million words becomes 24,000,000 / 400 = 60,000 chunks.
Embed and store: smaller than people expect
An embedding model reads a passage and outputs a fixed-length list of numbers, so that passages with similar meaning land near each other. At question time the question is embedded the same way and compared against the stored vectors. Which model to use, and how to compare two on your own queries, is the subject of choosing a local embedding model.
Storage is modest. pgvector documents that each vector takes 4 x dimensions + 8 bytes. For a model that outputs 1,024 dimensions:
- one vector: 4 x 1,024 + 8 = 4,104 bytes
- 60,000 chunks: 60,000 x 4,104 = about 246 MB
- the chunk text itself: 24 million words at roughly 6 bytes per word including spaces, about 144 MB
Add an approximate-nearest-neighbor index and some headroom and the whole store is still well under a few gigabytes. A firm of this size does not need a dedicated cluster. Whether the Postgres you already run is enough, or a separate engine earns its place, is weighed in pgvector versus a dedicated vector database.
Keep the text, vector, source ID, date and permission data in the same row. Splitting them across systems is how permissions drift out of sync.
Retrieve: vector search plus keywords
Pure vector search is good at meaning and weak at exact strings. A question about "invoice 40417" or a part number like "HX-220B" may retrieve passages about invoices in general. A common fix is to run two searches, one by vector similarity and one by keyword (Postgres full-text search works), and merge the result lists.
The number of passages handed to the model has a direct cost. Six chunks of 400 words is 2,400 words, which at about 0.75 words per token is 2,400 / 0.75 = 3,200 tokens. Add system instructions, the question and room for the answer, and a single request needs around 4,000 to 5,000 tokens of context. If the runtime's context setting is smaller than that, the runtime may truncate the prompt and the model never sees part of what was retrieved. Check the context setting before blaming the model.
Answer with citations
The prompt to the model carries the retrieved passages, each tagged with an ID, and instructions along these lines:
Answer only from the passages below.
After each claim, cite the passage ID in brackets.
If the passages do not contain the answer, say "Not found in the documents."
Citations do two jobs. Staff can click through and check the source in seconds, and the application can verify that each cited ID was actually among the retrieved passages. A cited ID that was never retrieved is a clear sign the model made something up. Further controls, such as checking that a quoted sentence exists in the source, belong to reducing hallucinations in document Q&A.
Allowing "not found" matters more than it looks. A system that always produces an answer will produce wrong ones when retrieval misses, and staff learn to distrust all of it.
Where document permissions must be enforced
Here is the failure that private hosting does not prevent. A file share has folders that only HR, partners or a specific matter team can open. Ingestion runs under a service account that can read everything. Every chunk goes into one index. Now any employee can ask "what is the salary range for the operations director role" and get a cited answer, from a document they could never have opened directly.
OWASP lists this under vector and embedding weaknesses in its 2025 LLM Top 10, recommending "fine-grained access controls and permission-aware vector and embedding stores" and strict partitioning so one group cannot reach another's data.
The rule: filter at retrieval, never after
Permissions have to be applied when passages are fetched, before anything reaches the model. Asking the model to "ignore documents the user may not see" does not work; once text is in the prompt, it can surface in the answer. Filtering the final answer does not work either, because the content has already shaped it.
In practice:
- At ingestion, copy each file's access list (users and groups) onto every chunk from that file.
- At question time, resolve the asking user's identity and group memberships from your directory.
- Retrieve only chunks whose access list intersects those groups. This is a database filter, not a prompt instruction.
- When permissions change at the source, update the chunks. When a file is deleted, delete its chunks.
Step 4 is the easiest to miss. If a document is moved out of a shared folder on Monday and the index syncs weekly, it stays answerable to the old audience until the next sync. Sync frequency is a security setting: run it at least nightly, incrementally by last-modified date, with an immediate removal path for files that must disappear at once.
Database details that bite
If the store is Postgres, row-level security can enforce the filter inside the database rather than in application code. Two details from the Postgres documentation matter. Superusers and roles with BYPASSRLS always bypass row security, and table owners normally do too unless the table is altered with FORCE ROW LEVEL SECURITY. The safe pattern is an application role that is neither a superuser nor the table owner. And once security is enabled with no policy defined, a default-deny policy applies and no rows are visible, which is the safe failure.
Filtering also interacts with vector indexes. The pgvector documentation notes that with approximate indexes, the WHERE filter is applied after the index scan, so a restrictive filter can return fewer results than requested. Its README gives the arithmetic: an HNSW index with the default hnsw.ef_search of 40 and a condition matching 10% of rows yields only about 4 matches on average. A user with access to a small slice of the archive might get two passages when six were asked for. pgvector added iterative index scans in version 0.8.0 to address this; per its changelog the current release in October 2026 is 0.8.7. Before rollout, test retrieval with a low-privilege account as well as an admin one.
When one index is the wrong design
For groups that must never mix, such as separate clients in a law or accounting firm, a separate collection or schema per group is simpler to audit than one index with filters. A filter bug leaks across groups; separate stores make that class of bug impossible. The cost is more indexes to maintain, which on a single server is usually acceptable.
A workable starting point: one shared index for documents everyone may read, plus separate stores for any group whose data must be walled off, and a test account per group that tries to retrieve another group's documents before go-live.
Frequently asked questions
Can restricted content leak through anything other than retrieval?
Yes, through features built around the model. An answer cache keyed only on the question text can serve an answer built from HR files to the next person who asks the same thing. Shared conversation threads and saved prompts can carry retrieved passages to people outside the original audience. Key any cache on the user's permission set as well as the question, and treat shared chats as copies of the source documents.
Should question and answer logs be treated as sensitive?
They should. Every logged answer contains text from the retrieved passages, so the log holds fragments of every confidential document anyone asked about. Restrict log access to the people who could read the most sensitive source, set a retention period, and remember that deleting a document from the index does not delete it from old logs.
What happens when a document is updated?
The old chunks for that file should be deleted and the new version chunked and embedded again. Leaving both versions in place means the model may cite an outdated clause next to the current one, and staff cannot tell which is authoritative.
Ready to Modernize Your Business?
Get your AI automation roadmap in minutes, not months.
Analyze Your Workflows →