What RAG actually does

Retrieval-augmented generation is three things at once, and that is exactly where the choices begin. It is an index of your documents, a search engine on top of it, and a language model that gets the retrieved chunks of text as context before it answers. Which of those three you outsource determines how much you spend and how much you hand over.

The temptation is to start with the model. That is the wrong side.

What RAG actually does

A language model knows what was in its training data, and nothing more. Ask it about a document written last month, and it makes up something plausible. That is not a bug you can prompt away, that is the architecture.

RAG reverses the order. Your question goes to a search index first. That returns the five or ten most relevant fragments from your knowledge base. Only then does the model formulate an answer, with those fragments alongside. The model is the writer, not the source.

That sounds like a detail. It is the whole point.

The index is the real work

Before you can look anything up, you have to have something to look in. That means chopping your documents into pieces, converting each piece into a vector, and putting those vectors into something that searches fast enough.

That is where most RAG projects founder. Not at the model, not at the prompt, but at the chunking. A PDF with a table that runs across three pages becomes unusable at every split. A manual with numbered steps that you cut in half in the middle of step 4 produces a fragment that nobody understands, the model included.

You can build this yourself. `langchain`, `llama_index`, a vector store of your choice. You get full control over chunk size, overlap, metadata and retrieval strategy. You also get a weekend of work per document type that looks just different enough from the previous one. Self-hosting sounds wonderful, until it turns out you spend ninety percent of your time fixing parsers instead of asking questions.

What you outsource and what you keep

The honest trade-off is this: a hosted RAG service handles the index, the chunking, the vector store and the retrieval. You supply documents and questions. That works, usually within an hour, and it costs you a subscription plus the certainty that someone else can read your documents.

Running locally means the opposite. LM Studio, Ollama, or your own stack on a server with enough RAM. Your data stays where it is. You pay in setup time, in memory, and in the kind of errors that only you can debug.

There is a third option in between that people often skip: your own server, with a hosted embedding API. The documents stay inside, the compute for converting to vectors comes from outside. That is a reasonable split if your machine has no GPU. It is also a split you have to explain to everyone whose documents are in it.

The model is the easiest choice

Here is the good news. For the generation step, it matters less than you think which model you pick. A 7B model that gets the right fragments answers better than a 70B model that has to guess.

That means you can run a local model on hardware you already have. A machine with 16 GB RAM can handle a 7B model in Q4 quantization. If you want 13B or 34B, you are talking about 32 GB or more, and about a GPU to keep up the pace.

The cost is not in the model. It is in the memory you have to reserve, and in the time it takes to generate an answer if you have no GPU. Asking one question of a 7B model on CPU is doable. Asking twenty questions is a lunch break.

Retrieval is where it goes wrong

You can have a perfect index and still retrieve the wrong fragments. That happens when your question and your documents are written in different languages — not Dutch versus English, but the language of a search query versus the language of a manual.

Embeddings solve this partly. Partly. A question like "why won't my container start" does not always look, in vector terms, like a fragment that begins with "systemd unit dependencies". You can improve this with hybrid search: combining vectors with classic full-text search. That is more work and it delivers more than any model upgrade.

There is a cheap trick you should always apply: make the model cite. Ask it not just for an answer, but for the fragment the answer is based on. If you get an answer without a source, the model drew on its general knowledge, and that is exactly what you did not want.

What it actually costs you

A local RAG stack on a Linux server is not an exotic project. You have enough RAM, enough disk for the vectors, and a model you download once. The software is open source and the parts can be swapped out individually.

The bill comes in maintenance. Documents change. A manual gets a new version. A folder of notes grows. Your index has to be redone, and that is not a button you press once.

With a hosted service, that is their problem. With your own setup, it is a cronjob you write yourself, and a cronjob that quietly stops without you noticing. You only notice when someone asks a question about a document that was updated three months ago and the answer still cites the old version.

What it depends on

Start with your documents, not your model. If there are twenty of them, a folder of text files and a simple index is enough. If there are twenty thousand, the quality of your parser is the entire project.

If there is anything in them that must not leave the building, half your options fall away and the rest get more expensive. If it may go outside, a hosted service is often better than what you cobble together yourself in a weekend.

And if you build it yourself: count on a week for the first version that works, and on a month for the version that keeps working when you are no longer looking at it.