"Should we fine-tune?" gets asked at every kickoff. Almost always, the honest answer is "not yet, and possibly not at all." Here is the decision pattern we use at Accolades, mapping three different techniques to three different problems.
Three techniques, three jobs
Prompt engineering solves behavior. You want the model to follow a specific persona, format outputs a specific way, or reason in a specific structure. The best prompt is the smallest prompt that hits your eval bar. Iterating on prompt + system instructions handles the vast majority of "the model is not doing what we want" complaints.
Retrieval-augmented generation (RAG) solves knowledge. You want the model to answer using information it could not possibly have memorized at training time — your private documents, your knowledge base, last week's policy update. RAG injects that context into the prompt at inference time. The model itself does not change.
Fine-tuning solves style and patterns that prompts cannot scale. You have thousands of examples of "what good looks like" in your specific domain — call transcripts, code in a niche framework, formal contracts — and you need the model to generate that style by default without a massive prompt. Fine-tuning teaches it.
When to combine them
Production systems almost always need two of three. Most commonly: prompt engineering to set the contract, plus RAG to ground the answer in your data. Fine-tuning enters the picture later, when prompts have become unmanageably long because they are doing too much pattern-matching.
The order matters. Start with prompt engineering against the strongest available foundation model. Add RAG once your accuracy plateau is "the model does not know our data." Only consider fine-tuning when you have shipped, you have an eval suite, and the cost case for shorter prompts at scale is real.
Most projects never reach step three. That is a feature, not a bug — every layer you add is a layer you have to maintain.