Fine-tuning vs prompt engineering: choosing with ROI in mind

The pattern we see every week

A company finds that GPT-4 consistently misclassifies support tickets in their vertical domain. The technical team's first reaction: «we need fine-tuning». Six weeks later, 40,000 annotated examples, €3,000 in compute costs, and the results are worse than a well-written prompt.

We've seen this sequence at least ten times in the last twelve months. Fine-tuning isn't wrong — premature fine-tuning is.

---

What actually separates the two approaches

Prompt engineering means instructing a base model at runtime by including examples, instructions, format constraints, and reasoning chains in the context. It does not modify model weights.

Fine-tuning means updating model weights on a specific dataset. The model «learns» patterns it no longer needs to receive via prompt.

The operational distinction:

| Dimension | Prompt Engineering | Fine-tuning | |---|---|---| | Upfront cost | Low (hours/days) | High (dataset, annotation, compute) | | Ongoing cost | High (long input tokens) | Low (short prompts) | | Latency | Depends on token count | Generally lower | | Updatability | Immediate | Requires re-training | | Data required | Zero | Minimum 500–1,000 examples |

---

When prompt engineering is enough (which is often)

If the target behavior can be described in natural language with in-context examples, fine-tuning is overengineering.

Concrete example: structured data extraction from PDF invoices. A few-shot prompt with 5–10 examples reaches 90–95% accuracy on standard invoices. Fine-tuning might push that to 97%, but the delta doesn't justify the costs unless volume exceeds 50,000 documents per month.

# Few-shot prompt example
EXTRACT the following fields from the invoice in JSON format:
- invoice_number
- issue_date
- total_amount
- supplier_vat_number

Example 1:
Input: "Invoice no. 2024/001 dated 15/01/2024 – Total £1,220.00 – VAT 12345678901"
Output: {"invoice_number": "2024/001", "issue_date": "2024-01-15", "total_amount": 1220.00, "supplier_vat_number": "12345678901"}

[add 4–9 similar examples with format variations]

Now extract from the following invoice:
{{input}}

This approach takes an afternoon, not a month.

---

When fine-tuning actually changes the numbers

There are three scenarios where fine-tuning makes economic sense:

1. High, stable token volume

If you're running GPT-4o with 2,000-token prompts across 500,000 calls per month, you're spending roughly €5,000/month on input tokens alone. A fine-tuned model with 200-token prompts cuts that cost by 70–80%. The training cost break-even lands in 2–3 months.

Quick formula:

break_even_months = finetuning_cost / monthly_token_savings

# Example:
# finetuning_cost = 8,000 €
# monthly_token_savings = 3,500 €
# break_even = 8,000 / 3,500 ≈ 2.3 months

2. Style or format that can't be described via prompt

A model that must generate technical documentation in an internal company style — with specific naming conventions, proprietary paragraph structure, precise legal tone — cannot be adequately instructed through text instructions alone. Training examples are required.

3. Latency-critical applications with long context

If your application requires sub-500ms responses and the necessary context is 3,000+ tokens, fine-tuning lets you eliminate most of that context. With smaller fine-tuned models (Mistral 7B, Llama 3 8B), p95 latency can drop from 2–3 seconds to 300–400ms.

---

The operational decision path

Before opening a fine-tuning project, answer these questions:

1. Do you have at least 500 high-quality annotated examples? If not, stop. Fine-tuning on less data almost always degrades the base model.
2. Is the target behavior describable via instructions? If yes, test a structured prompt with chain-of-thought first.
3. Does monthly volume exceed 100,000 calls? Below that threshold, token savings rarely offset fixed costs.
4. Does the domain change frequently? If business rules shift every week, a fine-tuned model becomes obsolete before it's profitable.

if annotated_samples < 500:
    return "prompt engineering"
elif monthly_volume < 100_000 and style_describable:
    return "prompt engineering"
elif latency_critical or monthly_volume > 500_000:
    return "evaluate fine-tuning"
else:
    return "benchmark both, then decide"

---

An underrated hybrid: RAG + prompt engineering

Many cases that appear to require fine-tuning are solved with Retrieval-Augmented Generation: relevant context is retrieved from a vector knowledge base and injected dynamically into the prompt. Setup costs comparable to fine-tuning, immediate updatability, no re-training.

When the problem is «the model doesn't know our domain», the answer is almost always RAG. When the problem is «the model doesn't behave the way we want», the answer might be fine-tuning.

---

Operational take-aways

• Always start with prompt engineering. It's reversible, fast, and measurable.
• Consider fine-tuning only after quantifying ROI with real numbers: dataset cost, training cost, expected token savings, time horizon.
• Calculate break-even before the project starts, not after.
• RAG is often the overlooked third option — evaluate it before investing in training.
• Open-source models (Llama 3, Mistral, Phi-3) dramatically lower fine-tuning costs: an 8B parameter model on an A100 GPU costs roughly €2–4/hour on cloud infrastructure.

---

If you're weighing which path to take for an internal AI project or a client engagement, Evviva Group's team can run a rapid feasibility analysis before the budget is committed.