How to evaluate a fine-tuned model before it reaches production

A finished run is not a result

The loss curve went down. The job completed. There is a checkpoint. None of that tells you whether the model is better at the thing you needed it to do.

Training loss measures how well the model predicts data it was trained on. It is a diagnostic for whether training worked mechanically, not a measure of capability. A model can achieve excellent training loss by memorising the dataset and be worse than the base model on anything else.

The only question worth answering is whether this version outperforms the version it would replace, on data neither of them has seen, on tasks that resemble what users will actually send.

Building a held-out set worth trusting

The evaluation set is the most valuable artefact your team will produce and usually gets the least attention.

It must be genuinely held out — never used in training, not even for a validation pass that influenced hyperparameters. Contamination is subtle and quietly common: a dataset that was deduplicated after the split, an example that appears in two source systems, a paraphrase that survived a naive deduplication.

It should reflect real traffic. If your evaluation set is composed of clean, well-formed examples and your users send fragments, typos and mixed-language input, you are measuring a task nobody performs.

It should include the cases you are afraid of: the edge cases, the ambiguous inputs, the ones where the right answer is to decline. A set made only of questions the model can answer measures nothing useful about failure behaviour.

Metrics that reflect the actual task

The right metric depends entirely on what the model does, and the generic ones are usually the wrong choice.

• Classification and extraction — precision, recall and F1 per class. Report per class, because aggregate scores conceal complete failure on your rarest and most important category.
• Structured output — schema validity rate first, then field-level accuracy. A response that is 95 percent correct but does not parse is a total failure to the calling system.
• Retrieval-grounded answers — citation correctness and groundedness. Does the cited source actually support the claim?
• Open-ended generation — pairwise human preference against the incumbent. Automated scoring correlates poorly with usefulness here, and pretending otherwise wastes months.
• Refusal behaviour — does the model decline when it should, and only when it should? Both directions of error matter.

Alongside quality, always record latency and cost per request. A version that is two percent better and forty percent slower is usually not an improvement, and that trade-off is invisible if you measure quality alone.

Compare against the incumbent, not against zero

A score in isolation is not decision-useful. The question is never 'is this model good' but 'is this model better than what is serving right now'.

Run both on the identical evaluation set, in the same conditions, and report the difference. Then look specifically for regressions. A version that improves the aggregate by four points while collapsing on one category is frequently a worse deployment than no change at all, and aggregate reporting hides exactly that.

Segment the comparison by input type, by language, by length, by user cohort — whatever dimensions matter for your application. Regressions concentrate in segments, and averages are very good at concealing them.

Promotion should be a deliberate act

The most consequential control in the whole pipeline is the simplest: a training run finishing should never place a model in front of users.

Promotion to a production endpoint should require a person with the authority to do it, and that approval should be recorded alongside the evaluation it was based on. Not because the person adds technical insight, but because it creates a point at which someone is accountable and a record of what was known at the time.

This is standard practice in regulated domains and unusual everywhere else. It costs almost nothing and it is the difference between being able to answer 'why is this model in production' and not.

Keep the failures

A version that did not beat the incumbent is a result, not a waste. Keep it in the registry with its configuration, its dataset version and its evaluation.

Six months later, someone will propose the approach that failed. If the negative result is recorded with enough detail to be reproduced, that conversation takes ten minutes. If it is not, it takes another three weeks of GPU time to rediscover.

This is the practical argument for lineage: not compliance, but not repeating your own work.