Your model’s 90% confidence interval is probably 85%

A price estimate that comes back as a single number is easy to disbelieve and impossible to check. A range is more useful — but only if the range means something. If a model says it is 90% confident the price falls between two figures, that claim is either measured or it is decoration.

Building Carmark, a used-car valuation platform, we trained a CatBoost model under a MultiQuantile loss at alpha 0.05, 0.50 and 0.95, so the lower bound, median and upper bound are learned together in one model rather than three independent ones. The median error was good: 5.97% median absolute percentage error on a held-out set of 30,786 listings. The intervals were not.

Quantile regression under-covers, almost always

Asked for a 90% interval, the raw quantile output contained the true price 85.24% of the time. Not catastrophic, but wrong in a specific and dangerous direction: the band was too narrow. Roughly one listing in seven fell outside a range the model presented as near-certain.

This is the standard failure mode, not a bug we introduced. Quantile regression optimises pinball loss on the training distribution; nothing in that objective forces the empirical coverage of the resulting interval to match its nominal target on unseen data. Finite samples, distribution shift between training and serving, and the optimiser’s preference for tighter loss all push the same way — toward intervals that look more confident than they have earned.

The perverse part is that under-coverage flatters the model. A narrow band demos better. It reads as precision. Nobody notices the failure until a real user is negotiating against a number that was never as reliable as it appeared.

Measure coverage before you trust it

The check is not complicated, and it is the step most teams skip. On a held-out set the model has never seen, count how often the true value falls inside the predicted interval, then compare that fraction against the nominal target.

• Nominal target: 90%, from the 0.05 and 0.95 quantiles.
• Empirical coverage, raw: 85.24%.
• Gap: 4.76 percentage points of overclaimed confidence.

That number is the whole argument. Without it you are asserting a confidence level; with it you know precisely how far off you are, and in which direction.

Conformal calibration, fit per price band

The fix is a conformal calibration layer. Rather than retraining the model or tuning the loss, you measure the residual behaviour on held-out data and widen the intervals by exactly the amount that data says they were short.

We fit the correction separately per price band, and that detail matters. Error is not uniform across the ladder — a single global adjustment would over-widen the dense middle of the market to compensate for the sparse ends.

Error is lowest through the 3M–10M band at 4.39%, where listing density is highest. It widens to 10.11% below 1M, where condition and history dominate price more than specification does, and to 6.80% above 10M, where only 1,400 comparable listings exist. A band-aware correction respects that structure.

The result

After calibration, empirical coverage lands at 90.39% against the 90% target — a gap of 0.39 percentage points, and erring slightly wide rather than slightly narrow. The median relative band width is 35.8%.

When the model says it is 90% confident, it is right 90% of the time. That is a claim you can put in front of a user, because it is a claim you measured.

The intervals got wider. That is the trade, and it is worth naming honestly: the calibrated model looks less impressive in a screenshot than the uncalibrated one did. It is also the only version of the two that a buyer can actually negotiate against and a seller can price to.

Ship the calibration with the model

One serving detail is easy to get wrong. The conformal deltas are not part of the model file — they are a separate artifact, derived from a specific held-out set at a specific point in time. If the two drift apart, the model quietly goes back to under-covering while still reporting a 90% interval.

We ship the deltas in a sidecar beside the model artifact, versioned together. A previously trained model keeps serving correctly as new features are added, because the correction that makes its confidence claim true travels with it.

What to take from this

1. If your model produces intervals, measure their empirical coverage on held-out data. Assume they under-cover until you have that number.
2. Fit the correction per segment where accuracy varies by segment. A global adjustment pays for the weakest slice everywhere else.
3. Version the calibration with the model. A correction that goes stale is worse than none, because the interface still promises 90%.
4. Accept the wider band. Narrow intervals are the easiest metric to fake and the fastest to lose a user’s trust.

Calibration is not sophisticated work. It is a measurement and a correction, perhaps a day of engineering. But it is the difference between a model that performs well and a product that can defend the claim printed next to its output.