What to log when the model is not yours

The first time an upstream model changes behaviour without an announcement, you discover exactly how good your logging is. In my experience the answer is usually "not good enough, by one field".

The reason is that people log what they need to debug their own code, and the thing they actually need is evidence about somebody else's. Those require different records.

The fields that turn out to matter

Request id, timestamp and status code are table stakes. Beyond those, five fields do the real work:

The model identifier the provider returned, not the one you sent. These differ more often than you would like. Providers alias names, route to variants under load, and silently promote a version. If you only log what you requested, you cannot distinguish "the model changed" from "our prompt changed", and that is the single most common question you will need to answer.

The full parameter set as actually sent, after all your defaulting has been applied. Defaults change when someone edits a config, and a behaviour shift that correlates with a deploy rather than an upstream change is a completely different investigation.

Latency, split into queue and generation if the provider exposes it, and total wall clock if it does not. Latency is the earliest signal of an upstream change — it moves before quality does, and it moves in a way you can chart without human judgement.

The raw error body, not a normalised error class. Everyone maps upstream errors into their own taxonomy, which is correct for handling and disastrous for diagnosis, because the mapping discards exactly the text that tells you what changed. Keep the original string.

Whether you were billed. This is the field people omit and later want most. It separates "the request was refused before inference" from "the request ran and produced something unusable", and those have opposite responses.

Two things worth logging that are not fields

A periodic canary. Run a fixed prompt with fixed parameters on a schedule and keep the outputs. It costs very little and it is the only way to notice gradual drift, which is otherwise invisible because no single user complains about it — they just quietly find the tool worse than they remember. A canary turns "it feels different lately" into a diff.

A sample of real outputs, with consent and a retention limit. Aggregate metrics will not tell you that the model started adding a warm colour cast. Someone has to look. Keep this small and delete it on a schedule; the point is a spot check, not a corpus.

Where the line is

It is worth saying what not to do, because the natural failure mode is to log everything and then never look.

Do not keep user prompts longer than you need for the canary comparison, and be explicit in your privacy policy about what you keep. Do not build a dashboard before you have had the first incident — you will build the wrong one. Do the canary and the raw error body first; those two have paid for themselves every time.

The part that faces the user

All of the above is internal. There is an external half, and it is the one that determines whether users trust you when something changes.

If your interface shows the model identifier on the output, then your users have the same evidence you do. When someone says results got worse last Tuesday, they can tell you which model they were on, and you can check it against your canary rather than debating impressions. It converts a support argument into a lookup.

It costs about an hour to add and it is the highest-leverage hour available in this category of product. On the interface I run, the identifier the API returned is printed under each result with the tier, the delivered size and the elapsed time — which is simply the user-facing projection of the log fields above, and it has settled more disputes than the dashboard has.