The report was short: spec generation is broken. You open a task, ask for a technical spec, and get back a form with every field empty. No error, no warning, just nothing. To the person looking at it, the AI had looked at their task and had nothing to say.
The real cause had nothing to do with their task. The AI provider account the platform runs on had run out of credit, and the provider was answering every request with a clear message saying so. That message never reached a screen. Finding it meant someone testing the production key against the provider's API by hand.
How a clear error became an empty form
The code that drafts a spec caught every exception and returned eight empty strings, one per section. The endpoint passed those through with a 200 OK. Nothing was technically wrong with any single step.
The pattern was not limited to one function. Our AI service has many handlers that turn an error into an empty or default result, and for one kind of failure that is the right call: when the model returns something we cannot parse, an empty draft the person can fill in is better than a crash. For a provider outage it is exactly wrong. An outage is platform-wide, it will fail again on the next try, and the customer cannot fix it. Treating the two the same is how "the provider says we are out of credit" became "the AI did nothing".
Runs had a version of the same problem. The coding agent's command-line tool reports an exhausted account as ordinary result text, not as a failure, and then exits with an error code. We only read the field that names the kind of failure, so the explanation was thrown away and the run said "Exit code 1". Two of our first five production runs failed that way.
Telling an outage apart from an empty answer
Spec drafting, the new-task chat and the other AI features now raise a distinct "AI unavailable" error when the provider failure is one the customer cannot fix: out of credit, a rejected key, rate limiting. Instead of a blank form, the person sees a sentence about what is wrong and what happens next. An answer we genuinely cannot read is no longer stored as a finished spec either; the draft is marked failed with "The AI's answer could not be read. Try again."
For runs, we keep the agent's result text even when it does not call itself a failure, and prefer it over a bare exit code. Every run error then goes through a classifier that maps the raw text onto a kind: out of credit, rate limited, repository credentials rejected, token cannot push, branch missing, task too large, timed out, and so on. Each kind comes with a summary, the next step, and whether retrying the same run could plausibly work. The run page shows that instead of the raw string. When nothing matches, it says the run failed and points at the log rather than guessing; a wrong next step costs more time than none.
Making it visible to us before a customer reports it
The second half of the fix is for our side. We added platform incidents: a failure in a dependency we run, the AI provider, payments, invoicing or hosting, is classified and recorded as one row per kind of failure, not one per occurrence. A credit exhaustion produces a failure for every attempted call, hundreds a minute, and a row for each would be a log, not a dashboard. Each row counts its occurrences and remembers when it was first and last seen, so it answers what is broken, since when, and whether it is still happening.
Active incidents show as a banner above every tab of our internal admin panel, which refreshes every minute. An incident clears only when a later call to the same service succeeds. We chose that over clearing on a timer, because a failure that quietly disappears because nothing happened to retry is worse than one that lingers a little.
Two rules shaped the code. Recording a failure must never cause one: the recording runs inside error handlers, so it swallows its own errors. And the incident classifier reuses the same vocabulary as the run error classifier, so the admin banner and a customer's run page cannot disagree about what went wrong.
What this does not do
- Classification is string matching on provider messages, because those providers do not give us typed errors. A provider that changes its wording, or a failure we have not seen yet, lands in the generic bucket until we add a rule.
- The incident banner is for our own admins. Customers see the error on the thing they were doing, not a platform status page.
- Organisations that bring their own AI provider key see their own account's errors in the same words, but those are theirs to fix, not ours.
If you build on top of an AI provider, the check worth doing this week is simple: search for every place an exception becomes an empty result, and ask which of those could be the provider saying no. We wrote about what else a run checks before a pull request opens in four checks an AI run should pass.