When we looked at our first five production runs, every one of them had failed. None of them failed because the AI wrote bad code. None of them got far enough for the code to matter.
The list is unglamorous: a repository whose default branch was not called main, a token that could not clone the repository, a token without permission to open a merge request, and twice a run that ended in "Exit code 1" while the log underneath plainly said the AI account was out of credit.
That matters more than it looks. Someone trying an AI coding tool for the first time does not read a failed run as "my token is missing a scope". They read it as "the AI does not work", and that impression decides whether they come back.
Two kinds of failure, both avoidable
Sorting the failures showed two groups, and neither needed a better model.
- Knowable before the run was queued. No repository connected, the wrong default branch, credentials that cannot push. All of this can be checked in seconds. Instead, a worker discovered it minutes later, and reported it in words our error classifier did not recognise.
- Known after the failure, but not shown. The agent's command-line tool reports an exhausted account as ordinary text and then exits with an error. We only read the field that names a failure, so the explanation was dropped and "Exit code 1" was all anyone saw.
The second group is its own story, about provider errors turning into empty results. This post is about the first.
Asking the remote, not the metadata
A repository connection now gets checked the way the first run will use it: against the real remote, with the real token, in three questions.
- Can we reach it? If not, the message says why in terms of the fix: the URL is wrong, the repository was not found or the token has no access to it, the credential was rejected, or an organisation needs SSO authorisation for the token.
- Does the branch we would run against exist? If the stored branch is missing, the check tells you what the remote says its default branch is and offers a button to use that one. An empty repository passes, because the first run creates the first commit.
- Can this token push? This is the one that cannot be read from a token's scopes. Branch protection, SSO enforcement and read-only deploy keys all produce a token that looks fine until the push. So we ask git: a dry-run push that deletes a branch which does not exist. The remote runs its full permission check and then nothing is applied. Nothing is ever written to your repository.
It runs when you press "Check connection" on a repository, and again automatically when you change the default branch, because a mistyped branch otherwise only shows up as a failed run. Every check is listed, not just the first failure: a token that can clone but not push is the one people are most surprised by, and it would be invisible behind a green "reachable".
A bug in how we found the default branch
While building this we found that our own default-branch detection was wrong. It listed the remote's branches and took the first line. That list is alphabetical, not default first, so a repository with a branch called bisect or develop got the wrong answer. The check now asks the remote directly which branch its HEAD points to.
Refusing a run that cannot work
Starting a run already checked the plan's limits, concurrency and whether the task was approved. It did not check whether the project had a repository at all, so the run queued and died later with a generic failure. That is now refused immediately, with a message saying to connect a repository first. The rule we took from it: if a condition can be known when someone presses the button, say so then, and do not let a worker find out minutes later.
What the check does not cover
- Pushing and opening a pull request are different permissions. The check proves the token can push; it does not prove the token may open a pull request or merge request through the provider's API.
- The full repository check is not run before every run. Starting a run only checks that a repository is connected. If a token is revoked next week, you find out from the run, which now says the repository rejected the credentials and how to reconnect.
- Each git call has a 30 second budget, so a very slow remote can fail the check while a run would have succeeded.
If you are evaluating an AI coding tool, run the connection check before judging the first result. And if the first run still fails, read the reason before reading the code. Connecting a repository is covered on GitHub, GitLab and Bitbucket, and what onboarding looks like is on implementation.