The pitch writes itself: an alert fires at three in the morning, an agent reads the stack trace, and by the time anyone wakes up there is a pull request with the fix. We built that. Most of the work was not in the part that writes the fix. It was in deciding when it should not start, because a monitoring system can send a hundred alerts in a minute and each one of them is a request to spend money on your behalf.
An alert becomes a bug task
You add an alert source to a project and get an ingest URL with a token in it, shown once. Your monitoring tool posts its JSON alert to that URL. We authenticate the token and turn the alert into a bugfix task: the title becomes the task title, the message and stack trace go into the bug report fields, and the full payload is attached so the agent has everything the monitor knew. If the source is linked to a repository, the task points there.
By default that is where it stops. The task lands as a draft, like any other bug report, for someone to look at. Each project has an Alerts tab that lists the tasks your monitoring created and the latest run on each, so you do not have to search the task list for them.
The same alert, once
A failing job alerts every few minutes until somebody fixes it. Without deduplication that is a new task every few minutes. Each alert gets a fingerprint: the payload's own fingerprint field if it sends one, otherwise a hash of its title, its service or environment, and the first line of its stack trace. If a task with the same fingerprint was created within the source's window, which defaults to 24 hours, the alert is folded into that task and nothing new is created.
On top of that each source accepts a bounded number of alerts per hour. Over the limit, alerts are quietly dropped rather than answered with an error, because a monitor that is told to retry makes the flood worse. The limit is there for a leaked token as much as for a noisy monitor.
Severity decides priority
Monitoring tools disagree on how to say how bad something is. We read severity, level or priority from the payload, or the severity label the way Alertmanager and Grafana send it, and map the words to the four task priorities: critical, fatal or sev1 become Critical; error, major or p2 become High; warning becomes Medium; info, notice and debug become Low. An alert whose severity we do not recognise is Medium.
When it is allowed to run by itself
A source can be set to auto-run. It is off unless you turn it on, and even then an alert only starts a run when every one of these holds:
- The source has auto-run on and an hourly cap above zero.
- The alert is High or Critical.
- The project has a verify command. If nothing can fail a bad fix, we will not open a pull request for it unattended.
- The hourly cap is not reached. It counts the runs started on alert-created tasks in the project in the last hour, and defaults to three.
- The organisation's run limits allow another run.
- Your spend controls allow it without asking. A run that would need someone to approve its cost is exactly the run that should not start while nobody is watching, so "needs approval" stops it as firmly as "refused".
When a guard says no, the task stays a draft and the response to the monitor says which guard it was. When all of them say yes, the run does what every run does: verification, self-review, a security scan, and a pull request. It never merges. We wrote about why the merge button stays yours, and an alert at three in the morning is the strongest case for it.
The guard that refused nothing
From the first release until early October, the severity guard did nothing. The first version of the intake set the priority to Critical when the severity was critical, fatal or error, and to High for everything else. Info, warning, a missing severity: all High. Since the auto-run guard lets High and Critical through, it passed every alert it was ever shown.
Nothing flagged it, because each piece was correct on its own. The mapping was a reasonable first guess and the guard checked exactly what it said. We found it while writing the help article for alerts and trying to explain which alerts would not run by themselves. The honest answer was none. The mapping now covers all four priorities, including the severity label, and unknown severity lands on Medium, which is the safe side of a guard that exists to keep noise from spending money.
The lesson we took from it: when you add a guard, write down one input it should refuse, and check that it does.
What it does not do
- There is no adapter per monitoring tool. The fingerprint uses top-level title, message, service and stack fields. If your tool keeps those elsewhere, send a fingerprint field, or different alerts will look the same to us and fold into one task.
- A deduplicated repeat is dropped. It does not add a counter or a new sample to the existing task.
- A "resolved" notification is not recognised as closing anything. It is treated like any other alert, and the task stays open until someone closes it.
- An auto-run starts from the alert, not from an approved spec. The bug report steers the agent to reproduce the problem before fixing it, but nobody has agreed what the fix should be until they review the pull request.
How we would roll it out
Connect one alert source with auto-run off and leave it for a week. Look at the Alerts tab: how many tasks, how many folded repeats, which priorities. If the priorities match what you would have chosen and the project has a verify command you trust, turn auto-run on with the default cap. For problems that need a person to look at production first, a troubleshoot session is often the better starting point than a fix.