Every CodeLynq run passes a security gate before its pull request opens. We were pleased with it, and we still are. But when we looked at what it actually covers, the answer was uncomfortable: it protects the code that is being changed, and most of a codebase is not being changed.
What a per-PR security gate sees
There are two checks on a run, described in four checks an AI run should pass. The self-review reads the diff. The security scan runs gitleaks, semgrep and the dependency audits on the repository the run changed, and feeds anything it finds back to the agent to fix.
Both are good at what they are for, which is keeping a new change from adding a problem. Three gaps follow from that design:
- Code nobody touches is never looked at. The admin endpoint written two years ago, the export job, the webhook handler from a contractor. If no task goes near it, no gate runs over it.
- The AI review only reads the change. A missing authorisation check is a fact about how an endpoint relates to the rest of the application. A reviewer looking at forty changed lines cannot see that the route next to them has none.
- Scanners match patterns. They are very good at a key in a config file or a known vulnerable dependency. They do not know that your invoices are meant to be visible only to their own customer.
And the world changes when your code does not. A dependency that was fine last month has an advisory this month, and nothing in a diff-based gate will notice until someone happens to touch that repository.
A security audit of the whole project
So we added a security check that reads everything. It clones every repository in the project, runs the same scanners over the full checkout, and then gives the whole codebase to a read-only AI agent with the scanner output alongside. The agent can read and search files; it cannot change them.
The brief is specific: missing authentication or authorisation on endpoints, injection of every kind including path traversal, SSRF, secrets in the repository, weak cryptography or token handling, unsafe deserialisation, CORS and CSRF mistakes, personal data logged or kept longer than needed, and vulnerable dependencies. It is told to prefer real, reachable problems over theoretical ones and to say how each could be exploited, and to confirm or dismiss what the scanners reported rather than repeat it.
Each finding comes back with a severity, a location and a recommendation. If the agent does not return its findings in a form we can read, the check fails and says so. It never records an empty result, because "nothing found" when nothing was checked is the worst answer a security tool can give.
Every week, without anyone asking
You can start a security check from a project's Checks tab at any time. The point, though, is that nobody should have to remember. With security monitoring turned on, a project gets a full security check every week. It is an optional add-on per project, billed monthly on the organisation's existing subscription, and only an organisation admin can turn it on or off. While it is on, the weekly checks are not charged to credit. If the plan itself lapses, scheduled checks pause and pick up again when it is active.
Nobody is waiting for a scheduled check, so the result goes to the organisation's admins as a notification, with the verdict and the number of new tasks.
Why only critical and high become tasks
Our go-live check files medium findings as tasks too. The weekly audit does not, and that difference is deliberate. A go-live check is something a person starts because they want to know everything that should be fixed before launch. A weekly scan runs unattended, and if it filed every medium, the backlog would fill with tickets nobody asked for. People learn to ignore a source of noise, and then they ignore the critical one too.
So only critical and high findings become tasks, as drafts with the problem, the location and the fix already written, ready for a run. Mediums and below stay on the audit report, where you can read them when you choose to. One audit files at most fifteen tasks, whatever it finds.
Weekly also means the same finding comes back every week until it is fixed. Each finding gets a fingerprint from its category, the kind of problem and the file, without the line number or the wording, both of which drift. A repeat is linked to the task that is already open instead of filed again. Once that task is closed and the problem returns, it gets a new one.
What it is not
- It reads code. It does not probe your running application, so it is not a penetration test and will not find a misconfigured load balancer.
- It does not run your test suite; that belongs to the go-live check.
- The scanners only run if they are available for your stack. When none apply, the check says so and the review relies on the AI alone.
- Like any AI review, it can miss things and it can be wrong. Findings are tasks for your team to judge, not verdicts to merge.
Both, not either
The per-PR gate and the weekly audit answer different questions. One asks whether this change makes things worse; the other asks how things are. If you only have the first, try one manual security check on your oldest project and see what is in the code nobody has touched in a year. More on how we handle code and data is on our security page.