Skip to Content

Quality gates for AI code: tests, review and security scan

The checks a run goes through before it opens a pull request - your verify command, a self-review, a security scan and documentation updates - and which apply on self-hosted runners.

A run does not open a pull request the moment the code is written. It first checks its own work, and fixes what it finds. You decide which checks apply per project.

Open the project and go to Settings → Context & conventions. The checks are under Verification gate. Changing them requires being an organization admin or the project's owner. Click Save Settings when done.

Verify command

Verify command is the command that proves the change works - your tests, build or linter. For example npm test, pytest -q or make ci.

  • After making its changes, the run executes this command in the repository (or in its subdirectory, if one is set).
  • If it fails, the AI gets the output and tries to fix the problem, then runs the command again.
  • Fix attempts on failure sets how many tries it gets (0 to 5, default 2).
  • If the command still fails after the last attempt, the run is marked Failed - you never get a pull request that breaks your build.

With several repositories, the command runs in each repository the run changed.

Leave Verify command empty to let the AI check its own work during the run instead. A verify command is strongly recommended: it is the most reliable way to get pull requests that pass CI. Analyse repository (see Projects) proposes one from your repository.

Automatic fixes from alerts only run when the project has a verify command.

Self-review before PR

With Self-review before PR on (the default), a reviewer agent reads the change before the pull request is opened.

  • Blocking problems are sent back and fixed automatically, and the change is verified again.
  • Smaller remarks are listed in the pull request description for your reviewer.
  • If a blocking problem is still open after the fix attempts, it is listed in the pull request description as unresolved, so your reviewer sees it.

Security scan before PR

With Security scan before PR on (the default), the changed code is scanned for security problems, leaked secrets and vulnerable dependencies before the pull request is opened.

  • Leave Security scan command (optional) empty to use the scanners that are available: semgrep, gitleaks, pip-audit and npm audit. If none is available, the scan is skipped.
  • Or enter your own command; a non-zero exit code counts as findings.
  • Findings are sent back to the AI to fix. Findings that remain after the fix attempts are flagged in the pull request description.

On self-hosted runners the verify command, the self-review, the security scan and the docs step do not run before the pull request; the architecture rules and the fixes for failing CI do. See Checks on self-hosted runs.

Generate docs in PRs

With Generate docs in PRs on (off by default), the run also updates the CHANGELOG and the documentation affected by the change, in the same pull request. If this step fails, the run continues without it.

Auto-iterate until green

Auto-iterate until green is not a check before the pull request, but after: when CI fails or a reviewer requests changes, CodeLynq starts a new round to fix it. See The CI feedback loop.

Architecture rules

If the project has architecture rules for a repository (see Architecture map and rules), a run also checks that its change adds no dependency the rules forbid. With On runs set to Fix, the run sends a broken rule back to the agent and verifies again afterwards. The pull request's Structure changes section lists what remains, together with what else the change did to the structure.

Which checks to use

  • Always set a Verify command if your project has tests or a build.
  • Keep Self-review before PR and Security scan before PR on; turn one off only if it gets in the way for this project.
  • Each fix attempt takes extra time and model usage, so a very high Fix attempts on failure makes failing runs more expensive. The default suits most projects.

Test it on a task from your own backlog

Send us one real task. We run CodeLynq on your repository and you review the pull request. Free, within 5 working days of approving the spec.