The comparison is their previous large model
GitHub described the new secret-scanning classifier today. In an online A/B test against the pipeline that called a large model, repository owner-weighted alert acceptance was 11.5% higher. Weighting by owner is meant to keep customers who resolve alerts in bulk from skewing a plain average. Median post-push scan time was 86% lower. Serving cost for the compact classifier was more than 98% lower than remote large-model calls.
On the same public traffic, an offline comparison found 13% higher precision and 19% higher recall.
[1]
Regexes collect candidates. A small model judges them
Scanning stays two stages. Regexes first find credential-like strings at high recall. A fine-tuned ModernBERT then reads the candidate with the file path, variable names, and surrounding code, and estimates whether it is a real secret. The model is exported to ONNX with 8-bit quantization.
Faster post-push scans are what made it practical to extend generic secret detection to push protection, so a result can come back before the change enters the repository. GitHub says the model does not replace the regexes.
[1]The training set includes synthetic hard cases
The training data includes product feedback with human review, regex candidates, and synthetic examples aimed at documentation placeholders, test fixtures, and checksums. Synthetic rows have to pass deterministic checks and a model quality review. The threshold is chosen offline, then checked on production-like traffic.
The gains are against GitHub's own previous pipeline. The post does not give an absolute false-positive rate, and it does not cite an outside audit.
[1]