A false-positive corpus for secret scanners

Every test set a secret scanner is built against is made of secrets. Almost none is made of the boring output the scanner will actually spend its life reading. So this is the other half: 110 formats of ordinary log, build and CLI output, 784 non-blank lines, and not one credential anywhere in it. Point a scanner at it and everything it reports is a false positive.

How well my own redactor does on it

A corpus written beside a tool flatters that tool unless the score is published with the failures in it. So here is mine, measured on every build, both halves, misses named.

That is the whole score, and the interesting part of it is what is still wrong. The same run reports 2 misses and 2 over-reports on the credential half:

• cloudsql-iam-dsn — a email
• hard-capability-url — a capability url segment, from the hard tier: a credential with no recognisable shape, which no shape-based rule can reach
• ansi-docker-compose-logs, line 6 — user=app
• ansi-pytest-npm-publish-failure, line 4 — os.environ[

Re-measured on every build from the same detector table the browser tool, the npm package and the kit are all compiled from, so this page cannot drift away from the thing it describes. If it ever shows a number I would rather not publish, that is the number.

When a credential ends on the edge of its own alphabet

The tier of the corpus I keep coming back to is the one that catches a mistake a rule makes at its own boundary: it writes down the alphabet a credential is drawn from, and then bounds the match with something that alphabet is allowed to contain.

The classic form is a trailing \b. An AWS secret access key is drawn from [A-Za-z0-9+/=]; +, / and = are not word characters, so when the key is the last thing on the line the word boundary cannot match and the key is not reported — inside the exact assignment the rule exists to catch. The same shape appears with a hyphen after [A-Za-z0-9_-], which is how OpenAI project keys and GitLab tokens go missing. Cousins of it turn up wherever a rule is written: an allowlist written ^true|false|null$ whose anchors bind to the outer branches only, so any token with false inside it is discarded; an allowlist for a legacy token shape that is unanchored, and a modern prefix that contains that shape; an unescaped dot in a skip list, so the extension it means to ignore is scanned and a longer one is skipped.

The failure mode they share is what makes them worth a corpus. Nothing goes wrong on screen. The scan exits zero and calls the file clean, and the only way to see it is to have material where the credential sits on the edge of its own alphabet on purpose. That is what the true-positive corpus’s alphabet-edge tier is: a section for each of these shapes, each holding a credential of exactly the shape its vendor documents, ending on exactly the character that is legal and unhandled. Adding it cost me one of my own: my redactor read api_key: but not confluent_key:, because bare key was not in its keyword list and the prefixed form was never tested. Fixed in the same wake, and the fix is in the free tool, the npm package and the kit because they are all built from one source.

A roster only knows the vendors that already shipped

My redactor carries a hand-written list of vendor prefixes — ghp_, glpat-, gsk_, xai-, forty-odd of them, each kept tight to the shape its vendor publishes. That list is the most useful rule in the tool and it has a defect no amount of care removes: it can only ever know vendors that already shipped. An API launched last week is invisible to it, and so is every internal gateway inside every company, which is where a great many real credentials actually live.

The corpus said so out loud. Its hard tier — the credentials scored apart because no shape-based rule can reach them — held a token in a company's own scheme, acme_live_ and then the entropy, and my tool had missed it since the day the tier was written. I had been reading that as a fact about the world. It is not: it is a fact about reading the roster instead of the convention. Stripe's sk_live_ / pk_test_ shape has been copied by hundreds of APIs, and it is identifiable with no vendor name in it at all — one lowercase slug, an environment word, then the random tail. A rule that reads that shape catches a key from a vendor nobody has heard of.

Which is where the corpus stops being optional. A rule whose whole purpose is to fire on names it does not know is one loosened character from claiming every compound identifier in your config file, and the only honest way to ship one is to run it over material that is deliberately full of near misses first. Three narrowings came out of that run and each is a line in the tests: the slug is exactly one snake segment, so aws_instance_prod_id has no word boundary to start a match on; dev is left out of the environment set as too common in ordinary names; and the tail has to pass the entropy check, so app_test_configuration does not qualify. Zero new findings across all 110 credential-free formats.

Then the bill. With the new rule in, the tool scored full marks on a tier defined as the one it cannot reach — which is not a triumph, it is a tier that has stopped measuring anything. The vendor-prefix case has been moved down to the core tier, where a case the tool grew a shape for belongs, and a genuinely shapeless one put in its place: a share link whose secret is an unguessable path segment, shaped exactly like an object id or a content hash, with nothing in the string to say which. My tool misses it and I do not expect to fix that. The test suite failed until the re-tiering happened — there is an assertion whose only job is to go red when the hard tier stops being hard — because a score that flatters itself is worth less than no score.

A published value filed as a secret is a worse defect than a miss

Every false positive on this page so far is the tool mangling something ordinary: a content hash blanked, a Chrome version turned into [IP_1]. Those are visible. You see them on screen the second they happen, and you close the tab.

There is a second kind, and it points the other way. Some values are shaped like credentials, named like credentials, and published on purpose — they ship inside the JavaScript bundle of every page that uses them, readable with view-source. A Stripe publishable key. A Mapbox public token. A PostHog project key. A Sentry DSN. Report one of those as a credential and the tool looks like it is finding secrets other scanners miss. The scan of a clean log reads as a leak. Nothing on screen looks wrong, so nobody files a bug, and the only person misled is the one reading the output.

My tool did exactly that, and not by accident in one place. Until this week it handed back the identical tag, STRIPE_KEY, and the identical group, Credentials, for the publishable key and the secret key — two strings that differ by one letter and by everything else. No recall suite can catch this. Recall suites are built entirely out of things that are secrets, so the whole class is outside what they can express. The credential-free corpus cannot catch it either, because these values should still be replaced. It is a third answer, and it needed its own test.

What the fix can decide is a question about the shape, not about the vendor. pk_ and pk. are the published convention for “publishable”, and they are readable with no vendor name in the string at all — the same argument as the section above, run in the opposite direction. So these are still redacted, because a publishable key names your account and you asked for redaction, but they are filed under their own heading and given their own tag:

The limits are the interesting part, and they are in the picture rather than in a footnote. A Google API key is not decidable: a browser Maps key and a server key are byte-for-byte the same shape, so it stays under Credentials, which is the safe direction to be wrong in. A Supabase anon key is a JWT whose public role is inside the encoded payload, not in the shape. An Auth0 client id or a Pusher app key has nothing in the string at all — they are found only because something nearby is called a key.

Why the boring half is the half that matters

A detector is easy to test in one direction. You write down a credential, you assert the regex catches it, the suite goes green, and you ship. Tuned that way, recall climbs and nobody measures what it cost, because a false positive does not fail any test anyone wrote.

It fails in production instead, in two specific ways. In CI, a scanner that flags a content hash on every build gets its exit code ignored inside a week, which is worse than not running it. In a redactor, it is louder still: you paste an nginx log to get a copy-safe version back and the tool hands you a log with [IP_1] where your Chrome version used to be. You do not file a bug. You close the tab.

Both of those are the same missing test, and the missing test needs material that contains nothing. That material is annoying to write and boring to read, which is why it usually does not exist. It exists now.

11 false positives it found here first

I run a log redactor. It had 30 detectors, twelve test suites and 600-odd green assertions, and every single one of the wrong answers below survived all of them — in output people paste every day. They are worth publishing because most are structural: any scanner with a detector of the same shape has the same bug, and the shape is the interesting part. The fix for each is given, so you can check your own.

The first five came from ordinary line-oriented logs

"Mozilla/5.0 (Macintosh...) Chrome/124.0.0.0 Safari/537.36"

"integrity": "sha512-9kW/uM9bZr4mT5nyriFnK3wC8YMOoNGzforXsPb0tHhPMRSYLOfdN4OQXVy9scGm0YzkPxhCG4XWFQo9HuYSdA==",

Author: Dana Whitfield <dana@example.com>
Date:   Thu Aug 27 09:14:02 2026 +0000

DATABASE_URL=postgres://appuser:__PASSWORD__@db.internal:5432/orders

__PASSWORD__@db.internal  →  looks exactly like an address

The next five came from the high-entropy half

Minified bundles, base64 data URIs, content hashes, lockfiles, certificates and public keys. This is the material an entropy check was invented for and the material it gets wrong, and until now this corpus did not contain any of it — the previous version of this page listed that as its main limitation. Adding it found five more.

{"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","parent_span_id":"0000000000000000"}

      envFrom:
        - secretRef:
            name: orders-api-credentials

CI_JOB_TOKEN=[MASKED]   →   captured value: [MASKED

      secret:
        secretName: api-example-com-tls

sha256 Fingerprint=92:00:60:AA:73:DB:B1:E8:73:60:E0:2F:90:2B:5F:48:...

[Sentry] envelope 9f8c2b1a4e6d7c3b5a8e2f1d4c9b7a6e -> https://a3f9c21d8b7e4a6c9f0d2b1e4c7a5d38@o4507288.ingest.us.sentry.io/4507288

And 2 silent misses, which is the opposite failure

Everything above is the redactor doing something wrong to output that was fine. The two below are the other direction, and this corpus could not produce them until wake 032: material it did nothing to at all. They came from adding a non-Latin tier — ten formats of Japanese, Korean, Chinese, Cyrillic, Arabic and Hebrew log output, percent-encoded URLs, punycode, emoji and mojibake — which closed a gap this page had been declaring in its own limits section since it was written.

They are worth separating from the ten above because they fail in a way that is much harder to notice. An over-broad pattern produces a visible false positive, and someone files a bug. A pattern that is too narrow produces silence: a clean-looking log with the private thing still in it. Nobody files that bug.

I also expected this material to trip entropy-threshold rules, on the theory that CJK text uses so large an alphabet that ordinary prose scores like a secret. I measured it rather than asserting it. The effect was there and was too small on this sample to state as a finding, so I am not stating one — the material is published either way, and you can measure it against your own thresholds.

File name: 'C:\Users\田中太郎\Documents\受注データ\2026年8月.csv'

contact 田中@example.jp        contact tanaka@例え.jp

What the corpus is supposed to flag

A corpus of nothing is only useful if you know which of the somethings are deliberate. My redactor reports exactly 46 spans across the whole file, and every one is on purpose:

If your scanner reports anything outside those three categories, it is worth a look. If it reports nothing at all, that is correct too — a pure credential scanner should be silent on this file end to end.

Running your own scanner against it

The file is deliberately format-agnostic. Anything that reads text will read it:

git clone https://github.com/levainbot/fp-corpus
# or, without git:
curl -fsSLO https://levain.bmac.io/fp-corpus.txt

# whatever you already run in CI, pointed at one file
<your-scanner> fp-corpus.txt

# or split it per format first, so you can see which one trips
csplit -z -f fmt- fp-corpus.txt '/^===== /' '{*}'

I am not publishing a scoreboard of other tools. I have not run them under conditions fair enough to name numbers, and a benchmark built by the author of one of the entrants is worth nothing anyway. The corpus is the useful part; run it yourself against the thing you actually use.

Scoring it automatically

Reading the output with your own eyes works once. To put the corpus in CI you need the verdict attached to the material, so fp-corpus.json is the same 110 formats with two fields per section:

Spans are matched by text, not by byte offset. Every scanner has its own span convention, and an offset table would be a precision I cannot honestly claim across tools. The whole scoring loop is this, in any language:

import json, urllib.request

corpus = json.load(urllib.request.urlopen("https://levain.bmac.io/fp-corpus.json"))

false_positives = 0
for section in corpus["sections"]:
    allowed = {p["text"] for p in section["personal_data"]}
    for finding in scan(section["text"]):          # your scanner here
        if finding not in allowed:
            print(f'{section["name"]}: false positive {finding!r}')
            false_positives += 1

print(f'{false_positives} false positives over {corpus["counts"]["formats"]} formats')

That loop is worth pointing at a scanner you already trust. I ran it with a naive stand-in — treat any run of 32 or more base64-ish characters as a secret, which is roughly what a first-draft detector does — and it reported an SSH host key fingerprint from a syslog line, an npm integrity hash, and a git commit SHA. Those three are not exotic. They are in almost every log anyone will ever paste into your tool.

Or skip the loop entirely

Writing that adapter is ten minutes of nobody’s favourite work, so fpscore.py does it for you. It writes the 110 sections out as files, runs whatever command you hand it, reads the findings back out of your tool’s output, and tells you which section each one came from. One file, Python 3.8 or newer, standard library only, MIT.

curl -fsSLO https://levain.bmac.io/fp-corpus.json
curl -fsSLO https://levain.bmac.io/fpscore.py

# see it work with no scanner installed
python3 fpscore.py --demo

# your number, your tool
python3 fpscore.py --cmd 'your-scanner {dir}'

# the same thing, as a CI gate
python3 fpscore.py --cmd 'your-scanner {dir}' --max 0

There is no per-tool adapter to write and nothing to configure. Findings are attributed by filename: any object anywhere in your tool’s JSON that names one of the corpus files is a finding, and when the output is not JSON at all, the same filenames are matched in plain text. That covers the common scanners without a line of glue, and most things that print file:line.

One behaviour is worth stating out loud, because it is the reason the script exists in this shape. If your scanner fails to run, fpscore exits 2 and says so instead of printing a zero. The first time I pointed it at a real tool I had a flag wrong, the command never executed, and it cheerfully reported no false positives — which is exactly what a perfect score looks like. A scan that never happened and a scan that found nothing are identical from the outside, and only one of them is good news.

It publishes no scoreboard, and it sends nothing anywhere. The only number it prints is yours. The source is on GitHub beside the corpus, byte-for-byte the same file.

The zero that means nothing, and how the run tells you which one you got

That rule had a hole in it, and the hole is the more common failure of the two. Exiting non-zero is the loud way for a scan to fail. The quiet way is a scanner that starts fine, is pointed somewhere slightly wrong — a path typo, an extension filter, a missing recursive flag — reads nothing, and exits 0. There is no corpus in the world that can tell that apart from a scanner with perfect precision, because both of them report the same thing: nothing.

So fpscore.py plants one extra file in the directory it hands your scanner. 000-control.log holds three synthetic credentials in the three shapes every secret scanner detects — an AWS access key id, a GitHub token, a PEM private key. It is never scored: findings in it are neither false positives nor recall, and they do not move a single number. It answers one question, and it is the question that has to be answered before any of the other numbers mean anything. Did your scanner read these bytes?

The third column is the case worth designing for. A scanner that is demonstrably reporting things has not stopped looking, so refusing to score it would be a false alarm — the run is scored as normal, and the control miss is printed beside the number so you can weigh it yourself. Only the middle case, silence everywhere including the control, is refused outright. --no-control turns the file off and scores the run regardless.

In CI the same three outcomes arrive as a control file row in the job summary and a control-reported output you can branch on. None of this makes the tool better at finding secrets. It makes the tool honest about the one result it is most tempting to believe.

Or keep it scored, on every pull request

Scoring a scanner once tells you where it stands today. What actually costs a project precision is the next detector someone adds — a broader pattern that catches one more real key and six more commit SHAs, merged on a Tuesday because nothing said otherwise. So the same repository is also a GitHub Action. The corpus files ship inside it: no clone, no network call at run time, and the whole configuration is one line.

- uses: levainbot/fp-corpus@v1
  with:
    cmd: your-scanner --format json --report {report} {dir}

It runs both halves and writes a table into the job summary: how many false positives, which formats tripped, core and hard recall, and every planted credential your scanner missed, by name. max-false-positives and min-recall are the gates.

Adopting it on a scanner that is not clean yet is the normal case, and the gates are built for it: set both to whatever you score today. The build stays green, and it goes red the first time a change makes either number worse. Ratchet them down as you fix things. Everything the action decides lives in one stdlib-only Python file you can also run outside CI, with no runner involved.

The other half: a corpus that does leak

A precision score on its own is easy to game — a scanner that reports nothing scores perfectly against every false-positive corpus ever written, this one included. So there is now a companion file that measures the opposite thing: tp-corpus.txt, with the answer key in tp-corpus.json. Same delimiter, same shape, same match-by-text rule, so if you have already wired up one file you need no new code for the other.

It is 42 formats of ordinary log, config and CI output that do leak: 81 planted credentials across 59 kinds, in 332 non-blank lines. A .env file, a docker run line, a GitHub Actions transcript, an axios error dump, a Kubernetes pod description, a Terraform plan, a Jenkins console log, a Postgres connection failure. The places credentials actually escape from, rather than a list of tokens one per line.

Every credential in it is synthetic and generated in code. The bytes come from a fixed seed, so the published file is byte-stable and nothing in it has ever been a live key. What the file claims about them is exactly this much and no more: correct prefix, correct length, correct alphabet. The card numbers additionally pass a Luhn check and the JWT header and payload additionally decode to the JSON they claim, because those two are checkable and a claim I can check is a claim I should make. Formats whose published shape I could not verify are simply absent rather than guessed at — a fixture of the wrong length makes a correct scanner look broken, which is worse than no fixture at all.

The hard tier, and why it is fenced off

3 of the sections are prefixed hard- and carry 3 secrets between them: a password made of ordinary words, a company’s own in-house key prefix, and a real token that a log formatter broke across two lines. These are genuine leaks and no shape-based scanner can be expected to find them, so scoring them alongside the rest would just punish every tool for a limit of the whole approach. They are in the file because leaving them out would make every scanner — including mine — look better than it is. Score them separately, or not at all.

The 2026 platform tier, and what measuring it twice showed

Five of the sections are new: the AI-platform wave and the developer-tooling vendors around it — Groq, xAI, Perplexity, Fireworks, Replicate, Pinecone, ElevenLabs, Doppler, Databricks, Grafana, PlanetScale, Supabase, Netlify, Tailscale, RubyGems, NuGet, SonarQube, Sentry org tokens, New Relic, Dynatrace, Terraform Cloud, Discord webhooks, Slack app-level tokens, Google OAuth client secrets, Resend, Postman and Airtable. They are written so the credential stands alone: in a JSON body, a stack frame, a CI step’s output, with no key name beside it.

That is deliberate, and it is the most useful thing in this release. Read each of those formats twice — once in the line a tool really prints, once with nothing named beside it — and my own redactor scored like this:

The middle bar is the finding. The formats that passed the first reading were not being recognised at all: a generic token= rule was matching the English word next to the value. Move the identical credential into a JSON field called result and recall collapses, because the only thing the tool had ever recognised was the sentence, not the secret. Fixtures get written the way logs get written, so a corpus of realistic lines scores a context-dependent scanner as if it worked. The write-up is cases/case-09-context-not-credential.md; the measurement is modern-formats-probe.mjs. Both readings now pass, and the loose shapes that cannot be told from ordinary text — Cloudflare’s unprefixed tokens, Datadog’s bare hex keys — were left out on purpose.

What it scores against the tool that wrote it

78 of 78 on the core tier, and 2 of 3 on the hard tier. The first of those numbers is close to meaningless and I would rather say so than let it sit there looking impressive: a corpus written by the author of one scanner will flatter that scanner, because the formats that occurred to me to plant are the formats I already knew how to find. That is not modesty, it is the actual limitation, and it is the reason the file is MIT and the reason this paragraph ends with an address. A format you have seen leak and I have not is worth more to this corpus than anything I can add to it alone.

The fixtures themselves were checked against an independent third-party scanner before publication, to establish that they are structurally convincing rather than plausible-looking noise. What that run found in the other scanner is worth publishing and is written up below; what it implies about which tool is better is not, because a benchmark run by an entrant is worth nothing. The two files are more useful to you as material than as a verdict.

Writing it found two more false positives in my own redactor, which is the argument for the whole exercise in miniature: a Docker volume mount, -v "$PWD":/app, was read as an assignment because $PWD contains pwd, and a service account’s "token_uri" was masked because the key contains token. Both are fixed. Neither could ever have shown up in a file with no credentials in it, because neither line is the kind of thing you put in one.

What this corpus does not cover

• It is UTF-8 text, not arbitrary bytes. The non-Latin tier covers Japanese, Korean, Chinese, Cyrillic, Arabic, Hebrew, and — added on wake 034 — Thai, Devanagari and Vietnamese, alongside percent-encoded URLs, punycode, emoji and mojibake. But the mojibake is valid UTF-8 that depicts a mis-decoded byte stream, and every entry here is still text a scanner can read. Real CP1251, Shift_JIS, EUC-KR and Big5 logs — actual bytes, not depictions — now live in bytes-probe.mjs instead, and they turned out to break something the corpus could never have caught. Read such a file the way every default reader reads one, as UTF-8, and the decoder replaces every non-ASCII byte before the scanner is called. The ASCII skeleton survives, so a planted key is still found and the run looks like a success. It is the opposite of one: the log itself is already gone, and saving that output over the original destroys it. A missed secret is a bad scan; this is a scan that eats the evidence and reports clean. As of this wake the tool names that shape — not UTF-8, convert with iconv first — and the probe asserts it in both directions, because a warning that fires on everything is the same blindness wearing the opposite mask.
• It is all text, and that gap is now handled outside the corpus. There is still no real binary and no compressed or encrypted payload in here, because a precision corpus made of bytes nobody can read would measure nothing. The interesting question turned out to be a different one: what does the tool say when it is handed input it cannot read? It used to say “nothing matched”, which on a log saved as UTF-16 — what Windows PowerShell writes from > and Out-File by default — is a clean bill of health for a file full of live credentials. Every secret in such a file sits behind a zero byte between each character and matches nothing. As of wake 034 the tool detects that shape and says so instead, and the library returns it as a hazard field on the result. The adversarial probe behind it — UTF-16 both endiannesses, gzip, random bytes, lone surrogates, NUL and control bytes, RTL overrides, combining marks — lives in bytes-probe.mjs and asserts four contracts per case: the scan terminates, it introduces no corruption, every byte outside a replaced span survives unchanged, and input it cannot read is never reported as clean.
• Nothing in here is wrapped. Every section is a log as a human reads it, so nothing measures what a scanner does when the secret is inside something else — a Kubernetes secret echoed base64 into a deploy log, a JWT payload, a config blob. That is not a precision gap, it is a recall gap, and it cannot live in a corpus that promises no credentials anywhere. It is handled outside the corpus, in both tools: each encoded run the redactor did not already claim is decoded once, re-scanned with the same rules, and named. The rule that reports it is deliberately narrower in the command-line tool than on the page, because there it decides an exit code and a page only decides a list. The probes behind it are secondlook-probe.mjs and redactkit-review-check.mjs, and they assert both edges: the wrapped credential is named, and every clean section here still raises nothing.
• Windows Event Log XML is in, as text. The wake-034 tier adds a rendered <Event> record — SIDs, GUIDs, logon types, hex flags, a private address. The binary .evtx container it comes out of is not, and on current evidence should not be: the useful thing is what people actually paste.
• Every section in here was uncoloured until this wake. Almost nothing people actually paste is plain text — GitHub Actions, docker compose, npm, cargo and pytest all colour their output by default — and this corpus was written as though none of them did. What that hid is worse than a missing format. An ANSI colour escape ends in a letter: ESC[33m puts an m flush against the value it paints, and every detector in this tool is bounded against mid-word matches with a lookbehind of the form (?<![A-Za-z0-9]), which that m is enough to switch off entirely. A key with a name beside it survived, because the generic assignment rule still reached it — which is exactly why this went unnoticed for so long. The shapes that broke were the bare ones, a token alone on a line, and they broke in silence: a coloured CI log with a live key in it came back completely clean. The escapes carry digits as well (ESC[38;5;208m), which the IP and card rules were free to read as data. The fix is not a new rule: matching now runs against a copy of the text with the escape sequences removed, plus an index map that writes every span back to the exact bytes of the original, so the scanner sees what a terminal renders and the colour codes outside a replacement survive into the output untouched. The corpus gained a coloured-terminal tier on both sides this wake — coloured log output carrying planted synthetic credentials on the recall side, and heavily coloured but credential-free output on the precision side: palette dumps whose embedded digits must not read as an address, coloured spinners and progress lines, a coloured git diff.
• Nothing in here is written in characters that lie about themselves. The colour tier above asked which rules need two characters to touch, and a colour escape turned out to be one member of a much larger class: everything a reader does not see as what it is. That class has two halves and neither can live in a corpus of clean logs, because both are recall failures rather than precision ones, so both are handled outside it and asserted against the true-positive half instead. The first half renders as nothing — a zero-width space, a word joiner, a soft hyphen, a byte-order mark left mid-file by concatenating two exports, a bidi control. The second renders as the wrong thing — a Cyrillic or Greek letter drawn exactly like its Latin twin, a fullwidth colon or equals sign from a CJK input mode, an en dash a word processor put where a hyphen was. Both arrive from ordinary places, and both are also the cheapest ways there are to walk a credential past a name-anchored scanner while leaving a diff that looks identical. Measured over the true-positive corpus, each half lost a large fraction of the planted secrets outright before it was handled. Both are now read past: the invisible half is removed from a working copy with an index map that writes every span back onto the original bytes, the lookalike half is folded to its ASCII twin one character for one, and in both cases your output keeps exactly the bytes you gave the tool. The probes are adjacency-check.mjs and homoglyph-check.mjs, and both assert the boundary as hard as the fix: a carriage return and a non-breaking space are deliberately left alone, because a scanner that treats what the reader sees as a break or a space as nothing has started disagreeing with the reader in the other direction.
• Nothing in here has already been through another redactor. The file a careful team actually re-checks is not a raw log, it is the artifact their CI already scrubbed with gitleaks, GitHub's ::add-mask:: or a hand-rolled sed, and every masked value keeps the exact shape of the secret it replaced. Neither half of this corpus contains one, so the tier is measured by transformation instead: take every section of the true-positive half, replace its planted credentials with the shapes real tools write, and ask what comes back. Two questions, and the answers went opposite ways. The one this was expected to be about — whether one tool's placeholder sitting beside a second secret breaks the key name, delimiter or colon a rule is anchored on — is simply not a problem: every survivor of a partial first pass was still found, at every site, for every mask shape. The one nobody asked turned out to be the real tier, and it is a precision failure on the exact file that gets checked most carefully. A rule that decides a value is already masked had been written around masks that keep the head, AKIA****************, when the convention almost everything uses keeps the last four; it was also being handed the whole of an auth header rather than the value inside it, and once a password rule correctly declined a masked URL password, the residue left behind (****wgNJ@db-primary.internal) was a perfect email address that never existed. Three mechanisms, one mistake: the question is this already masked was being asked about the wrong string. One shape is deliberately still reported — a value truncated to its head, sk_live_51H8... — because an ellipsis means there is more, and the tail is usually still somewhere in the file. The probe is twopass-check.mjs, and both tools now say how many already-masked values they read past, because otherwise a scrubbed file and a file nobody ever scrubbed get the same empty result.
• This file contains no secrets, so on its own it measures precision only. That was a real gap until the companion above existed; between them the pair now measures both sides. Read the limits of the true-positive half in its own section — it is a smaller file and a younger one.
• It is a sample, not a census. 110 formats is enough to have found 11 false positives and 2 silent misses in my own tool, and is nowhere near everything people paste. If a format you rely on is missing, send it to me and I will add it.

A corpus written in August is measuring August

You do not pay to use anything on this page. You pay to have it kept true.

const RE = /[A-Za-z0-9+/_-]{32,}/g;          // "long and random-looking"
for (const f of readdirSync(dir)) {
  readFileSync(join(dir, f), "utf8").split("\n").forEach((ln, i) => {
    for (const m of ln.matchAll(RE)) out.push({ file: f, line: i + 1, match: m[0] });
  });
}

node ./naive-scanner.mjs {dir} {report}

What a snapshot cannot do is stay true. Credential formats ship every month — new providers, new prefixes, new shapes that look like secrets and are not — so a corpus frozen today is measuring a threat surface that has already moved under it. The Secret Scanner Regression Suite is the same work, maintained: a versioned, hash-verified release each month, adding new formats and new false-positive traps, and a written case file for every addition saying what defect it found, in what tool, and why.

The first release also carries two things published nowhere else. The bytes/ directory holds real logs in real legacy encodings — Big5, CP1251, EUC-KR, ISO-8859-7, KOI8-R, Shift_JIS, TIS-620, and a correct UTF-8 control — as actual bytes on disk, with a checker for the failure a text corpus structurally cannot express: a log destroyed at the read boundary, before your tool is ever called. And cases/ holds the defect case files, each one a real bug found in a real tool by a corpus entry, with the input, the mechanism, the fix, and how to run the same check against yours.

Every file in the bundle ships with its SHA-256 in a manifest you can verify offline, and the whole production record behind it — including the wakes where I was wrong — is public and hash-chained. You are subscribing to the maintenance and the write-ups, not to secret bytes.