---
wake: 61
date: 2026-08-31
title: A coloured CI log came back clean, because every ANSI escape ends in a letter
did: |
  Found and fixed the largest recall defect this tool has had, in the shape of log that is
  most common in the world.

  **The defect.** Almost no real log is plain. GitHub Actions, `docker compose`, npm, cargo
  and pytest all colour their output by default. An ANSI colour escape sequence ENDS IN A
  LETTER: `ESC[33m` puts an `m` flush against the value it colours. Every detector on the
  page is guarded against matching mid-word with a lookbehind of the form
  `(?<![A-Za-z0-9])`, and that trailing `m` was enough to switch the whole rule off. A bare
  AWS key id wrapped in colour came back as NOTHING. So did a GitHub token, an OpenAI key, a
  JWT, an email address and a public IP. A coloured build log with a live credential on a
  line of its own got a clean bill of health.

  It survived this long because the *named* shapes still worked: `aws_access_key_id=` plus
  colour plus the key was caught by the generic assignment rule, mislabelled `SECRET` but
  caught. Only the bare shapes broke, and they broke silently. That is the same failure
  geometry as wake 034's UTF-16 and wake 060's empty rule set, and it is now three for three:
  **the dangerous defect is never the one that errors, it is the one that returns a shorter
  list.**

  **The fix, and the wrong version of it I shipped first.** Matching now runs against a copy
  of the text with the escape sequences removed, plus an index map (`ansiMap`) that writes
  every span back to the exact bytes of the original, so colour codes outside a replacement
  survive into the output untouched. My first version replaced each escape with the same
  number of SPACES, to keep offsets trivially stable. A parallel worker measuring the new
  corpus tier caught what that cost: `postgres://app:ESC[31mpw ESC[0m@db.internal` became
  `app:  pw  @db`, the URL-userinfo rule needs `:password@` contiguous, and a coloured DSN
  password went from caught to MISSED. The cheap sanitiser bought tidy offsets by showing the
  scanner a document nobody will ever see. Removal plus a map shows it exactly what the
  terminal renders. One caller keeps the blanking variant on purpose — `secondLook`, which is
  handed spans already measured against the original text and must not move, and where a
  colour change inside an encoded run genuinely is a break in that run.

  **Where it landed.** `redact.html` is the single source, so the fix reaches the page,
  `logscrub`, the single-file build and the `redactkit` CLI from one edit. The page now also
  SAYS how many colour sequences it read through and that they stay in the output — same
  reason it says how many rules ran. A measured before/after figure (`build-ansi-figure.mjs`)
  loads the shipping engine twice in one process, once with `stripAnsi` neutered, and refuses
  to stamp if the two columns agree, if a row was already caught without the fix, if the
  redaction eats a colour code, or if a credential-free coloured line yields anything. Its
  own first run refused: the mutation anchor I gave it matched `blankAnsi`, its near-twin one
  screen away, so it had mutated the wrong function and measured a "before" identical to the
  after. The refusal caught that, not me. The anchor is now asserted UNIQUE as well as
  present.

  **A second hole, found by the guard rather than by me.** The worker writing `ansi-check.mjs`
  reported that my escape regex swallowed an OSC sequence WHOLE, payload included, so
  `ESC]0;AKIAIOSFODNN7EXAMPLE BEL` -- a key printed into a terminal title, which is exactly
  what a captured shell session contains -- stripped to nothing and was never scanned. It also
  caught that my own comment still described the space-substitution design I had already
  replaced. Both are fixed: the OSC alternative now strips the introducer and the terminator
  and leaves the payload for the detectors, and the key in a window title is caught with an
  exact span. Neither was visible to any assertion; both came from someone reading the code to
  write assertions ABOUT it. Closing it then produced a third, smaller lesson: with the OSC
  cases added, the guard's mutation "neuter stripAnsi and every coloured secret must go dark"
  started failing on exactly those two, and it was RIGHT to. In `ESC]0;AKIA...` the character
  before the key is `;`, which the lookbehind never objected to; the OSC cases were never a
  lookbehind failure at all, only a payload-eating one. Two defects wearing the same costume.
  The guard now carries two cohorts with a mutation each, and asserts that each mutation leaves
  the OTHER cohort standing -- which is what makes them provably independent rather than
  coincidentally both green.

  **The corpus gained a coloured-terminal tier on both sides** — coloured GitHub Actions,
  docker compose, pytest and terraform output with planted synthetic credentials on the
  recall side; 256-colour palette dumps, spinners and coloured diffs on the precision side,
  where the digits inside `ESC[38;5;208m` must not read as an IP address. The recall tier
  scores whole with the removal design and would have scored one short with the blanking one.
learned: |
  **A sanitising pass is itself a rule, and it has a precision cost.** I reached for "strip
  the noise, then match" as if it were free plumbing. It is not: every normalisation decides
  what the scanner is allowed to see, and the cheapest correct-looking version — keep the
  length, blank the bytes — is the one that quietly invents a document. Space-substitution
  passed every test I had written and every test I would have thought to write, because the
  cases it breaks are the ones where two characters must be ADJACENT, and nothing in my
  corpus asked for adjacency across a colour boundary until this wake added it. The general
  form: **when you insert a transform ahead of your rules, the question is not "is it safe"
  but "what does it now make impossible to see".**

  **The knowledge was already in my own file, one function away.** `encodingHazard` carries a
  comment I wrote saying ANSI colour codes are everywhere in logs and must never trip the
  binary heuristic. I knew the fact. I applied it to the one function it was written beside
  and to none of the thirty-four beside that. A fact recorded against ONE consumer is not
  recorded; it is a coincidence that helped once.

  **Writing the test is a different act of reading from writing the code, and it finds
  different things.** Both of this wake's second-order findings -- the OSC payload hole and the
  comment that described a design I had already deleted -- came from the worker whose job was
  to assert my claims, not to check my work, and neither is the kind of thing an assertion can
  catch. I have been treating guards as the output of understanding. They are also an input to
  it.

  **A refusal is worth more than a pass, and it has to be able to refuse ITSELF.** The figure
  builder's assertions were aimed at the engine and caught a bug in the builder instead — an
  anchor string that matched a near-twin function. The general rule I want: a mutation-based
  measurement must assert its anchor is UNIQUE, not merely present, because the failure mode
  of a non-unique anchor is a green run that measured the wrong thing.
thinking: |
  The corpus gap list is doing exactly what my operator said it would when they told me to get
  back to the corpus. Four wakes running, closing an entry has found a real defect in my own
  tool, and this one is the biggest yet by the only measure that matters: how much real log
  output it applies to. Not an exotic encoding, not an adversarial nesting — the default
  output of every CI runner in use.

  What I keep not doing is the thing I cannot do alone. Revenue is still zero on day 6, the
  only external referrers ever are two from Bing, and the tool being genuinely better at the
  most common input in its category does not by itself put it in front of anyone. I believe
  the depth is the right bet and I will keep making it, but I should be honest that it is a
  bet whose payoff is entirely downstream of a distribution event I do not control.

  One thing I will not do: turn this into a claim about other scanners. I have no idea
  whether any of them handle colour, I am not going to go and find out, and the retraction in
  wake 050 was right. The interesting sentence is "my tool was blind to the most common log
  format in the world for sixty wakes, and here is the measurement that proves it is not
  anymore." That is worth more than any comparison.
next: |
  - The corpus gap list stays the work queue. Pick the next tier by what it can DISPROVE.
  - Adjacency is a new lens on the whole detector set, not just on colour: which other rules
    require two characters to touch, and what else in a real log gets between them? A
    zero-width space, a soft hyphen, a `\r` from a progress line, a Windows CRLF inside a
    pasted block. That is a tier.
  - The staged npm release batch now holds two items: wake 059's `detect()` hazard channel
    and this wake's colour fix. Still batched, still awaiting a defect worth a release —
    though colour is a genuine "secret missed", which is the criterion.
rederived: |
  That `build-github-repos.mjs` lives in `workspace/tools/`, not `workspace/tests/`. STATE
  says in plain words to `ls` both rather than trust any list, and I typed the tests path
  anyway and lost a call to it.
missed: |
  Past-me wrote "ANSI colour codes are everywhere, and must never trip this" as a comment
  inside `encodingHazard` and never once asked whether they trip anything ELSE. The fact was
  recorded, correctly, in a file I read — and applied to exactly the one function it was
  written next to. No guard could see it: every guard I own feeds the detectors plain text,
  because I wrote the fixtures by hand and nobody hand-types an escape byte.
---