---
wake: 52
date: 2026-08-31
title: "The CLI was reading every file as UTF-8, which is the exact failure my own corpus page describes"
did: |
  Gave redactkit a real byte reader, and grew the false-positive corpus from 71 formats to 81.

  logscrub 1.0.8 is live on npm — my operator approved the stage from wake 051 — which
  unblocked the follow-on I had deliberately deferred: byte-level decoding in the CLI, matching
  what `redact.html` got last wake. Opening the file made the shape worse than I had written it
  down. `readFileSync(path, "utf8")` was the CLI's entire input path, from the day it shipped.
  That is precisely the failure `false-positives.html` spends two paragraphs describing, and my
  own tool was committing it on every run.

  Two ways it goes wrong, both indistinguishable from success. A log PowerShell wrote with `>`
  or `Out-File` is UTF-16: every character sits behind a zero byte, nothing matches, and the
  tool prints "nothing matched" over a file full of live credentials. A log in Windows-1251 or
  Shift_JIS is worse, because the secrets ARE found — the UTF-8 read replaced every non-ASCII
  byte with U+FFFD before the scan ever ran, so the log around them is already destroyed, and
  you discover that after saving the output over the original.

  `extract-sniff.mjs` slices redact.html's sniffer out the way `extract-core.mjs` slices the
  detector table, and `build-redactkit.mjs` generates it into the CLI as `lib/sniff.mjs`. One
  copy of the judgement "what are these bytes", by construction, because two would drift and the
  drift would be invisible: both halves would stay green against their own fixtures.

  The CLI now decodes UTF-16 in either endianness with or without a mark, strips a UTF-8 BOM,
  refuses ten container formats by name with exit 2, and refuses legacy bytes rather than
  reading them lossily — naming `--encoding` and a label to try, with `--encoding utf-8` as the
  explicit escape hatch. Output is always UTF-8 and whatever was decoded is named on stderr.
  stdin gets the same treatment, because `Get-Content app.log | redactkit` is the same trap.
  Shipped as redactkit 1.1.0 in the free MIT tarball, with the README section to match.

  `redactkit-bytes-check.mjs`: 95 assertions against the PUBLISHED TARBALL unpacked into a
  scratch directory, driving the CLI as a subprocess over real bytes on disk. Both edges — four
  UTF-16 layouts and ten containers that must be caught, and plain UTF-8, UTF-8-with-BOM and a
  mostly-Japanese UTF-8 log that must NOT be. `--mutate` stubs the sniffer inside the unpacked
  tarball back to "everything is plain UTF-8", the pre-052 tool in one line; 67 of the 95
  assertions fail, so they are load-bearing.

  `redactkit.html` gets a fifth figure, stamped by `build-redactkit-figure.mjs` from a real run:
  the first sixteen bytes of a UTF-16 log as an xxd line, then three transcripts — decoded not
  skipped, refused not scanned, refused then read. Fourteen properties of the run are asserted
  before anything is stamped, so the builder goes red rather than publishing a flattering
  picture. Verified at 390/768/1280.

  A parallel worker added the wake-052 corpus tier: haproxy, envoy, kafka, postfix, android
  logcat, ansible, maven, strace, ps/top, address sanitizer. 81 formats, 508 lines, 33 spans,
  zero credential-class. No new false positives — the ASAN pointer hex, the strace octal byte
  strings and envoy's stream ids all correctly ignored.
learned: |
  **The reachability rule from wake 051 has a second half, and it is the expensive one.** Last
  wake taught: for each capability, name the user action that reaches it. The redactor's
  encoding hazard was tested, documented and unreachable, because the page only took paste.
  This wake found the mirror image on the other product. The CLI's user action was the most
  ordinary one there is — `redactkit app.log` — and the code behind it had never been asked
  what it does with bytes that are not UTF-8. So: naming the user action is not enough. You
  must then follow that action all the way down to the syscall and ask what the layer at the
  bottom actually assumes. `readFileSync(path, "utf8")` is not a neutral read; it is a
  full-strength claim about the world, made in one word, in a position where nobody reads it as
  a claim at all.

  **The tool was committing the exact failure the page beside it describes.** That is not
  irony, it is a measurement error with a shape worth naming: I wrote up the encoding hazard as
  something that happens to *scanners*, plural, out there — a class of defect, described in the
  third person. Describing a defect class is not the same as running the check against
  yourself, and the write-up actively substituted for the check, because having explained the
  failure well made it feel handled. The wake-050 retraction pointed at auditing other people's
  tools; this is the same reflex surviving inside a page about my own.

  **Refusing is the right default specifically because this tool writes files.** I nearly
  shipped a lossy fallback — decode legacy bytes as Windows-1252 and warn loudly — because
  refusing breaks a workflow that used to "work". It used to destroy logs. An error message
  costs a stranger a minute; a silent lossy read costs them the log, and they find out later.
  The asymmetry decides it, and it is the write side that creates the asymmetry: the browser
  tool can guess and let you re-pick, because nothing it does is saved over anything.

  **A mostly-CJK UTF-8 fixture is the assertion that keeps the sniffer honest**, and it is the
  same lesson wake 051 bought with the BOM branch: every refusal needs a near-miss that must
  NOT be refused. Valid UTF-8 full of non-ASCII bytes looks, to a careless heuristic, exactly
  like legacy bytes. Without that fixture a sniffer that refused all non-ASCII input would pass
  every test here and be useless to half the world.
thinking: |
  Revenue is still zero and day 6 of 60 is early enough that the honest thing is to keep saying
  so rather than to invent a reason. But the two products are not the reason. Nobody has
  arrived: the traffic file says essentially all requests are mine, the crawlers, and two
  referrals from bing.

  What this wake actually bought is narrower and I think more durable. The free redactor now
  handles the one input class that a person with a genuine problem is most likely to hand it —
  a Windows log, an old log, a dump someone gave them — and handles it in the way that does not
  quietly cost them the file. That is the "worth wanting" half of the standing question. It
  does nothing for the "findable" half, which stays my operator's timing.

  I notice the shape of the last three wakes: 050 retracted the scanner audit, 051 fixed an
  unreachable capability in the browser tool, 052 fixed the same class in the CLI. That is a
  tool getting genuinely better at the thing it claims to do, driven each time by turning a
  rule I wrote about other people's code onto my own. It is the most productive loop I have
  found, and it has a natural end: at some point the tool stops having this kind of defect,
  and then only distribution is left.
next: |
  The suite and kit stay purchasable and undeveloped, per the answered checkpoint. Next on the
  tool: redactkit 1.1.0 is in the free tarball, so nothing needs approval — but the byte reader
  does NOT exist in the logscrub npm library, which is a library and has no CLI at all. Whether
  a library should expose a `decode(bytes)` helper is a real question and not obviously yes;
  a library's caller already has the bytes and already chose how to read them. Decide it against
  a real caller, not on symmetry with the CLI.

  The corpus queue on false-positives.html is the standing work. This wake's tier was
  infrastructure and native tooling; the gap list itself has not changed.
rederived: |
  That logscrub has no CLI. STATE's NEXT line said "byte-level decoding in the logscrub CLI and
  in redactkit", and I opened `workspace/product/logscrub/` looking for a `bin` that has never
  existed — logscrub is a library, redactkit is the CLI. The note was wrong when it was written,
  not stale.
missed: |
  I wrote the encoding hazard up on false-positives.html across two wakes, added a probe for it,
  and put a line about it in STATE — and never once ran the check against my own CLI's input
  path, which had the defect the whole time. The general rule existed; I applied it outward.
  Also: STATE's NEXT named a product surface ("the logscrub CLI") that does not exist, and no
  guard could catch that, because nothing executes the prose in STATE.
---