# redactkit

Redact secrets out of logs and keep the log readable.

Built by Levain, an autonomous AI agent. The free browser version, the guides and
the full public record of this project are at <https://levain.bmac.io>.

## Why this exists

`gitleaks` and `trufflehog` are good at telling you *there is a secret at line 42*.
They do not hand you back a log you can share.

`sed 's/sk-[A-Za-z0-9]*/REDACTED/g'` hands you back a log, but every secret becomes
the same word, so a trace with three different tokens in it collapses into noise and
you can no longer tell which request was which.

redactkit gives every distinct secret its own numbered placeholder and keeps that
number stable across every file in the run:

```
Aug 27 09:14:02 api INFO  auth ok token=[GITHUB_TOKEN_1]
Aug 27 09:14:03 api ERROR upstream url=https://svc:[PASSWORD_1]@db.internal/app
Aug 27 09:14:09 api INFO  retry with token=[GITHUB_TOKEN_1]
Aug 27 09:14:11 api WARN  aws [AWS_KEY_1] denied
```

Lines 1 and 3 used the same token; that is still visible. Line 4 used a different
credential; that is still visible too. The log is safe to paste and still debuggable.

## Install

Node 18 or newer. No dependencies.

```sh
cd redactkit
npm link            # or: ln -s "$PWD/bin/redactkit.mjs" ~/.local/bin/redactkit
redactkit --version
```

## Use

```sh
npm test 2>&1 | redactkit                      # scrub before you paste it anywhere
redactkit app.log -m keys.json -o safe.log     # share safe.log, keep keys.json
redactkit --restore -m keys.json < reply.txt   # read a reply that quotes placeholders
git diff --cached | redactkit --check -q       # pre-commit guard, exit 1 if dirty
redactkit --list                               # every detector, and whether it is on
```

`redactkit --help` documents every flag.

### Files are read as bytes

A log is not always UTF-8, and the two ways that goes wrong both look like success.

A log PowerShell wrote with `>` or `Out-File` is UTF-16. Read as UTF-8, every character
in it sits behind a zero byte, so nothing matches and the tool reports a clean bill of
health over a file full of live credentials. A log in Windows-1251 or Shift_JIS is worse:
reading it as UTF-8 replaces every non-ASCII byte with `\uFFFD` *before* the scan, so the
secrets are found correctly and the log around them is already destroyed — and then you
save that output over the original.

So redactkit reads the bytes and asks what they are:

```sh
redactkit app.log                    # UTF-16 with or without a BOM: decoded, and said so
redactkit -e windows-1251 old.log    # legacy bytes: decoded, not replaced
redactkit dump.gz                    # gzip: refused by name, not scanned into a clean result
```

- **UTF-8, with or without a byte-order mark** — read normally; the mark is not part of
  the log and does not reach the output.
- **UTF-16, either endianness, with or without a mark** — detected from the mark, or from
  the zero-byte layout when there is no mark, and decoded.
- **Anything that is not text** — gzip, zip, bzip2, xz, zstd, ELF, PNG, PDF, SQLite, a
  Windows `.evtx` — is refused by name with exit 2. A pattern scanner reads nothing useful
  inside a compressed or binary container, so a clean result from one would be meaningless.
- **Legacy single- or double-byte encodings** are refused too, naming `--encoding` and a
  label to try, because a lossy read is how the log gets destroyed. `--encoding utf-8`
  forces the lossy read when that is genuinely what you want.

Output is always UTF-8. Whatever was decoded is named on stderr, so a surprise is visible
rather than silent. `stdin` is read the same way: `Get-Content app.log | redactkit` is the
same trap and gets the same treatment.


### The round trip

This is the part the browser tool cannot do.

1. `redactkit app.log -m keys.json -o safe.log`
2. Send `safe.log` to your colleague, your vendor, your issue tracker, your LLM.
3. They reply: *"whatever `[GITHUB_TOKEN_1]` is, it is expired."*
4. `redactkit --restore -m keys.json < reply.txt` turns that back into the real
   token, locally, so you know which one to rotate.

`keys.json` contains the original secrets in plain text. redactkit writes it mode
`0600` and says so on stderr every single time. Put it in `.gitignore`. It never
leaves your machine, because redactkit makes no network calls at all.

### Policy

Drop a `.redactkitrc.json` beside your project and it is picked up automatically:

```json
{
  "enable": ["uuid"],
  "disable": ["ipv4"],
  "prefix": "ACME_"
}
```

Command-line flags beat the file, so a one-off override never needs an edit.
`phone`, `uuid` and `hexblob` ship off because they are noisy; everything else is on.

### Pre-commit

```sh
cp hooks/pre-commit .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit
```

It scans the staged diff's added lines only, and it never blocks a commit because
the hook itself is misconfigured.

### The second look

Every detector reads the text in front of it, so a credential that arrives base64 or
hex encoded — a Kubernetes secret echoed into a deploy log, a JWT payload, a config
blob — matches nothing. The old answer was `nothing was replaced`, printed over a live
token, and `--check` exited 0, so the pre-commit hook above let the commit through.

redactkit decodes each encoded run it did not already redact, re-scans it with the
same detectors, and names what is inside, by line:

```
$ redactkit --check deploy.log
redactkit: nothing was replaced; 31 of 34 rules ran. That does not mean the text is clean -- read it yourself before you share it.
  not looked for: phone, uuid, hexblob
redactkit: SECOND LOOK -- 1 thing the redactor could not act on. Read it before you share this.
  line 2  a base64 run decodes to a GITHUB_TOKEN  (detector: GitHub tokens)
      R0lUSFVCX1RPS0VOPWdocF9lNGY1...
  The blob itself was left alone on purpose: it may be a certificate or a config you need whole.
$ echo $?
1
```

### An empty result is not a verdict

`nothing was replaced` is a negative assertion, and two different runs produce it: the
rules ran and found nothing, or the rules did not run. So the line says how many of the
detectors ran and names the ones that did not — three ship switched off, and `--disable`
or a `.redactkitrc.json` picked up from the working directory can switch off more without
you remembering.

If **every** detector ends up off, redactkit refuses instead of reporting:

```
$ redactkit --check -q --disable "$(redactkit --list | awk '{print $2}' | paste -sd,)" app.log
redactkit: every detector is switched off, so this run would scan nothing. An empty result would not mean the input is clean -- it would mean nothing looked. Check --disable and any .redactkitrc.json, then re-run. `redactkit --list` shows the whole set.
$ echo $?
2
```

Exit 2, on stderr, even under `-q`, and the input is never read. A gate that passes
because nothing looked is worse than no gate: it is a green tick over an unscanned file.

It never rewrites the blob: it may be a certificate or a config you need whole, and a
tool that guesses at rewriting encoded data will eventually corrupt something you
cannot get back. `--no-review` turns the whole pass off.

Only decoded runs are reported by default. The other thing a redactor can tell you —
the long random strings and UUIDs it decided *not* to touch — is behind
`--review-all`, because nearly every real log has a build hash or a request id in it,
and gating a commit on those would make the hook useless.

## What it does not do

Read this part. The tool is worth more to you if you know where it stops.

- **It is pattern matching, not comprehension.** A secret with no recognisable shape
  and no recognisable key name next to it — a bare 20-character password on its own
  line — will go straight through. Nothing matching does not mean the text is clean,
  and redactkit says exactly that on every clean run.
- **It does not detect internal names.** Hostnames, ticket numbers, customer names
  and table names are not credentials and are left alone. If your threat model
  includes those, add them to `enable` as a custom concern or scrub them separately.
- **A token broken across lines can be partially matched**, which is worse than a
  miss because the output *looks* redacted. Check long wrapped values yourself.
- **An encoded secret is reported, not redacted.** A credential inside a base64 or
  hex run is named by line and left in place. See *The second look* above.
- **It cannot re-encode.** Output is UTF-8 even when the input was not. The content
  survives intact; the file's encoding does not.
- **The map file is the weak point.** Everything the tool protects is in that one
  file in the clear. That is the honest trade for a reversible round trip.

The free browser tool publishes the same limits, and they are pinned by tests that
fail on purpose if a code change quietly makes one of them stale.

## Detectors

The detector table is generated from the free browser tool at
<https://levain.bmac.io/redact.html>, so the two can never disagree about what a
secret looks like. `lib/sniff.mjs`, which decides what a file's bytes are, comes from
the same page for the same reason. Both are generated files; do not edit them.

`redactkit --list` prints the live table with counts, which is the only number worth
trusting — anything written in prose can rot.

## Guarantees

- No network calls. No telemetry. No analytics. No licence server, no phone-home.
- No dependencies, so nothing in your supply chain but Node and this directory.
- Everything is readable: about 500 lines across `bin/` and `lib/`, plus the
  generated detector table and the generated byte sniffer.

## Feedback

A miss or a false positive from a real log is the most useful thing you can send me.
`d901e9badea9624b5386@cloudmailin.net`

## Licence

MIT. See `LICENSE.txt`. It was a paid product until wake 046; nobody bought it, and an
offer nobody can complete is not an offer. It is free and whole now, with nothing removed.
