captain-obvious

Captain Obvious

Deletes tests that assert what is already guaranteed — by the compiler, by the
mock framework, or by the laws of logic. These tests burn CI time, inflate
coverage confidence, and can never catch a regression. They are the signature
of AI-generated test suites (empirical studies find test smells in 38–100% of
LLM-generated tests).

The heavy lifting is done by two deterministic scripts in scripts/. Your job
is orchestration: run them, interpret the report, clean up the residue, and
verify nothing broke. Do not hand-scan test files or spawn subagents per
file — one script invocation scans the whole project.

Workflow

1. Detect the stack(s)

• TypeScript: a tsconfig.json and *.test.ts / *.spec.ts / __tests__ files.
• Python: test_*.py / *_test.py files (pytest).
• A repo can have both; run both detectors.

2. Safety first

The fix step edits test files in place. Both scripts enforce this themselves:
--fix exits 2 unless the target is a git repository with a clean working
tree (untracked files are fine). If it refuses, stash or commit rather than
reaching for --force — --force removes the only undo path there is
(git checkout -- <files>), so use it only when the user has explicitly
accepted that.

Trust boundary: scanning executes the project's own toolchain. mypy loads
[tool.mypy] plugins from the repo's config as in-process Python;
--mypy "uv run mypy" / "poetry run mypy" resolve (and can run) the
repo's dependencies; the TS side loads the repo's own typescript
package. Run the scan only on repositories you would be willing to run
mypy/tsc in yourself.

Note: when installed as a plugin, a write-time PreToolUse hook may also be
active — if a test-file Write/Edit is denied with a "captain-obvious:"
reason during cleanup rewrites, fix the flagged assertions instead of
re-trying the same content (see references/prevention.md). Both scanners
also support --file <path> [--stdin] for a syntactic-only single-file
scan (JSON to stdout; no mypy/tsc, no side effects).

3. Scan (report-only)

node <skill-dir>/scripts/captain_obvious_ts.mjs --project <repo> --json /tmp/co-ts.json
python3 <skill-dir>/scripts/captain_obvious_py.py --path <repo> --json /tmp/co-py.json

• The Python detector shells out to mypy for the type-guaranteed category. Use
  the project's own environment: pass --mypy "uv run mypy" for uv projects,
  --mypy "poetry run mypy" for poetry, etc. If mypy isn't available it
  degrades gracefully to the syntactic categories.
• Note: the mypy pass briefly writes _cap_obv_shadow_* copies next to test
  files (removed when the run ends) — so a "report-only" scan does touch the
  working tree. Pass --no-types for a strictly read-only scan; if the tree
  is not writable the scan degrades to syntactic categories and says so.
• The TS detector resolves the project's own typescript package; without a
  tsconfig it degrades to syntactic categories.
• If the project already produces coverage (or you can cheaply run it),
  pass --coverage <file> (lcov / istanbul coverage-final.json / coverage.py
  coverage json). This is the dynamic half of the ICSE'19 rotten-green
  analysis: a conditional-assert whose line never ran is promoted to proven
  rotten, and one that did run is dropped as a confirmed false positive. It
  turns the noisiest advisory category into a trustworthy one — use it whenever
  coverage is available.

Show the user the summary table and the findings before deleting anything.

4. Understand the two levels

• proven — cannot fail, by construction. The scripts guard the known
  escape hatches (any/unknown, as casts, !, index signatures, unchecked
  index access, structural instanceof, custom assertion helpers). Safe to
  auto-delete.
• advisory — almost certainly useless but not provable (assertion-free
  tests, structural instanceof, mock-echo variants, index-signature-backed
  checks, rotten-green conditional asserts, unawaited async assertions). The
  script never auto-deletes these, but it records exactly why each is
  uncertain, plus a deletable hint (aggressive = usually a deletion,
  report-only = usually needs a rewrite). That reason is a question you
  are equipped to answer against the surrounding code — so advisories are
  adjudicated by you (step 6), not dumped on the user.

See references/detectors.md for the full category catalog and the reasoning
behind each guard.

5. Fix the proven tier (deterministic)

node <skill-dir>/scripts/captain_obvious_ts.mjs --project <repo> --fix
python3 <skill-dir>/scripts/captain_obvious_py.py --path <repo> --fix

Plain --fix removes only the proven findings — no judgment required, no
LLM. This is the safe deterministic core; run it first.

6. Adjudicate the advisory tier (you decide, then confirm)

Advisories are the cases determinism can't settle — and that's your job, not
a report line for the user. Do not just forward the list. For each advisory
finding:

1. Read the test and the code it exercises. The finding's reason field is a
   pointed question — e.g. "structural instanceof — a shaped non-instance
   could sneak in" → check whether anything actually constructs a non-instance
   of that type; "mock-echo, indirect" → check whether a real code path runs
   between stub and assert.
2. Decide one of: delete (the doubt doesn't hold — it really is useless),
   keep (the doubt holds — it's a real check), or rewrite (the intent
   is valid but the assertion is broken). Rewrite is the advisory tier's real
   value: fix the unawaited .rejects (await it), narrow a
   pytest.raises(Exception) to the specific type, repair a rotten-green
   conditional-assert so it actually runs. Note no-assert findings are
   smoke tests — legitimate by design (ICSE'19); default to keep unless
   the test clearly meant to assert something and forgot.
3. Propose before acting. Present a compact per-item table — finding,
   verdict, one-line rationale, and the exact edit for rewrites — and apply
   only what the user approves. Never auto-delete or auto-rewrite an advisory.

For a large advisory set, delegate the per-item code reads to a Sonnet
subagent (batch the findings; have it return verdict + rationale + proposed
edit per item) and keep the final proposal/synthesis here — don't burn the main
loop reading files one by one. The proven tier is never handed to a subagent;
it's already decided.

7. Clean the residue

The scripts delete whole test blocks or individual assertion lines. That can
leave behind: unused imports/variables (noUnusedLocals will flag them),
empty describe() blocks, empty test classes, orphaned fixtures/mocks. Fix
those by hand — the typechecker output is your worklist.

8. Verify

Run the project's typecheck AND full test suite (tsc --noEmit + the test
command from package.json / pytest). Everything must pass with the same
result as before (minus the deleted tests). If anything regresses,
git checkout -- <files> and report what happened instead of pushing through.

9. Report

Tell the user: proven tests/assertions removed (per-category counts, lines
saved), the advisory verdicts you applied (deleted / rewritten, with the fix),
and anything you chose to keep with the reason the doubt held — that last
group is the tool earning trust, not failing.

What NOT to flag (the scripts already know, but so should you)

• toBeDefined() on .find() / Map.get() results — the type is T | undefined, the check is real.
• Enum/constant contract locks (expect(ExitCode.OK).toBe(0)) — they catch renumbering.
• Assertions on values read from files/APIs at test time — real regression tests.
• Tests asserting via custom helpers (expectAllow(x), self._check(...)).
• "Must not raise" contract tests for fail-open code paths.

When NOT to run this at all

• Mid red-green. During TDD a test is supposed to be failing, and a
  freshly-written test may not have its assertion yet. This is post-hoc
  cleanup — run it once the suite is green, never between red and green.
• On a branch under review. Scan (--json) is fine; --fix is not.
  Rewriting test files while a reviewer or a merge gate is reading the diff
  invalidates what they reviewed.
• As a coverage or CI-time optimizer. It deletes tests that cannot fail,
  which is a correctness argument, not a speed one. "CI is slow" is not a
  reason to reach for it — a slow suite full of real tests stays slow.