Daily harness signal

Pin the judge, not only the agent

July 26, 2026 · JST One fresh finding Verifier policy
An agent’s verifier is executable policy. Ruff 0.16 shows why both the tool version and its rule set must be pinned, upgraded on a branch, and admitted only after clean replay.
01 · Fresh · source date 2026-07-23

Version the acceptance oracle and its defaults

Use when: Your unattended coding agent treats a linter, formatter, type checker, or test runner as its acceptance oracle, especially when CI installs “latest” or configuration relies on upstream defaults.

Action: Pin the executable in the project lockfile and make the policy explicit. For Ruff migrations that need the pre-0.16 floor, add:

[tool.ruff.lint]
select = ["E4", "E7", "E9", "F"]

Then open an upgrade-only branch and run:

uvx [email protected] check . --statistics
uvx [email protected] check . --fix

The second command applies safe fixes only. Review that diff before any agent edits remaining violations; require a separate human-reviewed change for --unsafe-fixes. If Markdown formatting is not intended, explicitly exclude *.md from [tool.ruff.format]. Keep the full test suite as a separate gate.

Acceptance check: From a clean checkout, record ruff --version, the enabled settings (ruff check --show-settings path/to/canary.py), diagnostic counts, diff, and test result. Pass only if CI invokes the locked version, the configured rule set matches the reviewed baseline, safe-fix replay is idempotent, no unplanned Markdown changes appear, and lint plus the full tests exit zero twice. Fail promotion on any version, ruleset, file-scope, count, or exit-code drift.

Evidence: Ruff’s official 0.16.0 release changed its default from 59 to 413 rules and began formatting Python blocks in Markdown by default. Simon Willison’s previously green projects failed immediately with an unpinned dependency; one scan found 1,618 violations, of which 1,538 were auto-fixed and 80 remained. His inspectable LLM repair commit changed four files, while an open Ruff issue independently calls the default expansion disruptive for large repository fleets.

Caveat: Freezing the old rules forever discards useful bug and runtime checks. Treat the four-rule selection as a temporary compatibility floor, not the target policy. Expand categories deliberately, and remember that Ruff labels unsafe fixes unsafe because they may alter runtime behavior or remove comments.

Compact source notes

  1. Ruff 0.16.0 release and migration guide (2026-07-23). Official release, artifact attestations, breaking-change list, compatibility rule set, and Markdown-formatting behavior.
  2. Ruff fix-safety reference (retrieved 2026-07-26). Official distinction between safe and unsafe fixes, including runtime-behavior and comment-removal risks.
  3. Simon Willison’s field report (2026-07-25) and inspectable LLM repair commit. Concrete CI failures, counts, commands, test-matrix context, and resulting diff.
  4. Ruff issue #27177 (opened 2026-07-24 UTC). Independent fleet-maintainer failure report about non-fixable rules entering the expanded default set.