Make every skill beat its own absence

Treat “without skill” as the control condition

Use when. A new or revised SKILL.md appears helpful in demonstrations, but nobody has shown that it improves the same model on the same tasks. Apply this before enabling the skill broadly, and again after any material instruction or reference change.

Action. Freeze representative cases, target model, judge, parameters, tool definitions, and workspace fixtures. Run every case twice: once with the skill loaded and once with it absent. The open-source agent-skills-eval runner provides the exact baseline switch and retains JSON/JSONL artifacts plus a static comparison report:

OPENAI_API_KEY=... npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline --strict

Use deterministic assertions for observable behavior—required tool calls, arguments, files, or schema—and reserve model judging for qualities that cannot be checked mechanically. Keep both outputs and run metadata so a later model or skill revision can be compared rather than remembered.

Acceptance check. Each case exposes matched with_skill and without_skill artifacts produced under identical settings. The skill passes only if it improves the predeclared task metric without regressing critical deterministic assertions. A structural lint pass, nicer prose, or a judge score with no visible control output is a failure.

Caveat. The runner demonstrates a sound comparison shape, not universal validity. A model judge can share the target’s blind spots, prompt sensitivity can swamp small gains, and one run cannot establish stability. Calibrate subjective judges against human labels, repeat noisy cases, and prefer held-out tasks over examples used while authoring the skill.