Gate agent work with a checked-in eval harness skill
Cursor or Claude Code authors a SKILL.md that points at versioned eval cases and a pass/fail rubric so coding agents cannot green a task by vibes alone.
An evals/ folder, a skill that runs cases before merge, and a human-readable scorecard you can paste into the PR.
- 1
Write cases from real failures
Each case should come from a mistake you already saw in agent output. Synthetic happy-path cases only catch regressions you imagined.
- 2
Commit evals beside the skill
Put evals/ and the skill under the same skills root. The runner command in SKILL.md must match package.json or Makefile verbatim.
- 3
Block merge on the scorecard
Paste the scorecard into the PR and require a human to ack any failing case. Vibes-based 'I tested it' does not replace the runner output.
The prompt to paste
Write SKILL.md for an agent eval harness in this repository. The skill forces the agent to run defined cases and report pass/fail against a rubric — not self-grade with prose. Repo context: [language, how agents are invoked, test or script folder, CI job name if any] Eval scope (one job): [e.g. "support reply drafts", "SQL migration helper", "internal codegen template"] Deliver: ## SKILL.md Sections in order: 1. Purpose — one sentence: no task is done until eval cases pass or failures are listed explicitly. 2. When to use / when not to use (include: stop if eval runner command is missing). 3. Required inputs: branch or diff scope, path to evals/, and the rubric file name. 4. Eval layout the agent must respect: - evals/<job>/cases/*.yaml or *.json — each case: id, input, expected_checks (not necessarily exact string match; describe check type). - evals/<job>/rubric.md — how to score partial credit and what counts as automatic fail. 5. Steps: - Read rubric.md before editing code or prompts. - Run the eval runner command: [paste real command or write [confirm command]]. - For each failing case, fix the implementation or the case (never delete a case to green without human approval noted in PR). - Produce a scorecard markdown block: case id, pass/fail, one-line reason citing rubric section. 6. Two example prompts that invoke this skill with the eval path filled in. 7. Pitfalls: changing expected output without updating rubric, skipping slow cases, "looks good" without runner output, evals that only test happy path. 8. Done means: checklist including "runner exit 0" or "failures documented with case ids" and "scorecard pasted in PR." ## Starter files One example case file and a minimal rubric.md excerpt (three criteria) suitable for the scope above. Use placeholders, not fake metrics. Rules: - Do not invent pass rates or historical scores. - Do not claim integration with a vendor eval product unless named in repo context. - Output SKILL.md first, then example case + rubric excerpt.
Rubrics that say 'reasonable' or 'high quality' are not machine-checkable. Every criterion needs an observable check or an explicit human sign-off line.
Sample case written by Hidden Logic Lab. Not affiliated with Cursor or Claude Code.
More in Coding
Get new cases in your inbox
Weekly notes on new use cases and agent workflows. No product pitches.