Skip to content
Hidden Logic Lab

Quarantine flaky tests with a triage SKILL.md

Cursor or Claude Code follows a SKILL.md from CI flake reports: isolate the flaky test, quarantine it with owner and ticket, propose root-cause checks for timing/state/order issues, and provide a re-enable checklist.

·Coding·30 min·Working

A flaky-test skill in the repo, a quarantine commit for each confirmed flake with ticket link, and a root-cause checklist for the owner to verify before re-enabling.

  1. 1

    Quarantine one known flake as proof

    Pick a test your team already knows is flaky and run it through the skill. The output should match what a human would document.

  2. 2

    Link the skill in CI failure runbooks

    Add to AGENTS.md or CI docs: when a test flakes on main, use this skill before the postmortem. One skill beats ten Slack threads.

  3. 3

    Audit quarantined tests monthly

    The skill proposes tickets and owners, but humans must close the loop. Set a recurring review for tests quarantined > 30 days.

The prompt to paste

Write SKILL.md for triaging and quarantining flaky tests. The agent helps isolate the flake, records the context, and proposes root causes without disabling CI checks permanently.

Repository context:
[test framework: Jest, pytest, RSpec, etc., and how to run one test file or describe block]
[how to quarantine: skip() / xit() / @pytest.mark.skip / pending with ticket link in comment]
[CI tool: GitHub Actions, CircleCI, etc., and how to view test logs/artifacts]
[flake threshold: what count or time window defines a flake — e.g. failed 2+ times in 7 days on main]

Flake report (paste when using):
[test name, file path, failed run links, error messages from last N failures, environment — OS, browser, Node version if relevant]

SKILL.md must include:
1. Purpose — quarantine confirmed flakes with context so they do not block unrelated PRs, while preserving evidence for root-cause fixes.
2. When to use — after a test fails intermittently on main or flakes in CI without code changes; when not to use — first-time failures or tests that fail consistently.
3. Hard rules:
   - Never delete a flaky test or comment out assertions — quarantine only.
   - Never mark the whole suite as allow-failures — isolate the one test.
   - Always link a ticket in the quarantine comment with owner and target date to fix or delete.
4. Required inputs: test file path, test name, failed run links (at least 2), error message diff if multiple failure modes.
5. Triage steps the agent must follow:
   - Confirm flake: run the test locally 10 times (or note if not reproducible locally) — paste command and results.
   - Quarantine: add skip/xit with comment format: "// FLAKE: [ticket] — [one-line symptom] — owner @username — quarantined [date]".
   - Root-cause hypothesis list (agent proposes, human confirms):
     a. Timing: missing await, race condition, setTimeout without sufficient buffer, animation not settled.
     b. Shared state: test order dependency, global variable pollution, database/fixture not reset between runs.
     c. External dependency: network request not mocked, real API rate limit, third-party service flake.
     d. Concurrency: parallelized test runner collision on filesystem, port, or shared resource.
     e. Environment: CI-only (different Node version, timezone, screen resolution for browser tests).
   - For each hypothesis with "likely" confidence, propose verification command or code inspection target.
6. PR comment template for quarantine commit:
   - Test name and file
   - Flake evidence: N failures in M days, links to at least 2 distinct failures
   - Failure modes: list error messages if they differ
   - Hypotheses: bulleted list with confidence level
   - Quarantine commit: SHA and file changed
   - Re-enable checklist: "Fixed hypothesis X, ran test 50 times locally, ran in CI 10 times on this branch, removed skip and re-enabled."
7. Two example prompts: React component test flaking on animation timing vs pytest with database rollback race.
8. Pitfalls: quarantining without ticket (flakes never get fixed), marking flake as fixed after one green run, assuming timing when the real issue is order dependence, ignoring CI-only flakes because "works on my machine."
9. Done means: test quarantined with skip + comment, ticket created and linked, root-cause hypotheses proposed, re-enable checklist documented, no "disable the whole job" shortcut.

Rules:
- Use real test names and file paths from the flake report; do not invent test names.
- Do not claim a test is fixed until the re-enable checklist is complete.
- Output only SKILL.md markdown.

Flakes that fail with different error messages have multiple root causes. The skill should list each failure mode separately, not pick one.

Sample case written by Hidden Logic Lab. Not affiliated with Cursor or Claude Code.

More in Coding

Get new cases in your inbox

Weekly notes on new use cases and agent workflows. No product pitches.

Occasional notes on new cases. No product pitches in the first note.