Skip to content
Hidden Logic Lab

Triage a live alert with an on-call first-response skill

Claude or Cursor follows a SKILL.md when a page fires: confirm signal, narrow blast radius, safe mitigations first, and a status update template before deep debugging.

·Operations·30 min·Working

A first-response skill linked from your paging runbook, a timestamped triage note in the incident channel, and explicit escalate/stop rules so agents do not thrash production.

  1. 1

    Paste a real redacted alert

    Tune verify lines against your dashboards. Synthetic alerts in the skill produce synthetic confidence.

  2. 2

    Wire the skill next to the pager runbook

    Link from PagerDuty/Opsgenie runbook URL and from ops SKILLS folder. First response and postmortem skills should cross-link but stay separate files.

  3. 3

    Drill the five-minute block

    Run a game day with a fake page. If mitigations in the skill do not match what leadership allows, fix policy before the next real fire.

The prompt to paste

Write SKILL.md for first-response triage when an on-call alert pages. This is live response, not a postmortem write-up. The reader may be an agent assisting a human or a human following the same steps.

Alert context (paste when using):
[alert name, service, severity, link to dashboard or runbook, first error message]

Environment:
[observability tools — names only: Datadog, Grafana, Sentry, CloudWatch]
[safe mitigations allowed without approval: scale replicas, disable feature flag, rollback deploy — list what your team allows]
[kubectl context or CLI patterns — names, not credentials]

SKILL.md must include:
1. Purpose — stabilize or gather facts in the first 15 minutes; do not root-cause chase before impact is bounded.
2. When to use — any page matching [alert family]; when not to use — test pages, known maintenance windows marked in calendar.
3. Severity and time box: if customer impact confirmed, escalate to [role] within [N] minutes — use [define] if unknown.
4. First five minutes — numbered, each ending with Verify::
   - Acknowledge page in the incident channel with alert link and your role.
   - Confirm the signal: is the metric/log error still firing or a stale alert?
   - Identify blast radius: which regions, tenants, or endpoints — query snippets from existing dashboards only; write [add dashboard link] for gaps.
   - Check recent deploys, flag changes, and dependency status in the last 60 minutes.
   - Choose one safe mitigation from the allowed list or explicitly write "no mitigation yet — gathering facts."
5. Communication template: Impact (known/unknown), Current action, Next update time (15 min default).
6. Agent-specific guardrails if Claude/Cursor is driving:
   - Read-only production queries unless the human approves a listed mitigation command.
   - Never paste secrets; never run destructive commands (delete pod, drop table, purge queue) without human typing approval.
   - Stop and hand off if two mitigations fail or error messages contradict the runbook.
7. Deep dive gate: only after impact is bounded or mitigated, link to service-specific runbook SKILL.md paths [list or placeholder].
8. Pitfalls: restarting everything, scaling to zero, flushing caches without understanding, muting alerts without fixing, sharing customer data in chat.
9. Done means: page acknowledged, impact statement updated once, mitigation or escalate decision recorded, handoff note if going off-call.

Rules:
- Do not invent dashboard URLs; use placeholders from alert paste or [add link].
- Do not fabricate incident timelines.
- Markdown only.

Agents speed up bad mitigations. The allowed-mitigation list and read-only default matter more than extra debugging steps.

Sample case written by Hidden Logic Lab. Not affiliated with Claude or Claude Code or Cursor.

More in Operations

Get new cases in your inbox

Weekly notes on new use cases and agent workflows. No product pitches.

Occasional notes on new cases. No product pitches in the first note.