---
name: incident-triage
description: Triage the page, find the root cause, open the remedial PR, and write the post-mortem. Use this skill when: On the page — From an alert webhook or the incident channel.
---

# From incident to fix, without the scramble

Goal: triage the incident, open the fix, and write the post-mortem.

Connected tools this skill uses: github, linear, zendesk.

## When to use

- On the page — From an alert webhook or the incident channel.
- Or when the user asks directly to triage the incident, open the fix, and write the post-mortem.

## Method

A service is returning elevated errors. Investigate and fix.

1. Read the last hour of error logs and identify the failing endpoint and
   the error signature.
2. Correlate with the last few deploys to find the regression and the commit.
3. Open a PR that guards the failure path and adds a regression test.
4. File a P1 bug with the trace, draft a status note for the affected
   customers, and write a post-mortem once the fix ships.

Ask for approval before merging or rolling anything back.

## Root-cause tracing

Correlate the error signature with recent deploys to find the regression.

## Method
1. Read the last hour of error logs.
2. Identify the failing endpoint and the recurring error signature.
3. Pull the list of deploys in the same time window.
4. Correlate the error's onset time against each deploy's timestamp.
5. Narrow to the most likely deploy and commit.
6. Check the commit diff for a change that plausibly explains the error signature.

## Rules
- Correlate by time first, then confirm against the code diff — never name a cause from timing alone.
- Point to the specific commit and line that plausibly caused the error, not just "a recent deploy."
- If two deploys land in the same window, check both before choosing one.
- State a confidence level whenever the correlation is not conclusive.

## Output
A root-cause summary — failing endpoint, error signature, suspect commit, confidence — handed to PR and test authoring.

## PR + test authoring

Open a PR that guards the failing path and adds a regression test.

## Method
1. Locate the code path identified by root-cause tracing.
2. Write a guard that prevents the failure — validation, fallback, or bounds check.
3. Write a regression test that reproduces the original failure and confirms the guard fixes it.
4. Run the test suite locally to confirm the new test passes and nothing else breaks.
5. Open a PR containing the guard, the test, and a description linking the root cause.

## Rules
- Every fix ships with a regression test; never open a guard without a test proving it.
- The PR description states the root cause and the failing endpoint plainly.
- Never merge automatically — the PR always waits for human approval.
- Keep the guard scoped to the failing path; don't fold in unrelated refactors.

## Output
An open PR — guard, regression test, linked root-cause description — awaiting approval, plus the filed P1 bug.

## Post-mortem writing

Draft the RCA once the fix ships, held for review before it goes out.

## Method
1. Pull the incident timeline: alert time, triage start, root cause found, fix merged.
2. State customer impact: affected endpoint, duration, scope.
3. State the root cause in plain terms, linked to the commit.
4. List the fix and the regression test guarding against recurrence.
5. List any follow-up actions if the cause points to a broader gap.
6. Draft the customer-facing status note as a separate document from the internal RCA.

## Rules
- Draft only after the fix has shipped, never before.
- The customer status note and the internal post-mortem are separate documents, for separate audiences.
- Every timeline entry carries a timestamp; no approximate times.
- Follow-up actions stay open items — the post-mortem does not close them itself.

## Output
Two drafts held for review — an internal post-mortem (timeline, cause, fix, follow-ups) and a customer status note.

## Recurring jobs

This skill pairs with scheduled jobs. Offer to set each one up; only create the ones the user approves:

- Incident triage poll (Every 5 minutes, polling the alerts channel and pager feed) — https://eluu.ai/cookbook/incident-triage/jobs-setup/incident-triage-poll.md