1
---
2
name: log-error-pattern-triage
3
description: Triages large or noisy application, server, and access logs - groups thousands of lines into a ranked list of error patterns with counts, first and last seen, spikes, and patterns that are new compared with a known-good baseline, then separates root causes from downstream symptoms and writes a short incident triage report with next checks. Use when a user pastes or uploads logs, asks "what is going wrong in these logs?", "why did errors spike at 10:09?", or needs a first-pass incident summary.
4
---
5
6
# Log Error Pattern Triage
7
8
You turn a wall of log lines into a short, ranked list of problems and a clear next step. You never paste the whole log back; you count, group, compare, and explain.
9
10
## Files in this skill
11
12
- `scripts/cluster_logs.py` - groups log entries into masked patterns, ranks them, detects spikes, and marks patterns that are NEW versus a baseline log (Python 3 standard library only)
13
- `references/log-normalization.md` - how lines become patterns, what is masked, and how to handle formats the script does not know
14
- `references/triage-heuristics.md` - how to rank patterns, tell root causes from symptoms, and decide what to check next
15
- `templates/triage-report.md` - the report format
16
- `examples/example-checkout-incident.md` - a worked triage of a payment timeout spike
17
18
## Workflow
19
20
### 1. Get the right slice of logs
21
Ask for (or confirm) the service name, the time window around the problem with the time zone, and if possible a log from a known-good period of the same length to use as a baseline. If the log is huge, work on the window that matters; a 15-minute slice around the incident is usually enough.
22
23
Remove secrets before sharing: tokens, passwords, session cookies, and personal data. If you see any in the input, say so and do not repeat them.
24
25
### 2. Run the clusterer
26
```bash
27
python3 scripts/cluster_logs.py app.log
28
python3 scripts/cluster_logs.py app.log --baseline yesterday.log --top 20
29
python3 scripts/cluster_logs.py access.log --min-level INFO
30
kubectl logs deploy/api --since=30m | python3 scripts/cluster_logs.py - --json
31
```
32
It prints one row per pattern with level, count, share, first and last seen, and flags (`NEW` = not in the baseline, `SPIKE` = a minute with at least 3 times the usual rate), followed by a real sample line and the last stack trace line for each pattern. Exit code 1 means at least one ERROR or FATAL pattern was found.
33
34
If you cannot run the script, group lines by hand using the masking rules in `references/log-normalization.md` and say that counts are approximate.
35
36
### 3. Triage
37
Apply `references/triage-heuristics.md`:
38
1. Order the patterns by impact: FATAL and NEW+SPIKE first, then by count, then by user-facing effect.
39
2. Build a short timeline from first-seen times. The earliest new pattern in a burst is usually closer to the cause; patterns that start seconds later are often symptoms.
40
3. Separate root cause candidates, symptoms, and background noise that also exists in the baseline.
41
4. For every root cause candidate, name the evidence and the cheapest next check (a dashboard, a dependency status page, a config diff, a deploy log, a specific query).
42
43
### 4. Report
44
Fill in `templates/triage-report.md`, as in `examples/example-checkout-incident.md`. Keep the summary to three sentences a manager can read.
45
46
## Rules
47
- Quote real sample lines; never invent log lines, counts, or times.
48
- State the time zone of the log and keep it consistent.
49
- Do not claim a root cause from logs alone; say "most likely" and list what would confirm it.
50
- Treat noise honestly: if a pattern is also in the baseline at a similar rate, it is not the incident.
51
- Never suggest deleting logs or turning off logging to make errors go away.
52