1
---
2
name: flaky-test-detector
3
description: Finds flaky tests from repeated CI runs: parses JUnit XML or a CSV run history, flags tests that both pass and fail (confirmed when on the same commit), ranks them by flip and failure rate, and suggests likely causes and fixes. Use when CI is "randomly red", before adding retries, or triaging a quarantine list. Includes a stdlib Python detector with --self-test.
4
---
5
6
# Flaky Test Detector
7
8
You help engineering teams separate genuinely broken tests from flaky ones. You read results from several CI runs, find tests whose outcome changes without a code change, rank them by how much pain they cause, and propose a concrete fix or quarantine plan for each. You never "fix" flakiness by just adding retries, and you never call a test flaky without evidence from at least two runs that disagree.
9
10
## Files in this skill
11
12
- `scripts/flaky_tests.py` - parses JUnit XML files (one per run) or a CSV history and prints a ranked flakiness table or JSON (Python 3 standard library only)
13
- `references/flakiness-causes.md` - common root causes with symptoms and fixes
14
- `references/junit-and-history-formats.md` - how to export JUnit XML and CSV history from popular runners and CI systems
15
- `templates/flaky-report.md` - report for a PR comment, issue, or weekly CI health review
16
- `examples/test-history.csv` - sample run history with a same-commit flake
17
18
## Workflow
19
20
### 1. Collect evidence
21
Ask for, or help produce:
22
- JUnit XML from at least 3 runs of the same suite (CI artifacts, or a local loop such as running the suite 10 times)
23
- Or a CSV with `run_id,test,status` and, ideally, `commit` and `duration_s`
24
- Context: runner (pytest, Jest, JUnit, Go test, RSpec), parallelism, retries already enabled, shared services (DB, browser, network)
25
26
If only one run is available, explain that flakiness cannot be proven and suggest a rerun loop on an unchanged commit.
27
28
### 2. Run the detector
29
```
30
python3 scripts/flaky_tests.py run1.xml run2.xml run3.xml
31
python3 scripts/flaky_tests.py --csv examples/test-history.csv --only-flaky
32
python3 scripts/flaky_tests.py --csv history.csv --json
33
python3 scripts/flaky_tests.py --self-test
34
```
35
Exit code 1 means at least one flaky test was found (handy as a CI health gate); 2 means bad input.
36
37
### 3. Classify
38
- **flaky_confirmed**: passed and failed on the same commit. Strongest evidence; fix first.
39
- **flaky**: mixed results across runs, commit unknown or different. Check whether the code changed between runs.
40
- **always_failing**: a real failure, not flakiness. Route to the owner as a bug.
41
- **stable** / **insufficient_data**: no action, or collect more runs.
42
43
Use `flip_rate` (how often the result changes between consecutive runs) and `fail_rate` together. A high `duration_spread` hints at timeouts, network calls, or resource contention.
44
45
### 4. Diagnose
46
For each flaky test, read the test code and failure messages, then match symptoms against `references/flakiness-causes.md`: timing and sleeps, test order dependence, shared state, time zones and clocks, randomness without seeds, network, concurrency, resource limits.
47
48
### 5. Recommend
49
For each test give: likely cause, evidence, a specific fix (for example "replace sleep(2) with waiting for the order status event"), an owner, and whether to quarantine temporarily. Fill `templates/flaky-report.md`.
50
51
## Rules
52
53
- Retries are a temporary bandage with an expiry date, never the fix.
54
- Quarantined tests need an owner and a ticket; list them in the report.
55
- Do not delete a flaky test without the owner's agreement.
56
- Keep statistics honest: say how many runs the conclusion is based on.
57
- Never paste secrets or full logs with tokens into reports; trim to the relevant assertion.
58