1
---
2
name: time-series-gap-anomaly-checker
3
description: Checks CSV time series (metrics, sensor readings, sales by hour) for data problems before you chart or alert on them: missing intervals, duplicate or out-of-order timestamps, empty values, flatlines from stuck exporters, robust median/MAD spikes, and counter resets. Use when a dashboard looks wrong, before training a forecast, or when asking "is this data complete?". Includes a stdlib Python checker with --self-test.
4
---
5
6
# Time Series Gap and Anomaly Checker
7
8
You help analysts, SREs, and data engineers decide whether a time series can be trusted. You find where data is missing, repeated, stuck, or suspicious, explain the likely cause of each problem, and separate collection problems (the pipeline) from real events (the system being measured). You never silently fill gaps or delete outliers; you propose a treatment and say what it changes.
9
10
## Files in this skill
11
12
- `scripts/ts_check.py` - reads a CSV, infers the sampling interval, and reports gaps, duplicates, out-of-order rows, missing values, flatlines, spikes, and counter resets as markdown or JSON (Python 3 standard library only)
13
- `references/anomaly-methods.md` - how the checks work, how to tune thresholds, and when each method fails
14
- `references/gap-handling-playbook.md` - what to do with each kind of problem: fill, interpolate, drop, or annotate
15
- `templates/data-quality-report.md` - a report for a dashboard fix, incident review, or dataset hand-off
16
- `examples/api-metrics.csv` - three hours of 5-minute API metrics with a scrape outage, a duplicate, a stuck value, a latency spike, and a counter reset
17
18
## Workflow
19
20
### 1. Understand the series
21
Ask what each column measures, its unit, the expected sampling interval, and whether any column is a monotonic counter (requests_total, bytes_sent). Ask which time zone naive timestamps use; the script treats them as UTC.
22
23
### 2. Run the checker
24
```
25
python3 scripts/ts_check.py examples/api-metrics.csv --counter requests_total
26
python3 scripts/ts_check.py data.csv --time-col ts --cols temp_c,humidity --interval 10m
27
python3 scripts/ts_check.py data.csv --z 5 --window 12 --flatline-points 12 --json > report.json
28
python3 scripts/ts_check.py --self-test
29
```
30
Timestamps can be ISO 8601 or Unix epoch seconds or milliseconds. Exit code 1 when any issue is found, 2 on bad input, so it can run as a pipeline gate.
31
32
### 3. Read the results
33
- **Completeness** = rows present / rows expected over the range. Below 95% usually means aggregates and averages are biased.
34
- **gap**: spacing over 1.5x the interval. Long gaps at regular times often mean deploys or cron jobs; one long gap usually means an outage of the collector.
35
- **duplicate / out_of_order**: double scrapes, retries, or merged files.
36
- **flatline**: the same value repeated; real for on/off signals, suspicious for noisy measurements.
37
- **spike**: robust z-score against a rolling median. Check whether neighbouring columns spike at the same time (a real event) or not (a glitch).
38
- **reset**: a counter that went down, usually a process restart; rate calculations must handle it.
39
40
### 4. Recommend treatment
41
Use `references/gap-handling-playbook.md`. Tune thresholds with `references/anomaly-methods.md` if the data is seasonal, bursty, or very sparse.
42
43
### 5. Report
44
Fill `templates/data-quality-report.md` with the summary numbers, the top issues with times in the user's time zone, the suspected cause, and the agreed treatment.
45
46
## Rules
47
48
- Report problems; do not alter the source data. Write cleaned data to a new file and document every change.
49
- Do not call a spike an incident without corroboration (other metrics, logs, or a human).
50
- State the interval and thresholds you used so results are reproducible.
51
- Convert UTC times to the user's time zone when presenting them.
52