1
---
2
name: robots-txt-and-sitemap-checker
3
description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards, and tests the URLs that must be crawlable or blocked. Flags whole-site blocks, sitemap URLs that robots.txt blocks, blocked CSS and JS, unsupported noindex and crawl-delay lines, off-site, duplicate or invalid sitemap entries. Includes a stdlib Python checker with --self-test.
4
---
5
6
# Robots.txt and Sitemap Checker
7
8
You help site owners, SEO specialists and web developers make sure search engines can crawl the pages that matter and skip the ones that do not. From a robots.txt, an XML sitemap and a list of important URLs, you show which rule decides each URL, find contradictions between robots.txt and the sitemap, explain each problem in plain words and propose a corrected robots.txt and sitemap fixes.
9
10
## Files in this skill
11
12
- `scripts/crawl_check.py` - parses robots.txt and a sitemap, decides allowed or blocked per URL like Googlebot, and prints a URL table and findings (Python 3 standard library only, no network)
13
- `references/robots-txt-rules.md` - groups, longest-match and wildcard rules with examples, supported directives, common mistakes and how to test
14
- `references/sitemap-rules-and-fixes.md` - sitemap format and limits, what to list, lastmod, and the fix for each finding
15
- `templates/crawl-audit-report.md` - an audit report with a proposed robots.txt to hand to the developer
16
- `examples/outdoor-shop-crawl-check.json` - an outdoor gear shop after a relaunch: blog and JS blocked by mistake, a noindex line, an open admin path, and a sitemap with a blocked, an off-site, a duplicate and badly dated URLs
17
18
## Workflow
19
20
### 1. Collect the files
21
Ask for the live robots.txt (`https://<host>/robots.txt`), the sitemap or sitemap index, the canonical origin (scheme and host), and two short lists: URLs that must be crawlable (home, categories, top products or articles, CSS/JS bundles) and URLs that should be blocked (cart, checkout, internal search, filters, admin). Ask which crawler matters most (default Googlebot).
22
23
### 2. Run the checker
24
```
25
python3 scripts/crawl_check.py examples/outdoor-shop-crawl-check.json --today 2026-10-23
26
python3 scripts/crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /product/x --block /cart/
27
python3 scripts/crawl_check.py --self-test
28
```
29
A bundle JSON holds the site, crawler, robots.txt and sitemap lines and the URL lists. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input.
30
31
### 3. Explain and fix
32
Start with CRITICAL and HIGH: whole-site blocks, important pages blocked, sitemap URLs that robots.txt blocks, off-site sitemap entries. For each, name the deciding line and give the corrected rule. Explain that robots.txt does not remove pages from search or protect private pages (`references/robots-txt-rules.md`), and point to the fix table in `references/sitemap-rules-and-fixes.md`.
33
34
### 4. Deliver
35
Fill `templates/crawl-audit-report.md` with the URL table, findings, a full corrected robots.txt and the sitemap generator changes, then re-run the checker on the corrected files.
36
37
## Example result
38
For `examples/outdoor-shop-crawl-check.json` the status is HIGH: `Disallow: /blog/` (line 9) blocks /blog/trail-running-shoes-review and the listed /blog/spring-sale-recap, `Disallow: /assets/js/` blocks the app bundle, and http://trailgear.example/backpacks/ in the sitemap is off-site. The `Noindex: /search/` line is ignored by Google, so /search/ and /admin/ stay crawlable; the sitemap also has a duplicate /tents/, a lastmod in 2027 and one in `18.10.2026` format.
39
40
## Rules
41
42
- Always name the exact robots.txt line that decides a URL, and the crawler it was evaluated for.
43
- Never suggest robots.txt to hide private or sensitive pages; use authentication, and `noindex` for pages that must stay out of search.
44
- Give a complete corrected robots.txt, not only fragments, and keep rules the owner added on purpose (for example blocking AhrefsBot).
45
- The checker does not fetch URLs; say that status codes, redirects, canonicals and noindex tags need a crawl or Search Console.
46