dev.workers.quietworklabs.robots-check/robots-check · v1.0.0 · search-and-web
Robots.txt & AI Crawler Checker is healthy: it completed an MCP handshake in 514 ms and exposes 3 tools.
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Endpoint: https://robots-check.quietworklabs.workers.dev/mcp (streamable-http)
No repository listed · Website · JSON · Add to my inventory
| Users (last 30 days) | 0 |
| Handshake | Healthy |
| Auth needed | None |
| Tool count | 3 |
| Latency | 514 ms |
| Last checked (UTC) | 2026-10-18T09:30:35.479Z |
| Checked (UTC) | Status | Latency | Error |
|---|---|---|---|
| 2026-10-18T09:30:35.479Z | Healthy | 514 |
robots_check | Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line |
ai_crawler_access | AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, |
sitemap_check | Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Determini |