dev.workers.mike-tusa.robotsgate/robotsgate · v0.3.1 · search-and-web
RobotsGate is healthy: it completed an MCP handshake in 1541 ms and exposes 4 tools.
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Endpoint: https://robotsgate.mike-tusa.workers.dev/mcp (streamable-http)
No repository listed · Website · JSON · Add to my inventory
| Users (last 30 days) | 0 |
| Handshake | Healthy |
| Auth needed | None |
| Tool count | 4 |
| Latency | 1541 ms |
| Last checked (UTC) | 2026-10-16T14:55:49.745Z |
| Checked (UTC) | Status | Latency | Error |
|---|---|---|---|
| 2026-10-16T14:55:49.745Z | Healthy | 1541 |
list_crawlers | Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access |
generate_robots | Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns |
validate_robots | Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fi |
check_site | Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one G |