io.github.kriton-dev/ai-crawler-access-check · v1.0.2 · search-and-web
AI Crawler Checker — Robots.txt Policy for Six Bots is up but requires authentication (oauth).
Check declared robots.txt permissions for six AI crawlers at URL paths
Endpoint: https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check (streamable-http)
No repository listed · Website · JSON · Add to my inventory
| Users (last 30 days) | 0 |
| Handshake | Needs auth |
| Auth needed | oauth |
| Tool count | – |
| Latency | – |
| Last checked (UTC) | 2026-11-03T14:25:23.843Z |
| Checked (UTC) | Status | Latency | Error |
|---|---|---|---|
| 2026-11-03T14:25:23.843Z | Needs auth | 265 | initialize: HTTP 401 |
Tools are listed after a successful handshake.