# AI Crawler Policy Monitor - Robots.txt Evidence Compare robots.txt policies for named AI crawlers and paths. Export allow/disallow evaluations, source rules and saved policy changes. Canonical product page: https://mayd-it.com/data-tools/actors/ai-crawler-policy-monitor/ ## Example input ```json { "origins": [ "https://www.anthropic.com" ], "maxResults": 5, "itemsPerQuery": 3 } ``` ## Choose this tool Publishers and AI data teams reviewing declared crawler access policies. Per-agent, per-path rule evaluation includes the source rules and fingerprint. Useful output columns: `agent`, `testedPath`, `allowedByRobots`, `robotsHash`, `rules`. ## Practical workflow Enter website origins, exact crawler tokens and paths you care about. Save a baseline and compare policy fingerprints and evaluations. Review the source rules before changing crawler configuration; declared policy is not legal permission. ## Check the fit Evaluates supplied paths for GPTBot, ClaudeBot, Google-Extended and other selected tokens against a successfully fetched robots.txt. This is declared policy, not legal permission or observed crawler behavior. Missing/error files fail explicitly. No tested page is fetched. ## Real output sample One selected row from a 2026-10-18 run. One saved row with selected scalar fields. Nested structures remain available in full Actor output. Run: 0gs9aZiBQzKzgP5Ne. Summary status: ok. Incomplete: false. Historical bounded output from the recorded run. Current source values may differ. ```json { "recordId": "aade1dd02a1eddc204a2c1fe7d6b5e78021ac1f5d079c3c8faeb865d87f3514f", "title": "GPTBot /", "agent": "GPTBot", "testedPath": "/", "allowedByRobots": true, "robotsHash": "abe4a7beb88057ebc2e7ae2790d6304d7663b556eb2ead56e6b22173392b9c91", "url": "https://www.anthropic.com/robots.txt", "observedAt": "2026-10-18T15:34:02.465Z" } ``` ## Full guide # AI Crawler Policy Monitor - Robots.txt Evidence AI Crawler Policy Monitor - Robots.txt Evidence icon **Per-agent, per-path rule evaluation includes the source rules and fingerprint.** Compare robots.txt policies for named AI crawlers and paths. Export allow/disallow evaluations, source rules and saved policy changes. ## Start in under a minute 1. Paste public website origins into the first input field, or use the included sample. 2. Keep the five-result limit for your first run and choose **Start**. 3. Open **Observations**, **Scope and diagnostics**, or the **Readable run report** in Output. Export CSV or JSON from the dataset. ```json { "origins": [ "https://www.anthropic.com" ], "maxResults": 5, "itemsPerQuery": 3 } ``` No source account, API key, paid proxy or extra subscription is required for these public endpoints. Source availability and rate limits still apply. ## Who this helps Publishers and AI data teams reviewing declared crawler access policies. ## A real output sample Captured in a successful bounded Apify run; arrays and long text are shortened for readability. Values are snapshots, not live guarantees. ```json { "recordId": "aade1dd02a1eddc204a2c1fe7d6b5e78021ac1f5d079c3c8faeb865d87f3514f", "title": "GPTBot /", "agent": "GPTBot", "testedPath": "/", "allowedByRobots": true, "robotsHash": "abe4a7beb88057ebc2e7ae2790d6304d7663b556eb2ead56e6b22173392b9c91", "rules": [ "User-Agent: *", "Allow: /" ], "url": "https://www.anthropic.com/robots.txt", "observedAt": "2026-10-18T15:22:03.435Z", "comparisonStatus": "not_requested" } ``` ## What you receive One row per unique confirmed observation. Every row includes a stable `recordId`, title, source URL and observation time. | Field | Meaning | |---|---| | `agent` | Source-reported agent; missing optional values remain null. | | `testedPath` | Source-reported tested path; missing optional values remain null. | | `allowedByRobots` | Source-reported allowed by robots; missing optional values remain null. | | `robotsHash` | Source-reported robots hash; missing optional values remain null. | | `rules` | Source-reported rules; missing optional values remain null. | | `comparisonStatus` | not_requested, new, changed or unchanged | | `changedFields` | Fields different from your last saved observation | | `previous` | Previous saved source values when comparison is enabled | ## Saved comparisons Enable **Compare with saved observations**, choose a snapshot name, and reuse the same input scope. The first run establishes history; it cannot report changes before that point. Enable **Save only new or changed observations** for later runs if you want a change-only export. An unchanged run can correctly save zero rows. Only confirmed saved rows advance history. A bounded or failed run never establishes that a missing record was deleted. History is stored in a named key-value store in the running user's account. Use one writer per snapshot: do not overlap runs against the same snapshot. Change the snapshot name when starting a new independent workflow. ## Scope and limitations Evaluates supplied paths for GPTBot, ClaudeBot, Google-Extended and other selected tokens against a successfully fetched robots.txt. This is declared policy, not legal permission or observed crawler behavior. Missing/error files fail explicitly. No tested page is fetched. At most ten input values and five hundred saved rows per run. Each input uses a bounded source response. Shared feeds are cached within the run; public documents may require a robots.txt request. Requests are paced. Responses are capped at 8 MiB and share the run deadline. No authentication bypass or code execution is performed. HTTP 429 stops further requests. Source failures appear in SUMMARY and the HTML report, never as paid data rows. ## Pricing The configured base price is **$0.001 per saved observation**, plus **$0.00005 per start** at the default memory. Five observations cost $0.00505 at that base rate. Applicable plan discounts and the Store pricing panel govern actual charges. Set a maximum charge in Run options. Diagnostics, reports and unchanged skipped rows are not dataset result events. ## A practical workflow Publishers and AI data teams reviewing declared crawler access policies. Per-agent, per-path rule evaluation includes the source rules and fingerprint. Start with the included five-result sample, review source links, then save your own scope as an Actor task. Use a named snapshot for repeat runs and keep source metadata with downstream exports. ## Make it recurring Save an Actor task with your inputs, then attach an Apify Schedule. For change detection, keep the same snapshot and avoid overlapping runs. Add an integration to deliver results to your own spreadsheet or workflow. No external delivery is enabled automatically. ## Frequently asked questions **Why did a repeat run return zero rows?** With change-only mode enabled, unchanged records are skipped. Check SUMMARY to distinguish this from empty matches, limits, or an upstream error. **Does a missing record mean it was deleted?** No. Searches cover a bounded source window; a record can move outside it. Missing records never become deletion claims. **Can I export or integrate the data?** Yes. Use Apify dataset exports for CSV/JSON, or the Actor API and your own integrations. No external message or delivery is enabled by default. ## Examples 1. **Public website origins: https://www.anthropic.com** — A bounded https://www.anthropic.com observation sample. Review source evidence and export rows or compare them in a later run. 2. **Public website origins: https://en.wikipedia.org** — A bounded https://en.wikipedia.org observation sample. Review source evidence and export rows or compare them in a later run. 3. **Public website origins: https://www.cloudflare.com** — A bounded https://www.cloudflare.com observation sample. Review source evidence and export rows or compare them in a later run. ## Source and support [Official source documentation](https://www.rfc-editor.org/rfc/rfc9309). This independent Mayd IT tool is not affiliated with the source platform. Open an Actor issue with the run ID and expected result, or contact bmay@mayd-it.com. Never include credentials. ## Workflow library and related tools [Product guide](https://mayd-it.com/data-tools/actors/ai-crawler-policy-monitor/) · [Workflow and real output](https://mayd-it.com/data-tools/actors/ai-crawler-policy-monitor/workflow.md) · [All data tools](https://mayd-it.com/data-tools/) · [Use with AI](https://mayd-it.com/data-tools/ai/)