# AI Crawler Policy Monitor - Robots.txt Evidence — workflow

## Choose this tool

Publishers and AI data teams reviewing declared crawler access policies. Per-agent, per-path rule evaluation includes the source rules and fingerprint.

Useful output columns: `agent`, `testedPath`, `allowedByRobots`, `robotsHash`, `rules`.

## Practical workflow

Enter website origins, exact crawler tokens and paths you care about. Save a baseline and compare policy fingerprints and evaluations. Review the source rules before changing crawler configuration; declared policy is not legal permission.

## Check the fit

Evaluates supplied paths for GPTBot, ClaudeBot, Google-Extended and other selected tokens against a successfully fetched robots.txt. This is declared policy, not legal permission or observed crawler behavior. Missing/error files fail explicitly. No tested page is fetched.

## Real output sample

One selected row from a 2026-10-18 run. One saved row with selected scalar fields. Nested structures remain available in full Actor output.

Run: 0gs9aZiBQzKzgP5Ne. Summary status: ok. Incomplete: false.

Historical bounded output from the recorded run. Current source values may differ.

```json
{
  "recordId": "aade1dd02a1eddc204a2c1fe7d6b5e78021ac1f5d079c3c8faeb865d87f3514f",
  "title": "GPTBot /",
  "agent": "GPTBot",
  "testedPath": "/",
  "allowedByRobots": true,
  "robotsHash": "abe4a7beb88057ebc2e7ae2790d6304d7663b556eb2ead56e6b22173392b9c91",
  "url": "https://www.anthropic.com/robots.txt",
  "observedAt": "2026-10-18T15:34:02.465Z"
}
```