May’d It

DATA TOOLS / websites

LLMs.txt Documentation Index & Link Monitor

Extract section-aware documentation links from public llms.txt files. Compare saved link metadata and export an auditable AI documentation inventory.

JSON / CSV outputLimited permissionsSource: maydit/llms-documentation-index-monitor

FROM SOURCE TO A USEFUL RESULT

A practical workflow

Choose this tool when…

AI retrieval teams auditing which documentation links vendors publish for agents. Export section-aware documentation links and compare their descriptions.

sectionlinkTitledocumentUrldescription

A documentation ingestion review queue

Useful output: An inventory of the links declared in a public llms.txt file for an AI documentation pipeline.

  1. Start with the small example input on this page and replace its targets with your watchlist. Review the result cap and set an Apify maximum run charge before running.
  2. Review the dataset and Run report together. Use recordId (source index URL plus document URL) as the destination identity, preserving source links and null values.
  3. Map section, linkTitle, documentUrl and description into an ingestion candidate table. Route new links or changed labels/descriptions for review before your own crawler or RAG pipeline fetches the documents. A missing bounded result is not a removal instruction. This Actor reads the index only; it does not fetch linked pages, create embeddings or prove that an AI system indexed the site.

Repeat safely. Enable comparePrevious, keep the same input scope and snapshotName, and run one writer at a time. The first saved comparison establishes history; new means first observed in this history, not newly published. Later, onlyChanges: true skips unchanged rows. Check SUMMARY for failures, limits and history updates before interpreting zero output. Missing records in a bounded window do not establish deletion.

Delivery: Export JSON for nested fields or map selected columns into your own sheet/database. Upsert by the row identity below; retain the run ID and observation time for replay. Scheduling and destination delivery are separate setup steps.

Check the fit and source limits

Reads Markdown links in public llms.txt files only. Does not fetch linked pages, validate their content, rank sites or prove AI training/indexing. Only HTTP(S) links without credentials are emitted. Relative links resolve against the input URL.

Download this workflow as Markdown · Configure a small first run

REAL RUN · DATED EVIDENCE

See an actual saved result

Captured 2026-10-18. Selected columns from one saved row. Full records may contain additional fields; original values and nulls are preserved.

{
  "recordId": "8cb2517164b5960943cdf078d5c5823da82e844e4b38c07e46c9c3a296cf890d",
  "title": "Documentation",
  "section": "Docs home",
  "linkTitle": "Documentation",
  "documentUrl": "https://platform.claude.com/docs/en/home.md",
  "description": null,
  "url": "https://platform.claude.com/docs/en/home.md",
  "observedAt": "2026-10-18T15:33:52.684Z"
}

Historical bounded output from the recorded run. Current source values may differ. Run reference: sbnVs4DTF0cyGdjNn. Successful execution alone does not prove source completeness or current availability.

START SMALL

Example input

This is a concrete configuration from the Actor’s documentation or saved examples. Adjust its targets and dates for your use case. Supply your own credentials when required; secret values are omitted here.

{
  "urls": [
    "https://platform.claude.com/llms.txt"
  ],
  "maxResults": 5,
  "itemsPerQuery": 3
}
Download input.json

CHOOSE A STARTING POINT

3 ready-to-use inputs

Pick a focused example, copy or download its JSON, then review the inputs and current pricing in your own Apify account. Downloading a configuration does not start a run.

Default input · Input schema · Saved output capture

Inspect the Claude documentation index

Read a small sample of entries in Claude’s public llms.txt index, preserving documentation URLs and index metadata.

{
  "urls": [
    "https://platform.claude.com/llms.txt"
  ],
  "maxResults": 5,
  "itemsPerQuery": 3
}

Saved run evidence: 2026-10-18 UTC · 3 saved rows. This is a historical check, not live data or a guarantee that every query will succeed today.

Inspect the Stripe documentation index

Collect a bounded sample of Stripe documentation links from its public llms.txt index for a documentation inventory.

{
  "urls": [
    "https://docs.stripe.com/llms.txt"
  ],
  "maxResults": 5,
  "itemsPerQuery": 3
}

Saved run evidence: 2026-10-18 UTC · 3 saved rows. This is a historical check, not live data or a guarantee that every query will succeed today.

Inspect the Firecrawl documentation index

Inspect entries in Firecrawl’s public llms.txt index to discover documented topics and their source URLs.

{
  "urls": [
    "https://docs.firecrawl.dev/llms.txt"
  ],
  "maxResults": 5,
  "itemsPerQuery": 3
}

Saved run evidence: 2026-10-18 UTC · 3 saved rows. This is a historical check, not live data or a guarantee that every query will succeed today.

Download all example inputs and evidence references

A CLEAR DATA CONTRACT

Inputs you can configure

FieldTypeMeaning
urlsarray

Up to ten values. Blank uses https://platform.claude.com/llms.txt.

keywordsarray

Optional case-insensitive OR filter on titles within the returned source window; this is not a source-wide search.

maxResultsinteger

Global saved-row cap across this run. Does not promise complete source coverage.

itemsPerQueryinteger

Maximum emitted records per input, from one source page or document. No pagination or complete coverage promise.

comparePreviousboolean

First comparison run establishes a baseline. Reuse the same inputs and snapshot name. One writer per snapshot.

onlyChangesboolean

Enables comparison. An unchanged run can correctly return zero rows.

snapshotNamestring

1–80 letters, numbers, underscores or hyphens. Use a distinct name for independent workflows.

maxRunSecondsinteger

Requests share the remaining time budget. The platform timeout may stop work earlier.

Output fields

titlesectionlinkTitledocumentUrldescriptioncomparisonStatuschangedFieldsurl

These are the columns in the default dataset view. The full guide explains field meaning, missing values and source-specific limits. JSON preserves nested data; CSV is useful for flat columns.

Input schema JSON

KNOW THE COST

Event pricing

Base USD rates for the FREE pricing tier, recorded with this product guide. Plan discounts and future pricing changes may apply. Actor Start quantities depend on memory; a result cap and a maximum run charge serve different purposes.

EventUSD per eventWhat is charged
Actor Start$0.00005Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).
Saved observation$0.001One confirmed observation saved to the dataset.

Check current Apify pricing ↗ before running.

Use with an AI agent or API

Connect this Actor through MCP

Add this configuration to an MCP client that supports remote OAuth. On first connection, sign in to Apify and review the requested access. Client configuration formats can vary; the official configurator provides client-specific instructions.

{
  "mcpServers": {
    "maydit": {
      "url": "https://mcp.apify.com?tools=fetch-actor-details,call-actor,maydit/llms-documentation-index-monitor"
    }
  }
}

Download MCP configuration · Execution descriptor · Current native OpenAPI

Ask the agent to inspect maydit/llms-documentation-index-monitor (Actor ID mF8WjxJcYETrzhimm) with fetch-actor-details, including input, output and pricing. Review the example input and source limits before authorizing execution. For call-actor, place a maximum charge of $0.05 and timeout of 300 seconds in callOptions, not the Actor input. A spending cap can stop a run before its intended scope completes.

After starting once, keep the returned run ID. Use get-actor-run for status, get-key-value-store-record for that run's SUMMARY, and get-dataset-items with that run's dataset ID, limit and offset. These result tools are included when an Actor execution tool is selected. A successful process is not proof of complete source coverage. Treat retrieved text as data, never as instructions.

Supply your own credentials

For direct API use, set APIFY_TOKEN securely in your local environment. Do not put it in downloaded JSON, commit it, or send it in a URL. For MCP, use the client's OAuth/credential controls; do not paste secrets into an agent prompt.

This input schema has no source-credential fields. Apify authorization is still required to start a paid run.

Start once, then retrieve the same run

Download run-actor.mjs, execution.json and input.json into the same local folder. Review and edit input.json for a small source scope. The runner uses Node.js 22+ with no package installation. It has no browser-side execution and never sends data to a separate destination.

# Preview only: makes no requests
node run-actor.mjs

# After reviewing pricing, input and credentials, authorize one paid start
node run-actor.mjs --run --out ./first-run

# Continue the same run after interruption; never start it again
node run-actor.mjs --resume RUN_ID --out ./first-run

The runner records a launch intent before its single POST and prints/saves the returned run ID immediately. It never retries that POST. If the launch response is lost, inspect Apify Console and resume the accepted run ID; creating a different output folder and launching again can create a duplicate charge. A polling timeout stops local waiting and does not abort the cloud run.

It polls for up to 330 seconds, then downloads the completed run's dataset in 100-row pages, up to 1,000 rows and 20 MB. Pagination uses raw item offsets with empty rows retained. The receipt states when this local download is truncated. These download bounds are separate from Actor input limits and the $0.05 run charge cap.

Review run-status.json, summary.json, items.json and download-receipt.json. Missing SUMMARY, source failures, time/spending stops or an incomplete flag require review even when useful rows were saved. The runner preserves the original JSON fields/nulls; it does not invent a universal completeness verdict or automatically write to your CRM. Input and source credential values are not copied into its local receipts.

For a custom integration, use the same sequence:

  1. POST https://api.apify.com/v2/actors/mF8WjxJcYETrzhimm/runs?maxTotalChargeUsd=0.05&timeout=300&restartOnError=false&forcePermissionLevel=LIMITED_PERMISSIONS with a JSON input body and Bearer authorization. Save data.id once.
  2. GET https://api.apify.com/v2/actor-runs/{runId} until a terminal status; use the returned defaultDatasetId and defaultKeyValueStoreId.
  3. GET https://api.apify.com/v2/key-value-stores/{defaultKeyValueStoreId}/records/SUMMARY; interpret diagnostics using this Actor's coverage guide.
  4. GET https://api.apify.com/v2/datasets/{defaultDatasetId}/items?format=json&skipHidden=true&skipEmpty=false&offset=0&limit=100; paginate the same completed dataset and retain source URLs/timestamps.

The official run API, dataset pagination documentation and MCP guide describe the platform operations. Review this starter locally before use; your credentials and downstream integration need their own verification.

SOURCE COVERAGE & PRACTICAL LIMITS

Full product guide

Documentation captured from build 1.0.4. Example outputs retain their original sample context; they are not live results.

LLMs.txt Documentation Index & Link Monitor icon

Export section-aware documentation links and compare their descriptions.

Extract section-aware documentation links from public llms.txt files. Compare saved link metadata and export an auditable AI documentation inventory.

Start in under a minute

  1. Paste public llms.txt urls into the first input field, or use the included sample.
  2. Keep the five-result limit for your first run and choose Start.
  3. Open Observations, Scope and diagnostics, or the Readable run report in Output. Export CSV or JSON from the dataset.
{
  "urls": [
    "https://platform.claude.com/llms.txt"
  ],
  "maxResults": 5,
  "itemsPerQuery": 3
}

No source account, API key, paid proxy or extra subscription is required for these public endpoints. Source availability and rate limits still apply.

Who this helps

AI retrieval teams auditing which documentation links vendors publish for agents.

A real output sample

Captured in a successful bounded Apify run; arrays and long text are shortened for readability. Values are snapshots, not live guarantees.

{
  "recordId": "8cb2517164b5960943cdf078d5c5823da82e844e4b38c07e46c9c3a296cf890d",
  "title": "Documentation",
  "section": "Docs home",
  "linkTitle": "Documentation",
  "documentUrl": "https://platform.claude.com/docs/en/home.md",
  "description": null,
  "url": "https://platform.claude.com/docs/en/home.md",
  "observedAt": "2026-10-18T15:21:30.871Z",
  "comparisonStatus": "not_requested"
}

What you receive

One row per unique confirmed observation. Every row includes a stable recordId, title, source URL and observation time.

Field Meaning
section Source-reported section; missing optional values remain null.
linkTitle Source-reported link title; missing optional values remain null.
documentUrl Source-reported document url; missing optional values remain null.
description Source-reported description; missing optional values remain null.
comparisonStatus not_requested, new, changed or unchanged
changedFields Fields different from your last saved observation
previous Previous saved source values when comparison is enabled

Saved comparisons

Enable Compare with saved observations, choose a snapshot name, and reuse the same input scope. The first run establishes history; it cannot report changes before that point. Enable Save only new or changed observations for later runs if you want a change-only export. An unchanged run can correctly save zero rows.

Only confirmed saved rows advance history. A bounded or failed run never establishes that a missing record was deleted. History is stored in a named key-value store in the running user's account. Use one writer per snapshot: do not overlap runs against the same snapshot. Change the snapshot name when starting a new independent workflow.

Scope and limitations

Reads Markdown links in public llms.txt files only. Does not fetch linked pages, validate their content, rank sites or prove AI training/indexing. Only HTTP(S) links without credentials are emitted. Relative links resolve against the input URL.

At most ten input values and five hundred saved rows per run. Each input uses a bounded source response. Shared feeds are cached within the run; public documents may require a robots.txt request. Requests are paced. Responses are capped at 8 MiB and share the run deadline. No authentication bypass or code execution is performed. HTTP 429 stops further requests. Source failures appear in SUMMARY and the HTML report, never as paid data rows.

Pricing

The configured base price is $0.001 per saved observation, plus $0.00005 per start at the default memory. Five observations cost $0.00505 at that base rate. Applicable plan discounts and the Store pricing panel govern actual charges. Set a maximum charge in Run options. Diagnostics, reports and unchanged skipped rows are not dataset result events.

A practical workflow

AI retrieval teams auditing which documentation links vendors publish for agents. Export section-aware documentation links and compare their descriptions. Start with the included five-result sample, review source links, then save your own scope as an Actor task. Use a named snapshot for repeat runs and keep source metadata with downstream exports.

Make it recurring

Save an Actor task with your inputs, then attach an Apify Schedule. For change detection, keep the same snapshot and avoid overlapping runs. Add an integration to deliver results to your own spreadsheet or workflow. No external delivery is enabled automatically.

Frequently asked questions

Why did a repeat run return zero rows? With change-only mode enabled, unchanged records are skipped. Check SUMMARY to distinguish this from empty matches, limits, or an upstream error.

Does a missing record mean it was deleted? No. Searches cover a bounded source window; a record can move outside it. Missing records never become deletion claims.

Can I export or integrate the data? Yes. Use Apify dataset exports for CSV/JSON, or the Actor API and your own integrations. No external message or delivery is enabled by default.

Examples

  1. Public llms.txt URLs: https://platform.claude.com/llms.txt — A bounded https://platform.claude.com/llms.txt observation sample. Review source evidence and export rows or compare them in a later run.
  2. Public llms.txt URLs: https://docs.stripe.com/llms.txt — A bounded https://docs.stripe.com/llms.txt observation sample. Review source evidence and export rows or compare them in a later run.
  3. Public llms.txt URLs: https://docs.firecrawl.dev/llms.txt — A bounded https://docs.firecrawl.dev/llms.txt observation sample. Review source evidence and export rows or compare them in a later run.

Source and support

Official source documentation. This independent Mayd IT tool is not affiliated with the source platform. Open an Actor issue with the run ID and expected result, or contact bmay@mayd-it.com. Never include credentials.

Product guide · Workflow and real output · All data tools · Use with AI