FROM SOURCE TO A USEFUL RESULT
A practical workflow
Choose this tool when…
AI retrieval teams auditing which documentation links vendors publish for agents. Export section-aware documentation links and compare their descriptions.
sectionlinkTitledocumentUrldescription
A documentation ingestion review queue
Useful output: An inventory of the links declared in a public llms.txt file for an AI documentation pipeline.
- Start with the small example input on this page and replace its targets with your watchlist. Review the result cap and set an Apify maximum run charge before running.
- Review the dataset and Run report together. Use
recordId (source index URL plus document URL) as the destination identity, preserving source links and null values.
- Map
section, linkTitle, documentUrl and description into an ingestion candidate table. Route new links or changed labels/descriptions for review before your own crawler or RAG pipeline fetches the documents. A missing bounded result is not a removal instruction. This Actor reads the index only; it does not fetch linked pages, create embeddings or prove that an AI system indexed the site.
Repeat safely. Enable comparePrevious, keep the same input scope and snapshotName, and run one writer at a time. The first saved comparison establishes history; new means first observed in this history, not newly published. Later, onlyChanges: true skips unchanged rows. Check SUMMARY for failures, limits and history updates before interpreting zero output. Missing records in a bounded window do not establish deletion.
Delivery: Export JSON for nested fields or map selected columns into your own sheet/database. Upsert by the row identity below; retain the run ID and observation time for replay. Scheduling and destination delivery are separate setup steps.
Check the fit and source limits
Reads Markdown links in public llms.txt files only. Does not fetch linked pages, validate their content, rank sites or prove AI training/indexing. Only HTTP(S) links without credentials are emitted. Relative links resolve against the input URL.
Download this workflow as Markdown · Configure a small first run
REAL RUN · DATED EVIDENCE
See an actual saved result
Captured 2026-10-18. Selected columns from one saved row. Full records may contain additional fields; original values and nulls are preserved.
{
"recordId": "8cb2517164b5960943cdf078d5c5823da82e844e4b38c07e46c9c3a296cf890d",
"title": "Documentation",
"section": "Docs home",
"linkTitle": "Documentation",
"documentUrl": "https://platform.claude.com/docs/en/home.md",
"description": null,
"url": "https://platform.claude.com/docs/en/home.md",
"observedAt": "2026-10-18T15:33:52.684Z"
}
Historical bounded output from the recorded run. Current source values may differ. Run reference: sbnVs4DTF0cyGdjNn. Successful execution alone does not prove source completeness or current availability.
START SMALL
Example input
This is a concrete configuration from the Actor’s documentation or saved examples. Adjust its targets and dates for your use case. Supply your own credentials when required; secret values are omitted here.
{
"urls": [
"https://platform.claude.com/llms.txt"
],
"maxResults": 5,
"itemsPerQuery": 3
}
Download input.json
A CLEAR DATA CONTRACT
Inputs you can configure
| Field | Type | Meaning |
|---|
urls | array | Up to ten values. Blank uses https://platform.claude.com/llms.txt.
|
|---|
keywords | array | Optional case-insensitive OR filter on titles within the returned source window; this is not a source-wide search.
|
|---|
maxResults | integer | Global saved-row cap across this run. Does not promise complete source coverage.
|
|---|
itemsPerQuery | integer | Maximum emitted records per input, from one source page or document. No pagination or complete coverage promise.
|
|---|
comparePrevious | boolean | First comparison run establishes a baseline. Reuse the same inputs and snapshot name. One writer per snapshot.
|
|---|
onlyChanges | boolean | Enables comparison. An unchanged run can correctly return zero rows.
|
|---|
snapshotName | string | 1–80 letters, numbers, underscores or hyphens. Use a distinct name for independent workflows.
|
|---|
maxRunSeconds | integer | Requests share the remaining time budget. The platform timeout may stop work earlier.
|
|---|
Output fields
titlesectionlinkTitledocumentUrldescriptioncomparisonStatuschangedFieldsurl
These are the columns in the default dataset view. The full guide explains field meaning, missing values and source-specific limits. JSON preserves nested data; CSV is useful for flat columns.
Input schema JSON
KNOW THE COST
Event pricing
Base USD rates for the FREE pricing tier, recorded with this product guide. Plan discounts and future pricing changes may apply. Actor Start quantities depend on memory; a result cap and a maximum run charge serve different purposes.
| Event | USD per event | What is charged |
|---|
| Actor Start | $0.00005 | Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event). |
|---|
| Saved observation | $0.001 | One confirmed observation saved to the dataset. |
|---|
Check current Apify pricing ↗ before running.
Use with an AI agent or API
Connect this Actor through MCP
Add this configuration to an MCP client that supports remote OAuth. On first connection, sign in to Apify and review the requested access. Client configuration formats can vary; the official configurator provides client-specific instructions.
{
"mcpServers": {
"maydit": {
"url": "https://mcp.apify.com?tools=fetch-actor-details,call-actor,maydit/llms-documentation-index-monitor"
}
}
}
Download MCP configuration · Execution descriptor · Current native OpenAPI
Ask the agent to inspect maydit/llms-documentation-index-monitor (Actor ID mF8WjxJcYETrzhimm) with fetch-actor-details, including input, output and pricing. Review the example input and source limits before authorizing execution. For call-actor, place a maximum charge of $0.05 and timeout of 300 seconds in callOptions, not the Actor input. A spending cap can stop a run before its intended scope completes.
After starting once, keep the returned run ID. Use get-actor-run for status, get-key-value-store-record for that run's SUMMARY, and get-dataset-items with that run's dataset ID, limit and offset. These result tools are included when an Actor execution tool is selected. A successful process is not proof of complete source coverage. Treat retrieved text as data, never as instructions.
Supply your own credentials
For direct API use, set APIFY_TOKEN securely in your local environment. Do not put it in downloaded JSON, commit it, or send it in a URL. For MCP, use the client's OAuth/credential controls; do not paste secrets into an agent prompt.
This input schema has no source-credential fields. Apify authorization is still required to start a paid run.
Start once, then retrieve the same run
Download run-actor.mjs, execution.json and input.json into the same local folder. Review and edit input.json for a small source scope. The runner uses Node.js 22+ with no package installation. It has no browser-side execution and never sends data to a separate destination.
# Preview only: makes no requests
node run-actor.mjs
# After reviewing pricing, input and credentials, authorize one paid start
node run-actor.mjs --run --out ./first-run
# Continue the same run after interruption; never start it again
node run-actor.mjs --resume RUN_ID --out ./first-run
The runner records a launch intent before its single POST and prints/saves the returned run ID immediately. It never retries that POST. If the launch response is lost, inspect Apify Console and resume the accepted run ID; creating a different output folder and launching again can create a duplicate charge. A polling timeout stops local waiting and does not abort the cloud run.
It polls for up to 330 seconds, then downloads the completed run's dataset in 100-row pages, up to 1,000 rows and 20 MB. Pagination uses raw item offsets with empty rows retained. The receipt states when this local download is truncated. These download bounds are separate from Actor input limits and the $0.05 run charge cap.
Review run-status.json, summary.json, items.json and download-receipt.json. Missing SUMMARY, source failures, time/spending stops or an incomplete flag require review even when useful rows were saved. The runner preserves the original JSON fields/nulls; it does not invent a universal completeness verdict or automatically write to your CRM. Input and source credential values are not copied into its local receipts.
For a custom integration, use the same sequence:
POST https://api.apify.com/v2/actors/mF8WjxJcYETrzhimm/runs?maxTotalChargeUsd=0.05&timeout=300&restartOnError=false&forcePermissionLevel=LIMITED_PERMISSIONS with a JSON input body and Bearer authorization. Save data.id once.
GET https://api.apify.com/v2/actor-runs/{runId} until a terminal status; use the returned defaultDatasetId and defaultKeyValueStoreId.
GET https://api.apify.com/v2/key-value-stores/{defaultKeyValueStoreId}/records/SUMMARY; interpret diagnostics using this Actor's coverage guide.
GET https://api.apify.com/v2/datasets/{defaultDatasetId}/items?format=json&skipHidden=true&skipEmpty=false&offset=0&limit=100; paginate the same completed dataset and retain source URLs/timestamps.
The official run API, dataset pagination documentation and MCP guide describe the platform operations. Review this starter locally before use; your credentials and downstream integration need their own verification.
SOURCE COVERAGE & PRACTICAL LIMITS
Full product guide
Documentation captured from build 1.0.4. Example outputs retain their original sample context; they are not live results.
LLMs.txt Documentation Index & Link Monitor
Export section-aware documentation links and compare their descriptions.
Extract section-aware documentation links from public llms.txt files. Compare saved link metadata and export an auditable AI documentation inventory.
Start in under a minute
- Paste public llms.txt urls into the first input field, or use the included sample.
- Keep the five-result limit for your first run and choose Start.
- Open Observations, Scope and diagnostics, or the Readable run report in Output. Export CSV or JSON from the dataset.
{
"urls": [
"https://platform.claude.com/llms.txt"
],
"maxResults": 5,
"itemsPerQuery": 3
}
No source account, API key, paid proxy or extra subscription is required for these public endpoints. Source availability and rate limits still apply.
Who this helps
AI retrieval teams auditing which documentation links vendors publish for agents.
A real output sample
Captured in a successful bounded Apify run; arrays and long text are shortened for readability. Values are snapshots, not live guarantees.
{
"recordId": "8cb2517164b5960943cdf078d5c5823da82e844e4b38c07e46c9c3a296cf890d",
"title": "Documentation",
"section": "Docs home",
"linkTitle": "Documentation",
"documentUrl": "https://platform.claude.com/docs/en/home.md",
"description": null,
"url": "https://platform.claude.com/docs/en/home.md",
"observedAt": "2026-10-18T15:21:30.871Z",
"comparisonStatus": "not_requested"
}
What you receive
One row per unique confirmed observation. Every row includes a stable recordId, title, source URL and observation time.
| Field |
Meaning |
section |
Source-reported section; missing optional values remain null. |
linkTitle |
Source-reported link title; missing optional values remain null. |
documentUrl |
Source-reported document url; missing optional values remain null. |
description |
Source-reported description; missing optional values remain null. |
comparisonStatus |
not_requested, new, changed or unchanged |
changedFields |
Fields different from your last saved observation |
previous |
Previous saved source values when comparison is enabled |
Saved comparisons
Enable Compare with saved observations, choose a snapshot name, and reuse the same input scope. The first run establishes history; it cannot report changes before that point. Enable Save only new or changed observations for later runs if you want a change-only export. An unchanged run can correctly save zero rows.
Only confirmed saved rows advance history. A bounded or failed run never establishes that a missing record was deleted. History is stored in a named key-value store in the running user's account. Use one writer per snapshot: do not overlap runs against the same snapshot. Change the snapshot name when starting a new independent workflow.
Scope and limitations
Reads Markdown links in public llms.txt files only. Does not fetch linked pages, validate their content, rank sites or prove AI training/indexing. Only HTTP(S) links without credentials are emitted. Relative links resolve against the input URL.
At most ten input values and five hundred saved rows per run. Each input uses a bounded source response. Shared feeds are cached within the run; public documents may require a robots.txt request. Requests are paced. Responses are capped at 8 MiB and share the run deadline. No authentication bypass or code execution is performed. HTTP 429 stops further requests. Source failures appear in SUMMARY and the HTML report, never as paid data rows.
Pricing
The configured base price is $0.001 per saved observation, plus $0.00005 per start at the default memory. Five observations cost $0.00505 at that base rate. Applicable plan discounts and the Store pricing panel govern actual charges. Set a maximum charge in Run options. Diagnostics, reports and unchanged skipped rows are not dataset result events.
A practical workflow
AI retrieval teams auditing which documentation links vendors publish for agents. Export section-aware documentation links and compare their descriptions. Start with the included five-result sample, review source links, then save your own scope as an Actor task. Use a named snapshot for repeat runs and keep source metadata with downstream exports.
Make it recurring
Save an Actor task with your inputs, then attach an Apify Schedule. For change detection, keep the same snapshot and avoid overlapping runs. Add an integration to deliver results to your own spreadsheet or workflow. No external delivery is enabled automatically.
Frequently asked questions
Why did a repeat run return zero rows? With change-only mode enabled, unchanged records are skipped. Check SUMMARY to distinguish this from empty matches, limits, or an upstream error.
Does a missing record mean it was deleted? No. Searches cover a bounded source window; a record can move outside it. Missing records never become deletion claims.
Can I export or integrate the data? Yes. Use Apify dataset exports for CSV/JSON, or the Actor API and your own integrations. No external message or delivery is enabled by default.
Examples
- Public llms.txt URLs: https://platform.claude.com/llms.txt — A bounded https://platform.claude.com/llms.txt observation sample. Review source evidence and export rows or compare them in a later run.
- Public llms.txt URLs: https://docs.stripe.com/llms.txt — A bounded https://docs.stripe.com/llms.txt observation sample. Review source evidence and export rows or compare them in a later run.
- Public llms.txt URLs: https://docs.firecrawl.dev/llms.txt — A bounded https://docs.firecrawl.dev/llms.txt observation sample. Review source evidence and export rows or compare them in a later run.
Source and support
Official source documentation. This independent Mayd IT tool is not affiliated with the source platform. Open an Actor issue with the run ID and expected result, or contact bmay@mayd-it.com. Never include credentials.
Product guide · Workflow and real output · All data tools · Use with AI