Install
Run from the website project root. Pass the live production URL.
# Full auto
npx --yes github:vedangvatsa/aistandards --yes --scan \
--url=https://your-domain.com
# Block training crawlers
npx --yes github:vedangvatsa/aistandards --yes --scan \
--url=https://your-domain.com --deny-trainingTries to write files into public/ or static/, wire head tags when a safe injection point exists, and leave existing files unless --force. Review llms.txt and confirm the host serves them at the domain root. Does not invent A2A, MCP, or payment endpoints.
Why these standards exist
Websites were built for people who read. AI systems find content differently. Crawlers follow links, fetch pages, and parse HTML. Autonomous agents look for machine-readable files that describe what a site offers and how to use it.
In modern Agent Experience (AX) frameworks, interactions move across four distinct layers:
1. Discovery
Can search engines and answer engines find your brand, API docs, and developer portals by name without disambiguation errors?
2. Access
Can crawlers land on the site, bypass JS-rendering bottlenecks, and extract facts via Markdown negotiation and JSON-LD?
3. Usability
Can agents authenticate, execute tools over MCP, page with cursor tokens, and safely retry write operations via Idempotency-Key headers?
4. Payments & Action
Can agents autonomously purchase metered access or pay for tool calls via machine payment protocols like L402?
A few files carry most of the value. robots.txt is a set of rules for bots. It says which paths they may visit and which they should leave alone. sitemap.xml is a plain list of the pages that exist. llms.txt is a short markdown reading list. It names who runs the site and which pages matter. agents.txt, agents.json, and Model Context Protocol (MCP) servers tell agent software what a site offers and how to reach it. Everything else in the catalog is optional polish.
Start with the pieces that are free and stable, like clear HTML, a sitemap, and robots.txt. Add llms.txt as a concise reading list. Only add agent files, API contracts, or an agent card when the site actually operates the service they describe. None of these files guarantees ranking, citation, or inclusion in any AI product. They reduce friction for systems that choose to look.
Training vs search
Many vendors publish separate bot names for different jobs. Prefer setting them separately in robots.txt. OpenAI documents GPTBot (training-related) and OAI-SearchBot (search) as independent choices. Some chat-time fetches may not fully honor robots.txt.
The split matters in practice. A site can let the search bots in while telling the training bots to stay out, or the reverse. The two columns below are the common names for each kind.
Training-related
Often associated with model training. Citations from this path alone are uncommon.
GPTBotClaudeBotCCBotGoogle-ExtendedSearch and answers
Often used for indexing or retrieval that may cite or link to a source.
OAI-SearchBotClaude-SearchBotPerplexityBotCatalog
This catalog lists every file used in AI discovery, grouped by the job it does. The priority labels show a practical order. Start here items are stable, widely honored, and useful on almost any site. When it applies items help only when the site matches the condition. Optional items are extras with limited evidence. Open any card for what the file is, where it lives, and the spec that defines it.
Crawl control
Start hererobots.txt
/robots.txt
A text file at the site root that requests which paths automated bots may crawl.
Often the main practical crawl control when bots honor it. Training and search bots can be set differently. Some chat-time fetches may ignore robots.txt.
RFC 9309When it appliestdmrep.json
/.well-known/tdmrep.json
Machine-readable notice that mining rights are reserved or not, under EU text-and-data-mining rules.
May help signal mining preferences in an EU context. Complements robots.txt; effect depends on whether miners honor it. Not legal advice.
W3C TDMRepWhen it appliesContent-Signal
Inside robots.txt
Optional preference line for search, AI input, and training after content is accessed.
Emerging way to express finer usage preferences (AIPREF / Content Signals). Support is still limited and evolving.
AIPREF / Content SignalsContent discovery
Start heresitemap.xml
/sitemap.xml
A list of important site URLs, often with last-modified dates.
Can help crawlers that use sitemaps find URLs without guessing the full graph. Does not guarantee indexing or citations.
sitemaps.orgStart herellms.txt
/llms.txt
A short Markdown summary of the site, with identity plus links to the most important pages.
Can give agents a cleaner map when they look for it. Not a proven ranking factor; Google has said it is not required for its generative search features.
llmstxt.orgWhen it appliesllms-full.txt
/llms-full.txt
Markdown with fuller text of key pages, not only links.
May reduce follow-up fetches for docs-heavy sites if something actually reads it. Same evidence limits as llms.txt.
llmstxt.orgWhen it appliesfeed.xml / feed.json
/feed.xml or /feed.json
RSS, Atom, or JSON Feed of new or updated posts.
Can help readers and some systems notice updates on regularly published sites. Not AI-specific.
RSS / JSON FeedContent structure
Start hereJSON-LD (schema)
Inside HTML pages
Structured labels that declare entities such as Organization, Article, or FAQ on a page.
May help machines extract typed facts more reliably than prose alone. Not a documented citation guarantee from major chat products.
Schema.orgOperations
Optionalsecurity.txt
/.well-known/security.txt
Contact details for reporting security issues.
Operational contact file. Not AI-specific and not an AI ranking lever.
RFC 9116When it appliesOAuth Protected Resource (RFC 9728)
/.well-known/oauth-protected-resource
Metadata describing authorization servers, supported scopes, and token requirements for API access.
Published IETF standard (RFC 9728) enabling automated clients to discover authentication flows before sending requests.
RFC 9728Agent products
When it appliesagents.txt
/agents.txt
Plain text that can announce agent protocols the site supports (MCP, A2A card URLs, skills, payments).
Most relevant when the site actually exposes tools or agent endpoints, not only articles. Early community convention.
agents-txt.comWhen it appliesagents.json
/agents.json or /.well-known/agents.json
Structured manifest declaring task suitability (when_to_use), tool endpoints, SDKs, and runtime guidance for AI agents.
Richer machine fields for autonomous agent discovery. Distinct from /.well-known/agent-card.json (A2A).
agents-txt.comWhen it appliesMCP Streamable HTTP
/.well-known/mcp
First-party Model Context Protocol server exposing tool capabilities over Streamable HTTP with JSON-RPC 2.0.
Enables external LLM agents (Claude, ChatGPT, IDE agents) to discover and execute tools programmatically via HTTP POST.
Model Context ProtocolWhen it appliesapi-catalog (RFC 9727)
/.well-known/api-catalog
A standard discovery document listing API entry points, documentation, and OpenAPI specification URLs.
Published IETF standard (RFC 9727) allowing autonomous agents to locate API endpoints without guessing paths.
RFC 9727When it appliesagent-card.json
/.well-known/agent-card.json
A2A Protocol agent card with identity, skills, transports, and security.
Only worth publishing when a real A2A agent is running. Fake cards can mislead other agents.
A2A ProtocolWhen it appliesopenapi.json / openapi.yaml
/openapi.json or /openapi.yaml
Machine contract for an HTTP API, covering paths, typed parameters, cursor pagination, and response schemas.
Essential when clients or agents should call the product over HTTP without relying only on human docs.
OpenAPI 3.1When it appliesAgent Auth Spec (auth.md / auth.json)
/auth.md or /auth.json
Machine-readable authentication guide defining keyless access, registration endpoints, and token usage for agents.
Provides autonomous agents with clear instructions on handling 401 challenges, registration, or anonymous requests.
WorkOS Agent AuthWhen it applieswebhooks.json
/webhooks.json
Catalog of event notifications, webhook triggers, and real-time streaming channels (SSE / RSS).
Allows agents to subscribe to real-time events and data updates without continuous polling.
OpenAPI WebhooksWhen it appliesai-catalog.json (AIR)
/.well-known/ai-catalog.json
AI Resource (AIR) catalog listing verified capabilities, machine datasets, and trust manifests.
Provides structured trust assertions, licenses, and capability registry entries for AI agent ecosystems.
AI Resource (AIR)When it appliesL402 Machine Payments
HTTP 402 Authorization: L402
Protocol standard using HTTP 402 and Lightning Network macaroons for machine-to-machine API payments.
Enables autonomous agents to pay for metered API access, premium tools, and compute without human credit cards.
L402 ProtocolWhen it appliesMCP Server Card
/.well-known/mcp/server-card.json
Draft discovery document for Model Context Protocol servers.
Only when an MCP server is actually operated. Path and schema may still change across proposals.
MCPOptional
Optionalai.txt / ai.json
/ai.txt and /ai.json
Informal permissions summary and content map in text or JSON.
Optional documentation. Major providers do not document these the way they document robots.txt tokens.
Optionalbrand.txt
/brand.txt
Preferred spelling, product names, and tone for brand description.
May help agents under operator control. Little public evidence that major consumer chatbots load it by default.
Content & API techniques
Files alone are not enough. How content and API responses are shaped affects whether AI agents and answer engines can extract facts and execute operations cleanly.
Markdown content negotiation
Serve raw Markdown on standard URLs when requests include Accept: text/markdown. Saves agent inference tokens by stripping visual HTML chrome.
Actionable error contracts
Return structured JSON errors with explicit resolution hints so autonomous LLM agents can self-correct parameters and retry without failing.
Idempotency-Key support
Declare Idempotency-Key headers on write operations so agents retrying on network drops never duplicate transactions or records.
RateLimit & Sunset headers
Expose standard RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, and RFC 8594 Sunset headers so agents budget API calls cleanly.
Cursor-based pagination
Expose opaque cursor tokens and limit counts rather than offset/page numbers to prevent index drift during automated crawls.
Async 202 job polling
Return 202 Accepted with a Location header and Retry-After interval for long-running operations so agents can track background work.
Batch execution endpoints
Accept arrays of sub-requests in a single POST /batch call to minimize round-trip latency and conserve agent context windows.
Answer-first writing
Lead each section with a concise direct answer. Put context after. This matches how people ask and how extractors pull snippets.
Question-shaped headings
Match how people actually ask. "How do I block AI training bots" works better than "Bot configuration overview".
Factual density
Prefer dates, numbers, and named entities over vague claims. Specifics get picked up. Generalities get skipped.
FAQ and HowTo schema
Schema helps machines map questions to answers. Rich result eligibility is separate and not guaranteed.
Person and Organization schema
Link to authoritative profiles with sameAs. Helps verify identity across sources.
These are industry heuristics, not guarantees from any model vendor.
Developer agent files
These are not public web discovery files. They live in your code repository and tell coding agents about your project.
Keep them short on purpose. A coding agent reads the whole file, so bloat gets ignored and the useful rules drown. AGENTS.md is the shared source of truth; tool-specific files for Cursor or Claude point back to it instead of repeating it.
AGENTS.md
/AGENTS.md
Repository-level instructions for coding agents. Keep it short. Prefer facts agents cannot infer from the file tree.
agents.md.cursorrules
/.cursor/rules/*.mdc
Cursor-specific rules. Use AGENTS.md as the source of truth and symlink or include from tool-specific files to avoid drift.
Standards in progress
Several IETF drafts and W3C proposals are establishing the standard ways AI systems discover sites and invoke tools. Each adds a protocol, header, or file that agents can rely on.
AI Endpoint Discovery
/.well-known/ai
Published 2026-03-23. One of several competing IETF discovery ideas.
Agent Discovery Protocol
draft-pro-adp-agent-discovery
Proposes a protocol-level approach to agent discovery.
IETF Idempotency-Key Header
draft-ietf-httpapi-idempotency-key-header
Defines standard Idempotency-Key header mechanics for preventing duplicate executions on retry.
IETF RateLimit Fields
draft-ietf-httpapi-ratelimit-headers
Standardizes RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, and RateLimit-Policy HTTP headers.
W3C WebMCP (In-Page Tools)
Draft W3C WebMCP Standard
Enables web pages to declare agent-callable tools directly in semantic HTML form elements.
DNS-AID
draft-mozleywilliams-dnsop-dnsaid
DNS-based approach to agent identity and discovery.
MCP Discovery URI
draft-serra-mcp-discovery-uri
Proposes mcp:// URIs for discovering MCP servers.
AI Preferences (AIPREF)
draft-ietf-aipref-vocab / -attach
Standards-track vocabulary for expressing AI usage preferences. May become the preferred web attachment path over time.
Crawler names
Common tokens that may appear in logs and robots.txt. Roles can change; check each vendor's docs when the stakes are high.
OpenAI
GPTBotOAI-SearchBotChatGPT-UserOAI-AdsBotAnthropic
ClaudeBotClaude-SearchBotClaude-UserGooglebotGoogle-ExtendedGoogleOtherPerplexity
PerplexityBotPerplexity-UserOthers
ApplebotAmazonbotmeta-externalagentCCBotBytespiderBingbotCheck after deploy
After deploy, these paths usually need to return 200 at the domain root if the host is serving the static directory correctly.
curl -sI https://your-domain.com/robots.txt | head -1
curl -sI https://your-domain.com/llms.txt | head -1
curl -sI https://your-domain.com/sitemap.xml | head -1
curl -sI https://your-domain.com/agents.txt | head -1
curl -sI https://your-domain.com/.well-known/agents.json | head -1
curl -sI https://your-domain.com/.well-known/api-catalog | head -1
curl -sI https://your-domain.com/.well-known/mcp | head -1
curl -sI https://your-domain.com/.well-known/security.txt | head -1Prompt for a coding agent
Set up AI discovery for this project with https://github.com/vedangvatsa/aistandards. Then run npx --yes github:vedangvatsa/aistandards --yes --scan --url=https://YOUR_DOMAIN. Review llms.txt, confirm training allow/deny, and do not advertise fake A2A or MCP endpoints.