Install
Run from the website project root. Pass the live production URL.
# Full auto
npx --yes github:vedangvatsa/aistandards --yes --scan \
--url=https://your-domain.com
# Block training crawlers
npx --yes github:vedangvatsa/aistandards --yes --scan \
--url=https://your-domain.com --deny-trainingTries to write files into public/ or static/, wire head tags when a safe injection point exists, and leave existing files unless --force. Review llms.txt and confirm the host serves them at the domain root. Does not invent A2A, MCP, or payment endpoints.
Training vs search
Many vendors publish separate bot names for different jobs. Prefer setting them separately in robots.txt. OpenAI documents GPTBot (training-related) and OAI-SearchBot (search) as independent choices. Some chat-time fetches may not fully honor robots.txt.
Training-related
Often associated with model training. Citations from this path alone are uncommon.
GPTBotClaudeBotCCBotGoogle-ExtendedSearch and answers
Often used for indexing or retrieval that may cite or link to a source.
OAI-SearchBotClaude-SearchBotPerplexityBotCatalog
Crawl control
Start hererobots.txt
/robots.txt
A text file at the site root that requests which paths automated bots may crawl.
Often the main practical crawl control when bots honor it. Training and search bots can be set differently. Some chat-time fetches may ignore robots.txt.
RFC 9309When it appliestdmrep.json
/.well-known/tdmrep.json
Machine-readable notice that mining rights are reserved or not, under EU text-and-data-mining rules.
May help signal mining preferences in an EU context. Complements robots.txt; effect depends on whether miners honor it. Not legal advice.
W3C TDMRepWhen it appliesContent-Signal
Inside robots.txt
Optional preference line for search, AI input, and training after content is accessed.
Emerging way to express finer usage preferences (AIPREF / Content Signals). Support is still limited and evolving.
AIPREF / Content SignalsContent discovery
Start heresitemap.xml
/sitemap.xml
A list of important site URLs, often with last-modified dates.
Can help crawlers that use sitemaps find URLs without guessing the full graph. Does not guarantee indexing or citations.
sitemaps.orgStart herellms.txt
/llms.txt
A short Markdown summary of the site: identity plus links to the most important pages.
Can give agents a cleaner map when they look for it. Not a proven ranking factor; Google has said it is not required for its generative search features.
llmstxt.orgWhen it appliesllms-full.txt
/llms-full.txt
Markdown with fuller text of key pages, not only links.
May reduce follow-up fetches for docs-heavy sites if something actually reads it. Same evidence limits as llms.txt.
llmstxt.orgWhen it appliesfeed.xml / feed.json
/feed.xml or /feed.json
RSS, Atom, or JSON Feed of new or updated posts.
Can help readers and some systems notice updates on regularly published sites. Not AI-specific.
RSS / JSON FeedContent structure
Start hereJSON-LD (schema)
Inside HTML pages
Structured labels that declare entities such as Organization, Article, or FAQ on a page.
May help machines extract typed facts more reliably than prose alone. Not a documented citation guarantee from major chat products.
Schema.orgOperations
Optionalsecurity.txt
/.well-known/security.txt
Contact details for reporting security issues.
Operational contact file. Not AI-specific and not an AI ranking lever.
RFC 9116Agent products
When it appliesagents.txt
/agents.txt
Plain text that can announce agent protocols the site supports (MCP, A2A card URLs, skills, payments).
Most relevant when the site actually exposes tools or agent endpoints, not only articles. Early community convention.
agents-txt.comWhen it appliesagents.json
/agents.json
Structured companion to agents.txt at the site root (not the A2A card path).
Richer machine fields for the agents.txt convention. Distinct from /.well-known/agent-card.json (A2A).
agents-txt.comWhen it appliesagent-card.json
/.well-known/agent-card.json
A2A Protocol agent card: identity, skills, transports, security.
Only worth publishing when a real A2A agent is running. Fake cards can mislead other agents.
A2A ProtocolWhen it appliesopenapi.json / openapi.yaml
/openapi.json or /openapi.yaml
Machine contract for an HTTP API: paths, parameters, responses.
Useful when clients or agents should call the product over HTTP without relying only on human docs.
OpenAPI 3.1When it appliesMCP Server Card
/.well-known/mcp/server-card.json
Draft discovery document for Model Context Protocol servers.
Only when an MCP server is actually operated. Path and schema may still change across proposals.
MCPOptional
Optionalai.txt / ai.json
/ai.txt and /ai.json
Informal permissions summary and content map in text or JSON.
Optional documentation. Major providers do not document these the way they document robots.txt tokens.
Optionalbrand.txt
/brand.txt
Preferred spelling, product names, and tone for brand description.
May help agents under operator control. Little public evidence that major consumer chatbots load it by default.
Content techniques
Files alone are not enough. How content is written affects whether answer engines can extract and cite it.
Answer-first writing
Lead each section with a concise direct answer. Put context after. This matches how people ask and how extractors pull snippets.
Question-shaped headings
Match how people actually ask. "How do I block AI training bots" works better than "Bot configuration overview".
Factual density
Prefer dates, numbers, and named entities over vague claims. Specifics get picked up. Generalities get skipped.
FAQ and HowTo schema
Schema helps machines map questions to answers. Rich result eligibility is separate and not guaranteed.
Person and Organization schema
Link to authoritative profiles with sameAs. Helps verify identity across sources.
Multi-format presentation
Prose plus tables plus schema gives extractors multiple paths to the same fact. Reduces ambiguity.
These are industry heuristics, not guarantees from any model vendor.
Developer agent files
These are not public web discovery files. They live in your code repository and tell coding agents about your project.
AGENTS.md
/AGENTS.md
Repository-level instructions for coding agents. Keep it short. Prefer facts agents cannot infer from the file tree.
agents.md.cursorrules
/.cursor/rules/*.mdc
Cursor-specific rules. Use AGENTS.md as the source of truth and symlink or include from tool-specific files to avoid drift.
Standards landscape
Several IETF drafts are competing to become the standard way AI systems discover sites. None are RFCs yet. Track them if this space matters to your work.
AI Endpoint Discovery
/.well-known/ai
Published 2026-03-23. One of several competing IETF discovery ideas.
Agent Discovery Protocol
draft-pro-adp-agent-discovery
Proposes a protocol-level approach to agent discovery.
DNS-AID
draft-mozleywilliams-dnsop-dnsaid
DNS-based approach to agent identity and discovery.
MCP Discovery URI
draft-serra-mcp-discovery-uri
Proposes mcp:// URIs for discovering MCP servers.
AI Preferences (AIPREF)
draft-ietf-aipref-vocab / -attach
Standards-track vocabulary for expressing AI usage preferences. May become the preferred web attachment path over time.
Crawler names
Common tokens that may appear in logs and robots.txt. Roles can change; check each vendor's docs when the stakes are high.
OpenAI
GPTBotOAI-SearchBotChatGPT-UserOAI-AdsBotAnthropic
ClaudeBotClaude-SearchBotClaude-UserGooglebotGoogle-ExtendedGoogleOtherPerplexity
PerplexityBotPerplexity-UserOthers
ApplebotAmazonbotmeta-externalagentCCBotBytespiderBingbotCheck after deploy
After deploy, these paths usually need to return 200 at the domain root if the host is serving the static directory correctly.
curl -sI https://your-domain.com/robots.txt | head -1
curl -sI https://your-domain.com/llms.txt | head -1
curl -sI https://your-domain.com/sitemap.xml | head -1
curl -sI https://your-domain.com/agents.txt | head -1
curl -sI https://your-domain.com/.well-known/security.txt | head -1Prompt for a coding agent
Set up AI discovery for this project using:
https://github.com/vedangvatsa/aistandards
Run:
npx --yes github:vedangvatsa/aistandards --yes --scan --url=https://YOUR_DOMAIN
Review llms.txt, confirm training allow/deny, and do not advertise fake A2A or MCP endpoints.