index / uncategorized / x.com
F
14 / 100 · ● walled
x.com
pillars
worksa real agent runs one fixed task: landed, found, detail, actedpending
welcomeidentified agents: robots.txt stance, the door test, Content-Signal4/25
built for agentsllms.txt, JSON-LD, landmarks, server rendering, UCP, MCP, OpenAPI3/25
agents
Museuntested
ChatGPT Agentdisallowed
Claudedisallowed
Cometdisallowed
Grok Botdisallowed
Geminidisallowed
Manusdisallowed
Instinctuntested
evidence
- ×robots.txt disallows 17 of 19 user-directed agents via a default Disallow: / for unlisted bots: ChatGPT-User, Claude-User, Perplexity-User, GoogleAgent-Mariner, GoogleAgent-Shopping, Manus-User, MistralAI-User, Devin…
- –Homepage serves GoogleAgent-Mariner
- –robots.txt restricts some paths for Meta-ExternalFetcher, meta-externalfetcher
- –Also disallows 12 of 15 training/search crawlers (not scored; agents ≠ crawlers)
- –No Content-Signal in robots.txt
- –No llms.txt
- –No JSON-LD structured data on the homepage
- –Missing main/nav landmarks
- –No agent-protocol manifest (UCP, MCP, A2A, OpenAPI)
door test
- 402ChatGPT-User → 402
- 402Claude-User → 402
- 402Perplexity-User → 402
- 200GoogleAgent-Mariner → normal page
- 200browser user-agent → normal page
what would need to change
Give agents a front doorbuilt · up to 10 pts
A Universal Commerce Protocol manifest for shopping, or an MCP server card for everything else. An agent that can call an API never needs to hold a customer's password.
GET /.well-known/ucp
{ "ucp": { "version": "2026-01-23",
"services": { "dev.ucp.shopping": [{ "transport": "rest", "endpoint": "https://x.com/api/ucp/v1" }] },
"capabilities": { "dev.ucp.shopping.checkout": [ … ] } } }
# or
GET /.well-known/mcp.json
{ "name": "x.com", "endpoint": "https://x.com/mcp", "transport": "streamable-http" }Stop disallowing identified user-directed agents in robots.txtwelcome · up to 9 pts
You disallow 17 agents that act live for a person. Crawler rules (GPTBot, ClaudeBot) can stay; user-directed agents are the ones a customer sends.
User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: GoogleAgent-Mariner User-agent: GoogleAgent-Shopping User-agent: Manus-User Allow: /
Publish llms.txtbuilt · up to 4 pts
A short markdown map of what the site does, where the important pages are, and how to search or check out.
/llms.txt # x.com > One paragraph on what this site is for. - [Products](/products): how to search and filter - [Checkout](/checkout): steps and constraints
Say your stance with a Content-Signalwelcome · up to 3 pts
Two lines in robots.txt end the guessing game. ai-input covers agents reading pages on a person's behalf.
User-Agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /
Add JSON-LD structured databuilt · up to 3 pts
schema.org Organization on the homepage, Product + Offer on product pages, with price and availability.
<script type="application/ld+json">
{ "@context": "https://schema.org", "@type": "Organization", "name": "x.com", "url": "https://x.com/" }
</script>Use semantic landmarksbuilt · up to 3 pts
main, nav, header, footer and labeled forms. Everything an accessibility audit already asks for.
<header>…</header> <nav aria-label="Primary">…</nav> <main>…</main> <form aria-label="Site search"><label for="q">Search</label><input id="q" name="q"></form>
