Redditreddit.com
Score breakdown
Evidence
- ✕robots.txt disallows 19 of 19 user-directed agents via a default Disallow: / for unlisted bots: ChatGPT-User, Claude-User, Perplexity-User, GoogleAgent-Mariner, GoogleAgent-Shopping, Manus-User, MistralAI-User, Devin…
- ✕Homepage blocks identified agents: ChatGPT-User (HTTP 403); serves Claude-User, Perplexity-User, GoogleAgent-Mariner
- ◐Also disallows 15 of 15 training/search crawlers (not scored; agents ≠ crawlers)
- ◐No Content-Signal in robots.txt
- ◐No llms.txt
- ◐No JSON-LD structured data on the homepage
- ◐Missing main/nav landmarks
- ◐No agent-protocol manifest (UCP, MCP, A2A, OpenAPI)
| Identified as | HTTP | Result |
|---|---|---|
ChatGPT-User | 403 | blocked · “Blocked” |
Claude-User | 200 | normal page |
Perplexity-User | 200 | normal page |
GoogleAgent-Mariner | 200 | normal page |
browser (human) | 200 | normal page |
What Reddit would need to change
1Stop disallowing identified user-directed agents in robots.txtAccess · up to 17 pts
You disallow 19 agents that act live for a person. Crawler rules (GPTBot, ClaudeBot) can stay; user-directed agents are the ones a customer sends.
User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: GoogleAgent-Mariner User-agent: GoogleAgent-Shopping User-agent: Manus-User Allow: /
2Give agents a front doorProtocols · up to 12 pts
A Universal Commerce Protocol manifest for shopping, or an MCP server card for everything else. An agent that can call an API never needs to hold a customer's password.
GET /.well-known/ucp
{ "ucp": { "version": "2026-01-23",
"services": { "dev.ucp.shopping": [{ "transport": "rest", "endpoint": "https://www.reddit.com/api/ucp/v1" }] },
"capabilities": { "dev.ucp.shopping.checkout": [ … ] } } }
# or
GET /.well-known/mcp.json
{ "name": "Reddit", "endpoint": "https://www.reddit.com/mcp", "transport": "streamable-http" }3Publish llms.txtSurface · up to 8 pts
A short markdown map of what the site does, where the important pages are, and how to search or check out.
/llms.txt # Reddit > One paragraph on what this site is for. - [Products](/products): how to search and filter - [Checkout](/checkout): steps and constraints
4Verify agents cryptographically instead of blocking them at the doorAccess · up to 6 pts
Your edge answers ChatGPT-User with a block or interstitial. Web Bot Auth (HTTP Message Signatures, RFC 9421) lets an agent prove who it is per request; Cloudflare, Akamai, AWS WAF and Vercel can verify it. Allow verified agents and keep challenging the rest.
# Cloudflare: Security → Bots → Verified bots → allow category "AI Agent" # Or verify yourself: fetch the agent's key from # https://<signature-agent>/.well-known/http-message-signatures-directory # and check the Signature / Signature-Input headers (tag="web-bot-auth")
5Add JSON-LD structured dataSurface · up to 6 pts
schema.org Organization on the homepage, Product + Offer on product pages, with price and availability.
<script type="application/ld+json">
{ "@context": "https://schema.org", "@type": "Organization", "name": "Reddit", "url": "https://www.reddit.com/" }
</script>6Use semantic landmarksSurface · up to 5 pts
main, nav, header, footer and labeled forms. Everything an accessibility audit already asks for.
<header>…</header> <nav aria-label="Primary">…</nav> <main>…</main> <form aria-label="Site search"><label for="q">Search</label><input id="q" name="q"></form>
7Say your stance with a Content-SignalConsent · up to 3 pts
Two lines in robots.txt end the guessing game. ai-input covers agents reading pages on a person's behalf.
User-Agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /