Scan a URL
The Index › Social & media › nytimes.com
F

NY Timesnytimes.com

22/ 100 · Agentic Usability Score
WalledScanned 2026-09-22 · local-node · method v0.1Edge: DataDome
Muse
Meta
?Untested
ChatGPT Agent
OpenAI
⊘Disallowed
Claude
Anthropic
⊘Disallowed
Comet
Perplexity
⊘Disallowed
Grok Bot
xAI
◌Not disallowed
Gemini
Google
▮Walled
Manus
Manus
◌Not disallowed
Instinct
Spear Street
?Untested

Score breakdown

Five pillars. Task completion is pending until the live task runner ships; the other four are rescaled to 100.
Accessrobots.txt stance + live door test
17/40
Consent & identity pathContent-Signal, published agent policy
0/10
Machine-readable surfacellms.txt, JSON-LD, landmarks, SSR
3/25
Agent protocolsUCP, MCP, A2A, OpenAPI
0/15
Task completionlive agent run: search → cart → checkout
pending

Evidence

Generated by the scanner from this run. Every line is reproducible with the open-source probe.
  • ✕robots.txt disallows 6 of 19 user-directed agents: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, meta-externalfetcher, DuckAssistBot
  • ✕Homepage returns a bot interstitial (HTTP 403, “nytimes.com”) to every non-browser client, agents included
  • ◐robots.txt names 66 user-agents
  • ◐Also disallows 13 of 15 training/search crawlers (not scored; agents ≠ crawlers)
  • ◐No Content-Signal in robots.txt
  • ◐No llms.txt
  • ◐No agent-protocol manifest (UCP, MCP, A2A, OpenAPI)
Identified asHTTPResult
ChatGPT-User403blocked · “Not Authorized - The New York Times”
Claude-User403blocked · “Not Authorized - The New York Times”
Perplexity-User403blocked · “Not Authorized - The New York Times”
GoogleAgent-Mariner403blocked · “nytimes.com”
browser (human)403interstitial · “nytimes.com”

What NY Times would need to change

1Verify agents cryptographically instead of blocking them at the doorAccess · up to 23 pts

Your edge answers ChatGPT-User, Claude-User, Perplexity-User, GoogleAgent-Mariner with a block or interstitial. Web Bot Auth (HTTP Message Signatures, RFC 9421) lets an agent prove who it is per request; Cloudflare, Akamai, AWS WAF and Vercel can verify it. Allow verified agents and keep challenging the rest.

# Cloudflare: Security → Bots → Verified bots → allow category "AI Agent"
# Or verify yourself: fetch the agent's key from
#   https://<signature-agent>/.well-known/http-message-signatures-directory
# and check the Signature / Signature-Input headers (tag="web-bot-auth")

2Give agents a front doorProtocols · up to 12 pts

A Universal Commerce Protocol manifest for shopping, or an MCP server card for everything else. An agent that can call an API never needs to hold a customer's password.

GET /.well-known/ucp
{ "ucp": { "version": "2026-01-23",
  "services": { "dev.ucp.shopping": [{ "transport": "rest", "endpoint": "https://www.nytimes.com/api/ucp/v1" }] },
  "capabilities": { "dev.ucp.shopping.checkout": [ … ] } } }

# or
GET /.well-known/mcp.json
{ "name": "NY Times", "endpoint": "https://www.nytimes.com/mcp", "transport": "streamable-http" }

3Publish llms.txtSurface · up to 8 pts

A short markdown map of what the site does, where the important pages are, and how to search or check out.

/llms.txt
# NY Times
> One paragraph on what this site is for.
- [Products](/products): how to search and filter
- [Checkout](/checkout): steps and constraints

4Stop disallowing identified user-directed agents in robots.txtAccess · up to 5 pts

You disallow 6 agents that act live for a person. Crawler rules (GPTBot, ClaudeBot) can stay; user-directed agents are the ones a customer sends.

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
User-agent: DuckAssistBot
Allow: /

5Say your stance with a Content-SignalConsent · up to 3 pts

Two lines in robots.txt end the guessing game. ai-input covers agents reading pages on a person's behalf.

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /