Cloudflare reported on 1 July 2026 that 52% of crawler requests it sees are now for AI training, up from 22% in spring 2025, and that more than half of all internet traffic is non-human. Inside that half sits a smaller, commercially interesting slice: user-directed shopping agents, sent by someone who wants to buy. This is for e-commerce, travel and marketplace teams who need to know what those agents can actually reach on their store: a seven-signal scoring pass over the access layer, ending with four real retailers scored on the two signals anyone can verify from outside. You finish with a number out of 14 and a ranked list of what to fix first.
What this teardown covers, and what it does not
We score one question: can a user-directed agent reach and complete the steps a purchase needs on your normal site? Product attribute quality and feed hygiene are out of scope, and so is the measurement of AI-referred sessions; both have their own guides.
Start with access rather than a protocol integration, because the branded rails are narrower than the coverage suggests. OpenAI launched Instant Checkout on 29 September 2025 with Etsy sellers in the United States, then scaled it back in March 2026 and moved to retailer-built apps that hand checkout back to the retailer's own site. Google's Universal Commerce Protocol, published 11 January 2026 at version 2026-01-11, added Universal Cart at Google Marketing Live on 20 May 2026, but rollout is United States first, then Canada and Australia, then the United Kingdom. For a merchant in Turkey, the Gulf or most of Europe, neither rail is a 2026 plan. Your normal site is your agent surface.
The seven signals, scored 0 to 2
Score each signal 0 (absent or accidentally blocking), 1 (partial), or 2 (deliberate and complete). Maximum 14.
| # | Signal | What a 2 looks like | How to check |
|---|---|---|---|
| 1 | Agent-class user agents named separately from training crawlers in robots.txt | Training, search and agent tokens each get their own block with an intended answer | Read your own file. Anthropic documents ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for "when individuals ask questions to Claude" |
| 2 | The pages a purchase needs are reachable | Product detail and cart paths are not disallowed for the agent classes you want | Match disallow patterns against live PDP and cart URL templates, not assumed ones |
| 3 | Your CDN or WAF policy agrees with your robots.txt | The agent category is allowed or blocked as a decision, not by default | Cloudflare classifies verified bots as Search, Agent ("user-directed agents visiting a page on behalf of a human") and Training, one policy each. Defaults reportedly changed on 15 September 2026: Agent and Training blocked on ad-bearing pages |
| 4 | Signed-agent verification is possible | The origin verifies HTTP Message Signatures rather than IP ranges or user-agent strings | Web Bot Auth and Visa's Trusted Agent Protocol both use RFC 9421 signatures with keyid, alg, nonce and an expiry. Confirm where verification happens |
| 5 | A machine-readable commerce manifest exists | A discoverable capability document an agent reads before transacting | UCP expects capabilities exposed at /.well-known/ucp. Request that path on your domain |
| 6 | The payment step survives an agent | Step-up authentication and local payment options have a defined path with no human in the browser | Read your checkout for hard human dependencies. ACP's delegated token carries only reason (one_time), max_amount, currency and expires_at, and never mentions step-up authentication |
| 7 | Agent-placed orders are distinguishable afterwards | Order records carry the agent source, and returns and disputes have a written rule | Query your order table for a source dimension. ACP keeps you merchant of record "from fulfillment and returns to support and communication" |
Bands: 11 to 14 means deliberate choices, ready to integrate a rail when one reaches your market. 6 to 10 means partly accidental policy; fix signals 1 to 3 first, because they are cheap and gate the rest. 0 to 5 means your edge defaults are deciding for you.
Four real stores, scored on signals 1 and 2
Signals 1 and 2 are the only ones checkable from outside. Findings are the robots.txt text read on 28 September 2026: stated policy, not live behaviour, and an edge rule can override all of it.
| Store | Agent tokens named | Purchase-path directives | S1 | S2 |
|---|---|---|---|---|
| trendyol.com | None. Googlebot, Googlebot-Image and AdsBot have blocks; ia_archiver and archive.org_bot are fully disallowed | The * block disallows basket and checkout paths plus most filter parameters | 0 | 1 |
| hepsiburada.com | None. A single * block only | Disallows /product/, /api/, /m/ and four language prefixes, with one category exception. Cart paths unmentioned | 0 | 1 |
| boyner.com.tr | ClaudeBot disallowed entirely; Claude-User and Claude-SearchBot allowed with a crawl delay. No OpenAI, Google, Perplexity or Meta tokens | The * block disallows /cart, /sepet, /product/, /liste/ and /uyelik | 1 | 0 |
| etsy.com | None, despite being the first Instant Checkout merchant. Spinn3r disallowed | The * block disallows /api/, cart, messages and filtered result pages | 0 | 1 |
Three of the four files were written for search crawlers and have no opinion about a user-directed agent, so the * block answers for it by accident. Boyner alone separated a training crawler from two agent-class tokens, and for one vendor only. And a disallowed prefix such as /product/ may not match the live product URL template, which is why signal 2 must be checked against real URLs.
What this does not cover and where it breaks down
- robots.txt is a request, not a control. It works because the major operators say they honour it; Anthropic states its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt". Traffic that does not identify itself is untouched.
- Your edge outranks your file. A permissive robots.txt behind a rule that challenges unknown clients is a closed door with a welcome mat. Signal 3 exists because teams pass signal 1 and still block every agent in production.
- Stated policy is not behaviour. This tells you what you allowed, not how many agents arrive or what they do. And passing all seven means an agent can reach your store, not that it will choose it.
- Protocol specifics will move. ACP is at version 2026-04-17 and still beta; UCP is at 2026-01-11 and expanding by market. Re-run signals 5 and 6 quarterly, and signals 1 to 3 after any change to robots.txt, edge configuration or URL templates.
Three things you have probably read that are no longer true
"Get into Instant Checkout." It was scaled back in March 2026. Etsy told Modern Retail it "did not end up seeing a large volume of sales from Instant Checkout", and Walmart's Daniel Danker put conversion at three times lower inside the chatbot than for clickouts. Discovery in the assistant, purchase on your own site, is where the platforms landed.
"Universal Cart means agentic checkout is here." It is here in the United States. Published rollout is United States, then Canada and Australia, then the United Kingdom. Plan for the year it reaches your market; use the interval to fix the access layer.
"AP2 or ACP will handle authentication and disputes." They handle authorization and create an audit trail; AP2 frames the problem as authorization, authenticity and accountability, using signed mandates to evidence intent. Neither removes local checkout obligations. In Turkey, card checkout routinely involves a step-up authentication screen and a multi-instalment selection, and neither is expressible in a delegated payment token whose only allowance reason is one_time against a single max_amount. Turkey recorded 291,610,970 internet card payment transactions worth 926,544 million TL in July 2026 alone, so this is not a reason to ignore agents. It is a reason to write down, before integrating anything, which payment paths an agent can finish and which it cannot.
[INTERNAL DATA NEEDED: share of the client's Turkish card orders that hit a 3-D Secure step-up screen, and share using a multi-instalment plan, to quantify how much of the basket an agent could not complete today.]
FAQ
Should I block AI agents from my store?
Treat training crawlers, search crawlers and user-directed agents as three separate decisions with three different commercial consequences. Blocking a training crawler costs little in the short term. Blocking a user-directed agent blocks a person who asked to see your product.
Does robots.txt stop a shopping agent?
It stops the ones that honour it, including the documented agents from the major model providers. It does not stop unidentified traffic and it does not override a CDN rule. Check both.
What is the difference between ClaudeBot and Claude-User?
Anthropic documents ClaudeBot as collecting content that could contribute to model training, Claude-SearchBot as improving search result quality, and Claude-User as the agent that fetches a page when a person asks Claude a question. One rule for all three throws away the third for the sake of the first.
Do I need to implement ACP or UCP right now?
Only if you sell in a market where the rail is live. ACP is beta at version 2026-04-17; UCP is United States first. Signals 1 to 4 and 7 are worth doing regardless, because they concern your own site.
Who eats the chargeback on an agent-placed order?
Under ACP the merchant stays merchant of record across fulfillment, returns, support and communication, so disputes land with you. Decide now what evidence you keep for an order with no human browser session.
How do I verify that an agent is who it claims to be?
Cryptographically, not by user-agent string. Web Bot Auth and Visa's Trusted Agent Protocol both use RFC 9421 HTTP Message Signatures verified against a published key, with a nonce and a short expiry. This is an edge capability, not something you build in the application.
Run it, or have us run it
Score your own store on the seven signals this week, starting with your robots.txt and your live PDP and cart URL templates. If you would rather have the access layer, edge policy and payment paths audited together and handed back as a ranked fix list, talk to us about an agent access audit.
Sources
- Cloudflare, "Content Independence Day, one year on" (1 Jul 2026)
- Cloudflare, Verified bots docs
- Help Net Security on Cloudflare crawler controls (2 Jul 2026)
- Cloudflare, "The age of agents" (28 Aug 2025)
- Anthropic crawler documentation
- OpenAI, "Buy it in ChatGPT" (29 Sep 2025)
- Modern Retail, Instant Checkout (March 2026)
- Agentic Commerce Protocol repository (version 2026-04-17, beta)
- ACP Delegated Payment Spec
- Google Developers Blog, "Under the Hood: UCP" (11 Jan 2026)
- Search Engine Land on Universal Cart (20 May 2026)
- Google Cloud, AP2 announcement (16 Sep 2025)
- AP2 protocol documentation
- Visa Trusted Agent Protocol specifications
- Bankalararası Kart Merkezi, internet card payments (Jul 2026)
- robots.txt of trendyol.com, hepsiburada.com, boyner.com.tr, etsy.com (28 Sep 2026)
