Why the AI agent won't open a site: 403, robots.txt, and safe fallback

An AI agent and a regular browser are not the same thing

On October 17, The Verge gathered evidence regarding the blocking of browser agents on major services: from online stores to airlines and financial websites. Some companies explicitly do not allow agents to perform actions, some only agree through partnerships or licensed access, and elsewhere, automated traffic is stopped by standard anti-bot protection. This is a journalistic collection, not proof that every mentioned service blocks all agents in all countries.

OpenAI's official documentation also warns: a site may restrict a cloud browser, even if the same page opens on your device. The site owner decides whom to let in. A user's permission to open a domain does not override the site's rules, and permission to access a site is not automatic permission to confirm a payment or other irreversible action.

What 401, 403, 429, or CAPTCHA actually mean

robots.txt — a voluntary protocol. It helps well-behaved crawlers but does not protect secret URLs: a malicious client can ignore it. Real restrictions are enforced by authentication, access rights, WAF, rate limits, and server-side request verification.

AI feature architecture that doesn't break from a single 403

1. First, look for an official connector or API. It has more stable contracts, explicit permissions, and machine error codes. Leave the browser for supported UI scenarios.
2. Separate reading from consequences. Price searching can be automated more cautiously than payment, booking cancellation, or changing bank details.
3. Add fail-closed. If the agent does not see the final amount, domain, or confirmation, it stops rather than "guessing."
4. Preserve the manual path. The user should receive a direct link, a brief explanation of the blocker, and the ability to continue manually.
5. Design for re-runs. Steps must be idempotent: a retry after failure should not create a second order, email, or charge.
6. Log state, not secrets. Domain, error category, stage, and time are sufficient. Passwords, tokens, payment data, and private page content must not enter analytics.

A useful agent response format: "The site rejected automated access during the availability check. I did not bypass protection. Here is a direct link; check the price and complete the final confirmation yourself." This is worse than the magical one-click seen only in demos, but better for real trust.

As a site owner, do not confuse a useful agent with a scraper

In its documentation, Cloudflare distinguishes between search crawlers, training bots, and agents acting in real-time on behalf of a human. A single global "block all AI" button is simple, but it can cut off indexing, citations, or real customers along with unwanted data scraping.

OpenAI describes Web Bot Auth: cloud browser requests contain signed headers that a site owner can verify using a public key. This helps confirm the request's origin, but it does not mean automatic permission for any action. User authorization and specific endpoint rules are still required.

Local checker: will your site blocking function survive

Mark only what has already been implemented. Responses are counted in this browser and are not sent to the server; only the result range will be sent to analytics, without the site name.

Example: agent searches for and books a service

A reliable scenario has three different levels. First—the agent collects available options and sources. Second—the user chooses an offer and verifies the terms. Third—final booking or payment occurs only after explicit confirmation.

Implementation contractGoal: find three available options for [date/city]. Allowed: - read public pages and official APIs; - create a comparison with direct links; - prepare the form for the final step. Prohibited without new confirmation: - paying, booking, or agreeing to terms; - bypassing CAPTCHA, 403, rate limits, or site policy; - transferring passwords, tokens, or payment data to chat or logs. If the site blocks the agent: 1. stop automation; 2. name the stage and type of blocker; 3. provide a direct link; 4. suggest an official API, another allowed service, or manual continuation.

For functions with custom tools, continue with the material on secure function calling. If the agent works with code—use change verification checklist. Secret protection basics are collected in the guide on API keys in Vibe Coding.

One practical AI scenario per week: view the email example and subscribe with address confirmation.