Which websites block an AI agent running on a cloud server? We measured 83 URLs

What was tested

The September 3 battery followed an initial September 2 run. It used matching Chromium 149 builds against the same 83 URLs: a Linux cloud server on a Hetzner address, running headed and with --headless=new, and a Windows laptop running headed behind a US commercial VPN. Headed means the browser runs with a display, even if that display is virtual.

The sample covered reference material, news, search, shopping, travel, social sites, sign-in pages, feeds and API endpoints. Each browser visit used a fresh tab. The recorded evidence included the final URL, title, rendered text and challenge markers. A plain HTTP fetch pass used the same URL set with a desktop user-agent and a 25-second timeout.

A rendered homepage, listing or login screen counted as reaching that surface, not as completing a purchase, signing into an account or reading paywalled content. A timeout waiting for the browser’s load event was not automatically counted as a block when the page had rendered. Equally, an HTTP 200 response containing only a JavaScript shell was not a successful read.

Twelve sites changed with browser mode

On the same server address, these surfaces failed in headless mode and rendered in the headed pass:

This was a comparison of browser modes, not a controlled test of the user-agent string alone. The headless pass identified itself as HeadlessChrome, but other browser characteristics changed too. Nor are all headless implementations interchangeable: Playwright distinguishes the headless shell from Chromium’s new headless mode. The measured result supports trying the browser mode, not a claim that changing one header will reproduce it.

Nine sites differed between the two headed browsers

These sites failed on the cloud server while their tested surfaces rendered on the laptop/VPN connection:

This pattern is consistent with network-position filtering, but it does not prove that IP reputation was the sole cause. The operating system, geography and request timing also differed. Both addresses were datacenter-class: the laptop was not a residential control. These results cannot establish that a home exit or residential proxy would open the nine sites; the October 5 follow-up below tested a home connection.

Google web search presented an unusual-traffic wall at both positions; a later server browser turn reached a reCAPTCHA. Indeed, Glassdoor and Crunchbase also refused both headed positions. That describes these attempts, not every possible visitor or every form of authorized automation.

What a home connection changed (October 5)

On October 5, 2026 we tested the nine sites again, with Google search, through two headed browsers (a cloud server’s Chromium and the laptop’s Chrome) at three positions: the server’s own datacenter address, with the same provider as September’s server but a different address; a commercial VPN’s datacenter exit, on a different network from September’s; and a home fibre connection. Each browser started from a fresh profile with its timezone set to match the connection, and each position ran once, minutes apart. Control pages loaded in every combination.

Reddit and Ticketmaster refused both browsers at both datacenter addresses and opened for the server’s browser from the home connection. Expedia and Tripadvisor refused the server’s browser on its own address while opening for the laptop’s there, so on that address the browser mattered too. On the home connection, no site refused the server’s browser that opened for the laptop’s.

Three of the nine sites that refused September’s server (Bloomberg, Zillow and Quora) opened on October’s, so a refusal from one datacenter address did not hold for another address of the same provider a month later. Google search answered both browsers on that address; at the other two positions it answered one browser and not the other, so one pass says nothing stable about it. Etsy refused every combination, the laptop on its home connection included, and Reuters opened for each browser at some positions and not others, so neither says anything about addresses.

The home connection was a consumer line whose address its provider shares among customers, which may explain Bloomberg’s check there. One pass per position cannot show how a site treats repeated visits, and a dedicated residential proxy was not tested.

A fetch, a page load and a completed task are different results

The two plain-fetch passes agreed on 70 of 83 rows. Agreement included failures and empty shells, not 70 usable pages. Google search, for example, returned an HTTP 200 JavaScript shell to a plain fetch. Reading the response code alone would have called that a success.

Follow-up browser tasks on the server did retrieve Google Flights fares, Google Maps directions, a live airport departure board and a New York Times headline. One Flights attempt failed while driving the form; a direct results URL then worked. Reaching the site and successfully operating its interface were separate hurdles.

Geography changed the content too: consent screens appeared before some Google services, and shopping and local listings reflected the server’s location. For a price watch or local search, specify the intended country and currency rather than accepting the site’s location guess.

Before relying on a website in an agent workflow

Test the exact task from the environment that will run it. Record whether the failure is a challenge, a login requirement, an empty response or an interaction that did not complete. Compare a plain fetch with a browser when JavaScript is needed; compare browser modes before attributing a failure to the address. Where an official API supports the task and your account is authorized to use it, test that route separately.

For the causes behind these checks, see why websites block AI agents and how bot checks affect personal agents. Website rules change. The observations above are dated September 3 and October 5, 2026; re-test the required workflow before depending on it.