Which websites block an AI agent running on a cloud server? We measured 83 URLs
A datacenter address and a headless browser are two different walls. From a Hetzner server with a real headed Chromium, nine mainstream sites still refused what a laptop's identical browser got; the headless identity alone had closed twelve more.
Fewer than you would guess, and for two different reasons that need two different fixes. In early September 2026 we drove the same Chromium build through 83 URLs from a Hetzner cloud server and from a laptop. Most of the web was identical from both. What differed split cleanly: identity (a headless browser announces itself as HeadlessChrome, and that string alone closed twelve mainstream sites that opened for the same server’s browser with a display attached) and address (with a real headed browser, nine sites still refused the server’s IP where the laptop’s identical browser passed: Reuters, Bloomberg, Expedia, Reddit, Ticketmaster, Zillow, Quora, Tripadvisor, Etsy, plus Google web search behind a CAPTCHA). Fix the identity and you are most of the way there; the address list is what a residential route exists for.
How we measured, and the caveat that matters
Same browser on both sides: Playwright’s Chromium 149 build, headed and headless, driven by the same script against the same 83 URLs (reference sites, news, search, shopping, travel, social, job boards and other heavily defended sites, sign-in pages, public APIs). The server was a Hetzner cloud machine in Nuremberg on a recycled IPv4 address with no IPv6; the browser inside it ran on a virtual display so it presented a plain Chrome/149.0.0.0 identity, and a second pass ran it headless to measure the difference. We also ran a plain curl pass with a desktop user-agent, because a harness’s fetch tool behaves like curl, not like a browser.
The caveat: the laptop was behind a commercial VPN exit, which is a datacenter-class address, not a home connection. So where the two positions agreed, two datacenter positions agreed; where the server failed and the laptop passed, the Hetzner range is treated worse than a generic datacenter exit. A true residential control would likely open more sites than either. We say “the server’s address” below, not “datacenters”, for that reason.
Layer one: a plain fetch sees the same web from both places
With curl and a desktop user-agent, the two positions agreed on 70 of 83 rows. Every JavaScript challenge (Reuters, Bloomberg, the FT, Flightradar24, Stack Overflow, Medium, Quora, Crunchbase, Glassdoor, idealo, Etsy, Tripadvisor, booking.com’s shell) fired identically from both, which makes them a property of the site, not the address: any non-browser client gets them anywhere. Every API and feed (GitHub, open-meteo, npm, PyPI, Stripe, Discord, Slack, RSS) worked from both. The rows where the server alone was refused at this layer: Reddit (both hosts, a 403 “Blocked” page), Ticketmaster (a 403 identity-verification page), and IKEA Germany (a Cloudflare “Attention Required” page). Amazon’s European storefronts handed the server a soft “automated access” interstitial where the laptop got a normal 404. And one row went the other way: Craigslist blocked the VPN exit and served the server Nuremberg listings.
OpenClaw’s web_fetch tool mirrored this column line for line, because it executes no JavaScript. That is not a defect; it is what a fetch tool is for: APIs, feeds, server-rendered pages. Anything behind a JavaScript challenge needs the browser tool.
Layer two: the identity wall
A server has no display, so unless you give it one, Chromium runs headless and puts HeadlessChrome/149.0.0.0 in its user-agent. On the same server, same IP, same minute, these sites refused the headless browser and rendered for the headed one:
- The Financial Times (Cloudflare challenge, then the homepage)
- Brave Search (CAPTCHA page, then results)
- DuckDuckGo’s HTML results (CAPTCHA, then results; this is OpenClaw’s own key-free search provider)
- Walmart (“Robot or human?”, then the search results)
- IKEA Germany (Cloudflare block, then the homepage)
- Flightradar24 (“Just a moment…”, then the flight history)
- Medium, idealo, Zalando, Heathrow arrivals, WhatsApp Web, x.com (a shell or a challenge, then the site)
Indeed and Crunchbase went from a hard block to a Cloudflare challenge: softer, still shut. In an earlier pass we had spoofed a desktop user-agent string on the headless browser and watched fourteen sites flip on the string alone; the headed run confirmed it on the real fix rather than a spoofed one, and also cleared three sites (UNIQLO UK, Walmart, Best Buy) we had wrongly filed under the address. The fix is a virtual display in front of the Gateway; the sandbox guide ends with how.
Layer three: the address wall
With the display in place and a browser identical to the laptop’s, this is what the server still could not open:
| Site | Server, headed | Laptop (VPN), headed |
|---|---|---|
| reuters.com | the JavaScript-required shell | the homepage |
| bloomberg.com | “Are you a robot?” | the homepage |
| expedia.com | “Bot or Not?” | the homepage |
| reddit.com and old.reddit.com | “You’ve been blocked by network security” | the subreddit |
| ticketmaster.com | “Your Browsing Activity Has Been Paused” | the homepage |
| zillow.com | “Access to this page has been denied” | listings |
| quora.com | “Just a moment…” | the homepage |
| tripadvisor.com | a bare shell | the homepage |
| etsy.com | a bare shell | the homepage |
Reddit’s refusal extends to its JSON endpoints and its OAuth API hosts, so a Reddit-moderation use case is not rescued by “use the API instead”. Google web search deserves its own line: from the server the headed browser reached google.com/sorry and was offered a reCAPTCHA (the headless browser a day earlier got the page with no CAPTCHA at all), and from the VPN exit a CAPTCHA too. A CAPTCHA an agent cannot solve is a closed door, so plan on another search provider; Bing and DuckDuckGo returned results to the server, and Google Maps and Google Flights both worked (a fare comparison and a driving route came back complete). Three sites refused every position: Indeed, Glassdoor and Crunchbase. They are hostile to all automation and belong to no list of ours.
What is geography, not a block
A server in Germany is treated as a visitor from Germany. Google Flights and Maps showed the EU consent page first (one click), Yahoo Finance showed the EU privacy page, Craigslist opened on Nuremberg, storefronts defaulted to German. None of that blocked anything, but an agent asked for “prices near me” will answer for where its computer lives. One cosmetic oddity worth knowing about recycled cloud addresses: ours carried a stale reverse-DNS name from a previous tenant (a university domain in another country). It did not cause a refusal in our runs.
What worked, so the list above stays in proportion
Through the agent’s own browser, from the server: Google Flights results with three real fares, Dubai airport’s live departures board, Google Maps directions, the New York Times headline, Amazon.com’s product page (which handed the laptop a “Continue shopping” interstitial instead), Kleinanzeigen listings, idealo prices, the Google sign-in form to the email step, and a YouTube video’s captions via yt-dlp with no sign-in wall. The BBC, the Guardian, AP, Hacker News, Wikipedia, MDN, GitHub and every documentation site we tried were open from both positions at every layer. For research, monitoring and most shopping, a server’s browser sees the web you see.
What an operator can do
- Fix identity first. A display and a real browser identity are the cheap half and recover the larger list. Never fake the user-agent on a headless browser; sites read more than the string, and the spoof failed where the display succeeded.
- Route the few address-blocked sites through a residential connection. The practitioner fix is a Tailscale exit node on a device at home, so the server’s browser leaves the internet from your household address for those sites. A residential proxy service is the rented version of the same idea.
- Use the API where you have an account. Where a site offers one and you hold credentials, an API call from a server is expected traffic. Reddit is the exception above: its API hosts refuse the address too.
- Accept the rest. Ticketing, some retail and most job boards are hostile to any automation from anywhere. That is a rational position on their side, and the tooling behind it cannot tell your agent from a scraper.
Everything above was measured in the first week of September 2026. Edge rules move; treat the lists as a snapshot and re-test a site before you build a workflow on it.