apify-blocked-scrape-triage

A scrape came back wrong. Find out why before changing anything, then escalate in cost order. Most "blocks" are not bot detection, some are not even a live server, and the browser is the most expensive answer rather than the first one.

Work the steps in order. Step 0 ends more investigations than the whole ladder does.

• Apify account (sign up)
• Authentication via one of:
  • apify login (OAuth, if using the Apify CLI)
  • APIFY_TOKEN environment variable
  • Token from Apify Console → Settings → Integrations
• A one-time permission approval, before the first diagnostic run. apify/cheerio-scraper requires full account access and refuses to start until that is approved in the Console. Both paths refuse, with different wording: MCP call-actor answers requires full access to your account. You must approve its permissions before running it, while the CLI prints Error: Actor apify/cheerio-scraper requires full access to your Apify account and has not been approved yet. on stdout with exit code 0 — so an unattended CLI flow reads it as a normal result and stops without saying why. Approve it once at the URL the message prints, before you need a diagnosis. The same applies to apify/web-scraper and the browser Actors in Rung 5; apify/website-content-crawler does not require it.

One DNS pair, one navigation per egress per host, then stop and conclude. The unit is the host, not the organisation: one county published its records across three hostnames that refused a request in three different ways, and a budget spent on "the target" would have bought one of those three answers. The budget counts requests to hosts the target owns; looking something up on a third-party platform, a data catalogue or your own IP check is free. If you are four requests into one host and still cannot name the cause, you are no longer diagnosing, you are probing, and you should say what you know and what it would cost to learn more. One navigation means one page: a browser pulling forty subresources for that page is still one navigation, and a burst of repeated probes is not. The one exception worth spending on is the residential probe in Step 3, a single run that separates "your network" from "actually defended".

Sample before you commit to a fix, change one variable at a time, and remember that your diagnosis is traffic the target counts. Residential traffic and browser Actors are the two line items that surprise people.

One variable includes time. Measurements taken from one egress at 05:44 and from another at 05:57 differ by network and by half an hour, and a refusal that expired on its own is indistinguishable from one your second egress defeated. When you compare two egresses, interleave them inside the same minute. This bit a real investigation: three hosts looked like a clean "the network is the cause" result until the blocked egress was re-measured later and two of the three answers had changed.

Two requests, both free, both about you rather than the target. Skipping them is how a problem with your own address becomes a proxy purchase.

1. What your egress looks like from outside. Not its class in the abstract, the actual address, network and country the site sees:

curl -s "http://ip-api.com/json/?fields=query,country,as,org,hosting"

That service is free for non-commercial use and rate limited, and it tells you where you stand in its own headers: a live call returned X-Rl: 44 (calls left in the window) with X-Ttl: 60 (seconds until reset). Read them rather than discovering the limit as a failure, and for anything regular use a source whose terms fit your use. When you only need the address, the country and which POP you land on, an endpoint with no such conditions does that much:

curl -s "https://www.cloudflare.com/cdn-cgi/trace"

Measured side by side from one egress: the trace returned ip=64.176.60.193, loc=JP and colo=KIX, while the first call added what the trace does not carry, AS20473 The Constant Company and "hosting": true. The ASN and the hosting flag are the parts that predict a refusal, so the cheaper endpoint is a supplement rather than a replacement.

"hosting": true means every WAF you meet has classified you before reading a single header of yours, and that a 403 from a public site is the expected answer rather than a surprise. Egress below means exactly this: the network your request leaves from, as the site sees it. Two egresses of the same class, a hosting IP and a datacenter proxy, are usually one data point to a WAF rather than two.

Run this from every machine you are about to call a separate vantage point, and compare the addresses rather than the machines. A laptop behind a VPN client and the VPN's own server are one egress, not two, and the check says so in one line. Measured: two machines a continent apart returned the same query address, so every "second opinion" taken from the laptop was the first opinion again. What did differ was the path: the same request that timed out after 30 seconds from the laptop reported connect=0.000000s from the server, which is a silent TCP drop rather than a slow host, and only the second form of the measurement says which one you have.

2. Whether your DNS answers mean anything. Resolve over DoH rather than UDP, against two providers:

curl -s -H "accept: application/dns-json" "https://dns.google/resolve?name=HOST&type=A"
curl -s -H "accept: application/dns-json" "https://cloudflare-dns.com/dns-query?name=HOST&type=A"

If UDP answers hand back a non-routable address while DoH returns a real one, your own machine is answering, and the UDP answer is not evidence of anything. The usual cause is not hostile: proxy clients in the Clash, mihomo and sing-box family default to fake-ip mode, where the client invents an address out of 198.18.0.0/15 so it can route by hostname rather than by IP. Addressing the query to 1.1.1.1 or 8.8.8.8 changes nothing, because the answer never leaves the machine. Observed on exactly such a setup: both UDP queries returned 198.18.5.101 for a host whose real addresses came back over DoH from those same two providers.

If you are on a VPN or proxy client at all, assume this until DoH says otherwise, and never report a host as dead on a UDP answer alone.

DoH is the better witness here, not an infallible one. On a corporate or container network with split-horizon DNS, an internal name resolves only through the local resolver, and DoH will answer NXDOMAIN for a host that exists for you; some networks also block DoH outright. The rule above is about public sites, where the public answer is the one that matters. If your target is internal, invert it: trust the local resolver and use DoH only to prove the name is not public.

These checks cost at most one request each and they end the investigation outright. In a sweep of twenty-one public data sites, this section was the right answer more often than the entire ladder below. That sample is public-sector and open-data heavy, where defences are rare and publication is common, so read the ratio as an argument for checking cheaply first, not as a claim about the web at large: on commercial marketplaces and listing sites the balance moves the other way, and this section will end fewer of your investigations.

• A deliberate publication of the same data. An open data portal, an ArcGIS REST service, a bulk download page, a .json twin of the HTML route, sitemap.xml. A published feed is cheaper, more stable and more complete than anything you will scrape, and it often answers from an egress the main site is refusing.
• A login you do not own, or a paywall. Stop and say so.
• Terms that forbid this collection. Read the target page itself, not only robots.txt: assessment portals in particular tend to print the prohibition in the page body while robots.txt says nothing. Many sites publish no robots.txt at all, and its absence decides nothing either way. Where the terms forbid automated collection and a portal publishes the same records, the portal is not a workaround, it is the supported route.

Find the portal rather than guessing at it. Two families cover most public data, and both have a discovery step people skip because the URL template is easy to remember and useless without an id.

# Socrata: ask the federated catalog what a domain publishes, then fetch by id
curl -s "https://api.us.socrata.com/api/catalog/v1?domains=DOMAIN&q=TERM&limit=5"
curl -s "https://DOMAIN/resource/DATASET_ID.json?\$limit=5"

# ArcGIS: walk the directory instead of guessing a layer number
curl -s "https://HOST/arcgis/rest/services?f=json"
curl -s "https://HOST/arcgis/rest/services/FOLDER/SERVICE/MapServer?f=json"
curl -s "https://HOST/arcgis/rest/services/FOLDER/SERVICE/MapServer/LAYER/query?where=1%3D1&outFields=*&resultRecordCount=5&returnGeometry=false&f=json"

A hostname that answers is still a guess, and the check is the content rather than the status. The catalog endpoint above returns the whole federated catalog on any host, so a guessed portal name answered 200 with ten thousand results belonging to three other states, and the real portal returned the same rows. Asking the platform whether it knows the domain separated them in one request: ?domains=GUESSED_HOST answered {"error":"Domain not found: ..."} for the guess and its own datasets for the real one. Generalised: ask whether the records name the host you asked. The same rule kills wildcard DNS, where every name in a zone resolves and none of them is a service.

A hostname with no owner in it is usually a vendor serving hundreds of clients, so the block belongs to the vendor rather than to the body whose data you want, and that body often publishes the records itself. Check the CNAME before assuming independence: four portals under four different government domains resolved to the same three addresses, each CNAME naming the same vendor, sharing one rate limit and one point of failure that the hostnames do not show.

Everything after this step assumes a server answered you. Three failure classes produce no HTTP conversation at all, they are the cheapest things that can be wrong, and they are the most expensive to miss, because every rung below will fail identically against them at increasing cost.

Resolve the way the pre-flight section says, over DoH, against two providers. A local, corporate or captive resolver will happily answer for a name that does not exist publicly, and on an intercepted path it will do so in the name of whatever resolver you addressed. If two providers both return NXDOMAIN over DoH, the host does not exist and nothing below applies. One free query, and it can end the investigation.

A non-routable answer is a question, not a verdict. Transparent interception is common on corporate and VPN paths, and it works: your traffic still reaches the real origin.

An answer proves less than absence does. NXDOMAIN proves the name is not public; a real address proves almost nothing, because behind a shared CDN a wildcard record answers for every name in the zone. Confirm a guessed hostname the way Step 0 describes, by asking whether the content names the host you asked.

A verdict belongs to a host and a path together. Measured on one vendor platform in the same minute from one egress: robots.txt answered 200 while / and /sitemap.xml answered 403. The 200 came from the CDN's own managed copy of robots.txt and never reached the origin. Before recording "this host blocks us", fetch a second path as a control, and prefer one the CDN cannot be answering for.

Then check the URL is still the one the site publishes. The most ordinary cause of "I pulled nothing useful" is not a defence, it is a stale address. Sites move search, catalogues and APIs onto vendor platforms and leave the old hostname to rot for years. Open the site's own current navigation and confirm the path you are using is the one it links to today.

Stop rule for this step. If the host does not resolve, or the site no longer publishes that path, you are finished. There is nothing to unblock.

You are here only if a server answered. The single most expensive mistake now is reading every failure as "they detected me" and jumping to a browser with residential proxies. Match the symptom first, and read the body before you read the status.

The first two rows are one status code and two different situations. A hard block and a managed challenge both arrive as 403 from the same CDN vendor, they route in opposite directions, and the header separates them. Both were measured on the same afternoon from one egress: a government portal answered a plain HTTP client with a bare 403 and no cf-mitigated, a vendor platform answered the same client with cf-mitigated: challenge and Just a moment.... Your diagnostic Actor cannot see either header until you unblind it, and until then both arrive as an empty item. The one-line fix is in "Making refusals visible" below; apply it before using this table.

Name the edge before you use its marker, because cf-mitigated is one vendor's header and absence is not evidence. On any edge that is not Cloudflare the header cannot appear, so "no cf-mitigated" is guaranteed and tells you nothing, while the rule that reads it happily returns "your address was judged" for every one of them. Identify the vendor from what did arrive:

A refusal is a measurement, not a property of the site, and some refusals expire. The Akamai refusal above carried Server-Timing: cdn-cache; desc=HIT on a 403 whose own Cache-Control said max-age=0, meaning the error object was served from the edge cache and never reached the origin. Thirty-eight minutes later the same URL from the same address answered 302 and then 200 with 81,337 bytes, three times running. Nothing about the requester had changed. This is not a promise that refusals dissolve: an ASN or geo rule in a firewall is stable for as long as someone leaves it there, and re-measuring only confirms it. The point is the asymmetry of cost. One repeat request is free and it either upgrades a single observation into a stable fact or saves you a proxy purchase against a refusal that was never about you.

The parameter test, for an endpoint that answers 200 by refusing you. A missing argument and a credential you do not own look identical in the response, and they end in opposite places: one is Rung 3, the other stops the investigation at Step 0. Decide by where the value comes from, not by what it is called.

• The site itself hands it out to anyone who loads the page: a district code in a public dropdown, a build id in the HTML, a token the page fetches before its first search, a guest login printed in the UI. That is a parameter of a public route. Take it the way the page does, Rung 3.
• It is issued to a person, or it gates records the site does not otherwise publish. That is a credential you do not own, whatever the field is named. Stop and say so, and never guess, rotate or construct one.
• Unsure after one look at the page that issues it? Treat it as the second case and go to Step 0 to look for a published copy of the data. A source that requires an account almost always has a bulk or catalogue route that does not.

The form test, for a full page that carries a form instead of records. Server-rendered forms with hidden state are the default shape of government and utility record systems, and a GET-only crawler receives a valid 200 from one forever, with no error and no records. This is not the SPA shell row: there is no mount point and no bundle, the page is large, and the data is one POST away rather than one XHR away.

1. Issue the POST the form describes, carrying every hidden field it hands you (__VIEWSTATE, __VIEWSTATEGENERATOR, __EVENTVALIDATION and friends) plus the search field, with a Referer of the form page.
2. Read where the answer goes. If the response is a redirect to a URL carrying your query in its parameters, you have the record address rather than a session.
3. Then test that address cold, in a fresh client with no cookies and no Referer. This is the step that decides the whole job.

Measured on a county assessor's system: the form page answered 200 with 5,316 bytes and a __VIEWSTATE; the POST answered 302 to .../ParcelDetail.aspx?hdnParcel=<key>&hdnInstance=pcl7; and a cold curl of that URL, no cookies, no Referer, default user agent, answered 200 with 38,817 bytes of the actual record. One of the two redirect parameters turned out to be optional, so strip parameters one at a time before you build them into a crawl. A key the system does not know answered 200 with 747 bytes reading No record found, which is a third outcome to record rather than an empty row to drop.

When the cold GET works, the crawl collapses to one request per key, no browser, no session, no postback: fetch the keys from whatever Step 0 turned up and go straight to record URLs.

Separating a shell from a brochure, since both are a 200 with no data and the advice is opposite. Do not ask where a human would go; a human really does use the SPA's host. Ask the body. A shell is small, has a single empty mount element, loads a JS bundle, and returns the same bytes and the same ETag for different record ids. A brochure is a full rendered page whose record links point at another domain. Shell means stay, go to Rung 4. Brochure means the target is wrong, return to Step 1.

Compare the URL you asked for with the URL that answered, every time. Redirect chains cross domains silently, and a site that moved hosts is the cheapest diagnosis to miss. On one sweep loadedUrl was the only field that revealed a county assessor had moved onto the city's domain, behind a 202 that looked like nothing at all.

loadedUrl only exists on a request that completed, which is the case this skill is about. While every egress is refused you have no redirect chain and no way to see that two hostnames are one site, so two refusals from two different CDN vendors read as two independent targets and double your apparent problem. Two ways out, both cheap: compare the CNAME chains, or wait for the first egress that answers and read loadedUrl there. Measured: a .com hostname refused by Cloudflare and a .gov hostname behind Akamai turned out to be one site, and the loadedUrl from the egress that was admitted said so in one field.

Write down which row you are in. Every later decision depends on it.

Test A, different client, same egress. Fetch the same URL with a client that has a different stack; a real desktop browser is ideal, since it differs from a script in headers, TLS and JS execution at once. If you have no browser sharing your egress, this test is unavailable. Say so rather than substituting something that is not the same comparison.

Test B, different egress. On Apify that is an Actor run, not a one-line curl: write an input file, write a pageFunction, call the Actor, take the dataset id from the response, fetch the items. Budget a few minutes and a sub-cent run. The ready-made input is in references/diagnostic-run.md.

Test A moves more often than people expect, and it is free. Measured on a state assessment portal from one hosting egress: a default HTTP client got 403 and a real desktop Chrome on the same address got the full page, same minutes. The 403 was about the client, not the address, and every proxy in the world would have been the wrong purchase. That page then printed terms forbidding automated collection, which is a Step 0 answer and not a Rung 1 one.

Two refusals are not a verdict, and this is where people abandon reachable sites. A hosting IP and a datacenter proxy are the same class of address to a WAF, so refusing both is one fact, not two. Before calling a site defended, spend exactly one run on a residential egress in the country the site serves. If residential from the right country is also refused, the site is genuinely defended and "How this ends" applies.

That probe is billed traffic, so it is the one step to ask about rather than perform. Residential is charged by volume and this skill exists to prevent buying it reflexively; running it unattended, on someone's account, to satisfy a diagnosis is the same mistake at a smaller scale. State what the probe would cost and what it would settle, then run it once when the answer is yes. Everything before this point is free or sub-cent.

And prefer a free vantage point to a paid one whenever you have both. If you can issue a request from a second network yourself, a plain curl from there answers the same question as an Actor run, immediately and at no cost, and it shows you the things a crawler hides: whether TCP connected at all, the raw headers, the body before any parsing. Spend the run when your only second egress is the platform's.

When two same-class egresses disagree, stop before you buy anything. The rule above expects a hosting IP and a datacenter proxy to be judged alike. When they are not, the rule being applied is not about the class of network at all: it is about your specific address or its immediate neighbours, and the fix is another datacenter egress, which is cheap and probably already in your account. Residential is the wrong purchase here, and this is where the reflex costs the most for the least.

Measured on one county's hosts, one afternoon, two egresses of the same class:

Three refusal shapes from one county on one egress, and none of them from the other. Note also that this is Step 1's host-and-path rule again: a silent timeout, a CDN block and a short Access Denied are three different mechanisms, and reporting "the county blocks us" would have hidden all three.

With Test B alone, the normal case for an agent: B blocked means run the residential probe before deciding anything, B returns a body with no data means Rung 4 regardless of what A would have shown, and anything needing A is unresolved rather than guessed. B also changes the network and the client stack at once, so when the two disagree you cannot attribute it to either until you hold one constant.

A single outcome is not a rate. Six identical requests to one vendor platform, one run, returned three clean pages and three refusals from the same proxy pool. Draw conclusions from a sample. The four-request worked example behind this whole section, where the answer was the network and residential bought nothing, and the sample size that makes a block rate countable, are both in references/gotchas.md.

Enter it only if Step 2 or Step 3 sent you here. Classification decides; the ladder only remediates, cheapest rung first.

Rungs 1 to 3 are remediation you will recognise, and each is a paragraph rather than a page here, with the settings and the traps in references/rungs-1-3.md:

1. Rate and IP spread. Lower maxConcurrency, honour Retry-After, and run the residential probe Step 3 already described, once. Choose proxyRotation deliberately, because a stateful flow under PER_REQUEST looks far more suspicious than a steady crawler.
2. Request fidelity. Make your headers, their order and the Referer chain form one plausible client, and match Accept-Language to the proxy country. curl -H cannot control header order, so set them in preNavigationHooks. Below headers sits the TLS and HTTP/2 fingerprint, which no header work fixes: that layer means uTLS or curl-impersonate, or Rung 5.
3. Tokens and edge cookies from a legitimate flow. A build id, a CSRF or search token, a signed URL, or the cookies a WAF sets on first contact. Take it the way the page does, bind it to the session and address that earned it, respect its TTL. If Step 2 saw a sensor script on an otherwise clean page, carry those cookies into Rung 4.

Most investigations end without touching the ladder. All of these are finished results, not failures:

• The cause is named and it is not a block. A dead path, a stale URL, an SPA whose records come from an endpoint, data published in a feed. Write down the cause, the egresses you measured from, and the next concrete step.
• The cause is named and it is a block you fixed. Record which single change moved it.

Stop and tell the user when:

• The host does not resolve, the path is gone, or the data is not on this host. Re-target rather than unblock.
• The terms of the source forbid this collection. Say so at Step 0 rather than after a fix.
• A hard block survives residential from the country the site serves. That is the cheap stop, and it comes after one run rather than after a browser. A managed challenge is the other case: it earns one browser run first, because it is built to clear. On a Cloudflare edge cf-mitigated tells you which of the two you have; elsewhere, use the residential result itself as the discriminator rather than an absent header.
• Browser plus residential plus a coherent profile is still blocked. Cost per record now dominates; look for an official API, a data licence, or another source.
• Over a real sample, three rungs produced no change in block rate. The diagnosis is wrong, not the tooling.
• Cost per successful record exceeds what the record is worth. Compute it.
• The block is a login wall or a paywall.

Once a source is in production, track its failure rate and alert on it. A source that quietly degrades still produces something that looks like a dataset, and you want to hear about it from a graph rather than from a hole in the data.

Full table with the inputs that matter when blocked: references/actor-index.md.

Through the Apify MCP server, which is how an agent normally reaches the platform: call-actor with the Actor name and an input object, then get-dataset-items with the id from the reply. There is no input file and no shell redirection in this path, and the id you need is at storages.datasets.default.id, not defaultDatasetId. Do not paste the CLI's flags into MCP arguments; they are not parameters of these tools.

Two traps in that reply, both measured. The item count is read while the run is still flushing and undercounts: four runs on four different days reported 1 against six stored items, 4 against six, and 3 against four, so read the dataset before concluding how many items a run produced. And Validation errors: must be object does not name the field it means, and its wording points away from the cause: the same rejection appears for a valid array (must be object) and for an object in that same field (must be array). Do not go looking for your own escaping. Submit the small input from references/diagnostic-run.md, which validates as a whole, then add fields one at a time; the known offender is additionalMimeTypes, see Rung 4.

Through the CLI, if you have one, the call takes an input file and the reply names its storages differently (storage.defaultDatasetId, not storages.datasets.default.id). Commands and flags: references/actor-index.md.

• A target is missing from your dataset, or items arrive empty → Crawlee threw before your code ran. Add the hook from "Making refusals visible", then classify.
• 403 and you cannot tell a hard block from a challenge → name the edge first. On Cloudflare cf-mitigated decides it. On any other edge that header cannot appear, so its absence is not an answer and the refusal stays unclassified until a second egress speaks.
• A refusal that will not reproduce an hour later → a cached edge error or a rule that has since gone. Check for cdn-cache in Server-Timing on the refusal itself, and re-measure both egresses inside one minute before you conclude anything.
• A 200 carrying a full page, a form and no records → the form test in Step 2. Issue the POST, then check whether its redirect target answers a cold GET.
• Your two vantage points return the same address → they are one egress. Compare query from the egress check on each machine before treating them as independent.
• Your resolver answers 198.18.x.x → your own proxy client in fake-ip mode, not the site. Re-resolve over DoH.
• Your hosting IP is refused and a datacenter proxy is not → your specific address, not its class. Change datacenter egress, do not buy residential.
• A 200 whose whole body is Invalid Password or Not authorized → a missing argument or a credential. Apply the parameter test in Step 2.
• The JSON endpoint you found comes back empty through the Actor → not the MIME list; application/json is accepted by default. Read #debug.errorMessages on the item, which names the content type and the allowed list when the Actor really did skip it. For a type outside that list (text/plain, CSV), additionalMimeTypes is rejected through MCP whatever you encode it as — use the CLI, ask the service for f=html, or fetch the endpoint with your own code.
• No status, no headers, no body → Step 1, and check your own vantage point first.
• A non-routable address from your own resolver, but traffic works → transparent interception. Proceed, mark the vantage point confounded, require Test B before any verdict.
• 0 items, run succeeded → read SDK_CRAWLER_STATISTICS_0; a green run proves nothing.
• 403 with an XML body naming a storage service → a missing object, not a defence. The path is dead.
• 403 from two datacenter or hosting egresses → not a verdict. One residential run before you conclude.
• A guessed portal hostname answers 200 → confirm the records name that host before building on it.
• Intermittent 403, or a failure that takes the same number of seconds every time → look at concurrency and at configured timeouts, not at fingerprints.
• Detailed traps and recovery: references/gotchas.md.