WebScrapingAPI Docs

Web Scraping API

Request parameters, behavior, and examples for the /v2 Web Scraping API.

Web Scraping API uses the /v2 endpoint for HTTP scraping with optional JavaScript rendering, extraction rules, JSON wrappers, geolocation, and async jobs.

Use this endpoint for Web Scraping API requests:

Base URLSupported methodsPrimary worker
https://api.webscrapingapi.com/v2GET, POST, PUT, PATCHHTTP scraping with optional rendering or async jobs

Required parameters

ParameterTypeNotes
api_keystringRequired for authentication.
urlURL stringRequired unless you are polling an async request with snapshot_id. The decoded value must be a valid URL.

Request parameters

ParameterTypeValidation and behavior
methodGET, POST, PUT, PATCHSets the target request method. Use POST or PUT when forwarding a body.
bodystringForwarded to the target for body-capable Web Scraping API requests.
render_jsbooleanEnables JavaScript rendering.
country2-letter codeLowercased and passed to the unblocking layer.
statestringRequires country; only allowed with country=au or country=us.
citystringRequires country; also requires state for au and us. Spaces are removed and the value is lowercased.
asnstringRequires country; accepts formats such as AS12345 or 12345.
extract_rulesJSON stringExtracts selected data from HTML. Mutually exclusive with json_dom=1.
json_dombooleanReturns a parsed DOM-like JSON structure.
json_responsebooleanWraps response body, headers, status, type, and metadata in JSON.
asyncbooleanSubmits an async Web Scraping API job and returns a response id payload.
snapshot_idstringPolls a prior async result. Cannot be combined with url.
timeoutnumberMaximum request timeout.

/v2 does not allow custom target headers on normal GET requests. Header forwarding is available only for body-capable methods.

Extraction rules

extract_rules must be a JSON object encoded as a string. Each rule can be either a selector string or an object with:

OptionValues
selectorCSS or XPath selector
selector_typecss or xpath
outputoutput field or nested rule
all0 or 1
clean0 or 1
RULES='{"title":{"selector":"h1","output":"text","clean":1},"links":{"selector":"a","output":"@href","all":1}}'
curl --get "https://api.webscrapingapi.com/v2" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "extract_rules=$RULES"

Async flow

Submit an async scrape:

curl --get "https://api.webscrapingapi.com/v2" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "async=1" \
  --data-urlencode "country=us"

The response body contains a job id:

{
  "response_id": "s4t7w3619285042ra9dke1m8qx"
}

Poll the result:

curl --get "https://api.webscrapingapi.com/v2" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "snapshot_id=s4t7w3619285042ra9dke1m8qx" \
  --data-urlencode "json_response=1"

Polling returns the final scraped body or JSON wrapper once the job is ready. If the job is still running, wait and poll again with backoff.

For product-specific Web Scraping API docs, see Web Scraping API parameters and Web Scraping API examples.

On this page