Web Scraping API
Request parameters, behavior, and examples for the /v2 Web Scraping API.
Web Scraping API uses the /v2 endpoint for HTTP scraping with optional JavaScript rendering, extraction rules, JSON wrappers, geolocation, and async jobs.
Use this endpoint for Web Scraping API requests:
| Base URL | Supported methods | Primary worker |
|---|---|---|
https://api.webscrapingapi.com/v2 | GET, POST, PUT, PATCH | HTTP scraping with optional rendering or async jobs |
Required parameters
| Parameter | Type | Notes |
|---|---|---|
api_key | string | Required for authentication. |
url | URL string | Required unless you are polling an async request with snapshot_id. The decoded value must be a valid URL. |
Request parameters
| Parameter | Type | Validation and behavior |
|---|---|---|
method | GET, POST, PUT, PATCH | Sets the target request method. Use POST or PUT when forwarding a body. |
body | string | Forwarded to the target for body-capable Web Scraping API requests. |
render_js | boolean | Enables JavaScript rendering. |
country | 2-letter code | Lowercased and passed to the unblocking layer. |
state | string | Requires country; only allowed with country=au or country=us. |
city | string | Requires country; also requires state for au and us. Spaces are removed and the value is lowercased. |
asn | string | Requires country; accepts formats such as AS12345 or 12345. |
extract_rules | JSON string | Extracts selected data from HTML. Mutually exclusive with json_dom=1. |
json_dom | boolean | Returns a parsed DOM-like JSON structure. |
json_response | boolean | Wraps response body, headers, status, type, and metadata in JSON. |
async | boolean | Submits an async Web Scraping API job and returns a response id payload. |
snapshot_id | string | Polls a prior async result. Cannot be combined with url. |
timeout | number | Maximum request timeout. |
/v2 does not allow custom target headers on normal GET requests. Header forwarding is available only for body-capable methods.
Extraction rules
extract_rules must be a JSON object encoded as a string. Each rule can be either a selector string or an object with:
| Option | Values |
|---|---|
selector | CSS or XPath selector |
selector_type | css or xpath |
output | output field or nested rule |
all | 0 or 1 |
clean | 0 or 1 |
RULES='{"title":{"selector":"h1","output":"text","clean":1},"links":{"selector":"a","output":"@href","all":1}}'
curl --get "https://api.webscrapingapi.com/v2" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "extract_rules=$RULES"Async flow
Submit an async scrape:
curl --get "https://api.webscrapingapi.com/v2" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "async=1" \
--data-urlencode "country=us"The response body contains a job id:
{
"response_id": "s4t7w3619285042ra9dke1m8qx"
}Poll the result:
curl --get "https://api.webscrapingapi.com/v2" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "snapshot_id=s4t7w3619285042ra9dke1m8qx" \
--data-urlencode "json_response=1"Polling returns the final scraped body or JSON wrapper once the job is ready. If the job is still running, wait and poll again with backoff.
For product-specific Web Scraping API docs, see Web Scraping API parameters and Web Scraping API examples.