WebScrapingAPI Docs

JavaScript Instructions

Interact with the website you want to scrape and get an optimal response.

Interacting with the website you are scraping is very important, especially if you want to retrieve custom information. Browser API allows you to do so, by simply adding the js_instructions parameter to your request.

Available options

This parameter's value is a URL-encoded JSON array of instruction objects, executed in order. Each instruction accepts the following options:

ParameterRequiredTypeDescription
action✅stringThe action that you want to perform. See the available actions below.
selector❌stringThe CSS selector on which you want to perform your action.
block❌stringThe area of the HTML element identified by the specified CSS selector. Used with scrollTo. Accepted values are start, center, and end. The default value is start.
value❌stringThe value of an HTML input identified by the specified CSS selector. Used with value.
pixels❌stringThe vertical scrolling distance. Used with scrollPx.
count❌intThe number of vertical scrolls to perform. Used with scrollInf. The scrolling distance is calculated by the window height.
timeout❌intDelay after an instruction, measured in ms. For wait_for_css, it is the selector wait (5 seconds by default). For download_file_native, it caps resolving and downloading the file. For screenshot, it caps the capture. Downloads and captures add no delay after success.
file_url❌stringFile URL for a download action. For download_file_native, use an absolute or relative HTTP(S) URL; this takes precedence over selector.
allowed_content_types❌string[]For download_file_native, 1 to 32 MIME types to capture. Defaults to application/pdf and application/octet-stream.
name❌stringRequired for screenshot: a unique name of 1–64 ASCII letters, digits, underscores, or hyphens.
options❌objectPer-action capture options for screenshot. See multiple screenshots.
strict❌stringControls strict mode. Set it to "false" to continue when an ordinary instruction fails. Screenshot capture, checkpoint limit, and request deadline failures always fail the request.

Available actions

ActionDescription
clickClick an element specified by the selector option.
click_and_navigateSame as click, but also handles page navigation triggered by the click.
click_and_wait_responseClick an element and wait for a network response whose URL contains urlPattern.
delete_elementDelete the first element found for a given selector. If multiple is true, delete all matching elements.
focusSet focus on an HTML element.
load_full_pageWait for the configured timeout, then load iframe content before continuing.
scrollInfScroll a specified number of times using the count option.
scrollPxScroll a specified distance in pixels using the pixels option.
scrollToScroll to an element specified by the selector option.
solve_captchaSolve a reCAPTCHA or hCaptcha challenge.
submitSubmit a FORM element.
submit_and_navigateSame as submit, but also handles page navigation triggered by the submit.
typed_valueSame as value, but typed like a user would.
valueSet the value of an HTML input.
wait_for_cssWait for a CSS selector to exist in the page.
download_fileDownload a file from file_url, or from the link found by selector when file_url is not provided.
download_file_nativeCapture a native browser download from file_url, or the link URL found by selector. Returns the file as base64_string.
selectSelect an option value on a form control using selector and select_value.
screenshotCapture the current page at this step. Requires a unique name; returns an ordered screenshots collection. GET target requests only.

To use download_file or download_file_native, contact customer support and ask for the domain in your request's url parameter to be whitelisted. Both actions use the same whitelist.

Screenshot checkpoints

Insert screenshot actions after clicks and readiness checks to capture multiple page states in one request. The top-level screenshot=1 parameter still requests one final capture after all instructions; it is optional when using screenshot actions. See multiple screenshots for action options, the four-tab sequence, response format, and limits.

Native file downloads

For GET /v1 requests, download_file_native accepts either file_url or a CSS selector identifying a link. Omit selector when supplying file_url. The request's url must satisfy the download whitelist described above. The action does not rewrite target paths; supply the exact URL required by the target. HTTP(S) URLs are supported, including relative links, and embedded URL credentials are rejected. Target access restrictions still apply.

The optional instruction timeout accepts 10–60000 milliseconds and is capped by the request timeout. If omitted, the request timeout applies. Native downloading preserves the source page and its browser cookies, and forwards supplied WSA headers using the usual header filtering. An explicit WSA-Referer takes precedence over the source page URL. Source-page device emulation and resource-blocking rules are not copied to the download page.

Use allowed_content_types when the target serves another MIME type, such as text/csv or application/zip. Decode base64_string to recover the original file bytes. Existing download_file behavior is unchanged.

A full example of how this parameter would look like in production is:

js_instructions='[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]'

The js_instructions parameter will accept a URL encoded JSON string that represents an array. This is a mock example that uses multiple actions:

[{"action":"click","selector":"<selector>","timeout":2000, "strict":"false"},{"action":"click","selector":"<selector>","timeout":2000, "strict":"true"},{"action":"click","selector":"<selector>","timeout":2000},{"action":"scrollTo","selector":"<selector>","timeout": 2000, "block": "end"}]

And this is the URL encoded value from the example above:

%5B%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%2C%20%22strict%22%3A%22false%22%7D%2C%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%2C%20%22strict%22%3A%22true%22%7D%2C%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%7D%2C%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A%202000%2C%20%22block%22%3A%20%22end%22%7D%5D

JavaScript Instructions Integration Examples

The following examples shows how the js_instructions parameter is used in order to interact with the scraped url, before retrieving the HTML code.

The full GET request for the js_instructions should be:

https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D
curl "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D"
const response = await fetch("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D", {
  method: "GET"
});

const data = await response.text();
console.log(data);
import requests

API_KEY = '<YOUR_API_KEY>'
SCRAPER_URL = 'https://api.webscrapingapi.com/v1'

TARGET_URL = 'https://webscrapingapi.com/'

PARAMS = {
    "api_key":API_KEY,
    "url": TARGET_URL,    "js_instructions":'[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]'
}

response = requests.get(SCRAPER_URL, params=PARAMS)

print(response.text)
<?php

$curl = curl_init();

curl_setopt_array($curl, [
  CURLOPT_URL => "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => "",
  CURLOPT_MAXREDIRS => 10,
  CURLOPT_TIMEOUT => 30,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "GET",
]);

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}
package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
)

func main() {

	url := "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D"

	req, _ := http.NewRequest("GET", url, nil)

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
HttpResponse<String> response = Unirest.get("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D")
  .asString();
var client = new RestClient("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D");
var request = new RestRequest(Method.GET);
IRestResponse response = client.Execute(request);
require 'uri'
require 'net/http'
require 'openssl'

url = URI("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE

request = Net::HTTP::Get.new(url)

response = http.request(request)
puts response.read_body

Important! The url & js_instructions parameters have to be encoded.

Example: &url=https%3A%2F%2Fwww.webscrapingapi.com%2F&js_instructions=%5B%7B...%7D%5D

On this page