JavaScript Instructions
Interact with the website you want to scrape and get an optimal response.
Interacting with the website you are scraping is very important, especially if you want to retrieve custom information. Browser API allows you to do so, by simply adding the js_instructions parameter to your request.
Available options
This parameter's value is a URL-encoded JSON array of instruction objects, executed in order. Each instruction accepts the following options:
| Parameter | Required | Type | Description |
|---|---|---|---|
action | ✅ | string | The action that you want to perform. See the available actions below. |
selector | ❌ | string | The CSS selector on which you want to perform your action. |
block | ❌ | string | The area of the HTML element identified by the specified CSS selector. Used with scrollTo. Accepted values are start, center, and end. The default value is start. |
value | ❌ | string | The value of an HTML input identified by the specified CSS selector. Used with value. |
pixels | ❌ | string | The vertical scrolling distance. Used with scrollPx. |
count | ❌ | int | The number of vertical scrolls to perform. Used with scrollInf. The scrolling distance is calculated by the window height. |
timeout | ❌ | int | Delay after an instruction, measured in ms. For wait_for_css, it is the selector wait (5 seconds by default). For download_file_native, it caps resolving and downloading the file. For screenshot, it caps the capture. Downloads and captures add no delay after success. |
file_url | ❌ | string | File URL for a download action. For download_file_native, use an absolute or relative HTTP(S) URL; this takes precedence over selector. |
allowed_content_types | ❌ | string[] | For download_file_native, 1 to 32 MIME types to capture. Defaults to application/pdf and application/octet-stream. |
name | ❌ | string | Required for screenshot: a unique name of 1–64 ASCII letters, digits, underscores, or hyphens. |
options | ❌ | object | Per-action capture options for screenshot. See multiple screenshots. |
strict | ❌ | string | Controls strict mode. Set it to "false" to continue when an ordinary instruction fails. Screenshot capture, checkpoint limit, and request deadline failures always fail the request. |
Available actions
| Action | Description |
|---|---|
click | Click an element specified by the selector option. |
click_and_navigate | Same as click, but also handles page navigation triggered by the click. |
click_and_wait_response | Click an element and wait for a network response whose URL contains urlPattern. |
delete_element | Delete the first element found for a given selector. If multiple is true, delete all matching elements. |
focus | Set focus on an HTML element. |
load_full_page | Wait for the configured timeout, then load iframe content before continuing. |
scrollInf | Scroll a specified number of times using the count option. |
scrollPx | Scroll a specified distance in pixels using the pixels option. |
scrollTo | Scroll to an element specified by the selector option. |
solve_captcha | Solve a reCAPTCHA or hCaptcha challenge. |
submit | Submit a FORM element. |
submit_and_navigate | Same as submit, but also handles page navigation triggered by the submit. |
typed_value | Same as value, but typed like a user would. |
value | Set the value of an HTML input. |
wait_for_css | Wait for a CSS selector to exist in the page. |
download_file | Download a file from file_url, or from the link found by selector when file_url is not provided. |
download_file_native | Capture a native browser download from file_url, or the link URL found by selector. Returns the file as base64_string. |
select | Select an option value on a form control using selector and select_value. |
screenshot | Capture the current page at this step. Requires a unique name; returns an ordered screenshots collection. GET target requests only. |
To use download_file or download_file_native, contact customer support and ask for the domain in your request's url parameter to be whitelisted. Both actions use the same whitelist.
Screenshot checkpoints
Insert screenshot actions after clicks and readiness checks to capture multiple
page states in one request. The top-level screenshot=1 parameter still requests
one final capture after all instructions; it is optional when using screenshot
actions. See multiple screenshots
for action options, the four-tab sequence, response format, and limits.
Native file downloads
For GET /v1 requests, download_file_native accepts either file_url or a CSS
selector identifying a link. Omit selector when supplying file_url. The request's
url must satisfy the download whitelist described above. The action does not
rewrite target paths; supply the exact URL required by the target. HTTP(S) URLs are
supported, including relative links, and embedded URL
credentials are rejected. Target access restrictions still apply.
The optional instruction timeout accepts 10–60000 milliseconds and is capped by
the request timeout. If omitted, the request timeout applies. Native downloading
preserves the source page and its browser cookies, and forwards supplied WSA
headers using the usual header filtering. An explicit WSA-Referer takes precedence
over the source page URL. Source-page device emulation and resource-blocking rules
are not copied to the download page.
Use allowed_content_types when the target serves another MIME type, such as
text/csv or application/zip. Decode base64_string to recover the original
file bytes. Existing download_file behavior is unchanged.
A full example of how this parameter would look like in production is:
js_instructions='[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]'The js_instructions parameter will accept a URL encoded JSON string that represents an array. This is a mock example that uses multiple actions:
[{"action":"click","selector":"<selector>","timeout":2000, "strict":"false"},{"action":"click","selector":"<selector>","timeout":2000, "strict":"true"},{"action":"click","selector":"<selector>","timeout":2000},{"action":"scrollTo","selector":"<selector>","timeout": 2000, "block": "end"}]And this is the URL encoded value from the example above:
%5B%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%2C%20%22strict%22%3A%22false%22%7D%2C%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%2C%20%22strict%22%3A%22true%22%7D%2C%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%7D%2C%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A%202000%2C%20%22block%22%3A%20%22end%22%7D%5DJavaScript Instructions Integration Examples
The following examples shows how the js_instructions parameter is used in order to interact with the scraped url, before retrieving the HTML code.
The full GET request for the js_instructions should be:
https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5Dcurl "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D"const response = await fetch("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D", {
method: "GET"
});
const data = await response.text();
console.log(data);import requests
API_KEY = '<YOUR_API_KEY>'
SCRAPER_URL = 'https://api.webscrapingapi.com/v1'
TARGET_URL = 'https://webscrapingapi.com/'
PARAMS = {
"api_key":API_KEY,
"url": TARGET_URL, "js_instructions":'[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]'
}
response = requests.get(SCRAPER_URL, params=PARAMS)
print(response.text)<?php
$curl = curl_init();
curl_setopt_array($curl, [
CURLOPT_URL => "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
]);
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
url := "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D"
req, _ := http.NewRequest("GET", url, nil)
res, _ := http.DefaultClient.Do(req)
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
fmt.Println(res)
fmt.Println(string(body))
}HttpResponse<String> response = Unirest.get("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D")
.asString();var client = new RestClient("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D");
var request = new RestRequest(Method.GET);
IRestResponse response = client.Execute(request);require 'uri'
require 'net/http'
require 'openssl'
url = URI("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https%3A%2F%2Fwebscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A5000%2C%22block%22%3A%22start%22%7D%5D")
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(url)
response = http.request(request)
puts response.read_bodyImportant! The url & js_instructions parameters have to be encoded.
Example: &url=https%3A%2F%2Fwww.webscrapingapi.com%2F&js_instructions=%5B%7B...%7D%5D