WebScrapingAPI Docs
HomeLoginRegister
  • Introduction
    • Registration
    • Pricing
    • Free APIs
    • API Errors
    • Custom Response Headers
  • Browser API
    • Getting Started
      • Access the API
      • Authentication
      • API Parameters
    • Basic API Requests
      • GET Request
      • POST, PUT & PATCH Requests
      • Proxy Mode
    • Advanced API Features
      • Awaiting
      • Geolocation
      • Sessions
      • Forcing Timeouts
      • Custom Headers
      • Custom Cookies
      • Device Type
      • Viewport Sizing
      • Screenshot
      • Screenshot Options
      • Extraction Rules
      • JavaScript Instructions
      • Full JSON Response
      • JSON DOM Response
      • Block Resources
      • Captcha Solving
  • WebScrapingAPI
    • Getting Started
      • Access the API
      • API Parameters
    • Basic API Requests
      • GET Request
      • POST, PUT & PATCH Requests
    • Advanced API Features
      • Geolocation
      • Extraction Rules
      • Full JSON Response
      • JSON DOM Response
      • Rendering JavaScript
  • WebStealthProxy
    • Getting Started
      • Access the Proxy
      • Usage and Statistics
    • Basic Proxy Requests
    • Advanced Proxy Requests
      • Sessions
      • Geolocation
      • Custom Headers
      • Custom Cookies
      • Screenshots
  • Google Search API
    • Getting Started
      • Access the API
      • API Parameters
    • Google Search Engines
      • Google Search API
      • Google Maps API
      • Google Maps Reviews API
      • Google Reverse Image API
      • Google Trends API
        • Geo Parameter Options List
        • Google Trends Categories List
  • WB CUSTOM SEARCH API
    • Getting Started
      • Access the API
      • API Parameters
    • WB Custom Search Engines
      • Wayfair Product API
      • Best Buy Product API
      • Nordstrom Product API
  • Amazon Search API
    • Getting Started
      • API Parameters
      • Access the API
      • Supported Domains
    • Amazon Search Types
      • Amazon Search
      • Amazon Seller
        • Amazon Seller Products
        • Amazon Seller Profile
        • Amazon Seller Feedback
      • Amazon Product
      • Amazon Category
      • Amazon Bestsellers
      • Amazon New Releases
      • Amazon Deals
  • Bing Search API
    • Getting Started
      • Access the API
      • API Parameters
    • Basic API Requests
Powered by GitBook
On this page
  1. Browser API
  2. Advanced API Features

JavaScript Instructions

Interact with the website you want to scrape and get an optimal response.

Interacting with the website you are scraping is very important, especially if you want to retrieve custom information. BrowserAPI allows you to do so, by simply adding the js_instructions parameter to your request.

This parameter's value is a stringified object, which accepts the following options:

Parameter
Type
Description

action Required

string

The action that you want to perform. Accepted values are:

-click - click on an element specified by the selector option -click_and_navigate - same as click, except it also handles the page navigation that is triggered by the click

-scrollTo - scroll to an element specified by the selector option

-scrollPx - scroll a specified distance in pixels specified by the pixels option

-scrollInf - scroll a specified number of times specified by the count option

-delete_element - delete the first element found for a given selector. If the multiple property on the instuction is true, it will delete all the elements.

-focus - sets the focus on a HTML element

-value - sets the value of a HTML input

-typed_value - same as value, but the value is typed like a user would

-submit - works only on FORM elements -submit_and_navigate - same as submit, except it also handles the page navigation that is triggered by the submit -solve_captcha - solve any reCaptcha or hCaptcha challenge. - wait_for_css - wait for a CSS selector to exist in the page.

selector

string

The CSS selector on which you want to perform your action.

block

string

The area of the HTML element identified by the specified CSS selector. Used with the scrollTo action. Accepted values are start, center and end. The default value is start.

value

string

The value of a HTML input identified by the specified CSS selector. Used with the value action.

pixels

string

The distance for vertically scrolling. Used with the scrollPx action.

count

int

The number of times to perform a vertically scroll. Used with the scrollInf action. The scrolling distance is calculated by the window height. Useful for lazy-loaded lists.

timeout

int

Perform a delay after each JavaScript instruction. For the scrollInf action, the delay is performed after each individual scroll. For the wait_for_css action, it represents to time to wait for the CSS selector to appear and it defaults to 5 seconds. Measured in ms.

strict

string

By default, the JavaScript instructions are executed in "strict mode", which means that if an error occurs during the execution of one instruction, the request will be aborted and an error will be returned.

You have the option to deactivate this behaviour by assigning the strict key the value "false" within your instruction.

A full example of how this parameter would look like in production is:

js_instructions='[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]'

The js_instructions parameter will accept a URL encoded JSON string that represents an array. This is a mock example that uses multiple actions:

[{"action":"click","selector":"<selector>","timeout":2000, "strict":"false"},{"action":"click","selector":"<selector>","timeout":2000, "strict":"true"},{"action":"click","selector":"<selector>","timeout":2000},{"action":"scrollTo","selector":"<selector>","timeout": 2000, "block": "end"}]

And this is the URL encoded value from the example above:

%5B%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%2C%20%22strict%22%3A%22false%22%7D%2C%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%2C%20%22strict%22%3A%22true%22%7D%2C%7B%22action%22%3A%22click%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A2000%7D%2C%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22%3Cselector%3E%22%2C%22timeout%22%3A%202000%2C%20%22block%22%3A%20%22end%22%7D%5D

JavaScript Instructions Integration Examples

GET https://api.webscrapingapi.com/v1

The following examples shows how the js_instructions parameter is used in order to interact with the scraped url, before retrieving the HTML code.

Query Parameters

Name
Type
Description

api_key*

String

<YOUR_API_KEY>

url*

String

https://webscrapingapi.com

render_js

Integer

1

js_instructions

Object

[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]

The full GET request for the js_instructions should be:

https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A
curl "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A"
const http = require("https");

const options = {
  "method": "GET",
  "hostname": "api.webscrapingapi.com",
  "port": null,
  "path": "/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A",
  "headers": {}
};

const req = http.request(options, function (res) {
  const chunks = [];

  res.on("data", function (chunk) {
    chunks.push(chunk);
  });

  res.on("end", function () {
    const body = Buffer.concat(chunks);
    console.log(body.toString());
  });
});

req.end();
import requests

API_KEY = '<YOUR_API_KEY>'
SCRAPER_URL = 'https://api.webscrapingapi.com/v1'

TARGET_URL = 'https://webscrapingapi.com/'

PARAMS = {
    "api_key":API_KEY,
    "url": TARGET_URL,
    "render_js": 1,
    "js_instructions":'[{"action":"scrollTo","selector":"div.navFooterBackToTop","timeout": 5000, "block": "start"}]'
}

response = requests.get(SCRAPER_URL, params=PARAMS)

print(response.text)
<?php

$curl = curl_init();

curl_setopt_array($curl, [
  CURLOPT_URL => "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => "",
  CURLOPT_MAXREDIRS => 10,
  CURLOPT_TIMEOUT => 30,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "GET",
]);

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}
package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
)

func main() {

	url := "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A"

	req, _ := http.NewRequest("GET", url, nil)

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
HttpResponse<String> response = Unirest.get("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A")
  .asString();
var client = new RestClient("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A");
var request = new RestRequest(Method.GET);
IRestResponse response = client.Execute(request);
require 'uri'
require 'net/http'
require 'openssl'

url = URI("https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://webscrapingapi.com&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div.navFooterBackToTop%22%2C%22timeout%22%3A%205000%2C%20%22block%22%3A%20%22start%22%7D%5D%0A")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE

request = Net::HTTP::Get.new(url)

response = http.request(request)
puts response.read_body

Important! The url & js_instructions parameters have to be encoded.

( i.e. &url=https%3A%2F%2Fwww.webscrapingapi.com%2F&js_instructions=%5B%7B%22action%22%3A%22scrollTo%22%2C%22selector%22%3A%22div... )

PreviousExtraction RulesNextFull JSON Response

Last updated 7 months ago