Awaiting
When scraping the web, you may want to wait for various elements. WebScrapingAPI offers you the options to wait for almost everything related to the targeted website.
Passing the
wait_until parameter to your request tells the browser that a certain promise is still ongoing, hence preventing the service worker from terminating until the promise is resolved. Your full GET request should then be sent to the following address:
https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&wait_until=$promise
The currently supported options for the
wait_until parameter are:load- load event is fired.domcontentloaded- DOMContentLoaded event is fired.networkidle0- there are no more than 0 network connections for at least 500 ms.networkidle2- there are no more than 2 network connections for at least 500 ms.
This option comes in handy when dealing with websites that have a longer loading time. Passing the
wait_for parameter will instruct WebScrapingAPI to wait for a certain amount of time, before closing the connection. Your full GET request should then be sent to the following address:
https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&wait_for=$milliseconds
The values passed to the
wait_for parameter are expressed in milliseconds. We recommend that you use a value between 5000 and 10000.When you want to scrape a specific element of the page, passing the
wait_for_css parameter will make WebScrapingAPI wait for that specific CSS selector to be available in DOM before returning the result. If the CSS selector won't load within 10 seconds, the request is automatically marked as failed, and thus, you will not be charged for it.
Your full GET request should then be sent to the following address:
https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&wait_for_css=$cssSelector
We recommend that the
$cssSelector value should be encoded or escaped in order for your request to pass successfully. Last modified 1yr ago