Proxies

Using the WebScrapingAPI, you can bypass security rules on hard to scrape websites by sending requests from residential IP addresses.

The WebScrapingAPI is making use of a pool of 100+ million IP addresses worldwide, making your requests impossible to get blocked.

When interacting with the targeted URL, you can specify if your request should be processed through a residential or a datacenter proxy. By default, the proxy_type parameter is set to residential.

Your full GET request should then be sent to the following address:

https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&proxy_type=residential

What are Data Center proxies?

Datacenter proxies are proxies that are not affiliated with an Internet Service Provider (ISP). They come from a secondary corporation and provide you with completely private IP authentication and anonymity.

Usually coming from cloud server providers, datacenter proxies are used by many at the same time. Since they are not listed as ISP providers, these IPs can already be flagged by some targets and certain precautionary measures might be taken.

However, this is not the case with WebScrapingAPI datacenter proxies. All WebScrapingAPI datacenter proxies are private proxies, and ensure little to no IP blacklisting.

What are Residential Proxies?

A dedicated residential proxy is an IP address provided by an ISP (Internet Service Provider) to a homeowner. It is a real IP address attached to a physical location. So basically, whenever you move to a new location and set up your internet, your ISP will provide you with an IP address.

While these proxies are the most common proxies used on the Internet, they are also much more likely to get blocked on an IP-basis when attempting to scrape data.

The main difference between datacenter and residential proxies lies in the source of the IP. All other differences that arise rely on the differences between household machines and servers.

Enabling proxy_type=residential counts as 10 API calls when render_js is off and 25 API calls when the render_js is on.

The default value for proxy_type will be decided based on your plan. If your plan supports residential proxies then the default value will be residential. Otherwise the default value will be datacenter.

Proxies Integration Examples

#1: Datacenter Proxy Web Scraping: proxy_type=datacenter

curl --request GET --url "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://httpbin.org/get&proxy_type=datacenter"

Important! The url parameter has to be encoded. ( i.e. &url=https%3A%2F%2Fwww.webscrapingapi.com%2F )

Response Example
{
    "args": {},
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "Host": "httpbin.org",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
        "X-Amzn-Trace-Id": "Root=1-6267d94c-6139009e7aa96c016119654f"
    },
    "origin": "5.153.235.233",
    "url": "https://httpbin.org/get"
}

#2: Residential Proxy Web Scraping: proxy_type=residential

curl --request GET --url "https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=https://httpbin.org/get&proxy_type=residential
Response Example
{
    "args": {},
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "Host": "httpbin.org",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36",
        "X-Amzn-Trace-Id": "Root=1-6267d9e9-7b7217ea171b487901c4bfab"
    },
    "origin": "71.167.129.216",
    "url": "https://httpbin.org/get"
}

Last updated