Python Web Crawler: Python Web Crawler Essential Proxy IP Settings Tutorial

The hardcore operation of putting a crawler in a cloak of invisibility

Crawlers know that without a proxy IP is like running naked on the Internet, a minute by the site ban into a dog. Recently, many brothers asked how to Python crawler suite cloak, today we broke down to talk about this matter.

What the hell is wrong with proxy IPs?

Simply put, it is to find an intermediary to help you pass the data, as if ordering takeout and letting the rider pick up the meal on your behalf. Here's one.crux: Residential proxies most closely resemble real people surfing the Internet, and data center proxies are easy to identify, see this table for the differences:

Hands-on configuration of agents

Here's a chestnut using ipipgo's API to test the waters with the whole dynamic IP first:

import requests

def get_proxy().
     Fill in the link to the API provided by ipipgo.
    api_url = "https://api.ipipgo.com/getproxy"
    return requests.get(api_url).text

proxies = {
    'http': f'http://{get_proxy()}',
    'https': f'http://{get_proxy()}'
}

resp = requests.get('target site', proxies=proxies)

pay attention toChange IP for every requestDon't catch an IP and gripe hard, websites are not stupid.

Scrapy framework special poses

Old timers with Scrapy will have to get things going in middlewares, here's a labor-saving template:

class ProxyMiddleware.
    def process_request(self, request, spider): current_proxy = get_proxy() Call ipipgo API.
        current_proxy = get_proxy() call ipipgo's API
        request.meta['proxy'] = f "http://{current_proxy}"

Remember to activate this middleware in settings, it is recommended to work with theautomatic retry mechanismIt is more secure to use.

First Aid for Common Rollover Scenes

Don't panic when it comes to these three problems:

1. IP suddenly hangs all the time → Check your account balance and try switching protocol types
2. At a snail's pace. → Change of static residential agent or TK line
3. Always popping CAPTCHA