Best Python Crawler: Integrated Proxy IP Solution

Crawlers can't live more than three minutes without a proxy IP these days.

Crawler friends recently met and greeted have changed: "How many of your IP was blocked today?" Data capture is becoming more and more difficult, ordinary IP is like running naked on the battlefield. To cite a real case: an e-commerce monitoring program with a fixed IP to catch the price, just run half an hour to receive a 403 warning, change the IP to continue to catch, the results of the other side directly blocked the entire C section IP.

Proxy IP is what renews the life of contemporary crawlers. However, there is a mixed bag of proxy services on the market.Three Deadly PitsMost often stepped on:
1. claimed millions of IP pools, the actual use of less than 10%
2. Slower than a sloth to respond
3. Authentication mechanisms as complex as Morse code

Proxy Adaptation Guide for Python Family Bucket

Let's look at the basic operation first. Setting up a proxy with the requests library is renewed in three lines of code:

import requests

proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}
response = requests.get('destination URL', proxies=proxies)

But that's too easy to recognize! Gotta play a little trick:

from random import choice

ip_pool = [
    'gateway.ip ipgo.com:9020',
    'gateway.ipipgo.com:9021', 'gateway.ipipgo.com:9022', 'gateway.ipipgo.com:9022'
    'gateway.ipipgo.com:9022'
]

def random_proxy().
    return {'https': f'http://用户名:密码@{choice(ip_pool)}'}

 Change different ports for each request
requests.get(url, proxies=random_proxy(), timeout=(3,7))

Here's the point:Timeout settings should be like a Szechuan opera face turnDon't use fixed values. It is recommended that timeout=(2,5) to (3,7) be randomized to simulate the rhythm of a real person's operation.

Surviving Scrapy for Older Drivers

To do large-scale crawling you also need to look at Scrapy. add a dynamic proxy middleware in middlewares.py:

class RotateProxyMiddleware.
    def process_request(self, request, spider).
        request.meta['proxy'] = 'http://动态验证字符串@gateway.ipipgo.com:9020'
         It is recommended to use ipipgo's Tunnel Proxy Mode, which automatically changes the exit IPs.
        request.meta['download_timeout'] = 8 + random.randint(0,3)

Configuration parameters have to be played like this:

CONCURRENT_REQUESTS = 32 Adjusted according to proxy package
DOWNLOAD_DELAY = 0.5 + random.random() Random delay big method
AUTOTHROTTLE_ENABLED = True autotune must be on

Five hard indicators for choosing a proxy service provider

Here's a direct comparison table to visualize it better:

In particular.dynamic key: ipipgo's API can generate temporary authentication strings every 10 minutes, which is more than 10 times more secure than a fixed account.

Real-world pitfall avoidance Q&A

Q: What should I do if my proxy IP often times out?
A: Check the type of proxy package first, don't take a short-lived proxy for a long task. ipipgo's business package supports long TCP connections, suitable for continuous crawling scenarios.

Q: What should I do if I encounter human verification?
A: Don't be hardcore! Use ipipgo's Residential Proxy + Browser Fingerprint Emulation to get up to 80% success rate. Remember:Over validation should be a combination of punches, IP alone is not enough.

Q: How do I break the total agency fee overage?
A: Add a traffic statistics middleware in Scrapy to monitor consumption in real time. ipipgo has a dosage warning function in the background, and will send a reminder to you when you exceed the dosage.

One last piece of cold knowledge: be careful about DNS pollution even with proxy IPs. It is recommended to force DNS servers to be specified in the crawler, such as 8.8.8.8 and 114.114.114.114 alternately. This detail is handled well, can reduce the 20% resolution failure problem.