Scrapy how to implement proxy ip? middleware configuration and automatic rotation of the actual battle

First, why must your crawler use a proxy IP?

Crawler brothers understand that the site anti-climbing mechanism is like a neighborhood security - you every day from the same door in and out more than a dozen times, people do not stop you strange. At this time, the proxy IP is your spare key chain, every time you use a different key to open the door, the security guard can not remember what you look like.

To give a real example: an e-commerce platform with ordinary crawler continuous request, less than half an hour to be blocked IP. changed to dynamic proxy IP pool, continuous collection of three days did not trigger the blocking. This is the magic of IP rotation, so that the target website thinks that different users are visiting.

Second, Scrapy middleware configuration three steps

Let's start with the hard stuff and look directly at the core configuration code:

 Add these two lines to settings.py
DOWNLOADER_MIDDLEWARES = {
    'your_project.middlewares.ProxyMiddleware': 543,
}

 Write this class in middlewares.py
class ProxyMiddleware(object).
    def process_request(self, request, spider):
        proxy = "http://用户名:密码@gateway.ipipgo.com:端口"
        request.meta['proxy'] = proxy

Here's a pitfall to be aware of: many tutorials teach people to use free proxies, and they end up scratching their heads when they can't connect. It is recommended to useipipgo Dynamic Residential ProxyTheir proxy address format is gateway.ipipgo.com, remember to replace your own account password.

III. Automatic rotation of the tart operation

Changing proxies manually is too low, let's play with automation:

import random

class ProxyRotatorMiddleware.
    def __init__(self).
        self.proxy_list = [
            "http://user1:pass1@gateway.ipipgo.com:30001",
            "http://user2:pass2@gateway.ipipgo.com:30002", ...
            ... More proxy nodes
        ]

    def process_request(self, request, spider):
        proxy = random.choice(self.proxy_list)
        request.meta['proxy'] = proxy
        spider.logger.debug(f "Currently using proxy: {proxy}")

displacement (e.g. of gasoline or diesel fuel)ipipgo Dynamic Residential Enterprise PackageThe first is that the IP address of each country is automatically switched to a different country's IP address for each request, and a friend who is a cross-border e-commerce company has used this method to collect the prices of goods from 10 countries at the same time, and the success rate has soared from 47% to 92% directly.

IV. Anti-sealing practical experience kit

These are a couple of potholes I've personally stepped in:

• Don't use public proxy pools! Last year a crawler framework's public proxy interface was blocked on a large scale
• Don't be too stingy in setting the timeout, 3-5 seconds is recommended to be more prudent
• Don't get tough with CAPTCHA, it's recommended to useipipgo static residential proxyMaintaining long conversations

V. Why do you recommend ipipgo?

QA First Aid Kit

Q: What should I do if the agent suddenly fails?
A: Check the account expiration date, if it is a ipipgo user, the background has real-time usage monitoring, it is recommended to set the usage warning

Q: How do I test if the proxy is valid?
A: Use this command for quick detection: curl -proxy http://代理地址 -I https://www.example.com

Q: Do I need to work with multiple accounts?
A: Look at the size of the business, small-scale with ipipgo dynamic package enough. Day mining millions of data is recommended on the enterprise version + multiple sub-accounts