scrapy ip proxy settings: Scrapy crawler framework configuration proxy IP middleware

A Hands-On Approach to Cloaking Scrappy Crawlers

Crawler brothers understand that the site anti-climbing is like adding a security door to the data. At this time, the proxy IP is our master key, especially with Scrapy framework for work, do not learn to proxy settings equal to the bare Internet. Today we do not talk about false, directly on the hard food.

What the hell is proxy middleware?

Scrapy's middleware mechanism is like a sorting station, where every request goes through. All we have to do is change the "shipping address" of the request before it is sent. Specifically, we'll add a new address to theDOWNLOADER_MIDDLEWARESThe first thing you need to do is to get the proxy IP to be automatically included in every request.

 Add this to settings.py
DOWNLOADER_MIDDLEWARES = {
    'yourprojectname.middlewares.ProxyMiddleware': 543,
}

How to choose between dynamic vs. static proxies

Here's a pitfall to be warned about: don't assume that just any agent will work! It's important to choose a type based on your business needs:

Like ipipgo's.Dynamic Residential (Business)Package, more than 9 dollars 1G traffic, especially suitable for the need for high anonymity of the scene. Their Socks5 protocol support is very friendly to Scrapy, later will teach the specific how to match.

Real-world code templates (can be applied directly)

 middlewares.py
import random

class ProxyMiddleware(object): def process_request(self, request, spider): def
    def process_request(self, request, spider).
         Replace this with your own pool of proxies
        proxy_list = [
            'socks5://user:pass@ip.ipipgo.net:15236',
            'http://user:pass@gateway.ipipgo.com:2080'
        ]
        proxy = random.choice(proxy_list)
        request.meta['proxy'] = proxy
         It is recommended to add a timeout setting
        request.meta['download_timeout'] = 30

Attention! When using ipipgo's proxies, remember to generate the official website backendwhitelisted IP, otherwise authentication will fail. Their API gets the latest proxies in real time, which is a lot less work than maintaining them manually.

Guidelines for demining common pitfalls

Q: What should I do if I can't connect to the agent all the time?
A: First check the protocol type is not right, https site do not use http proxy. ipipgo's client has an automatic detection function, it is recommended to first use their test tool to verify the

Q：Set up the proxy instead of slower?
A: eighty percent of the data center agent, this kind of fast but easy to be blocked. Change into a residential agent, like ipipgo static residential although the unit price is higher (35 yuan / a), but the stability of the hanging ordinary agent!

Q: What if I need a multi-region IP?
A: Add the country code parameter after the proxy address, for example@gateway.ipipgo.com?country=us. They support 200+ countries and regions, do cross-border e-commerce data collection brother this function is very practical!

Tips for high-level play

1. Inretry middlewareAdd proxy switching logic to automatically change IP address when encountering 403.
2. MatchingCustomizing User-AgentUse to double the effectiveness of anti-blocking
3. Use of ipipgoTK LineDealing with special anti-climbing mechanisms, certain e-commerce platforms require this

One final point: don't waste your time on free proxies! The cost of maintaining your own proxy pool is definitely higher than buying an off-the-shelf service. Like ipipgo's dynamic package more than 7 yuan 1G, enough to climb hundreds of thousands of pages, have this effort not as much as write two more crawler scripts.