Scrapy proxy settings: Scrapy framework built-in proxy configuration program

Basic poses for Scrapy proxy setup

Crawler brothers know that the website anti-climbing mechanism is getting more and more perverted. Today we will nag how to use Scrapy comes with the proxy function to save life. Directly on the dry goods, Scrapy's proxy settings in fact, two strokes:Either change the settings configuration file or tinker with the middleware.The

Let's start with the saving grace, adding these two lines to settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

HTTPPROXY_ENABLED = True

This is the equivalent of putting a proxy switch on the crawler, but that's not enough. The point is, you have to stuff the proxy address into the request. For example, with ipipgo's dynamic residential proxy, the format looks like this:

yield scrapy.Request(
    url, meta={'proxy': ''}
    meta={'proxy': 'http://用户名:密码@gateway.ipipgo.com:9020'}
)

A fancy way to play middleware

The above method is suitable for a small game, really want to play a big one on the middleware. Let's write our own ProxyMiddleware, here is a pitfall to pay attention to---Rotation strategy for proxy IP pools. When using ipipgo's API to get proxies, it is recommended to change the IP for each request for a higher survival rate.

Real-world code example:

import random
from ipipgo_api import get_proxies This is the hypothetical official SDK for ipipgo_.

class RandomProxyMiddleware.
    def process_request(self, request, spider): proxy_list = get_proxies('web_scan_request(self, request, spider))
        proxy_list = get_proxies('web_scraping') call ipipgo's interface
        proxy = random.choice(proxy_list)
        request.meta['proxy'] = f "http://{proxy['auth']}@{proxy['ip_port']}"

Remember to activate this middleware in settings and set the priority to around 500 or so for best fit. This way, each request will be automatically hooked to a different proxy, and the anti-climbing system is basically blind.

A Guide to Avoiding the Pit (Lessons Learned Through Tears)

Here are a few common minefields that newbies step into:

Practical QA session

Q: What should I do if the agent often fails suddenly?
A: That's why it's important to use ipipgo's dynamic IP pool, their survival detection is refreshed on a 5-second scale and automatically filters failed nodes.

Q: Do I need multiple threads with different proxies at the same time?
A: Just give each request a separate proxy in middleware, Scrapy will handle concurrency itself.

Q: What should I do if I encounter a website asking for a verification code?
A: This situation is not enough to change the IP, it is recommended to cooperate with ipipgo'sResidential proxy + request header masqueradingPackage, pro-tested to reduce the CAPTCHA trigger rate of 90%.

Why recommend ipipgo

Honestly, there are tons of proxy service providers on the market. But those who do crawling know thatHigh Stash Residential AgencyIt's the king. ipipgo's top three killers:

1. Dynamic Residential IP in 200+ cities nationwide
2. Single request level IP switching (others are minute level)
3. Failure retry and auto-fuse mechanism.

Especially theirIntelligent Routing SystemThe best export node can be automatically matched to the target website. Last time there was an e-commerce project, the success rate of using ordinary agents is less than 30%, cut to ipipgo directly soared to 85%, the project manager almost sent me a banner.

Finally, a piece of advice: do not waste time on free agents, blocking the IP is a small matter, or to eat a lawyer's letter. Professional things to professional people, this agent fee compared to the risk of the project, really nothing.