Open Source Web Crawling Framework GitHub Featured Collection

当爬虫撞上反爬 你的代码需要隐身衣

Friends engaged in data collection understand that the anti-climbing mechanism of the site is now more stringent than the cell gates. Two days ago I used Python to write a simple crawler, the results just run half an hour to receive the operator warning text message, IP directly be blacklisted. At this time there is a need toproxy IPThis "cloak of invisibility" makes it impossible for the server to recognize your true identity.

To cite a real scenario: the collection of price data of an e-commerce platform, the first 50 pages of smooth to hand, turn to 51 pages suddenly popped up the CAPTCHA. At this time, if you bringipipgo Dynamic Residential ProxyThe CAPTCHA triggering rate can drop 70% or more by automatically switching between different city IP addresses, just like switching to a new cell phone login for each visit.

Recommended crawlers with over a thousand GitHub stars

These 5 open source frameworks are recommended to be saved in favorites to eat dust (but better not to eat dust):

Hands-on with Invisibility Cloak in Scrapy

Take the most commonly used Scrapy as an example, three steps to access the ipipgo proxy:

Step one:Add a custom middleware to middlewares.py, the key code is just these lines:

def process_request(self, request, spider).
    request.meta['proxy'] = 'http://username:password@gateway.ipipgo.com:端口'

Step two:To plug the API provided by ipipgo into the downloader middleware, it is recommended to set the5 seconds auto switchingIP, so that the collection success rate can go above 98%.

Step Three:Remember to open the retry mechanism in settings, encounter 403 status code automatically change IP retry, this combination of punches out, anti-climbing system is basically useless.

Why do older drivers go with ipipgo?

After using 7 or 8 proxy services, I finally locked down ipipgo for three reasons:

1. True Residential IPUnlike some service providers who take the IP of the server room to fool around, the collection of e-commerce evaluations must use real home broadband IP!

2. City of your choice: When you need to collect regional data, you can accurately locate the IP address of the county and city.

3. Traffic is not wasted: Billing based on actual usage, no need to hurt your wallet when running data in the middle of the night

Frequently Asked Questions First Aid Kit

Q: What should I do if I use a proxy and still get blocked?
A: Check three places: ① whether the request header with browser fingerprints ② whether the access frequency is too high ③ confirm the proxy IP type (recommended ipipgo mixed dialing services)

Q：What should I do if I need to collect data from foreign websites?
A：直接在ipipgo后台切换海外节点，注意选和网站服务器同区域的IP，能控制在200ms内。

Q: Do free proxies work?
A: short-term test can be, the official project do not! We have stepped on the pit: free agent 30% with Trojan horse, 50% response timeout, the remaining 20% has been pulled by the major sites as early as black.

One last piece of cold knowledge: with ipipgo'sDynamic Residential Proxy + Chrome Headless ModeThe success rate of collection is 4 times higher than the pure API solution. Next time you get crashed by anti-crawl, try this king bomb combination, remember to come back to thank me.