Crawler dynamic IP: Scrapy framework automatically switch IP module development

Teach you how to use Scrapy to get a "face" crawler

Recently, a lot of small partners asked me to use Scrapy to crawl the data of the old website IP blocking how to do? Today we will nag how to give the crawler to install a "smart face" function - in fact, dynamic switching proxy IP. do not panic, this thing is not as mysterious as you think, follow me step by step.

Why are your crawlers always recognized?

Website anti-climbing system is like a supermarket security door, focusing on abnormal behavior. If you always use the same IP crazy request, like wearing the same clothes every day to the supermarket to steal things, not catch you catch who? At this time it is necessary toProxy IP ServiceIt'll be your "dressing room."

Here we have to talk about ipipgo's dynamic residential IP, their IP pool has more than 90 million real home network addresses, every request can give you a new "armor". It's like changing your face every time you go to the supermarket, so the security guards won't recognize you.

Three Tips to Make Scrapy Learn the Art of Changing Faces

Tip #1: Give creepy crawlies a "checkroom."

First, add a middleware in settings.py, which is equivalent to a dressing room for the crawler. The code will look something like this:

class RotateProxyMiddleware(object).
    def process_request(self, request, spider).
        request.meta['proxy'] = get_random_ip()

The point is that this get_random_ip(), have to take fresh IP from ipipgo's API in real time. their interface return speed thief, basically can do milliseconds switching, with the magic like.

Tip #2: Set a "Dressing Rhythm"

Don't change the IP as if you were jerking off, you have to pay attention to the rhythm. It is recommended to change every 5-10 requests, depending on the sensitivity of the target site. In settings with a counter:

CUSTOM_PROXY_ROTATION = 5 IP change every 5 requests

One advantage of ipipgo's dynamic IP pool is that it supports concurrent acquisition, even if you open 20 crawler threads at the same time, you can also ensure that each thread gets a different IP, this design is really thoughtful.

Tip #3: Have a "contingency plan."

What if an IP fails? There has to be an automatic retry mechanism:

How to choose between Dynamic IP and Static IP?

Here to praise the service of ipipgo, his family dynamic static can support. Simply say the difference between the two:

• Dynamic IP: suitable for scenarios that require high-frequency switching, such as ticket-snatching software
• Static IP: suitable for those who need to maintain the session for a long period of time, such as hanging games

If it is an ordinary crawler project, it is recommended to use dynamic IP package. Their residential IPs are all real home broadband, much more reliable than those server room IPs, which are basically undetectable by websites.

Frequently Asked Questions QA

Q: Obviously used proxy IP or still blocked?
A: may be the IP quality is not good, it is recommended to change ipipgo'sHigh Stash Residential IP. Also be careful not to request too often, use with random delays

Q: What should I do if I lose my login status due to switching IPs?
A: In this case, you need to use static IP, ipipgo supports long time session maintenance, single IP can be maintained for up to 24 hours

Q: How do I play in the test environment?
A: first go to the official website of ipipgo to get a free trial, they send enough new users to test the traffic, enough for you to toss a small project!

Lastly, a word from the heart. In the reptile business.A stable proxy IP service is your lifeblood!. Instead of tossing free proxies, it is better to use ipipgo this kind of professional service provider. After all, people in more than 240 countries around the world cloth nodes, to what area of the IP can be adjusted at any time, save time and effort not to worry.