Crawling Agent: Professional Crawling Agent Service

A. Why is your crawler always blocked? Try this dirt method

engaged in crawling friends have encountered this bad thing: just run a good program suddenly stopped, a look at the log screen full of 403 errors. At this time do not rush to smash the keyboard, eighty percent of your IP is the target site off the small black house. Now the site are very fine, a little bit of abnormal traffic found on the IP seal, than the neighborhood security check health code is also strict.

It's time to bring out our savior-proxy IPThe first thing you need to do is to put a "mask" on the crawler. Simply put, it is to give the crawler to wear a "mask", each visit to a different IP address. Like you go to the supermarket to grab a limited number of goods, every time you change clothes to queue, the cashier simply can not recognize the same person.

import requests

 Example of proxy access for ipipgo (remember to change to your own account)
proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:9020',
    'https': 'http://用户名:密码@gateway.ipipgo.com:9020'
}

response = requests.get('destination URL', proxies=proxies, timeout=10)

Second, choose the proxy IP of the three main doorway

There are so many proxy service providers on the market, but not many of them are really reliable. Remember these three selection tips:

Special reminder: don't just look at the price is cheap, some agents of the IP are Internet cafes out of the second-hand IP, with a slower than dial-up Internet. Professional service providers like ipipgo, their IPs are allcomputer room directly operated by the server room+home broadbandHybrid resource pooling for both speed and real user profiles.

Third, hand to teach you to configure the crawler agent

Here to Python's Scrapy framework as an example, say a practical configuration skills. Many newbies will directly write a dead proxy in settings.py, which is long outdated! You have to use middleware to switch IPs dynamically.

class IpipgoProxyMiddleware.
    def process_request(self, request, spider).
        request.meta['proxy'] = 'http://用户名:密码@gateway.ipipgo.com:9020'
         It is recommended to enable the IP auto-refresh feature (configurable in the ipipgo backend)
        request.meta['dont_retry'] = True 

Take care to add the above code to your project's middlewares.py and go to settings to activate this middleware. If you're using ipipgo, it's recommended to enable theirIntelligent Routingfunction, the system will automatically select the fastest node, which is much less troublesome than polling IPs yourself.

Fourth, the senior engineer's private skills

Name a few real-world lessons that your peers won't tell you:

1. Don't use fixed intervals.: Manual operations have random pauses, it is recommended to randomly hibernate between 0.5 and 3 seconds.
2. Fake browser fingerprints: User-Agent should be matched with the full set, don't just change the UA without changing other headers
3. Failure to retry should be restrained: If you fail 3 times with the same IP, you should change it, you'll only expose yourself.
4. Make good use of proxy packages: like ipipgo'squantity-based packageIdeal for short-term bursts.monthly subscriptionSuitable for long-term monitoring

V. QA First Aid Kit

Q: What should I do if my proxy IP is not working after I use it?
A: Normal phenomenon, it is recommended to set the frequency of automatic replacement. ipipgo background can be set to automatically change a batch of IP every 5-30 minutes, this feature is recommended to open.

Q: How can I tell if an agent is highly anonymous?
A: Visit http://httpbin.org/ip, if the IP returned is the same as your proxy IP and there is no X-Forwarded-For header, it is a high stash proxy. ipipgo's all proxies are in high stash mode by default.

Q: What should I do if I encounter a website where I have to log in?
A: This is the time tosession holdfeature to allow the same IP to continuously process logins. ipipgo's Dedicated IP packages support this feature, never use a shared IP to process logins!

VI. Why do you recommend ipipgo?

After using so many proxy services, I finally locked ipipgo for three main reasons:

1. They have specializedCrawler optimization linesThe IP pool is completely isolated from normal users.
2. SupportAssign IPs by target siteFor example, an IP segment dedicated to a certain east and certain treasure
3. ExclusiveIP Health DetectionAutomatic filtering of blocked IPs
4. fast customer service response, the last time I had a problem in the middle of the night, there is actually a human technical support

Recently they had a free trial for new users, sign up and get 1G traffic. It is recommended to first take this test effect, after all, is not suitable for have to use to know. Anyway, my team is now more than a dozen reptile project all cut to ipipgo, the longest project ran for half a year has not overturned the car.