Crawler python: Python crawler proxy IP integration program

Hands-on with Python crawler to hook up proxy IPs

Brothers engaged in crawling understand that IP is blocked this thing is more common than eating. Don't panic, today we will nag how to use proxy IP to the reptile life. Remember ah, here are talking about the legal compliance of data collection, do not move the wrong idea.

Why do I have to use a proxy IP?

To cite a chestnut, you squat in the Internet cafe to play the game, the boss to see you play too high directly pull out the network cable. Proxy IP is like a new machine and then play, understand? Especially to catch e-commerce prices, price comparison sites, these places, no proxy IP simply can not play.

Three key scenarios:

• Requires high frequency visits to the same website
• Target sites are geographically restricted
• Multi-region data is required for collection tasks

Proxy IP Selection Guide

Sample code

With the requests library, the code looks like this:

import requests

 API address taken from ipipgo backend (remember to replace it with your own)
proxy_api = "https://api.ipipgo.com/getproxy"

def get_proxy():
    res = requests.get(proxy_api)
    return {'http': f'socks5://{res.text}', 'https': f'socks5://{res.text}'}

response = requests.get('destination URL', proxies=get_proxy(), timeout=10)
print(response.status_code)

If you use the Scrapy framework, the middleware has to be written like this:

class ProxyMiddleware(object).
    def process_request(self, request, spider): proxy = requests.get("ipipgo's API address").text.
        proxy = requests.get("ipipgo's API address").text
        request.meta['proxy'] = f "socks5://{proxy}"

Common pitfalls QA

Q: What should I do if my proxy IP suddenly fails?
A: Use ipipgo's dynamic residential package, which comes with an automatic IP pool switching. Remember to add the retrying mechanism in the code, it is recommended to use the retrying library

Q: How do I know the agent is in effect?
A: Print the current IP before and after the request, recommended to use httpbin.org/ip this interface detection

Q: Which one to choose, static or dynamic?
A: Static IP for websites that need to log in, and dynamic for general collection. ipipgo's Enterprise Edition dynamic package supports session hold, which is suitable for scenarios that need to be logged in.

Guide to avoiding the pit

1. Don't store the proxy IP in a local file, it's more reliable to store it in redis.
2. Check IP availability before each request, don't wait for an error to be reported before dealing with it
3. Pay attention to the type of protocol, http sites do not use socks5 proxy (although ipipgo are supported)
4. Remember to set a timeout, 5-10 seconds is recommended

One last thing about ipipgo's one-of-a-kind, their homeTK LineFor some special scenarios have a miraculous effect, encountered difficult to get the site can find customer service to test resources. New users are recommended to use the dynamic standard version, the amount of large and then turn to the enterprise version, can save a lot of silver.