Proxy IP Implementation for Python Requests Authentication: Requests Proxy IP Authentication

Teach you to use Python Requests to hang a proxy IP handily

Crawler partners know that the Requests library is the oil of gold in Python. However, if you send a request directly to the target site, you will be blocked by the target site's IP in a minute.stealth operationIt's a good idea. Today we take ipipgo family proxy service to give a chestnut, talk about how to play in Requests proxy authentication.

How exactly do you connect a proxy IP to a water pipe?

Imagine the proxy IP is a courier relay station, your request first to ipipgo's server to turn around, and then sent to the target site. The point is to put the account passwordCorrectly tucked request headerIt's like sticking the right delivery note on a courier package.

import requests

 Note that the tunnel address provided by ipipgo is used here exclusively
proxy = {
    'http': 'http://账号:密码@gateway.ipipgo.com:9020',
    'https': 'http://账号:密码@gateway.ipipgo.com:9020'
}

response = requests.get('http://目标网站.com', proxies=proxy)

Note that the account password in the middle of the code isEnglish colonI'm not sure if you're going to be able to do this, but I'm sure you're going to be able to do it in Chinese. Some newbies fall in here, dead can not connect still think it is a problem with the agent.

The Five Pitfalls of Certification Failure

Advanced Operations: Buffing Agents

if it needsHigh Frequency Switching IP, suggest using ipipgo's dynamic tunneling mode. Their API automatically refreshes the IP pool, which is much less work than manually changing IPs.

 Example of dynamic tunneling (remember to replace the authentication information with your own)
session = requests.Session()
session.proxies = {
    'http': 'http://动态账号:密码@rotate.ipipgo.com:9030',
    'https': 'http://动态账号:密码@rotate.ipipgo.com:9030'
}

 Automatic IP switching for successive requests
for _ in range(5)::
    print(session.get('http://检测网站.com').text)

Practical QA First Aid Kit

Q: Why can't I use the ipipgo proxy I just bought?
A: First check the whitelist IP is not bound, overseas agents to choose the right region, do not choose the domestic nodes!

Q: What should I do if the returned data is garbled?
A: Add an 'Accept-Encoding': 'identity' in the request header, some sites have odd compression methods

Q: How do I test if the proxy is working?
A: first use http://httpbin.org/ip测下, the return IP is not the local address means success!

Choosing an agent depends on the doorway

Proxy services on the market are a mixed bag, it is recommended to go directly to ipipgo this kind ofSpecializing in Residential AgentsThe. Their IP pool is updated daily, unlike some small workshops that fool around with virtual machine IPs. The most important thing is that the customer service response is fast, the last time I raised a work order at two o'clock in the middle of the night, actually 10 minutes to solve.

One last rant, using a proxy is not a get-out-of-jail-free card. The addition of request headers, control the frequency of access to the set can not be less. After all, the target site's anti-climbing is not vegetarian, we have to dolit. moisten things and make them quietIt's the highest level.