Python Proxy Setup: Requests Configuration Tutorial

I. Requests proxy settings for getting started with poses

Python crawler brother should have encountered the site blocked IP embarrassment, this time the proxy IP is a lifesaver. Requests library as the most commonly used HTTP tools in Python, with the proxy is actually very simple. Directly in the request to add a proxies parameter on the end of the matter:

import requests

proxies = {
  "http": "http://用户名:密码@gateway.ipipgo.com:9020",
  "https": "http://用户名:密码@gateway.ipipgo.com:9020"
}

response = requests.get("http://目标网站.com", proxies=proxies)

Note the use ofipipgoThe proxy address format of their home tunnel proxy does not need to maintain their own IP pool. Many tutorials teach people to use free proxies, but the survival rate of free proxies is measured to be less than 10%, and professional things are still reliable to professional service providers.

Second, the correct way to open the dynamic agent

Some scenarios require a different IP for each request, and this is where dynamic proxies come in. Takeipipgoof a short-acting proxy, for example, that automatically switches IPs with each request:

import requests

proxy = "http://用户编号:授权密码@dynamic-gw.ipipgo.com:9680"

session = requests.Session()
session.proxies = {"http": proxy, "https": proxy}

 Automatic IP switching for consecutive requests
for _ in range(5): print(session.get("")
    print(session.get("http://httpbin.org/ip").text)

This kind of revolving door proxy is especially suitable for scenes that require high-frequency IP replacement, such as price comparison and public opinion monitoring. The success rate of the request can soar from 30% to more than 95% after testing with dynamic proxy.

III. Guide to avoiding the pitfalls of certified agents

Proxies with account passwords often encounter authentication failure problems, here to teach you two anti-stepping mine skills:

There is one more piece of cold knowledge: the proxy authentication of Requests does not support special symbols, if the password contains the @ symbol, it will report an error directly. It is recommended to add the special symbols in theipipgoGenerating a dedicated authorization code in the background is much more secure than just using your own password.

Fourth, the alternative play of Socks proxy

In addition to HTTP proxy, some scenarios with Socks proxy is more appropriate. For example, game data collection that requires UDP protocol support, or API calls for some special scenarios:

pip install requests[socks] install dependencies first

proxies = {
    'http': 'socks5://user:pass@gateway.ipipgo.com:1080',
    'https': 'socks5://user:pass@gateway.ipipgo.com:1080'
}

displacement (e.g. of gasoline or diesel fuel)ipipgoSocks5 proxy, download large files faster than HTTP proxy 20% or so. However, be aware that some old websites do not support Socks proxy well, you still have to cut back to HTTP mode.

V. First aid kits for common problems

Q：Why is the local IP returned when the proxy is working?
A: 90% of the time the proxy format is written wrong, check if you missed the protocol header (http://或socks5://)

Q: How do I set up multiple proxies at the same time?
A: Poll the middleware with a proxy, or just go on theipipgoLoad Balancing Agent with Multi-Channel Switching

Q: What should I do if the proxy speed suddenly slows down?
A: First use curl to test the proxy delay, if it is indeed a proxy problem, contact theipipgoCustomer Service for Exit Node

As a final rant, don't just look at price when choosing an agency service. LikeipipgoThis kind of can provide real-time monitoring panel, at any time you can see the agent usage and success rate, out of the problem can also be quickly located, this is really save.