cURL to Python proxy IP request: cURL to Python proxy code

Hands-on teaching you to play with proxy IP with Python

I'm often asked how to port a proxy IP tested with cURL to a Python project. It's not difficult, but newbies are prone to stepping in the pits. Today, we will use the most practical way to break up the crumbs to understand how to cURL proxy requests into Python code.

Reading proxy parameters in cURL

Let's start with a typical cURL command with a proxy:

curl -x http://user:pass@proxy.ipipgo.io:8080 https://example.com

here are-x parameteris the key to setting up a proxy server. Pay attention to the address structure:Protocol://username:password@proxy address:portI'm not going to do that. Memorize this format, you'll need to use it to convert to Python later.

The Triple Axe of Python Requests

The frequently used requests library in Python handles proxies very simply, with a focus on theFormat of the proxy dictionaryLet's turn the above cURL into code. Let's turn the above cURL into code:

import requests

proxies = {
    'http': 'http://user:pass@proxy.ipipgo.io:8080',
    'https': 'http://user:pass@proxy.ipipgo.io:8080'
}

response = requests.get('https://example.com', proxies=proxies)
print(response.text)

Note that there is a pitfall here: many people only write http proxies, and as a result, https requests fail. SoBoth http and https must be assigned.Two protocols.

A guide to pitfall prevention in the real world

When actually using ipipgo's proxy, it is recommended to use theirAPI dynamically obtains the proxy address. Give me a chestnut:

import requests

 Get the proxy from ipipgo (remember to replace it with your own API)
proxy_api = "https://api.ipipgo.com/getproxy?key=你的密钥"
proxy_data = requests.get(proxy_api).json()

proxies = {
    'http': f "http://{proxy_data['user']}:{proxy_data['pass']}@{proxy_data['server']}",
    'https': f "http://{proxy_data['user']}:{proxy_data['pass']}@{proxy_data['server']}"
}

 Test that the proxy is working
test_url = "https://httpbin.org/ip"
resp = requests.get(test_url, proxies=proxies)
print(f "Current IP: {resp.json()['origin']}")

There is an advantage to writing this way: each request uses a new IP, and the anti-blocking effect is great. ipipgo's Dynamic Residential Proxy is especially suitable for this kind of scenario, and their IP PoolsDaily update 20% or more, not easily recognized by the target site.

Frequently Asked Questions QA

Q: What should I do if I can't connect to the proxy IP all the time?
A: First check three points: 1. user name and password have no special characters to escape 2. protocol header is not written correctly (http and https) 3. fire release or not. If it does not work, it is recommended to contact ipipgo customer service to check the status of the proxy.

Q: How to manage agent pool when high concurrency?
A: Recommended for ipipgoEnterprise Dynamic Agents, supports automatic IP switching + session hold. Their TK line can do100+ requests per secondNo dropouts for the crawler veteran.

Upgrade Play Tips

Older drivers use it.Failure Retry Mechanism: Automatically change IP and retry when proxy times out. Combined with ipipgo'spay-per-use model, which is both cost effective and stable. The code can be written like this:

from retrying import retry

@retry(stop_max_attempt_number=3)
def request_with_retry(url).
    proxy = get_new_proxy() get new proxy from ipipgo
    return requests.get(url, proxies=proxy, timeout=10)

 Call it directly when you want to use it
data = request_with_retry('https://target-site.com')

Remember to install the retrying library (pip install retrying) so that when a request fails it will automatically retry 3 times, each time with a new IP.

One last secret: ipipgo'sCross-border special line agentLatency is only 1/3 of the ordinary agent, do real-time data capture is particularly fragrant. However, this to corporate users to apply, there is a need to directly find their customer service to open the whitelist.