urllib set proxy ip: Python urllib library add proxy ip solution

Teach you how to hang a proxy with urllib.

Crawler brother should understand, sometimes the site anti-climbing too fierce, directly with their own IP hard just minutes to be blocked. This time you need to find an intermediary to help pass the message - that is, the proxy IP. a chestnut, like you go to buy milk tea do not want to queue up, looking for errand boys to buy the same.

urllib comes with a ProxyHandler in fact, the thief is good to use, but many tutorials are written like a book. Let's come to some real today, directly on the code. Note that there is a pit here:After creating the opener object with urllib.request.build_opener(), remember to register the global with install_opener()Otherwise the settings may not take effect.

import urllib.request

 Replace this with the proxy IP and port provided by ipipgo
proxy = 'http://用户名:密码@proxyIP:port'

proxy_handler = urllib.request.ProxyHandler({
    'http': proxy,
    'https': proxy
})

opener = urllib.request.build_opener(proxy_handler)
urllib.request.install_opener(opener)

 Test the request
response = urllib.request.urlopen('http://httpbin.org/ip')
print(response.read().decode())

How exactly do you choose a proxy IP?

There are all sorts of agent types on the market, so here's a quick highlight for the brothers:

The most important thing to remember is that if you're not using a free proxy on the Internet, nine out of ten of them are pits - either the speed is slow as a tortoise, or two minutes on the invalid.

Common Rollover Scene QA

Q: The code runs but the agent doesn't work?

A: First check if the proxy format is right, especially when it comes with account password. It is recommended to use the test site httpbin.org/ip to verify first!

Q: How can I tell if I've chosen the right type of agent?

A: you need a high stash of residential proxies, a large and cheap selection of data centers. ipipgo customer service can give a customized program, directly find them to test IP try!

Q: What should I do if the agent suddenly fails to connect?

A: First see if the IP is expired, dynamic proxy generally have survival time. It is recommended to add an exception retry mechanism in the code to automatically change to a new IP.

Say something from the heart.

I've used seven or eight proxy service providers, and I've used ipipgo for a long time mainly because I want to save my mind. Their client is really not blowing, white people can get started in three minutes. Focus on three practical functions:

1. SupportFilter IPs by regionIt's a great way to get a localized data collection.

2. Traffic packages can be suspended at any timeUnlike some platforms where it's a waste if you don't use it.

3. When encountering technical problems to find customer service, the speed of reply is faster than a delivery boy

Lastly, a reminder to newbies: proxy IP is not a panacea, the key is to work with a reasonable request frequency. Don't think that hanging a proxy can do whatever you want, the site is not a fool. Control the number of concurrency, coupled with a random dormant time, is the long-term solution.