Python request example: Python request proxy setup code tutorials

Why use a proxy IP?

Recently, an e-commerce friend complained to me that their company used Python to crawl commodity data, and as a result, the anti-climbing mechanism of the website directly blocked the company's IP, and the whole team was blinded. At this time, if you can use a proxy IP, change a vest to continue to work, which will delay the business? Here to teach you a few trickslife-saving operation, put a cloak on the program when requesting with Python.

Second, Requests library to set up the agent (hand-held version)

Install the essential libraries first:

pip install requests

Here's the kicker! When using ipipgo's proxy service, remember to replace the account password and IP address with your own. To give a live example:

import requests

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
}

response = requests.get('https://www.taobao.com', proxies=proxies)
print(response.status_code)

Notice the use ofhttp://The proxy address at the beginning is where a lot of people fall down. If you are using the Socks5 protocol, remember to install thepip install requests[socks]Re-operation.

III. Urllib library settings (old driver's version)

Some older projects still use urllib, and the setup method is slightly more roundabout:

from urllib import request

proxy_handler = request.ProxyHandler({
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
})

opener = request.build_opener(proxy_handler)
response = opener.open('https://www.douban.com')
print(response.read().decode('utf-8'))

IV. Practical guide to avoiding pitfalls

Here's a couple.lesson learned through blood and tears::

V. QA session (real user issues)

Q: Proxy setting is successful but not effective?
A: First check the proxy address format, pay special attention not to write http as https. use ipipgo client'sOne-Click TestingFunctions with the least amount of effort.

Q: How to switch between different proxies automatically?
A: Write an IP pool manager to work with ipipgo's API to get new IPs at regular intervals. the code snippet looks like this:

import random

ip_pool = [
    'http://ip1:端口',
    'http://ip2:端口'
]

proxies = {'http': random.choice(ip_pool)}

VI. Tips for choosing a package

Choose based on business needs:

• For crawling dataDynamic residential (standard), $7.67/GB is cheap enough
• For robbing secondsStatic homes35 bucks for a fixed IP. Steady as an old dog.
• Enterprise applications directly onCustomized SolutionsSupport for pay-as-you-go

One last cold fact: ipipgo's TK line is especially good for people who need toHigh Stash VisitsThe scenario that people who have used it have secretly renewed their subscription. The code is well written, but it's useless without a reliable agent, don't you think so?