Python requests library using proxies: proxy settings

Hands-on with Python requests with proxy IP configuration

What do you guys fear most about crawlers? Of course, the IP is blocked ah! Half of the hard-written code will be blacked out by the target site, and then you have to rely on the proxy IP to save the day. Today we take Python's requests library to teach you how to use proxy IP to avoid being blocked.

import requests

proxies = {
    'http': 'http://用户名:密码@ipaddress:port',
    'https': 'https://用户名:密码@ip address:port'
}

response = requests.get('destination URL', proxies=proxies)

Watch this.Proxies dictionaryThe first thing you need to do is to make sure that the protocol header is not written the other way around. A lot of newbies fall because the http and https proxy address confused, the result is not connected to the dead. Recommended directly withipipgoThe agent of the family, their family automatically adapted to the protocol, save a lot of heart.

Three ways to open a proxy IP

The first is for temporary use and is suitable for scenarios where you need to change IPs occasionally:

 Single request using proxies
requests.get('url', proxies={'http':'http://ip:port'})

The second is a global setting, suitable for cases where all requests go through the proxy:

 Global proxy settings
session = requests.Session()
session.proxies.update({'http':'http://ip:port'})

The third is an automatic rotation, and this will have to be on theipipgos dynamic proxy package too. Their proxy pool is large enough that they can't stop switching IPs automatically:

from itertools import cycle

proxy_list = ['ip1:port','ip2:port','ip3:port'] Fill in the list of proxies provided by ipipgo.
proxy_pool = cycle(proxy_list)

 Automatically change IPs for each request
for _ in range(10): proxy = next(proxy_pool)
    proxy = next(proxy_pool)
    requests.get('url', proxies={'http':f'http://{proxy}'})

A guide to avoiding the pit (a must see!)

1. timeout settingDon't forget: Proxy servers are often jerked around, without the timeout parameter, the program will be stuck in minutes!

requests.get('url', proxies=proxies, timeout=10)

2. Exception handlingTo put in place: it is recommended to wrap the request with try-except, and automatically retry in case of connection failure

3. IP qualityThe most critical: find their own free agent nine out of ten can not be used, recommended directly with theipipgocommercial agents, with measured availability up to 99%

Practical QA session

Q：Why can't I connect even though my agent is paired?
A: First check the proxy format, make sure it is "protocol://IP:port" format. If you useipipgoThe agent, pay attention to the background to the connection example, some of their packages need to add the account password!

Q: How do I know if the proxy is active?
A: You can use this test site: http://httpbin.org/ip, the return IP changed to indicate that the proxy is in effect!

Q: How to play with proxies in high concurrency scenarios?
A: UpipipgoThe exclusive proxy package with multithreading + proxy pooling, remember to control the frequency of requests

Why do you recommend ipipgo?

One last tip: useipipgoThe proxy remember to open their API to dynamically obtain IP, than their own maintenance of the proxy pool to save a lot of heartache. New user registration also sends 10G traffic, enough for you to test. The code is not as well written as the proxy is well chosen, this is really not blowing!