Python Request Library: Advanced Usage of Requests

Picking apart the proxy playbook of the Requests library

Old drivers who work with network requests know that the Requests library is the Swiss Army Knife of Python. However, many people are stuck in the proxy configuration of this link, especially the need to switch a lot of IP scenarios. Today, we will take the guys to unlock a few practical skills, to ensure that your bug crawling program like open flash skills.

Hardcore Configuration Method for Proxy IPs

Hooking up proxies in Requests is actually quite simple, but there are three pitfalls to be aware of:

import requests

proxies = {
    'http': 'http://user:password@proxy.ipipgo.cc:8000',
    'https': 'https://user:password@proxy.ipipgo.cc:8000'
}

response = requests.get('http://example.com', proxies=proxies, timeout=10)

Here's the point:

• Protocol headers should never be misspelled (http and https should be separate)
• Authentication information is recommended to use exclusive accounts do not use public pools
• Timeout settings should be reasonable, between 5-15 seconds is recommended

The tawdry operation of IP pool rotation

Single IP easily blocked? Try this auto-switching routine:

from itertools import cycle
from requests.exceptions import ProxyError

ip_pool = [
    'http://user:pass@proxy1.ipipgo.cc:8000',
    'http://user:pass@proxy2.ipipgo.cc:8000'
]

proxy_cycle = cycle(ip_pool)

for _ in range(5):
    try: current_proxy = next(proxy_cycle)
        current_proxy = next(proxy_cycle)
        response = requests.get(url, proxies={'http': current_proxy})
        break
    except ProxyError: print(f"{current_proxy}")
        print(f"{current_proxy} hung, move to the next one!")

This trick is especially good for crawler projects that need to run for a long time. If you use ipipgo's Dynamic Residential Agent Pool, they support automatic rotation by default, so you don't have to write your own wheels.

The Golden Rule of Agent Maintenance

Maintaining an agency pool is like keeping fish, you have to change the water regularly:

Practical QA session

Q: What should I do if the agent often fails suddenly?
A: It is recommended to use ipipgo's smart detection function, their API can return the list of available proxies in real time, which saves your mind than maintaining it by yourself.

Q: How to solve the problem of needing to process pictures and text at the same time?
A: Assign separate proxies to different request types, for example:

image_proxy = 'http://img-proxy.ipipgo.cc:8000'
text_proxy = 'http://text-proxy.ipipgo.cc:8000'

Q: What should I do if I encounter a website asking me to log in?
A: Use the Session object to maintain the session, and remember to bind a fixed proxy to the session:

session = requests.Session()
session.proxies.update({'http': 'http://sticky.ipipgo.cc:8000'})

Guide to avoiding pitfalls in agent selection

Agent service providers on the market are a mixed bag, to teach you a few identification tricks:

1. Look for responsiveness: useping (computing)命令测，超过200ms的直接pass
2. Measurement of availability: 20 consecutive requests, the success rate is less than 90% can not be wanted
3. Check IP purity: usehttps://ipcheck.ipipgo.ccChecking the level of anonymity

Lastly, I'd like to introduce my own product, ipipgo's Exclusive Proxy Package, which has recently added theautomatic retry mechanismIf you encounter connection problems, it will automatically cut to the backup line, especially suitable for commercial projects that require high stability. New user registration to send a 3-day trial, the old iron may wish to try to engage in crawlers.