Python Crawler Proxy: Python Exclusive API Interface Proxy IP

Proxy IPs are your airbags when crawlers hit counter-crawlers

Do crawl the old iron must have experienced such a magical scene: yesterday also ran up the script, today suddenly paralyzed. The server returned 403 error code like a machine gun burst you, this time to pull out the proxy IP this magic weapon. Let's take ipipgo family of dynamic proxy pool, their rotation mechanism can make your request like a Sichuan opera face like, every time you visit a new face.

import requests

proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('https://target-site.com', proxies=proxies)

The username and password in the above code block have to be replaced with the authentication credentials you received in the ipipgo backend. Pay attention to see the port number 9020, this is their home specifically for Python users to open a green channel, more than 30% more stable than the general port.

Choosing a proxy IP is like buying groceries, freshness is the difference between success and failure

There are so many proxy service providers on the market, but not many of them are reliable. I have summarized athe Three-Look Principle (in many contexts)::

Special mention to ipipgo'sIP warm-up mechanismThe system will automatically assign the IP address of a popular website to the IP address of that website. For example, if you want to climb an e-commerce platform, the system will automatically assign IPs that have successfully visited the site recently, a move that can save thirty percent of the cost of trial and error.

A practical guide to avoiding pitfalls: don't let low-level mistakes ruin your scripts

Seen too many cases of people using proxy IP as a sieve, here are two high-frequency stepping on mine points:

1. Time-out settings are too rigid

 Bug demonstration: timeout uniformly set to 3 seconds
requests.get(url, proxies=proxies, timeout=3)

 Correct posture: set it in stages
from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount('http://', HTTPAdapter(max_retries=3))
session.mount('https://', HTTPAdapter(max_retries=3))
response = session.get(url, proxies=proxies, timeout=(3, 7))

2. Request header forgetting to disguise

Even if you use a proxy IP, the User-Agent is still clearly python-requests, which is not the same as sticking "I'm a crawler" on your head. It is recommended to use it with fake_useragent library:

from fake_useragent import UserAgent
headers = {'User-Agent': UserAgent().random}

question-and-answer session

Q: What should I do if my proxy IP is not working after I use it?
A: This situation is eighty percent of the IP pool is not updated in time, ipipgo's solution is dual-channel rotation. Add an exception retry mechanism in the code to automatically switch the alternate access point:

proxy_list = [
    'gateway.ip ipgo.com:9020',
    'backup.ipipgo.com:9021'
]

Q: What is the safest way to control the frequency of crawling?
A: Don't be silly to use time.sleep(1), we suggest to use random delay + flow control double insurance. ipipgo backend can set theflow rate threshold, exceeding the set value automatically fuses, much more flexible than writing it to death in the code.

Q: How do I break the CAPTCHA when I encounter it?
A: First check if the proxy IP is exposed, with ipipgo's high stash of proxies can basically circumvent 90% of the CAPTCHA. The rest of the hardcore can be combined with OCR recognition libraries, such as ddddocr, the gods.

A final word of caution.

Proxy IP is not a panacea, but choosing the right service provider can extend the life of the crawler more than five times. I've used seven or eight providers, and in the end, ipipgo is the most resilient. They have aAutomatic compensation for abnormal IPThe mechanism, encountering invalid IP will automatically make up the time to the account, this kind of conscientious operation in the industry is really not common.

Recently found out they went liveGeographic orientationFunctions, such as specializing in obtaining a city's residential IP. last week to do a review of the site's merchant data collection, with this feature directly bypassing geographical restrictions, the efficiency of two times. If you need the old iron can go to the official website to take a look, new users to send 3G flow experience package, enough to run a small project.