Python crawler template open source : integrated proxy rotation + CAPTCHA recognition

This might be the most hassle-free Python crawler template you've ever seen!

Crawler old iron understand that the biggest headache is the IP is blocked and CAPTCHA interception. Today we do not talk about false, directly on the solution can run through. First of all, I would like to say a real case: last week there was a price comparison system brother, with the ordinary crawler half an hour was blocked 20 IP, replaced with our agent rotation program, ran for three days without turning over.

Proxy IP exactly how to play in order not to roll over

Many newbies think that just find a few free agents can be used, the result is that the code runs either timeout or blocked. Here are a fewlesson learned through blood and tears::

• Don't use the off-the-shelf proxy lists on the web, 99% is invalid.
• Don't use a single IP for more than 5 minutes, the site is not stupid!
• Remember to do IP quality pre-testing, don't wait for errors to be reported before dealing with them

recommendedIntelligent scheduling interface for ipipgo, fresh IPs that you can use when you get them directly. their API return format is like this:

{
  "proxy": "123.45.67.89:8000",
  "expire_time": 300,
  "region": "Shanghai"
}

Hands-On Integration of Operating Systems

A live code template is given here, focusing on the agent management section:

from ipipgo_client import IPPool This is their home SDK

def get_proxy().
    pool = IPPool(api_key="your key")
    return pool.get(protocol='http', count=5) Take 5 spares at a time

Remember to randomly switch User-Agent in the request header, this form is commonly configured:

Captcha Cracking in the Wild

Don't believe in any universal recognition library, the most stable one under real test is theddddocr+manual codingCombo. When the recognition fails more than 3 times, automatically invoke ipipgo'sHigh Stash Residential AgencyIf you have to change the IP address of a real person, you can try again. Here's a tip: Save the hash value of the CAPTCHA image, and directly check the cache for repeated occurrences.

Why do you recommend ipipgo?

Three hardcore advantages of using their home for over two years:

1. Dedicated IP pools are not watered down, every time you get one it is unused
2. Response speed control within 200ms, twice faster than many peers
3. There are specialized crawler optimization packages that support pay-per-use

Recently discovered a new feature: in the backend settingsIP geographic distribution strategyThe first is that you can specify that certain IPs will only be active at certain times, which is useful for thieves who want to get the job done when it's done.

Frequently Asked Questions QA

Q: What should I do if my proxy IP suddenly fails?
A: Enable auto refresh mode in ipipgo console, set the amount of redundancy of 10%, and switch automatically when abnormalities are detected

Q: Can't get the CAPTCHA recognition rate up?
A: Try to turn the picture to grayscale and then binarization, the accuracy rate can be improved by 30%. ipipgo's server room IP recognition is more difficult than residential IP, it is recommended to prioritize the use of mobile network resources

Q: How do I choose the best value for my package?
A: Crawling data volume of the selection of unlimited monthly packages, small-scale testing with per time billing. New users remember to take 5 yuan experience coupon, enough to run 20,000 requests

Finally, to tell the truth: do not expect a set of programs to eat all over the world, the site wind control changes every day. With ipipgo is mainly a figure of peace of mind, there are technical problems can be directly to their engineers, the response speed than some of the big companies much faster. Code templates I put GitHub, search "crawler anti-blocking practice" can be found, remember to point a star.