Lightweight Web Framework: Lightweight Proxy Web Framework

What to do when a crawler hits a counter-crawler? Try this native method

The old iron engaged in crawlers must have encountered this kind of thing - the target site suddenly blocked your IP. If you use traditional methods to change IP, you have to reboot the optical cat and wait for half a day, the efficiency is so low that it can kill you in a hurry. I have a wild way: with a lightweight Web framework + dynamic proxy IP, five minutes to build an automatic IP crawler system.

from flask import Flask
import requests
from ipipgo import get_proxy This is the ipipgo SDK we will use.

app = Flask(__name__)

@app.route('/crawl')
def crawl_page(): proxy = get_proxy()
    proxy = get_proxy() Automatically get a new IP for each request.
    res = requests.get('destination URL', proxies={'http': proxy})
    return res.text

if __name__ == '__main__'.
    app.run()

This code above uses the Flask framework, the key is in theipipgo.get_proxy()This method. This is not an ordinary proxy, it automatically picks a suitable IP from ipipgo's pool of millions of IPs, and when it gets blocked, it switches to the next one in seconds, which is at least 20 times faster than cutting IPs manually.

How do you play with dynamic IP pools without flipping?

The market is full of proxy service providers, but the choice is not good minutes to fall into the pit. Three points to avoid the pit guide to take good:
①IP survival timeDon't believe in nominal values, real-world testing is king;
②LocationBe able to be precise to the municipal level;
③Failure RetryMechanism must be with automatic switching

Here must be an amenity for the ipipgo family, they have a one-of-a-kind trick - theReal-time IP quality scoring system. Every IP has a health index, and an automatic waiver below 80 is much more reliable than those brainless rotations.

Practical anti-blocking guide (personally tested and effective)

Recently, when I helped an e-commerce company to do a price comparison system, I used ipipgo's proxy pool to get a riotous operation:

def smart_crawler(url):: for _ in range(3): for
    for _ in range(3): proxy = ipipgo.get_proxy(region='Shanghai')
        proxy = ipipgo.get_proxy(region='Shanghai') Specify the Shanghai region IP.
        try: res = requests.get(url).get_proxy(region='Shanghai')
            res = requests.get(url, proxies=proxy, timeout=5)
            if 'CAPTCHA' in res.text: ipipgo.report_base.text: ipipgo.report_base.text
                ipipgo.report_bad(proxy) flag the IP as problematic
                continue
            return parse_data(res)
        except.
            ipipgo.report_bad(proxy)
    raise CrawlerError("Failed three times in a row")

The trick is brilliant in two ways: 1,GeolockingMake requests look like real users; 2,Automatic reporting of invalid IPs, next time you won't get this crappy IP.

Common pitfalls for white people QA

Q: What should I do if I use a proxy IP and it times out?
A: 80% of the use of poor quality proxy. ipipgo IP default with 5 seconds heartbeat detection, to get the hand to ensure that it is a hot available IP!

Q: What if I need to initiate 1000 requests at the same time?
A: Don't build your own wheels! Go straight to ipipgo'sConcurrency PackageTheir API supports bulk IP groups, up to 500 unduplicated quality proxies at a time!

Q: It was fine in beta, but crashed online?
A: Check if there is a browser fingerprint in the request header, remember to turn it on when you use ipipgoReal equipment simulationMode to automatically generate mobile/PC UA information

Tell the truth.

Proxy IP this line of water is very deep, some small workshops sell cheap IP, in fact, is a million people ride the garbage pool. The last time I saw the most outrageous, 20 requests with 18 IP of the same room, which is not waiting to be blocked? ipipgo I used a small half a year, the biggest feeling is thatsteady as a dog-Doing data crawling has never dropped the ball because of IP issues, especially with their exclusive IP packages for long term projects.

Lastly, I'll throw in a dry run: use their code word for service times"Recommended by Lao Zhang"I can get a three-day premium package for free, so it's a waste of time not to pull the wool. After all, they have tried to know, than listening to others bragging is much more real.