http proxy pool: HTTP proxy pool maintenance program

HTTP proxy pool in the end how to raise? Hands-on guide to avoid the pit

Engaged in data collection of the old iron know, proxy IP is like a fish in the fish pond, have to regularly change the water to feed. Last year, I helped an e-commerce company to do price monitoring, the results of the free proxy was blocked twice in three days, angry operations girl straight jumping. Later, I figured out a set ofAgent Pool Maintenance Wildcard, today in its entirety.

I. Why does the agent pool always turn over?

A lot of newbies directly to the proxy IP thrown into the pool, no matter, which is the same as the fish thrown into the stinking ditch what is the difference? The common three major rollover site:

1. low survival rate like winning the lottery - 100 IPs in the morning, but only 20 in the afternoon can be used
2. Response speed is like a sloth - some IPs seem to work, but the actual request has to wait for half a minute.
3. Authentication information is messy - obviously bought a dedicated IP, but the result is always prompted authentication errors

Second, fish pond management four axes

Take the ipipgo proxy we use as an example, his API can directly export the list of available IP. But only will fish can not, have to be able to raise fish:

1. Water quality monitoring (survival testing)
Every day at 3:00 a.m. with a script batch ping Baidu, response more than 5 seconds directly kicked out of the pool. Be careful not to use the target site detection, easy to expose the business characteristics.

import requests
def check_proxy(proxy)::
    try: res = requests.get('', 'proxy').
        res = requests.get('http://www.baidu.com',
                         proxies={'http': proxy},
                         timeout=8)
        return True if res.status_code == 200 else False
    return False if res.status_code == 200 else False
        return False

2. Regular water changes (IP rotation)
Don't wait for all the IPs to hang up before restocking, like we automatically replace 30% IPs every 2 hours. ipipgo's Dynamic Residential Packages SupportAutomatic refill by volumeIt's a lot less work than manually recharging.

3. Pool farming (operational segregation)

Separating the crawler business from the account business is like not mixing piranhas with koi. We build separate pools for each line of business to avoid losing one for the other.

Dynamic Residential (Standard) 70% - Handling Common Requests
Static Residential 20% - Maintains high-powered accounts
TK Dedicated 10% - Specialized Business Requests