Free IP Authentication API Interface Development Guide

What's the point of this thing? Let's just figure out what we need.

Crawler old iron must have encountered this situation: just grabbed two pages of data on the IP blocked, this time you have to find a proxy IP to renew life, but the free proxy pool is mixed with a lot ofinvalid IPrespond in singingTurtle IP. Writing your own validation interface is like giving the IP a physical exam and sifting out the ones that work.

To give a real example: an e-commerce price monitoring system, with the unverified proxy IP to catch the data, six out of ten requests timeout. After using our ipipgo's proxy IP with the verification interface, the success rate directly soared to 85% upwards, which is a real productivity improvement.

API design triple axe

Don't get all fancy, just three core functions:

For anonymity detection, we recommend using the ipipgoCamouflage test interfaceI'm sure you'll be able to find out if it's a high stash proxy or not. Some of the bad proxies will leak the real IP, this kind of goods out before it's too late.

Hands on how to hook up ipipgo

Straight to the dry code (Python version):

import requests
proxies = {
  'http': 'http://user:pass@proxy.ipipgo.com:9020',
  'https': 'https://user:pass@proxy.ipipgo.com:9020'
}

def check_ip(target_url).
    try: resp = requests.get(target_url, proxies=proxies)
        resp = requests.get(target_url, proxies=proxies, timeout=5)
        return resp.status_code == 200
    except: resp.status_code == 200
        return False

Be careful to replace the user and pass with the authentication information you got from the ipipgo backend. It is recommended to enableAutomatic switching of IP poolsfunction, so that each request can use a new IP, anti-blocking effect.

A Guide to Avoiding the Pit (Lessons Learned Through Tears)

1. Don't be silly with free proxy detection interface, those interfaces are not stable. Suggested self-built detection services, with Ali Cloud / Tencent Cloud such asMulti-region serverBe a validation node

2. Don't immediately pass a death sentence when you encounter a validation timeout, as some sites will intentionally delay their response. SettingSecondary validation mechanismIf it fails the first time, switch to the alternate interface and try again.

3. 2 a.m. to 7 a.m. is the peak quality of the proxy IP, this time to do bulk verification can be screened out more than 30% available IP

Frequently Asked Questions QA

Q: What can I do if I don't have enough API calls in the free version?
A: ipipgo'sCorporate PackagesIt supports customized QPS and also enables smart routing mode. Last time, there was an old man who made a price comparison platform, and used this feature to improve the efficiency of data collection by 4 times!

Q: Inconsistency between test results and actual use results?
A: 80% of the target sites useBehavioral Validation. It is recommended to bring the real request header when detecting it, using ipipgo'sBrowser Fingerprint EmulationFunctions can break this.

Q：How can I respond to an emergency when my IP suddenly fails?
A: Add afusion mechanismThe IP pool is automatically switched when the IP pool fails 3 times in a row. ipipgo API supportFailure to redial automaticallyThis function is recommended to be on in the background

The doorway to choosing a service provider

There are many proxy IP service providers on the market, but not many of them are reliable. Focus on three indicators:

• IP survival time > 12 hours (ipipgo's dedicated IP is stable for 24 hours)
• Response time <800ms (measured ipipgo's BGP lines average around 560ms)
• Supports customization by business (e.g. e-commerce dedicated IP, social platform dedicated IP, etc.)

Finally, a piece of cold knowledge: when using a proxy IP for the authentication interface, remember to set theRandomized sleep interval. Too regular request intervals will be caught by the anti-crawl system, don't ask me how I know...