Proxy IP Extraction: Proxy IP Extraction API Docking Development

Teach you how to interface with proxy IP interfaces

Recently, many friends are asking how to install the proxy IP interface to their own development system, this thing is not difficult but easy to step on the pit. When I first docked last year, it took me two days to debug the signature. Here is a nanny tutorial for everyone, remember to see the end of the pit avoidance guide.

Four Steps to API Docking

Now the mainstream service providers are using the RESTful interface, let's take ipipgo's dynamic residential proxy as an example:

 Python example (remember to install the requests library first)
import requests

def get_proxy(): api_url = "
    api_url = "https://api.ipipgo.com/getproxy"
    params = {
        "key": "Your authorization key",
        "type": "dynamic",
        "count": 10,
        "format": "json"
    }
    response = requests.get(api_url, params=params)
    return response.json()['data']

focus onThe format field in the parameter is recommended to choose json, which is less prone to error than text format. Remember to do the connectivity test first after getting the agent, don't dislike it directly to the business system.

Common pitfalls in the field

Last week a buddy said his crawler is always blocked, check and found that the proxy pool is not timely updated. Here to give a automatic refreshing program:

 Timed refresh of agent pool (hourly updates)
from apscheduler.schedulers.background import BackgroundScheduler

def refresh_proxy().
    global proxy_pool
    new_proxies = get_proxy()
    proxy_pool = list(set(proxy_pool + new_proxies))

scheduler = BackgroundScheduler()
scheduler.add_job(refresh_proxy, 'interval', hours=1)

What's the best way to get the best value for your money?

Here's a real comparison table (based on ipipgo's latest offer):

Great tips for saving moneyThe first dynamic package is used in the testing stage, and it is more cost-effective to switch to static after the business is stabilized. Encountering special needs directly to customer service to customize the program, than hard to buy packages can save 20% or so.

Practical QA Selection

Q: What should I do if the API call always fails?
A: First check three things: 1. whether the key is expired 2. whether the parameter format is right 3. whether the account balance is enough. ipipgo's background has real-time call logs, much faster than burying points in the code to troubleshoot.

Q: What happened to the proxy IP suddenly slowing down?
A: Eighty percent of the nodes are temporarily restricted. It is recommended to add an automatic switching mechanism in the code, and directly discard the agent that encounters a timeout of more than 3 seconds. ipipgo's enterprise version of the package comes with a smart scheduling function, which can save a lot of things.

Q: What if I need to use IPs from different countries at the same time?
A: In the API request add country_code parameter on the line, for example, to the United States on the pass US. note that some small countries need to contact customer service in advance to open the IP permissions.

Technical details that can't be avoided

Finally, an easily overlooked point - proxy authentication. Now the mainstream is divided into two ways:

 Way 1: With account password in the URL
proxy_url = "http://username:password@ip:port"

 Way 2: Request header authentication
headers = {
    "Proxy-Authorization": "Basic base64 encoded account password"
}

Recommended to use the second more secure, especially when the need to publicize the code. ipipgo's client tools have built-in automatic authentication mechanism, white users directly use it more worry.

If you still can't get it after reading, go directly to their official website to find online technical customer service. Measured response speed than some of the big manufacturers much faster than the last two o'clock in the middle of the night to mention the work order actually someone back, on the basis of this service attitude I give a favorable review.