Python Processing JSON: Proxy IP Processing API Response Data

Hands-on with proxy IPs stuffed into Python scripts

play crawler old iron must have encountered the IP was blocked the bad things, here to teach you to use ipipgo home proxy IP service directly to the API. the first whole understand their interface to return the JSON long what kind of data we get is usually long like this:

{
  "code": 0,
  "data": [
    {"ip": "112.95.82.33", "port": 8000, "expire_time": "2024-03-20 12:00:00"}, {
    {"ip": "183.6.102.156", "port": 8080, "expire_time": "2024-03-20 12:30:00"}
  ]
}

See?code=0Success, the data is stuffed with working proxy IPs. we need to use the requests library to pull down the interface data, remember to replace your_api_key with the key given by the ipipgo backend.

How to unpack JSON data with the least amount of effort

Don't rush to use the data when you get it, you have to inspect it first. Many newbies planted inDidn't do a status code check.This pit. Look at this code:

import requests
import json

resp = requests.get('https://api.ipipgo.com/proxy?key=your_api_key')
if resp.status_code == 200: result = json.loads(resp.text)
    result = json.loads(resp.text)
    if result['code'] == 0.
        proxies = [f"{item['ip']}:{item['port']}" for item in result['data']]
        print("Caught valid IPs:", proxies)
    else: {item['port']}" for item in result['data']]
        print("Interface error: ", result.get('msg'))
else: print("Interface error: ", result.get('msg'))
    print("Network request pounced, status code: ", resp.status_code)

Here's the point:Check the HTTP status code first and then the business status codeDouble insurance. ipipgo's IPs are usually valid for 30 minutes, so remember to change them in time.

How proxy IPs are fed to crawlers

Here's a tawdry operation - dynamic proxy pooling. Save the proxy IPs you get into a list and recycle them, much more stable than a single IP:

from itertools import cycle

def get_proxy_pool()::
     This calls the ipipgo API.
    return proxies

proxy_pool = cycle(get_proxy_pool())

 This is how to get it when you use it
current_proxy = next(proxy_pool)
requests.get(url, proxies={"http": current_proxy, "https": current_proxy})

Remember to add an exception processing, encounter failure IP automatic switching. ipipgo's IP survival rate can reach 95% or more, more reliable than those free proxies.

First Aid Guidelines for Common Rollover Scenes

Q: What should I do if I keep getting SSL errors?

A: Eighty percent of the proxy protocol is not paired, https links have to use https proxy, do not confuse the protocol type. Check the protocol settings in the ipipgo background

Q：Returned JSON parsing failed?

A: 80% is a coding problem, try resp.content.decode('utf-8') hard decode. ipipgo's interface return are standard UTF-8

Q: Can't use the proxy IP when I just got it?

A: Check the local network fire prevention, or change a ipipgo's server room node. Their Jiangsu server room is particularly friendly to domestic lines

Why do you recommend ipipgo?

Let's compare the mainstream service providers in the market:

The key thing is that their home offersThe only IP quality checking tool in the countryThe function of getting a proxy IP to test and then use truly saves time. Now registered also send 10G traffic package, enough to test.

As a final note, when working with JSON data, remember to use thejson.dumps()Doing formatted output saves a lot of eye candy when debugging. Stuff a few more try-except blocks in your code and you're guaranteed to have a script that outlasts everyone else's.