JSON parsing function: JSON data parsing function application and proxy docking

What exactly do JSON parsing functions have to do with proxy IPs?

Many people think that JSON parsing is simply dealing with the data format, in fact, in the proxy IP application scenarios, this thing can help us solve big problems. For example, when we get the proxy list through the ipipgo API, the return is JSON format data, this time we have to rely on the parsing function to the need for IP addresses, ports, protocols, these key information out.

import json

 Let's say this is the response data fetched from the ipipgo API
response = '{"proxy_list": [{"ip": "1.1.1.1", "port":8080, "protocol": "socks5"},{"ip": "2.2.2.2", "port":8888, "protocol": "https"}]}'

data = json.loads(response)
for proxy in data['proxy_list'].
    print(f "Available proxies: {proxy['protocol']}://{proxy['ip']}:{proxy['port']}")

Anyone who's ever worked on a crawler knows that.Proxy IP Failure ReplacementIt's a family affair. Using JSON parsing can process IP pool updates quickly, at least 10 times faster than doing it manually. This automated process is especially critical when using ipipgo's Dynamic Residential package, which acquires different IPs each time.

Hands On Docking Agent Service

Take ipipgo's API docking as an example of a three-step process:

 Step 1: Get the proxy
import requests

api_url = "https://api.ipipgo.com/getproxy"
params = {
    "key": "Your API key",
    "protocol": "socks5",
    "count": 10
}

response = requests.get(api_url, params=params)
proxy_list = response.json()['data']

 Step 2: Rotate the proxies
for proxy in proxy_list:
    try.
        session = requests.Session()
        session.proxies = {
            "http": f"{proxy['protocol']}://{proxy['ip']}:{proxy['port']}",
            "https": f"{proxy['protocol']}://{proxy['ip']}:{proxy['port']}"
        }
         Put your business request code here
        except: {proxy['protocol']}:{proxy['ip']}
    except.
        print(f "Proxy {proxy['ip']} failed, automatically switching to next")

pay attention toexception capturerespond in singingautomatic retry mechanismThis is especially true when using dynamic IPs. ipipgo's Enterprise Dynamic package has good stability, but it's always safer to have a few extra IPs on hand.

How do you choose a package without stepping into a trap?

Guidelines on demining of common problems

Q: What should I do if I get an encoding error when parsing JSON?
A: 80% of the response data has special characters, it is recommended to add aresponse.encoding = 'utf-8'

Q: What should I do if the proxy IP is not working just after I use it?
A: First check whether the request interval is too short, if there is no problem you can change the static residential package, or contact ipipgo technical brother to check the quality of the API return

Q：What should I do if I need to use HTTP and Socks5 proxy at the same time?
A: In the API request parameters add a"protocol": "mixed"The JSON returned will then contain multiple protocol proxies

Personal experience sharing

Recently, while helping a client to connect to ipipgo's TK line, I found a problem.Hidden Tips: The JSON returned by their API actually has aregion_codefield, with this you can precisely select proxy nodes for a specific region. For example, to do localized content collection, add a"region": "us_ny"parameter, the IP of the New York node arrives immediately.

There's also a small pitfall to remind you: sometimes the parsed port number will be a string type, and splicing it directly will report an error. Remember to useint(proxy['port'])Under forced conversion, this detail pitted me for two hours...