Python parsing JSON data: Python proxy JSON parsing code

Hands-On JSON Data Processing for Proxy IPs with Python

We do data collection often have to deal with proxy IP, today take the actual scene to say how to use Python to deal with proxy IP service provider returned JSON data. For example, suppose we extract a batch of proxy IPs from ipipgo, and the return is in this format:

{
    "code": 200,
    "data": [
        {"ip": "1.1.1.1", "port": 8000, "expire_time": "2024-01-01 12:00:00"}, {"ip": "2.2.2.2", "port": 8080, "expire_time": "2024-01-01 12:05:00"}
        {"ip": "2.2.2.2", "port": 8080, "expire_time": "2024-01-01 12:05:00"}
    ]
}

Basic analysis of the three axes

Begin by teaching newbies three must-have actions:

import json

 Assuming the response is the raw data returned by ipipgo
raw_data = response.text

 The first style: to the dictionary
proxy_dict = json.loads(raw_data)

 Second: take the IP list
ip_list = proxy_dict['data']

 Third style: traversal processing
for proxy in ip_list.
    print(f "Available proxies: {proxy['ip']}:{proxy['port']}")

Note the pitfalls you may encounter here:Be sure to check that the code field is 200, if not then something is wrong with the request. Suggest adding a judgment:

if proxy_dict.get('code') ! = 200: raise Exception("Failed to extract proxy.
    raise Exception("Failed to extract proxy, error code: " + str(proxy_dict.get('code')))

Fancy maneuvers in real life

We have to consider these situations in the actual project:

Give a full example with proxy testing:

import requests
from concurrent.futures import ThreadPoolExecutor

def test_proxy(proxy)::
    try: resp = requests.get('')
        resp = requests.get('http://httpbin.org/ip',
            proxies={'http': f "http://{proxy['ip']}:{proxy['port']}"},
            timeout=5)
        return True if resp.status_code == 200 else False
    return False if resp.status_code == 200 else False except.
        return False

 Suppose we get 10 agents from ipipgo.
with ThreadPoolExecutor(max_workers=5) as executor: results = executor.map(test_proxy, ip_list)
    results = executor.map(test_proxy, ip_list)
    alive_proxies = [ip for ip, status in zip(ip_list, results) if status]

Frequently Asked Questions QA

Q: What should I do if I can't use the proxy IP?
A: First check the network connectivity and then use the online testing tool provided by ipipgo. It is recommended to prioritize static residential packages for higher stability

Q: What if there is nested data in the returned JSON?
A: For example, if you encounter this structure: data→region→city, you can chain get to fetch the value:

city = proxy.get('region', {}).get('city')

Q: How should I choose a package?
A: Based on business needs:
- Dynamic residential (standard): suitable for routine data collection
- Enterprise Edition: Choose when high concurrency is required
- Static residence: long-term fixed operational use

The unique advantages of ipipgo

Our agency service has three masterpieces:

1. Extraction is very convenient, the API directly returns the standard JSON format
2. SupportSocks5/HTTP Dual ProtocolIf you change a parameter in the code, you can switch
3. Client comes with IP detection function, do not have to write their own test scripts

Especially for cross-border business, with our TK dedicated package, the latency can drop 40% or more. The price is absolutely conscientious, dynamic residential packages7 bucks more for 1 gig of traffic, cheaper than buying a bottle of drink.

Finally, a thoughtful tip: remember to add exception capture when processing JSON, proxy IP this thing network fluctuations are inevitable. Encountered parsing errors first look at the document, ipipgo response format are standardized, according to the sample code to change the basic will not be overturned.