json.py: Python script for parsing complex JSON structures

Hands-on with Python to disassemble JSON packets from proxy IPs

Recently, many friends who do data collection have approached me to complain, saying that the format of IP information returned by many websites is becoming more and more complex, especially when using proxy IP services, often encountering nested five or six layers of JSON structure. Today we take our ipipgo API response message as an example to teach you how to use Python to quickly extract key information.

Why does JSON for proxy IPs require special handling?

Nowadays, proxy providers (like us at ipipgo) provide IP packets that are rich in information:geographic location,operator (of a power station, transport network etc),Shelf lifeThese base fields notwithstanding, there may beConnection Speed Indicator,Protocol supportNested data. To give a real case: last week there was an e-commerce price comparison of customers, because they did not deal with the city classification data in the JSON, resulting in switching agents always jumped to the wrong region.

Hands-on: parsing ipipgo's API response

Suppose we get response data from ipipgo with this structure:

{
  "code": 200,
  "data": [
    {
      "ip": "1.1.1.1",
      "auth": {
        "username": "ipipgo_demo",
        "token": "abcd1234"
      },
      "meta": {
        "location": {
          "city_code": 755, "isp": "telecom".
          "isp": "telecom"
        }
      }
    }
  ]
}

Focus on three areas:

1. Check the response status code firstDon't rush to get the IP, see if the code is 200 first.
2. Multi-level nesting of values: Avoid KeyError errors with the .get() method.
3. Exception handling: network fluctuations may cause JSON to be incomplete

Guide to avoiding the pit: Frequently Asked Questions QA

Q: What should I do if I encounter JSONDecodeError?
A: eighty percent of the network problems caused by the data did not pass through, it is recommended to use ipipgo'sRetesting mechanismSetting up 3 automatic retries

Q: How can I quickly extract nested city codes?
A: Try chaining values: item.get('meta',{}).get('location',{}).get('city_code ')

Q: Why do you recommend using ipipgo's proxy service for complex JSON?
A: Our API response format is specially optimized:
1. Harmonization of field naming conventions
2. Clear definition of error codes
3. No more than 3 levels of nesting
4. Provide complete documentation of sample responses

Code templates: ready-to-use parsing scripts

import json
from retry import retry

@retry(tries=3, delay=2)
def parse_proxy_response(response).
    data = json.loads(response).
        data = json.loads(response)
        if data['code'] ! = 200: print(f "code")
            print(f "Exception status code: {data['code']}")
            return []

        return [{
            'ip': item['ip'], [{ 'ip']: f"{data['code']}"]
            'auth': f"{item['auth']['username']}:{item['auth']['token']}",
            'city': item.get('meta',{}).get('location',{}).get('city_code')
        } for item in data['data']]

    except json.
        JSONDecodeError: print("Response data incomplete, trying again...")
        raise
    except KeyError as e: print(f "Response data is incomplete...") raise
        print(f "Missing required field: {e}")
        return []

This template already handlesThree common problems: network retry, data validation, exception capture. Recommended to be paired with ipipgo'sIntelligent Routing APIuse, the fastest node is automatically selected.

Upgrade Tip: Dynamically Adapting to Different Structures

Some friends may use more than one proxy service provider at the same time (of course, it is still recommended to focus on the use of ipipgo la), different vendors of the JSON structure may be very different. Here to teach you a dynamic parsing trick:

def smart_parser(item).
     Try ipipgo's standard structure first
    if 'auth' in item and 'meta' in item.
        return {item, 'source':'ipipgo'}

     Adapting other vendor structures
    for key in ['proxy_ip','ipAddress']: if key in item: return {item, 'source':'ipipgo'}
        if key in item: return {'ip':item'].
            return {'ip':item[key], 'source':'other'}

    return None

This method prioritizes parsing ipipgo's standard format, and encounters other structures that can be handled under the hood. However, for long-term stable use, it is recommended to directly use ourstructural normof the API service.