Python Parsing JSON Responses: API Data Handling Tips

When Python Meets Proxy IP: The Pitfalls of JSON Data Processing

Recently, when helping a friend to deal with the crawler project, I found that many newbies in Python to deal with the JSON data returned by the API, will always be in the proxy IP environment in a handful of things. Today, I just solved a real-world case last week, say how to elegantly handle JSON data in the proxy IP scenario.

The right posture for proxy IP requests

Many people always have problems with proxy settings when using the requests library. Remember this.Universal template::

import requests

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'https://用户名:密码@gateway.ipipgo.com:端口'
}

response = requests.get('https://api.example.com/data', proxies=proxies)

Here's one.Hidden Potholes: When using proxies like ipipgo that require authentication, be sure to put the account password in the URL. I've seen people put their authentication info in the headers, and they couldn't connect to the server.

Life-saving tips for JSON parsing

Don't rush json() when you get the response, do these three steps first:

 1. Check the status code
if response.status_code ! = 200.
    print(f "Request failed, current proxy IP: {proxies['http']}")

 2. Catch parsing exceptions
catch parsing exceptions. try: data = response.json()
    data = response.json()
except JSONDecodeError: print("The data was parsed by the response.json().
    print("Response is not legal JSON.")

 3. Validate the data structure
if 'results' not in data: print("Response content is not legal JSON")
    print("Data structure exception, check API documentation.")

Recently, when using ipipgo's rotating proxy, I've encountered that a node returns an HTML login page (presumably the proxy server is temporarily pumped), and I don't do these checks to crash the program directly.

Special handling in proxy IP environments

Consider agency in these situations:

Practical: the correct posture to deal with paged data

Look at this real-life example of crawling an e-commerce platform for review data:

def get_comments(page).
    try: with requests.Session() as s: with requests.
        with requests.Session() as s.
            s.proxies = proxies
            params = {'page': page, 'size': 50}
            response = s.get(api_url, params=params, timeout=10)

             Key Processing Logic
            if 'totalPages' in response.json():: return response.json()
                return response.json()['data']
            return []

    except Exception as e.
        print(f "Error capturing page {page}, switching proxies...")
         Automatically change the proxy node for ipipgo
        reset_proxy()
        return get_comments(page)

This write-up hasThe three essences1) Use Session to keep the connection 2) Timeout mechanism to prevent jamming 3) Replace the proxy node when auto retrying

Newbie FAQ QA

Q：Why the data returned after using proxy is not in the right format?
A: Ninety percent of the proxy server returned an error page, it is recommended to use curl to test whether the proxy is smooth!

Q: How to deal with the problem of blocked high-frequency requests?
A: Recommended for ipipgoconcurrent proxy poolTheir dynamic IP pool supports 200+ rotating requests per second!

Q: The json() method reports an error but prints response.text with data?
A: The probability is that the response header carries BOM characters, try to use response.content.decode('utf-8-sig')

The Ultimate Pit Avoidance Program

I recently discovered that ipipgo has akiller feature: Their API can directly return the cleaned JSON data. For projects that require rapid development, you can directly use their preprocessing services to save yourself the trouble of dealing with all kinds of dirty data.

One last reminder: when dealing with JSON be sure toA priori state reanalysisNetwork problems in a proxy environment are ten times more complex than local ones. Use ipipgo's IP health monitoring feature to detect failed nodes in advance and avoid wasting time on error handling.