Python parsing JSON methods: Python proxy JSON processing program

First, to engage in JSON data for what to pull on the proxy IP?

I'm sure you've encountered this before: when you're using Python to grab data, the target website suddenly dumps a"429 Too Many Requests"The first thing that you need to do is to get your hands on a proxy IP. At this time, the proxy IP will come in handy, as if playing the game to open a small number, change the identity to continue to wave. Especially when dealing with JSON interfaces, frequent requests are blacked out in minutes, using ipipgo's proxy pool to rotate IPs, just like a crawler installed with aAutomatic Face Changing MaskThe

import requests
from ipipgo import get_proxy Assuming this is their SDK

proxy = get_proxy(type='dynamic') dynamic residential IPs
response = requests.get('https://api.example.com/data', proxies={"https": proxy})
json_data = response.json()

Second, Python play JSON's three great tricks

first moveAvalanche Prevention Tips: Wrap json parsing with try-except to avoid a sudden and violent death of the program.

import json

raw_data = '{"name": "Zhang San", "age": 28}'
try: data = json.loads(raw_data)
    data = json.loads(raw_data)
    print(data['address']) This key does not exist!
except json.JSONDecodeError: print("The key does not exist!
    JSONDecodeError: print("This is not proper JSON!")
except KeyError as e.
    print(f "Dude, you wrote the wrong field: {e}")

second movedata grooming: Use the indent parameter of json.dumps to make the data typeset like a star resume:

ugly_data = {"company": "ipipgo", "services":["Proxy IP", "Cloud Server"]}
print(json.dumps(ugly_data, indent=2, ensure_ascii=False))

C. Proxy IP with JSON Practical Manual

With ipipgo.TK line agentWhen you do, remember to set up a timeout and retry mechanism, and the code should be written this way:

from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount('https://', HTTPAdapter(max_retries=3))

proxies = {
    'https': 'socks5://user:pass@tk.ipipgo.com:3000'
}
try.
    resp = session.get('https://api.xxx.com', proxies=proxies, timeout=5)
    print(resp.json().get('data'))
except Exception as e.
    print("Request pounced, change IP and keep going!")

Four, white common overturn scene QA

Q: What should I do if my proxy IP is not working?
A: ipipgo's dynamic residential IP comes with aAutomatic change function, set in code to change IP every 5 minutes, steady as an old dog.

Q: There is garbled code in the returned JSON data?
A: In the requests add aresponse.encoding='utf-8′, or just use response.content.decode('unicode-escape')

v. ipipgo package selection guide

One last flirty maneuver: using ipipgo'sSERP APIDirect access to search engine structured data, even the effort of parsing JSON are saved, true - lying flat development!