Python processing JSON: Python proxy JSON processing tutorials

First, why use proxy IP to deal with JSON data?

Do data capture brothers understand, directly take their IP hard to dislike the site is easy to eat the door. For example, you want to batch crawl weather data, commodity prices, these exist in the JSON format of the information, a dozen consecutive requests may be blocked IP. This time with ipipgo's proxy IP pool, just like to the crawler wearing thecloak of invisibility, change to a different vest for each request and the success rate is directly doubled.

Second, Python processing JSON three great skills

First, the whole understand the basic operation, here to teach you three practical tricks:

import json

 1. string to dictionary (like unpacking a courier)
data_str = '{"city": "Shanghai", "ip": "192.168.1.1"}'
data_dict = json.loads(data_str)

 2. dictionary to string (packaged for shipment)
new_data = {"status": 200}
json_str = json.dumps(new_data)

 3. file read and write operations (important data in the safe)
with open('data.json', 'w') as f.
    json.dump(data_dict, f)

Third, with the proxy IP real combat to catch the data

Take a real case: use ipipgo's dynamic residential proxy to capture a website's API data. Look at the proxy configuration section:

import requests

proxies = {
    'http': 'http://用户名:密码@proxy.ipipgo.com:端口',
    'https': 'http://用户名:密码@proxy.ipipgo.com:端口'
}

response = requests.get(
    'https://api.example.com/data',
    proxies=proxies,
    timeout=10
)

 Handling possible garbled code
if response.encoding == 'ISO-8859-1': response.encoding = response.encoding == 'ISO-8859-1'.
    response.encoding = response.apparent_encoding

data = response.json()
print(data.get('temperature'))

Key note: ipipgo's proxy address is in the backend"My package."You can find it in, support HTTP/HTTPS dual protocol, remember to replace your account password.

Fourth, the black technology to deal with multi-layer nested JSON

Don't panic when you come across this sick structure:

{
  "result": {
    "proxies": [
      {"ip": "1.1.1.1", "speed": 200}, {"ip": "2.2.2.2", "speed": 150}
      {"ip": "2.2.2.2", "speed": 150}
    ]
  }
}

Extract IP list in one step with jsonpath:

from jsonpath import jsonpath

ips = jsonpath(data, '$..proxies[].ip')
print(ips) output ['1.1.1.1.1', '2.2.2.2.2']

V. Practical common overturning scene QA

Q: What should I do if I can't connect to the proxy IP?
A: first check three elements: ① account password contains special characters (recommended URL encoding) ② port is not filled in the web page displayed ③ test TCP connectivity (telnet proxy.ipipgo.com port)

Q：json.decoder.JSONDecodeError报错怎么办？
A: eighty percent of the site returned an error page, first print (response.text) to see the original content, it is recommended to add try-except package:

try.
    data = response.json()
except JSONDecodeError: print("The returned data is not proper JSON!
    print("The returned JSON is not proper JSON!")

Six, ipipgo package how to choose the most cost-effective

Recommendations based on business scenarios:

Finally, a dry goods: using ipipgo client to manage proxies is more convenient than using API directly, and can automatically detect IP availability. Encounter technical problems directly to their technical customer service, the response speed is at least twice as fast as ordinary merchants - do not ask me how to know, are stepping out of the experience of stepping out of the pit.