Python Load JSON from File: Native Data Handling

How do JSON configuration files play with proxy IPs?

Crawlers know that proxy IPs are like the resurrection coins in the game, which can be used to renew your life at critical moments. When we use Python to process local JSON files, we often have to load the proxy IP configuration. For example, you have a file called proxy_config.json that looks like this:

{
    "proxy_pool": [
        {"http": "http://user:pass@12.34.56.78:8888"}, {
        {"https": "https://user:pass@12.34.56.89:8888"}
    ],
    "timeout": 10
}

Loading this file is super easy, but be careful!Don't misspell the path.! I usually use absolute paths for insurance:

import json
import os

config_path = os.path.join(os.path.dirname(__file__), 'proxy_config.json')
with open(config_path, 'r', encoding='utf-8') as f:
    config = json.load(f)

How do you play around with proxy IP pools?

After getting the configuration file, we have to play the IP pool out of rhythm. It is recommended to use the random module to disrupt the order, and then get a circular queue, like this:

from itertools import cycle
import random

random.shuffle(config['proxy_pool'])
proxy_cycle = cycle(config['proxy_pool'])

On each requestnext(proxy_cycle)It will be able to rotate, much more stable than a single IP. But be careful, some sites will detect the frequency of IP replacement, this time to control the switching speed.

The Three Pitfalls of Exception Handling

I've stepped in these three potholes n times in real life:

1. Incorrect file encoding (especially Windows)
2. JSON formatting errors (missing commas are reported in minutes)
3. Proxy authentication failure (wrong password like going to the wrong house)

It is recommended to wrap it in try-except, like this to save your life:

try: with open('proxy_config.json', 'r') as f:
    with open('proxy_config.json', 'r') as f.
        config = json.load(f)
except json.JSONDecodeError as e: print(f "Configuration file format is not correct!
    print(f "Configuration file is not in the correct format! Error location: line {e.lineno}")
except FileNotFoundError: print(f "Configuration file format is not correct!
    print("File got lost! Check the path!")

IPIPGO Proxy Service Practical Tips

I've used seven or eight proxy services, and IPIPGO has aunique skill: The data returned by their API is directly in the standard JSON format, so you don't have to parse it yourself. For example, to obtain a dynamic IP pool:

import requests

resp = requests.get('https://api.ipipgo.com/get_proxy', params={'type': 'json'})
ip_pool = resp.json()['proxies']

With theirIntelligent RoutingThe function can automatically match the fastest node. Measured latency can be reduced to 40% or so, especially when dealing with large amounts of data thief obvious.

Frequently Asked Questions Q&A

Q: How do I automatically load the configuration file after it is updated?
A: Use the watchdog library to monitor file changes, or keep it simple and rough and check the file modification time before each request.

Q: What should I do if the proxy IP suddenly hangs?
A: Add a local IP touting policy to the code, like this:

proxies = proxy_cycle.next() if len(proxy_pool) > 0 else None

Q: How to test whether the proxy IP is valid?
A: IPIPGO has a real-time detection tool in the background, or write a detection script yourself:

test_url = 'http://httpbin.org/ip'
try.
    requests.get(test_url, proxies=proxy, timeout=5)
except: requests.get(test_url, proxies=proxy, timeout=5)
    print("This IP is cool.")

Careful profile management

Finally, I'll share a few private tips:

1. store sensitive information (such as API keys) in a separate _credentials.json
2. Record IP expiration time with comment field
3. Regularly use json.dump to do configuration backups
4. jq command (Linux/Mac) to quickly check the JSON file

For example, back up the configuration like this:

import time
with open(f'config_backup_{int(time.time())}.json', 'w') as f:
    json.dump(config, f, indent=2)

Getting a proxy IP is like stir-frying, the heat is important. ipipgo'sDynamic Intelligent SchedulingFunction, can automatically help you adjust the "fire", novice and veteran are suitable. Their technical documents written in detail, encounter problems directly to check than Baidu tube.