Python Reading JSON Files: Proxy IP Processing Local Data

Hands-on with Python to call the local proxy IP library

Brothers engaged in network crawlers should understand that the local storage of hundreds of thousands of proxy IP is too normal. Today we take Python to disk a disk that exists in the JSON file proxy IP, teach you how to quickly filter out the quality of resources can be used. Don't panic, even if you are just getting started, follow the steps to go absolutely understand.

import json

 It is recommended to use relative paths.
with open('proxy_pool.json', 'r', encoding='utf-8') as f:: proxy_data = json.load(f)
    proxy_data = json.load(f)

print(f "Successfully loaded {len(proxy_data)} proxy configuration items.")

The key point of this code above is thatDocument encoding formatMany newbies fall into the trap of json files with Chinese comments or special symbols. If you report encoding errors, try to change the encoding parameter to gbk or delete non-essential content in the file.

Top 3 Tips for Filtering Effective Agents

Don't rush the raw data when you get it, do three rounds of screening first:

Here are the highlightsProtocol type judgmentMany proxy service providers (such as our ipipgo) provide support for multiple protocols at the same time. It is recommended to use type filtering to categorize the different protocols of the proxy, so that when you call later, you will not be strung out.

Real-world verification of proxy validity

The following validation code is recommended to be bookmarked to automatically exclude failed nodes:

import requests
from concurrent.futures import ThreadPoolExecutor

def check_proxy(proxy)::
    try: resp = requests.get('')
        resp = requests.get('http://httpbin.org/ip',
                          proxies={'http': proxy}, timeout=2))
                          timeout=2)
        return True if resp.status_code == 200 else False
    return False if resp.status_code == 200 else False
        return False

 Accelerating validation with a thread pool
with ThreadPoolExecutor(max_workers=20) as executor: results = executor.map(check_proxy, proxy_list)
    results = executor.map(check_proxy, proxy_list)

valid_proxies = [p for p, v in zip(proxy_list, results) if v]

Note that the test address do not use sensitive sites, easy to trigger anti-climbing. It is safe and reliable to use httpbin for testing, and it can also return the current IP information. If the pass rate is low, we recommend switching toipipgoThe stable agency service of their family can survive to 95% or more.

QA Session: A Guide to Avoiding Pitfalls

Q: What should I do if I read the JSON file and report an encoding error?
A: 90% of the probability is that the file is mixed with the BOM header, use Notepad to save as UTF-8 format, remember to select the "no BOM" option!

Q: What should I do if the program gets stuck when I verify the agent?
A: 80% is not set timeout parameters! requests timeout must not be less, it is recommended to set between 2-3 seconds!

Q: Is there a solution for local agent pools that are too cumbersome to maintain?
A: Direct access to ipipgo's API service, they provide real-time update of the proxy list, much more trouble-free than maintaining it yourself. New users can also get a 5G traffic trial, enough to run a small project!

Long-term maintenance tips

Lastly, I'd like to give you a sweet suggestion: run an auto-detection script with crontab or a scheduled task on a regular basis, and mark the invalid proxies. With ipipgo's dynamic IP pool as a supplement, you can basically say goodbye to the bad thing of IP being blocked. Remember, a stable proxy resource is the cornerstone of the success of the crawler, don't be stingy on the basic configuration.

If you're still confused after reading this, go directly to ipipgo's website and look at their technical documentation, which is much more detailed than what I have here. Especially the intelligent scheduling function, which can automatically match the best proxy according to the target website, whoever uses it will know.