Proxy IP Automation Test Script_Python Bulk Verification of Availability

Practical Guide: Quickly Verify Proxy IP Pool Quality with Python

Friends who do data collection know that the quality of proxy IP directly affects the success or failure of the project. The proxy IPs on the market are mixed, and manual testing is too inefficient. Today we teach you to use Python to write an automated validation script withipipgoThe high quality proxy resources, half an hour to get thousands of IP availability testing.

Build a basic testing framework

Prepare the three elements needed for testing first:

1. Proxy IP source: throughipipgoThe API to get the real-time IP list of their home residential IP covers 240+ regions worldwide, suitable for a variety of business scenarios

2. Detection targets: it is recommended to choose stable and well-known websites (such as the official website of search engines), while preparing multiple detection addresses

3. Validation metrics: three core metrics: response speed, status codes, content matching

import requests
from concurrent.futures import ThreadPoolExecutor

def check_proxy(proxy, test_url)::
    try: response = requests.get(test_url, test_url, test_url).
        response = requests.get(test_url,
                              proxies={"http": proxy, "https": proxy}, timeout=10))
                              timeout=10)
        if response.status_code == 200: return True, response.elapsed_code == 200.
            return True, response.elapsed.total_seconds()
    except.
        pass
    return False, 0

多线程代理ip检测

It takes 20 minutes to detect 100 IPs in a single thread, and the efficiency is improved significantly after changing to multi-threading. According to the computer configuration to adjust the number of threads, ordinary computers recommended 20-50 threads:

def batch_check(ip_list):
    results = []
    with ThreadPoolExecutor(max_workers=30) as executor:: [executor.submit(check_proxy, ip, '') for ip in ip_list].
        futures = [executor.submit(check_proxy, ip, 'https://检测地址') for ip in ip_list]
        for future in as_completed(futures): results.append(future.append(as_completed))
            results.append(future.result())
    return [ip for ip, (status, speed) in results if status]

Intelligent retry mechanism

Network environment is complex, it is recommended to set 2 retries for each IP to avoid misjudgment. Special attention:

- Separate detection of different protocols (HTTP/HTTPS/SOCKS5)

- Automatically add account passwords in case of 407 authentication errors

- Record the response rate of each IP for subsequent quality grading

Practical QA

Q：When the tested IP is actually used, it is invalid?

A: It is recommended to add the function of randomly accessing different websites in the script to avoid detecting websites being specially handled by agents

Q: How do I verify high anonymous proxies?

A: Add header parsing to the detection script to check for leaked fields such as X-Forwarded-For

Q: What should I do if the domestic agent is slow in detection?

A: Recommendedipipgo的区域定制服务，直接获取目标地区的住宅IP，实测可降低60%以上

Tips for Maintaining a Proxy Pool

Quality IPs filtered through scripts are recommended to be maintained this way:

1. Hourly automatic survival rate detection

2. Classified by response speed as fast/medium/slow

3. Automatic elimination of IPs with 3 consecutive detection failures

4. Prioritization of useipipgodynamic residential IPs, their IP survival cycle is 3-5 times longer than ordinary proxies

Final reminder: don't chase 100% availability, focus on keeping the agent pooldynamic equilibrium. Suggested Matchipipgos intelligent scheduling API to automatically replenish fresh IPs so that maintenance costs can be reduced by more than 70%.