Low-cost program: VPS build crawler proxy IP pool (Python practice)

First, why should I toss my own proxy IP pool?

Engaged in crawling old drivers understand, free proxy IP is like roadside stalls snacks - eat fragrant but easy to diarrhea. Either the speed is stuck to PPT, or when using it, it suddenlyhang up (a phone)The first thing you need to do is to build your own pool of agents. Building your own agency pool is like running a small kitchen at home, with controlled costs and fresh ingredients.

I recently helped a friend useAliCloud 2-core 2G burst energy instance(The annual fee is less than 300 dollars) built a stable operation of the proxy pool, the actual test can handle 50,000 + requests per day. The key is that you don't have to go around looking for free IPs every day, and the time you save to jerk a couple of games doesn't smell good, does it?

Second, VPS purchase anti-pit guide

Don't just look at the price when picking a VPS, these are the parameters you must keep an eye on:

Tested and foundLos AngelesThe machine is the most cost-effective, it is recommended to choose the KVM architecture, do not touch those over-sold serious business. Here secretly say a trick: many VPS merchants first month specials, you can woolgathering on the change of number (do not say that I teach)

III. Hands-on installation agent services

recommendedSquidBeing a proxy server, the configuration is as simple as building blocks:

 Install squid (Ubuntu as an example)
sudo apt-get install squid -y

 Modify the configuration file
sudo nano /etc/squid/squid.conf
 Add these two lines
http_access allow all
http_port 3128

 Restart the service
sudo systemctl restart squid

At this point your VPS is already a naked proxy server. But this is directly exposed to the public network, and you'll be swept into self-imposed isolation in minutes. You need to add aIP Whitelist Authentication::

 Creating a password file
sudo touch /etc/squid/passwd
sudo chown proxy /etc/squid/passwd

 Add user (replace your_password with your own)
sudo htpasswd /etc/squid/passwd ipipgo_user

 Modify squid configuration to add:
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
acl auth_users proxy_auth REQUIRED
http_access allow auth_users

Fourth, Python crawler access to the actual combat

Implementing proxy polling with the requests library, here's aAnti-Ban Tips: Randomly select a different exit IP for each request

import random
import requests

 Replace this with your own VPS information
proxies = [
    "http://user1:pass1@vps1_ip:3128", "http://user1:pass1@vps2_ip:3128", "http://user1:pass1@vps2_ip:3128",
    "http://user2:pass2@vps2_ip:3128", ...
    ... Add more proxies
]

def smart_request(url).
    try.
        proxy = random.choice(proxies)
        response = requests.get(url, proxies={"http": proxy, "https": proxy}, }
            proxies={"http": proxy, "https": proxy}, timeout=10)
            timeout=10)
        return response.text
    except Exception as e.
        print(f"{proxy} hung, move to the next one!")
        return smart_request(url) auto-retry

If it's too much of a hassle to maintain your own VPS, go straight to theProxy services for ipipgoLess hassle. Their API access is super easy:

 Get dynamic proxy from ipipgo (remember to replace API_KEY)
import requests

def get_ipipgo_proxy():
    api_url = "https://api.ipipgo.com/get?key=YOUR_API_KEY"
    return requests.get(api_url).text

 Example of use
proxy = get_ipipgo_proxy()
requests.get("target url", proxies={"http": proxy})

V. Handbook on Mine Clearance of Common Problems

Q: How much does it cost to build my own agent pool?
A: VPS annual fee of about 300 + ipipgo supplemental IP (9.9 / 10,000 requests), cheaper than using a dedicated proxy more than half!

Q: What should I do if my agent is slow?
A: ① check whether the VPS bandwidth runs full ② use ipipgo'shigh speed channelNode ③ Reduce the number of SSL handshakes

Q: What should I do if I keep getting my IP blocked?
A: three tricks: ① random User-Agent per request ② control access frequency ③ use ipipgo'scommercial agent(comes with automatic IP replacement)

VI. Maintaining the proxy pool for tart operations

1. Early morning each dayautomatic rebootsquid service (to prevent memory leaks)
2. Use crontab to check for agent survival at regular intervals:

 Survival Detection Script
! /bin/bash
if curl -x "http://你的代理" --retry 2 -m 10 ipipgo.com/check > /dev/null; then
    echo "Proxy alive"
else
    echo "Proxy hangs!" | mail -s "alert" your mailbox
mail -s "Alarm" to your mailbox

3. When encountering a focused target site, it is recommended to call it directly.Exclusive IP for ipipgoThe stability takes off as one IP is dedicated to a single service.

Finally, a big truth: maintaining your own agent pool is like having a pet, it takes time to take care of it. If you are doing a lot of business, it is recommended to useipipgo's commercial level agentsThey have a professional team to maintain the IP pool, but also with intelligent switching and request statistics, than their own toss to save a lot of heartache. Newbies are advised to use theirFree Trial PackageExperience how silky smooth professional agency services can be.