Microblog Crawler Proxy Pool: Microblog Data Collection Proxy Pool Building Solution

The biggest headache for microblogging crawlers: what to do about IP blocking?

The old iron engaged in microblogging data collection know that the most crushing is just run up to be blocked IP. is like going to the supermarket to buy snacks, just take two bags of potato chips on the security guards will not be allowed to enter. This is the time to learn"Change of armor."The proxy IP pool is your arsenal of a hundred different vests.

Proxy pools aren't casual. You have to be smart about it.

Many people think that the proxy IP is to buy a bunch of random can be used on the line, the results found that some IP even microblogging login page can not open. Here to teach you three must see indicators:

It's important to name names here.ipipgo's Static Residential PackageThe actual test can be stabilized to disguise as a real user in different provinces of the country, 35 dollars an IP with a whole month, cheaper than buying milk tea.

Teach you how to build a proxy pool by hand

Let's start with the core principle:Recycling + automatic phase-outIt's like eating rotary sushi. It's like eating rotary sushi, where fresh IPs are constantly replenished and those that fail are immediately removed. Here's a Python example:

import requests
 Pull the latest IP pools from ipipgo
def get_ips():
    api_url = "https://api.ipipgo.com/fetch?type=static"
    resp = requests.get(api_url).json()
    return [f"{ip}:{port}" for ip in resp['data']]

 Check if the IP is available
def check_ip(proxy): [f"{ip}:{port}" for ip in resp['data']]
    test_url = "".
        test_url = "https://weibo.com"
        resp = requests.get(test_url, proxies={'http':proxy}, timeout=5)
        return True if 'tweet' in resp.text else False
    else False
        return False

Be careful to set theRandomized sleep time, don't let Twitter think you're a robot that doesn't sleep 24 hours a day. Suggest adding a random.uniform(1,3) delay after each request.

Maintaining the agent pool for troll operations

Don't ever think you're done after building, here are two life saving tips:

1. 3am automatic blood change: Use crontab to update the IP of 30% in the early hours of every day, the wind control of microblogging is relatively lax at this time of the day.

2. IP quality scoring system: Record the number of successes, response rate for each IP, and prioritize the use of high scores, like this:

ip_score = {
    '122.96.1.1:8080': {'success':18, 'speed':1.2},
    '183.207.1.2:80': {'success':3, 'speed':4.5}
}

A must-see QA session for the little guy

Q: How many IPs should be enough for the proxy pool?
A: Ordinary collection of 200-300 dynamic IP is enough, if you do such high-frequency operation as public opinion monitoring, it is recommended to go on ipipgo's enterprise package, which supports doubling the number of concurrency.

Q: How to deal with the emergency when the IP is blocked?
A: Immediately do three things: 1. deactivate the IP 2. check the frequency of requests 3. switch IPs in different geographic areas. recommended in the code to add an automatic melting mechanism, 3 consecutive failures to trigger the alarm.

Q: Choose dynamic or static IP?
A: short-term collection with dynamic ($7.67/GB), long-term monitoring with static ($35/IP). There is a tawdry operation to mix it up: use dynamic IP for data collection and static IP for login state maintenance.

Let's get down to brass tacks.

Finally, we remind you, don't buy those cheap junk IP sold by the pound. before I saw someone with 0.5 yuan / GB proxy, the result of 40% IP even Baidu can not open. ipipipgo has a hidden function - - ipipipgo has a hidden function - - ipipipgo has a hidden function - - ipipipgo has a hidden function.Per request billing, especially for newbies who aren't sure how much to use, it doesn't hurt to use as much as you need.

If you come across a particularly tricky anti-climbing strategy, you can just ask their tech guy for theCustomized SolutionsWe have a project that needs to switch IP and UA at the same time. Last time we had a project that needed to switch IP and UA at the same time, they gave us an auto-association solution, which saved us half a month of time compared to tossing it out on our own.