Twitter Crawl: Compliant Tools for Getting Tweets

Hands-on teaching you to use proxy IP to securely glean Twitter data

Recently, many of my friends in overseas markets have complained to me about IP blocking when they use scripts to capture Twitter data.ipipgoThe dynamic IP pool of the only complete solution. Today, I'll break down my real-world experience and make sure you can play around with Twitter data collection after reading this.

Why is your crawler always blocked?

Twitter's anti-crawl mechanism is more savvy than its own bosses, staring at three main metrics:

Dynamic IP pooling is the real deal

Before, using a fixed proxy IP was like taking a shower in a raincoat - you had to get wet. Then I switched toipipgoThe residential dynamic IP, each request automatically change the real user IP. measured 12 hours of continuous capture, the success rate is stable at 98% or more.

import requests
from itertools import cycle

 The address of the proxy pool provided by ipipgo
proxy_pool = [
    '103.21.163.76:8000',
    '45.89.123.142:3128', '198.55.112.89:8080', '198.55.112.89:8080'
    '198.55.112.89:8080'
]

proxies = cycle(proxy_pool)

for page in range(1, 100): current_proxy = next(proxies)
    current_proxy = next(proxies)
    current_proxy = next(proxies)
        response = requests.get(
            'https://api.twitter.com/xxx',
            proxies={'http': current_proxy},
            timeout=10
        )
         Processing data...
    except Exception as e.
        print(f "Changing IP to continue: {current_proxy} kneeling")

A guide to avoiding the pitfalls (a must-see for beginners)

Don't use a data center IP!Twitter now recognizes server room IP segments, and using such IPs is tantamount to blowing yourself up. Suggested choicesipipgoThe residential IP packages, their IPs are all real home broadband, and they are personally tested to be effective.

Don't be too regular in your request intervals, all human operations have shaky hands. It is recommended to use a random delay:

import random
import time

 Randomly wait 3-8 seconds
time.sleep(random.randint(3,8))

QA First Aid Kit

Q: Why do I still get blocked with a proxy IP?
A: 80% of the IP quality is not good, or the request frequency is too high. Replace it withipipgoof a pool of quality IPs, while cranking up the request interval to 5 seconds or more.

Q: How many IPs are needed to be sufficient?
A: 50 rotating IPs are enough if you pick 10,000 pieces of data per day. Don't be greedy.ipipgoThe base package is perfectly adequate to make.

Q: What should I do if I encounter a CAPTCHA?
A: Immediately deactivate the current IP, change the new IP to reduce the collection speed. Really can't get it can private message me, give you a anti-CAPTCHA tart operation.

Tell the truth.

Don't believe in those free proxies, either the speed is slow or the survival time is short. I used a free IP at first, but I didn't get much data, but I was implanted with mining scripts. Now useipipgoThe monthly package, 1G bandwidth + exclusive IP, converted to only two dollars a day, much cheaper than buying coffee.