Tweets Grabber: Twitter Data Grabber API

For the data nerds out there, here's a look at the most stable position for Twitter crawling.

Recently, a lot of friends who do social media analytics have been complaining to me about how gleaning Twitter data the normal way is always limited. I know this too well! Last year, when I did competitive analysis, I used my own crawler script for three days in a row, and as a result, the IP was directly shut down in a small black room. Later, I found that using proxy IP rotation is the king's way, and today I will share this set of wild ways with you.

Why do your crawlers always flop?

Many newbies tend to fall into these potholes:
1. Single IP High Frequency Request: It's like trying food over and over again at the supermarket and not paying for it, and the clerk isn't on to you in a minute?
2. Too much concentration of IP segments: It's all IPs starting with 192.168 that go knocking on doors, and any fool knows it's the same people.
3. It doesn't simulate a real person.: Mechanical timed requests, not even mouse trajectory simulation

Last year, a customer doing public opinion monitoring used 10 fixed IPs to catch data in rotation, and all of them were banned on the third day, and then changed to use our ipipgo's dynamic residential IPs with random request intervals, and it ran stably for two months without overturning.

How to choose a reliable proxy IP?

Here's the kicker.Dynamic Residential IPThe IPs are exactly the same as those used by real users to access the internet. Like ipipgo's pool has 20 million+ such IPs, which are automatically switched with each request, so the platform can't tell whether they are real people or machines. Last time, there was a team doing Netflix monitoring, using their 1C package (5,000 IPs per day) to engage in cross-region data comparisons, and it properly ran for three months.

Hands-on API Configuration

Take Python for example, with the requests library + ipipgo proxy service:

import requests
from itertools import cycle

proxies = cycle([
    "http://user:pass@gateway.ipipgo.io:8000", "http://user:pass@gateway.ipipgo.io:8000", "http://user:pass@gateway.ipipgo.io:8000", "http://user:pass@gateway.ipipgo.io:8000", "http://user:pass@gateway.ipipgo.io:8000
    "http://user:pass@gateway.ipipgo.io:8001",
     Add more ports...
])

def get_tweets(keyword).
    current_proxy = next(proxies)
    try: current_proxy = next(proxies)
        res = requests.get(
            url="https://api.twitter.com/2/tweets/search/recent",
            params={"query": keyword},
            proxies={"http": current_proxy}, timeout=10
            timeout=10
        )
        return res.json()
    except.
        print(f"{current_proxy} hung, automatically switching to next node")
        return get_tweets(keyword)

focus onRemember to set a random delay (0.5-3 seconds), don't use a fixed SLEEP time. It is recommended to make the User-Agent into a polling pool, we ipipgo background has a ready-made UA generator can be gleaned directly.

Old Driver QA Time

Q: Why is it still blocked after using a proxy?
A: Ninety percent of the problem is the quality of the IP. Don't be cheap and use free proxies, those IPs have long been marked rotten. It is recommended to use ipipgo with automatic cleaning mechanism, their system will kick off the blacklisted IP in real time.

Q: What package should I choose to capture 100,000 levels of data?
A: Directly on the ipipgo enterprise customized version, support concurrency without limit. Last time, a 4A company invested in overseas projects, using their exclusive channel to pick 500,000 tweets a day, data cleaning directly into the BI system.

Q: What should I do if the API returns a 429 error?
A: This is triggering a rate limit. Three steps: 1. check request frequency 2. switch ipipgo's other geographic nodes 3. add retry-after logic to the request header

One last nag: now that the wind control of each platform has been upgraded, simply changing the IP is not enough. It is recommended to match the ipipgoBrowser Fingerprint Emulationfunction, disguising the canvas, webgl, and all these parameters, which is the true - stealth mode.