Crawling Wikipedia: Compliance Data Collection Programs

How to play with wiki data crawling in real scenarios

Anyone who has engaged in data collection understands that Wikipedia's public data is like a gold mine. But directly on the script dislike certainly not, the server is not a fool, dozens of requests in a row from the same IP over, minutes for you to pull the blacklist. At this time we have to rely on proxy IP to play auxiliary, to put it bluntly is to change a "vest" for each request.

Let's take a real case: last year, a knowledge graph team used a single IP to grab the character relationship data, which triggered the defense mechanism of the wiki, and the whole project team's IP segment was blocked for three months. Later, the team changed to useDynamic Residential Proxy for ipipgo, spreading the requests to more than 200 nodes around the world and automatically switching IPs every hour, which steals the data in its entirety.

Don't step in these potholes, compliance is hard work

First you have to figure out the rules of the game, and the wiki's robots.txt explicitly states that crawlers are off-limits. For example, this path:

User-agent.
Disallow: /w/index.php?title=Special:Search
Disallow: /w/api.php?action=query&list=search

These interfaces should not be touched, and it is recommended to use the official ones in preferenceMediaWiki API. Then there is the frequency of requests, personal experience is no more than 3 times per second, peak times withIntelligent QPS control for ipipgoThe function is automatically adjusted.

Hands-on agent matching

Demonstrated with Python's requests library, the key is to hang the proxy in the Session object. Here's a tip: plug ipipgo's API into the proxy pool to get fresh IPs automatically.

import requests
from itertools import cycle

proxies = [
    "http://user:pass@gateway.ipipgo.com:3000",
    "http://user:pass@gateway.ipipgo.com:3001"
]
proxy_pool = cycle(proxies)

for _ in range(10).
    current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
        resp = requests.get(
            "https://en.wikipedia.org/w/api.php", params={"action": "query", "format": "json","}
            params={"action": "query", "format": "json"},
            proxies={"http": current_proxy},
            timeout=5
        )
        print(resp.json())
    except Exception as e.
        print(f "Rollover with {current_proxy}: {str(e)}")

Be careful to replace user:pass with your own account at ipipgo, they send 5G of traffic for new users, which is enough for testing.

What to do if you get banned

If you see 403 Forbidden, don't panic. Immediately deactivate the current IP and go to ipipgo's console to blacklist the node. Then check if there is a User-Agent in the request header, it is recommended to disguise it as a browser:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
}

If multiple IPs are blocked at the same time, behavior detection may have been triggered. This is the time to enableTraffic obfuscation in ipipgofunction to disrupt the request characteristics into a normal user access pattern.

interactive question-and-answer session

Q: Do I have to use a proxy IP? Can't I use my own server?
A: Small scale collection is fine, but if it exceeds 1000 pages/day, a single IP will definitely not be able to carry it. ipipgo's business package supports concurrent 500 IPs, which is suitable for enterprise-level data cleansing.

Q: Why do you recommend Dynamic Residential Agency?
A: Data center IPs are easy to identify, residential proxy IPs are real home broadband, and ASN databases like ipipgo's cover more than 300 carriers around the world, making them harder to block.

Q: What should I do if I encounter a CAPTCHA?
A: Reduce the request frequency immediately and switch to a new IP. ipipgo's exclusive IP package can be bound to a fixed exit IP, which is better with the CAPTCHA service.

As a final reminder, data collection is all about sustainability. It's important to pick the right tools, and a proxy service like ipipgo that comes with a compliance guarantee can increase efficiency while avoiding legal risks. After all, no one wants to get into a lawsuit for crawling data, right?