Academic Data Collection: An Example of Journal Paper Crawler Development

Why do academics need to build their own crawlers?

Recently, I've been helping a few graduate students with their dissertation data, and I found that they are using the most primitive method: downloading articles from journal websites manually. One of them clicked the mouse for two days to download 300 articles, and the website blocked his IP address, which made me realize that many academics need to download articles from journals.Automated Acquisition Tools, but are afraid that the technological barriers are too high.

In fact, nowadays, writing basic crawlers in Python is as easy as learning to make scrambled eggs with tomatoes. The key problem is that the anti-crawl mechanism of many journal platforms is stricter than the neighborhood gates. This is when you need toproxy IPTo be your "cloak of invisibility", especially like ipipgo this kind of service providers specializing in dynamic IP pools, can let you like the Monkey King pulling hairs to change the split like, easy to bypass the access restrictions.

Teach you to match the proxy crawler by hand

First prepare three things: Python environment (recommended 3.8 or more), requests library, and ipipgo API key. Here is a small pit to pay attention to, don't directly use the free proxy, nine out of ten are pits. Last year, I tried a certain treasure to buy a cheap proxy, the result of the downloaded paper mixed into the small yellow text, the scene was once very embarrassing.

Core Configuration Steps:
1. Go to the official ipipgo website and register, then select theirAcademic-only packages(with high anonymity)
2. In the code to set up a rotating proxy, it is recommended that every 5-10 requests to change the IP
3. Remember to add a random delay, do not let the site find out that you are a robot

To give a real case: when climbing a core journal website, the 7th request will be blocked if you don't add a proxy. After using ipipgo's dynamic IP, it ran 2000 requests in a row and was as stable as an old dog. Their API call is simple, just add a proxies parameter to the requests:

proxies = {
    "http": "http://用户名:密码@gateway.ipipgo.com:端口",
    "https": "https://用户名:密码@gateway.ipipgo.com:端口"
}

Avoiding the tawdry maneuver of backcrawling

Now the journal site of the anti-climbing sets more and more, in addition to changing IP also pay attention to these:

Here's a cold one: with ipipgo'sResidential AgentsIt is not easy to be recognized than the data center proxy. The last time I climbed know.com, the success rate of using ordinary proxy is only 60%, change to residential proxy directly soared to 92%. but pay attention to the academic ethics, don't make people's servers crashed.

Five common pitfalls that white people step into

Q: Why does my crawler work at first and then suddenly fail?
A: eighty percent of the IP was pulled black, remember to change the IP as often as changing socks. suggested in ipipgo background settings automatic switching frequency

Q：Why can't I open the downloaded PDF?
A: It may have triggered the anti-climbing mechanism of the website and returned an error page. Teach you a trick: add a file header check in the code, found that the file is less than 10KB automatically retry!

Q: What if the crawler is slow as a snail?
A: Don't open multiple threads and rush, spread out the requests like a guerrilla war. Use ipipgo's API with asynchronous requests for a 3-5x speed boost!

Q: Will I be held legally responsible?
A: comply with the robots agreement + control the intensity of access + only for academic purposes, generally no problem. Last year I used this method to help my tutor crawl over 80,000 documents, and now the papers are published

Q: How do I choose a package for ipipgo?
A: Beginners are advised to chooseFlexible Traffic PackI bought 50G first to test the waters. Their traffic calculations are very realistic, unlike some platforms that are watered down

Speak from the heart.

Academic data collection is like fighting a tunnel war, which requires both technology and strategy. Proxy IP in this matter is like the energy treasure of the Transformers, and choosing the right one is twice the result with half the effort. After using ipipgo for half a year, the biggest feeling is their IP pool.Updated fast enough.I'm not sure if I've ever seen a customer who's been on the phone for more than a year, but I'm sure if I've ever seen a customer who's been on the phone for more than a year.

Finally, I would like to remind you that there are thousands of rules for crawlers, but the first rule is to obey the law. Don't try to paralyze other people's websites, we do academic to talk about morality. If you are really unsure, ipipgo's technical support can help you look at the code for free, and remember to pull the wool.