Google Scholar API: Thesis Data Collection Interface

Google Scholar messing with data? A handy guide to avoiding the pitfalls with proxy IPs

Academics know that Google Scholar is a big treasure trove. But really want to batch pick thesis data, the official API has long been closed to the public. At this time, we have to show their skills, many technology geeks choose to write their own crawler. But the problem is--Your IP will be blocked in minutes.I'm not sure if you're going to be able to do that! Today we'll be chattering about how to use proxy IPs to glean data safely and efficiently.

Why your crawler doesn't live more than three minutes?

Google's anti-crawl mechanism is not vegetarian, mainly look at these three indicators:

1. the frequency of requests from a single IP
2. whether the request header looks like a real person
3. JavaScript authentication levels

Especially the first one, the average home broadband on a public IP, crazy request, light flow limit heavy blocking. Last month, a doctoral student told me that he wrote a script to run at 2:00 a.m., the result is that at 3:00 a.m., the IP was blocked, and the thesis was almost open sky window.

Proxy IPs are the way to go.

The principle of this thing is as simple asLet different couriers deliver your packagesDynamic Residential Proxy from ipipgo is the best, why? Look at this comparison table:

ipipgo's residential agents realistically tested down theAuthentication is not triggered by 500 consecutive requests. The key is that their IP pool is updated daily with 20% and is not easily tagged.

The actual code is written like this

Using Python as an example, remember toRandomly switch User-Agentrespond in singingControl request interval::

import requests
from itertools import cycle

proxies = cycle(ipipgo.get_proxy_list()) get dynamic IP pools

headers_list = [
    {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0)...'} ,
    {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel...'}
]

for page in range(1, 100): proxy = next(proxies).
    proxy = next(proxies)
    try: response = requests.get()
        response = requests.get(
            'https://scholar.google.com/scholar', proxies={"http": proxy, "https": proxy}
            proxies={"http": proxy, "https": proxy},
            headers=random.choice(headers_list), timeout=10
            timeout=10
        )
         Processing the data here...
        time.sleep(random.uniform(2,5)) Random pause
    except Exception as e.
        print(f "Flipped with {proxy}, switch to the next one!")

Common Rollover Scene QA

Q: Why do I still get blocked after using a proxy?
A: Three possibilities: 1. IP quality is not good 2. request header did not randomly change 3. speed is too fast. It is recommended to use ipipgo's intelligent rotation package, comes with request frequency control.

Q：What package should I choose if I want to collect 100,000 pieces of data?
A: directly find ipipgo customer service to customize the program, academic use has exclusive discounts. Personal use choose the monthly payment of 199 package is enough, enterprise use is recommended to buy concurrent packages.

Q: Is this illegal?
A: Academic use is basically fine as long as it is not commercial or maliciously offensive. Remember to add in the headers'Referer': 'https://scholar.google.com/'Safer.

Tell the truth.

Don't believe in those free agents, nine out of ten are pits. I've seen people use free IPs before, and as a result, they climbed to the data of all the phishing sites. ipipgo costs money, but the IP pool has aReal Life Housing IP, and can also be billed by volume. Especially with their smart routing feature, which automatically avoids the IP of being, the saving is not a little bit.

Last reminder: don't write dead IP addresses in your code! It is best to use the API they provide to get the latest proxy in real time, so that even if a certain IP hangs, it can be automatically switched. It's not easy to be an academic, so climb and cherish it.