Google Trend Crawler: Keyword Data Collection

How does Google Trend Crawler really work?

Old-timers who are into number dwelling analytics know that Google Trends is a goldmine. But directly on the hand to pick the data? Minutes to your IP whole rest! This is the time to use ourThe Proxy IP MethodIt's like playing a game and opening a small number, changing your armor every time you log in. It's like playing a game with a small number and changing your vest every time you log in, the system doesn't recognize who you are.

Why do I have to use a proxy IP?

To give a chestnut: you continuously 10 times to check the "double eleven promotions", Google immediately pull your IP blacklist. If you use ipipgo's dynamic proxy pool, each request automatically change IP, like a Sichuan opera face like, guaranteed to catch the data catch fly up. The point is:

import requests
proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}
response = requests.get('https://trends.google.com/trends/api/dailytrends', proxies=proxies)

The username and password in this code, remember to replace them with your own authentication information that you got in the backend of ipipgo. Port number 9020 is their exclusive high-speed channel, measured more than 3 times faster than ordinary ports.

Grabber build in four steps

1. Fake BrowserRemember to add User-Agent, so you don't look like a robot.
2. IP Rotation Strategy: ipipgo supports automatic switching of nodes by number of requests
3. Exception handling: Hibernate for 30 seconds after encountering error code 429.
4. Data Cleaning: Google returned JSON with a bunch of garbled prefixes, have to use the string interception method

Guidelines for demining common pitfalls

Q: Why can't I catch the data all of a sudden?
A: eighty percent of the IP pool quality is not good, recommended to use ipipgoCommercial level agentsThey have a 99.2% IP survival rate.

Q：数据怎么破？
A: Add this parameter in the request header:

headers = {'Cache-Control': 'no-cache'}

Q: What should I do if I encounter a CAPTCHA?
A: hurry up and change ipipgo'sResidential AgentsThe program is designed to be a real user's behavior. Tested to catch 8 hours continuously without stopping

Exclusive advantages of ipipgo revealed

Lastly, don't try to use a free proxy, that thing nine out of ten is a pit. To engage in long-term stability of data, or have to use ipipgo such professional service providers, new user registration also send 10G traffic trial, incense very much!