Federal Data Resources: Open Government Data Sets

When Government Data Meets Proxy IP

Recently, many friends doing data analysis have complained to me that there is obviously a gold mine in the government's public datasets, but collecting them is like playing with theGopher game--Just grabbed a few data IP is blocked. Last week, the old king in order to get traffic flow data, hard to use their own router to change the broadband 8 times, the result is pulled into the small black room by the operator.

The Data Mover's Survival Guide

Anti-crawling mechanisms for government websites are now getting more and more sophisticated, just like mall security guards remembering your face and not letting you in. This is the time toproxy IPto be a data mover "dressing props". For example, if you use ipipgo's residential agent, every time you request data, it's like changing into new clothes, and the site won't recognize the same person.

import requests

proxies = {
    'http': 'http://user:pass@gateway.ipipgo.com:9020',
    'https': 'http://user:pass@gateway.ipipgo.com:9020'
}

response = requests.get('https://data.gov.cn/api', proxies=proxies)
print(response.text)

Three must-haves in the real world

1. IP rotation frequencyDon't be stupid and cut IPs per second, adjust intelligently to site response. ipipgo's backend can be set upFailure auto switchIt's like a crash airbag for reptiles.

2. Don't panic when encountering CAPTCHA, spread the request to different regional nodes. Last week with ipipgo's Jiangsu + Anhui double node, the rate of CAPTCHA straight down 60%

3. Do not use the IP type of the morning collection of work hours, residential agents are more active at night, this cold knowledge most people do not know

Common pitfalls warning for white people

question-and-answer session

Q: Is it legal to collect with a proxy IP?
A: Just like registering accounts with different cell phone numbers, the government encourages the fair use of public data as long as it doesn't break the system and adheres to the robots protocol.

Q: What's unique about ipipgo?
A: His family has aIntelligent RoutingThe function can automatically match the most suitable export IP. last time to collect a certain economic data platform, the success rate from 47% directly pull to 89%, really fragrant!

Q: Does it burn a lot of money in the long run?
A: Compared to the business interruption caused by the blocked IP, the proxy cost is about the same as buying an insurance policy. ipipgo's hourly billing model is particularly suitable for intermittent collection needs

Finally, a cold knowledge: the government data platform of the anti-climbing system will be updated on the 1st of each month rules, remember to use ipipgo in advance of thetrial packageDo compatibility testing. After all, data collection, like fishing to choose the right bait, find the right tool to get twice the result with half the effort.