How to use public data without stepping on the line? Handy guide to avoid the pitfalls
Nowadays, friends engaged in data research are facing a headache: there is so much public information on the Internet, how in the world to use it legally? Last year, a university team was sued for crawling enterprise information, which gave the industry a wake-up call. Let's be honest here.Using a proxy IP doesn't let you steal data, it helps you work safely within the rulesThe
First, the three forbidden areas of data use should not be touched
1. Personal privacy is a high-pressure lineID card number, cell phone number of these sensitive information, even if the public in the web page can not be collected. Last year, a company in Hangzhou collected cell phone numbers when crawling user reviews, and ended up with a 500,000 dollar fine!
2. Don't reach for trade secrets.
3. Reptile don't be a demolition crewSome novice technicians in order to catch up with the progress, open the multi-threaded crazy request, the other server to crash a few examples. This time with dynamic proxy IP rotation request, just like the car installed transmission, both to ensure speed and will not burst the cylinder!
| take | Hazardous operations | correct posture |
|---|---|---|
| Price monitoring | 24-hour non-stop crawling | Capture in 3 periods per day, each time with different IPs |
| Public Opinion Analysis | Grab user comments + personal information | Capture only the content of the public text |
Second, the correct way to open the proxy IP
We have to apologize for this.ipipgoThe services of their family's originalBusiness Scenario Matching ModelIt does work well. For example, if you do academic research, pick theirDedicated Academic AccessThe IP pool automatically controls the frequency of requests and also intelligently avoids sensitive websites.
Take a real case: an e-commerce team needs to compare prices, with ordinary proxy IP request 500 times per hour was blocked. Change to ipipgoBusiness Compliance PackageAfter the system automatically disperses the request to 200 IPs, each IP is only sent 2-3 times per hour, the data is collected as usual, and the platform does not notice any abnormality.
Third, the small white must see the operation of the manual
1. Look at the robots protocol first: just like entering someone's home to knock on the door first, the website / robots.txt file will write which directories are not allowed to climb!
2. Setting the acquisition interval: Set the request interval to more than 5 seconds in the ipipgo backend, don't act like a hungry man grabbing his food!
3. Data desensitization: Coding of user nicknames, such as "Zhangsan" and "Li".
IV. Answers to frequently asked questions
Q: Is it illegal to collect data with a proxy IP?
A: The tool itself is fine, it depends on how it is used. Just like a kitchen knife can cut vegetables or hurt people, it is recommended to choose a service provider like ipipgo that provides compliance guidance
Q: Why am I blocked when others can use the same proxy IP?
A: Many newbies are planted on UA settings. Remember to add random User-Agent in the collection code, ipipgo's API supports one-click injection of this function!
Q: Is it cost-effective for companies to use a self-built proxy pool or buy a service?
A: Unless there is a professional operation and maintenance team, it is truly recommended to use the ready-made. The IP blocking rate of self-built proxy pools is generally above 40%, and the commercial version of ipipgo can suppress the blocking rate to less than 5%.
At the end of the day, data use is like getting water from a river.Neither draining the river nor polluting the water supply. Choosing the right tool is only the first step, the key is to have a scale in your mind. Next time you're in a situation where you're not sure what to do, take a look at the ipipgo website and check out their compliance whitepaper, which is written in a way that's more understandable than a lot of legal documents.
English 
简体中文 
Español 
Deutsch 
Français