When the API hit the privilege control proxy IP can do what? Recently, many friends doing API development complained to me that their interfaces are always called by unidentified guys. Just like your security door is unlocked, anyone can come in and take things. This time we have to rely on the proxy IP + authority control of the golden couple. Proxy I...

When the IP is blocked, your crawler is still running naked? Brothers who engage in crawlers understand that they are most afraid of waking up to find that their IPs have been blacked out by the target website. The hard-written scripts are suddenly mute, and the data flow is more difficult than the Internet. At this time, if you can receive an alert in advance, it is like installing a pacemaker for the crawler, when it is critical...

When the ELK system meets the proxy IP anomaly, what should be done? Recently encountered a very interesting case: a company to do log analysis with ELK, found that every morning at three o'clock always appear a large number of request failure. The operation and maintenance brother tossed half a month did not find the reason, and finally found that the fixed IP is the target site as a robot to...

When the crawler hit the counter-crawler, your IP is okay? The old iron engaged in crawlers must have experienced such a scene: yesterday also ran a good script, today suddenly 403. At this time, do not rush to smash the keyboard, eighty percent of your IP is the site stared at. Just like you go to the supermarket to try to eat always remembered by the clerk looks, crawlers ...

Turn wget into a secret weapon of data harvester Those of us who are engaged in data collection know very well that using wget to download something is like driving a tractor to harvest wheat - simple and rough, but with a lot of noise. If you don't disguise yourself well, you will be exterminated by the target website as a pest in a minute. Today we will teach the guys how to give this tractor ...

Why is mobile data collection always blocked? Brothers who have engaged in mobile data collection know that the biggest headache is that the IP is blocked. Especially when using Appium to do automated testing, the device is connected to the same WiFi to run scripts, and the target server can recognize it in minutes. Last week there was a little brother who did e-commerce price comparison with me...

When the crawler encounters data mutilation, is your cleaning process hardcore enough? Brothers engaged in data collection understand that the hard work of crawling down the data is often missing arms and legs. Just like we go to the supermarket to buy special goods, there are always a few empty spaces on the shelves especially eye-catching. If you don't know how to deal with the missing values at this time, the subsequent sub...

Dude, is your proxy IP reliable or not? The old Zhang, who is engaged in crawling, has a big head recently, and the thousands of proxy IPs saved in his hand are like opening a blind box. The script that just ran yesterday suddenly went on strike today, and he was so angry that he slammed the table. This is something I know too well, batch verify the survival rate of proxy IP...

Teach you how to use Selenium to hang the proxy brothers engaged in automated testing understand that sometimes do not hang the proxy simply can not run up. Today we will nag how to use Python, Java, C, the three languages to Selenium set on the proxy, focusing on our old buddy ipipgo's proxy service. Do not organize those false ...