Crawler log analysis: automatic diagnosis system for abnormal requests

Why do crawlers always get pinched by websites?

Crawler veterans understand that the biggest headache is the site suddenly give you a face. Obviously, the code is written smoothly, the results of the log suddenly appeared a bunch of 403, 429, it is time to take out a magnifying glass to see the log. But manually check the log is like a needle in a haystack, especially with a fixed IP, the site wind control a catch.

To cite a real case: last year, there is a team of e-commerce price comparison, three consecutive days of data volume waist cut. Check the logs found that they used the fixed IP of the Beijing server room to catch a certain platform, the first 200 requests well, the 201st time directly ate the door. This is a typicalIP exposure characteristics are recognizedIt's the same thing as wearing the same clothes and going to the mall every day.

Anomaly diagnostic system four great skills

We've got to be able to build our own auto-diagnostic system:

How is IP health calculated?

Here's a wild card formula:
Health Score = (Number of Successes × 0.7) - (Number of Exceptions × 0.3) - (Response Time/1000)
For example, if an IP has 100 successes, 20 exceptions, and an average response of 800ms, the score would be (100×0.7)-(20×0.3)-0.8=68.2 points. Set a passing line of 60 points, below which the IP is automatically laid off.

Gotta settle for ipipgo's here.Dynamic Residential Agents, they have more than 2 million residential IPs in their IP pool, and each IP is used for up to 5 minutes to change. We've tested it in real life, and with the healthiness algorithm, we were able to keep the banning rate down to below 3%.

Real-world configuration tutorials

1. Log collector installs a Filebeat and pushes to ES
2. Get a dashboard with Kibana and focus on monitoring:
- Hourly Distribution of Abnormal Status Codes
- Top 10 Problematic IPs
- Average Response Time Curve
3. Write a Python script to poll ES data and call ipipgo's API to change the IP when the threshold is triggered

Focusing on ipipgo'sAPI AccessThe design of their home interface is thievery simple:

import requests
def get_new_ip(): url = "": url = "ip": url = "ip".
    url = "https://api.ipipgo.com/replace"
    params = {
        "key": "Your key",
        "type": "residential"
    }
    return requests.get(url, params=params).json()['ip']

Frequently Asked Questions QA

Q: How do I choose a proxy IP without stepping on mines?
A: Remember the three don'ts: don't use a data center IP (easy to be identified), don't use a shared IP (you take the blame for your neighbor's death), and don't be greedy for cheap (anything less than 50 cents/GB is definitely a problem). Exclusive residential proxy like ipipgo is a bit more expensive but steady as an old dog.

Q: How often do I engage in log analysis?
A: The logs are swept every 15 minutes during peak business hours, and can be relaxed to 1 hour during non-peak hours. If you find abnormal IPs, isolate them immediately, and don't feel bad about the proxy fee.

Q: Will IP switching too often be recognized instead?
A: This is where ipipgo is smart, their IP allocation strategy will mimic the rhythm of a real person's operation. For example, in the morning, they cut IPs more quickly, and late at night, they reduce the frequency of switching, so as to synchronize with the work and rest of real people.

What is the biggest benefit of having this system? Last month, a customer used the automatic diagnosis + ipipgo proxy, originally 3 hours a day to deal with the blocking problem, now the system itself to deal with, the operation and maintenance brother can finally on time off work.