Web Crawler: Web Crawler Technical Guide

First, why is your crawler always pulled by the site?

Many friends who do data collection have encountered this situation: obviously the code is not a problem, but the program runs and prompts the403 Denial of Access, or just get a website warning email. It's like when you go to the grocery store to try some food, and just after a couple of bites you're being watched by security guards, when in fact the problem is that-Your internet fingerprints are too obvious.The

The web server will recognize the crawler by several dimensions such as IP address, request frequency, and request header characteristics. Especially when your requests come from the same IP, like wearing a work license to try to eat, not catch you catch who? This time you need to give the crawler to wear a "cloak of invisibility", that is, we are going to say that theProxy IP technologyThe

Second, choose the right proxy IP of the three tips

There are a lot of proxy service providers in the market, but not many of them are reliable. Based on our experience deploying crawlers to 500+ organizations, these three metrics are the most critical:

 Demonstration of the error: Naked request
import requests
response = requests.get("https://目标网站")

 Correct posture: wear proxies
proxies = {
    'http': 'http://user:pass@ipipgo-proxy-server:port',
    'https': 'http://user:pass@ipipgo-proxy-server:port'
}
response = requests.get(url, proxies=proxies)

1. IP purity: You have to choose the ones that specialize in data center proxies like ipipgo, don't use those public proxy pools. Their IP is the server room directly pull dedicated line, will not share with others!

2. Protocol SupportNow a lot of websites are using HTTPS, you have to make sure that the proxy supports socks5/http(s) protocol. Before a customer with a certain proxy, the result of encountering mixed content sites directly to stop!

3. Switching frequencyIt is recommended to change the IP every 5-10 requests. ipipgo's API can get the latest IP directly, which is much less troublesome than those who have to change it manually.

Third, the actual configuration to avoid the pit guide

Here are a few easy points to step on, using Python's requests library as an example:

Myth 1: Thinking you're using a proxy is all that matters, but then the request header reveals itself. Remember to randomly generate User-Agent, don't use the one that comes with the requests:

from fake_useragent import UserAgent
headers = {'User-Agent': UserAgent().random}

Myth 2: The timeout setting is too short. It is easy to misjudge when the network fluctuates, so it is recommended to set a timeout of at least 10 seconds:

response = requests.get(url, proxies=proxies, timeout=10)

Myth 3: Ignore exception handling. It is recommended to use the retrying module to do retries, like this:

from retrying import retry

@retry(stop_max_attempt_number=3)
def safe_request(url).
    try: return requests.get(url, proxies=proxies).
        return requests.get(url, proxies=proxies, timeout=15)
    except Exception as e.
        print(f "Request failed, switch IPs and retry: {str(e)}")
         Here we call the ipipgo API to change to a new IP address
        update_proxy()
        raise e

IV. Frequently Asked Questions QA

Q: What should I do if I use a proxy IP and still get blocked?
A: First check if it is a high anonymity proxy (like ipipgo's are all high stash), then reduce the frequency of requests, preferably adding random delays (0.5-3 seconds) between requests.

Q: Proxy IP speed is too slow to affect efficiency?
A: It is recommended to choose the package billed by bandwidth, ipipgo's BGP line average latency of 80ms or less, more than 3 times faster than ordinary agents!

Q: How do I test if the agent is valid?
A: You can periodically visit http://ipipgo.com/checkip This detection interface will return the IP and anonymity currently in use

V. Maintenance strategy and cost control

Many newbies are prone to make the mistake of frantically grabbing data in the early stages, and as a result, the project runs and finds that the agency fee is overpriced. Here to teach you two tricks:

1. Intelligent switching strategyThe static page with ordinary proxy, encounter anti-climbing strict page and then switch to high-quality proxies. ipipgo support graded call by quality, can save 30% cost!

2. Local caching mechanism: Set local cache time for data that does not change often. For example, the price of goods can be cached for 6 hours to reduce the number of requests without affecting business.

3. anomaly monitoring: It is recommended to use Prometheus + grafana to do the monitoring of the large disk, when the success rate is lower than 95% automatic alarms, timely troubleshooting is a proxy problem or site revisions

Finally, to be honest, do crawl this line of tools to choose the right half of the success. Like our technical department is now unified with ipipgo proxy service, stability than before the self-built proxy pool is too strong, the key is their technical customer service is really 7 × 24 hours online, the last time three o'clock in the morning to mention the work order is actually a second back, this point is really convincing.