Extracting Web Text: Proxy IP for More Efficient Content Capture

Teach you to use a proxy IP to pick up web page data

The old iron to engage in network crawlers know that the biggest headache is the target site blocking IP. hard work to write the crawler running a sudden break, check the logs to see all the 403 errors, this time if you do not have a proxy IP, it is really not looking for the tune of the cry.

To cite a real case: last year there is a small team of price comparison website, their crawler every day to catch hundreds of thousands of commodity data. As a result, one day was suddenly blocked by an e-commerce platform server IP, which directly led to the day of the data cutoff. Later, they usedipipgoThe dynamic residential proxy, which spreads the requests to different regional IPs, is what stabilizes the data source.

import requests

proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('destination URL', proxies=proxies, timeout=10)
print(response.text)

What are the doors to look for when choosing a proxy IP?

There are all sorts of agent types on the market, so let's explain the difference in layman's terms:

Based on empirical experience.ipipgoThe mixed proxy pools work best. They can intelligently schedule the three types of proxies, such as using data center IPs for common pages, cutting important data to residential proxies, and then going to mobile IPs when encountering difficult websites, which not only saves costs but also ensures the success rate.

Avoiding the tawdry maneuver of backcrawling

It's not enough to have an agent, you have to know these combos:

1. randomized sleep: don't request like a robot, stop randomly between 2-5 seconds

2. Replacement of UA: Have 10 different browser versions of the request header to rotate through

3. request frequency control: Don't exceed 500 requests per hour from a single IP (with theipipgo(If you do, you can relax to 800 times)

Focus on the pitfall of cookie handling. Some sites will track via cookies, which need to be emptied periodically. when using the session object of requests, remember to reset it every 50 requests:

session = requests.Session()

    if i % 50 == 0: session = requests.
        session = requests.Session() rebuild session
     Normal request code...

Practical QA session

Q: What should I do if my proxy IP often times out?

A: It is recommended to enable ipipgo's intelligent routing function, their API can automatically eliminate slow nodes. In addition, add retry mechanism in the code, set 3 retries + 2 seconds interval basically can be solved.

Q: How can I tell if a proxy is in effect?

A: A visit to http://ip.ipipgo.com/checkip这个专属检测接口 can return the currently used exit IP and geographic location.

Q: What should I pay attention to when collecting offshore websites?

A: Be sure to choose the corresponding region of the proxy node. For example, if you use ipipgo's Tokyo server room IP to catch Japanese websites, the speed can be increased by more than 3 times.

Save the Streams Summary

There are just three things at the heart of using a good proxy IP:Multiple IP rotation, simulation of real-life operation, selection of reliable service providersIt's a good idea to have a good deal of time to work on your own. Beginners suggest directly on the ipipgo package, their IP pool updated daily 20% or more, comes with the failure of automatic switching function, than their own maintenance agent pool to save too much effort. Recently see the official website there are new users free trial activities, register to send 1G flow, enough small-scale collection needs.