Website data collection: website data collection proxy IP configuration program

First, the website data collection for why have to use proxy IP?

Engaged in data collection know that the target site is very sensitive to the frequency of visits. For example, a treasure product details page, continuous use of the same IP brush half an hour, Iron will trigger the anti-climbing mechanism. At this time, the proxy IP is likecloak of invisibility, allowing the acquisition program to switch back and forth between different identities.

To cite a real case: there is a price comparison system team, with their own server to directly collect an e-commerce platform, the results of the next day the entire server room IP are blocked. Later, they changed to use ipipgo's dynamic residential proxy to disperse the request to different areas of the IP pool, and the collection success rate was directly pulled to 95% or more.

Proxy IP configuration manual

Here's a demo of the proxy configuration for the Python requests library for the guys, pay attention to the details in the code:

import requests

 Proxy address extracted from ipipgo (example)
proxy = "http://user:password@gateway.ipipgo.com:9020"

try.
    response = requests.get(
        'https://目标网站.com/api',
        proxies={'http': proxy, 'https': proxy},
        timeout=10
    )
    print(response.text)
except Exception as e.
    print("Request failed, try again with another IP:", str(e))

Highlight a few pitfalls:

1. Don't exceed 15 seconds timeout, otherwise it will affect the collection efficiency
2. Remember to handle SSL certificate validation (verify parameter)
3. Dynamic residential IPs are recommended to be changed on every request

Third, the Scrapy framework proxy middleware configuration

For those of you who are old enough to use Scrapy, look here and add this to middlewares.py:

class IpProxyMiddleware.
    def process_request(self, request, spider).
         Get the latest proxy from the ipipgo API
        current_proxy = get_ipipgo_proxy()
        request.meta['proxy'] = current_proxy
         Remember to add the random UA
        request.headers['User-Agent'] = random.choice(USER_AGENTS)

Here's a little trick: in settings.py put theCONCURRENT_REQUESTSTune it to 20-50, with a proxy IP pool to maximize collection speed.

IV. First aid guide to common rollover scenes

V. QA session

Q: How do I choose between a static IP and a dynamic IP?
A: need to maintain the login status of the selection of static (such as the collection of the need to log in the page), the ordinary collection of dynamic more cost-effective. ipipgo static residential 35 yuan / a / month, enterprise-level business is recommended to choose this.

Q: How do I break the CAPTCHA when I encounter it?
A: Don't hard just, two programs: 1. reduce the collection frequency 2. with the coding platform. At the same time, it is recommended to use ipipgo's TK line, which has a higher probability of IP being labeled as a normal user.

VI. ipipgo package selection guide

Based on our real-world experience:

• Startup team: choose Dynamic Residential Standard Edition ($7.67/GB), suitable for small and medium-sized collection
• Enterprise users: directly on the enterprise version of Dynamic Residential ($9.47/GB) with exclusive API channel
• Special needs: such as the need for fixed IP login, with 35 yuan / month of static residential

Lastly, I would like to say: don't try to use a free proxy, I've seen some people collect half of the data and mix it with spinach advertisements, and only after half a day's investigation did I realize that the proxy had been contaminated. Professional things or to ipipgo this kind of regular service providers reliable, after all, they have more than 200 countries operator resources at the bottom.