Data Collection Services: Enterprise Automated Collection Solutions

The biggest headache in data collection.

Do data collection brothers understand, the most afraid of encountering the site to give you a trip. In the morning, the script is still running well, and in the afternoon, it suddenly reports 403 errors, just like being stopped by the security guard in front of the shopping mall. At this time, if you use your own broadband hard just, light IP is blocked, heavy paralyzed the entire project - this kind of thing I've seen too much, there is a price comparison system for three consecutive days by an e-commerce platform blocked more than 200 IP, the boss almost gnawed on the keyboard.

That's when it's time to useProxy IP's dry run. Like a martial arts film in the disguise, each visit to change the face, so that the site's anti-climbing system can not recognize that you are the same person. However, the proxy services on the market are uneven, some claim to be a million IP pool, the actual use of all the duplicate addresses, than the supermarket promotion of the expiration date of the yogurt is not reliable.

The core three axes of an enterprise solution

A truly reliable automated capture solution has to meet these three hard criteria:

Take the case we did for a financial company, they need to collect data from 20 information websites in real time. With ipipgo's dynamic residential proxy, together with the intelligent switching strategy, the collection success rate was successfully pulled from 47% to 92%. here is a tip:Don't switch IPs at fixed intervalsThe response speed of the target website should be adjusted dynamically, like an old driver who will change gears according to the road conditions.

Teach you to build a collection system by hand

Here's a real Python example in use, using the Scrapy framework combined with the ipipgo API:

import random
from scrapy.downloadermiddlewares.retry import RetryMiddleware

class ProxyMiddleware(object): def process_request(self, request, spider): process_request(self, request, spider)
    def process_request(self, request, spider): proxy_server = random.choice_proxy(ip_list).
        proxy_server = random.choice(ipipgo.get_proxy_list())
        request.meta['proxy'] = f "http://{proxy_server['ip']}:{proxy_server['port']}"
        request.headers['X-Proxy-Secret'] = ipipgo.get_auth_token()

    def process_exception(self, request, exception, spider).
        return RetryMiddleware().process_exception(request, exception, spider)

Be careful to set theDifferentiated request headersDon't make all requests carry the same User-Agent, just as you can't go to a masquerade party and have everyone wear the same fox mask.

A practical guide to avoiding the pit

Recently encountered a typical case: a cross-border e-commerce customers collect product data, obviously used the proxy IP is still recognized. Later, it was found that there was a problem with cookie processing - although the IP was changed, the cookie still carried the previous information, just like changing clothes without changing the perfume smell.

The solution is simple: add these two lines to scrapy's settings.py

COOKIES_ENABLED = False
DOWNLOAD_DELAY = random.uniform(1,3)

Coupled with ipipgo'sSession-holding agents, the perfect solution to the identity leakage problem. It's like giving every crawler a temporary work license, use it or burn it.

QA First Aid Kit

Q: Why is it still blocked after using a proxy?
A: Check three places: 1. whether the request frequency is too fierce 2. whether the proxy is a transparent proxy (you must use a high stash of proxies) 3. whether the TLS fingerprints have done randomization

Q: What's unique about ipipgo?
A: Their homehybrid protocol poolIndeed, there are two brushes, can automatically identify the target site type, in the HTTP/Socks5 intelligent switching between. Last week to help customers docking travel platform, with the regular proxy can not pick up data, cut to their socks5 line immediately see the effect.

Q: Which package should business users buy most?
A: If it's a long-term project, go straight toCustomized Exclusive IP PoolI have a client who is doing public opinion monitoring and has bought 500 fixed IPs for scheduling. There is a customer who does public opinion monitoring and bought 500 fixed IPs for scheduling by himself, together with the intelligent routing function of ipipgo, and there has not been any large-scale blocking for half a year in a row.

At the end of the day, proxy IP is not a panacea, but just like a good wok for stir-frying, the key is toChoose the right tool for the jobThe first thing I'd like to say is that I've used seven or eight proxy service providers. Used seven or eight proxy service providers, ipipgo in the stability and technical support can really beat, especially their engineers can help tune the collection strategy, this point many big companies can not do.