Enterprise-level agency services: large-scale data collection project solutions

First, why is data collection always stuck? Let's see if your IP is being targeted.

Brothers who have engaged in data crawling understand that the most fearful thing is that the program is running and suddenly stuck. Last month an e-commerce friend and I complained, they climbed the price of competitors, just grabbed 2000 data on the target site pinched neck. I let him turn out the logs to see - good guy, the same IP address sent more than 800 consecutive requests, the site is not a fool, not seal you seal who?

It's time to move outproxy IP poolThis is a great tool. Simply put, it is to prepare a bunch of different IP addresses, like a shift like rotation. For example, with ipipgo's dynamic residential proxy, each request automatically switches between different regions of the real user IP, the site simply can not distinguish between a machine or a real person.

import requests
from itertools import cycle

 List of proxies from the ipipgo backend
proxies = [
    "http://user:pass@gateway.ipipgo.com:8001",
    "http://user:pass@gateway.ipipgo.com:8002".
     ... Prepare at least 20 more
]
proxy_pool = cycle(proxies)

for page in range(1,100): current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
        response = requests.get(url, proxies={"http": current_proxy})
         Processing data...
    except: print(f "IP {current_proxy}")
        print(f "IP {current_proxy} failed, automatically switch to next")

Second, the three major propositions of the selected agent service provider

There are a lot of proxy service providers in the market, but not many of them can carry enterprise-level projects. Last year, we did public opinion monitoring for a bank and tested 7 service providers, and in the end, only ipipgo was able to withstand 5 million requests per day. Here are a few key points for selection:

In particular.IP purityMany service providers blow their own IP more, in fact, are data center IP, this one catch a pass. ipipgo's residential agent are real home broadband, we have done the test: the same target site with the average agent to hold up to 300 requests, with his family can run to 2000 + times before triggering the verification.

Third, the actual battle in the tawdry operation

It is not enough to have an agent, you have to be able to play a combination of punches. Last year, during the double eleven to help a brand to do the whole network price comparison, relying on these moves 7 days to catch 12 million data:

1. Traffic camouflageInstead of using Python's default User-Agent, have 50 major browser logos to rotate through. ipipgo has a ready-made UA library in the backend that you can call directly.

2. Rhythm Master ModeDon't send out requests like chicken blood, set random intervals of 0.5-3 seconds. We've written a smart speed controller that automatically slows down when it encounters a CAPTCHA.

3. geographical relayFor example, if you want to catch a website in the United States, don't just use the IP of New York, mix the IP of Chicago and Los Angeles. ipipgo's city-level location function can directly specify the zip code.

IV. Pits you must have encountered (with solutions)

QA1：What should I do if I use a proxy IP and it becomes slow?
The IP is tagged by the target website, hurry to change a batch. ipipgo's proxy pool automatically updates 20% IP every 15 minutes, it is recommended to set the maximum number of times to use, do not exceed 100 times for a single IP.

QA2: How do I manage IPs with 100 threads open at the same time?
Use a connection pooling tool! For example, Scrapy's middleware, with ipipgo's API to get available IPs in real time. remember to bind each thread to a separate IP, don't get confused!

QA3: How to solve the problem when encountering CAPTCHA?
Three steps: 1) Switch IP immediately 2) Reduce the request frequency 3) Get on a coding platform (but you have to pay extra). We usually set 5%'s CAPTCHA trigger rate threshold, and send an alert if it exceeds it

V. Why die for ipipgo?

After using the proxy service for more than three years, the final selection of ipipgo is not without reason. Once at 3 am docking API, their technology actually returned the message in seconds, and later realized that it was a 24-hour shift system. And then say a hardcore: they have aIntelligent Routingfunction, can automatically select the fastest line. Once we catch Japanese website, the system automatically cut to the node in Tokyo, the speed is faster than direct access.

Recently releasedBusiness Assurance ModelMore perverted, you can reserve an exclusive IP pool in advance. Last month to a car group to do competitive analysis, 2 million stable requests per day, 15 consecutive days zero ban. This level of stability, the market really can not find the second.

(concluded)