Distributed Crawling System: Celery Practical Examples

Celery meets proxy IP, data capture problem is solved!

Do data crawl friends understand, stand-alone crawler is like drinking pearl milk tea with a straw - suck to the back is always a bunch of pearls left to suck up. This time we have to move out of the distributed crawling system, and Celery this task queue tool is definitely a good helper. But today we focus on how to give it with a proxy IP this "plug-in", especially with ipipgo service to break through the capture bottleneck.

Why do you have to use a proxy IP?

Take a real case: last year, there is a team doing e-commerce price comparison, their Celery cluster to catch 3 million pieces of commodity data every day. As a result, one day suddenly found thatThe target site blocked all their IP segments.The whole business simply shut down. This is a typical lesson of "putting eggs in one basket".

This is where ipipgo's dynamic residential IP pools come in handy. Their services are supported:

Hands-on Configuration of Celery + Proxy IPs

Here's a practical tip:Don't write proxy configuration in code! The right thing to do is to manage it with environment variables:

 In the Celery configuration
BROKER_URL = 'redis://localhost:6379/0'
IPIPGO_PROXY = os.environ.get('IPIPGO_PROXY')

Then pass the parameter this way when starting the worker:

IPIPGO_PROXY="http://user:pass@gateway.ipipgo.com:9021" celery -A proj worker

The advantage of this is that you don't have to change the code when switching proxies, which is especially good for people who need toMulti-geographic IP rotationscenarios. ipipgo's API can directly generate exit IPs for different cities, which is especially useful for projects that need to simulate the distribution of real users.

A Guide to Avoiding the Pit (Blood and Tears)

1. Don't be cheap and use free proxiesAs tested before, the average response time of free proxies is more than 8 seconds, while ipipgo's premium lines can be pressed within 1.2 seconds!

2. Set up a reasonable retry mechanism: it is recommended to use an exponential backoff algorithm, like this:

@task(
    autoretry_for=(TimeoutError, ),
    retry_backoff=30,
    max_retries=3
)

3. IP quality testing can't be understatedThe ipipgo admin backend actually comes with this feature, but it's safer to write your own double insurance.

Practical Q&A QA

Q: How does a Celery cluster manage a large number of proxy IPs?
A: recommended IP pool queue with redis, with lua script to achieve atomic operations. ipipgo API can directly return multiple IP, with RPUSH command stuffed into the queue on the line!

Q: What should I do if I encounter a CAPTCHA?
A: It would have to be in conjunction with ipipgo'sLong-lasting static IPup. Fix tasks that require CAPTCHA recognition to specific IPs for easy processing by subsequent coding platforms

Q: How do I test the actual results of the agent?
A: build their own detection service, regular visits to http://httpbin.org/ip. ipipgo users can directly use the detection interface they provide, the return information can be seen in the IP remaining validity period

Why ipipgo?

I finally locked him down after using seven or eight agency services for three main reasons:

1. There are specializedData crawl optimization linesUnlike some service providers who mix crawler traffic with regular users
2. Customer service response is fast, the last time I encountered IP can not connect, 10 minutes to change the new channel!
3. Transparent fees without hidden pitfalls, per-use billing model is particularly friendly to small teams

They recently came out with a newPay per successThe model of the failed crawl is not billed, which is a boon for projects that need to control costs. Need to experience can go directly to the official website to get a 3-day trial, remember to choose "distributed crawler special" that package.

One last piece of cold knowledge: the more Celery workers you have, the better. As a rule of thumb.2-3 workers per CPU coreThe most cost-effective program is to combine the IP pool size of ipipgo. For example, 8-core machine with 20 workers, while maintaining 50 available IP, this ratio has been verified by a number of projects, crawling efficiency can be improved by more than 4 times.