How to crawl an entire website: Site-wide crawler architecture

What the hell is site-wide crawling doing?

Many people think that the whole site crawler is a brainless pickpocket web page, in fact, here is a lot to talk about. The larger the site is more likely to trigger the anti-climbing mechanism, as if you go to the supermarket to try to eat, if you do not change clothes every day to go, the security guards do not stare at you to stare at who? This time we have to use theproxy IPThis dress-up artifact disguises itself as a different customer each time you visit.

How do you pick your core gear?

Engage in full-site capture is like playing a game of chicken, equipment selection error minutes into the box. You must get a reliable proxy IP service, here must be amenable!ipipgoHome service, their IP pool is big enough to swim in and comes with smart switching. See this comparison table for a specific equipment list:

What does a real-world architecture look like?

Let's use an e-commerce site as an example, where the architecture is layered like an onion:

 Proxy Middleware Configuration Example (Python Version)
import random
from ipipgo import get_proxy Here we use the SDK of ipipgo.

def get_random_proxy():
    proxies = get_proxy(pool_size=50) take 50 IPs at a time to spare
    return {'http': f'http://{random.choice(proxies)}'}

 This is how to use it when making a request
response = requests.get(url, proxies=get_random_proxy(), timeout=10)

Watch this.The pool_size parameterNot the bigger the better, it is recommended to adjust according to the strength of the site anti-climbing, just like eating a buffet to take a small number of times to get food, do not one-time end away from the entire dining table.

The Five Best Tips for Staying Alive

1. IP Rotation Strategy: Don't be stupid and use the IPs in order, ipipgo's random assignment mode can disrupt the usage trajectory
2. Request for fingerprint disguise: User-Agents have to be changed as often as Sichuan Opera Changing Faces
3. abnormal melting mechanism: Suspend the IP if it fails 3 times in a row and ipipgo will automatically replenish it with new IPs.
4. speed control：模仿人类浏览节奏，半夜可以适当代理ip
5. Data de-duplication: Memory savings with Bloom filters over traditional de-duplication

Common Rollover Scene QA

Q: What should I do if I always get my IP blocked?
A: Check three places: 1. whether to use the high stash proxy (ipipgo default is) 2. whether the request header with browser fingerprints 3. whether the frequency of access to the mutation

Q: How to grab image resources efficiently?
A: Use an independent download channel, ipipgo support line forwarding, diversion of image requests to different IP pools, do not and API requests crowded together!

Q: How do I break the CAPTCHA when I encounter it?
A: Don't be rigid! Immediately switch IP (ipipgo's second cut function) + change access portals, save more money than using coding platforms!

Tell the truth.

Engaging in site-wide crawling is like playing a game of cat and mouse, where the focus is not on how awesome the technology is, but on theIs it enough of a disguise to look like a normal person?. Having used seven or eight agency services, ipipgo is the most hassle-free of them all!Traffic obfuscation techniquesThe first thing you can do is to disguise the crawler traffic as normal user behavior, which is something that other companies really can't do. Remember don't be cheap with a free agent, that's equivalent to wearing prison clothes to the bank vault - asking for trouble.