Selenium vs Scrapy: Crawling Framework Selection Guide

Hands-on teaching you to choose a crawler tool: Selenium and Scrapy in the end which is better?

Crawler old iron people most often ask the question is: with Selenium or Scrapy, these two goods look at the data can be grabbed, but the difference between the use of it can be a big go. Today we will break open the crumbs said, especially how to use with the proxy IP to not overturn the car.

I. Applicable scenarios are very different

Let's start with the conclusion:Selenium for real people, Scrapy for speed and quantity.The first thing you need to do is to use Selenium to simulate the operation of a real person. For example, if you want to catch the evaluation of a product, you have to log in and then turn the page, then use Selenium can perfectly simulate the operation of real people. But if you want to grab enterprise yellow pages in bulk, Scrapy can get dozens of pages a second.

Here's a pitfall to be aware of: it's especially easy to get IP blocked when using Selenium because the browser characteristics are so obvious. It's time to rely on theDynamic Residential Proxy for ipipgoIf you want to change your IP address automatically every time you visit, you can reduce the probability of 90%'s blocking.

Proxy IP use posture

Adding proxies in Scrapy is super easy, two lines in settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 543,
}
HTTP_PROXY = "http://用户名:密码@gateway.ipipgo.com:9020"

And Selenium has to be messed with like this (using Chrome as an example):

from selenium import webdriver
proxy = "gateway.ipipgo.com:9020"
options.add_argument(f'--proxy-server=http://{proxy}')

III. Guide to avoiding pitfalls in actual combat

I recently flipped out while helping a client crawl a certain business information site. Using Scrapy to request directly, the result was all CAPTCHA pages returned. Later, I switched to Selenium+ipipgo'sBrowser Fingerprinting ProxyThe problem is perfectly solved. Here's a tip: remember to set a random wait time, don't let the site find out it's a robot operating.

If you run into slider validation, don't head iron hard. Try ipipgo'sFixed Session ProxyIf you want to keep the same IP to complete the whole set of operations, the success rate can be improved a lot.

IV. Answers to frequently asked questions

Q: What should I do if I always get my IP blocked?
A: Three tricks: 1) Reduce the frequency of requests 2) Use ipipgo's rotating proxy 3) Randomly change the User-Agent

Q: How do I get a website that requires a login?
A: First use Selenium to simulate login to get cookies, then use Scrapy to batch operation. Remember to pair it with ipipgo'sLong-lasting proxy IP, keeping the login status uninterrupted.

V. Recommendations for final selection

Give a universal formula:
Data volume <1000/day ➜ Selenium+ipipgo Residential Agent
Data volume >1000/day ➜ Scrapy+ipipgo Data Center Proxy

Lastly, I would like to remind you: don't try to use a free proxy, last time a customer was blocked IP segment, the site directly black the entire C segment. With ipipgo's exclusive proxy although more expensive, but the success rate is guaranteed, the calculation is actually more cost-effective.