Scrapy vs BeautifulSoup: Crawler Framework Selection

Tool positioning is not the way to go

Let's first break these two tools to understand. Scrapy this thing is a serious crawler framework, from network requests to data storage all inclusive, like building a house to give you a good scaffolding. BeautifulSoup is an HTML parser, like eating crabs with tweezers, specifically from the web page to pick up data with. For example, you want to use Scrapy to engage in web crawling, they come with automatic retry, concurrency control mechanism, but if you use BeautifulSoup, you have to do their own with a requests library to do the network request, and manually deal with all kinds of exceptions.

On the matter of proxy IPs, Scrapy has thebuilt-in middleware mechanismIf you use BeautifulSoup, you have to stuff the proxy parameters separately every time you requests.get(), and if you need to change the IP frequently, the code can give you a chance. If you use BeautifulSoup, you have to stuff the proxy parameter in each requests.get(), and if you need to change IP frequently, the code can wind you into a ball of wool.

There's a generational difference in the way we handle data.

BeautifulSoup's specialty is parsing messy HTML documents, and it can smooth out any tag nesting errors or poorly formatted web pages. XPath and CSS selectors are also easy to use, and are particularly suitable forprecise positioningPage Elements. But Scrapy's own Selector isn't really that bad, and it works seamlessly with the crawling logic.

Focusing on proxy IP fit issues. When using ipipgo's rotating proxies, Scrapy is able toDownload MiddlewareThe level of automatic IP switching, for example, set every crawl 5 pages to change the proxy. With BeautifulSoup+requests, you have to write your own IP pool manager, and there may be cases where the proxy is not replaced in time, leading to IP blocking.

Anti-Crawl Response Capability

现在网站的反爬机制越来越狠，这时候就看出框架的优势了。Scrapy自带User-Agent轮换、请求设置，配合ipipgo的高匿代理，能把你的爬虫伪装成Real user accessThe most important thing to remember is that you can use a free proxy to climb an e-commerce platform. Recently to help people do the project will have a deep understanding of the use of free proxies to climb an e-commerce platform, 10 minutes to be blocked, replaced with ipipgo's exclusive IP, ran for three days in a row are all right.

If you use BeautifulSoup program, these anti-climbing measures have to build their own wheels. For example, dealing with CAPTCHA, maintaining cookie pools, no experience in the newbie easy to blind. Here to insert a sentence of truth, really want to do business-class crawler, or have to use Scrapy + professional proxy services to save heart.

Applicable scenarios to the right place

It's more visual to give the guys the whole comparison table:

To give a real case: before climbing a recruitment website, with BeautifulSoup script at first okay, and so climbed to the 200th page of the IP will be blocked. Later changed to Scrapy framework, in settings.py configure ipipgo API interface, set CONCURRENT_REQUESTS = 32, the agent automatically rotate, the speed of direct takeoff has not been blocked.

Frequently Asked Questions QA

Q: Which tool should I choose when I'm just starting out?
A: If you just grab a few pages and play around with them, BeautifulSoup + ipipgo's pay-per-use agent is enough. But if you want to do the project seriously, it is recommended to learn Scrapy directly, which is more efficient in the long run.

Q: What's wrong with using a proxy IP and still getting blocked?
A: The quality of the proxy may not be good, and many free proxies are blacklisted. We recommend using ipipgo's pure residential proxy, we have tested the blocking rate can be controlled at 5% below.

Q: How does Scrapy configure ipipgo's proxy?
A: Something about adding three lines of code to middlewares.py:
"`python
request.meta['proxy'] = "http://用户名:密码@gateway.ipipgo.com:端口"
“`
Remember to open the download middleware in the settings, and go to the official website to see the documentation.

Lastly, I would like to say that the tool itself is not high or low, the key is to match the scene. But if it comes to commercial use, don't save the proxy money, use ipipgo's quality IP can reduce a lot of detours. Two days ago, there are customers trying to cheaply use free proxy, the result of the data did not catch much, but in turn the server IP to take into account, a big loss, is not it?