BeautifulSoup vs Scrapy: Crawling Framework Selection

First, why do crawlers have to use proxy IP?

Brothers who do data crawling should have experienced it, just running for two minutes and getting403 ForbiddenThe tips. At this time, if you do not hang the agent, light is the day in vain, or directly by the site black. Take the e-commerce price comparison project I did last year, with real IP caught less than 100 times, the whole team computer was blocked for three days.

at this momentipipgoThe rotation of the proxy will come in handy - each request for a different exit IP, the site can not tell whether you are a real person or a machine. Especially the need for long-term operation of the task, no proxy is like running naked into the battlefield, sooner or later be shot dead.

Second, the survival of BeautifulSoup

This thing isn't technically a frame, it's more likeHTML parser. For example, you want to pick the content of a forum post, with requests + bs4 combination is the most appropriate. To cite a practical scenario: one day operation suddenly want to competing sites 500 commodity title, this time now learn Scrapy is too late.

import requests
from bs4 import BeautifulSoup
from ipipgo import get_proxy This plugs into the ipipgo SDK.

proxy = get_proxy()
headers = {'User-Agent': 'Mozilla/5.0'}
resp = requests.get('https://target-site.com',
                   proxies={"http": proxy}, headers=headers)
                   headers=headers)
soup = BeautifulSoup(resp.text, 'lxml')
titles = soup.select('.product-title')

Notice in the code above that theget_proxy()That's what I'm talking about.ipipgoDynamic IP acquisition interface provided. Every time you run the script automatically change IP, more than ten times more stable than using a free proxy.

Third, Scrapy's industrial-grade gameplay

When demand becomesGrab 100,000 pieces of data on a regular basis every dayIt's time to bring out Scrapy. Its middleware mechanism is much more friendly to proxy support, especially in combination with theipipgoof concurrent IP pools that enable truly distributed crawling.

Configure the proxy middleware in settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 543,
}

IPIPGO_API = "your_account:your_token@gateway.ipipgo.com:8000"

With this setup, each request is passed through theipipgoThe proxy channel. Measured in the gigabit bandwidth, 8 hours can crawl 800,000 pieces of data and zero banning.

Fourth, which frame should be chosen?

Delineate the focus:Scrapy must be paired with a high-quality agent pool., ordinary proxies simply can't carry high-frequency requests. This is why it is recommended thatipipgoof commercial-grade services, people have optimized QPS specifically for crawlers.

V. Practical guide to avoiding pitfalls

I have encountered the most pitiful situation: a free proxy, the result of the return data are alla commercial. It was later switched toipipgoof the Enterprise Edition, only to realize that the previously captured data 30% are contaminated.

Suggest adding an IP checking link to the code:

def check_proxy(ip).
    try.
        requests.get('http://ip.ipipgo.com/check',
                    proxies={'http':ip},
                    timeout=5)
        return True
    except.
        return False

This detection interface isipipgoUnique service that confirms whether the current agent is available in real time to avoid crawling to dirty data.

VI. Frequently Asked Questions QA

Q: Which framework should newbies learn first?
A: temporary needs with BeautifulSoup + Requests, long-term projects directly on Scrapy. regardless of which to choose, remember to match theipipgoof agency services.

Q: How often should I change my proxy IP?
A: Ordinary websites change once every 5 minutes, and websites with strict anti-climbing are recommended to change every request. InipipgoThe background can be set to change the frequency automatically.

Q: Why is it still blocked after using a proxy?
A: Check if you are using a transparent proxy.ipipgoThe high stash of proxies will hide the X-Forwarded-For header, and websites won't see the real IP at all.