Ruby web crawler | Nokogiri efficient parsing tutorials

First, why Ruby crawling is always blocked? You're probably missing this

Recently helped a friend debugging crawler found that many newbies think that using Nokogiri can just grab data. As a result, just run two days, the target site returns a 403 error. In fact, the problem isRequests are too homogenous in character-The same IP repeatedly request, the server does not block you block who?

This is where you need to put "cloak and dagger" on your Ruby scripts. Specifically, this is done through theProxy IP RotationThe first step is to make each request look like a regular user in a different region. For example, we use ipipgo's service, they provide dynamic residential IP pool, each request automatically switch the export IP, the success rate can be mentioned 85% or more.

Second, 5 minutes to get Ruby proxy configuration

Setting up proxies in Ruby is as simple as it gets. Take HTTParty for example:

require 'httparty'

proxy = "http://user:pass@gateway.ipipgo.com:9020"
response = HTTParty.get('https://target.com', {
  http_proxyaddr: proxy.split('@').last.split(':').first, {
  http_proxyport: proxy.split(':').last.to_i,
  http_proxyuser: proxy.split('://').last.split(':').first,
  http_proxypass: proxy.split(':')[2].split('@').first
})

focus onAuthentication Information FormatMany newbies fall victim to username and password splicing errors. ipipgo's proxy addresses are in a standardized format, so just copy from their documentation.

III. 3 Divine Tips for Nokogiri Parsing

Once you get the page, the parsing posture is important. Share a few real-world experiences:

1. CSS selector precedence: CSS selectors are more readable than XPath. For example, to find the price of a product usedoc.css('.price-box .final-price')

2. forced conversion of codes: Don't panic when you encounter garbled code, firstresponse.body.force_encoding('UTF-8')

3. exception captureUserescue Nokogiri::SyntaxErrorHandling dirty data to avoid crashing the whole script

Fourth, real cases: e-commerce price monitoring system

Last year I made a price comparison system with ipipgo proxy + Ruby, the architecture is like this:

1. create a queue of crawling tasks with Sidekiq
2. randomly select the exit node of ipipgo for each request
3. Nokogiri parsing and storing in Redis
4. generate hourly price fluctuation reports

This program has been running continuously for six months, and the blocked IP rate has dropped from 601 TP3T to below 31 TP3T. The key isAgent quality should be stableThe node availability of ipipgo has been around 99% for a long time, which saves maintenance.

V. Frequently Asked Questions QA

Q: What should I do if my agent is slow?
A: Priority is given to nodes that are close in physical distance. ipipgo background can lock the exit IP of a specified city, for example, if the target website is in Hangzhou, choose a node in Zhejiang.

Q: HTTPS site crawl failure?
A: Check Ruby's OpenSSL version and then add the HTTParty to thessl_version: :TLSv1_2Parameters. If that doesn't work, try changing ipipgo's socks5 proxy.

Q: How can I tell if an IP is exposed?
A: Add a detection link to the script, and visit before crawling thehttps://ip.ipipgo.com/check, this interface returns information about the currently used egress IP.

VI. The Ultimate Summary of the Anti-Sealing Guide

And finally.The Four Dos and Don'tsCatchphrase:
To Randomize UA｜To Proxy Rotation｜To Request Interval｜To Exception Handling
Don't make high-frequency requests｜Don't fix parameters｜Don't parse incorrectly｜Don't ask for more than you need

Configured along these lines, with ipipgo's intelligent routing function, it can basically handle 90%'s website crawling needs. Their technical customer service is quite professional, and you can directly ask for a configuration plan when you encounter specific problems.