Ruby Web Crawling: Ruby Crawling Tutorials

Why do Ruby crawlers always get blocked? Try this

Recently, a lot of small partners in the use of Ruby to write a crawler have encountered a headache - the target site does not move on the blocking of the IP. this is something I also planted last year, when three days in a row to write the crawler script can not run, so angry that I almost smashed the keyboard. Later, I realized thatThe problem is that the IP is recognized as machine traffic.The web server is not a foodie. Just like you go to the supermarket and always buy the same kind of instant noodles, the cashier can remember your face, the web server is not vegetarian.

Ruby Scripts in Cloak & Dagger

The key to not getting caught with a crawler is to learn tolit. rotate vestsThe first thing you need to do is to get a new IP address, and then you have to change it. The vests mentioned here are proxy IPs, like changing into different clothes and wearing a wig every time you go to the supermarket. Take ipipgo's service as a chestnut, their home has a huge amount of residential IP pool, with a special hand:

require 'net/http'

proxy_addr = 'gateway.ipipgo.com'
proxy_port = 9021

uri = URI('http://目标网站.com')
Net::HTTP.start(uri.host, uri.port, proxy_addr, proxy_port) do |http|
  response = http.get(uri.path)
  puts response.body
end

Notice in the code thatproxy_addrrespond in singingproxy_portThese two parameters, this is our stealth props. ipipgo's proxy server supports a variety of authentication methods, we recommend using theirUsername+Password Binding IPThe model is much less of a problem than those that require dynamic CAPTCHA.

A practical guide to avoiding the pit

It's not enough to be able to use proxies, here are a few potholes I've stepped into:

To focus on the User-Agent thing, don't try to save time and use Ruby's default one. It is recommended to make an array of dozens of common browser identifiers, and randomly select one for each request.

Crawler Maintenance Tips

Keeping reptiles is like keeping a pet, you have to feed and maintain them regularly:

1. Check the IP availability rate every day, ipipgo background can see the success rate of each IP
2. Set up smart switching policy to automatically change IP when encountering 3 consecutive failures
3. Running large data volume tasks from 2-5am, when the site's defense mechanisms are more lax

Once I was lazy half a month without maintenance, the results of a day suddenly found that the efficiency of the crawler down 70%. later found that the target site updated anti-climbing strategy, timely adjustment of the request interval to save back.

Frequently Asked Questions QA

Q: Do free proxies work?
A: Never! I tried using free proxies last year, and 8 out of 10 didn't work, and I encountered phishing proxies. Then I switched to ipipgo's paid service, and the stability went up three steps.

Q: Do I need to change the proxy IP often?
A: Depends on the business scenario. If it is a high-frequency collection, it is recommended to change IP for each request. ipipgo's dynamic pool has millions of IPs, which is completely enough.

Q: What should I do if I encounter a website asking for a verification code?
A: In this case, simply changing the IP may not be enough, you have to cooperate with the request frequency control. Setting the request interval randomly at 3-8 seconds can effectively reduce the probability of triggering CAPTCHA.

As a final rant, it's important to do the crawlerSustainable developmentThe first thing you need to do is to use ipipgo's proxy service + intelligent scheduling strategy. Last month to help a friend's company tuning crawler system, with ipipgo's proxy service + intelligent scheduling strategy, continuous and stable operation of 28 days without being blocked, the collection of efficiency is also improved by 40%. this thing with the guerrilla warfare, like, flexible and changeable is the way.