Ruby Web Crawler | API Docking Data Collection Solution

What is the use of proxy IPs for data crawling?

Do data crawl old iron must have encountered this situation: just grab two pages on the target site blocked IP, or API interface suddenly returned 403 error. At this time you need toproxy IPTo be your "stand-in actor", especially like ipipgo this can automatically change the IP service, is simply the programmer's "cloak of invisibility".

To give a chestnut, an e-commerce platform can only check the price 500 times per hour per IP, with ipipgo's dynamic IP pool, it is equivalent to wearing the program with aA transforming battle armorThe IP address is automatically changed every minute, and the crawling efficiency is directly doubled. Measured with the ordinary way to capture the success rate of less than 30%, plus ipipgo agent can be stabilized at 95% or more.

Ruby Crawl Triple Axe Configuration Guide

Ruby does web crawling mainly with these three guys:
Net::HTTP(self-contained library),Faraday(Third party artifacts),Mechanize(Automation Expert). Regardless of which one you use, the routine for configuring the agent is pretty much the same:

"`Ruby
Take Faraday for example
conn = Faraday.new(
url: 'https://目标网站.com',
proxy: {
uri: "http://用户名:密码@proxy.ipipgo.com:端口号",
headers: {'User-Agent' => 'Mozilla/5.0'}
}
)
response = conn.get('/api/data')
“`

Note that you have to go to the ipipgo website firstRegister for an accountGetting the authentication information, their proxy address format is generallyArea code.proxy.ipipgo.comFor example, the Shanghai node issh.proxy.ipipgo.comThe

API docking anti-blocking practical skills

Many sites now have open APIs, but as usual, they limit traffic. Compare the two ways of docking:

The right posture for docking APIs with ipipgo:

"`Ruby
require 'json'

ipipgo_proxy = "http://your_account:password@rotate.proxy.ipipgo.com:9020"
response = Faraday.get("https://api.xxx.com/data",
{param: 'value'},
proxy: {uri: ipipgo_proxy},
headers: {Authorization: 'Bearer xxx'}
)

data = JSON.parse(response.body)
“`

Highlight it three times:Be sure to add a timeout setting!Be sure to add a timeout setting!Be sure to add a timeout setting! It is recommended to set it to 10 seconds to prevent individual agent nodes from jamming the whole process.

Real Case: E-commerce Price Monitoring System

Last year to help friends do the price comparison system, with the ordinary method to catch the data of a certain East, the average lasted no more than 2 hours to be blocked. After changing to ipipgo, I configured three strategies:

1. Randomization of User-Agent per request
2. Automatic switching of IP segments every 5 minutes
3. Automatically switch to alternate IP when encountering CAPTCHA

The results were immediate, and it ran for two weeks straight without flipping. The key code looks like this:

"`Ruby
def fetch_product_price(url)
3. times do |i|
begin
proxy = ipipgo.get_proxy Call ipipgo's API to get a new IP
return Faraday.get(url, proxy: proxy).body
rescue => e
Rails.logger.error "The {i+1}th attempt failed: {e.message}"
end
end
raise "Failed to capture"
end
“`

A must-see QA session for the little guy

Q: What should I do if my proxy IP is slow?
A: Preferred ipipgo'sExclusive use of high-speed linesIf you want to use a public proxy pool, don't use a public proxy pool. Check if the target site itself is slow loading, you can use curl to test the direct access speed first.

Q: How do I test if the proxy is valid?
A: Test with this command:curl -x http://代理IP:端口 https://httpbin.org/ipto see if the returned IP is a proxy IP.

Q: Will the blocked IP be used again?
A: ipipgo's intelligent scheduling system will automatically isolate tagged IPs and will not reuse them within 24 hours, which is more reliable than many small factories.

Pitfall Avoidance Guide and Core Tips

Three common mistakes newbies make:

1. Write the proxy configuration in the code (it should be put in an environment variable)
2. Failure to set up a reasonable retry mechanism
3. Ignoring HTTP header masquerading

Recommended for ipipgoIntelligent Routingfeature that automatically selects the fastest node. Their control panel allows you to see IP usage in real time, something that is especially useful for debugging.

A final reminder: while proxy IPs can solve a lot of problems, they also have to beComply with the website's robots.txt rulesDon't hang their servers. After all, we are only reasonable data collection, not a cyber attack.

Brothers who need to test can go to the official website of ipipgo to get theFree Trial Pack, 1G of free traffic for new users, enough for a small test. Remember to use the promo codeRUBY2024You can also get 20% more traffic, tested and valid~.