Capturing Data with R: A Practical Guide to rvest Packets

The best way to prevent IP blocking when using rvest to capture data

Recently, a lot of data analysis partners with me to complain, with rvest packet data always encounter IP blocked broken thing. Like last week, a buddy climbed the price of e-commerce, just grabbed 200 pages on the blocked IP, so angry that he almost smashed the keyboard. Today we will teach you how to use proxy IP this magic weapon to get this bad problem.

Don't step in the pits with rvest basics

First, to fill in the blanks, rvest packages do work well like a Swiss army knife. But a lot of people tend to fall into these pits:

 Examples of common errors for newcomers
library(rvest)
url <- "https://example.com"
html <- read_html(url) direct bare access, wait to be blocked!

Here's the point!read_html() This function is in naked mode by defaultIf you are not blocked, continuous access is like square dancing in front of the webmaster's eyes, so who will be blocked if not you?

Proxy IP's Life-Saving Trio

This is the time to offer up our life preserving trio:

 The right way to open
library(httr)
proxy <- "http://username:password@gateway.ipipgo.com:9020"
response <- GET(url, use_proxy(proxy))
html <- read_html(response)

There's a little doorway here that many tutorials don't make clear:username should be replaced with the authentication string generated by the ipipgo backendThe password field can actually be left blank. One advantage of using their proxy is that it can automatically switch the export IP, which is much less troublesome than manually changing the IP.

Practical case: e-commerce price monitoring

Taking a real-life example, let's say we want to catch the price of a cell phone on a certain store:

library(httr)
library(rvest)

ipipgo_proxy <- "http://user-3k9d8s@gateway.ipipgo.com:9020"

for(i in 1:100){
  url <- paste0("https://item.taobao.com?page=",i)
  resp <- GET(url, use_proxy(ipipgo_proxy), timeout(30))
  if(status_code(resp)==200){
     Code to parse the price data...
  } else {
    print("Crawl triggered, switching IPs automatically.")
     The ipipgo API interface will automatically rotate the IPs
  }
  Sys.sleep(runif(1,1,3)) Random hibernation is more realistic
}

Watch this.runif(1,1,3)The trick to make the access intervals irregular. Combined with ipipgo's automatic IP pool switching, it's basically as stable as an old dog.

Frequently Asked Questions QA

Q: What should I do if the proxy IP is invalid after I use it?
A: This situation is recommended to use ipipgo's dynamic residential proxy, their IP pool is updated every day 200,000 + IP, much more stable than a static proxy!

Q: Setting up a proxy or being blocked?
A: Check three points: 1. whether the proxy is highly anonymous 2. whether the request header has browser fingerprints 3. whether the access frequency is too high. If you use ipipgo, remember to turn on the "auto request header disguise" function.

Q: How to solve the problem of slow agent speed?
A: When choosing nodes try to pickGeographically close server room，ipipgo的国内BGP线路能控制在50ms以内

Why recommend ipipgo

Used seven or eight agent service providers, the last long-term use of ipipgo on three reasons: 1. price is really (students can afford to play) 2. optimized line specifically for crawlers 3. customer service response fast, last two o'clock in the morning to mention the work order actually someone back!

Finally, a piece of advice: don't be greedy and use a free agent, or data leakage, or reverse crawling. Professional things to professional people to do, save time to write a few more data analysis model does not smell?