R language crawler combat: rvest package details

First, hand to teach you to use rvest to play around with web crawling

Brothers engaged in network crawling know that the rvest package in the R language is as good as a Swiss army knife. To give a chestnut, want to catch the price of a certain East goods, three lines of code to get it done:

library(rvest)
jd_page % html_text()

But don't get too happy too soon! When you grab a couple dozen pages in a row, the site immediately gives you IP on theblacklistsThe first thing you need to do is to use a proxy IP address. This is where the importance of proxy IPs comes into play - it's like a cloak of invisibility when playing a game of chicken, so that the server doesn't recognize your real address.

Second, why is the proxy IP crawler life preserver?

The real-world partners know that playing with crawlers without an agent is like running naked:

Here's to the homegrown productsipipgoTheir API can switch IPs in seconds, which is especially suitable for scenarios that require a large number of requests. For example, when doing price monitoring, use their dynamic residential proxy to easily bypass the anti-climbing mechanism.

Third, rvest + proxy IP practical tips

Configuring proxies in R is actually super easy, the point is to use the right posture. Take ipipgo's proxy as an example:

library(httr)

proxy_config <- use_proxy(
  url = "gateway.ipipgo.com",
  url = "gateway.ipipgo.com", port = 9021,
  url = "gateway.ipipgo.com", port = 9021, username = "your_account",
  password = "your_token"
)

 Request with proxy
safe_read_html %
    content("parsed")
}

take note ofThree key points::
1. Always use the GET/POST method of the HTTR package
2. Authentication information should not be written directly into the code (environment variables are recommended)
3. Timeout settings should preferably be limited to 5-10 seconds

IV. Guide to avoiding the pit: Frequently asked questions QA

Q: What should I do if the proxy always times out the connection?
A: 80% of the IP pool quality problems. Recommended to use ipipgo'sExclusive use of high-speed linesThe measured latency can be squeezed to within 200ms.

Q: What if I need to change my country IP?
A: Just choose the locale code in the ipipgo background. For example, if you want Japanese IP, change the proxy address tojp.gateway.ipipgo.com

Q: Do free proxies work?
A: Blood lesson! 9 out of 10 free proxies are bad, leaving 1 that may steal data. You still have to use a paid service for important items. ipipgo new users have1 Dollar Trial Package, experience it for yourself is the most real.

Fifth, the masters are using advanced techniques

Share a few private tips:
1. Automatic IP switching: Use httr's retry function + ipipgo's API to realize the blocked automatic IP change.
2. Request for fingerprint disguise: work with the fake_useragent package to randomly generate UAs
3. flow control: Controlling requests per minute with the ratelimit package

 Example of automatic retry
library(ratelimit)

throttled_get <- throttle(GET, rate(n=50, period=60))

retry_request <- function(url){
  RETRY("GET", url.
        times = 3,
        terminate_on = 404,
        httr_config = proxy_config)
}

Finally, we remind all crawlers that using proxies should also comply with therobots.txtRules. After all, we just engage in data, don't make other people's websites collapse. Use good tools, compliant collection, is the way to the long term.