Curl Crawling Techniques: Example of Command Line Web Page Capture

Curl crawling meet IP blocked? Teach you to use a proxy to break through

engage in data capture of the old iron should understand, with curl script just run two days, the target site on our IP to seal. This time do not be anxious to drop the keyboard, today to teach you a trick ---The Great Proxy IP Dynamic RotationThe first thing we're going to do is to use ipipgo's service as an example. Let's take the services of ipipgo's family as an example, and we guarantee that you will be able to do it practically after reading it.

Why proxy IPs are a lifesaver for curl crawling

Website anti-climbing is like subway security, the same face (IP) frequent appearances are sure to be stared at. ipipgo provides a dynamic proxy pool likeThe Mask of a Thousand FacesThe fact that each request has a different face makes it impossible for the anti-climbing system to understand the rules. The actual test with their residential agent, 30 days of continuous capture did not trigger the ban.

Zero-Basic Curl Proxy Configuration Guide

Adding proxies to the command line is as simple as it gets, remember this catch-all format:

curl -x http://用户名:密码@proxy address:port Destination URL

As a chestnut, use the socks5 proxy provided by ipipgo (their proprietary protocol is more stable):

curl -x socks5://vip123:abcd1234@gateway.ipipgo.net:30001 https://target.com


Practice: dynamic IP rotation collection of e-commerce prices
Single proxy is not enough? Go to ipipgo's API to change IPs automatically (their interface is very responsive).Within 200ms):
! /bin/bash
for i in {1..100}
do
   proxy=$(curl -s api.ipipgo.net/getproxy?key=your key)
   curl -x $proxy https://shop.com/item_$i >> prices.txt
   sleep $[RANDOM%5+1] Randomly wait for anti-regulation
done

Here's the key point.Get a new agent for each loop, in conjunction with random hibernation, perfectly avoids anti-climbing monitoring.
3 Must-Have Tips for Avoiding Pitfalls


pothole
prescription


Proxy connection timeout
Add the --connect-timeout 10 parameter to curl


Web content garbled
Add -H "Accept-Encoding: gzip" request header


Certificate Validation Failure
-k parameter to skip SSL authentication (use with caution for sensitive data)


Frequently Asked Questions Demining Area
Q: Proxy lagging with use?

A: 80% of the IP quality is not good. It is recommended to change ip ipgoExclusive use of high-speed linesThey carry 5Gbps of bandwidth per IP, and the download speeds are personally tested to run the full local broadband.
Q: How can I tell if a proxy is in effect?

A: First use curl to access ipipgo's detection interface:
curl -x proxy address api.ipipgo.net/checkip
Seeing a change in the returned IP indicates that the configuration was successful.
Q: What if I need to process a CAPTCHA?

A: ipipgo'sLong-lasting static proxiesIt is more appropriate to cooperate with the coding platform, and a single IP survives for 24 hours, which is enough to complete complex operations.
One last tip: writing the proxy configuration to an environment variable can save you a lot of work, add it in .bashrc:
export ALL_PROXY="http://用户名:密码@gateway.ipipgo.net:30000"
This way all curl requests will automatically go to the proxy, the degree of worry directly pull full. Encounter technical difficulties do not own hard just, ipipgo technical customer service 24 hours online, report my name can also send more 10G flow (laughs).