Example of using cURL proxy: Command Line Crawling Tutorial

Teach you to use cURL to hang proxy to catch data

Crawlers should understand the old iron, direct bare access to the site in minutes on the blocked IP. this time we have to sacrifice the proxy IP this magic weapon, today we take the Swiss Army Knife in the command line cURL to demonstrate how to use the proxy IP to safely grip the data.

Let's start with something dry.The proxy server address for ipipgo looks like this: http://username:password@gateway.ipipgo.com:8020 Just replace username and password with your account's authentication information. Memorize this address, you'll have to take the test later.

Agent Configuration Triple Axe

When typing commands in the terminal, add a-xparameter will be able to hang the proxy. As an example:

curl -x http://user123:pass456@gateway.ipipgo.com:8020 https://target-site.com/data.json

Here's something to keep in mind.Three key pointsThe proxy type should be correct (http/https/socks5), the port should not be written incorrectly, and the authentication information should not be omitted. If the 407 error code is returned, it is likely that the account password is wrong.

Real-world capture mini-theater

Suppose you want to capture the price of an e-commerce site, first use ipipgo'stachymeter interfacePick a fast node:

curl -x http://user:pass@gateway.ipipgo.com:8020 https://speedtest.ipipgo.com/latency

Seeing that the returned response time is within 200ms indicates that this node is fast enough. Then on to the real thing:

curl -x http://user:pass@gateway.ipipgo.com:8020
-H "User-Agent: Mozilla/5.0"
-L https://mall.com/product/12345
-o price_data.html

put the emphasis onRemember to add the -L parameter to automatically follow the redirection, and -o to save the result to a file. If you encounter a certificate error, you can add the -insecure parameter to skip the verification, but the formal environment should be used with caution.

First aid kit for difficult cases

Q: What can I do if the agent can't connect?
A: First ping gateway.ipipgo.com to see if it goes through, then check the account expiration date. Recommended to use ipipgoSurvival Detection Interface:: https://status.ipipgo.com/checkip

Q: How do I switch between different regional IPs?
A: In the ipipgo control panel, select the access point for the specified region, for example, if you want the Shanghai node, use sh.gateway.ipipgo.com:8030, remember that the ports may be different in different regions.

Q: What should I do if I am frequently blocked?
A: In the cURL command add -proxy-negotiate parameter to enable automatic switching, with ipipgo's dynamic rotation service, every minute automatically change the IP, measured live rate can be more than 98%.

Private Tips for Connoisseurs

1. Use-connect-timeoutSet a 5-second timeout. Don't wait for the node to fail.
2. Write the proxy parameter in the ~/.curlrc configuration file, so that you don't have to type the command every time.
3. Remember to change the protocol header to https:// when you need https proxy.
4. Enterprise support for ipipgoAPI Dynamic Acquisition ProxyThe IP pools are suitable for scenarios that require a large number of IP pools.

Lastly, anon, ipipgo has recently been on thepay-per-use packageI'm not sure if you're a new user, but I'd like to give you a 5G traffic trial. The quality of their residential agent is really top, last time I opened 200 threads at the same time were not blocked, do data collection old brother can try.