curl download file: command line file fetch

When downloads slow to a turtle's pace? Try this hidden trick

Programming should have encountered this situation: use curl to download foreign resources when the progress bar half a day does not move, anxious to smash the keyboard. This time do not rush to change the download tool, you may just lack the proxy IP this secret weapon. Just like driving in a traffic jam to take a detour, the network request also need to find aTraffic-free routesThe

Recently, I helped a colleague to deal with a particularly typical case: their team to use curl batch download AWS mirror package, the results of 10 files under the 3 hours. After switching to ipipgo's proxy IP, the same operation was performed.Seven minutes.The secret is to add a -proxy parameter to the curl command. The secret is to add a -proxy parameter to the curl command, stuff the proxy address into it, and the speed will take off.

Hands-on teaching you to put "cloak of invisibility" on curl

Let's look at a most basic usage first:

curl -x http://username:password@proxy.ipipgo.com:8080 https://目标文件地址

Here's a couple.Easy to step in the pitThe place:
1. Remember to use % to escape any special symbols in the password.
2. Proxy type should be the same as the one given by the service provider (http/https/socks5)
3. Don't get the port number wrong.

What to do when you encounter problems in the real world? Look here

QA 1: What should I do if my proxy IP suddenly fails?
In this case, it is recommended to add an auto-replacement logic in the script. ipipgo's API supports pay-per-use billing mode, so you can get new IPs in real time. remember to set the number of retries, so that the program doesn't get stuck.

QA 2: How do I verify if the proxy is working?
First use this command to test: curl -x proxy address http://ip.ipipgo.com/ip, the return IP if the proxy server means success. Don't be lazy and use Baidu to check the IP directly, some websites will block the proxy request.

QA 3: Is downloading large files always interrupted?
Try these two parameter combinations:

curl -C - -L -O --retry 3 --retry-delay 5

-C supports intermittent transfer, -retry set auto retry, pro-tested to solve the network pumping problem of 90%.

Why ipipgo?

A bunch of proxy service providers on the market, but really want to stabilize also have to look at the professional players. I've used an unknown proxy before, and the peak latency is not moving on 2000ms. after changing to ipipgo, I've been able to get the best out of it, and I've been able to get the best out of it:

• Average Response Speed80ms or less
• Support pay-per-use without wasting money
• Professional technical customer service is always available

Especially theirDynamic Residential IP PoolI'm not sure if I'm going to be able to do that, but I'm going to be able to use it to download overseas resources. The last time I helped a friend under the Docker image, direct connection speed 20KB / s, cut to ipipgo's Japanese nodes directly run full bandwidth.

Finally, a reminder: remember to follow local laws and regulations when using proxies, and don't use them for weird stuff. Using tools wisely is the best way to get twice the result with half the effort, don't you think?