cURL get file: cURL download file method

Teach you how to use cURL to grab files when hanging proxy

Web developers know that cURL is a tool like a Swiss army knife that can do anything. But if the target site set up access restrictions, grab a file can make you anxious out of gray hair. This time we have to ask the proxy IP this "courier" to help us run errands.

Why do I need a proxy IP to download?

For example, like going to the supermarket to buy a limited number of goods, the same ID card can only buy once. If you use a proxy IP, the equivalent of each time to change a different ID card to purchase, so that not only will not be limited to purchase, but also to protect their real information. Especially for those who do data collection, using ipipgo's dynamic proxy pool can effectively avoid the embarrassment of IP being blocked.

Practical operation guide

Here's an example of ipipgo's proxy service to teach you how to juggle it up on the command line. First, get a proxy address that works, and look for HTTP proxy access information in the ipipgo backend, which usually looks like this:

curl -x http://username:password@gateway.ipipgo.com:9020 https://target.com/file.zip

Watch this.-xThe parameter is the switch that specifies the proxy, followed by the address format must not be written incorrectly. A common mistake made by novices is to leave out the port number or fill in the reverse authentication information, it is recommended that the account password is stored in a separate environment variable:

export PROXY_USER="your_username"
export PROXY_PASS="your_password"
curl -x http://${PROXY_USER}:${PROXY_PASS}@gateway.ipipgo.com:9020 https://example.com/data.csv

Mine clearance for common problems

Q: What should I do if I keep reporting 407 agent authentication errors?
A: First check the three elements: username, password, port is not the same as the ipipgo background. Pay special attention to whether there are special characters in the password, if so, you have to use % plus ASCII code escape.

Q: What if I get disconnected in the middle of downloading a large file?
A: Plus-C-parameter will be able to disconnect, like this:

curl -C - -x http://代理地址 -O https://大文件地址

Advanced Skills Package

If you need to download in bulk, it is recommended to work with ipipgo's API to dynamically obtain proxies. For example, every 5 downloads, change the IP, so as to avoid triggering the wind control to the greatest extent possible. Here is a shell script example:

for i in {1..100}
do
  Proxy = $ (curl -s ipipgo.com/api/getproxy)
  curl -x $ proxy -O "https://资源地址/file_${i}.rar"
  sleep 3
done

Here are the highlightssleepFor this pause command, don't set the interval too short. According to the actual test, with ipipgo's enterprise-level proxy, the interval is set to 3-5 seconds to ensure efficiency and not easy to be blocked.

What to look for in an agency?

Proxy services on the market are mixed, it is recommended to recognize a reliable service provider like ipipgo. Their server room nodes cover 200+ countries and regions, and the response speed is basically at the millisecond level. In particular, friends doing cross-border e-commerce, using their static residential IP to catch the competitor's data, the success rate can be increased by seven or eight percent.

The last nagging word, with the proxy is not open plug-ins, have to comply with the site's terms of service. ipipgo's background has a detailed dosage monitoring, it is recommended that you choose the right package according to the actual needs, do not buy too much traffic all of a sudden waste of money.