Using proxies with curl: Examples from the command line

Teach you how to use curl to hang a proxy!

engage in network development know, curl this thing is like a Swiss army knife, what protocols are supported. But when it comes to the need for proxy scenarios, many newbies will be blind. Today we take ipipgo proxy service as an example, nag how to play in the command line proxy settings.

Basic posture: put a "cloak of invisibility" on the curl.

The simplest proxy setup is two parameters:-x maybe -proxy. For example, with ipipgo's HTTP proxy:

curl -x http://user:pass@proxy.ipipgo.com:8080 https://target-site.com

Note the formatting here:Protocol://username:password@proxy address:portIf you don't have a password, just write the address. If you don't have a password you can just write the address. such as a temporary proxy for development and testing:

curl --proxy socks5://test-proxy.ipipgo.com:1080 http://example.com

Advanced play: split between different protocols

There's a cold fact about curl:-proxyrespond in singing-socks5It is possible to mix and match. For example, using a SOCKS5 proxy to access an HTTP site:

curl --socks5 proxy.ipipgo.com:1080 -L http://login-page.site

The -L parameter here is an automatic follow-up redirection, which is particularly handy when used with proxies. Another example is when you need to test with multiple proxies at the same time:

curl --proxy http://proxy1.ipipgo.com:8080 --proxy http://proxy2.ipipgo.com:8081 -U user:pass https://api.service.com

Guide to avoiding pitfalls: common moths to deal with

1. EncounterProxy CONNECT abortedError: Most likely the proxy server is jerking, change the node of ipipgo try
2. TimeoutProblem: Add a -connect-timeout parameter, for example, set a timeout of 5 seconds:

curl --proxy http://proxy.ipipgo.com:8080 --connect-timeout 5 https://slow.site

3. Certificate error: add a -k parameter to ignore SSL authentication (for test environments, use with caution in production environments)

Real-world scenarios: a few chestnuts

Scene 1:Test API interface connectivity
Bulk test with ipipgo's rotating proxy pool:

for i in {1..5}; do
  curl -x http://proxy$i.ipipgo.com:9090 https://api.example.com/health
done

Scene 2:Capture pages that require login
Works with cookies:

curl -x socks5://auth-proxy.ipipgo.com:1080 -b "sessionid=xxxx" https://member.website.com

QA time: you may want to ask

Q: What should I do if I can't connect to the agent all the time?
A: First check the proxy address/port is correct, then use telnet to measure the connectivity:
telnet proxy.ipipgo.com 8080
If you can't connect it, change to other nodes of ipipgo, their nodes have a survival rate of 95% or more.

Q: What if I need to switch proxies frequently?
A: Write the proxy configuration to an environment variable:
export http_proxy=http://proxy.ipipgo.com:8080
export https_proxy=$http_proxy
This way all curl requests automatically go to the proxy

Q: Slow agents affect business?
A: Recommended for ipipgoExclusive use of high-speed lines, measured download speeds can go up to 50Mbps+. When testing, you can add a-wparameter to see the time consumed:
curl -x http://vip-proxy.ipipgo.com:8080 -w "time: %{time_total}s" https://file.server.com

Curl with proxy is not difficult to say, but the doorway is quite a lot. The key to choose a reliable proxy service, like ipipgo with intelligent routing, different business scenarios have corresponding solutions. Finally, remember:Always use a certified agent for production environmentsDon't use public free proxies, you'll get your data intercepted in minutes.