curl set request header: curl proxy request header setting

Hands-on with curl to play with proxy request header

Brothers engaged in network development must have touched the curl Swiss Army knife, but when it comes to the request header with proxy configuration, many newbies are prone to confusion. Today we will specifically nag about this, especially with the ipipgo family of proxy services how to use it smoothly.

Curl basics to memorize

Let's start with a quick lesson for the completely uninitiated. curl's most basic proxy setup command looks like this:

curl -x http://代理IP:端口 Destination URL

For example, with ipipgo provides dynamic residential agent (their official website has detailed package description), assuming that get an address is 123.45.67.89:8000, access to Baidu should be so written:

curl -x http://123.45.67.89:8000 https://www.baidu.com

Request head camouflage trio

It's not enough to use a proxy IP, some sites will detect request header characteristics. Here are three surefire ways to teach you:

1. User-Agent randomization: change the browser ID for each request, ipipgo's control panel can generate a list of random UA's directly

curl -x http://123.45.67.89:8000 -H "User-Agent: Mozilla/5.0 (Windows NT 10.0)..." https://目标网站

2. The protocol version should look like: don't use the protocol header that comes with curl, manually masquerade as a browser

-H "Accept: text/html,application/xhtml+xml..."
-H "Accept-Language: zh-CN,zh;q=0.9"

3. X-Forwarded-For don't miss it!: This header helps proxy servers to pass the original IP information correctly, which they will handle automatically if you use ipipgo.

Troublesome maneuvers in the real world

What to do when you come across a difficult website? Try this combo:

curl -x http://123.45.67.89:8000
-H "User-Agent: randomly generated UA" -H "Accept-Encoding: gzip, deflate"
-H "Accept-Encoding: gzip, deflate"
-H "Connection: keep-alive"
https://难搞的网站.com

Look at the backslashes in the arguments, this is how line breaks are written in Linux. If you use Windows cmd, you have to change it to use the ^ symbol.

Common Rollover Scene QA

Q: Why is the 407 error code returned?
A: This is a proxy authentication failure, check the authentication method of ipipgo account. They support both username and password and whitelist authentication, don't get confused!

Q: How do I break the request timeout?
A: First, use the connectivity test interface provided by ipipgo to check if the proxy is normal, their node survival rate is usually above 99%. If there is no problem, adjust the timeout parameter appropriately:

curl --connect-timeout 30 -x http://代理IP...

Q: What if I need to deal with cookies?
A: Use the -c and -b parameters with the proxy, and remember to put the cookie storage file in a safe location:

curl -x http://代理IP -c cookie.txt -b cookie.txt Destination URL

ipipgo exclusive optimization tips

Their proxies have a hidden feature where adding specific identifiers in the request header can trigger smart routing:

-H "X-IPIPGO-Mode: auto_retry"

This header allows the system to automatically switch the exit node of the failed request, measured to improve the success rate of 20%. Specific usage to their official website to check the development documents, here do not expand on it.

Lastly, I would like to remind newbies that proxy configuration is not a one-off solution. Suggested with ipipgo API timed IP replacement, especially when doing data collection, do not catch an IP to death. What do not understand, directly to their technical support, the response speed is much faster than most counterparts.