curl -header option: Curl header settings

How does curl's -header option really play out?

guys with curl test interface, often encountered 403, 404 this headache. Today we will chatter how to use -header option with proxy ip to solve these moths. First, remember the truth:The server looks at the request like a security guard checking an ID card, without the correct header it will stop you in minutes.The

Basic operations: putting a hat on curl

As a chestnut, trying to masquerade as a proper browser visit, write it like this:

curl --header "User-Agent: Mozilla/5.0" http://example.com

Here the request is given a browser logo hat. But the reality is more complicated, many sites will detect the source of the request, and this is where proxy ip is needed to fight the battle of cooperation.

The golden combination of proxy ip and header

When using ipipgo's proxy service, you often have to deal with two kinds of headers:

curl --proxy http://user:pass@proxy.ipipgo.io:8080
--header "Authorization: Bearer your_token"
https://api.target.com

watch carefullyHere are two key points: the proxy server address should be written correctly, the authentication information do not put in the wrong place. ipipgo users often planted in the account password written directly in the header, in fact, should be placed in the proxy address of that part.

Real-world anti-blocking techniques

Recently, I helped a customer to deal with a case: they used curl to collect data in bulk and it was always blocked. Later, they found that the Accept-Language in the header was not set, and added this immediately:

curl --header "Accept-Language: zh-CN,zh;q=0.9"
--proxy http://ipipgo动态代理池地址
https://target-site.com/data

With ipipgo's auto-switching ip function, it was tested to run continuously for 12 hours without being banned.

QA time

Q：Why is the header still 403?
A: Ninety percent of the proxy ip is the target site pulled black, change ipipgo high stash proxy to try, remember to check the Referer header!

Q: What should I do if I can't connect to the proxy ip?
A: First check whether the network is open, and then make sure the authentication information is correct. ipipgo's technical support response is quite fast, and they can also check the connection log in the background.

Q: How to view the header returned by the server?
A: Add a -i argument:

curl -i --header "X-Requested-With: XMLHttpRequest" https://example.com

the right tool saves effort and leads better results

When tossing header configurations, the biggest fear is encountering ip blocking. There is an advantage to using ipipgo's proxy service - theirDynamic residential ip poolCan automatically change the header fingerprints, with the -header option of the curl is simply a combination of two swords. Last time there is a price monitoring customer, the original maintenance of their own proxy server burns more than 20,000 per month, replaced with ipipgo after the cost directly cut in half.

A final reminder for newbies:Don't expose real information in the headerIf you have a problem, ipipgo's technical support can help you analyze the packet data, which is much better than doing it yourself. When you encounter problems, ipipgo's technical support can help analyze the packet data, which is much stronger than doing it yourself.