curl Requests with Headers: Anti-Detection Configuration Example

Hands on with curl to hide the horse's feet

The old iron engaged in network requests know that now the site anti-climbing mechanism is more and more refined. Proxy IP is not enough, you have to request the head of the whole operation like a real person. Today we take the most common curl command to open the knife, say how to configure the target site can not recognize you as a machine.

Why did the requestor toss it?

A lot of newbies think that hanging a proxy IP will be all right, in factThere's a lot of holes in the request header.. Give me a chestnut:

curl http://example.com

This default request exposes your curl version, and webmasters will see this feature header and block your IP in minutes. all we have to do is change the request header to be exactly the same as the browser.

Live Configuration Templates

Here's a tested and working configuration (remember to change the proxy to your own):

curl -x http://username:password@gateway.ipipgo.net:9020
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
-H "Accept-Language: zh-CN,zh;q=0.9"
-H "Referer: https://www.google.com/"
-H "Sec-Fetch-Dest: document"
https://target-site.com

A few key points:

Easy to step on the pit

1. User-Agent expired: Don't use an old version of browser UA, it is recommended to update the version number once a month
2. Language parameter conflicts: Accept-Language without en, unless you want to visit an English site.
3. Agent instabilityRecommended to use ipipgo's exclusive proxy, the actual failure rate can be pressed to 0.3% or less!

QA time

Q: Do I still have to change the request header with the high anonymity proxy?
A: It must be changed! High Stash can only hide the IP, the request header characteristics will still be recognized

Q: How do I choose a proxy address for ipipgo?
A: There is a ready-made code generator in the backend of the webpage, choose "curl+anti-detection" mode and copy it directly.

Q: Is more request header parameters better?
A: Big mistake! Adding too many non-useful parameters is suspicious, according to the template we give enough!

advanced skill

1. Random delay: add sleep $((RANDOM%5)) before the command to pause randomly for 0-5 seconds.
2. Rotate UA: store commonly used UA in a txt file and read it randomly for each request.
3. protocol camouflage: ipipgo's proxy supports both socks5 and http modes, so it is recommended to mix them up.

The last thing I want to remind you is that the configuration is perfect, but the quality of the proxy is not good enough. Recently, when I helped a customer debugging, I found that the success rate of the ordinary proxy is only 78%, but the business package of ipipgo directly soared to 99.6%. This thing is similar to wearing clothes, the jacket (proxy) and underwear (request header) have to pay attention to.