Example of curl-header: Example of request header settings

I. Why toss the request header?

Folks use curl to do data crawling, often encountered by the site anti-climbing situation, right? This timeRequest HeaderIt's your cloak of invisibility. For example, some sites see you using the default User-Agent of curl and just block the request. A proxy IP with a specific request header allows you to sneak around the page like a real human browser.

Second, curl play request header three axes

Remember this universal formula:curl -H "header field: value". Three real-world scenarios are demonstrated below:

 Fake Chrome browser
curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
--proxy http://user:pass@ipipgo-proxy.com:8080 https://target-site.com

 Customize the content type
curl -H "Content-Type: application/json"
--proxy socks5://ipipgo-proxy.com:1080 -X POST -d '{"key": "value"}' https://api.example.com

 Carry login credentials
curl -H "Authorization: Bearer your_token_here"
--proxy http://ipipgo-proxy.com:3128 https://member-only.site

Third, the proxy IP and request header double sword combination

Proxy IP alone is like wearing a nightclothes shopping, plus the request header camouflage is the real - stealth. Here we recommend usingipipgo Dynamic Residential ProxyThe IP pool of their home is automatically replaced every day, with the following combination of scripts, anti-blocking effect:

! /bin/bash
for i in {1..10}; do
  curl -H @headers.txt
  --proxy $(shuf -n 1 ip ipgo-ip-list.txt)
  https://data-scraping-site.com/page=$i
  sleep $((RANDOM % 5 + 2))
done

Remember to save the proxy address provided by ipipgo to theipipgo-ip-list.txtThe request header parameters are placed separatelyheaders.txtManaged in the file.

IV. Pitfalls often stepped on by novices

- Case-sensitive header fields (Content-Type ≠ content-type)
- Forgot to deal with redirects (plus)-L(Parameters)
- Proxy protocols are confused (http proxies can't connect to https sites)
- Frequent changes of User-Agent instead of triggering risk control

V. QA First Aid Kit

Q: Do I still need to set the request header with ipipgo proxy?
A: It is necessary! Proxy IP solves the problem of IP blocking and request header solves the problem of identification, they are complementary.

Q: Why is my curl command with proxy still banned?
A: Check three points: 1. whether the proxy IP is effective 2. whether the request header is complete 3. whether the access frequency is too high. It is recommended to use ipipgo'sIntelligent Rotation AgentsPackage with its own frequency control.

Q: How to manage multiple request headers in bulk?
A: Recommended-H @filenamesyntax to store request headers for different scenarios into multiple files, for example:

curl -H @mobile_headers.txt --proxy ipipgo-proxy.com:8888 https://m.site.com
curl -H @desktop_headers.txt --proxy ipipgo-proxy.com:8888 https://www.site.com

VI. Private room tuning skills

1. Randomly generated Accept-Language field
2. Adding a no-trace browsing feature header (e.g., DNT: 1)
3. Mixed use of ipipgo's static long-lived IPs and dynamic IPs
4. Adaptation of header information according to the type of target web server (Nginx/Apache is treated differently)

One last nag: don't be a fool and use the default User-Agent, the webmaster sees thecurl/7.68.0This kind of logo is a no brainer to pull the plug in minutes. Use ipipgo'sEnterprise-level agency servicesTheir tech guys can also help you customize your anti-blocking strategy, which is a lot less stressful than tossing it yourself.