curl set request header | custom UA and Referer

Hands-on request header masquerading with curl

Folks in the use of curl grab data, is not often encountered in the website dead not give response? This is something I encountered every day last year when I was doing e-commerce price monitoring. Later found thatNot camouflaging a request header is like surfing the web naked.The website can recognize you as a crawler at a glance. Today we will talk about how to use the proxy IP with curl set request header, focusing on the recommended home to use the smooth handipipgoAgency services.

Why toss the request header?

举个实际例子：去年双十一我想抓某平台的促销数据，用自己电脑的IP，刚发几个请求就被封。后来给curl挂上ipipgo的动态住宅代理，再改下UA和Referer，连续跑了3天都没事。这就像Wearing a human skin mask to a masquerade ball.The website won't even recognize who you are.

curl sets the request header core parameters

Remember these three mandatory changes:
– -H "User-Agent: ..."(Equipment fingerprints)
– -H "Referer: ..."(incoming page)
– -x Proxy server address(Suggested socks5 proxy with ipipgo)

The actual order looks like this:

curl -x socks5://user:pass@gateway.ipipgo.io:20000
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
-H "Referer: https://www.example.com/product/123"
https://target-site.com/data

Proxy IP Selection with Care

After using 7 or 8 proxy service providers, I finally settled on ipipgo for two main reasons:
1. The residential IP pool is large enough (I've heard 20 million +)
2. Automatic session maintenance (especially save when doing operations that require logging in)
Be careful with the format of their home proxy address:gateway.ipipgo.io This domain is a fixed entry, don't use it wrong.

Common Rollover Scene QA

Q: What should I do if the order of parameters is always mixed up?
A: Remember the mnemonic: proxy settings (-x) at the top, header information (-H) in the middle of the row, the target URL last with the

Q: Does UA use the mobile or computer version?
A: Look at the target website traffic source, e-commerce class more cell phone UA, enterprise official website more PC. ipipgo background has ready-made UA library can be copied directly.

Q: How is the dynamic request header implemented?
A: We recommend using ipipgo's intelligent routing function, which can automatically rotate UA and Referer, and saves much more work than writing scripts by yourself!

Guide to avoiding the pit

I recently discovered that some websites detectHeader integrityLast week, a customer failed to add the Accept-Language header. Last week, a customer fell into the Accept-Language header was not added, obviously all other parameters are correct, but still recognized as a robot. It is recommended to use ipipgo's request header checkup function to automatically complete the necessary parameters.

Finally, a cold knowledge: remember to turn off the system proxy settings when using proxy IP! Once debugging half a day without results, and finally found that the computer on the global proxy, two proxies lead to timeout, this low-level error do not make.