cURL Modify User-Agent: Proxy IP Request Header Settings

Don't let the User-Agent betray you!

You old iron in the use of crawlers grip data, certainly encountered the website dead not to visit the situation, right? At this timeUser-AgentIt's your lifeblood. To give a chestnut, you use the default cURL request header, the server took a look: "Oh, curl/7.68.0 this little brat came to steal data again", directly to you 403 blocked without negotiation.

 Finding a way to request a hit
curl https://example.com

Nice disguise.

This is the time to offerUser-Agent CamouflageThe following is an example of how the success rate can be increased by more than three times with the following pose. Our ipipgo's tech guy tested it, and the success rate can go up more than three times with the following pose:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36..."
--proxy http://user:pass@proxy.ipipgo.com:8888
https://target-site.com

Here's the point:-A parameterIt is to change the User-Agent switch, and the string of Martian text connected to the back should look like a proper browser. It is recommended to copy the latest version of UA from the browser developer tool, don't use those outdated templates on the internet.

Proxy IP's Gold Partner

Changing your UA alone is not enough, smart sites will remember your IP, and that's where we come in.Dynamic Residential Proxy for ipipgoThe effect is like putting ten layers of stockings on a request:

The actual code is written like this:

curl -x http://user:pass@proxy.ipipgo.com:8888
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)..."
-H "Accept-Language: zh-CN,zh;q=0.9"
https://data-source.com/api

Note the -H parameterHere in addition to the UA also added the language identification. Some sites calibrate multi-dimensional information, so don't get in the gutter.

Guide to avoiding the pit

A common rollover scene for newbies:

1. UA and IP locale mismatch (e.g. using a Chinese UA with a US IP)
2. Forgetting to set the timeout parameter causes a jam
3. Forgetting to switch IP pools for high frequency requests

Recommended for ipipgoIntelligent Rotation Modelthat automatically handles IP switching. Code example:

curl --proxy http://user:pass@auto-rotate.ipipgo.com:8899
---max-time 30
-A "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X)..."
https://mobile-site.com

Frequently Asked Questions QA

Q: What if the UA is disguised or still recognized?
A: Eighty percent of the other requesting heads showed up with thecurl -vView the full request header and generate a full set of artifacts with ipipgo's request header template tool.

Q: How to solve the problem of slow proxy IP speed?
A: Switching in the ipipgo consolelow latency nodePlus--connect-timeout 10parameter, which is automatically discarded after 10 seconds.

Q: What if I need a multi-threaded request?
A: Use ipipgo'sconcurrent authorizationfunction, each thread with a separate proxy, never share an IP with multiple threads.

As a final rant, a lot of websites now haveBehavior Analysis SystemIf you want to change your UA and IP, you should remember to control the frequency of requests. If you really can't do it, ipipgo's technical support is available 24/7, and you can get free test hours by reporting the code word "UA".