curl with request header | simulate browser access settings tips

I. Why do I need to install a browser to access it?

A lot of sites are now with the thief like, see the direct curl send a request to dump face. For example, you use the command line tool curl to grab data directly, nine times out of ten will receive 403 forbidden. this time you have to give the requestput on makeup, making the server think that a real person is operating with a browser.

Here is a pitfall to note: some sites not only check User-Agent, but also look at Accept-Language, Referer and these parameters. It's like when you go to a fancy restaurant, you don't just have to look at what you're wearing, but you also have to say the right password to get in.

Second, the hand to teach you curl camouflage art

Let's look at the simplest example first:

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" https:/ /example.com

here are-H parameterIt's the switch that stuffs things into the request header. Focus on the field to be forged look at this table:

Third, with proxy IP more insurance

It's not enough to change the request header, some sites hold grudges - they block IPs when they find a suspicious visit, so it's time to use theProxy services for ipipgoto fight guerrilla warfare. This is done in three steps:

1. First go to ipipgo official website to get a package, it is recommended to choose dynamic residential IP (this is the most like a real person)
2. get the API address and port number
3. Add the proxy parameter to the curl command.

The full command looks like this:

curl -x http://username:password@proxy.ipipgo.com:port -H "User-Agent: Browser UA" https://目标网站

IV. Practical QA First Aid Kit

Q: What should I do if I add a request header and still get recognized?
A: It is likely that the cookie is exposed. First, use the browser to visit the normal once, copy the cookie out completely, and then use the-H "Cookie: xxx"Parameters to bring on.

Q: How many times will the proxy IP be invalidated?
A: This situation is recommended to change ipipgo'squantity-based billing packageTheir IP pool is updated quickly and automatically changes IPs with each request, saving you the trouble of manually tossing them.

Q: What if I need to simulate mobile access?
A: Change the User-Agent to the mobile version, for example:

Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1

V. Advanced player skills

When you come across a particularly difficult website, try these damaging tricks:
- Random hibernation: prefix the curl command withsleep $((RANDOM%5+2)) Pretend to think about life.
- Mixed use: residential IPs and server room IPs swapped (ipipgo has both types)
- Request header shuffle: each time randomly adjust the order of the header, so that the site can not catch the law

One final note: when using a proxy IP, remember to choose something like ipipgoProvide full HTTP header informationservice providers. Some cheap proxies will omit to pass X-Forwarded-For these key headers, and are easily exposed instead.