cURL Add Request Header: Proxy IP Emulation Browser

Browser Manipulation: cURL Disguise Secrets

Have you ever encountered this situation? With the program to capture data is always intercepted by the site, people can see that you are machine access. Today to teach you a masterpiece - with cURL cloaked in a browser, with theipipgoproxy IPs so that the server does not recognize the real one!

Why the browser vest?

Web servers are now refined and can recognize machine access by two features:

1. Request headers are too clean (browsers come with dozens of parameters)
2. Fixed IP high-frequency access (immediately triggers risk control)

As a chestnut, a naked cURL request looks like this:

curl http://example.com

The server sees this kind of bare board request and just flips you off with a 403 error. We'll have to put some makeup on the request package and put a proxy vest on it.

Request Head Makeup Guide

Focus on camouflaging these three parameters:

Real-world code example (note that the -H parameter is safer to write in segments):

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
     -H "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8"
     -H "Referer: https://www.baidu.com/"
     http://target-site.com

The right way to open a proxy IP

It's not enough to change the request header, the IP address is exposed and you're still screwed. It is recommended to use theipipgoThe Dynamic Residential Proxy, which is massively simple to operate:

curl -x http://username:password@gateway.ipipgo.io:9021
     -H "User-Agent: ..."
     http://target-site.com

Watch this space.-xParameters specified proxy server, ipipgo's proxy pool will automatically rotate IP, more than ten times more stable than a single IP. Tested 500 consecutive requests have not triggered blocking, tested effective!

Anti-Blocking Package

Use a combination of the two tricks for better results:

 Start by defining an array of request headers
headers=(
    "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15"
    "Accept: text/html,application/xhtml+xml"
    "Connection: keep-alive"
)

 Make a request with a proxy
curl -x http://user123:pass456@proxy.ipipgo.io:9021
     $(printf "-H '%s' " "${headers[@]}")
     https://target-site.com/api/data

This script uses ipipgo's intelligent routing, automatically switching the exit IP for each request, with random request header parameters, the anti-blocking rate of 90% or more.

Frequently asked questions on demining

Q: What should I do if I add a request header and still get recognized?
A: Check for cookies and SSL fingerprinting, recommended to be turned on in the ipipgo consoleBrowser Fingerprint Emulationfunctionality

Q: What should I do if my proxy IP often times out?
A: After the curl command, add--connect-timeout 10parameters, while contacting ipipgo technical support to optimize the line

Q: What if I need a multi-threaded request?
A: Parallel execution with xargs with ipipgo'sMulti-Channel Concurrent PackageSpeed is straight off the charts.

Lastly, I would like to remind you not to be greedy for cheap proxy services. I've used a couple of small shops before, and their IP purity is not even close to the same.ipipgoThe ratio is more than that. People have their own server room + operator cooperation resources, the success rate can be 99.2%, do data collection know how rare this number. New user registration also sends 20G traffic package, do not try for nothing, right?