Example of proxy IP setting cURL request header: Example of cURL proxy IP request header

Teach you how to use cURL to hang proxy IP handily

Crawlers should understand the old iron, direct bare request site minutes to be blocked IP. this time the need for proxy IP to cover, today we take the most common cURL tool to open, say how to give it to wear a "cloak and dagger".

Let's look at the most basic way to play: suppose you start with theipipgoI got a proxy IP. It looks like this.123.45.67.89:8080, the pose of stuffing into the cURL looks like this:

curl -x http://123.45.67.89:8080 https://目标网站.com

Watch this space.-xThe parameter is the old driver that specifies the proxy, http://开头说明用的是HTTP协议代理. If the proxy server requires account password authentication, remember to write it this way:

curl -x http://用户名:密码@123.45.67.89:8080 https://目标网站.com

The Great Request Header Disguise

Some websites are so sophisticated that it is not enough to change the IP address, but also to change the request header. This is the time to be on the-Hparameters to get a facelift:

curl -x http://123.45.67.89:8080
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0) decent browser"
-H "Accept-Language: zh-CN"
https://目标网站.com

Focus on disguising these three header messages:

IP auto-rotation trick

Single IP used for a long time will still be recognized, this time to ipipgo on thedynamic agent pool. Their home agent address looks like this:gateway.ipipgo.com:20000, automatically changing IPs with each request:

curl -x http://账号:密码@gateway.ipipgo.com:20000 https://目标网站.com

Pay attention to the port number, ipipgo home different ports correspond to different services:

• Port 20000: Domestic Mixed Dialing Agent
• 30000 ports: Overseas quality lines
• Port 40,000: Highly Anonymized Enterprise Proxy

A guide to avoiding the pitfalls of HTTPS requests

When you encounter an HTTPS site, it may report an SSL error, so you should add a-proxy-sslParameters:

curl --proxy-ssl -x http://123.45.67.89:8080 https://加密网站.com

If that doesn't work, try ignoring certificate validation (don't use it for important operations):

curl -k --proxy-ssl -x http://123.45.67.89:8080 https://加密网站.com

Real-world QA triple play

Q: Proxy settings are successful but I can't connect?
A: First remove the proxy and directly ping the gateway address, for exampleping gateway.ipipgo.comSee if you can get through, then check if the account password was copied wrong.

Q: What should I do if I return a 407 agent authentication error?
A: Ninety percent of the account password format is not correct, the correct posture isUsername:Password@Proxy Address, note that special characters should be URL-encoded.

Q: How can I check the actual export IP used?
A: Hook up the proxy and access the detection interface of ipipgo's home:
curl -x http://代理IP:端口 https://api.ipipgo.com/ip

The doorway to choosing a proxy service

The last word of caution is that you have to look at three things when choosing an agency service:

1. Is the IP pool big enough (ipipgo home is updated daily with millions of IPs)
2. Is there an automatic authentication feature (their smart scheduling system is real flavor)
3. Steady response speed (measured domestic node latency <50ms)

If you want to save yourself the hassle you can just use the one provided by ipipgocURL dedicated proxy channelThey also prepared a ready-made code template, the official website documents can be used to pull down. Remember that the agent of this matter with the same wear vest, the key is to often change to wear, do not let the target site to recognize your real body.