curl download images: curl proxy download images

Teach you how to use curl to hang proxy to download images

Crawlers should have encountered this situation: under the next picture was suddenly blocked by the site IP! At this timeproxy IPis a life saver. Today, we will take the actual case to teach you how to use curl to hang the proxy under the picture, guaranteed to look at the end of the hand.

Why do you need to put up a picture of the agent?

To give a real example: last week I wanted to batch download an e-commerce platform of goods, the first 50 are fine, to the 51st suddenly returned 403 error. This is a typicalIPs are recognized as crawlersI've got it. After hooking up a proxy IP, the program automatically switches between different IPs and downloads 500 images without any problems.

 Direct download (will be blocked)
curl -O https://example.com/image1.jpg

 Proxy download (unimpeded)
curl -x http://ipipgo-proxy:8000 -O https://example.com/image1.jpg

Setting up a curl proxy in three steps

Here's the kicker! Setting up with ipipgo's proxy service is particularly easy:

1. Log in to the ipipgo backend to get the proxy address (format: ip:port)
2. After the curl command, add-xparameters
3. Remember to replace your account password (packages without passwords can be used directly)

 Write with account password
curl -x http://user:pass@proxy.ipipgo.cn:23333 -O https://target.com/img.jpg

How do you choose the type of agent?

Common Rollover Scene QA

Q: What should I do if the proxy IP shows connection timeout?
A: first ping the proxy server address, can pass, then may be the target site blocked the current IP. ipipgo with theautomatic switchingfunction to set the number of failure retries in the code.

Q: What should I do if I get disconnected in the middle of the download?
A: curl plus-C-The parameters continue to be transmitted, in conjunction with ipipgo'sLong Connection ProxyPackage, Stability Enhancement 80%

Q: How do I verify if the agent is in effect?
A: Use this command to check the current exit IP:
curl -x http://代理IP -sS whatismyip.ipipgo.net

Guide to avoiding the pit

A common mistake newbies make isLack of attention to concurrency control. Even with a proxy, high frequency access from the same IP will still be recognized. Suggestion:

1. Control of no more than 3 requests per second
2. With ipipgoRotation agent pool(5000+ IP per day)
3. Setting random waiting times (0.5-2 seconds)

One final piece of cold knowledge: some sites will detectTCP fingerprintThis is when normal proxies may not work. In this case you have to use ipipgo'sAdvanced Protocol Supportservice, their tech guy can help you customize your solution.