curlGET request: command line data fetch

Hands on teaching you to play with proxy IP requests with curl

Crawlers should understand that the old iron, with curl direct request site is like a naked run - minutes by the target site ban off the IP. this time you need a proxy IP to be your stealth vest, today we take the command line artifact curl to open the knife, teach you how to use ipipipgo proxy service to safely engage in the data.

Curl Basic Operations Guide

Let's start with an appetizer. A normal curl request looks like this:

curl https://example.com/api/data

But this directly expose your real IP, like in the live room face open plug-in, not be blocked only strange. This time we have to pull out ourThe Proxy IP MethodThe

Proxy IP real-world configuration

Take ipipgo's proxy service as an example (their dynamic IP pool is genuinely solid), add a proxy parameter to curl and take off:

curl -x http://username:password@proxy.ipipgo.io:8080 https://target-site.com

There are three key points to note here:
1. Proxy protocol selection http/https according to the requirements of the service provider to
2. The username and password must be the authorization code given by the ipipgo backend.
3. Do not fill in the wrong port number, which may be different for different packages.

High Frequency Pitfall Warning

Three common mistakes newbies make:
1. Write the proxy address as the IP of your own router (what's the difference between this and not wearing a vest)
2. forget to replace username/password in the example (it's a crap shoot if you use a fake password)
3. Failure to pay attention to the concurrency limits of proxy packages (ipipgo's concurrency varies greatly from package to package)

Upgrading of practical skills

When you need to switch IPs, use ipipgo's API to get new proxies in real time:

NEW_IP=$ (curl -s api.ipipgo.com/getproxy)
curl -x $NEW_IP https://target-site.com

Add the random request header for more insurance:

curl -x http://proxy.ipipgo.io -H "User-Agent: $(shuf -n 1 ua-list.txt)" https://target-site.com

Troubleshooting QA

Q: What about proxy request timeout?
A: First check the remaining traffic in the background of ipipgo, and then test the proxy server ping value, the delay of nodes in different regions is much worse!

Q：Return 407 Agent Authentication Error?
A: Ninety percent of the account password is wrong, ipipgo password case sensitive, it is recommended to directly copy the console code examples

Q: How can I tell if a proxy is in effect?
A: Use this command to check the exit IP:

curl -x http://代理信息 -s ip.ipipgo.com/checkip

Performance Optimization Tips

Multi-threaded concurrent requests with xargs to squeeze the IP pooling advantage out of ipipgo:

cat urls.txt | xargs -P 10 -I {} curl -x http://动态代理 -s {}

Remember to adjust the number after -P, don't exceed the maximum number of concurrency allowed by the package or you will be automatically limited by ipipgo's system.

Lastly, anecdotally, ipipgo has recently added the newpay-per-use packageThe first one is that you don't need to tie the card to use it, which is especially suitable for the old man who does the short-term crawler project. Their residential agent to play the verification code success rate is super high, personally tested than the IP is much more stable, there is a need to go directly to the official website to jerk test hours.