Curl Basic Authentication: Example of a request with permissions

Handyman teaches you to play with proxy IP requests with passwords using curl

What do crawlers fear the most? Just two minutes after the collection of IP is blocked! This time the proxy IP is your lifesaver. Today we will take the most common curl command to start, say how to take the account password through the proxy to access the site.

Don't let the basic certification scare you down.

A lot of websites are like ticket gates at an amusement park, gotta get theUsername + PasswordWhen tickets. Remember this universal format if you use curl:

curl -U "account:password" http://目标网址

But watch out! This is the equivalent of writing your password in plain sight, and if you use a proxy... (more on that later)

The right way to open a proxy IP

Using ipipgo's proxy as an example (his proxy with authentication), you have to put the authentication infoAddress of the agent::

curl -x http://用户名:密码@proxy.ipipgo.cn:8080 http://目标网站

Here's the point!Don't use special symbols for username and password, especially @ and these, explode your request in place in minutes.

Practical: crawler old driver's guide

Let's say you bought a short-term package from ipipgo to capture the price of an e-commerce site:

curl -x http://vipuser:8k9d!m2@proxy.ipipgo.cn:3128
-H "User-Agent: Mozilla/5.0"
https://target-site.com/products

Note the three points:

Common Rollover Scene First Aid Kit

Q: What can I do if the agent can't connect?
A: Take this command to test the network first:telnet proxy.ipipgo.co.uk 3128To see the word Connected

Q: Always prompting certificate error?
A: At the end of the curl add-kParameters, but don't use this for sensitive operations

Q: Which ipipgo package is the best deal?
A: For crawlingDynamic Residential AgentsI'm not going to register.Long-lasting static IPRemember to use coupon code IPGO7 to get 30% off your first order for new customers.

Tips from the best of the best

1. Write the proxy configuration into the~/.curlrcfile, saving you from knocking out commands every time:
proxy = http://用户:密码@proxy.ipipgo.cn:端口

2. When you need to change proxy IP, use ipipgo'sAPI interfaceDynamic Acquisition:
curl api.ipipgo.com/getproxy?token=your key

Remember, a good proxy is like gasoline for a sports car, and going with a provider like ipipgo that has its own server room is much more stable than those used proxies. Recently they made aFlow can be carried forwardThe campaigns that run out of traffic can be built again next month, especially for teams with unstable projects.