curl crawl site: command line harvesting tutorial

Why does curl use a proxy ip to catch websites?

Engaged in the old driver of the network crawler understand, directly with their own computer IP hard rigid web server, as dangerous as wearing pants standing in the snow. The site's anti-crawler mechanism is not vegetarian.The lesser case is to block the IP for half an hour, the more serious case is to directly pull the black listThe server will not be able to tell who is who. At this point the proxy ip is like wearing a vest for curl, each request changes identity, the server can not tell who is who.

For example, an e-commerce platform is limited to 500 visits per hour, with their own broadband to last up to 5 minutes on the break. If you use ipipgo's Dynamic Residential Proxy, which automatically changes the IP address for each request.Acquisition efficiency directly ten timesAnd without taking a breath. Here's the kicker, there are three metrics to look for when choosing an agent:

Hands-on teaching curl with proxies

Don't be intimidated by the command line, it's really just a few more parameters than regular curl. Let's say you've signed up for ipipgo and got a socks5 proxy account:

curl -x socks5://username:password@gateway.ipipgo.com:1080 https://target.com

There are a few pitfalls to watch out for here:

1. 密码含特殊符号记得用%编码，比如@要写成%40
2. https sites must use high stash proxies, otherwise the real IP will be exposed
3. We recommend adding the -connect-timeout 30 parameter to the timeout setting.

Practical anti-blocking techniques are given out.

It's not enough to be able to use proxies, you have to learn to disguise yourself as a normal person. Here are three tricks for you:

Trick #1: Random Hibernation

sleep $((RANDOM%5+1)) Random pause 1-5 seconds

Tip #2: Request Header Obfuscation

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
     -H "Accept-Language: zh-CN,zh;q=0.9"
     -x http://ipipgo-proxy.cn:8080

Tip #3: IP Rotation

Use ipipgo's API to get the proxy pool dynamically, it is recommended to call the interface to change the IP before each request:

API_URL="http://api.ipipgo.com/getproxy?key=YOUR_KEY&protocol=socks5"
PROXY=$(curl -s $API_URL)
curl -x $PROXY https://target.com

Frequently Asked Questions QA

Q: What should I do if my proxy IP is not working?
A: eighty percent of the IP was the target station pulled the black, hurry to switch ipipgo automatic rotation mode, their home pool is updated every day 200,000 +IP

Q: Why is it still recognized even if I use a proxy?
A: Check if you are using a transparent proxy, ipipgo's high stash of proxies will completely hide the X-Forwarded-For header

Q: What configuration is required for enterprise-level acquisition?
A: It is recommended that the enterprise version of ipipgo, support for concurrency 500 +, with automatic retry and failure rate monitoring Kanban

How to choose a reliable proxy service

Agency services on the market are a mixed bag, so remember these three guidelines for avoiding pitfalls:

1. Don't believe in perpetually free services that either limit speed or sell data
2. See if multiple protocols are supported, like ipipgo supports both HTTP/S and SOCKS5
3. Test IP purity, use this command to check for X-Real-IP header leakage:

curl -x proxy IP http://httpbin.org/headers

Lastly, I'd like to apologize for the fact that ipipgo has been doing a lot of activity lately, and new users are getting 10G of traffic to try out. Their dynamic residential agent is particularly suitable for long-term collection projects, IP survival time than other parents 3 times, the key is the customer service response, the last two o'clock in the middle of the night to mention the work order actually seconds back....