First, the first whole understand what is the proxy IP?
Let's say you fill in the wrong shipping address when you shop online, the courier guy will deliver the package to the wrong place. Proxy IP is like a temporary change in the delivery address, so that the target server thinks you are a visitor from another place. For example, if you visit a Beijing website with a Jiangsu IP, the other side will see that the source of the visit is Jiangsu.
The most common situation that our crawler friends encounter:Continuously use the same IP to grab data, the site directly to you blacklisted!The first thing you need to do is to rotate your proxy IPs to avoid being blocked. This time you need to proxy IP rotation, to avoid being blocked. However, special attention should be paid to find a proxy service provider do not choose those who come from an unknown source, such as we use ipipgo such a formal qualification is reliable.
Second, Python to engage the proxy of the two poses
Let's start with the most commonly used requests library, adding proxies is huge and simple:
import requests
proxies = {
'http': 'http://用户名:密码@ipipgo proxies:port',
'https': 'https://用户名:密码@ipipgo proxy:port'
}
resp = requests.get('destination URL', proxies=proxies, timeout=10)
print(resp.text)
It's a little trickier if you use urllib, but you don't have to install third-party libraries:
from urllib.request import ProxyHandler, build_opener
proxy = ProxyHandler({
'https': 'https://ipipgo提供的代理地址:端口'
})
opener = build_opener(proxy)
response = opener.open('https://要访问的网站')
print(response.read().decode('utf-8'))
Third, avoid these pits can save 80% time
Three of the easiest places for newbies to roll over:
1. Wrong proxy format: pay special attention to http and https should be written separately, there are special symbols in the password remember to escape with %
2. The timeout is not set.: It is recommended that the timeout should not exceed 15 seconds, otherwise the program will be jammed.
3. IP quality is not good
That's why we recommend ipipgo, which is optimized for crawlers.Dynamic Residential PackageI've been using the standard version with a dynamic IP address for three days and it didn't trigger a counter-script. Tested with the standard version of the dynamic IP, running for three consecutive days script did not trigger anti-climbing. To maximize agent results, remember this golden combination: 1. Randomly change the IP for each request (ipipgo's API can directly batch obtain) 2. With User-Agent random switching 3. Controlling the frequency of visits Give a real-world code example: Q: What should I do if the proxy IP is invalidated after using it? Q: What if I need a lot of IP for my company's projects? Q: The code looks fine but I can't connect to the agent? I've used 7 or 8 proxy services and finally locked in to ipipgo for three main reasons: 1. The protocol supports full: socks5 and https both work, some sites only recognize specific protocols 2. Client Saving: Their Windows client is a real handicap boon, you can switch IPs in a couple of clicks! 3. Price transparencyUnlike some platforms that hide surcharges, the standard version of Dynamic Residential starts at $7.67/G, making the cost of data collection manageable! Lastly, don't just look at the cheap proxy service. Before the cheap buy 5 dollars 100G, the result is that half of the IP are waste, but delayed. Now with ipipgo's standard version of dynamic IP, collection efficiency at least 3 times, the key is not to toss all day to change the IP.IV. Practical skills to give away
import random
import requests
def get_ipipgo_proxy():: Here we change the API address to the one we got from ipipgo_proxy.
Here we replace it with our own API address from the ipipgo backend
api_url = "https://api.ipipgo.com/getproxy"
proxies = requests.get(api_url).json()['data']
return random.choice(proxies)
headers_list = [
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0)...'} ,
{'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...'}
]
resp = requests.get(
url=target url, proxies={'https
proxies={'https': get_ipipgo_proxy()},
headers=random.choice(headers_list),
timeout=12
)
V. Quick questions and answers to frequently asked questions
A: This situation is eighty percent of the use of poor-quality agents, ipipgo static residential package stability is much better, 35 dollars an IP can be used for a whole month!
A: directly to their customer service to enterprise customization program, can adjust the billing according to the volume of business, than pay by volume cost-effective!
A: first check the whitelist settings, ipipgo background to bind the local IP. if you go account password authentication, remember the format is user:pass@ip:portVI. Why do you recommend ipipgo?
English 
简体中文 
Español 
Deutsch 
Français