Web Resolver: Web Resolver + Proxy IP Integration Solution

Teaching you how to "buff" a web parser with a proxy IP

Anyone who has ever done data collection knows that web parsers are afraid of encounteringIP blocked. It's like being banned from a game and not being able to get the data. This time you need to give the parser a "vest" - proxy IP. we do not talk about false today, directly on the hardcore operation guide.

Why do I have to use a proxy IP?

To give a chestnut: a treasure commodity price comparison tool, if you only use your own broadband IP to catch data, half an hour quasi-blocked. With a proxy IP is likeseventy-two changesIf you change your IP address for each request, the site won't even recognize who you are.

Three-step access to proxy IP

Here's an example of ipipgo's API, his proxy poolSurvival rate can be as high as 99%.This is a good example of how to use Python. Code samples written in Python, other language principles are similar.

import requests

 Proxy configuration from ipipgo
proxy = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:9020',
    'https': 'http://用户名:密码@gateway.ipipgo.com:9020'
}

 Request with proxy
response = requests.get('destination URL', proxies=proxy, timeout=10)
print(response.text)

pay attention toRandom IP switching! It is recommended to use ipipgo's dynamic proxy service, they automatically assign a different exit IP per request, which is much less work than switching manually.

Anti-blocking Practice Tips

1. Ask the frequency not to jerk off.: Even if you have a proxy, don't swipe it to death, set a random interval of 3-5 seconds.
2. Camouflage should be in place: Remember to rotate User-Agents, don't use the same one all the time!
3. Abnormalities to be monitored: When encountering CAPTCHA or 403, immediately cut the IP, do not head iron hard to fight!

QA First Aid Kit

Q: What should I do if my proxy IP is slow?
A: Go with ipipgo'sBGP High Speed LineThe delay can be reduced to 200ms. Be careful to check the timeout setting in the program, don't set it too short.

Q: What should I do if I always encounter CAPTCHA?
A: two programs: 1. change ipipgo high stash proxy (transparent proxy easy to expose) 2. on the automated coding service two-pronged approach

Q: What if I need an overseas IP?
A: ipipgo has 20+ country nodes to choose from, so remember to select theNative IP Poolpackages, those server room IPs are easy to recognize

Guide to avoiding the pit

Don't use free proxies for cheap! Blood lesson: I tested a free proxy before, and the result was50%IP are black.I'm not sure if you're going to be able to get a good deal on your own. Professional things to professional people to do, ipipgo this kind of formal service providers have IP quality monitoring, with a solid.

One last piece of cold knowledge: when setting up a proxy remember toTurn off the system agent.The first thing you need to do is to get your hands on a proxy! A lot of newbies planted in this, the program to go proxy, but the system is still using the local IP, equivalent to the work for nothing. What do not understand, go directly to the official website of ipipgo technical customer service, the reply is quite fast not ink.