Beautifulsoup Documentation: The Official Manual

When Crawler Meets Beautifulsoup

Engaged in network crawler brothers understand that data capture is most afraid of encountering web page structure is as complex as a maze. This is the time to sacrifice Beautifulsoup this weapon, it is like a smart locksmith, can be arranged in a clear web page tags. However, only will parse the page can not be enough, if the site to give you an IP ban, and then powerful parsing tools have to rest.

import requests
from bs4 import BeautifulSoup

 Remember to replace the ipipgo proxies with the following configuration
proxies = {
    'http': 'http://username:password@proxy.ipipgo.com:9020',
    'https': 'http://username:password@proxy.ipipgo.com:9020'
}

response = requests.get('destination URL', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')

The right way to open a proxy IP

Many newbies are prone to make the mistake of writing dead IP addresses directly in the code. This is not only easy to be blocked, but also a waste of resources. Use ipipgo's dynamic proxy pool is the proper way, their family'sAutomatic IP Rotation FunctionEspecially good for long crawling tasks. Remember the three key points:

Pitfalls and countermeasures in practice

Last week, a customer crawled an e-commerce site with an ordinary IP, just ran for half an hour and was blocked 20 IPs. after changing to ipipgo's high stash of proxies, it ran for three days in a row and was fine. Here is a little trick: in requests.Session() configure the proxy, than a single request to set more trouble.

session = requests.Session()
session.proxies.update({
    'http': 'http://user:pass@proxy.ipipgo.com:9020',
    'https': 'http://user:pass@proxy.ipipgo.com:9020'
})

Frequently Asked Questions First Aid Kit

Q: Why is it still blocked after using a proxy?
A: Check if you are using a transparent proxy, ipipgo'sHigh Stash AgentsWill completely hide the real IP

Q: Do I need to maintain my own IP pool?
A: No need at all, ipipgo's API can return a list of available IPs, remember to set the automatic switching interval

Q: What about HTTPS sites?
A: In the proxy configuration https and http should be written, some sites will be mixed loading resources

Why ipipgo?

It's not for nothing that I tried 7 or 8 proxy providers and finally locked in on ipipgo. Theirs.Dedicated bandwidthThe design is especially suitable for projects that require stable connections, unlike shared proxies that can't move without dropping the line. There is also a hidden benefit - technical support response is super fast, three o'clock in the middle of the night to raise a work order actually someone back!

The recently discovered new feature is even better: setting up directly in the backendIP whitelistingThe first is that you don't have to enter your password every time. For projects to be deployed to the server, security is directly upgraded by two notches. But remember to regularly update the access credentials, this no matter which one you use can not be lazy.

The last nagging sentence of the truth: tools and then cattle also have to see how to use. I have seen someone open ipipgo 100 megabyte proxy, the result is too high because of the frequency of crawling by the target site to pull black. Reasonable set request interval + quality proxy, is the king of sustainable crawling.