Python Beautiful Soup library guide: HTML parsing in action

When reptiles meet HTML: don't crash into the South just yet!

Engaged in the old iron of the network crawler must have encountered this situation: it is difficult to write a good script, suddenly the target site has been pulled black. At this time you need two brushes - theBoth parsing web content and learning to protect yourselfThe first thing we're going to talk about today is that Beautiful Soup is like a Swiss Army knife. The Beautiful Soup we're talking about today is like a Swiss Army knife, specializing in all kinds of HTML messes.

Let's start with the role of proxy IPs in this play. Suppose you want to visit a certain website continuously to check the data, using a fixed IP is like wearing a fluorescent suit to track - exposed in minutes. This timeProxy pool for ipipgoIt comes in handy, as if you have hundreds of cross-dressing props for you.

 As a chestnut, use requests with a proxy
import requests
from bs4 import BeautifulSoup

proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('https://target-site.com', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')

HTML Anatomy Lesson: Don't Get Wrapped Up in Tags

Beautiful Soup is the most awesome place is that the mess of HTML can be cleaned up in a satisfactory manner. Let's look at a few common tricks:

1. Finding things is like checking the water meter: Using find() and find_all() is like holding a search warrant, and the CSS selector is your GPS navigation. For example, if you want to grab all item prices:

price_tags = soup.select('.product-price')
for price in price_tags.
    print(price.get_text())

2. Don't miss the attribute values: When you encounter an image or link, remember to pull out the href or src. Give an example of grabbing an image:

images = soup.find_all('img')
for img in images: print(img['src'])
    print(img['src']) Note that you may get an error here, so remember to add an exception!

The right way to open a proxy IP

Here's the kicker! A pitfall that many newbies tend to step into:

Recommended for ipipgoIntelligent switching modeThe API will automatically assign available IPs, and it has been tested, and the continuous collection has not been blocked for 3 hours, which is much more reliable than some proxy services that are sold on the market.

Troublesome maneuvers in the real world

Don't panic when you run into a CAPTCHA, try these tricks:

1. Reduce the frequency of requests and randomize sleep with the random module
2. Switch the User-Agent to a different browser.
3. Immediately change ipipgo's alternate IP in the event of a ban.
4. Important data are collected in batches, so don't try to get fat in one go.

 Example of masquerading as a browser
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
response = requests.get(url, headers=headers, proxies=proxies)

question-and-answer session

Q: Why use ipipgo instead of a free proxy?
A: Free proxy is like a public toilet, anyone can use it but the hygiene is not guaranteed. ipipgo's exclusive proxy is like your own bathroom, clean and hygienic.

Q: What should I do if I encounter dynamically loaded data?
A: You can work with Selenium, remember to hang proxy for Selenium as well. ipipgo supports socks5 protocol, which is suitable for this scenario.

Q: How can I tell if an IP is exposed?
A: Regularly visit http://httpbin.org/ip检测 and if the IP returned does not match what is expected, hurry up and change ipipgo's new IP.

One last rant, be a crawler and be martial. Don't screw up other people's sites with ipipgo'sIntelligent QPS controlFunctionality, both to ensure efficiency and improper network hooligan. Encountered a difficult website, their technical customer service can also give customized solutions, this point is really more thoughtful than peers.