BeautifulSoup Python Crawl: Web Page Parsing in Action

Teach you to use Python + proxy IP to get the webpage capture

Recently, I was helping a friend with a price comparison site and realized that a lot of platforms are starting to play withIP blockingThe trick. For example, 30 consecutive visits to the IP blocking, so that the data crawl is particularly difficult. This time you need a proxy IP tocover upToday, we will use real-world examples to teach you how to use BeautifulSoup with proxy IP to get the data.

import requests
from bs4 import BeautifulSoup

 Replace this with the proxies provided by ipipgo
proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('destination URL', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')
 The parsing code follows...

Three great scenarios for proxy IP

Many people think that proxy IP can only do crawlers, in fact, there are many uses:

A practical guide to avoiding the pit

Pro-tested to be effective! Be aware of these with ipipgo's proxy service:

1. The request header must masquerade as a browser (User-Agent don't use Python defaults)
2. Randomization of access intervals (don't make it look like a robot)
3. Don't fight with CAPTCHA, change IP and try again!

 Example of disguising browser headers
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36...' , 'Accept-Language': 'Accept-Language'.
    'Accept-Language': 'zh-CN,zh;q=0.9'
}

 Randomize the wait time
import random
time.sleep(random.uniform(1,3))

Frequently Asked Questions QA

Q: What should I do if my proxy IP is not working?
A: It is recommended to use ipipgo's Dynamic Residential Proxy, their IP pool is updated daily with 8 million+, and the stability is quite a bit higher than that of a static proxy, as pro-tested.

Q: What about slow crawling?
A: You can try ipipgo's exclusive bandwidth service with a multi-threaded crawler. But pay attention to the number of threads do not exceed the concurrency limit of the proxy package.

Q: What should I do if I encounter an SSL certificate error?
A: Add verify=False parameter to requests, or ask ipipgo technical support to help troubleshoot proxy configuration.

The doorway to choosing a proxy service

There are a variety of agency services on the market and it is recommended to focus on these points:

• IP survival time (ipipgo's residential proxies last an average of 5 minutes)
• Geographic coverage (they support 200+ country locations)
• Protocol support (HTTP/HTTPS/SOCKS5 are required)

Finally, to remind the newbie: free proxy ten have nine pits, before the free IP to the crawler crashed three times. Now I'm using ipipgo's monthly package with automatic IP replacement, which saves me a lot of heartache. Especially theirIntelligent Routingfunction, can automatically select the fastest node, crawl speed directly doubled.