Beautiful Soup Tutorial: Python Parsing Guide

First, why use proxy IP with Beautiful Soup?

Guys who have engaged in data crawling know that the website anti-climbing mechanism is now more and more strict. You take an ordinary IP to grab data, light speed limit, heavy direct seal. At this timeproxy IPIt becomes a life saver - especially like ipipgo, a service provider specializing in high stash proxies, each request for a different IP, the site can't tell if you're a real person or a crawler.

To cite a real scenario: you want to catch the price of an e-commerce platform, with their own home broadband connected to the request 50 times, the results of the third time on the seal. Switch to ipipgo's dynamic proxy pool, each request randomly switch the country's different regions of the IP, the success rate directly pull to 95% or more.

import requests
from bs4 import BeautifulSoup

proxies = {
  'http': 'http://username:password@gateway.ipipgo.com:9020',
  'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('https://target-site.com', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')

Second, configure the proxy IP of the three major pitfalls

The easiest place for a novice to fall:

1. Wrong authentication method: ipipgo's proxy requires dual authentication with account and password, and many people leave out the authorization parameter in the code

2. Protocol mismatch: Accessing a https site but using a http proxy is like taking a bus card and using a subway gate.

3. IP Survival Time

Now on the market agent service providers of varying quality, some claimed millions of IP pools, the actual availability of less than 30%. ipipipgo is mainly interested in theirSurvival detection mechanismThe system automatically eliminates failed nodes every minute. Measured continuous crawling for 6 hours, the number of request interruptions does not exceed 3 times.

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.7113.93 Safari/537.36',
    'Accept-Language': 'en-US,en;q=0.5'
}

for page in range(1, 100): 'Accept-Language': 'en-US,en;q=0.5' }
    time.sleep(random.uniform(1, 3)) random wait
    response = requests.get(f'https://xxx.com/page/{page}', headers=headers, proxies=proxies)