BeautifulSoup Web Crawl: Python Parsing Dynamic Pages

The earthy way to pick apart dynamic web pages and make sense of them

Engaged in web crawling know that many sites are now learning fine, data loading with the trick like. With traditional requests + BeautifulSoup combination to catch, often catch a lonely - the page on the hairline data are not. This time it is necessary to use somedishonest practices, such as leaving the browser kernel on to simulate a real person's actions.

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get('https://目标网站')
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
 Here's where you start your show...

But it's easy to get caught by websites playing this way, and that's when we have to bring out ourlife-saving device--ipipgo's proxy IP service. Their home IP pool is large enough that the site can't tell if you're a person or a machine if you change your vest with each request.

Putting an invisibility cloak on a reptile

Here's a trick for you to configure the crawler with ipipgo's proxy service. For example, if you use the requests library, you can do this:

import requests

proxies = {
    'http': 'http://用户名:密码@ipipgo proxies:port',
    'https': 'https://用户名:密码@ipipgo proxy:port'
}

response = requests.get('destination URL', proxies=proxies, timeout=10)

Here's the kicker! ipipgo's proxies areThree packagesOptional:

Dynamic Page Crawl

When you come across the kind of site that you have to scroll down to load, you have to use a browser automation tool in conjunction with a proxy. Here's an example using selenium:

from selenium.webdriver import ChromeOptions

options = ChromeOptions()
options.add_argument('--proxy-server=http://ipipgo代理地址:端口')
driver = webdriver.Chrome(options=options)
 The rest of the process is the same as normal

Remember to put in the backend of ipipgowhitelisted IPSet it up so that authentication doesn't jam the proxy. If you get a captcha block, reduce the frequency of requests appropriately, or try switching to ipipgo's high stash package.

Frequently Asked Questions QA

Q: What should I do if I keep getting my IP blocked by websites?
A: Use ipipgo's rotating proxy pool to change different exit IPs for each request. their IP pool is updated every day, and automatically changes to a new one when it's closed.

Q: How do I break a website that requires a login?
A: It is recommended to use ipipgo's long-lasting static IP to keep the login status uninterrupted. Remember to set the cookie expiration date, don't let the session expire.

Q: Do free proxies work?
A: Never! Nine out of ten free proxies are either slow or have been hacked by the website. ipipgo's paid proxies have been verified at the enterprise level and are much more reliable.

As a final rant, dynamic page capture is a cat and mouse game. The key is toSimulation of real-life behaviorThe first thing you need to do is to use ipipgo's proxy service to catch data. With ipipgo's proxy service, grabbing data is just like strolling in your own backyard garden, you want to stroll as much as you want. They recently put on a new mixed dialing package, the measured capture success rate can be 98% or more, it is worth a try.