BeautifulSoup crawler crash course: e-commerce data collection

First, white people can also play the reptile introductory position

Want to get into e-commerce data but don't know how to program? Don't panic, let's use Python'sBeautifulSoupThis artifact is a three-legged effort to get started. Load these two first:Requests LibraryResponsible for catching web pages.beautifulsoup4Responsible for disassembling the data. Remember the install command:

pip install requests beautifulsoup4

For example, if you want to catch the price of a certain commodity, the skeleton of the code is probably long like this:

import requests
from bs4 import BeautifulSoup

url = 'https://某电商网站/product/123'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
price = soup.find('span', class_='price').text
print(f'Current price: {price}')

Why use a proxy IP, which is a life preserver?

A lot of newbies fall intoIP blockedOn this matter. E-commerce sites are very smart, the same IP crazy request, minutes to blacklist you. This is the time to rely onipipgoThe proxy IP service to renew the life, the principle is like guerrilla warfare - each request changes to a different IP address.

Highlight it three times:Don't use free agents! Don't use free proxies! Don't use free proxies!Those contraptions are either slow as snails or have long since been marked as blacklisted by websites. With ipipgo's exclusive proxy pool, every IP is guaranteed to be freshly baked.

Third, the correct way to open the proxy IP

Take ipipgo as an example, after getting the API interface, get a new IP before each request. focus on thetimeout settingrespond in singingException handlingThe code is changed this way:

proxies = {
    'http': 'http://用户名:密码@ipipgo proxies:port',
    'https': 'http://用户名:密码@ipipgo proxy address:port'
}

try.
    response = requests.get(url, proxies=proxies, timeout=10)
except Exception as e.
    print(f'Request failed, change to next IP: {str(e)}')

Fourth, the actual combat: capture commodity details

Target the structure of an e-commerce page and use the developer tool (F12) to find the HTML tags for price, inventory, and other data. For example, find the price hidden in the<div class="”product-price”">In it, the code is written like this:

price_tag = soup.select_one('div.product-price')
if price_tag.
    current_price = price_tag.text.strip().replace('¥','')
else: current_price = price_tag.text.strip('¥',')
    print('The price tag may have been revamped!)

Remember to userandom hibernation(time.sleep(1~3 seconds)) to simulate the operation of a real person, don't swipe wildly like a robot.

v. guide to demining common problems

Q: What should I do if the proxy IP suddenly fails to connect?
A: First check if the account password is correct, then try to manually ping the proxy address. If ipipgo shows normal IP in the background, it may be a temporary jerk of the target website.

Q：Data capture back is garbled?
A: add in requests.get()response.encoding = 'utf-8', or adjusted according to the charset in the page source code.

Q: How can I tell if a proxy is in effect?
A: Visit https://httpbin.org/ip to see if the IP returned is a proxy address.

VI. Hidden benefits of ipipgo

They have a family.Intelligent SwitchingThe features are pretty hassle-free, and it automatically changes to a new one when it encounters an IP block. Recently also came outvolumetric billingpackage, especially friendly to small-scale collection. Newbies are advised to practice with the experience package first, and then go on the large traffic package after familiarizing themselves with it.

The last nagging sentence: do data collection to speak of virtue, don't make other people's websites collapse. Control the frequency of requests, don't be stingy when it comes to using proxies, after all!ipipgoThe agents are not expensive, and it's a real loss if you get banned.