Web page data automatically imported into Excel: web page data agent + Excel export

How much trouble is it to manually import web data?

Guys who have engaged in data processing know that manually copying web forms is a disaster. Especially e-commerce price monitoring or industry statistics such work, moving from dozens of pages to pick up data. Last week my colleague Wang because of frequent refreshing of a wholesale website, the result of the IP directly blocked - this unlucky child hard squatting in Starbucks to rub public WiFi to finish the job.

The Three Axes of Automatic Grabbing

To save time and effort, you must take care of these three pieces:Web Crawler + Proxy IP + Excel AutomationThe first thing you need to know is that you can't get a good deal of information about the website. Here is a pitfall to pay attention to: many sites are particularly sensitive to frequent visits, just like your downstairs kiosk owner, always keep an eye on the familiar customers who often come to buy instant noodles.

import requests
from bs4 import BeautifulSoup
import pandas as pd

 Example of proxy setup
proxies = {
    'http': 'http://用户名:密码@ipipgo proxies:port',
    'https': 'http://用户名:密码@ipipgo proxy address:port'
}

response = requests.get('destination URL', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')
 Here's the data parsing code...

How to choose a reliable proxy IP?

There are all kinds of agency services on the market, but we have to recognize three hard indicators:

I have to settle for ipipgo's high stash of proxies here.Dynamic rotation mechanismIt's really good. Last time I used their service to grab the data of a certain platform for 3 days in a row, and it didn't trigger the anti-climbing mechanism - it's like wearing a cloak of invisibility.

A guide to avoiding the pitfalls of Excel automation

Data guide Excel most afraid of encountering coding problems, share a universal code template:

 Data export section
data = {'Title': [], 'Price': [], 'Inventory': []} Modified as appropriate

 Populate the data...
df = pd.DataFrame(data)
 Solve Chinese garbled code
df.to_excel('data report.xlsx', index=False, engine='openpyxl')  

If you can't open the exported file, nine times out of ten, it's not installed.openpyxllibrary, remember to hit the command line with apip install openpyxlIt's done.

Frequently Asked Questions QA

Q: Why is it still blocked after using a proxy?
A: Mostly the quality of the proxy is not good, ipipgo's exclusive proxy pool is updated frequently, we suggest trying their commercial packages.

Q: What should I do if the data capture is always interrupted?
A: Add atry-exceptException catching, with ipipgo's automatic node switching feature, remember to set the timeout in the code:

response = requests.get(url, proxies=proxies, timeout=30)

Q：Exported Excel data misalignment how to do?
A: Check if the web page element contains merged cells by using thepandasRemember to specify theheaderParameters.

Practical advice for newcomers

1. Start with ipipgo'sFree Trial PackagePractice, their family gets 1G of traffic for new users
2. Remember to add important datatry... . finallyhandle sth. exceptionally
3. Regularly clean up cookies, just like taking out the garbage every day to make it a habit
4. Complex pages are prioritizedSelenium (computing)+ Agent's program

Lastly, I would like to say one last thing: data collection should be done in a long term, don't just grab it like a rash person. With ipipgo's intelligent scheduling strategy, set a reasonable collection interval, in order to efficiently and safely handle the data into the database. Recently found that their control panel addedSuccess rate monitoringfeature, which is particularly helpful for debugging programs, is worth a try.