XPath Text Include Queries: Writing Expressions to Pinpoint Web Elements

Hands on text query with XPath to grab data

The most annoying thing about data collection is that the structure of the web page changes, today to teach the guys a trick ---XPath's contains() functionThis is like a GPS for web elements. It's like a GPS for web elements, so you can pinpoint your page even if it's been styled differently.

For example, if you want to capture the price of a product but find that the price element in the web page does not have a fixed id, use the//span[contains(text(),'¥')]Just catch all the tags with the ¥ symbol, no matter what nook or cranny it's hidden in.

//div[contains(@class,'price-box')]//span[contains(text(),'¥')]

Proxy IP and XPath how to get laid

When you're using crawlers to frantically grab data, website fire prevention is no picnic. That's when it's time to bring out ouripipgo Dynamic Residential ProxyIts IP pool is bottomless, and with XPath's fuzzy matching, the collection efficiency is directly doubled.

For example, if you want to grab the price of a certain e-commerce company's national division, use ipipgo'sCity-level location agentsToggle different regional IPs and use this XPath again to catch prices:

//[contains(@id,'price_') and contains(text(),'discounted price')]

Remember to configure ipipgo's proxy authentication in the code:

import requests
proxies = {
  'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
  'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
}
response = requests.get(url, proxies=proxies)

A practical guide to avoiding the pit

Three common mistakes newbies make:

Focus on the third point, when the site has an anti-crawl mechanism, remember to integrate ipipgo in the XPath collection code of theAutomatic proxy switchingThis way, each request is exported from a different IP, which has been tested to be effective in preventing blocking.

question-and-answer session

Q: What can I do if I can't find the element in XPath?
A: First check if the content is dynamically loaded, this case has to work with Selenium, remember to configure ipipgo's proxy in Selenium, his family supports theBrowser-level proxy settingsThe

Q: What should I do if my proxy IP keeps timing out?
A: 80% of them are using spam proxies. ipipgo's exclusive proxy lines are5-second guaranteed connection, and the real-world rush hour is as steady as an old dog.

Q：What should I do if I need to collect overseas websites?
A: ipipgo's overseas server room agent covers 50+ countries, with XPath's multi-language text query, for example, to find the price of the German website with thecontains(text(),'€'), a catch.

Lastly, XPath is something you need to practice more before you are familiar with it. Don't be hard on complex pages, use contains() flexibly with our ipipgo'sIntelligent Resolution AgentThe first thing you need to do is to get your hands on a website that you can't afford to lose. Code to write tired to his home background grip a free trial, new users to send 5G traffic package it.