XPath with Sibling Nodes: Element Positioning Tips

Practical: Grabbing dynamic data with sibling nodes

Suppose we want to crawl the prices of an e-commerce platform, and the page structure looks like this:

  advertising position
  ￥999 
  time-limited discount

The correct XPath should be:

//div[@class='product']/span[2]

But this is easy to get pitted by ad position changes. It's safer to use sibling node positioning instead:

//span[contains(text(),'¥')]/preceding-sibling::span[1]/following-sibling::span[1]

Why must proxy IPs work with XPath?

expense or outlayipipgoWhen using the proxy service, you often encounter situations where servers in different regions return different page structures. For example:

That's when it's time toDynamically adjusting XPath, use the different regional IPs provided by ipipgo to do structural probing and find the most stable way to locate them.

Three Tips for Avoiding Detection

1. Random Waiting Time: add 0.5-3 seconds random delay before XPath operation
2. hybrid localization: Use both class and sibling node localization
3. IP pool rotation: Use ipipgo's exclusive IP pool to switch to a different exit IP for each request.

 Python Sample Code
from selenium import webdriver
from ipipgo import get_proxy call ipipgo SDK

proxy = get_proxy(region='East China')
options = webdriver.ChromeOptions()
options.add_argument(f'--proxy-server={proxy}')
driver = webdriver.Chrome(options=options)

 Compound location with sibling nodes
price = driver.find_element_by_xpath('//div[contains(@class, "price-box")]//following-sibling::span[1]')

Frequently Asked Questions QA

Q: Why can't I catch data even if I use sibling node positioning?
A: eighty percent of the web page with dynamic loading, first use ipipgo's residential agent to simulate the real user environment, wait for the elements to finish loading and then grab the

Q: How to deal with multi-layer nested structures when encountered?
A: Try combined axis positioning, such asancestor::divbecome man and wifefollowing-siblingIf you can't figure it out, use ipipgo's page structure analysis tool.

Q: What should I do if XPath behaves inconsistently in different browsers?
A: It is recommended to use Chromium kernel fixed, with ipipgo's browser fingerprint management function

Practical advice from ipipgo

When we did technical support for our customers, we found that usingExclusive IP + Intelligent RoutingThe combination of can make XPath positioning success rate increase more than 60%. Especially recommended when doing price monitoring:

1. use ipipgo's East China/North China dual line
2. set up automatic retry mechanism
3. update the XPath rule base once a week

Lastly, many of my peers fall into the trap of theStick to the technology and don't change the IPThe fact is that using the right tools is much more useful than hard code. In fact, using the right tool is more useful than hard code, like ipipgo's intelligent scheduling system can automatically match the optimal node, much more efficient than manual switching. A brother tested, the same XPath script, with a good proxy IP data acquisition can be tripled, this is the reality of the gap.