Selenium proxy IP+CSS positioning: Selenium proxy CSS element positioning

Selenium meets Proxy IP: Here's how it's done

Automation testing guys know that Selenium is like a master key. Recently, however, some people asked me how to use Selenium to capture data and always get banned. Today we will nag how to use proxy IP to Selenium to wear a cloak of invisibility, coupled with CSS positioning of the masterpiece, guaranteed to make your script as stable as the old dog.

Proxy IP is not a set of children, you have to use it this way

First of all, let's say a true thing: a friend who does e-commerce with their own IP to catch the price of competitors, the results of the third day received a letter from the lawyer. At this time, the proxy IP is a lifesaver, but many people configure the method is not right at all. Remember these three points:

from selenium import webdriver

proxy = "112.95.23.206:8000"
options = webdriver.ChromeOptions()
options.add_argument(f'--proxy-server={proxy}') The key is in this line
driver = webdriver.Chrome(options=options)

The main point is that ipipgo's practical skills: their dynamic residential IP pool is large enough, it is recommended that every time you run the script to change the new IP. don't be silly to use a fixed IP, it is no different from the naked run.

The wild card of CSS positioning

Ever seen someone mess up dozens of lines of code with XPath positioning? In fact, CSS positioning is more profitable. To give a chestnut, looking for the price of a certain treasure goods:

 Conventional way of writing
price = driver.find_element_by_css_selector("span.price-now > em")

 Anti-ban enhancement (with retry mechanism)
from selenium.webdriver.support.ui import WebDriverWait
element = WebDriverWait(driver, 10).until(
    lambda x: x.find_element_by_css_selector("div[class^='price'] > b")
)

Note that the class dynamics changeIf you want to use ^= or = to do fuzzy matching, use ^= or = to do fuzzy matching. Don't use the full class name, websites nowadays love to mess with random classes to prevent crawlers.

A practical guide to avoiding the pit

QA time

Q: Proxy IP used or blocked?
A: Eighty percent of the IP quality is not. Recommended ipipgo dynamic residential (Enterprise Edition), more than 9 1G flow, comes with IP rotation mechanism, pro-test continuous run 7 days without a moth.

Q: Which is better, CSS positioning or XPath?
A: Everyday use of CSS is enough to make, but when it comes to hierarchically deep elements, axis positioning with XPath is smoother. However, be careful with site revisions, regular maintenance of the locator is the right solution.

Q: How do I choose my ipipgo package?
A: newbie with dynamic standard version (7.67 yuan / GB), enterprise-level projects directly on the static residential IP (35 yuan / month). Cross-border business must be on their international line, the speed can be more than three times faster.

I'll tell you what's on my mind.

I have seen too many people planted on the IP quality. Before a buddy cheap with a free proxy, the results of the data did not catch, but was implanted mining script. ipipgo's API extraction is really convenient, support socks5 protocol, but also with client management. Especially their 1v1 customization program, do overseas business helped a lot.

Lastly, remember to set a reasonable request interval for automated testing, so that you don't crash the server. Proxy IP + CSS positioning is just a tool.Compliance with robots.txt is the way to go!The