Web Crawling with Selenium: Browser Automation Tutorials

Teach you to use Selenium to catch web pages, proxy IP so play to be stable!

Recently, a brother always asked me, using Selenium to engage in data always be blocked IP how to do? Today we will nag about this matter. First of all, let's be honest, now the site is very smart, you take a local IP repeatedly brush, not block you block who? This time we have to invite our savior - proxy IP.

Why do I have to use a proxy IP?

For example, if you go to the same kiosk every day to buy cigarettes, the boss will remember you on the third day, right? Proxy IP is like changing different clothes every day to buy cigarettes, the boss simply do not recognize. Especially with Selenium this kind of real browser operation, with dynamic IP, the site is basically no way.

Here's a good one.ipipgoHis IP pool is as big as the Pacific Ocean, with more than 300 regions around the world to choose from. The most important thing isDedicated IP not duplicated, using the special peace of mind, my own project real test continuous running 72 hours without problems.

Selenium Configuration Proxy in Three Steps

from selenium import webdriver

proxy = "123.123.123.123:8888" proxy address provided by ipipgo
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'--proxy-server=http://{proxy}')

 Remember to replace the password with your own ipipgo account password
chrome_options.add_argument('--proxy-auth=username:password')

driver = webdriver.Chrome(options=chrome_options)
driver.get("http://example.com")

Note that there are two potholes here:
1. Don't use free proxies! Not to mention the slow speed, 90% are all leftover IPs used by others.
2. Accreditation informationDon't make a mistake.If you have a key generator in the background of ipipgo, we recommend you to copy and paste it directly.

Real-world anti-blocking techniques

It's not enough to have an agent, you have to learn the combinations:

Frequently Asked Questions QA

Q: What should I do if the proxy IP is invalidated after using it?
A: This situation is eighty percent of the use of public proxy pool, it is recommended to change to ipipgoExclusive IP packageThe IP address of each IP with independent authentication, stability is directly pulled full.

Q: The code runs and prompts a certificate error?
A: Try adding this parameter to options:
chrome_options.add_argument('--ignore-certificate-errors')

Q: How can I tell if a proxy is in effect?
A: Visitshttp://ipipgo.pro/checkipThis page, you can see the current use of the export IP, pro-test effective.

Upgraded Play: Intelligent Switching Strategies

Sharing a high level configuration for the guys, using ipipgo's API for automatic IP changing:

import requests

def refresh_proxy(): resp = requests.get("")
    resp = requests.get("https://api.ipipgo.pro/getproxy")
    return resp.json()['proxy']

 Automatically change the IP address for every 20 page captures
if count % 20 == 0.
    driver.quit()
    current_proxy = refresh_proxy()
     Reinitialize the browser...

This program is tested to capture millions of data no problem, the key is to choose the right proxy service provider. I've used seven or eight providers, but in the end, it's stillipipgoThe most reliable, the response speed is basically within 200ms, and the drop rate is less than 0.1%.

Finally, the proxy IP is not a panacea, with a reasonable request frequency and page resolution strategy. What specific problems welcome to ipipgo official website to find technical customer service nagging, they are 24 hours online, more professional than me this wild road.