Python Parsing HTML: Python Proxy Parsing HTML in Action

When the crawler meets the anti-climbing, the proxy IP is the real brotherhood

Engaged in the data crawl know that the site is now very fine. The same IP request frequently, light speed limit, heavy seal. Last week an e-commerce friend touted, they use ordinary IP to catch the price of competitors, half a day was blocked more than a dozen times. At this time, we have to offer up the proxy IP this magic weapon, especially like ipipgo such can provideDynamic rotation of IP poolsof service providers.

import requests
from bs4 import BeautifulSoup

proxies = {
    'http': 'http://用户名:密码@proxy.ipipgo.cc:端口',
    'https': 'http://用户名:密码@proxy.ipipgo.cc:端口'
}

response = requests.get('destination URL', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')
 Here's where the parsing logic comes in...

Three Tips to Teach You to Play with Agents + Analysis

Tip #1: Dynamic IP rotation
With ipipgo's dynamic residential package, each request automatically change IP. test an e-commerce platform, a single IP to support up to 20 requests, with a dynamic IP after 200 consecutive times did not trigger the wind control.

Tip #2: Have a full set of disguises
It is not enough to just change the IP, remember to bring a random User-Agent, here we recommend fake_useragent library, and proxy IP with better results:

from fake_useragent import UserAgent

headers = {'User-Agent': UserAgent().random}
response = requests.get(url, headers=headers, proxies=proxies)

Tip #3: Don't be lazy about exception handling
When encountering 403/503 status code, don't be tough. Setting up a retry mechanism + automatic IP switching is the right solution:

retries = 3
for _ in range(retries):
    try: response = requests.get(url, proxies=proxies, timeout=10)
        response = requests.get(url, proxies=proxies, timeout=10)
        if response.status_code == 200: if response.status_code == 200: if response.status_code == 200
            if response.status_code == 200: break
    except.
         Here we call the ipipgo API to change the IP address.
        update_proxy()

A practical guide to avoiding the pit

Old Driver QA Time

Q: Proxy IPs are not working when I use them?
A: Choose ipipgo's exclusive static package, a single IP can be used for 1 month. If you use dynamic package, remember to set the auto change frequency, their API supports changing IP by time/times.

Q: How can I improve the efficiency of data collection?
A：两个路子：1）上多线程，每个线程配不同代理 2）用ipipgo的TK专线，能压到200ms以内。

Q: Which ipipgo package is the best deal?
A: Use Dynamic Residence Standard Edition ($7.67/GB) for small-scale collection, choose Enterprise Edition Dynamic Package for enterprise-level business, and choose Static Residence at $35/month for those who need fixed IP.

I'll tell you what's on my mind.

Proxy IP this thing, stability is ten times more important than the price. I've used others before for cheap, and often encountered problems with high duplication of IP pools and slow response. ipipgo has a cold but useful feature - theFilter IPs by country cityIt's a great way to get the most out of your data collection. Their customer service can help write a customized collection program, suitable for lazy novice.

Lastly, I would like to remind you that using a proxy is not a gold medal, and you need to cooperate with request frequency control and request header camouflage in order to maximize the effect. When you encounter a particularly difficult website, directly on their cloud server business, local deployment of proxy nodes is more worrying.