Python HTML Parser: Python Parsing HTML

What to do when a crawler meets a counter-crawler? Try this combo

You do data crawl old iron must have encountered this situation: just write a good crawler script, running suddenly run by the target site blocked IP. At this time, do not be in a hurry to smash the keyboard, we want to talk about today'sProxy IP + HTML parsingA combination of punches that specializes in all kinds of anti-climbing quandaries.

HTML parsing three big killers how to choose

There are so many libraries for handling HTML in Python, let's focus on the three most useful ones:

I usually like to use BeautifulSoup + lxml the golden pair, both to ensure the speed of parsing, writing and smooth. Take a chestnut:

from bs4 import BeautifulSoup
import requests

 Remember to replace the proxies with ipipgo proxies here
proxies = {
  'http': 'http://username:password@gateway.ipipgo.com:9020',
  'https': 'https://username:password@gateway.ipipgo.com:9020'
}

resp = requests.get('destination URL', proxies=proxies)
soup = BeautifulSoup(resp.text, 'lxml')
title = soup.find('h1', class_='title').text

The right way to open a proxy IP

A lot of newbies tend to fall into the pit is the proxy configuration, and here's where to draw the line:

1. Don't confuse the authentication information: The username and password for ipipgo should be clearly stated in the proxy address.
2. The protocol should correspond: Proxy addresses for http and https should be configured separately.
3. Timeout settings can't be beat: It is recommended to add timeout=10 to requests.

Here's a recommendation for ipipgoDynamic Residential AgentsThe IP survival rate of their home can reach more than 95%. Especially when doing e-commerce data collection, with their static residential agent, an IP can be used for 24 hours without turning over.

A practical guide to avoiding the pit

Recently a friend doing cross-border e-commerce came to me for help, they used a normal proxy to catch Amazon data which was always blocked. Then they switched to ipipgoIntelligent Rotation Agents, with the following code structure, the problem is solved:

import random
from itertools import cycle

 Proxy pool from ipipgo
proxy_pool = [
  'http://user:pass@gateway.ipipgo.com:9020',
  'http://user:pass@gateway2.ipipgo.com:9020', ...
   ... More proxy addresses
]

proxy_cycle = cycle(proxy_pool)

for page in range(1, 100):
    current_proxy = next(proxy_cycle)
    current_proxy = next(proxy_cycle)
        resp = requests.get(url, proxies={'http': current_proxy}, timeout=8)
         Parsing Logic...
    except Exception as e.
        print(f "Failed with {current_proxy}, move to the next one!")

Common QA for white people

Q: Why am I still blocked when I use a proxy?
A: The probability is that the quality of the agent is not good, free agents are basically into the site blacklist. It is recommended to use ipipgo this kind of professional service provider, they update ten million IP pool every day!

Q: Do I need to maintain my own agent pool?
A: No need at all! ipipgo's background will automatically filter invalid IPs, and you can also customize the export nodes according to the region, which is much less troublesome than doing it yourself!

Q: How do I break the CAPTCHA when I encounter it?
A: This is going to be on ipipgo'sHigh Stash Agents+ request frequency control now. Suggest adding random.sleep(random.uniform(1,3)) to the code to simulate a real person operation

Say something from the heart.

搞数据抓取这行，代理IP就像战士的防弹衣。用过七八家服务商，最后长期续费的还是ipipgo。他们家有两点特别戳我：一是客服响应快，半夜三点提工单都有人回；二是API设计简单，拿来就能直接塞代码里用。最近看官网在做618活动，新用户首月才9.9，想试水的可以去。

A final reminder for newbies:Don't save money on proxy IPs! Cheap shared proxies look like a good deal, but the actual time wasted is enough to buy ten years of VIP. Choose the right tool to get twice the result with half the effort, don't you think so?