Python parsing XML: Python proxy XML parsing

Hands-on teaching you to use Python to parse XML when hanging proxy

Recently a lot of data collection brothers asked, with Python parsing XML when the target site is always blocked IP. this thing I did last year when the e-commerce price comparison system also encountered, then used a stupid way - every 200 times to parse a new IP. later found that with ipipipgo's proxy service can be directly dealt with today! Today, I'm going to share my practical experience with you.

import requests
from lxml import etree

proxies = {
    'http': 'http://用户名:密码@proxy.ipipgo.cc:9020',
    'https': 'http://用户名:密码@proxy.ipipgo.cc:9020'
}

response = requests.get('Target site', proxies=proxies)
xml_data = etree.fromstring(response.content)

watch carefullyProxies dictionaryThe writeup here uses the account verification method provided by ipipgo. Their proxy server address with .cc domain name, don't get confused with those unreliable merchants. I have tested, with this configuration, continuous running for 8 hours without a verification code.

Three Great Uses for Proxy IP in XML Parsing

1. anti-blocking: Last year, when climbing an automobile website, using a single IP to parse the XML quote data, 10 minutes to be blocked. Later, I hung up ipipgo's rotating proxy and cut 3 IPs per second, and I was able to survive the whole promotion season.

2. geographic positioning: The XML data of some websites will show different content by region. For example, the price of a product parsed by Shanghai IP may be 50 dollars cheaper than that seen by Chengdu IP.

3. Breaking the Frequency LimitFor example, the seat information interface of a ticketing website can only be resolved 50 times per hour by a single IP. Using a proxy pool can magnify this limit by a factor of N.

Practical skills: proxy IP tuning program

Here's the kicker.Exception Handling for Dynamic IP: Add a proxy reconnect mechanism in the try-except block. I had a project where I wrote this and the parse failure rate dropped from 12% to 0.7%

try.
     XML parsing code
except etree.XMLSyntaxError:
    requests.get('http://ip.ipipgo.cc/release_ip?key=你的密钥')
     Immediately release the current problem IP

Frequently Asked Questions Q&A

Q: What should I do if my proxy IP suddenly fails?
A: It is recommended to add heartbeat detection in the code and ping ipipgo's verification interface every 5 minutes. They have remaining traffic alerts in their API return, which makes it easy to renew in advance

Q: Encountering XML interfaces that require certificate validation?
A: In requests requests request plus verify=False parameter, at the same time remember in ipipgo background open HTTPS proxy support. Last year to climb the bank exchange rate data to do so

Q: Does proxy speed affect resolution efficiency?
A: Choose ipipgo BGP line agent, measured delay can be controlled within 200ms. Don't be greedy for cheap overseas nodes, the last time I used a U.S. agent to parse a domestic website, an XML waited 6 seconds!

Lastly, I would like to remind you that the XML parsed User-Agent should be replaced randomly, and the effect is better when used with proxy IPs. Once I forgot to change the UA, although the IP cut 30, but still be recognized crawler behavior. Now I use ipipgo's browser fingerprinting proxy, and I don't have this problem anymore.