Python crawler combat: BeautifulSoup fast crawl web data

Hands-on teaching you to use proxy IP to avoid anti-climbing traps

Recently, a number of old iron to do data capture with me complained that the use of Python's BeautifulSoup to capture data is always blocked by the site's IP. this thing, it is the same as playing the game was banned number a reason -The site is monitoring that you're sending too many requests in a short period of time.The first thing you need to do is to get a proxy IP to disguise your real identity. At this time it is necessary to rely on proxy IP to disguise the real identity, ipipgo home dynamic IP pool tested to be able to carry a continuous 8 hours of high-frequency requests.

First of all, to the novice popularize a cold knowledge: many websites anti-climbing mechanism will count the frequency of access to a single IP. When you use your own broadband connected to send a request, not half an hour quasi blacklisted. Last year, there is an e-commerce comparison of buddies, because there is no hanging proxy, the company network IP to get blocked for three days, the boss almost let him compensate for the broadband fee.

Practical configuration of proxy IP tao operation

Start by loading the essential three-piece suit:

Here's the kicker! The proxy service with ipipgo has to be configured like this:

import requests
from bs4 import BeautifulSoup

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
}

headers = { 'User-Agent': 'Randomly generated UA'}
response = requests.get('Target URL', proxies=proxies, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

Here's a pitfall to watch out for:Remember to use urllib.parse.quote if your password contains special symbols.I've had a brother who couldn't connect to the proxy without the @ symbol being encoded. I had a brother who had the @ symbol not transcoded and couldn't connect to the proxy, and it took two hours of troubleshooting to find the problem.

An advanced play on dynamic IP rotation

Relying on a proxy IP alone is not stable enough, you have to learn to playIP pool rotationThe API interface of ipipgo can directly get the latest IP list, use this script to realize the automatic switch:

import random

def get_ip_list().
     Call the ipipgo API to get the latest IP pool.
    return [
        '111.222.33.44:8000',
        '112.233.45.67:8080', ...
         ... Other IPs
    ]

current_ip = random.choice(get_ip_list())

It is recommended to set every 30-50 requests to change the IP, so that it is not easy to trigger anti-climbing, but also to maintain the collection efficiency. Tested with this method, an e-commerce site to collect 30,000 consecutive commodity data are not overturned.

A newbie's guide to avoiding the pitfalls

1. Don't use free proxies for cheapNine out of ten of those public free IPs are pits, either slow or pulled by the site long ago.
2. HTTPS site must be matched with the https protocol agent, protocol mismatch will report SSL errors
3. 403 error first check the User-Agent has not been randomly switching
4. Important data collection is recommended with ipipgo's exclusive IP package, the stability of direct pull full

Frequently Asked Questions QA

Q: What should I do if my proxy IP is slow?
A: Pick the node that is close to the target server. For example, if you collect websites in North China, choose ipipgo's Beijing server room node.

Q: How can I tell if a proxy is in effect?
A: Use requests.get('http://httpbin.org/ip') to see if the returned IP address has changed

Q: What should I be aware of when starting multiple crawler threads at the same time?
A: Each thread should be assigned a different proxy IP, it is recommended to use ipipgo's concurrent authorization package, which supports multiple threads to fetch different IPs at the same time

Q: Can I use the blocked IP again?
A: Ordinary proxy IP is blocked need to wait 24 hours, ipipgo's high-quality proxy pool will automatically filter the invalid IP, real-time update of available resources

Finally, a piece of advice: don't save money on proxy IP! I've seen people buy low-quality proxies on the cheap before, and as a result, the data collected was mixed with competitors' induced information, which led to the company's marketing strategy to make a complete mistake. With ipipgo's enterprise-level proxy, there are special people to do IP quality verification, which can save a lot of late data cleaning trouble.