Web Crawling with PythonBeautifulSoup: Practical Examples

First, why use a proxy IP to engage in web crawling?

The old iron engaged in network crawlers must have encountered such a situation - just grabbed two pages of data, the site will be your IP blocked. At this time do not be silly with their own real IP hard just, with a proxy IP is the king. To give a chestnut, like playing the game to open a small number, was blocked to change a number to continue to play, proxy IP is the reason.

We have to give a shout out to ouripipgo proxy serviceIt specializes in dynamic residential proxies and has a pool of real residential IPs from more than 200 regions around the world.Not easily recognized as a crawlerAfter all, each request is changed to a different region of the real user IP, the site simply can not distinguish between real people visit or machine operation.

II. Hands on environment

Load these guys up first:

pip install requests beautifulsoup4

Don't forget to prepare the API key for ipipgo, you will get the exclusive access address and port after registration. It is recommended to store the configuration information in an environment variable so that the code looks fresh:

import os
PROXY_USER = os.getenv('IPIPGO_USER')
PROXY_PASS = os.getenv('IPIPGO_PASSWORD')

Third, the basic grasp of five steps

Let's use an e-commerce site as a target to demonstrate how to grab price data safely:

from bs4 import BeautifulSoup
import requests

def basic_crawler(url):: response = requests.
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
     Here we change the selector according to the actual page structure
    prices = soup.select('.price-section')
    return [p.text.strip() for p in prices]

But this bare running operation, not less than 10 minutes absolutely blocked. Next, let's put a "bulletproof vest" on it.

IV. Putting a Proxy Shield on the Crawler

Modify the session object of requests to integrate ipipgo's proxy service:

session = requests.Session()
session.proxies = {
    'http': f'http://{PROXY_USER}:{PROXY_PASS}@gateway.ip ipgo.com:8080',
    'https': f'http://{PROXY_USER}:{PROXY_PASS}@gateway.ipipgo.com:8080'
}

def safe_crawler(url).
    try.
        response = session.get(url, timeout=10)
        response.raise_for_status()
         Handling parsing logic...
    except requests.exceptions.RequestException as e:: print(f "f", "f", "f", "f", "f")
        print(f "Request flopped: {str(e)}")
         Retry logic for automatic IP switching

Here's the point:ipipgo's proxy server comes with an automatic IP rotation function, so each request may use a different exit IP. if you eat with a random User-Agent, the effect is even better.

Fifth, the actual combat: capture commodity data does not roll over

A complete case study combining proxy IPs and anti-anti-crawling strategies:

import random
from fake_useragent import UserAgent

ua = UserAgent()
headers = {'User-Agent': ua.random}

def super_crawler(url):: {'User-Agent': ua.random}
    try: with session.get(url, headers=headers) as resp.
        with session.get(url, headers=headers) as resp.
            if 'CAPTCHA' in resp.text: if 'CAPTCHA' in resp.text: if 'CAPTCHA' in resp.text.
                print("CAPTCHA triggered!")
                 Here you can access the coding platform
                return None
            soup = BeautifulSoup(resp.text, 'lxml')
             Data parsing logic...
    except Exception as e.
        print(f "Rollover site: {e}")
        return None

With this set of consecutively grabbed 3 days of a certain East commodity data, ipipgo's agent pool froze without being ban, visible dynamic residential agent is indeed reliable.

VI. Guidelines for demining common problems

Q: Why am I still blocked even though I use a proxy?
A: Check three points: 1. whether the proxy type is used correctly (recommended residential proxy) 2. whether the request frequency is too high 3. whether to bring a random request header

Q: What is the difference between ipipgo and other agents?
A: The best feature of his house isReal Life Housing IPIt is not as easy to be recognized as a server room agent. Under the same request volume, the blocking rate is lower than others by more than 60%.

Q: How do I break the CAPTCHA when I encounter it?
A: Appropriately reduce the request frequency + random mouse movement track simulation. If there are really too many CAPTCHAs, it is recommended to access professional coding services.

Q: How can I tell if a proxy is in effect?
A: Visit the address http://ip.ipipgo.com/checkip to see the exit IP and geographic location currently in use.

VII. Summary of anti-sealing tips

1. Proxy IP selectionipipgo residential agentDon't use free agents.
2. Randomization of User-Agent per request
3. Control the frequency of requests, don't burst like a machine gun.
4. Critical data captureWith automatic retrycode logic
5. Regularly check proxy connectivity and replace failed IPs in a timely manner.

Finally remind all crawlers, with proxy IP is not a gold medal, comply with the website robots agreement is the way to go. Need long-term stable collection, it is recommended to directly contact ipipgo customer service customized exclusive proxy program, their technical brother tuned program can make the collection of several times more efficient.