How to Crawl Websites with Python: A Getting Started Tutorial

Hands-on with Python to grab data without blocking it

Recently, a lot of friends asked me to use Python to climb the website is always blocked IP how to do? Today we will chatter about this matter. To put it bluntly, the site is like a neighborhood gatekeeper, see strangers always come to the door will pull the blacklist. This time you have to learn"Change of armor.", that is, disguise yourself with a proxy IP.

import requests
from random import choice

 Proxies pool from ipipgo
proxies_pool = [
    {"http": "http://123.34.56.78:8080"}, {"http": "http://123.34.56.78:8080"}, [
    {"http": "http://45.67.89.12:3128"}, ...
     ... More proxies provided by ipipgo
]

url = 'https://目标网站.com'

try.
    response = requests.get(
        url,
        proxies=choice(proxies_pool),
        timeout=10
    )
    print(response.text)
except Exception as e.
    print(f "Crawl failed, try another IP: {str(e)}")

How exactly do you use a proxy IP to be reliable?

There are three key points here that are easy to step on:

A real case in point: a buddy who does price comparison always drops out with free proxies. He switched to ipipgo.Dynamic Residential AgentsAfter the collection efficiency is directly doubled, the key to people's IP pool updated every day ten million IP, simply can not be used up.

QA Time: Frequently Asked Questions for Newbies

Q: Does it cost money to proxy IP? Does the free one work?
A: You can use free for short-term small quantities, but for serious projects it is recommended to use ipipgo's paid service. Their IP survival rate can reach more than 95%, which is much more trouble-free than tossing it yourself.

Q: What's wrong with the code running and reporting errors?
A: 80% is IP failure, remember to add exception handling in the code. ipipgo's API can also detect the IP status in real time, use their interface to get IP success rate is higher.

Practical Tips and Tricks

1. Before each request, check if the IP is valid, you can do this:

def check_proxy(proxy).
    try.
        requests.get('http://httpbin.org/ip',
                    requests.get('', proxies=proxy, timeout=5)
                    timeout=5)
        return True
    except: requests.get(''), proxies=proxy, timeout=5
        return False

2. Don't panic when you encounter a captcha, use ipipgo'sHigh Stash Agents+Random UA head combo, pro-tested to bypass 90%'s counter-crawl

3. Important data collection is recommended to use their API to obtain IP dynamically, code example:

import ipipgo Assuming this is their SDK

def get_fresh_ip().
    client = ipipgo.Client(api_key="your key")
    return client.get_proxy(type='http')

Why do you recommend ipipgo?

This is not an advertisement! The real-world comparison reveals:

• Response time is 2-3 times faster than others
• There are special anti-blocking IP packages
• Supporting pay-as-you-go without waste

The bottom line is that their homeIP Survival TimeIt is especially long, unlike some service providers that give you an IP that will be invalid after a few minutes of use. The last time I helped a client to do public opinion monitoring, it ran for a week without being blocked, so I do have two brushes.

Lastly, I would like to say: although the crawler is good, don't be greedy! Control the collection frequency, with a reliable proxy IP, in order to get the data in the long run. What do not understand, welcome to the comments section nagging ~!