Python Web Crawler GitHub Resources: Python Crawler Agent GitHub Project Practice

Crawler is blocked IP, teach you how to whore GitHub resources with proxy IP.

Recently, when GitHub pulls the source code of the project, it is always blocked by 403. Tried a variety of user-agent camouflage or not, and then asked an old driver to do data capture, only to know that now the site have learned to be refined, directly blocking the IP address. At this time it is necessary to proxy IP to act as a stand-in actor, so that the server thinks that each visit is a different person.

Why use a residential proxy? Server room IPs are outdated.

A lot of newbies are still using free IPs, and as a result, they just climbed two pages to be blocked. Now the website anti-climbing mechanism thief fine, see the IP section of the room directly black. ipipgo dynamic residential agent with a real home broadband IP, like a real person surfing the Internet, the success rate is directly doubled.

import requests
from itertools import cycle

 List of proxies from ipipgo
proxies = [
    'http://user:pass@gateway.ipipgo.net:3000',
    'http://user:pass@gateway.ipipgo.net:3001',
    'http://user:pass@gateway.ipipgo.net:3002'
]
proxy_pool = cycle(proxies)

url = 'https://github.com/search?q=python+spider'
for page in range(1,6): proxy = next(proxy_pool)
    proxy = next(proxy_pool)
    try.
        response = requests.get(
            f"{url}&p={page}",
            proxies={"http": proxy, "https": proxy}, timeout=10
            timeout=10
        )
        print(f "Page {page} crawled successfully")
    except.
        print("Change IP and keep doing it!")

Three tricks to play with ipipgo proxy pools

First move:Create a "crawler-only" channel in the background, choose the Dynamic Residential Standard Edition package, which supports pay-per-use without waste. It is recommended to open more than 3 channels at the same time, so that you can switch to other channels in seconds when you are banned.

Second move:Use their API to get IP dynamically, remember to set 3 seconds timeout to switch automatically. Measurement of 50 times per hour to change the IP, running for 12 hours without triggering anti-climbing.

Third move:Add an exception retrying mechanism in the crawler code. It is recommended to use python's retrying library, configure 10 retry intervals, and personally test to catch GitHub's star history record as steady as an old dog.

White Common Pitfalls Fact Sheet

Q:Why are you still blocked even though you used a proxy?
A:The quality of the proxy is not good, the free proxy is often shared by many people. ipipgo's exclusive static residential IP, 35 bucks a month that, specifically to solve this problem.

Q:How come the crawler speed is not increasing?
A:Don't use a single thread! Do asynchronous requests with aiohttp, open 20 connections at the same time and remember to use a different proxy channel for each connection.

Q:What if I need to process a CAPTCHA?
A:In their background to open the TK dedicated line service, this line comes with a human verification crack, suitable for grabbing open source projects limited time STAR such a tawdry operation.

Tell the truth.

I've used seven or eight proxy services, and the most amazing thing about ipipgo is the "IP warm-up" feature. In the official start crawling before the first proxy IP to visit a few ordinary sites, and then use the IP after the site's wind control observation period, this trick makes my collection success rate soared from 47% to 89%.

Recently they've come up with a new feature that allows you to see the geographic location and carrier information for each IP directly in the client. Once I found out that a certain UK IP was actually a Vodafone line, and I used it to crawl the London company's public data, and it was rock solid!