How to get the proxy ip in bulk?API interface and crawler collection of dual program details

Hands-on teaching you to engage in bulk proxy IP, two programs directly on the dry goods

Now engage in data collection, batch registration of these operations, no proxy IP is like frying vegetables without salt. There are two common ways on the market: either use the ready-made API interface, or write their own crawlers to glean free resources. Let's break it up today and say, which situation should be used in which position.

Option 1: API interface steady as an old dog

First of all, let's talk about the way to save time, directly connect to the service provider's API. take ipipgo's dynamic residential proxy as an example, their interface is designed like a stupid camera. Sign up for an account, get the key, and follow the documentation to adjust the interface on the line.

import requests

def get_proxies(): api_url =
    api_url = "https://api.ipipgo.com/dynamic/get"
    params = {
        "key": "Your key",
        "country": "us",
        "protocol": "socks5",
        "quantity": 10
    }
    resp = requests.get(api_url, params=params)
    return [f"{p['protocol']}://{p['ip']}:{p['port']}" for p in resp.json()['data']]

Note a few key parameters:Country code selected by countryThe protocol is to look at the business needs to choose the protocol, quantity do not want too much at a time. ipipgo's interface response speed thief, measured per second can spit 200 + valid IP, than some platforms card half a day much stronger.

Option 2: A Complete List of Crawler Collection Wildcards

Those who do not want to spend money can try free proxy sites, but be prepared - nine out of ten of these IPs are pits. Here's a basic collection script:

from bs4 import BeautifulSoup
import requests

def scrape_free_proxies():
    proxies = []
    try: resp = requests.get('', timeout=10)
        resp = requests.get('https://example-proxy-site.com', timeout=10)
        soup = BeautifulSoup(resp.text, 'lxml')
        for row in soup.select('table tr'):: cells = row.
            cells = row.find_all('td')
            if len(cells)>=2.
                proxies.append(f"{cells[0].text}:{cells[1].text}")
    except Exception as e.
        print('Capture failed:', str(e))
    return proxies

There are three major potholes in this law:Low survival rate, slow, easily blockedIf you are doing serious business, it is recommended that you don't bother with the free service. If you are doing serious business, it is recommended that you don't bother with the free ones, or you may not get the data, and your own IP will be blacked out.

API vs Crawler

How to choose a ipipgo package without stepping on the line?

Their home is mainly divided intoDynamic Residential (Standard/Enterprise Edition)respond in singingStatic homesTwo kinds:

• Dynamic standard version: suitable for short-term projects, IP automatically changed every 15 minutes, pay by volume does not hurt!
• Dynamic enterprise version: with exclusive channel and fixed regional IP, do cross-border e-commerce closed eyes into the
• Static residence: a must for long-term number raising, an IP can be used for 30 days without change

Frequently Asked Questions QA

Q: What should I do if my IP is always blocked?
A: Check if you are using a transparent proxy, change the high stash proxy and control the access frequency. ipipgo's dynamic IP comes with request header camouflage, which is stronger than ordinary proxies to resist blocking.

Q: How do I test if the agent is valid?
A: Use this detection script:

def check_proxy(proxy).
    try.
        resp = requests.get('http://httpbin.org/ip',
                          proxies={'http': proxy, 'https': proxy}, timeout=5))
                          timeout=5)
        return resp.json()['origin'] in proxy
    except.
        return False

Q: How many IPs do I need to use at the same time?
A: According to the amount of business, ordinary collection 1 minute to change 1 enough. If you are doing spike type business, it is recommended to use ipipgo's rotation mode to cut different IPs per second.

Finally, to say a big truth: free agents look to save money, the actual cost of time and risk, it is really not as good as directly on the reliable paid services. In particular, the need for long-term stability of the business, choose ipipgo can customize the IP time limit, not a little half a point.