Search Engine Results API: SERP Data Interface

This could be the SERP data collection solution you've never seen before!

The data collection of the old iron know, directly call the search engine API is like running naked - minutes to be blocked IP. those so-called official API interface, either outrageously expensive, or more restrictions like a tightrope. Today we talk about some wild ways, with proxy IP to play around with search engine results collection.

Why do traditional methods always fail?

A lot of newbies come in and jack the code, only to find out:

import requests
response = requests.get('https://api.search.com?q=关键词')
 Half an hour later... Your IP has been restricted

the problem is...Requests are too homogenous in characterThe search engine is not a fool. Search engine is not a fool, the same IP high-frequency request, with toes can guess is the machine operation. At this time it is necessary to proxy IP to cover, so that each request is like a different user in the operation.

The right way to open a proxy IP

There are three hard indicators to look for when choosing a proxy IP (take ipipgo for example):

Here's the kicker.Request interval settings: Don't be silly and use fixed time intervals, stop randomly like a real person searching. Like this:

import random
import time

def random_delay(): time.sleep(random.uniform(1.5, 5.8))
    time.sleep(random.uniform(1.5, 5.8)) randomly wait 1.5-5.8 seconds

Troublesome maneuvers in the real world

Remember to pair these tips when using ipipgo's proxy pool:

1. UA camouflage: Don't just use one browser logo, prepare 20+ common UA rotations
2. Request header randomization: The Accept-Language, Referer parameters have to be changed every time.
3. Failure Retry Mechanism: Auto switch IP retry when encountering 429 status code

Take a look at a full example:

from ipipgo import ProxyPool This is the main library to use.
import fake_useragent

proxy = ProxyPool(token='your key') get from ipipgo backend
ua = fake_useragent.UserAgent()

def search(keyword).
    headers = {
        'User-Agent': ua.random,
        'Accept-Language': 'zh-CN,zh;q=0.9'
    }
    proxies = proxy.get_proxy() Automatically gets the latest IPs

    try.
        response = requests.get(
            f'https://api.search.com?q={keyword}',
            headers=headers,
            headers=headers, proxies=proxies,
            timeout=8
        )
        return response.json()
    except Exception as e.
        proxy.report_error(proxies['ip']) flag the IP in question
        return search(keyword) auto-retry

Guide to avoiding the pit (QA session)

Q: Why do I still get blocked after using a proxy?
A: Check three points: 1. whether the request header is set 2. whether the IP quality is up to standard 3. whether the request frequency is too high

Q: How fast can I collect?
A: With ipipgo's concurrency program, measured 1 hour can pick 30,000 pieces of data. But be careful not to greedy fast, control 2-3 requests per second is safer!

Q: What's wrong with the data suddenly becoming less?
A: The probability of triggering the anti-climbing mechanism. Suggestions: 1. Replace the IP segment 2. Increase the mouse movement track simulation 3. Randomly add search keyword suffixes

Say something from the heart.

Proxy IPs are like consumables, don't try to buy cheap junk IPs, I've used a certain IP before, 6 out of 10 are blacklisted by search engines. Later, I switched to ipipgo, mainly because of their family'sIP cleansing mechanismThe IP pool is kept pure by automatically eliminating flagged IPs on a daily basis.

Finally remind: collect data to comply with the rules of the platform, do not catch a search engine to the death grip. Reasonable set collection strategy, with high-quality proxy IP, is the long-term solution. If you need to test, you can go to ipipgo official website to get a free trial package, new users to send 1G flow enough to toss.