DuckDuckGo Search API: DuckDuckGo Proxy API Calls

Hands-on DuckDuckGo Search with Proxy IPs

Recently, a lot of friends who do data crawling have asked Lao Zhang, saying that the API of DuckDuckGo is often blocked IP how to do. In fact, this matter is just like playing the game dropped line, change the network channel will be solved. Today we will nag how to use proxy IP to call the DuckDuckGo interface in a stable place.

Why do you have to use a proxy IP?

For example, if you go to the supermarket to buy eggs, if you run ten times a minute to the cashier, the security guards must be on you.DuckDuckGo interface is also the same reason, the same IP frequent requests will be limited. This time you have to use a proxy IP toDecentralization of request pressure, as if opening ten cashier lanes to check out at the same time.

Three Steps to Real-World Configuration

Here use ipipgo proxy service to demonstrate, their API interface is really save trouble. First the whole Python code to see:

import requests

def ddg_search(query):
    proxies = {
        "http": "http://用户名:密码@gateway.ipipgo.com:端口",
        "https": "http://用户名:密码@gateway.ipipgo.com:端口"
    }

    response = requests.get(
        "https://api.duckduckgo.com",
        params={
            "q": query,
            "format": "json"
        },
        proxies=proxies
    )
    return response.json()

 Example of a call
print(ddg_search("weather"))

Focus on the proxy parameter in theUser name and passwordrespond in singinggateway addressThese can be obtained directly from ipipgo backstage. They support HTTP/HTTPS dual protocols, no need to toss certificates and all that crap.

There is a way to choose a package

There are three main packages for ipipgo, and Lao Zhang gives the guys an understandable list:

If you're just starting out, we recommend starting with theDynamic Standard EditionThe first thing you need to do is get started, and use as much as you can without any pain. Do batch crawler brother directly on the enterprise version, IP pool is larger and more resistant to manufacturing.

Frequently asked questions on demining

Q: Why was I still blocked even though I used a proxy?
A: 80% of the IP is reused, add a logic in the code to automatically change the proxy. ipipgo's API can return a list of available IPs, remember to rotate them.

Q：What should I do if the return data is always garbled?
A：Check the Accept-Encoding parameter in the request header, we suggest to set it to "gzip, deflate". If it doesn't work, go to ipipgo technical support directly, they are very quick to respond.

Guide to avoiding the pit

Recently, I have seen people trying to use free agents cheaply, and the result is that the data are all off. A friend of mine who does e-commerce price comparison has been planted, and the price data captured are all false. Later changed the ipipgo TK line is stable, this cross-border business or have to be a professional player.

Lastly, I would like to say that adjusting the API is like fishing, you have to find the right fish pond (proxy IP) and also know how to hit the nest (request strategy). ipipgo background that the usage statistics function is quite practical, you can see the consumption of each IP, it is convenient to adjust the strategy.