DoorDash Dataset: Takeout Industry Data Kit

How hard is it to capture takeout data?

Anyone who owns a takeaway business knows that it's harder than ever to get to the bottom of your competitors. Especially for platforms like DoorDash, the data protection is very strict. Last week, an old pizza store customer complained to me that his family operation manually transcribed competitors' information, and as a result, the account was blocked the next day, and he was in a hurry to jump to his feet.

It's time to bring out ourproxy IPThe magic weapon. For example, if you want to collect the delivery range of burger restaurants in the city, if you use your own network to request repeatedly, DoorDash's fire prevention will label you as a "robot" in minutes. But if you change the IP address for each request, it's like having a different person visit the store, and the platform won't even notice the anomaly.

import requests
from ipipgo import get_proxy The SDK for ipipgo is called here.

def fetch_menu(store_id):
    proxies = {
        "http": get_proxy("residential"), use residential proxy for more realism
        "https": get_proxy("residential")
    }
    url = f "https://api.doordash.com/v2/stores/{store_id}/menu"
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}

    try: response = requests.get(url)
        response = requests.get(url, headers=headers, proxies=proxies, timeout=10)
        return response.json()
    except Exception as e.
        print(f "Crawl error: {str(e)}")
        return None

Poor choice of proxy IP, data collection all for nothing!

Proxy service providers on the market are a mixed bag, I have seen the most pitiful is a certain claim to millions of IP pool, the result is that 15 out of 20 requests are recognized by DoorDash. Later, I changedipipgo(used form a nominal expression)Dynamic Residential IPThe effect is immediate. They have a unique skill - IP survival time control is the same as a real user, there will not be just logged on the front foot after the foot disconnected embarrassment.

Real-world example: monitoring pop-up dishes with ipipgo

Last year, we helped a sushi chain do a competitive analysis, and they wanted to monitor in real time all the hibachi restaurants within 10 kilometers of each other in the neighborhoodNew products on the shelvesrespond in singingPromotions. We operate like this:

1. Adoption of ipipgo'sCity-level IP positioningThe following are some examples of how to ensure that each request comes from the target area
2. Set the IP address to change every 5 minutes
3. In conjunction with browser fingerprinting camouflage technology (more on this next time)
4. Automatic recognition of menu changes and triggering of e-mail alerts

It turned out that the next week we caught our competitors pushing$9.90 package, the customer immediately adjusted their lunchtime offer and turnover went up by 23% that month.

A must-see QA session for beginners

Q: Is it legal to collect data with a proxy IP?
A: As long as you don't crack the website or steal user information, simply collecting public data is like looking at a street view through a telescope, and it's perfectly fine. Of course, it depends on the platform's terms and conditions, and it is recommended that you consult a legal advisor before collection.

Q: Why do you recommend ipipgo?
A: Three hardcore reasons: ① His IP pool covers200+ countries②ExclusiveRequest Success Rate GuaranteeThe number of hours is lower than 95% automatically ③ Customer service response is faster than takeout, the last time 3:00 a.m. problem work orders, 5 minutes to receive a solution.

Q: What details should I pay attention to when collecting?
A: focus! ① control the frequency of requests, don't make it look like a DDoS attack ② randomize the operation interval, the real order which is not so punctual ③ regular cleaning Cookies, don't let the platform remember your "digital fingerprints".

Guide to avoiding pitfalls: don't step on these mines

Recently, I've found that some of my peers have fallen prey toIP purityOn. A friend doing data analysis bought a second-hand proxy on the cheap, and as a result the DoorDash account was blocked en masse. Later switched to ipipgo'sExclusive IP packageIt's the only way to fix the problem, it's more expensive but saves you the loss of being blocked.

A final word of advice: don't believe those who claim to be"Permanently effective"proxy IPs, there is simply no one-size-fits-all method in this business. It is recommended to choose a proxy like ipipgo that canReal-time IP changeThe service is like putting a golden bell on data collection, which is safe and hassle-free.