Automobile Sales Data: Automobile Sales Data Acquisition

How do you mess with car sales data? Teach you to use proxy IP to bypass the pit

Attention newbies who want to mess with car sales data! Many websites are now loaded withIP identification systemYou can check it dozens of times in a row and it will be blocked immediately. Last week, a buddy used his home broadband to check the 4S store offer, the result of the next day the entire neighborhood network can not open the site.

Why is IP blocked? Read it and you'll understand

Websites are learning the ropes now, and three characteristics immediately lock you in:

1. the same IP frequent requests (more than 30 times / minute)
2. request time is too regular (such as every 5 seconds to grab data)
3. User-Agent does not change (always use the same browser logo)

It's like if you go to the grocery store to try something out and wear the same red dress every time, the sales clerk should kick you out the third time she sees you.

The right way to open a proxy IP

Recommended hereDynamic Residential Proxy for ipipgo, there are three main advantages of their IP pool:

The actual test to catch the official website data of a car enterprise, with ordinary proxy 1 hour was sealed, replaced with ipipgo lasted 6 hours is still fine.

Hands-on Crawler Scripting

In Python, for example, the key code has to be written like this (remember to install the requests library):

import requests
from random import choice

 The format of the proxies provided by ipipgo
proxies = {
    "http": "http://用户名:密码@gateway.ipipgo.com:端口",
    "https": "http://用户名:密码@gateway.ipipgo.com:端口"
}

headers = {
     Always change the browser logo here
    "User-Agent": choice([
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64)..." ,
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)..."
    ])
}

 The point: each request is randomly spaced
response = requests.get("destination URL", proxies=proxies, headers=headers, timeout=(3,7))

Be careful to puttimeoutSet it to interval values, don't use a fixed duration, it's more like a real person's operation.

Frequently Asked Questions QA

Q: Can't I use a free proxy?
A: car data sites are now on the AI wind control, free agent 99% are blacklisted, you use is to send head.

Q: How does ipipgo charge?
A: They are more cost-effective to bill by traffic, and the 10GB traffic package can grab about 100,000 data. The first registration sends 1GB trial, so it is recommended to try before you buy.

Q: What should I do if I encounter a CAPTCHA?
A: Two ways: 1) control the request frequency not to exceed 20 times/minute 2) use ipipgo'sHigh Stash AgentsThe CAPTCHA trigger rate for such IPs is low 60%

Guide to avoiding the pit

Three final reminders:
1. Don't write a dead proxy IP in the code, use dynamic rotation
2. Higher success rate of data capture at 2-5 a.m.
3. Stop for half an hour when encountering a 403 error, change to a new IP address and try again.

If you can't handle it yourself, you can just use ipipgo'sCustomized Capture ServicesThey can help you configure the whole package, which is much less stressful than tossing it yourself. Recently, a customer used their services, a week to catch the real-time quotes from 3,000 4S stores across the country.