Map crawling tools: geo-data crawling program

Map crawler tool's biggest headache pitfalls

Engaged in geographic data capture of the old iron should understand, hard work to write a crawler script, the results just run ten minutes IP was blocked. In particular, the climb of Goddard, Baidu, these large map platform, anti-climbing mechanism than the cell access control is also strict. Once I witnessed a colleague's script ran 287 requests on the cool, the page jumped directly to the CAPTCHA, who tried to who knows.

Here's the kicker.IP Access Frequency MonitoringThe first thing you need to do is to get the information you need from your computer. Many platforms will count the number of requests from a single IP, like a hungry rider taking orders, and trigger an alert if they take too many. What's more, some sites will detect the geographic location of the IP, for example, you obviously logged in with a Beijing IP, and suddenly started to frantically request map data from Shanghai, which is very suspicious.

How to use proxy IPs as "cloak and dagger" for crawlers?

At this time it is necessary to proxy IP to play with the war, the principle is like playing hide-and-seek when constantly changing hiding places. For example, to climb the national chain store data, you can do so:

import requests
from itertools import cycle

 Proxy pool provided by ipipgo (example)
proxies = [
    "http://user:pass@123.123.123.123:8888",
    "http://user:pass@124.124.124.124:8888", ...
     ... More ipipgo proxy nodes
]
proxy_pool = cycle(proxies)

for page in range(1,100):
    current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
        response = requests.get(
            "https://mapapi.com/search",
            proxies={"http": current_proxy},
            timeout=10
        )
         Processing data...
    except.
        print(f "Flipped with {current_proxy}, switch to the next one.")

The key to this routine isIP rotation frequencyIt is recommended that you change your IP address every 50-100 requests, like changing your clothes to prevent collisions. According to the experience of testing, it is recommended to change the IP every 50-100 requests, like changing clothes to prevent collision. If you encounter a particularly strict site, you may need to shorten to 20 times a change.

What to look for in a proxy IP

There are all kinds of agency services on the market, but getting a map crawl to get recognized these hard indicators:

Special ReminderProtocol typeThe socks5 protocol, like ipipgo, is more suitable for high concurrency scenarios. There is a friend who does logistics data before, using the wrong http proxy, the result of concurrency open to 50 on the crazy drop line.

A practical guide to avoiding the pit

Name a couple of common fallouts for newbies:

1. IP pool too smallSome people try to buy 10 IPs cheaply to climb the province's data, and the result is that they are blacked out in half an hour. It is recommended to prepare at least 200+ dynamic IP pools, like ipipgo's flexible package is more cost-effective!

2. The request header's not disguised.: Remember to switch User-Agents randomly, so that all requests don't have "python-requests" on them.

3. Timeout settings are too deadSome proxy nodes may be jerked, timeout time is recommended to be set between 8-15 seconds, don't wait for a response!

Frequently Asked Questions QA

Q: Is it okay to use a free proxy?
A: Never! Free agents are like toilet seats in public toilets, which are actually full of mines. Previously tested, the availability of free agents less than 15%, and many of them are honeypot systems!

Q: How many IPs are needed to be sufficient?
A: Look at the data level. Municipal data 200 IP is enough, provincial recommendations 500 +. ipipgo business package to send IP automatic expansion and contraction of capacity, suitable for fluctuations in demand

Q: How do I break the CAPTCHA when I encounter it?
A: three countermeasures: ① reduce the frequency of requests ② change to a more anonymous agent ③ with the coding platform. Recommended to use ipipgoHigh Stash Residential Agency, measured probability of triggering CAPTCHA reduced by 70%

Q: What should I do if my proxy IP is slow?
A: Check three points: ① geographic location of the proxy node ② protocol type ③ local network environment. You can try ipipgo'sBGP High Speed LineThe support for automatic selection of the optimal node

Finally, the data crawl is a long-lasting war. Last week a customer with ipipgo rotation program, ran for 72 hours without being blocked, single machine daily average crawl from 30,000 to 270,000. This line of fighting is who's tool is more stable and more hidden, choose the right agent service provider can really less three years of detours.