Open Data Website: Open Data Proxy Collection Program

Why is open data collection always blocked? Try this wildcard.

Brothers who engage in data collection understand that the crawler runs and is choked by the website. Either the IP is blocked, or the frequency of access restrictions, the most disgusting is that some sites directly give you a pop-up CAPTCHA. At this time we have to use proxy IP to play guerrilla warfare - to put it bluntly is to use different IP rounds, so that the site thinks it is a group of people in the visit.

For example, you want to climb a city's public traffic data, the same IP access to 50 times in a row, the server immediately black. But if each request for a different IP address, the site wind control system will be confused. There is a key point here:The quality of the proxy IP directly determines the collection efficiencyThe problem is that there are many different proxies on the market. Proxy services on the market are a mixed bag, and some of the cheaper ones are used to realize that the IP survival time is only 3 seconds, or they can't connect at all.

Three Tips for Choosing the Right Type of Agent

Proxy IP is divided into three major schools, use the right to get twice the result with half the effort:

Focusing on dynamic residential IP, this thing is most suitable for collecting public data. Because it goes to the real home broadband, each request automatically change IP, the site can not tell whether it is a real person or a machine. Like ipipgo's dynamic proxy pool covers more than 200 countries, and it can also specify city-level location, which is good for capturing geographical data.

Teach you to pick up agents by hand

Here's a real-world example given in Python, using the requests library + proxy IP to collect data:

import requests

 Proxy API address from ipipgo
proxy_api = "https://api.ipipgo.com/getproxy?key=你的密钥"

def get_data(url).
     Get fresh proxy IP
    proxy = requests.get(proxy_api).json()['proxy']

    proxies = {
        "http": f "http://{proxy}",
        "https": f "http://{proxy}"
    }

    try.
        response = requests.get(url, proxies=proxies, timeout=10)
        return response.text
    except Exception as e.
        print(f "Request failed, automatically changing IP: {str(e)}")
        return get_data(url) auto-retry

 Example of collecting public data
traffic_data = get_data("http://data.example.com/traffic-info")

Be careful to putrequest intervalControl in 3-8 seconds random, too regular easy to be recognized. ipipgo client comes with intelligent scheduling function, can automatically control the switching frequency, than to write their own polling to save time.

A guide to stepping through the pits (QA session)

Q: What should I do if I use a proxy IP and it becomes slow?
A: 80% is the quality of IP pool is not good. Select supportReal-time speed measurementof service providers, like the ipipgo client that displays the latency of each node and manually blocks slow nodes.

Q: What should I do if I am bombarded with CAPTCHAs?
A: two programs: 1) reduce the collection frequency, each IP does not exceed 500 requests per hour 2) on the static residential IP, this type of IP survival time is long, it is not easy to trigger verification

Q: How do I break the need to collect foreign public data?
A: with cross-border dedicated agent, such as ipipgo's TK line goes to the local family broadband, much more stable than the ordinary server room IP. The actual test to catch the European public dataset, the success rate can be more than 98%.

Why do you recommend ipipgo?

There are three great things about this agency's services:
1. Capabilityhourly rateNo need to buy a monthly subscription for a temporary program.
2. Client built-inIP Health CheckAutomatically kicks out failed nodes
3. SupportSocks5 protocolIt's easy to interface with Python, Java, and so on.
In particular, their dynamic residential agent, the actual test collection of a government open platform, continuous running 12 hours without being blocked, the cost only spent less than 20 dollars.

Finally, don't just look at the price when choosing a proxy service. Some cheap packages with recycled IP (recycled IP), has long been pulled by the major sites black. It is recommended to get a test package to try the water, such as ipipgo new users to send 500MB traffic, enough to run a small project to verify the effect.