Web Crawling with Go: Go Language Crawling

First, why do you have to use Go to get crawlers?

Nine out of ten people who work with crawlers these days are using Python, but old timers know that the Go thing was born to do this. Look at that.concurrentThe Go net/http library is a great way to get the most out of other languages, with hundreds of threads and less memory. Especially when you need to make a lot of requests, Go's net/http libraries are a great way to get rid of other languages.

There's a pitfall to be aware of though, many sites are nowStaring at IP address blockingThe first thing you need to do is to get a 403 forbidden. You have to write half a day's code, and then just run for ten minutes to receive a 403 forbidden, which is a matter of who will blow up. This is the time to bring out our secret weapon - ipipgo's proxy IP service.

// For example, setting up a proxy with Go is as simple as this
client := &http.Client{
    Transport: &http.Transport{
        Proxy: http.ProxyURL(&url.URL{
            Scheme: "http",
            Host: "your-account:password@proxy.ipipgo.io:9021",
        }),
    },
    Timeout: 30 time.
Second, }

Second, how did the proxy IP become a lifesaver?

Recently, I helped a friend to get a price comparison crawler, the target site anti-climbing strict as hell. At first, the local IP hard just, the results of half an hour was blocked. Later changed to ipipgo's dynamic residential proxy, good guy, ran for two consecutive days are fine.

ipipgo's pool of proxies is large enough, especially with theirIntelligent Route SwitchingFunction, automatically select the fastest node for you. I remember the last time I did e-commerce data collection, using their API to change the IP every 5 minutes, the data grabbed that called a smooth.

Third, hand to teach you to match the agent

Here is a trick for newcomers, do not directly use the default http.Get, with a Client instance is the right way. The following code you directly copy homework on the line:

func createProxyClient(proxyURL string) (http.Client, error) {
    proxy, err := url.Parse(proxyURL)
    if err ! = nil {
        return nil, err
    }

    return &http.Client{
        Transport: &http.Transport{
            Proxy: &http.ProxyURL(proxy), &http.
            TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, // Skip certificate verification.
        }, // Skip certificate validation.
        Timeout: 20 time.
    Second, }, nil
}

Remember to replace the proxyURL with the address given to you by ipipgo in the format ofhttp://用户名:密码@GatewayAddress:PortIf you have a problem with the certificate, you can temporarily skip it as above. If you run into certificate problems, you can temporarily skip validation as above, but it's better to have a proper certificate for a formal environment.

IV. Record of actual pitfalls

The last time I used Go to climb a travel site, the proxy was set correctly, but I couldn't connect. Later, I realized that it wasconnection poolIt wasn't paired properly and the default Keep-Alive was taking up the connection. The workaround was to add two lines to Transport:

Transport: &http.Transport{
    MaxIdleConns: 100,
    IdleConnTimeout: 90 time.
    TLSHandshakeTimeout: 10 time.
Second, TLSHandshakeTimeout: 10 time.Second, }

There is also a more evil, with ipipgo's proxy access is always timeout. Later found that the default node of their home is the HTTP protocol, the target site is forced to HTTPS. change to support HTTPS proxy package immediately resolved.

V. Frequently Asked Questions QA

Q: What should I do if I can't connect to the proxy IP all the time?
A: First check the account password is not right, especially special symbols to use URL code. Then try to test directly with the curl command, if it does not work quickly contact ipipgo customer service to change the node.

Q: What's the deal with crawling and suddenly getting blocked?
A: 80% of the request frequency is too high, even if the IP change has to control the rhythm. It is recommended to add a random delay in the code, using time.Sleep with rand.Intn to get some randomness.

Q: How is ipipgo better than others?
A: Other than that, two things are hard enough: aReal Life Housing IP, pseudo-degree pull full; second, the exclusive development of intelligent routing, automatically avoiding the ASN of the ban.

Sixth, say something heartfelt

In the crawler business, technique is just the basics.Resources are king.The problem is that you can't afford to have a proxy server. I've seen too many people build their own proxy servers, and as a result the maintenance costs are frighteningly high. Might as well just use ipipgo this kind of professional services, save time to write a few more crawlers do not smell?

Lastly, a reminder to newbies. Don't just crawl the data.Compliance with the Robots protocolIt's important. Some sites explicitly do not allow climbing, let's not hard, after all, do not do the project in order to eat lawyer's letter, right?