Golang HTML Parser: Parsing HTML in Go

What happens when a crawler meets an anti-blocking mechanism?

Do data collection of the old iron know that the target site's anti-climbing mechanism is like a summer mosquito - indefensible. Yesterday you can visit the page normally, today suddenly give you a pop-up verification code, or directly blocked IP. this time you need to give the programPut on a vest.And proxy IPs are the best cloak of invisibility.

For example, when you write a crawler with Golang, if you always use the same IP to make crazy requests, the server will pull the plug on you in a minute. It's like going to the cafeteria to eat, if you are a person to cut in line more than ten times in a row, the aunty will definitely take the rice spoon to knock your head.

// Example of a basic request (written to death)
resp, err := http.Get("https://target-site.com/data")
// This gets the IP blacklisted the next day

The right way to parse HTML in Go!

Handling HTML parsing is recommended with the goquery library, which is much smoother than the official parser that comes with it. Just like eating noodles with chopsticks, it's always easier than grabbing them with your hands. Installation is very simple:

go get github.com/PuerkitoBio/goquery

In practice, with the proxy IP use better, here to demonstrate how to integrate ipipgo proxy service into the code:

func fetchWithProxy(url string) (goquery.Document, error) {
    // Fetch the proxy address from ipipgo.
    proxyUrl, _ := url.Parse("http://user:pass@proxy.ipipgo.com:9023")

    client := &http.Client{
        Transport: &http.Transport{Proxy: http.ProxyURL(proxyUrl)},
        Timeout: 15 time.
    }

    resp, err := client.Get(url)
    if err ! resp, err := client.Get(url) if err !
        return nil, err
    }
    defer resp.Body.Close()

    return goquery.NewDocumentFromReader(resp.Body)
}

Anti-blocking Practical Skills Manual

Here's a list of a few life-saving tips for the guys:

Guidelines on demining of common problems

Q: How many times will the proxy IP be invalidated?
A: In this case it is recommended to use ipipgo's Dynamic Residential Proxy, their IP pool is updated every day with 200,000+ fresh IPs, fresher than the vegetables in the market.

Q: HTTPS website resolution failure?
A: Add TLS configuration in Transport, like this:

Transport: &http.Transport{
    Proxy: http.ProxyURL(proxyUrl),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}

Q: How can I tell if a proxy is in effect?
A: Add an IP detection step in the code, for example, visit http://ip.ipipgo.com/checkip, the returned IP is the proxy address means it is successful.

Let the program learn seventy-two changes

One last advanced suggestion: plug ipipgo's API into the error retry mechanism. When the program detects a request failure, automatically call their interface to get a new IP, like a chameleon to switch identity at any time. In this way, even if the target site has eyes of fire, but also can not recognize your crawler's real body.

Here's a pseudo-code logic for a self-healing crawler:

for retry := 0; retry < 3; retry++ {
    doc, err := fetchWithProxy(url)
    if err == nil {
        if err == nil { break
    }
    // Automatically change ipipgo's proxy node
    updateProxy()
    time.Sleep(2 time.Second)
time.Sleep(2) }

Remember, a good crawler has to learn to play guerrilla warfare, ipipgo's million IP pool is your ammunition depot. Don't use free proxies anymore, those IPs have long been played with, like a horse poke in a public restroom, anyone has used them, they can't hide your tracks at all.