C# Parsing HTML: AngleSharp Library Tutorials

How does the AngleSharp library really work? Hands-On Web Page Grabbing with C

Brothers who have engaged in web crawling know that IP blocked is a common occurrence. At this time we have to offer our best - theproxy IPThe first thing you need to know is how to use a proxy service to parse HTML. Today we take ipipgo home proxy service, with C's AngleSharp library, teach you how to play in the code of HTML parsing.

Don't be lazy about environmental preparation

First install a NuGet package, open VS's package manager and knock this:

Install-Package AngleSharp.
Install-Package AngleSharp.

Proxy configuration piece to focus on, using the HTTP proxy provided by ipipgo, remember that their format isip:port:account:password. Give me a chestnut:

var config = Configuration.
    .WithRequesters()
    .WithDefaultLoader(new LoaderOptions
    WithDefaultLoader(new LoaderOptions) {
        IsResourceLoadingEnabled = true,
        Filter = request =>
        {
            request.Headers["User-Agent"] = "Mozilla/5.0";
            request.Headers["User-Agent"] = "Mozilla/5.0"; return true;
        }
    })
    .WithProxy(new ProxyOptions
    {
        new Uri("http://ipipgo-proxy.com:8000"),
        Credentials = new NetworkCredential("Your Account", "Password")
    });

Basic analysis of the three axes

Assuming that you want to catch the price of an e-commerce site, look at this code first:

var context = BrowsingContext.New(config); var document = await context.OpenAsync("Target URL"); var context = BrowsingContext.
var document = await context.OpenAsync("Target URL");

var priceNodes = document.QuerySelectorAll(".price-class"); var priceNodes = document.
foreach (var node in priceNodes)
{
    Console.WriteLine(node.TextContent.Trim()); }
}

take note ofQuerySelectorAllThis magic tool, with CSS selector with play like. When you encounter dynamically loading pages, remember to pair it with ipipgo'srotational agentFunction to change different exit IPs for each request.

A practical guide to avoiding the pit

Here to share a real case: a customer with a single IP to capture data, 10 minutes to be blocked. Later changed to use ipipgoIntelligent Agent Pool, configured as such:

var proxyList = new List
{
    "ip1:port:user:pass",
    "ip2:port:user:pass",
    //... Get the latest proxy list from the ipipgo backend
};

var randomProxy = proxyList[new Random().Next(proxyList.Count)];

The key point is thatRandom selection of agents, with AngleSharp's auto-retry mechanism, the success rate is pulled straight to full.

Frequently asked questions on demining

Q: What should I do if I can't connect to the agent?
A: First check if the package is in effect on the ipipgo backend, try theirConnectivity testing tools. Remember to set the timeout in the code:

var requester = new HttpRequester();
requester.Timeout = TimeSpan.FromSeconds(15);

Q: Page elements can't be caught?
A: Eighty percent of the selector is written wrong, use the browser developer tools to confirm the element path. ipipgo'sHigh Stash AgentsCan avoid being recognized as a crawler by websites.

Q: Need to handle JavaScript rendering?
A: AngleSharp itself does not support JS, this time you have to PuppeteerSharp, but remember to configure ipipgo proxy in headless browser:

var options = new LaunchOptions
{
    Args = new[] { $"--proxy-server=http://{randomProxy}" }
};

Why ipipgo?

Real-world comparison of the three service providers, ipipgo wins in three areas:

Especially theirCommercial-level agent poolIt supports launching 500+ connections at the same time without lagging, which is suitable for enterprise-level crawler projects.

As a final rant, web crawling is aboutprofessional ethicsThe first thing you need to do is to control the frequency of requests. Control the frequency of requests, don't hang people's servers. Use ipipgo.Intelligent Speed Limiting AgentIt can automatically adjust the request interval, which is both efficient and safe.