C# Web Crawl: HttpClient Library Tutorials

When the crawler meets anti-climbing: HttpClient how to use proxy IP to break through restrictions

The most headache is to engage in web crawling IP is blocked, especially with the C data collection brothers should have a deep understanding. We do not organize those false today, directly on the dry goods to teach you how to use HttpClient with proxy IP, focusing on how to use ipipgo service to save life.

HttpClient Basic Operations

First of all, understand how to use HttpClient to send requests, this thing can be much better than WebClient. Remember to set a timeout when initializing, don't let the program get stuck:

var handler = new HttpClientHandler(); var client = new HttpClient(handler){
var client = new HttpClient(handler){
    Timeout = TimeSpan.FromSeconds(15)
}

To send a GET request to play like this, remember to add User-Agent to disguise the browser:

client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0) ...") ;)

Three Tips to Save Your Life with Proxy IPs

When the403 Forbiddenor429 Too Many RequestsIf you have a proxy IP, it's time to change it. Configuring it with ipipgo's service is extraordinarily easy:

Real-world anti-blocking techniques

1. The IP pool has to be big enough: Use ipipgo's API to get new IPs on a regular basis, it is recommended to change to a different proxy for each request.
2. Don't be too regular in your request intervals: set random wait seconds with Random.Next(3,8)
3. Failure auto switch: Change agent immediately when an abnormal status code is encountered

try{
    var response = await client.GetAsync(url);
}
catch{
    // Call ipipgo's API to change IPs
    SwapProxy(handler);
}

Frequently Asked Questions QA

Q: What should I do if my proxy IP is not working after I use it?
A: This is 80% of the IP is marked by the target site, with ipipgo's dynamic IP pool will be automatically refreshed, remember to set the failure retry mechanism in the code.

Q: HTTPS website crawling always report certificate error?
A: Add this configuration to the HttpClientHandler:
handler.ServerCertificateCustomValidationCallback = (msg, cert, chain, err) => true;

Q: How to determine whether the proxy IP is effective?
A: Visiting the address http://ip.ipipgo.com/checkip returns the IP address of the proxy currently in use.

Real Case Demonstration

Recently a brother did e-commerce price monitoring and used the solution I gave him:
1. Each request randomly selected ipipgo's domestic server room IP
2. Sleep for 2 minutes for every 50 requests completed
3. Automatically switch city nodes when encountering CAPTCHA
As a result, it ran for 7 consecutive days without being blocked, and the collection success rate soared from 37% to 92%.

Lastly, I would like to remind you that you have to look for quality in choosing a proxy service. Support like ipipgopay per volume,IP survival rate of 95% or moreThe service provider, it is indeed more worrying to use than to build their own proxy pool. Don't wait until the IP is blocked into a sieve before you remember to add a proxy, then the data has long been picked collapse.