Node.js fetch module: Node web requests

How to use proxy ip for fetch in Node.js?

Recently, when I helped a friend to do a data collection project, I found that if I used the original IP to request directly, it would be blocked in less than half an hour. That's when I remembered thatProxy ip is simply a programmer's life-sustaining tool.Today we're going to talk about how to stuff proxy ip into Node.js fetch requests. Today let's talk about how to stuff proxy ip in Node.js fetch request, focusing on my commonly used ipipgo service.

Practical: 5 lines of code to the request to wear a vest

Let's look at the simplest example first, which can be achieved with the package http-proxy-agent:

import fetch from 'node-fetch';
import { HttpProxyAgent } from 'http-proxy-agent';

const agent = new HttpProxyAgent('http://username:password@gateway.ipipgo.com:8080');
const response = await fetch('https://目标网站.com', { agent });

Here pay attention to replace the real address and port you get in the ipipgo background. username and password must not use 123456 this kind of weak password, don't ask me how to know ...

Proxy ip type selection guide

There are three types of proxies provided by ipipgo, and it is important to choose the right type according to your business needs:

Common Pitfalls Lightning Avoidance Manual

The most dodgy situation I've encountered was when the proxy suddenly failed, which I later found out was because I didn't do the exception handling. Now I have to add this to my code:

try {
  const res = await fetch(url, { agent, timeout: 5000 });
} catch (err) {
  console.log('IP failed! Switch to alternate address'); } catch (err) { console.log('IP failed!
  // Automatically switch ipipgo's alternate node.
}

QA: 3 Frequently Asked Questions by Developers

Q：代理请求比慢怎么办？
A：建议用ipipgo的BGP线路，实测能降40%左右。另外注意设置合理的超时时间，别让程序死等。

Q: How can I prevent cookies from leaking my real IP?
A: Remember to add 'X-Forwarded-For' camouflage in the request header, ipipgo's premium package comes with this feature.

Q: Proxy IPs suddenly fail in a big way?
A: First check your account balance, then contact ipipgo customer service for an IP pool refresh interface. Don't mess around with it yourself, last time I manually changed the IP and crashed the program.

Maintenance Tips: Keeping Agents Alive

Share my three tips for staying alive:
1. Change IP every 50 requests (ipipgo's API can automatically rotate)
2. Randomly spaced requests, don't let the server see a pattern
3. 3:00 a.m. automatic detection of IP survival status

Lastly, the big truth is that choosing the right proxy service provider can save you half the trouble. I've been using ipipgo for over two years now, and they've got thatIP Survival Monitoring PanelIt works really well, you can see clearly which set of IPs is about to expire. If you are looking for a reliable agent, you may want to go to their website to gather a test package to try.