fetch: Data Request Module in NodeJS Tutorial

Hands-on teaching you to use NodeJS fetch with a proxy IP!

What do you fear most when you do data crawling? The server will block your IP! This time we have to rely on proxy IP to save the day. Let's take the NodeJS fetch module as a chestnut today, teaching you how to use ipipgo's proxy service to keep you safe.

Let's figure out what's going on with the proxy IP.

Proxy IP is like a middleman, helping you forward network requests. Take a practical scenario: you want to collect data from a certain website in bulk, and directly dislike it with your own server IP, and it will be blacked out in minutes. At this time with ipipgo's dynamic IP pool, each request for a new vest, the other site simply can not understand the routine.

// A basic fetch example
const fetch = require('node-fetch');
async function basicRequest() {
  const response = await fetch('https://target-site.com/data'); console.log(await response.text()); async function basicRequest() {
  console.log(await response.text());
}

Put a proxy vest on fetch.

NodeJS native fetch does not support direct proxy matching, you have to use thehttps-proxy-agentThis artifact. Load it first dammit:

npm install https-proxy-agent --save

Then we use the proxy address provided by ipipgo (remember to go to the personal center of the official website to copy the exclusive proxy address):

const HttpsProxyAgent = require('https-proxy-agent'); // This is the example address.
const proxyUrl = 'http://username:password@gateway.ipipgo.com:9021'; // this is the example address

async function proxyRequest() {
  const agent = new HttpsProxyAgent(proxyUrl);

  const response = await fetch('https://target-site.com/protected-data', {
    agent: agent, {
    headers: {'User-Agent': 'Mozilla/5.0'}
  });

  console.log('Status Code:', response.status);
  
}

A practical guide to avoiding the pit

Here are a few bloody lessons that must be said:

1. Timeout settings to be added

Don't let the request get stuck there, it's recommended to use AbortController to control the duration

2. Error retry mechanism

Encountered 403/429 status code, automatically change ipipgo's IP and try again

3. UA camouflage is important

Don't use the default NodeJS User-Agent, it's easy to be broken.

Frequently Asked Questions QA

Q: What should I do if my proxy IP suddenly doesn't work?
A: First check the account status, ipipgo has real-time usage statistics in the background. It is recommended to use their API to get IP dynamically, which is more stable than a static proxy

Q: How do I test if the proxy is working?
A: Try it with this test interface:

fetch('https://api.ipipgo.com/check-ip', { agent: proxyAgent })

Q: What if I need to switch IP frequently?
A: ipipgo'spay-per-use packageSupports automatic IP change per request, just add &change=true parameter after the proxy address.

Why ipipgo?

A couple of real ones: their homeResponse speed average 80msThe IP pool is updated 200,000+ times daily, and there is a dedicated technical customer service on standby. Recently new users to send5G Traffic Pack, enough for a novice to build it vigorously.

Lastly, I would like to remind you that data collection should be done in a martial arts manner and the frequency of requests should be controlled. With ipipgo's intelligent flow limiting function, set a reasonable QPS parameter, guarantee that you can run the data safely and securely without overturning the car.