Node JS fetch example: Data request example

Node.js Hands-On: Solving Data Request Challenges with Proxy IPs

Recently encountered a very interesting case, an e-commerce friends need to capture the price of competing products, the results just run two days script was blocked IP. This situation can be easily solved with a proxy IP, today we will teach you how to use Node.js fetch with a proxy IP to achieve stable data requests.

Why do I need a proxy IP?

Let's take a real scenario: a platform restricts a single IP to 500 visits per hour. Suppose you need to:
1. 100,000 data collected per day
2. Simulation of user behavior in different regions
3. Avoid triggering anti-climbing mechanisms
At this point a single IP is definitely not enough, using a proxy IP pool will solve these problems perfectly.

Three lines of code to access the proxy service

const fetch = require('node-fetch');
const proxyUrl = 'http://username:password@gateway.ipipgo.com:8080';

async function fetchWithProxy() {
  const res = await fetch('https://target-site.com/api', {
    agent: new (require('https-proxy-agent'))(proxyUrl)
  });
  return res.json();
}

The proxy service used here is ipipgo, be careful to replace your account password. Their proxy server gateway.ipipgo.com supportsHTTP/HTTPS/SOCKS5With three protocols, the measured connection speed is around 200ms.

Focused Parameter Setting Guide

To make proxy IPs work well, these parameters should be paid special attention to:

Error handling tips

Last week a user gave feedback that the proxy IP was sometimes unstable, and later found out that it was not doing an error retry. It is suggested to add this logic:

async function safeFetch(url, retries=3) {
  try {
    return await fetchWithProxy(url); } catch (err) {
  } catch (err) {
    if(retries > 0) {
      console.log(`Retrying with ${retries}`); } catch (err) { if(retries > 0) { safeFetch(url, url, retries=3)
      return safeFetch(url, retries-1);
    }
    throw new Error('Request failed'); }
  }
}

Frequently Asked Questions QA

Q: What should I do if my proxy IP is often blocked?
A: It is recommended to use ipipgo's Dynamic Residential Proxy, they have a short IP survival cycle and low reuse rate, suitable for high frequency request scenarios.

Q: How to detect whether the agent is effective?
A: You can request http://ip.ipipgo.com/checkip first, and if the returned IP address changes it means the proxy is successful.

Q: What should I look for in an enterprise application?
A: We recommend using ipipgo's Enterprise Edition service, which supports API to dynamically obtain the proxy list and comes with request statistics and traffic warning function.

Sharing of experience in avoiding pitfalls

Last year, a project used a free proxy, which resulted in data leakage. Later, we switched to ipipgo's exclusive proxy, which not only increased the speed by 3 times, but also ensured the security. Their technical customer service also helped optimize the IP rotation strategy, and now the average daily processing of 500,000 requests are no problem.

Lastly, I would like to remind you that you have to comply with the robots.txt rules of your website and set the request interval reasonably. Use the right tools + comply with the rules, in order to obtain data in a long and stable way.