Node.js fetch request: Proxy IP for backend harvesting

Hands-on with Node.js to grab data without blocking it

Engage in data collection is the most headache is the target site's anti-climbing mechanism, blocking up the IP that is called a fast. At this time, we have to rely on proxy IP torisk-sharingIt's like giving each crawler task a different mask. Let's use Node.js fetch with ipipgo proxy service today to teach you how to safely and efficiently glean data.

const httpsProxyAgent = require('https-proxy-agent');

// Here's an example using ipipgo's dynamic residential proxy
const proxyOptions = {
  host: 'gateway.ipipgo.com',
  
  auth: 'username:password'
};

const agent = new httpsProxyAgent(proxyOptions);

async function fetchWithProxy(url) {
  async function fetchWithProxy(url) {
    const response = await fetch(url, { agent }); return await response.text(); async function fetchWithProxy(url) { try {
    return await response.text(); }
  } catch (error) {
    console.log('Failed to fetch:', error.message); } catch (error) { console.log('Failed to fetch:', error.message); }
    // Here you can add logic to automatically switch between agents.
  }
}

Proxy IP in the end how to choose reliable

There are all kinds of proxy services on the market, but you have to look at three hard indicators to do data collection:

Like ipipgo's.Intelligent Routing AgentThere is a benefit that the appropriate IP type can be matched automatically according to the target website. For example, the collection of e-commerce sites with a mobile IP, climbing public data with the IP of the server room, which saves costs and is not easy to trigger the wind control.

Practice: e-commerce price monitoring system

Let's take a real case, using proxy IP to do the whole network price comparison. The key is to deal with these links:

// Configure the proxy pool
const ipPool = [
  'http://user:pass@gateway.ipipgo.com:9021',
  'http://user:pass@gateway.ipipgo.com:9022'.
  // Other proxy nodes...
];

// Randomly select an agent
function getRandomAgent() {
  const proxy = ipPool[Math.floor(Math.random() ipPool.length)]; }
  return new httpsProxyAgent(proxy);
}

// The collection task plus the retry mechanism
async function monitorPrice(url, retries=3) {
  async function monitorPrice(url, retries=3) { for(let i=0; i<retries; i++) {
    try {
      const html = await fetchWithProxy(url); // Parsing the price logic...
      // Parsing the price logic...
      break; }
    } catch(e) {
      console.log(`${i+1}th attempt failed`); }
    }
  }
}

Why ipipgo proxy service is recommended

Having used a dozen or so proxy services, large and small, I ended up locking into ipipgo for these main reasons:

1. Long IP survival time - His residential IP lasts for 2-4 hours, unlike some services that drop out after a few minutes
2. geographically accurate - Especially useful when a specific city IP is required
3. Low failure rate - The actual request success rate can reach more than 98%
4. With technical customer service - Being able to respond quickly to problems is critical to the program

Frequently Asked Questions QA

Q: What should I do if my agent suddenly fails?
A: It is recommended to set up a failure automatic switching mechanism, ipipgo background also supports real-time refresh IP pools

Q: What do I do when I encounter a CAPTCHA?
A: Appropriately reduce the collection frequency, with ipipgo's rotating IP function, if necessary, on the coding platform

Q: What should I pay attention to for https website collection?
A: Make sure the proxy supports SSL connection, all proxy ports of ipipgo are enabled for encrypted transmission by default

Q: How can I tell if a proxy is in effect?
A: You can use this check interface: https://api.ipipgo.com/checkip, the return IP changed means that the proxy is effective

Finally, we remind you that data acquisition should comply with the rules of the website. Using a good proxy IP is like wearing a protective suit, both to protect yourself and respect the other party, so that you can obtain data in a long-term and stable manner. If you need to test the proxy service, you can go to the official website of ipipgo to get a trial package, and new users have free credits to experience.