JavaScript set proxy IP: Node.js environment HTTP request proxy method

First, why use a proxy IP to get Node.js requests?

Do crawl or data crawl brother understand, the target site if you find a certain IP frequent visits, click on the block. This time you have to use a proxy IP toDecentralization of request pressureIt's like opening a small number to play a game to avoid being blocked. Residential proxy IPs like our ipipgo can be disguised as real users' online behavior and are more difficult to identify than ordinary server room IPs.

Second, hand to teach you two kinds of agent setup method

Here are two of the most commonly used routines, proxy configuration with axios and node-fetch, remember to install these two libraries in advance.

Method 1: axios + tunneling proxy

const axios = require('axios');
const proxyUrl = 'http://用户名:密码@gateway.ipipgo.com:端口';

async function fetchData() {
  try {
    const response = await axios.get('https://目标网站.com', {
      proxy: {
        host: 'gateway.ipipgo.com', port: Port, {
        port: port, {
        auth: {
          username: 'Your account', {
          password: 'Your key'
        }
      }
    });
    console.log(response.data);
  } catch (error) {
    console.log('Rollover:', error); }
  }
}

Method 2: node-fetch + native proxy

const fetch = require('node-fetch');
const { HttpsProxyAgent } = require('https-proxy-agent');

const agent = new HttpsProxyAgent('http://账号:密码@api.ipipgo.com:端口');

fetch('https://目标网站.com', { agent })
  .then(res => res.text())
  .then(data => console.log(data))
  .catch(err => console.log('Something's wrong:', err));

Third, it is important to choose the right type of agent

ipipgo has three major diamond packages to choose from depending on your business needs:

IV. Practical guide to avoiding pitfalls

A few common pitfalls encountered by newbies:

1. Authentication error 401: Check whether the account password with special characters, it is recommended to use encodeURIComponent transcoding
2. Connection timeout: Setting the proxy server response timeout to more than 30 seconds
3. IP blockedDynamic residential packages automatically change IPs every 5-10 minutes, more secure than static IPs.

V. QA First Aid Kit

Q: Proxy setting is successful but the request is not responding?
A: First test the proxy connectivity with the curl command: curl -x http://代理IP:端口 www.ipipgo.com/check

Q: How can I improve agent stability?
A: add a retry mechanism in the code, recommend using the promise-retry library, the failure of the automatic IP retry 3 times

Q: What if I need to use more than one agent at the same time?
A: Use ipipgo's API to extract multiple IPs, with round-robin algorithm polling call, code examples in the official website developer documentation has ready-made

VI. Special benefits

Attention brothers with ipipgo, theirSERP API servicesYou can directly return to the rendered web page data, than to write their own crawler to save time. If you encounter technical problems, find customer service to report the code [NODE666] can be inserted into the queue to deal with, and personally test the response speed is 3 times faster than ordinary work orders.