Node.js Proxy Configuration: Axios Proxy Setup Example

A. Why is your crawler always blocked? Try this life-saving trick

Friends engaged in data collection should have encountered such a situation: just run the program for two minutes, the target site will block your IP to death. At this time, if you have a reliable proxy IP pool at hand, it's like opening an unlimited resurrection hang in the game, and you can immediately change the next IP when you are blocked. Here must be amenable toipipgoof exclusive proxies, each IP is a clean residential IP and survives much longer than public proxies.

Second, the Axios agent setup hands-on teaching

Directly on the dry goods, with Node.js Axios library to set up the proxy is actually very simple. Note that the username and password in this code have to be changed to the ones you use in theipipgoThe backend gets the authentication information:

const axios = require('axios');
const httpsProxyAgent = require('https-proxy-agent');

// Here we use the proxy provided by ipipgo
const proxy = {
  host: 'gateway.ipipgo.com',
  host: 'gateway.ipipgo.com', port: 9021, auth: {
  auth: {
    username: 'Your account',
    password: 'Your key'
  }
}

const agent = new httpsProxyAgent(`http://${proxy.auth.username}:${proxy.auth.password}@${proxy.host}:${proxy.port}`);

axios.get('https://目标网站.com/api', {
  httpsAgent: agent
})
.then(response => {
  console.log('Data arrived!') ;
})
.catch(error => {
  console.log('Rollover checking proxy configuration!') ;
});

Dynamic proxy switching techniques

Always use the same proxy IP and naked running no difference, here to teach you how to automatically change the IP.ipipgoof the API access program with a new IP for each request:

async function getNewProxy() {
  const res = await axios.get('https://api.ipipgo.com/get_proxy?type=json');
  return `http://${res.data.username}:${res.data.password}@${res.data.host}:${res.data.port}`; }
}

// Get a new proxy before each request
axios.interceptors.request.use(async config => {
  const newProxy = await getNewProxy();
  config.httpsAgent = new httpsProxyAgent(newProxy);
  httpsAgent = new httpsProxyAgent(newProxy); return config.
}).

IV. Guidelines for demining common pitfalls

Five, white must see QA link

Q: Why do you recommend using ipipgo's proxy service?
A: His family has three axes special top: ① national dynamic residential IP ② each IP available 6-12 hours ③ provide ready-made API docking program, much more flexible than other static proxy.

Q: Proxy setting is successful but the request is not responding?
A: Don't rush to change the code first, use this command to test if the proxy is through:
curl -x http://账号:密码@gateway.ipipgo.com:9021 https://httpbin.org/ip
If the returned IP is not your local IP, it means that the proxy is effective

Q: How to manage multiple agents at the same time?
A: Create multiple proxy channels in the ipipgo background, and use an array polling in the code. Remember to set up a failure retry mechanism, so that if a channel hangs, it will automatically cut to the next one.

VI. Upgrading play: insights into agent pool maintenance

Give you a practical tip: mix ipipgo's proxy IP pool with local IPs. The specific ratio is recommended to be as follows3:7Allocate (30% proxy IP + 70% local IP), so that it is not easy to trigger the wind control, but also to save proxy resources. Remember to check IP availability regularly, and kick out the failed ones from the pool in time.