Node.js Proxy: HTTP Middleware Building Tutorials

Hands-on with Node.js to build a proxy middleware

Old iron should have encountered the need for proxy IP scenarios, right? For example, crawling data by the site blocked IP, or to test the effect of access to different regions. Today we use Node.js the whole simple HTTP proxy middleware, without those complex frameworks, directly on the native module to engage in!

Prepare a new project first. Knock it in the terminal.npm init -yBuild a package.json. Here's the kicker, the core module we're going to use is thehttp-proxy, this one is easier to use than making instant noodles. Installation commands:

npm install http-proxy --save

Three lines of code to start a proxy service

Create a new proxy.js file and code this:

const http = require('http');
const httpProxy = require('http-proxy');

const proxy = httpProxy.createProxyServer();
const server = http.createServer((req, res) => {
  proxy.web(req, res, {
    target: 'http://your-backend-server.com',
    changeOrigin: true
  });
});

server.listen(3000, () => {
  console.log('The proxy service is running on port 3000!) ;
});

Here's a pitfall to watch out for.changeOriginThis parameter must be set to true, otherwise the target site may not recognize your request. It's like if you go to someone's house without knocking, you'll be blocked, right?

Put an IP vest on the proxy.

It's time to bring out ouripipgo proxy serviceup, change the code to this:

const proxy = httpProxy.createProxyServer({
  agent: new http.Agent({
    agent: new http.Agent({ keepAlive: true, proxy: '')
    proxy: 'http://username:password@gateway.ipipgo.com:8080'
  })
});

Here username and password to go to the official website of ipipgo to apply, their proxy IP pool is very large, nodes all over the country. Especially suitable for scenes that require frequent IP switching, such as grabbing shoes and tickets or something (manual dog head).

How to play around with dynamic IP pools

If you need to switch IPs automatically, you can modify it like this:

const ipPool = [
  'http://ip1.ipipgo.com:8080',
  'http://ip2.ipipgo.com:8080'.
  //... More IPs
];

function getRandomIP() {
  return ipPool[Math.floor(Math.random() ipPool.length)];
}

// Use this for proxy requests
proxy.web(req, res, {
  target: 'http://your-backend-server.com',
  agent: new http.Agent({
    proxy: getRandomIP()
  })
});

It is recommended to use the dynamic API provided by ipipgo to get IPs, their IPs are highly anonymized, which makes them more stable to use. I've used a number of proxies before, but in the end, ipipgo's survival rate is the highest, basically will not encounter a dead IP can not be used.

Guidelines on demining of common problems

Practical Tips and Tricks

1. Remember to set a timeout so that the program doesn't die waiting:

proxy.web(req, res, {
  target: 'http://your-backend-server.com',
  timeout: 5000 // 5 seconds to pinch the line
}).

2. This is how to deal with HTTPS sites:

proxy.on('error', (err, req, res) => {
  res.writeHead(500, {
    'Content-Type': 'text/plain'
  });
  res.end('The proxy is jerking around, check it out!) ;
});

3. To record logs, you can hang a morgan middleware, or write a simple log middleware:

app.use((req, res, next) => {
  console.log(`${new Date().toISOString()} - ${req.method} ${req.url}`);
  next();
});

QA time

Q: What should I do if the proxy IP always fails?
A: It is recommended to use ipipgo's dynamic IP pool, their IP survival rate can reach 98% or more, automatic switching without worrying about the

Q: How to handle high concurrency scenarios?
A: Node.js itself event loop mechanism is suitable for IO-intensive, but to control the number of concurrency, you can work with async library parallelLimit

Q: How do I choose a proxy service provider?
A: Focus on three aspects: IP pool size, response speed, protocol support. Like ipipgo support socks5 and http dual protocols with more flexible

Finally, to do crawlers and other things to comply with the rules of the site, do not get hung up on their servers. Proxy IP should also pay attention to the basic law, we are technical exchanges, do not take it to do bad things ah!