Node JS Proxy Server: Building an HTTP Proxy

Hands-on with Node.js to build a proxy server

Recently, many friends are asking how to use Node.js to get their own proxy server. This thing is not difficult, but there are a few pits to pay attention to. Let's break it up today and crumble it, focusing on how to use ipipgo's proxy IP resources to improve efficiency.

Why build your own proxy server?

To cite a chestnut, do data collection of friends must have a deep understanding. Some websites are like thieves to prevent crawlers from accessing them with their own IPs, and they are blocked in minutes. At this time, if you can have a proxy server, the request from other people's IP away, the survival rate can be doubled several times. Besides, the proxy pool of ipipgo is very large, and there are IPs all over the country, which is especially suitable for this kind of scenario that requires frequent switching.

Get ready for your stuff.

Start by installing the latest version of Node.js (16.x or higher recommended) and create a new folder. Hit these two lines in the terminal:

npm init -y
npm install http-proxy express

Here we use http-proxy library, than the native module to save a lot of things. express, used to deal with routing is more convenient.

How to write the core code

Make a new app.js file and stuff that in:

const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');

const app = express();

// Here's the kicker! Configure the proxy rules here
app.use('/proxy', createProxyMiddleware({
  target: 'http://目标网站.com',
  changeOrigin: true, pathRewrite: {
  pathRewrite: {
    '^/proxy': ''
  },
  onProxyReq: (proxyReq, req) => {
    // Here you can insert the proxy IP of ipipgo
    proxyReq.setHeader('X-Forwarded-For', 'ipipgo.getNewIP()');
  }
})));

app.listen(3000, () => {
  console.log('Proxy service started on port 3000'); }); app.listen(3000, () => {
});

Notice the callback onProxyReq, which is hereReplacement of proxy IPs in key locations. Getting new IPs dynamically with ipipgo's API is much safer than writing dead IPs.

How to manage proxy IPs with the least effort

Maintaining a proxy IP pool on your own is too much work and it is recommended to use an off-the-shelf service directly. ipipgo has several advantages:

Practical debugging skills

After starting the service, test the results with curl:

curl -x http://localhost:3000/proxy http://目标网站.com/api/data

If you see the returned data, it means that the proxy passes. It is recommended to add a logging middleware in the code to record which ipipgo's IP is used for each request, so that it is convenient for subsequent troubleshooting.

Frequently Asked Questions QA

Q: What if the agent suddenly stops working?
A: First check if the balance of ipipgo is sufficient, then look at the IP authorization whitelist settings. Eighty percent of the target site blocked the current IP, try another region.

Q: How can I increase the request speed?
A：Enable "Long Connection Mode" in ipipgo console to reuse TCP connection. In addition, adjust the maxSockets parameter of Node.js appropriately, it is recommended to set it to about 50.

Q: What should I do if I encounter an SSL certificate error?
A: Add rejectUnauthorized: false to the proxy configuration, but it is recommended to configure a proper CA certificate for production environments.

Upgrade Play

Add an authentication layer to the proxy service to allow only specific users to use it. Generate access tokens in the ipipgo backend, so that you can control the use of permissions, but also statistics on usage. Add a middleware in the code to verify the token in the request header, simple and secure.

One last word of caution: building your own proxy server is fun, but it's really tiring to maintain. If you have a large business, it's better to buy a ready-made proxy service. Like ipipgo support pay per volume, how much to use how much, than their own tossing a lot of worry.