Node.js native fetch using proxies: request configuration

Hands-on with Node.js fetch with proxies

咱们搞爬虫的兄弟都懂，用原生fetch目标网站，分分钟就被封IP。这时候就得请出代理IP这个救兵了。今天不讲那些虚头巴脑的理论，直接上硬菜，教你怎么用Node.js最新版自带的fetch配代理。

Why do I need a proxy IP?

To give a chestnut, your courier wears the same clothes every day to deliver, the doorman is not quickly remember him? Proxy IP is like a variety of vests prepared for the courier, every time you change wear, the doorman will not recognize. UseipipgoThe dynamic IP pool of the home is equivalent to having hundreds of couriers on standby to collect data in a stable batch.

Three Steps to Configuring Agents

import fetch from 'node-fetch';
import { HttpProxyAgent } from 'http-proxy-agent';

// Replace this with your own ipipgo proxy channel
const proxyUrl = 'http://username:password@gateway.ipipgo.com:9021';

const response = await fetch('https://目标网站.com', {
  agent: new HttpProxyAgent(proxyUrl)
}).

take note ofThree priorities::
1. Proxy address should be in the right format, such as user name and password do not write the opposite
2. http and https protocols should be clearly differentiated
3. Do not copy the port number, it depends on the service provider to give the what

A practical guide to avoiding the pit

You can add a configuration item if you have problems with certificates:

const response = await fetch(url, {
  agent: new HttpProxyAgent(proxyUrl), rejectUnauthorized: false // Skip certificate validation.
  rejectUnauthorized: false // skip certificate validation
});

However, this is not safe and it is recommended that theipipgo consoleIt's only reliable to download their CA certificate and assign it to the system.

Proxy Parameter Comparison Table

Frequently Asked Questions QA

Q: What can I do if the agent can't connect?
A: First check the three pieces of the puzzle: address, port, and password. Use theipipgoThe in-line inspection tool measures the status of the channel.

Q: Requests suddenly slow down?
A: It may be that the current IP is limited, add an automatic IP switching logic in the code, or use the smart rotation API provided by their family.

Q: An ECONNRESET error occurs?
A: 80% of the proxy is not stable, change a high anonymous package. Suggest to tryipipgo EnterpriseThe exclusive line is as steady as an old dog.

I'll tell you what's on my mind.

Agents are used well and come home early from work. The key has to choose the right service provider, likeipipgoThis kind of real-time view of the remaining traffic, using the heart is solid. Their technical support reply speed is also fast, the last time at two o'clock in the middle of the night to mention the work order, ten minutes to solve.

Finally, to remind the newbie friends, the test phase first buy a pay-per-volume package, do not come up to charge the annual fee. Remember to do a good job of retrying errors in the code, do not let an IP hang up on the whole program. If you don't understand anything, go directly to their official website to find online customer service, report my name...forget it, report my name is not discounted (laughs).