Puppeteer Proxy IP Configuration: Puppeteer Proxy Settings Tutorial

Puppeteer play the right posture of proxy IP

Crawlers should know that the old iron, browser automation tools Puppeteer, although good, but not a set of proxy IP is like running naked on the battlefield. Today we will nag how to Puppeteer wear a good "protective armor", focusing on how to use ipipgo's proxy service to work steadily.

Basic configuration of the three axes

Stuffing an args parameter into the launch method when starting a browser instance is the straightest ball of play. Note that you have to use the-proxy-serverThe parameters specify the protocol type and address, and the format has to be whole and correct to work:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    args: ['--proxy-server=http://用户名:密码@gateway.ipipgo.net:端口']
  });
  // Follow up...
})();

Here's a pitfall to watch out for:Protocol type alias string. For example, ipipgo's Socks5 proxy should be written as socks5:// at the beginning, and with http proxy, it should be written as http://. If you get the protocol header wrong, you won't be able to connect to the server in minutes.

There is something to be said for certified processing

When encountering proxies that require account password authentication, it is recommended to use the page.authenticate method to handle it. This is much safer than writing the password directly in the URL, especially when working in a team without leaking credentials:

const page = await browser.newPage();
await page.authenticate({
  username: 'ipipgo account', password: 'Exclusive password', 'password', 'password', 'password', 'password', 'password')
  password: 'Exclusive password'
}).

If you encounter an authentication failure, first check theAccount Validityrespond in singingWhitelistingipipgo's proxy is bound to use IP by default, remember to add the local public IP in the background. if you use a dynamic residential proxy, it is recommended to enable the automatic IP whitelisting function.

A practical guide to avoiding the pit

Here are a few bloody lessons to share:

1. When opening multiple pages, each page has to be authenticated individually
2. When running in headless mode, the proxy failure rate will be higher, it is recommended to turn on visual debugging first!
3. When encountering ETIMEDOUT errors, prioritize checking proxy package margins (don't laugh, some newbies really make this mistake)

Package Selection Comparison Table

Frequently asked questions on demining

Q: I can't open the web page even though the proxy is connected?
A: First remove the proxy to test the basic network, and then use the online testing tool provided by ipipgo to measure the proxy status. It may be that the target website has blocked the residential IP segment, try another country node.

Q: ERR_PROXY_CONNECTION_FAILED appears?
A: 80% is the protocol type mismatch. http proxy port can not connect with socks5 protocol, and vice versa. Check the connection information given by the console, and pay attention to the case of letters.

Q: How to realize the automatic switching of proxy?
A: It is recommended to use ipipgo's API to dynamically obtain the proxy pool, together with tools like puppeteer-cluster to do the rotation. The Enterprise Edition package supports adding load balancing parameters to the connection string to directly realize intelligent switching.

As a final note, when configuring the proxyNever use a free agent.The actual fact is that the actual business is not a serious one, but it's a good thing that you've got a lot of money. I've seen some people get cheap, and their accounts were blocked and they lost all their data. ipipgo's dynamic residential packages start at 7 bucks for 1G, which is cheaper than drinking milk tea, so there's no need to take that risk.