Puppeteer Web Screenshot: Puppeteer Agent Screenshot Solution

First, Puppeteer screenshots for what to build a proxy?

Recently, some of my friends who are doing data capture asked me what to do if the target website always blocks the IP of the screenshot with Puppeteer. This thing is like eating hot pot is spicy to the throat, you have to find the right way to solve the spicy method. For example, if you continuously take screenshots with the same IP address, the website will immediately post a"Suspicious visits"The labeling of the company is a direct ban no matter what.

At this point the proxy IP is the equivalent ofcloak of invisibilityIf you change your clothes every time you take a screenshot, the website won't recognize who you are. Like our commonly used ipipgo dynamic residential agent, each request can automatically switch IP address, than the supermarket cash register change is also sharp.

Second, the actual operation: to Puppeteer wear a cloak of invisibility

First of all, we need to understand how to plug the proxy IP into Puppeteer. Here's the key parameter--proxy-server, dropping it into the startup parameters is akin to putting a mask on the browser:

const puppeteer = require('puppeteer');

async function screenshotWithProxy(url) {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=http://用户名:密码@ipipgo proxy server address:port'
    ]
  });

  const page = await browser.newPage();
  await page.goto(url); await page.screenshot({ url)
  await page.screenshot({path: 'example.png'});
  await browser.close(); }
}

Note that there is a pitfall here, many newbies directly copy the online code, and as a result, they miss theauthenticationThis step. ipipgo's proxy service requires the account password to be embedded in the proxy address in a format similar to filling in an address for a courier, which must be accurate to the door number.

Third, the proxy IP selection doorway

There are several types of agents on the market, let's compare them in a table:

If you are doing 24/7 screenshot tasks, Rift recommends ipipgo's residential proxies. Their IP pool is as big as a swimming pool, and they assign new IPs with every request, so there's no fear of being blocked.

IV. Guide to avoiding the pit: 5 common rollover sites

1. What if the screenshot always fails?
First check the proxy address is not a mistake, especially the colon, slash and these symbols. It is recommended to directly copy the sample code provided by ipipgo background, guaranteed not wrong.

2. What should I do if the page is not fully loaded?
Add a waitUntil parameter after page.goto(), for example:
await page.goto(url, {waitUntil: 'networkidle2'});
This is the equivalent of waiting for the page to finish loading before taking a screenshot.

3. What happens when a proxy suddenly fails?
It may be that the IP has been pulled by the target website. This is the time to turn on the automatic rotation function of ipipgo, just like a car shifting gears, automatically changing to a new IP every few minutes.

V. QA First Aid Kit

Q: Is it okay to use a free proxy?
A: Never! The free agent is just like the toilet in the public restroom, who have used it. Before a buddy trying to save trouble with free agents, the results of the screenshot is full of gambling ads, the site directly to him blocked.

Q: How are ipipgo proxies billed?
A: They have two kinds of packages according to the flow and the number of IPs. If you do screenshots such as the need to change IP frequently, it is recommended to choose the number of IP packages, just like a buffet, randomly change without pain.

Q: How do I hide Puppeteer features when taking screenshots?
A: Add these parameters at startup:
args: ['--disable-blink-features=AutomationControlled']
Combined with ipipgo's proxy, it basically masquerades as a normal browser.

As a final rant, making automated screenshots is about astable characterThe first thing you need to do is to choose the right proxy service provider. Choose the right proxy service provider will be half successful, like ipipgo can provide API real-time extraction of the proxy, with the use of driving an automatic transmission car like, save heart and effort. If you don't understand anything, go directly to their official website to find customer service, the speed of reply is faster than the delivery of food by the delivery boy.