Node Web Crawler: Puppeteer in Action

Why does the Puppeteer crawler always get blocked?

When many brothers use Puppeteer to grab data, they often come across the403 Denial of AccessorCAPTCHA bombing. Last month I helped a client to catch the price of e-commerce, just run half an hour IP was pulled. Later, I found out that it was the target website that recognized the crawler by three features: request frequency, device fingerprint, and most damaginglyRepeated visits from fixed IPsThe

The right way to open a proxy IP

Here's a tip for the guys: use theResidential Proxy Pool Rotation IP. For example, with ipipgo's dynamic residential IP, each visit automatically switches the exit address. The actual test of an e-commerce platform for 3 days without triggering the wind control, the key code is long like this:

const puppeteer = require('puppeteer');
const ipipgo = {
  host: 'gateway.ipipgo.net',
  
  auth: 'username:password' // remember to change to your own key
};

(async () => {
  const browser = await puppeteer.launch({
    args: [`--proxy-server=http://${ipipgo.host}:${ipipgo.port}`]
  });
  //... Subsequent operations
})().

Avoiding the tawdry maneuvers of fingerprint detection

It's not enough to change IPs, you have to learnMasquerading as a real person. Here's a practical skill combo to share:

Suggest adding random wait times to the code, don't open the page in seconds like a robot:

function humanDelay() {
  return Math.random() 2000 + 1000; // 1-3 seconds random wait
}

await page.waitForTimeout(humanDelay());

QA time: the pitfalls you may have encountered

Q: What should I do if my proxy IP often times out?
A: Preferred ipipgo'sLong-lasting static residential IPTheir lines support long connections, and their measured stability is 40% higher than that of ordinary dynamic IPs.

Q: How can I tell if an IP is exposed?
A: Add a detection link in the code, visit https://httpbin.org/ip, if the returned IP does not match the expected, immediately change the proxy

Q: What if I need high concurrency?
A: Use ipipgo'sMulti-Threading PackageWith the cluster deployment, pay attention to control the amount of requests per second do not exceed the threshold of the target site to withstand

Commissioning tips: Seeing is believing

It is recommended to add the startup parameterHeadless mode visualization debugging, see the crawler behavior first hand:

const browser = await puppeteer.launch({
  headless: false, //see the actual running screen
  slowMo: 50, //slow down the operation
  args: [`--proxy-server=http://${ipipgo.host}:${ipipgo.port}`]
});

Finally, we remind you to choose the agent service to recognize theipipgo this support auto switching + failure retry mechanismThe service provider. Last time I used their failover auto-switching feature, the crawl success rate directly soared from 67% to 92%, so fragrant!