Puppeteer gets all p-tags: proxy IP for automation

When crawlers meet CAPTCHA? Try this combo

Recently, a friend who is an e-commerce company complained to me that they were always recognized as bots when they used Puppeteer to capture product details. I am familiar with this situation! Last year, when I was helping someone to make a price comparison tool, I encountered the situation of IP being blocked every three days. Then I found a trick--Putting Proxy IP Armor on Puppeteer, the success rate is directly doubled.

const puppeteer = require('puppeteer');
const ipipgoProxy = 'http://user:pass@gateway.ipipgo.com:9021';

(async () => {
  const browser = await puppeteer.launch({
    args: [ `--proxy-server=${ipipgoProxy}` ]
  });
  const page = await browser.newPage(); await page.goto(''); }
  await page.goto('https://目标网站.com');

  const paragraphs = await page.$$eval('p', elements =>
    elements.map(el => el.innerText)
  ); console.log(paragraphs);
  console.log(paragraphs);

  await browser.close();
})().

Proxy IP Selection with Care

There are all kinds of proxy services on the market, but you have to pay special attention to three things with Puppeteer:

I've used ipipgo before.Dynamic Residential IP PoolThe function of automatically changing IP per request is really fragrant. Especially suitable for the need to continuously operate multiple pages of the scene, do not have to manually switch this point to save a lot of things.

A practical guide to avoiding the pit

Last week, I stepped into a pit when helping a customer do news gathering: obviously, I used a proxy, but it still triggered the CAPTCHA. I found that the browser fingerprint was exposed, and then added these two configurations to get an immediate effect:

const browser = await puppeteer.launch({
  headless: true,
  args: [
    '--disable-blink-features=AutomationControlled',
    ` --proxy-server=${ipipgoProxy}`
  ]
});

Remember to randomize the settings in the codeUser-AgentThe ipipgo backend can directly download ready-made UA listings, a detail handled quite thoughtfully.

Frequently Asked Questions First Aid Kit

Q: What should I do if my proxy IP suddenly fails?
A: Check if the account balance is sufficient, and it is recommended to turn on ipipgo's auto-renewal feature. Contact their technical support directly in case of emergency, the response speed is about half an hour faster than the counterparts.

Q: How do I assign IPs to multiple browser instances open at the same time?
A: Use ipipgo's API to dynamically obtain IP pools and assign independent proxies to each instance. Their interface return speed control within 200ms, measured than directly write a dead configuration much more flexible.

Efficiency Improvement Tips

I recently discovered that the ipipgo backend has aIntelligent Routingfeature that automatically selects the node with the lowest latency. With Puppeteer's request interception function, loading speed increase of 40% is not a dream:

await page.setRequestInterception(true);
page.on('request', request => {
  if(request.resourceType() === 'image')
    request.abort();
  request.abort(); else
    request.continue();
});

This method is particularly suitable for the collection of plain text content, to save the bandwidth used to maintain the stability of the proxy connection, pro-test effective.

Say something from the heart.

Using a proxy IP is like wearing a protective suit, good or bad quality directly affects the survival rate. After several projects to test, ipipgo in thehigh concurrency scenarioThe performance under it is really solid, especially their IP cleaning algorithm, basically there is no case of just buying an IP and having it blocked. The recent double eleven event seems to have a 30% discount, friends who need to renew can go and take a look.