Node.js crawl: Node.js data crawl

Proxy potholes you must know to do Node.js crawling

Recently, I was helping a friend to build a price comparison website, and when I used Node.js to capture the data, I was always getting banned.Proxy IP playThe first thing you need to do is to get a good deal of information about the company. For example, a certain continuous capture of an e-commerce platform, less than half an hour was blocked, and then changed ipipgo's dynamic residential agent, immediately effective.

const axios = require('axios');
const tunnel = require('tunnel');

const agent = tunnel.httpsOverHttp({
  proxy: {
    host: 'gw.ipipgo.com',
    port: 9021,
    proxyAuth: 'Your account:password'
  }
}).

axios.get('https://target-site.com', {
  httpsAgent: agent,
  timeout: 8000
}).then(res => console.log(res.data))

What are the hard indicators to look for when choosing an agent

There are three types of agents on the market, so I'll give you a solid comparison table:

Residential Agents | Server Room Agents | Mobile Agents
— | — | —
Real User IP | Cloud Server IP | Cell Phone Base Station IP
High anonymity | Easily recognized | Medium anonymity
Suitable for long term tasks | Suitable for short bursts | For specific scenarios

Like ipipgo's large pool of residential proxies, I've tested the crawl for three days in a row without triggering a back crawl. Special attention should be paid toShelf lifeThis parameter, which some agents say is valid for 5 minutes, actually drops in 2 minutes.

Real-world proxy configuration of the tart operation

Remember to add startup parameters if you're using puppeteer, don't be stupid and run naked:

const puppeteer = require('puppeteer');

async function crawlWithProxy() {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=http://gw.ipipgo.com:9021',
      '--disable-gpu'
    ]
  });
  //... Subsequent operations
}

The most pitiful thing I've ever encountered isSSL Certificate IssuesSome sites will detect the proxy's certificate fingerprints. This time use ipipgo's HTTPS proxy solution, their family's certificates are regularly updated, save your heart.

Self-help Guide to Common Rollover Scenes

QA 1:What if the proxy suddenly fails?
First check the return status code, 403/429 should change IP. ipipgo API supports automatic switching, it is recommended to set up a failure retry mechanism.

QA 2:Slow as a snail to crawl?
Try concurrent requests + proxy pool rotation. But pay attention not to open too many threads, generally control in 10-20 concurrent, depending on the target site affordability.

QA 3:Do free proxies work?
Blood lesson! I've used free proxies before to save time, but the data was mixed with ad code. Now I use ipipgo's exclusive proxy, and the data quality is very stable.

It's all for naught if you don't pay attention to these details

1. In the request headerX-Forwarded-ForRandomize, don't use fixed values
2. Per-proxy IP recommendations5-10 minutesone-off replacement
3. Don't be tough when you encounter CAPTCHA, use ipipgo's overseas proxies to change the regional IP to try.
4. log remember to record the use of proxy IP, easy to troubleshoot the problem

Finally, a piece of cold knowledge: some sites will detect the mouse track, with headless mode remember to add theuser-agent masquerading. My common configuration scheme is ipipgo proxy + random UA library, which is a combo down to 90% sites.