Proxy IP combined with npm crawler module: Node.js crawler module proxy IP configuration

When Crawlers Meet Anti-Crawlers? Try putting a 'cloak of invisibility' on Node.js

Engaged in data crawling understand, the site's anti-climbing mechanism is like a neighborhood security - found that unfamiliar faces frequently in and out of the direct black. At this time it is necessary toproxy IPTo be a cloak of invisibility, so that the server can not recognize your true identity. We do not talk about false today, hand in hand to teach you how to play in the Node.js crawl module proxy configuration.

Don't run naked! Put a shield on axios.

Sending requests with axios is like surfing the web naked, you'll get caught by the target site in minutes. Using ipipgo's proxy service to dress the request is so easy to configure that you'll cry:

const axios = require('axios');
const tunnel = require('tunnel');

const agent = tunnel.httpsOverHttp({
  proxy: {
    host: 'api.ipipgo.com', //proxy server address
    port: 9000, //exclusive port
    proxyAuth: 'Your account:password' //authentication information
  }
}).

axios.get('Target URL', {
  httpsAgent: agent, {
  timeout: 5000
}).then(res => {
  console.log('Successfully cloaked! Data in hand');
});

Focused Reminder:Remember to replace the proxy authentication information with your own real credentials obtained from the ipipgo backend, don't be silly and copy the fake data from the sample code.

Headless browsers have to change their vests too

When you use a headless browser like puppeteer, many people think it's troublesome to open a proxy. In fact, with ipipgo's socks5 proxy, three lines of configuration is done:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=socks5://api.ipipgo.com:9001',
      '--disable-gpu'
    ]
  });
  //... Subsequent operations
})().

At this time your browser instance is like wearing a human skin mask, each visit will be randomly switched to the exit IP. measured with this method, an e-commerce platform for 3 hours of continuous collection did not trigger the verification.

A must-see guide to de-mining proxy configurations for the uninitiated

Why choose ipipgo?Old Driver Test Report

The market is a mixed bag of agency services, and after using seven or eight of them, theipipgoIt's captivated me on three key points:

1. Exclusive IP pool without serial number (measured 300,000 requests with 0 duplicates)
2. Response speed stabilized within 200ms
3. 7 × 24 hours technical customer service seconds back to the work order

The key thing is that their homeDynamic Residential AgentsThe first time I saw this, I was able to get a good deal on the website, and I was able to get a good deal on the website. Last time to help customers climb a government website public data, with other agents 5 minutes to be blocked, change ip ipgo hard to run all night.

Frequently Asked Questions First Aid Kit

Q: What should I do if my agent suddenly fails?
A: First check whether the account balance is sufficient, and then try to switch the protocol type (for example, from HTTP to SOCKS5). ipipgo has real-time usage monitoring in the background, and will actively push alerts for abnormal situations.

Q: How do I test if the proxy is working?
A: Add a test step in the code to visit http://ip.ipipgo.com/ip.php to see if the IP returned is a proxy IP.

Q: Do I need to maintain my own IP pool?
A: No need at all! ipipgo's API supports automatic rotation, each request with a ?cycle=1 parameter will be able to change the new IP, worry-free to burst.

Tell the truth.

Proxy IP is not a panacea, and works better with these tips:

• Random User-Agent don't be lazy (user-agents library recommended)
• Control request frequency not to be too wavy (add a sleep function)
• Buy exclusive IP for important projects (ipipgo Enterprise supports customization)

Finally, to remind newcomers: do not try to buy a cheap junk agent, light collection of data is incomplete, heavy anti-climbing mechanism anti-kill. In ipipgo new registration to send 2000 times trial, first test and then order is safe.