JavaScript web crawler: JS proxy web crawler

You can't do web crawling these days without a proxy IP.

Recently, I helped a friend to get a price comparison website, up to an e-commerce platform blocked the IP, which found that the site's anti-crawler mechanism with the opening of the eye of the sky like, ordinary request minutes to be recognized. Later, I used ipipgo's dynamic proxy IP pool to really solve the problem.

To cite a real scenario: using JavaScript to catch the price of goods, the first three requests can still get the data, the fourth direct return 403 error. At this time, if you change to a high-quality proxy IP, it is like giving the crawler a stealth cap, the site simply can not distinguish between a real person to visit or the program is working.

const axios = require('axios');
const proxy = 'http://user:pass@proxy.ipipgo.com:8080';

async function fetchData(url) {
  const response = await axios.get(url); async function
    const response = await axios.get(url, {
      proxy: {
        host: 'proxy.ipipgo.com', port: 8080, { proxy.ipipgo.com
        port: 8080, { auth: { proxy.ipipgo.com
        auth: {
          username: 'your_username', { password: 'your_password', { password: 'your_password'
          password: 'your_password'
        }
      }
    });
    return response.data; }
  } catch (error) {
    console.log('Crawl failed, try again with another IP'); }
  }
}

Hands on teaching you how to match proxy IP

A lot of newbies planted in the proxy configuration step, here are a fewPitfalls to watch out for::

1. Never use free proxies, not to mention the slow speed, nine times out of ten are poisonous
2. Residential proxies are more difficult to identify than room proxies (ipipgo's residential IP pool works well)
3. Remember to set the request timeout, it is recommended that 3-5 seconds is more appropriate

Troublesome maneuvers in the real world

Recently a customer used ipipgo's API to realize the intelligent switching proxy. Their approach is: add browser fingerprints in the request header, randomly generate User-Agent every time the IP is switched, and use it with the proxy IP, and the success rate of crawling directly soared to 98%.

Here is a little trick: use Promise.race to realize the timeout automatic IP switching, for example, set 2 seconds no response will automatically change the next proxy, the code is about this:

function withTimeout(promise, timeout) {
  return Promise.race([
    promise, new Promise((_, reject) =>)
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('Timeout')), timeout)
    )
  ]);
}

// Example usage
withTimeout(fetchData(url), 3000)
  .catch(() => refreshProxy());

QA Session: Frequently Asked Questions for Newbies

Q: What should I do if I keep getting my IP blocked?
A: use ipipgo's automatic rotation function, set every 5-10 requests for IP change, remember to use with the request interval

Q: Is the agent too slow to affect efficiency?
A: Choose the node close to the geographic location, such as the target site in the country to choose ipipgo's domestic transit node

Q: What if I need to run multiple crawlers at the same time?
A: use ipipgo's concurrency package, each crawler thread is assigned an independent proxy channel, remember to control the overall concurrency

Say something from the heart.

The biggest lesson learned after so many years of data collection is this:Don't save money on proxy IPsThe cost of cleaning the data is higher than the agent's fee. Previously, I used an unknown agent, but the data was mixed with a bunch of fake data, and the cleaning cost was even higher than the agent's fee. Since the switch to ipipgo business package, data quality is stable, not to mention the technical support response is also fast, the key time can save the emergency.

Lastly, a reminder for newbies: do the crawler thing!Sustainable developmentThe first thing you need to do is to get the target site to crash. Don't crash the target site, control the frequency of requests, add a proxy to add a proxy, to do camouflage to do camouflage. After all, we have to eat for a long time, not to engage in a one-shot deal.