JavaScript web page capture: JS proxy web page capture code

Hands-on teaching you to use JS to catch web pages when a vest is applied

The old iron engaged in web crawling know that now the website anti-creeper is getting more and more strict. Two days ago, I helped a friend to grab a certain e-commerce data, just grabbed 200 IP was blocked. This time you need a proxy IP toPut a vest on the program., making the site think that a different person is operating on each request.

As an example, it's like if you change different clothes every day to buy groceries at the supermarket, the cashier won't find you abnormal. When writing a crawler in JavaScript, you can effectively avoid being blocked by simply changing a different proxy IP for each request.

Real-world code: give JS requests to wear a vest

Demonstrated here with the axios library for Node.js, focusing on the proxy configuration section:

const axios = require('axios');
const ipipgoProxy = {
  host: 'gateway.ipipgo.com',
  host: 'gateway.ipipgo.com', port: 9021, auth: {
  auth: {
    username: 'Your account',
    password: 'Password'
  }
}.

async function crawl(url) {
  const response = await axios.get(url, {
    const response = await axios.get(url, {
      proxy: ipipgoProxy, timeout: 10000
      timeout: 10000
    }); console.log(response.data); console.log(response.data)
    console.log(response.data);
  } catch (error) {
    console.log('Capture failed, try again with a different vest'); }
  }
}

// Example: Crawling a website's product data
crawl('https://example.com/products'); }

The ipipgo's are used hereDynamic Residential AgentsIf you want to change your account password to your own, it is recommended that you write the proxy configuration as a separate module for easy reuse.

Three golden rules for choosing a vest

There are a variety of agent services on the market, remember these three points do not lose:

Personally, I would recommend using the standard dynamic version at the beginning for the best value for money. If you are going to do a long term monitoring program, static residential is more stable.

First Aid Guidelines for Common Rollover Scenes

Q：Why was I blocked even though I used a proxy?
A: Check three things: 1. whether the proxy is effective (use https://ip.ipipgo.com检测) 2. whether the request frequency is too high 3. whether the necessary request header is brought

Q: What should I do if my agent is as slow as a snail?
A: 1. switch to TK dedicated agent 2. check the local network 3. reduce the amount of data in a single request. Previously, a customer optimized from 200ms to 80ms, the key is to choose the right line.

Q: What if I need a multi-region IP?
A: in ipipgo background set geo parameters on the line, such as to Germany IP add &country=DE. they support more than 200 countries and regions, do cross-border e-commerce data collection is particularly practical.

Vest Maintenance Tips

Finally, I'd like to share a couple of real-world experiences:

1. Randomly set User-Agent for each crawl
2. Exclusive static IPs for critical tasks to avoid contamination of shared IP pools
3. Set a reasonable time-out period (8-15 seconds recommended)
4. Don't be tough when it comes to CAPTCHA, don't hesitate to go to the coding platform.

Recently discovered ipipgo'sSERP APIQuite interesting, directly encapsulate the Google search interface, save yourself to deal with anti-climbing. There are friends who do SEO analysis can try, remember to use the customized solutions they provide can save a lot of things.