Crawling the Web with JavaScript: Browser Solutions

Handy Browser Scripting Data Gathering

Often partners ask, how to use JS to grab data in the browser is not blocked? This thing is not difficult to say, the key to find the right way. Let's nag some real today, teach you how to use the browser environment with a proxy IP stable local data capture.

Let's talk about the pain point, many sites are now installed "electronic eye", caught the same IP repeatedly visit the black. This time we need to invite our "cloak of invisibility" - proxy IP. take ipipgo home services, for example, his family IP pool has more than 200 areas of resources, with the use of the change is particularly convenient.

Playing around with proxies in the browser

Here's a trick to teach you guys, change proxy settings with a browser plugin. Don't be intimidated, it's actually just three steps:

// Start by installing a chrome.proxy API
chrome.proxy.settings.set({
  scope: 'regular', value: {
  value: {
    mode: 'fixed_servers',
    rules: {
      singleProxy: {
        scheme: 'http', host: 'gateway.ipipgo.com', // fill in their portal here
        host: 'gateway.ipipgo.com', // fill in their home portal here
        port: 9021
      }
    }
  }
}, function() {});

Note that the replacement of your own ipipgo account information ha. After setting up this, all requests from the browser go through the proxy channel, just like wearing a cloak of invisibility.

A guide to avoiding the pitfalls of real-world capture

It's not enough to be able to use proxies, you have to be strategic. Here are a few life-saving tips to share:

1. Don't wait too long for randomization

Don't send requests like a hungry wolf, add a random delay:

function randomDelay(min, max) {
  return Math.floor(Math.random() (max - min + 1)) + min; }
}

// Use it like this
await new Promise(resolve => setTimeout(resolve, randomDelay(1500, 4000)));

2. Request headers should be camouflaged

Remember to make the User-Agent look like a real person's browser, and don't use the default scripted UA. ipipgo's client has a ready-made list of UA's that you can just copy for homework.

Frequently Asked Questions First Aid Kit

Q: What should I do if I use a proxy and still get recognized?
A: eighty percent of the IP quality is not good, change ipipgo's exclusive IP pool, their family IP are real equipment raised, not easy to show up!

Q: What if I need to wait for the page to render for data loading?
A: on Puppeteer! With the proxy settings more fragrant, remember to use their dynamic port function, an account can open more than one channel does not fight!

Q: Encountering a CAPTCHA blocker?
A: ipipgo has a matching coding service, pass the CAPTCHA image to their API, and the results will be returned within 5 seconds!

Why do you recommend ipipgo?

Finally, new registered users remember to lead a 3-day trial, enough to run through the whole process. Encounter technical problems directly to their customer service, the speed of reply is faster than the delivery boy. Grab the data thing, tools with the right way, not to mention the doubling of efficiency, account security is not guaranteed?