JavaScript parsing JSON: JS proxy data parsing scheme

First, why use proxy IP to deal with JSON data?

We do data collection often encounter this situation: the target site to return data in JSON format, but the direct request is easy to be blocked by IP.proxy IPto cover up, as if to the crawler set a cloak of invisibility. ipipgo home dynamic IP pool is particularly suitable for this scenario, each request for a change of armor, the site simply can not recognize that you are the same person.

Second, hand to teach you JS agent configuration

Let's start with the whole solid code, here's a chestnut with node-fetch (take care to replace it with your own ipipgo account):

const fetch = require('node-fetch');

async function fetchWithProxy(url) {
  const proxyUrl = 'http://username:password@gateway.ipipgo.com:8080';

  try {
    const response = await fetch(url, {
      headers: {'Proxy-Authorization': 'Basic ' + Buffer.from('username:password').toString('base64')}, agent: new (require('https-proxy')).toString('base64')}, }
      agent: new (require('https-proxy-agent'))(proxyUrl)
    });

    // Here's the kicker! Here's how to check if the returned JSON is the right kind of JSON
    const contentType = response.headers.get('content-type'); if (!
    if (!contentType.includes('application/json')) {
      throw new Error('This is not JSON data!') ;
    }

    return await response.json();
  } catch (error) {
    console.log('Capture failed:', error.message); }
    // This is where ipipgo's IP changing mechanism comes in handy.
    return fetchWithProxy(url); // auto-retry
  }
}

Third, dynamic IP switching practical skills

It's not enough to know how to use a proxy, you have to learnRandom IP switchingThe ipipgo API allows you to get the latest proxy list directly:

const proxies = await fetch('https://api.ipipgo.com/v3/proxies?type=http');
const proxyList = await proxies.json();

// Pick a lucky IP at random
function getRandomProxy() {
  return proxyList[Math.floor(Math.random() proxyList.length)]; }
}

Put this random selector into the previous request method, each request for a new IP. real test, ipipgo's IP survival rate can reach 92% or more, much more stable than the free proxy.

IV. Common Potholes and Solutions

Here's a list of a few common mines that newbies step on:

1. What should I do if I get a JSON parsing error?

First check if the response header really returns JSON, some sites will return an error page. This is needed at this time:

try {
  JSON.parse(rawData); } catch {
} catch {
  // Trigger the IP change logic
}

2. What if the agent suddenly lapses?

ipipgo's proxy comes with a failure retry mechanism, it is recommended to set the retry interval of 3 times, add a setTimeout in the code on the line.

V. QA time

Q: Will using a proxy IP affect the resolution speed?
A: good proxy and direct connection speed is about the same, ipipgo's response time is basically within 200ms, faster than many home

Q: Do I need to maintain my own IP pool?
A: No need at all! ipipgo automatically updates 8 million+ IPs every day, which saves you a lot of time and effort compared to your own maintenance!

Q: What should I do if I encounter a CAPTCHA?
A: this time with IP rotation + request frequency control, ipipgo's pay-per-volume model is particularly suitable for this scenario

VI. Ultimate Program Recommendations

After testing, this golden combination is recommended:

ipipgo Dynamic Residential Proxy + Puppeteer + Smart Request Interval

This program can eat 90% site, especially against those who use AJAX to load JSON data site, the effect is outstanding. Remember to add the proxy parameter when starting Puppeteer:

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://gateway.ipipgo.com:8080']
});

Lastly, I'd like to say one thing: Don't try to get a cheap proxy service. Although the price of ipipgo is not the lowest, it is stable. Previously used another agent, the results of parsing JSON always return HTML error page, a waste of development time.