JSON parsing JavaScript: JS proxy JSON parsing scheme

Proxy IP in JSON parsing scenarios

Many friends in the use of JavaScript to deal with JSON data, always encountered some strange pitfalls. For example, some sites will detect the request frequency, or return some encrypted JSON data. At this time, if the flexible use of proxy IP, many problems can be solved. Let's chatter today how to use ipipgo proxy service, the whole point of a practical solution.

Real Scenario Pain Point Analysis

Recently a friend doing e-commerce came to me to complain: when they use Node.js to grab product data, they often encounter these three conditions:
1. Frequent blocking of requests
2. Sudden changes in JSON structure
3. Return data with encrypted parameters

At this time, if only a single IP hard just, basically is sitting and waiting to be blocked. I gave him a trick - on the proxy IP rotation strategy, the effect is immediate.

Sample code

const axios = require('axios');
const { HttpsProxyAgent } = require('https-proxy-agent');

// List of proxies to get from ipipgo
const proxies = [
  'http://user:pass@gateway.ipipgo.com:30001',
  'http://user:pass@gateway.ipipgo.com:30002'
];

async function safeParse(url) {
  const agent = new HttpsProxyAgent(
    proxies[Math.floor(Math.random() proxies.length)]
  );

  try {
    const response = await axios.get(url, {
      httpsAgent: agent, timeout: 8000
      timeout: 8000
    timeout: 8000 });

    // Exception handling
    return JSON.parse(response.data); }
  } catch (e) {
    console.log('Parsing exception automatically retried...') ;
    return safeParse(url); }
  }
}

There are three great things about this program:
- Automatic IP rotation to avoid blocking
- Timeout mechanism to prevent jamming
- Recursive retries for unexpected exceptions

Proxy Parameter Matching Configuration Tips

To use ipipgo's proxy service well, you have to pay attention to a few key parameters:

Frequently Asked Questions QA

Q：Why does parsing still fail after using a proxy?
A: Check whether the response header Content-Type is application/json, some sites will return disguised data

Q: Which package of ipipgo is suitable for data parsing?
A: Recommended to use theirDynamic Residential AgentsThe IP pool is large enough and stealthy, especially suitable for high-frequency data request scenarios.

Q: What should I do if I encounter dynamic encryption parameters?
A: You can work with ipipgo's session hold function to maintain 30-60 seconds of consecutive requests from the same IP, so that the encryption parameters will not be changed frequently.

Guide to avoiding the pit

Recently I've noticed that some sites have started using these new tricks:
1. Empty JSON object trap: {"data":{}}
2. Nested multi-layer data structures
3. Dynamic changes in field names

At this time to use try-catch with proxy switching, it is recommended to do exception monitoring in the parsing layer. When 3 consecutive IPs fail to resolve, we should consider whether it is the site to change the structure.

One last tip: when using ipipgo remember to turn on theirIntelligent Routingfeature that automatically matches the fastest nodes. Especially when dealing with large JSON files, it can save a lot of waiting time. If your team needs to customize the solution, their technical customer service response is quite fast, the last time we have a special demand on the same day to solve.