Node fetch sends a POST request: API call example

Hands-on with Node fetch to send POST requests

Recently, a lot of data collection buddies asked, using Node.js to send a request to the target site is always blocked IP how to do? This matter is in fact the same as playing the game was banned number a reason, the solution is also very simple ---The Great Proxy IP RotationThe first thing you need to know is how to use Node fetch to send a POST request. Today we will take ipipgo's service as an example and say how to use Node fetch to send POST requests without being blocked.

Basic equipment preparation

Install a node-fetch library first, don't be stupid and use the native module. Command line knock this:

npm install node-fetch@2 // Note that compatibility is better with older versions.

Prepare a reliable proxy service, demonstrated here with ipipgo's Dynamic Residential Proxy. One good thing about their home is thatAutomatic IP switchingYou don't have to maintain your own IP pool. After registering and getting the API key, the proxy address looks like this:

http://username:password@gateway.ipipgo.com:8080

Real-world code disassembly

Here's the point! The posture of the matching agent should be memorized:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

// proxy assignment value (be careful to replace your own account)
const proxyAgent = new HttpsProxyAgent(
  'http://你的账号_不带中文:密码@gateway.ipipgo.com:8080'
).

async function sendPost(url, data) {
  async function sendPost(url, data) {
    const response = await fetch(url, {
      method: 'POST', body: JSON.stringify(data), async
      
      headers: {'Content-Type': 'application/json'}, agent: proxyAgent // The key is in the proxyAgent.
      agent: proxyAgent // The key is to hook up the proxy here.
    }); return await response.json()
    return await response.json();
  } catch (error) {
    console.log('Request flopped:', error); }
  }
}

// Example usage
sendPost('https://api.某某网站.com/login', {
  username: 'test',
  password: '123456'
}).

Watch out for potholes:

• In the proxy accountDon't bring Chinese.Otherwise, authentication fails.
• If the target site uses HTTPS, remember to match the https-proxy-agent.
• Setting a timeout suggests adding a setTimeout, so you don't have to wait.

Common Rollover Scene QA

Q: What can I do if the agent can't connect?
A: First check the proxy address format is correct, use curl command to test it:
curl -x http://账号:密码@gateway.ipipgo.com:8080 https://httpbin.org/ip

Q：Return 407 Agent Authentication Error?
A: Ninety-nine percent of the account password is wrong, or the package has expired. Go to ipipgo background to see the remaining traffic

Q: How to realize the change of IP per request?
A: Open in the ipipgo consoleSwitch IP on requestmode without changing anything in the code

What to look for in an agency?

There are all kinds of proxy services on the market, but you have to recognize these points to do data collection:

Like ipipgo's.Dynamic Residential AgentsIt's more suitable for crawler scenarios, and although it's a bit more expensive than a server room agent, it's less likely to get banned. another great thing about their home is thatPrecise localizationFor example, if you want to collect weather data of a certain place, you can directly lock the exit IP of the corresponding city.

Tips for saving traffic

Finally, I'd like to share a great money-saving tip: ipipgo's traffic packages use thetime-based billingThe model is more cost-effective than the monthly subscription, especially suitable for the initial testing phase of the project. When the business is stabilized, then cut the monthly package, so it is not easy to spend money.

If you're in a pinch, you can start with theirs.trial package(New users get 1G traffic). However, be careful not to use to engage in violent crawling, the trial package of IP quality will be poorer, the official work also have to pay on the version.