curl send POST request | form data submission tutorial

Hands-on with curl to play with POST requests

engaged in interface testing know, curl this tool is like a Swiss army knife as omnipotent. Today we focus on nagging how to use curl POST request to submit form data, incidentally, the use of proxy IP magic. First on the basic command:

curl -X POST -d "username=test&password=123456" http://example.com/login

here are-X POSTSpecify the request method.-dFollowed by the form data to be submitted. Note that the parameters should be connected with & symbols, do not use spaces, otherwise the server will not receive the complete data.

The right way to open a proxy IP

Directly naked to send a request is easy to trigger the site protection mechanism, this time the proxy IP on the field. Take ipipgo's service for example, first get a proxy address:

Add proxy parameters to curl:

curl -x http://ipipgo_user123:passw0rd!@123.45.67.89:8080
     -X POST -d "content=test data" http://api.example.com/submit

expense or outlay-xThe parameter specifies the proxy server in the format "protocol://account:password@IP:port". ipipgo's proxy pool is updated frequently, which is especially suitable for scenarios where a large number of IPs need to be changed.

Form Submission Pitfalls and Countermeasures

Ever had a form submit and return a 403 error? Eighty percent is that the request header is not set properly. The full pose should be:

curl -x http://ipipgo代理地址
     -H "Content-Type: application/x-www-form-urlencoded"
     -H "User-Agent: Mozilla/5.0"
     -X POST -d "phone=13800138000&code=6688"
     http://service.example.com/verify

There are two key points here:The Content-Type must be set correctly.Otherwise the server does not recognize the data;User-Agent to masquerade as a browserThe default UA of curl is recognized in minutes.

Real-world scenario case analysis

Take a real case: the price monitoring of an e-commerce platform. Use ipipgo's rotating proxy with curl script to crawl every hour:

! /bin/bash
for i in {1..50}
do
   curl -x http://ipipgo动态代理
        -d "product_id=100${i}&type=price"
        -X POST https://mall.example.com/api
        >> result.log
   sleep 60
done

Here the dynamic proxy automatically switch IP, to avoid a single IP high-frequency access to the ban. ipipgo's concurrent connections support to 5000 +, engage in batch operation is not afraid of jam.

Frequently Asked Questions QA

Q: What should I do if the proxy keeps timing out the connection?
A: First check if the proxy address is correct, it is recommended to use the API provided by ipipgo to get the list of available proxies in real time. The timeout can be set with the -connect-timeout parameter

Q：What should I do if I submit garbled Chinese data?
A: Precede the -d parameter with a-data-urlencode, or wrap the data in quotes: -d "name=$(printf %s 'Zhang San' | jq -sRr @uri)"

Q: What if I need to go to HTTPS proxy?
A: Just replace the http with https in the -x parameter. ipipgo's proxy servers all support dual protocols, and remember to pair the certificates.

Proxy Services Selection Guide

You have to look at a few hard indicators to choose an agency service:

• IP pool size: ipipgo has over 2 million nodes worldwide
• Connectivity: Measured to remain above 99.2%
• Protocol support: HTTP/HTTPS/Socks5 full compatibility
• Authentication: In addition to basic authentication, IP whitelisting is also supported.

Recently, I helped a client to do a stress test, and used ipipgo's exclusive proxy to run for 72 hours without dropping the line, which is really more reliable than some cheap services.