Curl sends a POST request from a file: Curl File Submission

How easy is it to transfer data with files these days?

Old iron must have encountered this scenario: to upload a large pile of form data to the server, manually knocking parameters can exhaust individuals. That's when you need to use curl's-d @file pathThe big idea is to store the data directly in a txt file and stuff it like a feed to the server. To give a chestnut:

curl -X POST -d @order.json https://api.example.com

Note that the @ symbol is not a decoration, without it curl will not recognize you. In the file, you can honestly presskey=valueformat, or just go to JSON format. If you have a picky server, remember to add a-H "Content-Type: application/json"of the request header.

How does a proxy IP group CP with curl?

Now for the main point - how to make curl requests go out with a proxy IP "mask". After the command line, add a-proxyParameters will do the trick:

curl --proxy http://username:password@ip:port -X POST -d @data.txt https://target.com

I'd like to introduce you to ouripipgoservice, it has been tested that their dynamic residential proxies are particularly suitable for high-frequency requests. It is recommended to store the proxy information in a separate configuration file, such as a file calledproxy.conf::

proxy=http://user123:abcde@gateway.ipipgo.net:31028
retry=3
timeout=10

The call is made directlycurl -K proxy.conf ..., it's a lot less work than knocking on the command every time. Emphasis added three times:Passwords should be encrypted.,Passwords should be encrypted.,Passwords should be encrypted.! You can use openssl to make a base64 encoded anti-snooping.

Diary of a real-world pit stop

Last week, I was helping a customer docking logistics system and encountered an evil problem: uploading CSV files with proxy IP always reported 413 errors. Later I realized thatipipgoBy default, some nodes limit the size of the request body, just turn on the "large file transfer" switch in the console. Here is a list of common errors:

You must be asking QA

Q: What should I do if I can't connect to the proxy IP all the time?
A: First, remove the proxy parameter and make a direct request to confirm whether it is a local problem or a proxy problem. If it is a proxy problem, measure the speed in the "node diagnosis" function in the background of ipipgo and change the node with low latency.

Q：When uploading files, the speed is slow as a dog?
A: Two tricks: 1) Use the-Compressed.Parameter Enable Compression 2) Select nodes of the same carrier in ipipgo console. It can shorten the transmission time by more than 60%.

Q: What if I need to use more than one agent at the same time?
A: Write a shell script to poll the list of proxies with the-proxyThe parameters are called in a round robin. Remember to buy a package from ipipgo with enough "concurrent sessions" so that you don't get limited.

Say something from the heart.

Engaged in technology is afraid of deadlock, like proxy IP infrastructure or to the professional team reliable. I have used seven or eight service providers, and the last long-term renewal is stillipipgo. Their smart router is really good at avoiding IP segments that are blocked by target websites. Recently, there is also a "request fingerprint camouflage" function, which is particularly useful for anti-crawler.

A final reminder for newbies: the testing phase should always be used-verboseParameters look at the detailed log! Sometimes the error message is hidden in the response header and not visible to the naked eye. If you encounter a weird problem, submit a request log in ipipgo's work order system, their tech support responds much faster than some cloud vendors.