JSON to CSV tool: proxy data format conversion

Hands-on teaching you to change proxy IP data from JSON to CSV

Those of you who often work with proxy IPs must have encountered this situation: the IP list you get from the service provider is in JSON format, but the crawler tool you use only recognizes CSV tables. This format mismatch is like trying to eat a steak with chopsticks - not impossible, but always awkward.

Take our ipipgo users as an example, the proxy data exported from the backend is a regular JSON structure by default. For example, it looks like this:

{
  "proxies": [
    {
      "ip": "203.34.56.78",
      "port": 8866,
      "protocol": "socks5", "location": "Xuzhou, Jiangsu Province", "徐州
      "location": "Xuzhou, Jiangsu"
    }, {
    {
      "ip": "118.23.45.67", "port": 3128, {
      
      "protocol": "http", "location": "shenzhen, guangdong" }
      "location": "Shenzhen, Guangdong"
    }
  ]
}

But if you want to stuff these data into Excel for screening, or imported into some only eat CSV format software, you have to come to a format change. Here to teach you two practical conversion methods, to ensure that more secure than the use of online conversion sites - after all, the proxy IP such sensitive data, do not just pass to a third-party site.

Python comes with a great format converter.

Prepare a text editor and create a new .py file. The following code is a conversion script written by our tech guy specifically for ipipgo users:

import csv
import json

 Remember to change the path to your own file
with open('ipipgo_data.json') as f:: data = json.load(f)['proxies']['proxies']['proxies']
    data = json.load(f)['proxies']

csv_columns = ['ip', 'port', 'protocol', 'location']

with open('output.csv', 'w', newline='') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
    writer.writeheader()
    for row in data.
        writer.writerow(row)

After running this program, there will be multiple output.csv files in the same level directory. Open with Excel to see the regular table, but also by region, protocol type to do filtering. For example, if you want to find a socks5 agent in Jiangsu, you can sift through the table directly and get it done.

Why is it recommended to store proxy IPs with CSV?

Here are a few tangible benefits:

Especially with ipipgo's proxy pool, often need to screen different regions of the IP according to business needs. into a CSV, even if the operation of the colleague to deal with will not be blind, after all, the table operation is a person will.

Frequently Asked Questions QA

Q: What about nested data in JSON?
For example, some agents have validation information in their data:

"auth": {
  "username": "ipipgo_user",
  "password": "123456"
}

Add a field like 'auth.username' to csv_columns when processing, and the code will expand it into a separate column with the corresponding processing.

Q: What should I do if I encounter garbled codes when converting?
Add an encoding parameter to the open function, e.g. encoding='utf-8-sig', which is especially useful when dealing with proxy regions with Chinese characters.

Q: Can I automatically time the conversion?
Use Windows task planner or Linux crontab with ipipgo's API to get the latest proxy list automatically, and update the CSV file automatically at dawn every day.

Hidden benefits for ipipgo users

In fact, we've prepared aQuick Export ChannelIf you want to convert your data to CSV format, you can click [Data Export] → [CSV Format] after logging in and get a ready-made form file in three seconds. This feature is specifically for users who often have to do data analysis, than with a script to convert much more trouble.

If you need to interface with other systems of the scene, it is recommended to use ipipgo API directly. in the request parameter add a format=csv, the return is ready-made CSV data stream, eliminating the need for conversion steps. Specific documents in the user's background in the [Development Guide], according to the sample code can be used to change.

Lastly, I would like to say one thing: Proxy IP data involves account security, so pay attention to the file storage location when converting the format. Especially with the authentication information of the proxy, remember to delete the temporary files in time after the conversion, do not leave a backdoor for those who are interested.