First, ten million proxy IP data stored for what purpose? Why do we have to optimize?
We do crawler brother understand, hand no millions of proxy IP are embarrassed to go out. But really saved to the ten million quantitative time, the problem comes--Normal databases just fall apart for you.The other day, an old iron told me that they used MySQL to store 8 million IPs, and it took half a minute to check the available IPs. A couple days ago, an old iron told me that they used MySQL to store 8 million IPs, and they had to wait half a minute to check the available IPs, so what's the point of playing with a hammer?
The most damning thing here are the three pits:
1. Queries crawl like a tortoise when the volume of data is large
2. Hard disk space is running out.
3. Rising maintenance costs
Second, the practical school of storage optimization three axes
Tip #1: Turn the whole thing into pieces
Don't put the eggs in one basket, let's cut the IP by geographical area. For example, the Beijing server room 1 section of the IP stored separately, Shanghai server room 2 section of the other storage. Take ipipgo's proxy pool as an example.Intelligent Segmentation TechnologyIt can automatically pack and store IPs in the same region, and directly locate the specific slice when checking, and the speed can be more than 5 times faster.
Tip #2: Check the RAM before the hard disk
Get a two-tier caching mechanism and put recently used IPs in Redis. Here's a little trick:
Hot data (used in the last 5 minutes) goes in the first tier, warm data (used on the same day) goes in the second tier, and only the rest goes to the database. The measured response time can be reduced from 3 seconds to 200 milliseconds.
| data type | storage location | response time |
|---|---|---|
| thermal data | memory cache | ≤50ms |
| temperature data | SSD hard drive | ≤200ms |
| cold data | mechanical hard drive | ≥1s |
Tip #3: Multi-threaded Parallel Queries
Don't be stupid and check the library in a single thread, open 10 threads at the same time to check different splits. Be careful to set thetime-out fuse mechanismIf you're not sure what you're looking for, you'll be able to find a way to get it to work for you, so don't let it drag the whole thing down. ipipgo's API interface has this built in to automate query assignment. 1. Go to the vital 2. Choosing the right compression algorithm 3. The Great Separation of Hot and Cold Q: Does IP de-duplication affect usage? Q: How to query the compressed data quickly? Q: Does sharding storage increase maintenance costs? It's too much work to toss storage optimization on your own, just go straight to theipipgo EnterpriseAnd it's done. Their home storage system has three killer features: The last time I helped a friend's company migrate to ipipgo, the original 20,000 per month server costs were cut directly to 4,000 per month. The key is theirData Visualization PanelDoing it like a thief, IP usage, survival rate and all that data at a glance. When it comes to data storage, it's better to leave the professional work to the professionals. It is better to build wheels from scratch than to stand on the shoulders of giants. Especially now that the competition in the proxy IP market is so fierce, wouldn't it smell good to save the time and money and take it to expand your business?Third, compression black technology to save 80% space
The same IP segment is represented by CIDR. For example, 192.168.1.1 to 192.168.1.254, directly written as 192.168.1.0/24, saving 90% storage space.
Tested and tested these work best:
- LZ4: fast compression but average compression rate
- Zstandard: the balanced player
- Brotli: highest compression rate but CPU intensive
It is recommended to choose according to business needs, to speed with LZ4, to save space with Brotli.
Transfer 30 days of unused IPs to cold storage with ipipgo'sIntelligent archiving functionAutomated Processing. Their home cold data storage costs can be reduced to 1/10th of hot data.IV. Frequently Asked Questions QA
A: No effect at all! The de-duplication is just a storage level optimization, the system will automatically expand it when you actually call it.
A: Recommended for ipipgosolve-it-and-find-ittechnique that does not unpack the entire dataset and directly locates the desired chunks of data.
A: It's more cost effective to use an off-the-shelf solution. For example, ipipgo's storage solution can be deployed in 10 minutes with an auto-sharding cluster.V. Recommendations for a heart-saving program
1. Intelligent compression algorithm automatically adapts to business scenarios
2. Distributed query engine supporting millisecond response
3. Automatic tiering of hot and cold data, the storage cost is reduced by 80%.
English 
简体中文 
Español 
Deutsch 
Français