Data de-duplication technology: BloomFilter algorithm application details

What happens when proxy IP pools hit 10 million data?

Friends who have engaged in proxy IP pools should understand that each time the IP address is captured back like cabbage in the market - a large amount of pipe enough but repeated more. Last week, an old man said he used a traditional database de-duplication, the results of millions of data directly into the PPT card. this time we should offer us today to talk about theBloomFilterThis big Buddha now.

There's something special about this sieve.

Imagine you have a magic sieve, pour a basket of IP addresses into it, and the duplicates will automatically disappear.BloomFilter is the principle, but is much more memory-efficient than traditional databases. Specifically:

Here's the kicker! When using ipipgo's dynamic IP service, their IP pool updates tens of thousands of addresses every hour. At this time with the traditional method of de-duplication, the server will explode in minutes. And BloomFilter is like a smart security guard that can instantly recognize which are the new IPs that have just been added to the pool.

Hands on to build a de-duplication system

Here's a real-world example: let's say we're dealing with ipipgo's multi-million IP repository

Step 1: Pick an appropriate number of hash functions. In general3-5That's enough, don't make too much of it to affect performance.

Step 2: Calculate the bit array size. There is a simple formula:m = - (n ln p)/(ln2)^2, where n is the number of elements and p is the desired misclassification rate. For example, a million data wanting 1% misclassification would require roughly 1.7MB of memory.

Step 3: Write a timed cleaning mechanism. Because most of ipipgo's IP validity period is 4-6 hours, let's set the filter to be cleared every 2 hours to avoid expired IP residue.

Guide to Avoiding Pitfalls and Practical Tips

A common mistake newbies make isUsing parameters blindly without looking at the business scenario. For example, if you do real-time verification, the false positive rate should be pressed below 0.1%; if you do historical data analysis, 1% is acceptable.

Recommended for ipipgoCity-level IP librariesDo the test, their addresses are clearly categorized and it's easy to verify the de-duplication effect. There's a tricky way to do this: import known duplicate IP segments and see the filter's block rate.

And here's a cold one: the BloomFilter miscue is actuallyone-dimensional. That is, it may mistake a new IP for an old one, but it will never miss a true duplicate item. This is rather safer for proxy IP pool management - at best it wastes a bit of resources and doesn't use duplicate IPs.

QA time

Q: Does a high misclassification rate affect business?
A: Depends on the specific scenario. If it is a dynamic IP service like ipipgo, the IP is inherently time-sensitive, and it is recommended to use it with the timed refresh mechanism.

Q: How to choose a hash function?
A: We recommend MurmurHash3, an algorithm that balances speed and even distribution, and there are ready-made open-source implementations on the Internet.

Q: What should I do if the IP address format is not standardized?
A: Use the standardized interface provided by ipipgo to process first, and convert both IPv4 and IPv6 to a unified format before processing.

Finally, when using ipipgo's proxy service, their API returns directly to theList of IPs after de-duplicationIt saves you the effort of tossing your own filters. Especially to do distributed crawler project, directly call the ready-made interface is much more cost-effective than building their own systems.