[erlang-questions] Packets deduplication

dmkolesnikov@REDACTED dmkolesnikov@REDACTED
Thu Feb 18 19:28:24 CET 2016

Hello Alexandr,

The right data structure is either bloom filter or scalable bloom filter.

I've played with standard bloom here

20k might require you to do tuning and reflect the filter to ETS.

The ETS as such or any cache based on ETS might help you as well. Cache is needed to implement TTL for your ID's 

Best Regards,

Sent from my iPhone

> On 18 Feb 2016, at 14:01, Alexander Petrovsky <askjuise@REDACTED> wrote:
> Hi!
> I have the stream of packets with ID (int), and I need to check is the packet is uniq (by ID) or not?
> Incoming rate is about 20k pps and ID is monotonically grows. What's the best way and data structure fit for this problem?
> -- 
> Петровский Александр / Alexander Petrovsky,
> Skype: askjuise
> Phone: +7 914 8 820 815
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

More information about the erlang-questions mailing list