[erlang-questions] Packets deduplication

[dmkolesnikov@REDACTED]

Hello Alexandr,

The right data structure is either bloom filter or scalable bloom filter.

I've played with standard bloom here
https://github.com/fogfish/feta/blob/master/src/bloom.erl

20k might require you to do tuning and reflect the filter to ETS.

The ETS as such or any cache based on ETS might help you as well. Cache is needed to implement TTL for your ID's 

Best Regards,
Dmitry

Sent from my iPhone

> On 18 Feb 2016, at 14:01, Alexander Petrovsky <askjuise@REDACTED> wrote:
> 
> Hi!
> 
> I have the stream of packets with ID (int), and I need to check is the packet is uniq (by ID) or not?
> 
> Incoming rate is about 20k pps and ID is monotonically grows. What's the best way and data structure fit for this problem?
> 
> -- 
> Петровский Александр / Alexander Petrovsky,
> 
> Skype: askjuise
> Phone: +7 914 8 820 815
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions