[erlang-questions] How to do this in Erlang -- any suggestions ?

[Banibrata Dutta]

On Sun, Jun 12, 2011 at 10:16 PM, Darach Ennis <darach@REDACTED> wrote:

>
> An excellent book introducing event processing is Opher Etzion's Event
> Processing in action:
>
> http://www.manning.com/etzion/
>

Read reviews and some sample pages on google-books, looks very interesting.


> The Event Processing Technical Society also has a wealth of resources & use
> cases online:
>
> http://www.ep-ts.com/
>

Okay, will go through. One of my doubts still is, whether I really need a
CEP framework (based on little I've read on CEP so far -- mostly on Esper
site). Given your extensive background in this domain, and my requirements
(happy to answer more questions if needed), do you believe CEP is the way to
go ? Or, can I hope to achieve with a pure FSM based approach ?


> Generally speaking you will want to separate the mechanics of windowing and
> correlation/combination from their utilisation
>

I think I understand this part...


> so that you can compose synthetic events derived from external or
> 'ordinary' events easily and reuse windowed or other functions processed in
> the
>
context of a window or other correlation/combination.
>

...though not too sure about this. My external / raw events could be SNMP
traps, lines from log-file or even be OA&M activity. I intend to convert
them to a homogeneous form, which at-least identifies the event-type, some
kind of hierarchical position (if possible), and other data.

A good example of windowing in Erlang projects is slide.erl/spiraltime.erl
> in riak core:
>
> https://github.com/basho/riak_core/blob/master/src/slide.erl
>
> https://github.com/basho/riak_core/blob/master/src/spiraltime.erl
>
> Event processing engines such as the open source Esper engine or commercial
> StreamBase (disclaimer: I work there) provide documentation online that
> could also be good references.
>

Thanks for all the tips. I am sure I'd be spending sometime going through
some of that. I've gone thru the introduction to Esper -- definitely like
what I see, although I am not quite sure (yet), if I really need all of
that.


On 12 Jun 2011, at 16:38, Banibrata Dutta <banibrata.dutta@REDACTED> wrote:

> gr8 questions, and they certainly need clarification.
> cc'ing the group s.t. others could contribute too.
>
> On Sun, Jun 12, 2011 at 8:48 PM, Mihai Balea <mihai@REDACTED> wrote:
>
>>
>> On Jun 12, 2011, at 10:51 AM, Banibrata Dutta wrote:
>>
>> Prematurely sent.
>>
>> On Sun, Jun 12, 2011 at 7:59 PM, Banibrata Dutta <
>> banibrata.dutta@REDACTED> wrote:
>>
>>>
>>> What would be a good way to correlate asynchronous events, spot patterns
>>> over a sliding window (s.a. of no. of events elapsed or time elapsed), with
>>> millions of events occurring simultaneously, using Erlang ?
>>>
>>> The set of possible events is known, and any unknown event is just
>>> flagged as 'unknown' (so all unknowns are similar). The set of possible
>>> event patterns can be enumerated, but is possibly quite a large set of
>>> patterns.
>>>
>>
>> Was wondering as to what could be the approach taken to implement such a
>> thing in pure Erlang. My initial thoughts were along the line of maintaining
>> FSMs per event source, but with so many events and so many possible/valid
>> patterns, the thing seems kind of unwieldy. Also, I'd like a non-programmer
>> to be able to define new events and valid event patterns.
>>
>> I believe 'Complex Event Processing' is quite likely to be the standard
>> approach for such things, as I've found from some posts, and solutions exist
>> in Java world for same, but both as an academic exercise (for the fun of
>> learning) and for a potentially simpler + better solution, would like to try
>> doing this is Erlang.
>>
>>
>> I think you need to define your problem better.
>>
>
> Sure, let me try.
>
>
>> What exactly do you mean by "millions of events occurring simultaneously"?
>>
>>
>
> Okay, so I can say something like 500 events/second handled for correlation
> would be a more realistic number.
>
>
>> At exactly the same time?
>>
>
> Yes... some of the events might be from same source, but spaced by as
> little as 50ms, but mostly from different sources. There could be some
> heirarchical relationship between sources. Very typical case of network
> management scenario. E.g. a fault port on a switch, could probably cause
> hundreds of destination unreachable events, application response timeouts,
> heartbeat losses etc..
>
>
>> Millions of events per second? Minute? Is that peak rate, average rate or
>> minimum rate?
>>
>
> Okay, I got over-enthusiastic :-) . Say 100 events/second typical, 500
> events/second peak, no real minimum.
>
> What exactly is a pattern?
>>
>
> Node-A failed, Power in room-X where Node-A is kept failed, Nodes B,C,D
> which are served thru Node-A became unreachable, due to which Services L & M
> became unavailable, and due to which another dependent service N started
> giving inconsistent answers. So this is a pattern. However in this case,
> there's a possibility that Power-failure had nothing to do with Noda-A's
> failure, as backup power was available.
>
> Another pattern is, Power in room X failed, then Noda A failed, leading to
> failure of only Node D, because somehow Nodes B & C were dynamically
> configured to reroute. This is another pattern.
>
> What do you mean by "quite a large set of patterns"? Hundreds, thousands,
>> millions?
>>
>
> Several hundreds is a distinct possibility, and thousands are not
> impossible, but millions -- probably not.
>
>
>> How long is that sliding window?
>>
>
> From few minutes (for certain type of events), to few days (for another
> type of events).
>
>
>> Can patterns encompass events coming from multiple sources or just one
>> source?
>>
>
> Yes, indeed. However in this case, there needs to a "relationship" between
> the event sources, that is pre-defined. E.g. some sense of "topology"
> exists. However it is likely that only 2% of the event sources are
> interrelated.
>
>
>> Are patterns concerned only with event ordering and occurrence or there
>> are timing issues involved as well?
>>
>
> Ordering, Timing, or any kind of causal relationship.
>
> --
regards,
Banibrata
http://www.linkedin.com/in/bdutta
http://twitter.com/edgeliving
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110613/5f2141de/attachment.htm>