[erlang-questions] How to do this in Erlang -- any suggestions ?

[Banibrata Dutta]

On Sun, Jun 12, 2011 at 9:48 PM, Mihai Balea <mihai@REDACTED> wrote:

>
> On Jun 12, 2011, at 11:38 AM, Banibrata Dutta wrote:
>
> gr8 questions, and they certainly need clarification.
> cc'ing the group s.t. others could contribute too.
>
> On Sun, Jun 12, 2011 at 8:48 PM, Mihai Balea <mihai@REDACTED> wrote:
>
>>
>> On Jun 12, 2011, at 10:51 AM, Banibrata Dutta wrote:
>>
>> Prematurely sent.
>>
>> On Sun, Jun 12, 2011 at 7:59 PM, Banibrata Dutta <
>> banibrata.dutta@REDACTED> wrote:
>>
>>>
>>> What would be a good way to correlate asynchronous events, spot patterns
>>> over a sliding window (s.a. of no. of events elapsed or time elapsed), with
>>> millions of events occurring simultaneously, using Erlang ?
>>>
>>> The set of possible events is known, and any unknown event is just
>>> flagged as 'unknown' (so all unknowns are similar). The set of possible
>>> event patterns can be enumerated, but is possibly quite a large set of
>>> patterns.
>>>
>>
>> Was wondering as to what could be the approach taken to implement such a
>> thing in pure Erlang. My initial thoughts were along the line of maintaining
>> FSMs per event source, but with so many events and so many possible/valid
>> patterns, the thing seems kind of unwieldy. Also, I'd like a non-programmer
>> to be able to define new events and valid event patterns.
>>
>> I believe 'Complex Event Processing' is quite likely to be the standard
>> approach for such things, as I've found from some posts, and solutions exist
>> in Java world for same, but both as an academic exercise (for the fun of
>> learning) and for a potentially simpler + better solution, would like to try
>> doing this is Erlang.
>>
>>
>> I think you need to define your problem better.
>>
>
> Sure, let me try.
>
>
>> What exactly do you mean by "millions of events occurring simultaneously"?
>>
>>
>
> Okay, so I can say something like 500 events/second handled for correlation
> would be a more realistic number.
>
>
>> At exactly the same time?
>>
>
> Yes... some of the events might be from same source, but spaced by as
> little as 50ms, but mostly from different sources. There could be some
> heirarchical relationship between sources. Very typical case of network
> management scenario. E.g. a fault port on a switch, could probably cause
> hundreds of destination unreachable events, application response timeouts,
> heartbeat losses etc..
>
>
>> Millions of events per second? Minute? Is that peak rate, average rate or
>> minimum rate?
>>
>
> Okay, I got over-enthusiastic :-) . Say 100 events/second typical, 500
> events/second peak, no real minimum.
>
> What exactly is a pattern?
>>
>
> Node-A failed, Power in room-X where Node-A is kept failed, Nodes B,C,D
> which are served thru Node-A became unreachable, due to which Services L & M
> became unavailable, and due to which another dependent service N started
> giving inconsistent answers. So this is a pattern. However in this case,
> there's a possibility that Power-failure had nothing to do with Noda-A's
> failure, as backup power was available.
>
> Another pattern is, Power in room X failed, then Noda A failed, leading to
> failure of only Node D, because somehow Nodes B & C were dynamically
> configured to reroute. This is another pattern.
>
> What do you mean by "quite a large set of patterns"? Hundreds, thousands,
>> millions?
>>
>
> Several hundreds is a distinct possibility, and thousands are not
> impossible, but millions -- probably not.
>
>
>> How long is that sliding window?
>>
>
> From few minutes (for certain type of events), to few days (for another
> type of events).
>
>
>> Can patterns encompass events coming from multiple sources or just one
>> source?
>>
>
> Yes, indeed. However in this case, there needs to a "relationship" between
> the event sources, that is pre-defined. E.g. some sense of "topology"
> exists. However it is likely that only 2% of the event sources are
> interrelated.
>
>
>> Are patterns concerned only with event ordering and occurrence or there
>> are timing issues involved as well?
>>
>
> Ordering, Timing, or any kind of causal relationship.
>
> Thanks for answering.


> Okay, that's a bit more descriptive :)
>
> First of all, don't dismiss your idea of using FSMs,
>

Well, I haven't. In fact, starting with a small set of possible patterns,
seems to be the most intuitive approach. When I bring in the thought of
permitting non-programmers to write "something" (e.g. natural language
rules, or a simple DSL snippet), and then such a thing modifying the FSM,
without me having to code anything further in Erlang, is when I am hitting a
mental roadblock. I've coded fairly complex FSM's in C/C++ using the
table-based approach. Does gen_fsm take a similar approach ? Will it be a
good, natural fit ? Maybe a rather naive question for people extremely
familiar with Erlang, but I'm still very much on the learning curve's
positive vector.


> however you might want to make them a bit more flexible. Maybe have a bunch
> of processes running pattern recognizers, say one process per pattern or
> class of patterns.
>

My natural tendency was to think more in terms of 1 process per source, e.g.
1st event from a source causes an Erlang process to be created, which then
embodies the entire possible FSM. The main issue with pattern based
processes (if I understood it correctly), is determining the right process
to hand-off the event to, and once handed-off, backtracking would mean extra
work.


> Maybe you can filter events based on source so certain recognizers will
> only get certain events.
>

Yes, that was the starting point, as I had imagined.


> If you have a low correlation between event sources maybe you can even
> design your system to distribute the processing of unrelated events to
> different nodes in a cluster (big scalability gain if you can do that).
>

Excellent thought... indeed. Source based sharding.


> Another way of approaching this would be to duplicate the event stream to
> multiple nodes in your cluster and have each node only look for certain
> subsets of patterns.
>

Actually, my bad, I should've mentioned, I wanted to do all this in
Real-Time. So I don't really have a pattern-string, so to say. I don't,
without that I can really take this approach. Or can I ?

However, for a peak of 500 events/sec, you most likely can get away with
> running everything on a decently powerful machine (speaking from experience,
> we had a relatively similar, though simpler, system running just fine on a
> dual core server, handling up to 800 events/sec)
>
> If you intend to let non programmers define events and patterns you'll
> probably want to define some sort of DSL. Try as best as you can to make it
> declarative only, this way you can probably get away with files containing
> Erlang terms.
>

Thanks for the tip. A declarative DSL sounds good.


> Just a bunch of thoughts, hopefully this helps
>

They certainly do ! Thanks.


-- 
regards,
Banibrata
http://www.linkedin.com/in/bdutta
http://twitter.com/edgeliving
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110613/b6dfc545/attachment.htm>