[erlang-questions] How to do this in Erlang -- any suggestions ?

[Banibrata Dutta]

On Tue, Jun 14, 2011 at 10:09 PM, Edmond Begumisa <
ebegumisa@REDACTED> wrote:

> The more I think about it, the more I think Banibrata is really onto
> something here!
>

:-) , well, the more I read about CEP (Complex Event Processing), the more I
feel, people have been onto this thing for quite a while now. The use-cases
all seem very real.


> A generic tool of this nature could be rather useful. If such a tool
> allowed me to say things like...
>
> "If 30 'EXIT' messages appear in supervision subtree Y over a period of
> 1sec and 60% of these contain {tcp_error, _} then call
> sasl:set_alarm({'network_down', Y})" in which case an SMS would be sent to
> my phone.
>

Precisely. It is extremely close to what I was thinking.

>
> I can think of a million uses for such a tool :)
>

Indeed. This is the reason, the more I think of FSM (in gen_fsm terms, for
instance), the more I get the feeling of getting into something that is
"static", something that is embedded deep within an application. Mihai here,
mentioned DSL to represent the patterns & actions, and EPL (Event Processing
Language) in the CEP terminology, appears to be just that. Thanks to Darach
for recommending a really interesting book as well.

And to answer your questions from the previous mail -- by real-time I mean
soft real-time, or more like nearly synchronous processing, and not an
offline/batch processing for post-facto analysis. Also all this processing,
and looking into to time window to correlate events, needs to be done by the
computer. No human looking here... well, other than to fix things if they
are broken.

On Tue, 14 Jun 2011 06:13:56 +1000, Mihai Balea <mihai@REDACTED> wrote:


> On Jun 13, 2011, at 9:12 AM, Banibrata Dutta wrote:
>
>
>> Well, I haven't. In fact, starting with a small set of possible patterns,
>> seems to be the most intuitive approach. When I bring in the thought of
>> permitting non-programmers to write "something" (e.g. natural language
>> rules, or a simple DSL snippet), and then such a thing modifying the FSM,
>> without me having to code anything further in Erlang, is when I am hitting a
>> mental roadblock. I've coded fairly complex FSM's in C/C++ using the
>> table-based approach. Does gen_fsm take a similar approach ? Will it be a
>> good, natural fit ? Maybe a rather naive question for people extremely
>> familiar with Erlang, but I'm still very much on the learning curve's
>> positive vector.
>>
>
> Erlang gen_fsms are quite simple and standard, they define a bunch of
> states and use pattern matching to select actions for each state/event
> combination. Pretty much equivalent to a table-based fsm, only there's no
> distinct table to speak of.
>
>
>
>> however you might want to make them a bit more flexible. Maybe have a
>> bunch of processes running pattern recognizers, say one process per pattern
>> or class of patterns.
>>
>> My natural tendency was to think more in terms of 1 process per source,
>> e.g. 1st event from a source causes an Erlang process to be created, which
>> then embodies the entire possible FSM. The main issue with pattern based
>> processes (if I understood it correctly), is determining the right process
>> to hand-off the event to, and once handed-off, backtracking would mean extra
>> work.
>>
>> Maybe you can filter events based on source so certain recognizers will
>> only get certain events.
>>
>> Yes, that was the starting point, as I had imagined.
>>
>
> Event forwarding based on source is technically easy in Erlang, just use a
> process registry (proc, grpoc, even the built in process registrar) and use
> sources as keys.
> The problem with this approach is you might have at least some correlation
> between sources. Events originating from different sources but belonging to
> the same pattern (i.e. server going black due to switch failing in some
> other room, due to power outage and the like).
>
>
>> If you have a low correlation between event sources maybe you can even
>> design your system to distribute the processing of unrelated events to
>> different nodes in a cluster (big scalability gain if you can do that).
>>
>> Excellent thought... indeed. Source based sharding.
>>
>> Another way of approaching this would be to duplicate the event stream to
>> multiple nodes in your cluster and have each node only look for certain
>> subsets of patterns.
>>
>> Actually, my bad, I should've mentioned, I wanted to do all this in
>> Real-Time. So I don't really have a pattern-string, so to say. I don't,
>> without that I can really take this approach. Or can I ?
>>
>
> I think you can. If your patterns are independent, or at least can be
> grouped in independent categories, what would prevent you from assigning
> sets of independent patterns to different nodes? Not saying you need to do
> that, but it might be helpful if you find out that recognizing and
> processing thousands of patterns take up too much cpu.
>
> Mihai
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110614/80a7c3b9/attachment.htm>