<div dir="ltr"><p class="MsoNormal">Hi
Darach,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thank
you for your elaborated post!</p>
<p class="MsoNormal">Frankly
speaking, this project was started by pure curiosity rather than a necessity.
That is why I announced it in this early stage (may be a bit prematurely) – to gather
opinions and ideas.</p>
<p class="MsoNormal">Data
parallelism on a single query is out of scope (for now). I was more concerned about
“scaling out” scenario, but surely this is something to think about.</p>
<p class="MsoNormal">The
main goal was to have the ability to execute operations over multiple event
streams, where each stream contains a single event type. Currently, only “implicit
joins” are supported in the form of “select…. from event1, event2 where
event1.paramX = event2.paramX”. That is why I am using ETS (actually provided
by “folsom”). I gave it some thinking, but failed to find how to join multiple
streams without a temporary storage. On the other hand – if there are no joins
in the query, it can be done without using ETS, simply by updating aggregations
on the fly, as the event arrives. That is exactly what I am working on at the
moment.</p>
<p class="MsoNormal">Btw,
my background in the CEP theory is minimal (something I plan to catch up on),
do the above-mentioned stream “joins” qualify as “stream combinators”?</p>
<p class="MsoNormal">My
immediate plans are to extend the DSL with:</p><p class="MsoNormal"><span style><br></span></p><p class="MsoNormal"><span style>- “Named windows” (similar to ESPER ones) – basically a new streams, each
one built from a query result, which can be used by other queries.</span></p><p class="MsoNormal"><span style>- Defining the events, so that event behaviour (event1.erl and event2.erl
in the test suite) to be generated automatically</span></p><p class="MsoNormal"><span style>- Event patterns over streams and named windows</span></p><p class="MsoNormal"><span style>- Batch (trigger result on specific time intervals) and tumbling (trigger
result on specific number of inputs) windows</span></p><p class="" style></p>
<p class="MsoNormal">I’ll
definitely think about using eep-erl and beam-erl <span style="font-family:Wingdings">J</span></p>
<p class="MsoNormal">Thank
you again!</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Best
regards,</p>
<p class="MsoNormal">Vasco</p><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 6, 2014 at 2:31 AM, Darach Ennis <span dir="ltr"><<a href="mailto:darach@gmail.com" target="_blank">darach@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi Vasco,</span><div>
<font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">[Resent to all]<br></font><div><div class="h5"><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Looks like you're off to a good start here towards a SQL based CEP engine for Erlang/OTP.</div><div style="font-family:arial,sans-serif;font-size:13px">
There are arguably 4 key constructs that Rivus would need to support to qualify as a CEP engine:</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
1. Continuous Query</div><div style="font-family:arial,sans-serif;font-size:13px">2. Windows (aggregation)</div><div style="font-family:arial,sans-serif;font-size:13px">3. 'Complex' pattern matching (eg: temporal patterns across streams, stream combinators, ...)</div>
<div style="font-family:arial,sans-serif;font-size:13px">4. A domain specific language.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">So, Rivus would basically qualify in principle under 1, 2 and 4 here but not entirely under 3. I saw</div>
<div style="font-family:arial,sans-serif;font-size:13px">no support for combinators, temporal pattern matching, state (tables or variables), concurrency or</div><div style="font-family:arial,sans-serif;font-size:13px">data parallelism. However, as the syntax allows the definition of multiple correlations it could be said</div>
<div style="font-family:arial,sans-serif;font-size:13px">that simultaneity (of queries) is supported - and this is essential.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
A potential issue depending on your target audience is that the facilities in core erlang for</div><div style="font-family:arial,sans-serif;font-size:13px">expressive pattern matching and tuple processing aren't leveraged, nor is the native support</div>
<div style="font-family:arial,sans-serif;font-size:13px">for concurrency and parallelism through providing concurrency and data parallelism of queries</div><div style="font-family:arial,sans-serif;font-size:13px">in the DSL. StreamBase (note: I used to work for them so I'm probably biased) support concurrency</div>
<div style="font-family:arial,sans-serif;font-size:13px">and (data) parallelism in their SQL and visual (flow-based) languages and in the hands of an</div><div style="font-family:arial,sans-serif;font-size:13px">expert (eg: someone intimate with the runtime) is very powerful.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Apama is worth looking it - its runtime was inspired by the Erlang concurrency model (allegedly)</div>
<div style="font-family:arial,sans-serif;font-size:13px">and its monitorscript language supports processes.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
In CEP languages and environments where these and more features are provided - it doesn't</div><div style="font-family:arial,sans-serif;font-size:13px">actually help the poor CEP developer - it hinders. Large complex CEP algorithms are difficult to</div>
<div style="font-family:arial,sans-serif;font-size:13px">evolve, maintain and support in most organisations. The tools typically have weak debugging and</div><div style="font-family:arial,sans-serif;font-size:13px">refactoring support if at all and the DSLs aren't standard and often have constructs peculiar to the</div>
<div style="font-family:arial,sans-serif;font-size:13px">lineage of the technology (it may have started out life as an active database or as a log processing</div><div style="font-family:arial,sans-serif;font-size:13px">
tool or as a captured packet analyser ...).</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">StreamBase, by far, has the best tooling (yup, I'm biased), IDE, debugging and refactoring support.</div>
<div style="font-family:arial,sans-serif;font-size:13px">But a significant component of any successful CEP solution is native code. A question I've been</div><div style="font-family:arial,sans-serif;font-size:13px">grappling with is, what are the useful bits in CEP that would be useful within Erlang as a library or</div>
<div style="font-family:arial,sans-serif;font-size:13px">service? I've experimented with what I think are the most useful two, namely: aggregate window</div><div style="font-family:arial,sans-serif;font-size:13px">processing and data flow algorithm definition. Rather than define a SQL to enshrine the conditions</div>
<div style="font-family:arial,sans-serif;font-size:13px">under which they can be leveraged - plain old erlang provides a richer environment for these to be</div><div style="font-family:arial,sans-serif;font-size:13px">used fruitfully. Why write a DSL and Erlang is a 'real language' - one that makes pattern matching</div>
<div style="font-family:arial,sans-serif;font-size:13px">easy with distribution and concurrency built in if you need it?</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
<a href="https://github.com/darach/beam-erl" target="_blank">https://github.com/darach/beam-erl</a><br></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
Embeddable data flow library. Branch, Pipe, Combine, Filter and Transform data.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><a href="https://github.com/darach/eep-erl" target="_blank">https://github.com/darach/eep-erl</a><br>
</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Sliding, Tumbling, Periodic and Monotonic aggregation.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">In the case of aggregation once the window semantics are defined the window functions</div><div style="font-family:arial,sans-serif;font-size:13px">can be provided by extension (use an erlang behaviour). So you could use this to allow</div>
<div style="font-family:arial,sans-serif;font-size:13px">user defined (aggregate) functions in your SQL dialect. The SQL statements and clauses</div><div style="font-family:arial,sans-serif;font-size:13px">could be compiled to an intermediary form that supports a more primitive flow language</div>
<div style="font-family:arial,sans-serif;font-size:13px">and runtime making building out the CEP engine a little bit easier. This also favors plugging</div><div style="font-family:arial,sans-serif;font-size:13px">in user defined constructs allowing the language itself to be extended through user defined</div>
<div style="font-family:arial,sans-serif;font-size:13px">operations.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I haven't implemented temporal pattern matching or interesting complex combinators because I haven't</div>
<div style="font-family:arial,sans-serif;font-size:13px">needed them myself of late. Even the window aggregation is simple. Some CEP engines allow aggregates of multiple</div><div style="font-family:arial,sans-serif;font-size:13px">
dimensions (data, time, value, predicate expression) but these can typically be (far) more easily defined</div><div style="font-family:arial,sans-serif;font-size:13px">through composition.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Judging from the SQL dialect syntax the need so far is for expressivity in filtering / detection</div><div style="font-family:arial,sans-serif;font-size:13px">
based on fairly simple windowed events or windowed temporal processing of event data. Complex</div><div style="font-family:arial,sans-serif;font-size:13px">processing of multiple event sources against complex scenarios (eg: near real-time collusion or layering</div>
<div style="font-family:arial,sans-serif;font-size:13px">fraud detection in capital markets market abuse and surveillance) without extending the language dialect</div><div style="font-family:arial,sans-serif;font-size:13px">
would be hard/impossible in its current form.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">It would be interesting to hear what the plans for the language are and if/where plain old Erlang/OTP</div>
<div style="font-family:arial,sans-serif;font-size:13px">could be leveraged to extend the capabilities of the engine. Once concern though is that aggregation</div><div style="font-family:arial,sans-serif;font-size:13px">
windows are based on folsom, which in turn depends on ETS. If any of your queries contain large</div>
<div style="font-family:arial,sans-serif;font-size:13px">numbers of aggregate operations - you'll spend a lot of time in ETS. If your intermediate operation</div><div style="font-family:arial,sans-serif;font-size:13px">
results aren't critical - thats a lot of overhead you can avoid by providing your own aggregation/windowing</div><div style="font-family:arial,sans-serif;font-size:13px">logic (or fork/steal eep-erls)</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Promising start to a SQL based CEP in Erlang/OTP! Thanks for sharing so early in its development!</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Cheers,</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Darach.</div></div>
</div></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div class="h5">On Mon, Jan 6, 2014 at 1:02 AM, Vassil Kolarov <span dir="ltr"><<a href="mailto:vasco@vas.io" target="_blank">vasco@vas.io</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div class="h5">
<div dir="ltr">Hi all,<div><br></div><div>I'd like to announce a project called 'Rivus'. The goal of the project is implementation of complex events processing application in Erlang, which uses a DSL similar to ESPER's EPL. It is in a very early stage, but could be considered sort of a "MVP".</div>
<div><br></div><div>Here is the GitHub repository: <a href="https://github.com/vascokk/rivus_cep" target="_blank">https://github.com/vascokk/rivus_cep</a> </div><div><br></div><div>Hope you'll find it interesting and useful.<br>
</div>
<div><br></div><div>Best regards,</div><div>Vasco</div></div>
<br></div></div><div class="im">_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br></div></div>