<div dir="ltr">Hi Vasco,<div><br></div><div>Combinators are, ugh, complicated. You can roughly break then down semantically:</div><div><br></div><div>* Merge - take two (or more) partially ordered streams and create a single totally ordered resultant stream</div>
<div>* Gather - works in concert with splitting (or scattering) a stream of events down multiple sub streams.</div><div> Most implementations of this aren't tolerant of losing events that should be gathered</div><div>
* Pattern - Uses some form of temporal logic to define match criteria</div><div>* Join - Typically windowed, find matching events in two streams by value or predicate expression</div><div><div>* Union - the most general streaming combinator is stateless and simply passes any input events to outputs.</div>
</div><div>* There are others.</div><div><br></div><div>Union and Gather are typically more commonly found in the wild in my experience than Merge or Join</div><div>but all are useful some of the time. Pattern is useful if you're checking causal order - "did event B occur</div>
<div>after and only after event A whilst event C was non occurring" kind of thing but composition "... within the</div><div>last 5 minutes / events" is king.</div><div><br></div><div>But stateful combinators are quite common. In fact, robust data recovery is the hard part of the problem.</div>
<div>The database community has some discipline here (CRDTs) that a distributed CEP may want to consider</div><div>for first order support that doesn't get handled elegantly by most traditional CEP engines. Bloom's exploitation</div>
<div>of out of order processing is another interesting thing that would be nice to see influencing CEP engine design.</div><div><br></div><div>Anyway, StreamBase's SQL dialect is well documented and more complete than Esper:</div>
<div><br></div><div><a href="http://docs.streambase.com/sb73/topic/com.streambase.sb.ide.help/data/html/streamsql/ssql-dml.html">http://docs.streambase.com/sb73/topic/com.streambase.sb.ide.help/data/html/streamsql/ssql-dml.html</a><br>
</div><div><br></div><div>The temporal pattern language may also be of interest:</div><div><br></div><div><a href="http://docs.streambase.com/sb73/topic/com.streambase.sb.ide.help/data/html/reference/patternquery.html">http://docs.streambase.com/sb73/topic/com.streambase.sb.ide.help/data/html/reference/patternquery.html</a><br>
</div><div><br></div><div>IBM's SPL is also very well documented and worth exploring:</div><div><br></div><div><a href="http://pic.dhe.ibm.com/infocenter/streams/v2r0/index.jsp?topic=%2Fcom.ibm.swg.im.infosphere.streams.whats-new.doc%2Fdoc%2Fibminfospherestreams-whats-new-spl4.html">http://pic.dhe.ibm.com/infocenter/streams/v2r0/index.jsp?topic=%2Fcom.ibm.swg.im.infosphere.streams.whats-new.doc%2Fdoc%2Fibminfospherestreams-whats-new-spl4.html</a><br>
</div><div><br></div><div>It would certainly be worth your time surveying the many different flavors of CEP and ESP currently out</div><div>there. Ultimately, the kind of streaming algorithms and continuous processing you are embarking upon will</div>
<div>have a more significant influence. What you won't get from a lot of the lower level mechanics are the higher</div><div>level flow patterns that an experienced practitioner will use but you may not always find directly in syntax...</div>
<div><br></div><div>These typically influence the direction and evolution of CEP languages but as its a domain that is still very</div><div>much evolving (technically) you will find quite a lot of variation. I'd advise only implementing that which you</div>
<div>need. Its easier to extend later than remove a bad construct. A good example is multi-dimensional windows.</div><div>Pretty much all CEP engines I know of that support this got the model wrong... worse, the human brain doesn't</div>
<div>really like handling more than one dimension simultaneously so favoring a dialect that allows ease of composition</div><div>of simple operations over complex operations in simple flows.</div><div><br></div><div>Another omission in most CEP engines is good distribution, fault tolerance and clustering out of the box.</div>
<div>By 'good' I mean with respect to some formal notion of recovery (can an algorithm or query recover</div><div>precisely without loss or not). This is hard and existing CEP engines get this wrong - it turns out that solving</div>
<div>this wasn't commercially compelling until recently - but near real-time market abuse and surveillance and continuous</div><div>big data use cases need better solutions here specifically.</div><div><br></div><div>
All part of the fun of building a full blown CEP engine ...</div><div><br></div><div>Cheers,</div><div><br></div><div>Darach.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 6, 2014 at 11:06 AM, Vassil Kolarov <span dir="ltr"><<a href="mailto:vasco@vas.io" target="_blank">vasco@vas.io</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p class="MsoNormal">Hi
Darach,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thank
you for your elaborated post!</p>
<p class="MsoNormal">Frankly
speaking, this project was started by pure curiosity rather than a necessity.
That is why I announced it in this early stage (may be a bit prematurely) – to gather
opinions and ideas.</p>
<p class="MsoNormal">Data
parallelism on a single query is out of scope (for now). I was more concerned about
“scaling out” scenario, but surely this is something to think about.</p>
<p class="MsoNormal">The
main goal was to have the ability to execute operations over multiple event
streams, where each stream contains a single event type. Currently, only “implicit
joins” are supported in the form of “select…. from event1, event2 where
event1.paramX = event2.paramX”. That is why I am using ETS (actually provided
by “folsom”). I gave it some thinking, but failed to find how to join multiple
streams without a temporary storage. On the other hand – if there are no joins
in the query, it can be done without using ETS, simply by updating aggregations
on the fly, as the event arrives. That is exactly what I am working on at the
moment.</p>
<p class="MsoNormal">Btw,
my background in the CEP theory is minimal (something I plan to catch up on),
do the above-mentioned stream “joins” qualify as “stream combinators”?</p>
<p class="MsoNormal">My
immediate plans are to extend the DSL with:</p><p class="MsoNormal"><span><br></span></p><p class="MsoNormal"><span>- “Named windows” (similar to ESPER ones) – basically a new streams, each
one built from a query result, which can be used by other queries.</span></p><p class="MsoNormal"><span>- Defining the events, so that event behaviour (event1.erl and event2.erl
in the test suite) to be generated automatically</span></p><p class="MsoNormal"><span>- Event patterns over streams and named windows</span></p><p class="MsoNormal"><span>- Batch (trigger result on specific time intervals) and tumbling (trigger
result on specific number of inputs) windows</span></p><p></p>
<p class="MsoNormal">I’ll
definitely think about using eep-erl and beam-erl <span style="font-family:Wingdings">J</span></p>
<p class="MsoNormal">Thank
you again!</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Best
regards,</p>
<p class="MsoNormal">Vasco</p><div><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 6, 2014 at 2:31 AM, Darach Ennis <span dir="ltr"><<a href="mailto:darach@gmail.com" target="_blank">darach@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi Vasco,</span><div>
<font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">[Resent to all]<br></font><div><div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Looks like you're off to a good start here towards a SQL based CEP engine for Erlang/OTP.</div><div style="font-family:arial,sans-serif;font-size:13px">
There are arguably 4 key constructs that Rivus would need to support to qualify as a CEP engine:</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
1. Continuous Query</div><div style="font-family:arial,sans-serif;font-size:13px">2. Windows (aggregation)</div><div style="font-family:arial,sans-serif;font-size:13px">3. 'Complex' pattern matching (eg: temporal patterns across streams, stream combinators, ...)</div>
<div style="font-family:arial,sans-serif;font-size:13px">4. A domain specific language.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">So, Rivus would basically qualify in principle under 1, 2 and 4 here but not entirely under 3. I saw</div>
<div style="font-family:arial,sans-serif;font-size:13px">no support for combinators, temporal pattern matching, state (tables or variables), concurrency or</div><div style="font-family:arial,sans-serif;font-size:13px">data parallelism. However, as the syntax allows the definition of multiple correlations it could be said</div>
<div style="font-family:arial,sans-serif;font-size:13px">that simultaneity (of queries) is supported - and this is essential.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
A potential issue depending on your target audience is that the facilities in core erlang for</div><div style="font-family:arial,sans-serif;font-size:13px">expressive pattern matching and tuple processing aren't leveraged, nor is the native support</div>
<div style="font-family:arial,sans-serif;font-size:13px">for concurrency and parallelism through providing concurrency and data parallelism of queries</div><div style="font-family:arial,sans-serif;font-size:13px">in the DSL. StreamBase (note: I used to work for them so I'm probably biased) support concurrency</div>
<div style="font-family:arial,sans-serif;font-size:13px">and (data) parallelism in their SQL and visual (flow-based) languages and in the hands of an</div><div style="font-family:arial,sans-serif;font-size:13px">expert (eg: someone intimate with the runtime) is very powerful.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Apama is worth looking it - its runtime was inspired by the Erlang concurrency model (allegedly)</div>
<div style="font-family:arial,sans-serif;font-size:13px">and its monitorscript language supports processes.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
In CEP languages and environments where these and more features are provided - it doesn't</div><div style="font-family:arial,sans-serif;font-size:13px">actually help the poor CEP developer - it hinders. Large complex CEP algorithms are difficult to</div>
<div style="font-family:arial,sans-serif;font-size:13px">evolve, maintain and support in most organisations. The tools typically have weak debugging and</div><div style="font-family:arial,sans-serif;font-size:13px">refactoring support if at all and the DSLs aren't standard and often have constructs peculiar to the</div>
<div style="font-family:arial,sans-serif;font-size:13px">lineage of the technology (it may have started out life as an active database or as a log processing</div><div style="font-family:arial,sans-serif;font-size:13px">
tool or as a captured packet analyser ...).</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">StreamBase, by far, has the best tooling (yup, I'm biased), IDE, debugging and refactoring support.</div>
<div style="font-family:arial,sans-serif;font-size:13px">But a significant component of any successful CEP solution is native code. A question I've been</div><div style="font-family:arial,sans-serif;font-size:13px">grappling with is, what are the useful bits in CEP that would be useful within Erlang as a library or</div>
<div style="font-family:arial,sans-serif;font-size:13px">service? I've experimented with what I think are the most useful two, namely: aggregate window</div><div style="font-family:arial,sans-serif;font-size:13px">processing and data flow algorithm definition. Rather than define a SQL to enshrine the conditions</div>
<div style="font-family:arial,sans-serif;font-size:13px">under which they can be leveraged - plain old erlang provides a richer environment for these to be</div><div style="font-family:arial,sans-serif;font-size:13px">used fruitfully. Why write a DSL and Erlang is a 'real language' - one that makes pattern matching</div>
<div style="font-family:arial,sans-serif;font-size:13px">easy with distribution and concurrency built in if you need it?</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
<a href="https://github.com/darach/beam-erl" target="_blank">https://github.com/darach/beam-erl</a><br></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
Embeddable data flow library. Branch, Pipe, Combine, Filter and Transform data.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><a href="https://github.com/darach/eep-erl" target="_blank">https://github.com/darach/eep-erl</a><br>
</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Sliding, Tumbling, Periodic and Monotonic aggregation.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">In the case of aggregation once the window semantics are defined the window functions</div><div style="font-family:arial,sans-serif;font-size:13px">can be provided by extension (use an erlang behaviour). So you could use this to allow</div>
<div style="font-family:arial,sans-serif;font-size:13px">user defined (aggregate) functions in your SQL dialect. The SQL statements and clauses</div><div style="font-family:arial,sans-serif;font-size:13px">could be compiled to an intermediary form that supports a more primitive flow language</div>
<div style="font-family:arial,sans-serif;font-size:13px">and runtime making building out the CEP engine a little bit easier. This also favors plugging</div><div style="font-family:arial,sans-serif;font-size:13px">in user defined constructs allowing the language itself to be extended through user defined</div>
<div style="font-family:arial,sans-serif;font-size:13px">operations.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I haven't implemented temporal pattern matching or interesting complex combinators because I haven't</div>
<div style="font-family:arial,sans-serif;font-size:13px">needed them myself of late. Even the window aggregation is simple. Some CEP engines allow aggregates of multiple</div><div style="font-family:arial,sans-serif;font-size:13px">
dimensions (data, time, value, predicate expression) but these can typically be (far) more easily defined</div><div style="font-family:arial,sans-serif;font-size:13px">through composition.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Judging from the SQL dialect syntax the need so far is for expressivity in filtering / detection</div><div style="font-family:arial,sans-serif;font-size:13px">
based on fairly simple windowed events or windowed temporal processing of event data. Complex</div><div style="font-family:arial,sans-serif;font-size:13px">processing of multiple event sources against complex scenarios (eg: near real-time collusion or layering</div>
<div style="font-family:arial,sans-serif;font-size:13px">fraud detection in capital markets market abuse and surveillance) without extending the language dialect</div><div style="font-family:arial,sans-serif;font-size:13px">
would be hard/impossible in its current form.</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">It would be interesting to hear what the plans for the language are and if/where plain old Erlang/OTP</div>
<div style="font-family:arial,sans-serif;font-size:13px">could be leveraged to extend the capabilities of the engine. Once concern though is that aggregation</div><div style="font-family:arial,sans-serif;font-size:13px">
windows are based on folsom, which in turn depends on ETS. If any of your queries contain large</div>
<div style="font-family:arial,sans-serif;font-size:13px">numbers of aggregate operations - you'll spend a lot of time in ETS. If your intermediate operation</div><div style="font-family:arial,sans-serif;font-size:13px">
results aren't critical - thats a lot of overhead you can avoid by providing your own aggregation/windowing</div><div style="font-family:arial,sans-serif;font-size:13px">logic (or fork/steal eep-erls)</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Promising start to a SQL based CEP in Erlang/OTP! Thanks for sharing so early in its development!</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Cheers,</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Darach.</div></div>
</div></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Mon, Jan 6, 2014 at 1:02 AM, Vassil Kolarov <span dir="ltr"><<a href="mailto:vasco@vas.io" target="_blank">vasco@vas.io</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>
<div dir="ltr">Hi all,<div><br></div><div>I'd like to announce a project called 'Rivus'. The goal of the project is implementation of complex events processing application in Erlang, which uses a DSL similar to ESPER's EPL. It is in a very early stage, but could be considered sort of a "MVP".</div>
<div><br></div><div>Here is the GitHub repository: <a href="https://github.com/vascokk/rivus_cep" target="_blank">https://github.com/vascokk/rivus_cep</a> </div><div><br></div><div>Hope you'll find it interesting and useful.<br>
</div>
<div><br></div><div>Best regards,</div><div>Vasco</div></div>
<br></div></div><div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>