From yoursurrogategod@REDACTED Mon Jul 1 02:40:28 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Sun, 30 Jun 2013 20:40:28 -0400 Subject: [erlang-questions] How to copy the results from the debug window? Message-ID: Hi all, I'm debugging my Erlang code (with the built-in debugger) and would like to copy the results from the output of a variable (located in bottom-left in window when debugging a process) into a text editor in order to better visualize it. However, every time I try to copy/paste it, it's as if it's not picked up at all. What am I missing? Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jul 1 06:15:21 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 1 Jul 2013 16:15:21 +1200 Subject: [erlang-questions] Erlang and/or MPI In-Reply-To: References: Message-ID: On 28/06/2013, at 11:13 PM, Yvan Godin wrote: > Hello, > you could also give a try to Racket-lang There are several things to say about Racket. First, I was a happy PLT-Scheme user for several years. There's something for everyone in Racket. Second, when it comes to concurrency, the model it offers is the same old same old shared memory with threads and synchronisation model depressingly familiar from C, _without_ the safety features of languages like Concurrent Pascal, Occam, or Ada. Third, there's some fine print in the guide: "threads never run truly in parallel, even if the hardware and operating system support parallelism." (Section 18.12, about half way down http://docs.racket-lang.org/guide/performance.html#%28part._effective-futures%29 ) There is a _different_ mechanism for things that can happen at the same time, and the guide says of that: "Futures run in parallel as long as they can do so safely, but the notion of ?future safe? is inherently tied to the implementation. The distinction between ?future safe? and ?future unsafe? operations may be far from apparent at the level of a Racket program." I spent several hours on Saturday trying to debug the Complex class in my Smalltalk. The basics had been tested long ago, but not the trig and hyperbolic functions. The third complete rewrite of the inverse functions finally did the trick. Here, for example, is inverse hyperbolic tangent: arcTanh "This has a pole at (-1,0)." ?Complex re: ((4 * re) / (im squared + (1 - re) squared) + 1) ln / 4 im: (-2 * im arcTan: 1 - im squared - re squared) / (-2) That's not the actual code; I've cleaned it up a bit. I wonder if I broke anything? The number of trivial changes that leave the code syntactically correct and would satisfy the pickiest type checker is dizzying. Even amongst things that usually compute the right values, apparently tiny changes can make big differences to the branch cuts. And this code certainly doesn't take any care at all with IEEE special values... I've seen C code for this one function that ran to a couple of hundred lines, mostly comments explaining with extreme care every little detail of why the code was the way it was. The last round of changes broke certain tests. For example, 1 cosh asComplex arcCos closeTo: (0 i: 1) broke because the answer was coming out as (0 i: -1). Thing is, (0 i: 1) cos and (0 i: -1) cos are equal and both are indeed equal to (1 cosh). Branch cuts! *That* is numeric code. The support or lack of it for 80-bit floats (not normally part of the 64-bit model) is not really relevant. In the initial stages, the speed of the generated code is not really relevant. What matters is HOW MANY DAYS IT TAKES TO GET THE CODE WORKING. Three things matter here. (1) How hard is it to *accurately* transcribe a correct description of an algorithm into the language in question? Here Ada, Fortran, Haskell, and Ada shine. Smalltalk and Scheme, for all their other virtues, stink. Erlang is somewhere in the middle. (2) How much do the language & compiler help to *prevent* errors? There are all sorts of errors. The way that Ada and Haskell can let you create new distinct types isomorphic to old ones with zero-cost conversion between old and new doesn't help for arcTanh but can help a _lot_ with array code. The way Haskell and Erlang don't let you smash variables also helps a lot. (3) How much support is there for *testing*? (Does anyone have a mutation testing tool for Erlang?) From wbin00@REDACTED Mon Jul 1 05:58:42 2013 From: wbin00@REDACTED (Bin Wang) Date: Mon, 1 Jul 2013 11:58:42 +0800 Subject: [erlang-questions] How to use gpb? In-Reply-To: References: Message-ID: Hi, Tomas Thanks for answering. But if I put all .proto files in directory proto, how do I write the hooks to compile them all? I tried to use *.proto instead of x.proto in pre_hooks, but it came with error: init terminating in do_boot 2013/6/29 Tomas Abrahamsson : > Hi, Rebar by default uses another protobuf compiler for any *.proto files it > finds in the src subdirectory. > > One way to use gpb with rebar is to put your *.proto in another > subdirectory, and use hooks to compile it. I got it working with the > following stuff in rebar.config: > > %% -*- erlang -*- > {pre_hooks, > [{compile, "mkdir -p include"}, %% ensure the include dir exists > {compile, > "erl +B -noinput -pa /path/to/gpb/ebin " > "-I`pwd`/proto -o-erl src -o-hrl include " > "-s gpb_compile c `pwd`/proto/x.proto" > }]}. > > {post_hooks, > [{clean, "rm -f src/x.erl include/x.hrl"}]}. > > {erl_opts, [{i, "/path/to/gpb/include"}]}. > > and the following initial file/directory layout: > > prompt# find . > . > ./proto > ./proto/x.proto > ./src > ./src/g_sup.erl > ./src/g.app.src > ./src/g_app.erl > ./rebar.config > > BRs > Tomas > > > On Fri, Jun 28, 2013 at 11:08 AM, Bin Wang wrote: >> >> Hi, >> >> I'm using google protocol buffers. I found an erlang plugin named >> gpb(https://github.com/tomas-abrahamsson/gpb). But I don't know how >> to use it. When I add it as a dependency of my app using rebar, while >> compile, it always comes out this error: >> >> ERROR: Protobuffs library not present in code path! >> >> Which I searched in google and only found it in erlang_protobuffs >> code. I do use it as a dependency before, but dose it matters? >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From ok@REDACTED Mon Jul 1 07:10:14 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 1 Jul 2013 17:10:14 +1200 Subject: [erlang-questions] Erlang and/or MPI In-Reply-To: References: <51CAF879.1080700@llaisdy.com> Message-ID: On 1/07/2013, at 7:21 AM, Jesper Louis Andersen wrote: > On Thu, Jun 27, 2013 at 3:26 AM, Richard A. O'Keefe wrote: > > Except for commands typed at the shell, Erlang is compiled, not interpreted. > And there is a native code generator; c(File, [native]) does the trick if > native code generation was enabled at build time. > And the native code generator exploits -spec declarations (and some type > inference) to avoid boxing floats some of the time. > > > It is bytecode interpreted to be precise. It is faster than blind AST-interpretation, but it is not really what I would call compiled in the sense of compiled-to-native-code. The native code generator can help for certain types of programs, but it does not yield enough speedup to make it interesting in my opinion. If the native code back end was installed when your Erlang system was installed - and it should be under Linux and Solaris at least - and you compile with c(File, [native]) you will get *native* code, not 'bytecode interpreted'. Way back when Quintus was making the world's finest Prolog system, our system compiled to threaded code (not byte code) that was emulated in macro-generated assembler (not C). We observed that - on tiny programs, native code systems were about a factor of 2 faster; - on medium programs, we were neck and neck; - on large programs (defined as 100 pages of source code), we were at least a factor of 2 faster. I leave it as an exercise for the reader to figure out why this might be so. There was one time I wanted to analyse some benchmark data, and typed in a Fortran program from a book. I just couldn't get it working, so I rewrote it from scratch in Prolog. Once I had done that I was able to find the bug fairly quickly. The amusing thing was that the Prolog code was faster than the native-compiled Fortran. That one I'll explain: I used a better data structure. Asymptotic factors really can win over constant factors. In a distributed system, it's astonishingly easy for costs to be dominated by the cost of getting data from one place to another, with the actual time to compute being minor. No, I'm not saying that it's _always_ the case, just that it's easy to write a program where that's true _without meaning to_. For anyone using MPI, http://www.netlib.org/utk/people/JackDongarra/PAPERS/perftools-review2.pdf may still be useful. Erlang's debugging and tracing tools could be quite helpful. From kolorahl@REDACTED Mon Jul 1 09:03:18 2013 From: kolorahl@REDACTED (Kolo Rahl) Date: Mon, 1 Jul 2013 00:03:18 -0700 Subject: [erlang-questions] Help Using Bullet + BERT Message-ID: I've been messing around with Bullet and Cowboy recently and am loving how easy it is to perform websocket handling. But then I tried integrating a BERT JavaScript library with it (the one from 5HT/n2o) and noticed that the messages hitting my server are in a form that can't be parsed by binary_to_term/1, which is the whole point to to using BERT encoding. Even just sending a string or an integer results in an error on the server. Client/JavaScript: bullet.send(Bert.encode("testing")) Server/Erlang: ** Cowboy handler bullet_handler terminating in websocket_handle/3 for the reason error:badarg ** Message was {text,<<194,131,107,0,7,116,101,115,116,105,110,103>>} [...] ** Stacktrace: [{erlang,binary_to_term, [<<194,131,107,0,7,116,101,115,116,105,110,103>>],[]}, [...] Calling term_to_binary/1 using "testing" returns <<131,107,0,7,116,101,115,116,105,110,103>>, so I can only imagine it's that "194" part at the head of the binary object that's the problem, but I don't know why it's there and whether that's a result of the internals of the client or of the server. Any help? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rexxe98@REDACTED Mon Jul 1 09:32:42 2013 From: rexxe98@REDACTED (Andrew Berman) Date: Mon, 1 Jul 2013 00:32:42 -0700 Subject: [erlang-questions] Help Using Bullet + BERT In-Reply-To: References: Message-ID: Shouldn't you be using Bert's own decode function on the server side: https://github.com/mojombo/bert.erl On Mon, Jul 1, 2013 at 12:03 AM, Kolo Rahl wrote: > I've been messing around with Bullet and Cowboy recently and am loving how > easy it is to perform websocket handling. But then I tried integrating a > BERT JavaScript library with it (the one from 5HT/n2o) and noticed that the > messages hitting my server are in a form that can't be parsed by > binary_to_term/1, which is the whole point to to using BERT encoding. Even > just sending a string or an integer results in an error on the server. > > Client/JavaScript: > bullet.send(Bert.encode("testing")) > > Server/Erlang: > ** Cowboy handler bullet_handler terminating in websocket_handle/3 > for the reason error:badarg > ** Message was {text,<<194,131,107,0,7,116,101,115,116,105,110,103>>} > [...] > ** Stacktrace: [{erlang,binary_to_term, > [<<194,131,107,0,7,116,101,115,116,105,110,103>>],[]}, > [...] > > Calling term_to_binary/1 using "testing" returns > <<131,107,0,7,116,101,115,116,105,110,103>>, so I can only imagine it's > that "194" part at the head of the binary object that's the problem, but I > don't know why it's there and whether that's a result of the internals of > the client or of the server. Any help? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Mon Jul 1 09:43:29 2013 From: dangud@REDACTED (Dan Gudmundsson) Date: Mon, 1 Jul 2013 09:43:29 +0200 Subject: [erlang-questions] How to copy the results from the debug window? In-Reply-To: References: Message-ID: Including some relevant information would be great :-) Platform? Wx or gs? In wx some text fields are not by default copied to clipboard. So the programmer have to add that explicitly, trouble is that the event is not implemented yet in wx, it is on my todo list. But for this particular case it works for me, using linux and wx debugger. /Dan On Mon, Jul 1, 2013 at 2:40 AM, Yves S. Garret wrote: > Hi all, > > I'm debugging my Erlang code (with the built-in debugger) and would like > to copy the > results from the output of a variable (located in bottom-left in window > when > debugging a process) into a text editor in order to better visualize it. > However, > every time I try to copy/paste it, it's as if it's not picked up at all. > What am I > missing? > > Thanks in advance. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus@REDACTED Mon Jul 1 11:10:46 2013 From: magnus@REDACTED (Magnus Henoch) Date: Mon, 01 Jul 2013 10:10:46 +0100 Subject: [erlang-questions] Waiting for a node to start using run_erl In-Reply-To: (Garrett Smith's message of "Sat, 29 Jun 2013 09:13:01 -0500") References: Message-ID: Garrett Smith writes: > What is "started" in this case? > > A node start is a system startup, much like an OS start. There should > be a point that you system is "ready" to server/act/etc. Once you > establish that point, test for for it. > > Just as an example of why this isn't always straight forward, consider > a socket server (e.g. a web server). You might define "ready" as soon > as the server is accepting connections on a port, which is easy to > test for. But the server might in fact be ready several seconds later > as database connections are opened, caches warmed, whatever. So a > ready test might be a 200 response on a particular http request (which > is able to test for all those conditions). A definition of "started" that I have wished for more than once for run_erl is "all applications have started", i.e. the boot script has reached {progress,started}. Regards, Magnus From mrtndimitrov@REDACTED Mon Jul 1 11:13:19 2013 From: mrtndimitrov@REDACTED (Martin Dimitrov) Date: Mon, 01 Jul 2013 12:13:19 +0300 Subject: [erlang-questions] Restarting supervision tree Message-ID: <51D1482F.5090800@gmail.com> Hello, In our app, I would like to reset (restart) a whole supervision tree. If I return {stop, normal, []} from a permanent worker with supervisor's restart strategy set to one_for_all, the supervisor will terminate and restart the whole tree, right? But this will count towards the maximum restart frequency and I have to avoid this. If I call supervisor:terminate_child/2 and then supervisor:restart_child/2 on the permanent worker, will this affect the maximum restart frequency? Thank you. Martin From tomas.abrahamsson@REDACTED Mon Jul 1 11:33:14 2013 From: tomas.abrahamsson@REDACTED (Tomas Abrahamsson) Date: Mon, 1 Jul 2013 11:33:14 +0200 Subject: [erlang-questions] How to use gpb? In-Reply-To: References: Message-ID: Hi, you might try the to loop over proto files, like below: %% -*- erlang -*- {pre_hooks, [{compile, "mkdir -p include"}, %% ensure the include exists {compile, "bash -c 'for f in proto/*.proto; " "do " " erl +B -noinput -pa /path/to/gpb/ebin " " -I$(pwd)/proto -o-erl src -o-hrl include " " -s gpb_compile c $(pwd)/$f ; " "done'" }]}. {post_hooks, [{clean, "bash -c 'for f in proto/*.proto; " "do " " rm -f src/$(basename $f .proto).erl; " " rm -f include/$(basename $f .proto).hrl; " "done'" }]}. {erl_opts, [{i, "/path/to/gpb/include"}]}. It is currently not possible to specify several files using something like *.proto, but that's obviously a useful idea, I will fix that. BRs Tomas On Mon, Jul 1, 2013 at 5:58 AM, Bin Wang wrote: > Hi, Tomas > > Thanks for answering. But if I put all .proto files in directory > proto, how do I write the hooks to compile them all? I tried to use > *.proto instead of x.proto in pre_hooks, but it came with error: init > terminating in do_boot > > > 2013/6/29 Tomas Abrahamsson : > > Hi, Rebar by default uses another protobuf compiler for any *.proto > files it > > finds in the src subdirectory. > > > > One way to use gpb with rebar is to put your *.proto in another > > subdirectory, and use hooks to compile it. I got it working with the > > following stuff in rebar.config: > > > > %% -*- erlang -*- > > {pre_hooks, > > [{compile, "mkdir -p include"}, %% ensure the include dir exists > > {compile, > > "erl +B -noinput -pa /path/to/gpb/ebin " > > "-I`pwd`/proto -o-erl src -o-hrl include " > > "-s gpb_compile c `pwd`/proto/x.proto" > > }]}. > > > > {post_hooks, > > [{clean, "rm -f src/x.erl include/x.hrl"}]}. > > > > {erl_opts, [{i, "/path/to/gpb/include"}]}. > > > > and the following initial file/directory layout: > > > > prompt# find . > > . > > ./proto > > ./proto/x.proto > > ./src > > ./src/g_sup.erl > > ./src/g.app.src > > ./src/g_app.erl > > ./rebar.config > > > > BRs > > Tomas > > > > > > On Fri, Jun 28, 2013 at 11:08 AM, Bin Wang wrote: > >> > >> Hi, > >> > >> I'm using google protocol buffers. I found an erlang plugin named > >> gpb(https://github.com/tomas-abrahamsson/gpb). But I don't know how > >> to use it. When I add it as a dependency of my app using rebar, while > >> compile, it always comes out this error: > >> > >> ERROR: Protobuffs library not present in code path! > >> > >> Which I searched in google and only found it in erlang_protobuffs > >> code. I do use it as a dependency before, but dose it matters? > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Mon Jul 1 12:15:46 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Mon, 01 Jul 2013 11:15:46 +0100 Subject: [erlang-questions] Erlang and/or MPI In-Reply-To: References: Message-ID: <51D156D2.8090208@llaisdy.com> On 01/07/2013 05:15, Richard A. O'Keefe wrote: > ... > (1) How hard is it to *accurately* transcribe a correct description > of an algorithm into the language in question? > > Here Ada, Fortran, Haskell, and Ada shine. Is there a secret message here about Ada? -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From essen@REDACTED Mon Jul 1 12:57:03 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 01 Jul 2013 12:57:03 +0200 Subject: [erlang-questions] Help Using Bullet + BERT In-Reply-To: References: Message-ID: <51D1607F.50706@ninenines.eu> Note that bullet only supports text data right now which must be valid UTF-8 with some transports like Websocket. So what I'm guessing happens is that the browser encodes your binary to be valid UTF-8 and messes things up. To use binary data with bullet you need to base64 encode it or something similar. On 07/01/2013 09:03 AM, Kolo Rahl wrote: > I've been messing around with Bullet and Cowboy recently and am loving > how easy it is to perform websocket handling. But then I tried > integrating a BERT JavaScript library with it (the one from 5HT/n2o) and > noticed that the messages hitting my server are in a form that can't be > parsed by binary_to_term/1, which is the whole point to to using BERT > encoding. Even just sending a string or an integer results in an error > on the server. > > Client/JavaScript: > bullet.send(Bert.encode("testing")) > > Server/Erlang: > ** Cowboy handler bullet_handler terminating in websocket_handle/3 > for the reason error:badarg > ** Message was {text,<<194,131,107,0,7,116,101,115,116,105,110,103>>} > [...] > ** Stacktrace: [{erlang,binary_to_term, > [<<194,131,107,0,7,116,101,115,116,105,110,103>>],[]}, > [...] > > Calling term_to_binary/1 using "testing" returns > <<131,107,0,7,116,101,115,116,105,110,103>>, so I can only imagine it's > that "194" part at the head of the binary object that's the problem, but > I don't know why it's there and whether that's a result of the internals > of the client or of the server. Any help? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From dmercer@REDACTED Mon Jul 1 16:34:46 2013 From: dmercer@REDACTED (David Mercer) Date: Mon, 1 Jul 2013 09:34:46 -0500 Subject: [erlang-questions] Erlang and/or MPI In-Reply-To: References: Message-ID: <005b01ce7668$230c0630$69241290$@gmail.com> On Sunday, June 30, 2013, Richard A. O'Keefe wrote: > (1) How hard is it to *accurately* transcribe a correct description > of an algorithm into the language in question? > > Here Ada, Fortran, Haskell, and Ada shine. > Smalltalk and Scheme, for all their other virtues, stink. > Erlang is somewhere in the middle. Can you expand on this? I've usually thought most programming languages make it fairly easy to transcribe an algorithm. But even if not, why would Scheme be particularly bad at it? Cheers, DBM From danbenderr@REDACTED Mon Jul 1 15:10:49 2013 From: danbenderr@REDACTED (Dan Bender) Date: Mon, 1 Jul 2013 16:10:49 +0300 Subject: [erlang-questions] Real time audio transcoding with a port driver Message-ID: Hi I need to transcode live audio streams with avconv with a port driver. I need to be able to handle hundreds of concurrent streams. What is the cost of creating a port driver? Is it ok to create a separate driver instance for each stream or should a single driver handle all the streams? I know that I can create hundreds of thousands of erlang processes. Is this also true for port drivers or am I supposed to use no more than a few dozen. What is the cost of sending binary audio packets to the driver and receiving the binary result? Is it important to do as much work as possible in a single port driver or is it ok for example to separate the encoding and decoding to two separate drivers? Will using a port driver scale when using a large number of streams? Will the port driver block erlang processes? Is there a better option than a port driver? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Jul 1 16:52:56 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Jul 2013 18:52:56 +0400 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: You have following choices: 1) start program based on libav, one per stream and speak with it via port. You cannot use avconv directly because it will be unmaintainable. 2) launch driver based on libav and schedule transcoding to async threads 3) transcode in NIF. I think that if you have really hundreds of streams, than 2-nd choice is best, because it is much better to schedule 20-40 threads than 400 processes. From yoursurrogategod@REDACTED Mon Jul 1 17:29:16 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Mon, 1 Jul 2013 11:29:16 -0400 Subject: [erlang-questions] How to copy the results from the debug window? In-Reply-To: References: Message-ID: Every time I startup the debugger, I get this message: 1> debugger:start(). =ERROR REPORT==== 1-Jul-2013::11:28:08 === WX Failed loading "wxe_driver"@"/usr/local/lib/erlang/lib/wx-1.0/priv" {ok,<0.36.0>} I'm assuming that it's the standard Erlang GUI. gs I presume? On Mon, Jul 1, 2013 at 3:43 AM, Dan Gudmundsson wrote: > Including some relevant information would be great :-) > Platform? Wx or gs? > > In wx some text fields are not by default copied to clipboard. > > So the programmer have to add that explicitly, trouble is that the event > is not implemented yet > in wx, it is on my todo list. > > But for this particular case it works for me, using linux and wx debugger. > > /Dan > > > > On Mon, Jul 1, 2013 at 2:40 AM, Yves S. Garret > wrote: > >> Hi all, >> >> I'm debugging my Erlang code (with the built-in debugger) and would like >> to copy the >> results from the output of a variable (located in bottom-left in window >> when >> debugging a process) into a text editor in order to better visualize it. >> However, >> every time I try to copy/paste it, it's as if it's not picked up at all. >> What am I >> missing? >> >> Thanks in advance. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Mon Jul 1 17:55:48 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 1 Jul 2013 17:55:48 +0200 Subject: [erlang-questions] [ANN] nat_upnp Message-ID: Just released nat_upnp a simple library that provides you a way to map a local port to the external using UPNP IGD [1] : https://github.com/benoitc/nat_upnp While the old library I extracted fron etorrent [2] was working it was pretty complicated to use, and you couldn't dynamically add or remove a port from your router. This one does it, also the code and the design is more simple. It is only using the bare minimum of the UPNP spec and no gen_server. Also this library is only based on the standard library in Erlang. Any feedback is welcome, Enjoy ! [1] http://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol [2] https://github.com/benoitc/upnp From max.lapshin@REDACTED Mon Jul 1 18:10:06 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Jul 2013 20:10:06 +0400 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: Sorry, but what for do you required to start inets? There is call to xmerl_scan which doesn't require started inets. On Mon, Jul 1, 2013 at 7:55 PM, Benoit Chesneau wrote: > Just released nat_upnp a simple library that provides you a way to > map a local port to the external using UPNP IGD [1] : > > https://github.com/benoitc/nat_upnp > > While the old library I extracted fron etorrent [2] was working it was > pretty complicated to use, and you couldn't dynamically add or remove > a port from your router. This one does it, also the code and the > design is more simple. It is only using the bare minimum of the UPNP > spec and no gen_server. Also this library is only based on the > standard library in Erlang. > > Any feedback is welcome, > > Enjoy ! > > [1] http://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol > [2] https://github.com/benoitc/upnp > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bchesneau@REDACTED Mon Jul 1 18:15:52 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 1 Jul 2013 18:15:52 +0200 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: On Mon, Jul 1, 2013 at 6:10 PM, Max Lapshin wrote: > Sorry, but what for do you required to start inets? There is call to > xmerl_scan which doesn't require started inets. inets need to be started to use httpc : https://github.com/benoitc/nat_upnp/blob/master/src/nat_upnp_proto.erl#L193 I didn't want to use any external lib for this one, and httpc can't be used without having inets started unfortunately. Or did I miss something? - benoit > > > On Mon, Jul 1, 2013 at 7:55 PM, Benoit Chesneau wrote: >> Just released nat_upnp a simple library that provides you a way to >> map a local port to the external using UPNP IGD [1] : >> >> https://github.com/benoitc/nat_upnp >> >> While the old library I extracted fron etorrent [2] was working it was >> pretty complicated to use, and you couldn't dynamically add or remove >> a port from your router. This one does it, also the code and the >> design is more simple. It is only using the bare minimum of the UPNP >> spec and no gen_server. Also this library is only based on the >> standard library in Erlang. >> >> Any feedback is welcome, >> >> Enjoy ! >> >> [1] http://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol >> [2] https://github.com/benoitc/upnp >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Mon Jul 1 18:17:02 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Jul 2013 20:17:02 +0400 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: You are right, I've missed this. From straightflush@REDACTED Mon Jul 1 19:06:02 2013 From: straightflush@REDACTED (AD) Date: Mon, 1 Jul 2013 13:06:02 -0400 Subject: [erlang-questions] Help Using Bullet + BERT In-Reply-To: <51D1607F.50706@ninenines.eu> References: <51D1607F.50706@ninenines.eu> Message-ID: I believe this is why tools like SockJS use their own JSON binary encoding on the wire On Mon, Jul 1, 2013 at 6:57 AM, Lo?c Hoguin wrote: > Note that bullet only supports text data right now which must be valid > UTF-8 with some transports like Websocket. So what I'm guessing happens is > that the browser encodes your binary to be valid UTF-8 and messes things > up. To use binary data with bullet you need to base64 encode it or > something similar. > > > On 07/01/2013 09:03 AM, Kolo Rahl wrote: > >> I've been messing around with Bullet and Cowboy recently and am loving >> how easy it is to perform websocket handling. But then I tried >> integrating a BERT JavaScript library with it (the one from 5HT/n2o) and >> noticed that the messages hitting my server are in a form that can't be >> parsed by binary_to_term/1, which is the whole point to to using BERT >> encoding. Even just sending a string or an integer results in an error >> on the server. >> >> Client/JavaScript: >> bullet.send(Bert.encode("**testing")) >> >> Server/Erlang: >> ** Cowboy handler bullet_handler terminating in websocket_handle/3 >> for the reason error:badarg >> ** Message was {text,<<194,131,107,0,7,116,**101,115,116,105,110,103>>} >> [...] >> ** Stacktrace: [{erlang,binary_to_term, >> [<<194,131,107,0,7,116,101,**115,116,105,110,103>>],[]}, >> [...] >> >> Calling term_to_binary/1 using "testing" returns >> <<131,107,0,7,116,101,115,116,**105,110,103>>, so I can only imagine it's >> that "194" part at the head of the binary object that's the problem, but >> I don't know why it's there and whether that's a result of the internals >> of the client or of the server. Any help? >> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> >> > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Jul 1 21:37:37 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Jul 2013 23:37:37 +0400 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: I'll take a look if it is easy to add MediaServer to upnp registration. From steven.charles.davis@REDACTED Tue Jul 2 00:54:07 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Mon, 1 Jul 2013 15:54:07 -0700 (PDT) Subject: [erlang-questions] Help Using Bullet + BERT In-Reply-To: References: Message-ID: Hi, Not sure, but you may be looking for this js snippet: ws = new WebSocket(ws_url, ws_protocol); ws.binaryType = "arraybuffer"; /s On Monday, July 1, 2013 2:03:18 AM UTC-5, Kolo Rahl wrote: > > I've been messing around with Bullet and Cowboy recently and am loving how > easy it is to perform websocket handling. But then I tried integrating a > BERT JavaScript library with it (the one from 5HT/n2o) and noticed that the > messages hitting my server are in a form that can't be parsed by > binary_to_term/1, which is the whole point to to using BERT encoding. Even > just sending a string or an integer results in an error on the server. > > Client/JavaScript: > bullet.send(Bert.encode("testing")) > > Server/Erlang: > ** Cowboy handler bullet_handler terminating in websocket_handle/3 > for the reason error:badarg > ** Message was {text,<<194,131,107,0,7,116,101,115,116,105,110,103>>} > [...] > ** Stacktrace: [{erlang,binary_to_term, > [<<194,131,107,0,7,116,101,115,116,105,110,103>>],[]}, > [...] > > Calling term_to_binary/1 using "testing" returns > <<131,107,0,7,116,101,115,116,105,110,103>>, so I can only imagine it's > that "194" part at the head of the binary object that's the problem, but I > don't know why it's there and whether that's a result of the internals of > the client or of the server. Any help? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Jul 2 03:20:54 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 2 Jul 2013 13:20:54 +1200 Subject: [erlang-questions] Erlang and/or MPI In-Reply-To: <005b01ce7668$230c0630$69241290$@gmail.com> References: <005b01ce7668$230c0630$69241290$@gmail.com> Message-ID: On 2/07/2013, at 2:34 AM, David Mercer wrote: > On Sunday, June 30, 2013, Richard A. O'Keefe wrote: > >> (1) How hard is it to *accurately* transcribe a correct description >> of an algorithm into the language in question? >> >> Here Ada, Fortran, Haskell, and Ada shine. >> Smalltalk and Scheme, for all their other virtues, stink. >> Erlang is somewhere in the middle. > > Can you expand on this? I've usually thought most programming languages make it fairly easy to transcribe an algorithm. But even if not, why would Scheme be particularly bad at it? Because the syntax is utterly unlike the original presentation. Let's take the square root of a Quaternion as an example. Let's hope I don't break it in the process! Maths: sqrt (w,x,y,z) = (r/2,x/r,y/r,z/r) where r = sqrt((sqrt(w^2+n^2)+w)*2), if n > 0 (0,sqrt(-w),0,0), if n = 0 and w < 0 (sqrt(w),0,0,0), if n = 0 and w >= 0 where n = sqrt(x^2+y^2+z^2) Haskell: sqrt q@(Quaternion w x y z) | n > 0 = let r = sqrt ((hypot w n + w)*2 in Quaternion (r/2) (x/r) (y/r) (z/r) | w < 0 = Quaternion 0 (sqrt (negate w)) 0 0 | w >= 0 = Quaternion (sqrt w) 0 0 0 where n = vector_part_abs q This is very close to the mathematical form. The guards and bodies are flipped (Miranda had them the other way around). The other difference is that it uses hypot w n and vector_part_abs q instead of sqrt(w^2+n^2) and sqrt(x^2+y^2+z^2) for computational reasons. Erlang: qsqrt(Q = {quaternion,W,X,Y,Z}) -> N = vector_part_abs(Q), if N > 0 -> R = math:sqrt((math:hypot(W, N) + N)*2), {quaternion,R/2,X/R,Y/R,Z/R} ; W < 0 -> {quaternion,0,math:sqrt(-W),0,0} ; true -> {quaternion,math:sqrt(W),0,0,0} end. Not too bad, actually, except that it doesn't work. For some reason beyond the ability of mortals to discern, math:sqrt/1 exists but math:hypot/2 does not. (It's only been part of standard C for the last 14 years...) erf/1 and erfc/1 there but hypot/2 not? It's touch clumsy with repeating the 'math:' prefix so much, but I don't actually have to do that. I _could_ -import(math, [sqrt/1,hypot/2]). if math:hypot/2 existed and then I wouldn't need the prefixes. C99: quaternion qsqrt(quaternion q) { double const n = vector_part_abs(q); double const w = q.w; if (n > 0) { double const r = sqrt((hypot(w, n) + w)*2); return (quaternion){ .w = r/2, .x = q.x/r, .y = q.y/r, .z = q.z/r}; } else if (w < 0) { return (quaternion){.w = 0, .x = sqrt(-w), .y = 0, .z = 0}; } else { return (quaternion){.w = sqrt(w), .x = 0, .y = 0, .z = 0}; } } Still pretty close to the mathematics; a little clunky at making records and a lot clunky at taking them apart. Smalltalk: sqrt |n r| n := self vectorPartAbs. ^n > 0 ifTrue: [ r := ((w hypot: n) + w) * 2) sqrt. Quaternion re: r/2 im: x/r jm: y/r km: z/r] ifFalse: [ w < 0 ifTrue: [Quaternion re: 0 im: w negated sqrt jm: 0 km: 0] ifFalse: [Quaternion re: w sqrt im: 0 jm: 0 km: 0]] I think you will agree that converting sqrt(-w) to w negated sqrt is a little distracting. What you may not know is that the operator precedence rules for Smalltalk, well, technically Smalltalk doesn't have operators, so 1 + 2 * 3 is 9. Moving unary function names from the left to the right, and adding extra parentheses whenever there are two operators in the same expression is just enough restructuring to make it painfully easy to mess up. Now let's look at Scheme. Let's suppose we have a constructor and accessors available already. R6RS Scheme has define-record-type; R5RS Scheme did not, but there were plenty of free equivalents. There are also free add-ons to give you pattern matching of a sort; I don't know how well they handle R6RS records though. Let's also skip over the Erlang-like nonexistence of hypot. (define (quaternion-sqrt Q) (let ((N (vector-part-abs Q)) (W (quaternion-re Q))) (cond ((> N 0) (let ((R (sqrt (* (+ (hypot W N) W) 2)))) (make-quaternion (/ R 2) (/ (quaternion-im Q) R) (/ (quaternion-jm Q) R) (/ (quaternion-km Q) R)))) ((< W 0) (make-quaternion 0 (sqrt (- W)) 0 0)) (else (make-quaternion (sqrt W) 0 0 0))))) This particular example isn't _too_ horrible. But I invite you to consider (1) r = sqrt ((hypot w n + w)*2 vs (R (sqrt (* (+ (hypot W N) W) 2))) Transcribing that was not pleasant and I don't have a high degree of confidence that it's right. (2) the fact that the arithmetic operators can be overloaded in Haskell, Ada, R, Smalltalk, Eiffel, Fortran,... to apply to Quaternions, but they cannot be in Erlang or Scheme. Yes, I know about the possibility of enclosing a block of code in (let ((outer-plus +) (outer-minus -) (outer-times *) ...) (define (+ x y) (if (quaternion? x) (quaternion-plus x y) (if (quaternion? y) (quaternion-plus y x) (outer-plus x y)))) ...) but I really don't want to do that, thanks. The thing is, once Quaternions are properly set up in Haskell or Fortran, I can write abs(cos(x+y) - (cos(x)*cos(y) - sin(x)*sin(y))) and it will Just Work. (The reason this is an interesting thing to compute is that for real and complex numbers, the answer should be zero, but for Quaternions in general it will not be.) Again, in Haskell and Smalltalk I can set things up so that A*V + W does a vector operation -- and Haskell can let me ensure that it won't create an intermediate object, thanks to Fusion -- while in Fortran and R it already does. This entirely eliminates wrong-loop-variable and off-by-one errors from the calculation AND gives me a mathematically appropriate notation. I don't deny that Scheme is a fine language. I don't deny that Scheme is perfectly up to the task of describing any reasonable algorithm. Nor am I making any claim about the *performance* of any particular Scheme system, such as Chez (which has some impressive theory behind its compiler) or Racket. All I say is that the act of *transcribing* a *numerical* algorithm from a published paper or technical report to a computational form is likely to have fewer errors the closer the two notational systems are. From kolorahl@REDACTED Tue Jul 2 01:01:58 2013 From: kolorahl@REDACTED (Kolo Rahl) Date: Mon, 1 Jul 2013 16:01:58 -0700 Subject: [erlang-questions] Help Using Bullet + BERT In-Reply-To: References: Message-ID: @Andrew Looking at the source code, bertl.erl looks to call binary_to_term, with the only difference being that it wraps calls in a tuple to signify success or error. So calling binary_to_term/1 directly should work if it's a properly-formatted BERT message. @Lo?c Yeah, I figured it was along those lines, but was hoping it was something more easily fixed =P @Steve I'll give that a try, thanks. On Mon, Jul 1, 2013 at 3:54 PM, Steve Davis wrote: > Hi, > > Not sure, but you may be looking for this js snippet: > ws = new WebSocket(ws_url, ws_protocol); > ws.binaryType = "arraybuffer"; > > /s > > On Monday, July 1, 2013 2:03:18 AM UTC-5, Kolo Rahl wrote: >> >> I've been messing around with Bullet and Cowboy recently and am loving >> how easy it is to perform websocket handling. But then I tried integrating >> a BERT JavaScript library with it (the one from 5HT/n2o) and noticed that >> the messages hitting my server are in a form that can't be parsed by >> binary_to_term/1, which is the whole point to to using BERT encoding. Even >> just sending a string or an integer results in an error on the server. >> >> Client/JavaScript: >> bullet.send(Bert.encode("**testing")) >> >> Server/Erlang: >> ** Cowboy handler bullet_handler terminating in websocket_handle/3 >> for the reason error:badarg >> ** Message was {text,<<194,131,107,0,7,116,**101,115,116,105,110,103>>} >> [...] >> ** Stacktrace: [{erlang,binary_to_term, [<<194,131,107,0,7,116,101,** >> 115,116,105,110,103>>],[]}, >> [...] >> >> Calling term_to_binary/1 using "testing" returns >> <<131,107,0,7,116,101,115,116,**105,110,103>>, so I can only imagine >> it's that "194" part at the head of the binary object that's the problem, >> but I don't know why it's there and whether that's a result of the >> internals of the client or of the server. Any help? >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Tue Jul 2 08:53:01 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 2 Jul 2013 08:53:01 +0200 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: On Mon, Jul 1, 2013 at 9:37 PM, Max Lapshin wrote: > I'll take a look if it is easy to add MediaServer to upnp registration. Cool :) Let me know if you find anything. - benoit From drormein@REDACTED Tue Jul 2 08:47:33 2013 From: drormein@REDACTED (Dror Mein) Date: Mon, 1 Jul 2013 23:47:33 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed Message-ID: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> I'm trying to send data between two nodes on two different servers, that talk through a 10 Gbps switch. Running iperf on the connection shows consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, when looking through iptraf, and all I do is send data from one node to a dummy process on the other. Is there a limitation on connection speed on erlang? Is there a flag that need to be configured? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Tue Jul 2 09:11:09 2013 From: gleber.p@REDACTED (Gleb Peregud) Date: Tue, 2 Jul 2013 09:11:09 +0200 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> Message-ID: I was able to get 5+gbps over multiple connections out of a single Erlang VM. Max probably did even more. There is no explicit limit in connection speed. Show us your code and we'll probably be able to tell you why it can not get good throughput. On Tue, Jul 2, 2013 at 8:47 AM, Dror Mein wrote: > I'm trying to send data between two nodes on two different servers, that > talk through a 10 Gbps switch. Running iperf on the connection shows > consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, when > looking through iptraf, and all I do is send data from one node to a dummy > process on the other. > Is there a limitation on connection speed on erlang? Is there a flag that > need to be configured? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From drormein@REDACTED Tue Jul 2 09:20:47 2013 From: drormein@REDACTED (Dror Mein) Date: Tue, 2 Jul 2013 00:20:47 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> Message-ID: <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> >From shell: 1> SendFunc = fun(Pid, Msg, Times, Func) when Times>0 -> Pid ! Msg, Func(Pid, Msg, Times-1. Func); (_,_,0,_)-> ok end. 2> F = SendFunc(Pid, <<00>>, 300000000, SendFunc). 3> spawn(F). where Pid is the pid of a simple receive function on the other node, that counts messages.? ________________________________ From: Gleb Peregud To: Dror Mein Cc: Erlang-Questions Questions Sent: Tuesday, July 2, 2013 10:11 AM Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed I was able to get 5+gbps over multiple connections out of a single Erlang VM. Max probably did even more. There is no explicit limit in connection speed. Show us your code and we'll probably be able to tell you why it can not get good throughput. On Tue, Jul 2, 2013 at 8:47 AM, Dror Mein wrote: > I'm trying to send data between two nodes on two different servers, that > talk through a 10 Gbps switch. Running iperf on the connection shows > consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, when > looking through iptraf, and all I do is send data from one node to a dummy > process on the other. > Is there a limitation on connection speed on erlang? Is there a flag that > need to be configured? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Tue Jul 2 09:41:41 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Tue, 2 Jul 2013 09:41:41 +0200 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> Message-ID: It only goes up to 300Mbps because you are sending a huge number of very small messages, between two processes over a single TCP connection. Is this your actual use case? Sergej On Jul 2, 2013, at 9:20 AM, Dror Mein wrote: > From shell: > 1> SendFunc = fun(Pid, Msg, Times, Func) when Times>0 -> Pid ! Msg, Func(Pid, Msg, Times-1. Func); (_,_,0,_)-> ok end. > 2> F = SendFunc(Pid, <<00>>, 300000000, SendFunc). > 3> spawn(F). > > where Pid is the pid of a simple receive function on the other node, that counts messages. > > From: Gleb Peregud > To: Dror Mein > Cc: Erlang-Questions Questions > Sent: Tuesday, July 2, 2013 10:11 AM > Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed > > I was able to get 5+gbps over multiple connections out of a single > Erlang VM. Max probably did even more. There is no explicit limit in > connection speed. Show us your code and we'll probably be able to tell > you why it can not get good throughput. > > On Tue, Jul 2, 2013 at 8:47 AM, Dror Mein wrote: > > I'm trying to send data between two nodes on two different servers, that > > talk through a 10 Gbps switch. Running iperf on the connection shows > > consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, when > > looking through iptraf, and all I do is send data from one node to a dummy > > process on the other. > > Is there a limitation on connection speed on erlang? Is there a flag that > > need to be configured? > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From drormein@REDACTED Tue Jul 2 10:08:56 2013 From: drormein@REDACTED (Dror Mein) Date: Tue, 2 Jul 2013 01:08:56 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> Message-ID: <1372752536.21228.YahooMailNeo@web164503.mail.gq1.yahoo.com> the message size will be around 100 bytes, but basically - that's what I want, yeah. when I use more servers, this bandwidth spreads between them. ________________________________ From: Sergej Jurecko To: Dror Mein Cc: Gleb Peregud ; Erlang-Questions Questions Sent: Tuesday, July 2, 2013 10:41 AM Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed It only goes up to 300Mbps because you are sending a huge number of very small messages, between two processes over a single TCP connection.? Is this your actual use case?? Sergej On Jul 2, 2013, at 9:20 AM, Dror Mein wrote: From shell: >1> SendFunc = fun(Pid, Msg, Times, Func) when Times>0 -> Pid ! Msg, Func(Pid, Msg, Times-1. Func); (_,_,0,_)-> ok end. >2> F = SendFunc(Pid, <<00>>, 300000000, SendFunc). >3> spawn(F). > > >where Pid is the pid of a simple receive function on the other node, that counts messages.? > > > >________________________________ > From: Gleb Peregud >To: Dror Mein >Cc: Erlang-Questions Questions >Sent: Tuesday, July 2, 2013 10:11 AM >Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed > > >I was able to get 5+gbps over multiple connections out of a single >Erlang VM. Max probably did even more. There is no explicit limit in >connection speed. Show us your code and we'll probably be able to tell >you why it can not get good throughput. > >On Tue, Jul 2, 2013 at 8:47 AM, Dror Mein wrote: >> I'm trying to send data between two nodes on two different servers, that >> talk through a 10 Gbps switch. Running iperf on the connection shows >> consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, when >> looking through iptraf, and all I do is send data from one node to a dummy >> process on the other. >> Is there a limitation on connection speed on erlang? Is there a flag that >> need to be configured? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Tue Jul 2 10:16:43 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 2 Jul 2013 10:16:43 +0200 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <1372752536.21228.YahooMailNeo@web164503.mail.gq1.yahoo.com> References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> <1372752536.21228.YahooMailNeo@web164503.mail.gq1.yahoo.com> Message-ID: How about the "between two processes" part, is that accurate? Or are there more senders and receivers involved? Den 02/07/2013 10.09 skrev "Dror Mein" : > the message size will be around 100 bytes, but basically - that's what I > want, yeah. when I use more servers, this bandwidth spreads between them. > > ------------------------------ > *From:* Sergej Jurecko > *To:* Dror Mein > *Cc:* Gleb Peregud ; Erlang-Questions Questions < > erlang-questions@REDACTED> > *Sent:* Tuesday, July 2, 2013 10:41 AM > *Subject:* Re: [erlang-questions] Is there a lomit on erlang vm nodes > connection speed > > It only goes up to 300Mbps because you are sending a huge number of very > small messages, between two processes over a single TCP connection. > > Is this your actual use case? > > > Sergej > > On Jul 2, 2013, at 9:20 AM, Dror Mein wrote: > > From shell: > 1> SendFunc = fun(Pid, Msg, Times, Func) when Times>0 -> Pid ! Msg, > Func(Pid, Msg, Times-1. Func); (_,_,0,_)-> ok end. > 2> F = SendFunc(Pid, <<00>>, 300000000, SendFunc). > 3> spawn(F). > > where Pid is the pid of a simple receive function on the other node, that > counts messages. > > ------------------------------ > *From:* Gleb Peregud > *To:* Dror Mein > *Cc:* Erlang-Questions Questions > *Sent:* Tuesday, July 2, 2013 10:11 AM > *Subject:* Re: [erlang-questions] Is there a lomit on erlang vm nodes > connection speed > > I was able to get 5+gbps over multiple connections out of a single > Erlang VM. Max probably did even more. There is no explicit limit in > connection speed. Show us your code and we'll probably be able to tell > you why it can not get good throughput. > > On Tue, Jul 2, 2013 at 8:47 AM, Dror Mein wrote: > > I'm trying to send data between two nodes on two different servers, that > > talk through a 10 Gbps switch. Running iperf on the connection shows > > consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, > when > > looking through iptraf, and all I do is send data from one node to a > dummy > > process on the other. > > Is there a limitation on connection speed on erlang? Is there a flag that > > need to be configured? > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drormein@REDACTED Tue Jul 2 10:35:57 2013 From: drormein@REDACTED (Dror Mein) Date: Tue, 2 Jul 2013 01:35:57 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> <1372752536.21228.YahooMailNeo@web164503.mail.gq1.yahoo.com> Message-ID: <1372754157.36974.YahooMailNeo@web164503.mail.gq1.yahoo.com> currently only two processes, perhaps more in the future. To add load I tried to multiply the senders, and it didn't help. checking the message queue length in the receiver, It's almost always 0 durimg the load test, so I assume there is no problem in the receiver. ________________________________ From: Erik S?e S?rensen To: Dror Mein Cc: Sergej Jurecko ; Erlang Questions Sent: Tuesday, July 2, 2013 11:16 AM Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed How about the "between two processes" part, is that accurate? Or are there more senders and receivers involved? Den 02/07/2013 10.09 skrev "Dror Mein" : the message size will be around 100 bytes, but basically - that's what I want, yeah. when I use more servers, this bandwidth spreads between them. > > > >________________________________ > From: Sergej Jurecko >To: Dror Mein >Cc: Gleb Peregud ; Erlang-Questions Questions >Sent: Tuesday, July 2, 2013 10:41 AM >Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed > > > >It only goes up to 300Mbps because you are sending a huge number of very small messages, between two processes over a single TCP connection.? > > > >Is this your actual use case?? > > > > >Sergej > > >On Jul 2, 2013, at 9:20 AM, Dror Mein wrote: > >From shell: >>1> SendFunc = fun(Pid, Msg, Times, Func) when Times>0 -> Pid ! Msg, Func(Pid, Msg, Times-1. Func); (_,_,0,_)-> ok end. >>2> F = SendFunc(Pid, <<00>>, 300000000, SendFunc). >>3> spawn(F). >> >> >>where Pid is the pid of a simple receive function on the other node, that counts messages.? >> >> >> >>________________________________ >> From: Gleb Peregud >>To: Dror Mein >>Cc: Erlang-Questions Questions >>Sent: Tuesday, July 2, 2013 10:11 AM >>Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed >> >> >>I was able to get 5+gbps over multiple connections out of a single >>Erlang VM. Max probably did even more. There is no explicit limit in >>connection speed. Show us your code and we'll probably be able to tell >>you why it can not get good throughput. >> >>On Tue, Jul 2, 2013 at 8:47 AM, Dror Mein wrote: >>> I'm trying to send data between two nodes on two different servers, that >>> talk through a 10 Gbps switch. Running iperf on the connection shows >>> consistent 5-7 gbps throughput. But with Erlang I can't exceed 300Mbps, when >>> looking through iptraf, and all I do is send data from one node to a dummy >>> process on the other. >>> Is there a limitation on connection speed on erlang? Is there a flag that >>> need to be configured? >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >>_______________________________________________ >>erlang-questions mailing list >>erlang-questions@REDACTED >>http://erlang.org/mailman/listinfo/erlang-questions >> > > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Jul 2 10:52:09 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 2 Jul 2013 12:52:09 +0400 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: > When you suggest using libav, do you mean something like the following? > https://github.com/libav/libav/blob/master/libavcodec/api-example.c > Yes, something like this. But I had to add additional thread that controls parent process and stdin and exits immediately. Also I had to add additional protocol to ffmpeg that packs frames into erlang #video_frame{} record in external format and sends it in packet,4 mode to stdout. > Using a port driver per stream will not scale? Are you speaking about port (external program) or about driver (library running inside erlang VM)? > Is there an example for using async threads to do a similar task? > Will my transcoding compete with file I/O because they are using the same threads? Yes. > How do I know how many threads to use? > Practice. > Why not use NIF? > Encoding/decoding a single frame will probably take about 3ms. > If the NIF will bump reductions, will this be ok? I can't tell you how will it work with new erlang. From danbenderr@REDACTED Tue Jul 2 10:41:17 2013 From: danbenderr@REDACTED (Dan Bender) Date: Tue, 2 Jul 2013 11:41:17 +0300 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: When you suggest using libav, do you mean something like the following? https://github.com/libav/libav/blob/master/libavcodec/api-example.c Using a port driver per stream will not scale? Is there an example for using async threads to do a similar task? Will my transcoding compete with file I/O because they are using the same threads? How do I know how many threads to use? Why not use NIF? Encoding/decoding a single frame will probably take about 3ms. If the NIF will bump reductions, will this be ok? On Mon, Jul 1, 2013 at 5:52 PM, Max Lapshin wrote: > You have following choices: > > 1) start program based on libav, one per stream and speak with it via > port. You cannot use avconv directly because it will be > unmaintainable. > 2) launch driver based on libav and schedule transcoding to async threads > 3) transcode in NIF. > > I think that if you have really hundreds of streams, than 2-nd choice > is best, because it is much better to schedule 20-40 threads than 400 > processes. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jul 2 11:07:06 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 2 Jul 2013 11:07:06 +0200 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 10:41 AM, Dan Bender wrote: > Why not use NIF? > Encoding/decoding a single frame will probably take about 3ms. > If the NIF will bump reductions, will this be ok? > A NIF should run for at most 1ms. So you will have to break the frame into multiple cooperative/reentrant calls and call your NIF several times, or you need to spawn a background thread which runs the code and then replies when it is done, usually through a message. Otherwise, you are blocking the scheduler and this is a problem which can mess up the internal scheduler timing in Erlang. This problem *might* be solved in R17 where so-called dirty schedulers will be present. These are probably able to solve this problem in another way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jul 2 11:13:23 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 2 Jul 2013 11:13:23 +0200 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> Message-ID: On Tue, Jul 2, 2013 at 9:20 AM, Dror Mein wrote: > From shell: Note: The shell is interpreted and you are building a closure for no reason. Try pushing your code into a module and then compiling that module to bytecode and run it again. Chances are you are basically just being bitten by interpretation overhead. Also, you should probably increase the message size to around 100 bytes if that is what you expect to send and receive. There is a messaging overhead which you are paying as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Jul 2 11:17:16 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 2 Jul 2013 13:17:16 +0400 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: It is also possible to revert to R15, but in any case if NIF will block scheduler, you will have problems with managing hunders of streams. From drormein@REDACTED Tue Jul 2 12:10:01 2013 From: drormein@REDACTED (Dror Mein) Date: Tue, 2 Jul 2013 03:10:01 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> Message-ID: <1372759801.10432.YahooMailNeo@web164505.mail.gq1.yahoo.com> You were right. increasing the message size to 10000 allowed me to reach 1-2Gbps. I guess the simple solution is to collate messages in a list and send them together. Is there another way you can think of? Can I configure the erlang vm to open more tcp connections between two nodes? will that solve the problem? ________________________________ From: Jesper Louis Andersen To: Dror Mein Cc: Gleb Peregud ; Erlang-Questions Questions Sent: Tuesday, July 2, 2013 12:13 PM Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed On Tue, Jul 2, 2013 at 9:20 AM, Dror Mein wrote: >From shell: Note: The shell is interpreted and you are building a closure for no reason. Try pushing your code into a module and then compiling that module to bytecode and run it again. Chances are you are basically just being bitten by interpretation overhead. Also, you should probably increase the message size to around 100 bytes if that is what you expect to send and receive. There is a messaging overhead which you are paying as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.lundin@REDACTED Tue Jul 2 12:16:59 2013 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Tue, 2 Jul 2013 12:16:59 +0200 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: Hi Max, I don't know why you are thinking that this will work better in R15 than in R16? Can you point to any specific change in R16 that causes problems? It has always been a bad idea to execute for a long time (before returning to the Erlang scheduler) inside a NIF, BIF or linked in driver. In R16B01 some BIFs and NIFs which potentially could execute for a long time (based on the size of their input) have been made interruptible. More specifically this concens the BIF term_to_binary and the crypto NIFs which handle stream input. /Kenneth, Erlang/OTP, Ericsson On Tue, Jul 2, 2013 at 11:17 AM, Max Lapshin wrote: > It is also possible to revert to R15, but in any case if NIF will > block scheduler, you will have problems with managing hunders of > streams. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jul 2 12:37:30 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 2 Jul 2013 12:37:30 +0200 Subject: [erlang-questions] Restarting supervision tree In-Reply-To: <51D1482F.5090800@gmail.com> References: <51D1482F.5090800@gmail.com> Message-ID: On Mon, Jul 1, 2013 at 11:13 AM, Martin Dimitrov wrote: > If I call > supervisor:terminate_child/2 and then supervisor:restart_child/2 on the > permanent worker, will this affect the maximum restart frequency? > No, it will not affect the maximum restart frequency in this case. The way I usually figure this out is by reading the source code of the supervisror module however, and I am not sure that is the best way to have it documented. But at least that is one way to figure it out :P -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrtndimitrov@REDACTED Tue Jul 2 12:49:28 2013 From: mrtndimitrov@REDACTED (Martin Dimitrov) Date: Tue, 02 Jul 2013 13:49:28 +0300 Subject: [erlang-questions] Restarting supervision tree In-Reply-To: References: <51D1482F.5090800@gmail.com> Message-ID: <51D2B038.2030705@gmail.com> Thanks. I figured that supervisor:terminate_child/2 will actually not trigger the recovery mechanism at all. On 7/2/2013 1:37 PM, Jesper Louis Andersen wrote: > On Mon, Jul 1, 2013 at 11:13 AM, Martin Dimitrov wrote: > >> If I call >> supervisor:terminate_child/2 and then supervisor:restart_child/2 on the >> permanent worker, will this affect the maximum restart frequency? >> > > No, it will not affect the maximum restart frequency in this case. The way > I usually figure this out > is by reading the source code of the supervisror module however, and I am > not sure that is the best > way to have it documented. But at least that is one way to figure it out :P > From dmkolesnikov@REDACTED Tue Jul 2 12:54:02 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 2 Jul 2013 13:54:02 +0300 Subject: [erlang-questions] Restarting supervision tree In-Reply-To: <51D2B038.2030705@gmail.com> References: <51D1482F.5090800@gmail.com> <51D2B038.2030705@gmail.com> Message-ID: Hello, You have to call erlang:exit(SupPid, shutdown) or erlang:exit(SupPid, kill) to restart whole supervisor tree. - Dmitry On Jul 2, 2013, at 1:49 PM, Martin Dimitrov wrote: > Thanks. I figured that supervisor:terminate_child/2 will actually not > trigger the recovery mechanism at all. > > On 7/2/2013 1:37 PM, Jesper Louis Andersen wrote: >> On Mon, Jul 1, 2013 at 11:13 AM, Martin Dimitrov wrote: >> >>> If I call >>> supervisor:terminate_child/2 and then supervisor:restart_child/2 on the >>> permanent worker, will this affect the maximum restart frequency? >>> >> >> No, it will not affect the maximum restart frequency in this case. The way >> I usually figure this out >> is by reading the source code of the supervisror module however, and I am >> not sure that is the best >> way to have it documented. But at least that is one way to figure it out :P >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From danbenderr@REDACTED Tue Jul 2 11:40:27 2013 From: danbenderr@REDACTED (Dan Bender) Date: Tue, 2 Jul 2013 12:40:27 +0300 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: Jesper, where can I read about dirty schedulers? Is there an ETA for R17? I saw something about native processes. Is there info about it? On Tue, Jul 2, 2013 at 12:17 PM, Max Lapshin wrote: > It is also possible to revert to R15, but in any case if NIF will > block scheduler, you will have problems with managing hunders of > streams. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danbenderr@REDACTED Tue Jul 2 12:53:27 2013 From: danbenderr@REDACTED (Dan Bender) Date: Tue, 2 Jul 2013 13:53:27 +0300 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: I checked with timer:tc/3 and calling a port driver with encode or decode takes 4ms: timer:tc(?MODULE, command, [Port, encode, Binary]) command(Port, Type, Binary) -> port_control(Port, Type, Binary). Is this the correct way to measure it? How do I choose how much to bump reductions assuming that it takes 4ms to run? Is it reasonable to assume that a more modern and powerful CPU will result with 1/10 time which will be < 1 ms? I'm trying to understand if I can use NIF today and take advantage of dirty schedulers and native processes in a year or two. On Tue, Jul 2, 2013 at 1:16 PM, Kenneth Lundin wrote: > Hi Max, > > I don't know why you are thinking that this will work better in R15 than > in R16? > Can you point to any specific change in R16 that causes problems? > > It has always been a bad idea to execute for a long time (before returning > to the Erlang scheduler) inside a NIF, BIF or linked in driver. > > In R16B01 some BIFs and NIFs which potentially could execute for a long > time (based on the size of their input) have been made interruptible. More > specifically this concens the BIF term_to_binary and > the crypto NIFs which handle stream input. > > /Kenneth, Erlang/OTP, Ericsson > > > On Tue, Jul 2, 2013 at 11:17 AM, Max Lapshin wrote: > >> It is also possible to revert to R15, but in any case if NIF will >> block scheduler, you will have problems with managing hunders of >> streams. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Jul 2 13:53:45 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 2 Jul 2013 14:53:45 +0300 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <1372759801.10432.YahooMailNeo@web164505.mail.gq1.yahoo.com> References: <1372747653.8839.YahooMailNeo@web164502.mail.gq1.yahoo.com> <1372749647.13799.YahooMailNeo@web164501.mail.gq1.yahoo.com> <1372759801.10432.YahooMailNeo@web164505.mail.gq1.yahoo.com> Message-ID: <58279B27-3F1B-494F-9755-05EEA0728D2A@gmail.com> Hello, You can have only one TCP/IP connection between node. Yes, your idea of message collation is fine. There was even message post about this. http://www.ostinelli.net/boost-message-passing-between-erlang-nodes/ I think you should split data and signalling to different channels, at least this is my approach. Use erlang distribution only for cluster coordination but use plain TCP/IP sockets for data path. - Dmitry On Jul 2, 2013, at 1:10 PM, Dror Mein wrote: > You were right. increasing the message size to 10000 allowed me to reach 1-2Gbps. I guess the simple solution is to collate messages in a list and send them together. Is there another way you can think of? Can I configure the erlang vm to open more tcp connections between two nodes? will that solve the problem? > > From: Jesper Louis Andersen > To: Dror Mein > Cc: Gleb Peregud ; Erlang-Questions Questions > Sent: Tuesday, July 2, 2013 12:13 PM > Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed > > > On Tue, Jul 2, 2013 at 9:20 AM, Dror Mein wrote: > From shell: > > Note: The shell is interpreted and you are building a closure for no reason. Try pushing your code into a module and then compiling that module to bytecode and run it again. Chances are you are basically just being bitten by interpretation overhead. Also, you should probably increase the message size to around 100 bytes if that is what you expect to send and receive. There is a messaging overhead which you are paying as well. > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Tue Jul 2 14:05:24 2013 From: jon@REDACTED (Jon Schneider) Date: Tue, 2 Jul 2013 13:05:24 +0100 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed Message-ID: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> I notice message packets have TCP PSH set which makes sense at low message rates. In the interests of efficiency (rather than strict latency) at high rates would it useful if this didn't happen allowing messages to get combined by the TCP stack ? I wonder what would happen if one found the appropriate bit of code and knocked out the PSH. Jon > On Jul 2, 2013, at 1:10 PM, Dror Mein wrote: > >> You were right. increasing the message size to 10000 allowed me to reach >> 1-2Gbps. I guess the simple solution is to collate messages in a list >> and send them together. Is there another way you can think of? Can I >> configure the erlang vm to open more tcp connections between two nodes? >> will that solve the problem? From dmkolesnikov@REDACTED Tue Jul 2 14:49:12 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 2 Jul 2013 15:49:12 +0300 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> References: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> Message-ID: PSH bit is extremely important when we are talking about signal plane. I would see a negative impact on Erlang distribution if it is knocked out. Thus, I was talking to separate concern into different perspectives. Best Regards, Dmitry >-|-|-*> On 2.7.2013, at 15.05, "Jon Schneider" wrote: > I notice message packets have TCP PSH set which makes sense at low message > rates. > > In the interests of efficiency (rather than strict latency) at high rates > would it useful if this didn't happen allowing messages to get combined by > the TCP stack ? > > I wonder what would happen if one found the appropriate bit of code and > knocked out the PSH. > > Jon > >> On Jul 2, 2013, at 1:10 PM, Dror Mein wrote: >> >>> You were right. increasing the message size to 10000 allowed me to reach >>> 1-2Gbps. I guess the simple solution is to collate messages in a list >>> and send them together. Is there another way you can think of? Can I >>> configure the erlang vm to open more tcp connections between two nodes? >>> will that solve the problem? > > From jon@REDACTED Tue Jul 2 15:11:39 2013 From: jon@REDACTED (Jon Schneider) Date: Tue, 2 Jul 2013 14:11:39 +0100 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: References: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> Message-ID: I was of course mixed up and meant TCP_NODELAY if indeed that is the cause of bitty message traffic. TCP (v4) overhead on ethernet starts at 78 bytes. Jon > PSH bit is extremely important when we are talking about signal plane. I > would see a negative impact on Erlang distribution if it is knocked out. > Thus, I was talking to separate concern into different perspectives. > > Best Regards, > Dmitry >-|-|-*> From stefan@REDACTED Tue Jul 2 15:13:47 2013 From: stefan@REDACTED (Stefan Jahn) Date: Tue, 2 Jul 2013 15:13:47 +0200 Subject: [erlang-questions] predefined Erlang version macros Message-ID: <35de19ade1e460a26ee6d187495983c2.squirrel@service.rules.org> Dear Erlang'ers, sometimes it would be nice to enable version-dependend code such as needed here: %% R16B: use new ssh behaviour %-behaviour(ssh_daemon_channel). %% R15: use old ssh behaviour -behaviour(ssh_channel). Is there anything suitable in Erlang? Or is it unwanted because of strict versioning requirements of modules and applications? Thank you in advance, Stefan. From mononcqc@REDACTED Tue Jul 2 15:53:50 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 2 Jul 2013 09:53:50 -0400 Subject: [erlang-questions] predefined Erlang version macros In-Reply-To: <35de19ade1e460a26ee6d187495983c2.squirrel@service.rules.org> References: <35de19ade1e460a26ee6d187495983c2.squirrel@service.rules.org> Message-ID: <20130702135349.GD4510@ferdair.local> A lot of people ask for this, but the problem is that 'R15' or 'R16B' and so on are release names. This means that if I package my own Erlang release with a subset (or superset) of OTP applications shipped with R16B01, I can give it a new name and the macros will start breaking. This includes the potential (but yet to be seen) use case of making a release that uses libraries from two (or more) distinct OTP releases at once. Instead, what you should do is find a way to do feature-detection: load the application at compile-time, find the version it has, and do some optional branching based on that. It's much more complex, but it uses an application-based mechanism instead of trying to rely on the release name, which is not fixed. Here's an example that could be done with a parse-transform. For the user of the code: -module(some). -export([main/0]). -compile({parse_transform, ssh_behaviour}). %% R16B: use new ssh behaviour -behavior(ssh_daemon_channel). %% R15: use old ssh behaviour -behaviour(ssh_channel). main() -> ok. And the parse transform that verifies whether 'ssh_daemon_channel' exists or not to pick what to include: -module(ssh_behaviour). -export([parse_transform/2]). parse_transform(Forms, _Options) -> lists:filter(fun pred/1, Forms). pred({attribute, _Ln, behaviour, ssh_daemon_channel}) -> %% always include when available is_available(ssh_daemon_channel); pred({attribute, _Ln, behaviour, ssh_channel}) -> %% only include when ssh_daemon_channel is not available not is_available(ssh_daemon_channel); %% Same predicates but for americans pred({attribute, _Ln, behavior, ssh_daemon_channel}) -> is_available(ssh_daemon_channel); pred({attribute, _Ln, behavior, ssh_channel}) -> not is_available(ssh_daemon_channel); %% Skip the rest pred(_Form) -> true. is_available(Mod) -> try %% if the module loads, it means it exists _ = Mod:module_info(), true catch error:undef -> % doesn't exist false end. Now the code can be shorter if you only want to support 'behavior' or 'behaviour', but this macro will make sure only one of the two modules will be used for the behaviour, at compile time, based on the assumptions that you will compile and run code with the same version, and that you do not compile with the VM in embedded mode (which forbids loading modules dynamically). This one should work for all releases without a problem, given it works by doing feature detection rather than trying to find out about a name. I just wish it were easier to do something like this than needing to write custom macros. Regards, Fred. On 07/02, Stefan Jahn wrote: > Dear Erlang'ers, > > sometimes it would be nice to enable version-dependend code such as > needed here: > > %% R16B: use new ssh behaviour > %-behaviour(ssh_daemon_channel). > > %% R15: use old ssh behaviour > -behaviour(ssh_channel). > > > Is there anything suitable in Erlang? Or is it unwanted because of > strict versioning requirements of modules and applications? > > Thank you in advance, > Stefan. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From norton@REDACTED Tue Jul 2 16:04:50 2013 From: norton@REDACTED (Joseph Wayne Norton) Date: Tue, 2 Jul 2013 09:04:50 -0500 Subject: [erlang-questions] predefined Erlang version macros In-Reply-To: <35de19ade1e460a26ee6d187495983c2.squirrel@service.rules.org> References: <35de19ade1e460a26ee6d187495983c2.squirrel@service.rules.org> Message-ID: <64E7601F-CA92-47C5-82D5-3AE2B8801546@lovely.email.ne.jp> Stefan - I typically use macros with rebar's platform_define feature for such situations. Here is an example for R14 and old inets behavior (https://github.com/ubf/ubf-jsonrpc/blob/master/rebar.config#L6). This method is simple but works only at compile-time. Regards, Joe N. On 2013/07/02, at 8:13, "Stefan Jahn" wrote: > Dear Erlang'ers, > > sometimes it would be nice to enable version-dependend code such as > needed here: > > %% R16B: use new ssh behaviour > %-behaviour(ssh_daemon_channel). > > %% R15: use old ssh behaviour > -behaviour(ssh_channel). > > > Is there anything suitable in Erlang? Or is it unwanted because of > strict versioning requirements of modules and applications? > > Thank you in advance, > Stefan. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jul 2 16:27:43 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 2 Jul 2013 16:27:43 +0200 Subject: [erlang-questions] Real time audio transcoding with a port driver In-Reply-To: References: Message-ID: On Tue, Jul 2, 2013 at 11:40 AM, Dan Bender wrote: > Jesper, where can I read about dirty schedulers? > Is there an ETA for R17? > I saw something about native processes. Is there info about it? > Dirty schedulers are currently scheduled for R17 which is in turn planned for a release around February 2014 I think. DS's and Native processes are not there yet, so documentation on them are currently not existing and you cannot in general rely on them being there in the next release. You can use a NIF today, but you will have to solve the problem by either breaking up your 3-4ms frame encoding in 4+ blocks and then call the NIF multiple times. You should also be calling enif_consume_timeslice() frequently while doing so. The other solution is to have a background thread which communicates back with a message when it is done. The NIF can then invoke this background thread with work. I do not have an example however, of this being done. -------------- next part -------------- An HTML attachment was scrubbed... URL: From drormein@REDACTED Tue Jul 2 16:58:47 2013 From: drormein@REDACTED (Dror Mein) Date: Tue, 2 Jul 2013 07:58:47 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: References: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> Message-ID: <1372777127.7434.YahooMailNeo@web164505.mail.gq1.yahoo.com> I did inet:i(), and prim_inet:get_opt(S, nodelay), and found out that the tcp nodelay flag is actually?true; so , no help there. ________________________________ From: Jon Schneider To: Dmitry Kolesnikov Cc: Erlang Questions Sent: Tuesday, July 2, 2013 4:11 PM Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed I was of course mixed up and meant TCP_NODELAY if indeed that is the cause of bitty message traffic. TCP (v4) overhead on ethernet starts at 78 bytes. Jon > PSH bit is extremely important when we are talking about signal plane. I > would see a negative impact on Erlang distribution if it is knocked out. > Thus, I was talking to separate concern into different perspectives. > > Best Regards, > Dmitry >-|-|-*> _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Tue Jul 2 17:06:07 2013 From: jon@REDACTED (Jon Schneider) Date: Tue, 2 Jul 2013 16:06:07 +0100 Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <1372777127.7434.YahooMailNeo@web164505.mail.gq1.yahoo.com> References: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> <1372777127.7434.YahooMailNeo@web164505.mail.gq1.yahoo.com> Message-ID: <44230afee179c0082513d7420931d497.squirrel@mail.jschneider.net> > I did inet:i(), and prim_inet:get_opt(S, nodelay), and found out that the > tcp nodelay flag is actually?true; so , no help there. The idea was to disable ndelay. If you can try clearing this bit. Jon > ________________________________ > From: Jon Schneider > To: Dmitry Kolesnikov > Cc: Erlang Questions > Sent: Tuesday, July 2, 2013 4:11 PM > Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes > connection speed > > > I was of course mixed up and meant TCP_NODELAY if indeed that is the cause > of bitty message traffic. > > TCP (v4) overhead on ethernet starts at 78 bytes. > > Jon > >> PSH bit is extremely important when we are talking about signal plane. I >> would see a negative impact on Erlang distribution if it is knocked out. >> Thus, I was talking to separate concern into different perspectives. >> >> Best Regards, >> Dmitry >-|-|-*> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From drormein@REDACTED Tue Jul 2 17:55:28 2013 From: drormein@REDACTED (Dror Mein) Date: Tue, 2 Jul 2013 08:55:28 -0700 (PDT) Subject: [erlang-questions] Is there a lomit on erlang vm nodes connection speed In-Reply-To: <44230afee179c0082513d7420931d497.squirrel@mail.jschneider.net> References: <216623c8618cc7bb81682da6f2d40e1d.squirrel@mail.jschneider.net> <1372777127.7434.YahooMailNeo@web164505.mail.gq1.yahoo.com> <44230afee179c0082513d7420931d497.squirrel@mail.jschneider.net> Message-ID: <1372780528.70813.YahooMailNeo@web164505.mail.gq1.yahoo.com> Ah ok, I disabled nodelay, and it helped me only slightly. thanks anyway. ________________________________ From: Jon Schneider To: Dror Mein Cc: Dmitry Kolesnikov ; Erlang Questions Sent: Tuesday, July 2, 2013 6:06 PM Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes connection speed > I did inet:i(), and prim_inet:get_opt(S, nodelay), and found out that the > tcp nodelay flag is actually?true; so , no help there. The idea was to disable ndelay. If you can try clearing this bit. Jon > ________________________________ >? From: Jon Schneider > To: Dmitry Kolesnikov > Cc: Erlang Questions > Sent: Tuesday, July 2, 2013 4:11 PM > Subject: Re: [erlang-questions] Is there a lomit on erlang vm nodes > connection speed > > > I was of course mixed up and meant TCP_NODELAY if indeed that is the cause > of bitty message traffic. > > TCP (v4) overhead on ethernet starts at 78 bytes. > > Jon > >> PSH bit is extremely important when we are talking about signal plane. I >> would see a negative impact on Erlang distribution if it is knocked out. >> Thus, I was talking to separate concern into different perspectives. >> >> Best Regards, >> Dmitry >-|-|-*> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ransomr@REDACTED Tue Jul 2 18:57:39 2013 From: ransomr@REDACTED (Ransom Richardson) Date: Tue, 2 Jul 2013 16:57:39 +0000 Subject: [erlang-questions] iPhone unable to connect over SSL after upgrading to R16B01 In-Reply-To: References: <5f2a8a61-7cad-4896-a2b5-be02df1ad1ad@googlegroups.com> , Message-ID: <9da2873ba4274c0e86bca0a24c36e645@BL2PR08MB002.namprd08.prod.outlook.com> Ingela, thanks for looking at this. Do you still need additional information? The issue seems to be client specific, and I don't have an isolated repro. Based on what I'm seeing it looks like {ecdhe_rsa,aes_256_cbc,sha} works fine when I connect with a python client, but not when I connect with an iOS client. This is the case even if I also support {rsa,aes_256_cbc,sha256}, which does work with the iOS client. So the iOS client/Erlang R16B01 server are choosing an EC cipher that then doesn't work. I think you will see the issue if you implement just a basic ssl server and connect with an iOS client. Please let me know if I can provide more information. Ransom ________________________________ From: erlang-questions-bounces@REDACTED on behalf of Andrew Shu Sent: Monday, June 24, 2013 10:49 AM To: Ingela Andin Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] iPhone unable to connect over SSL after upgrading to R16B01 Hi Ingela thanks for troubleshooting this. I ran the openssl server and pointed Chromium at https://localhost:4433, using the same certs from the Cowboy example. It works, so I don't think the web browser client is the issue. Here's the long HTML printout by openssl server, when I hit https://localhost:4433 using Chromium: s_server -accept 4433 -CAfile cowboy-ca.crt -cert server.crt -key server.key -www Ciphers supported in s_server binary TLSv1/SSLv3:ECDHE-RSA-AES256-GCM-SHA384TLSv1/SSLv3:ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1/SSLv3:ECDHE-RSA-AES256-SHA384 TLSv1/SSLv3:ECDHE-ECDSA-AES256-SHA384 TLSv1/SSLv3:ECDHE-RSA-AES256-SHA TLSv1/SSLv3:ECDHE-ECDSA-AES256-SHA TLSv1/SSLv3:SRP-DSS-AES-256-CBC-SHA TLSv1/SSLv3:SRP-RSA-AES-256-CBC-SHA TLSv1/SSLv3:DHE-DSS-AES256-GCM-SHA384TLSv1/SSLv3:DHE-RSA-AES256-GCM-SHA384 TLSv1/SSLv3:DHE-RSA-AES256-SHA256 TLSv1/SSLv3:DHE-DSS-AES256-SHA256 TLSv1/SSLv3:DHE-RSA-AES256-SHA TLSv1/SSLv3:DHE-DSS-AES256-SHA TLSv1/SSLv3:DHE-RSA-CAMELLIA256-SHA TLSv1/SSLv3:DHE-DSS-CAMELLIA256-SHA TLSv1/SSLv3:ECDH-RSA-AES256-GCM-SHA384TLSv1/SSLv3:ECDH-ECDSA-AES256-GCM-SHA384 TLSv1/SSLv3:ECDH-RSA-AES256-SHA384 TLSv1/SSLv3:ECDH-ECDSA-AES256-SHA384 TLSv1/SSLv3:ECDH-RSA-AES256-SHA TLSv1/SSLv3:ECDH-ECDSA-AES256-SHA TLSv1/SSLv3:AES256-GCM-SHA384 TLSv1/SSLv3:AES256-SHA256 TLSv1/SSLv3:AES256-SHA TLSv1/SSLv3:CAMELLIA256-SHA TLSv1/SSLv3:PSK-AES256-CBC-SHA TLSv1/SSLv3:ECDHE-RSA-DES-CBC3-SHA TLSv1/SSLv3:ECDHE-ECDSA-DES-CBC3-SHA TLSv1/SSLv3:SRP-DSS-3DES-EDE-CBC-SHA TLSv1/SSLv3:SRP-RSA-3DES-EDE-CBC-SHA TLSv1/SSLv3:EDH-RSA-DES-CBC3-SHA TLSv1/SSLv3:EDH-DSS-DES-CBC3-SHA TLSv1/SSLv3:ECDH-RSA-DES-CBC3-SHA TLSv1/SSLv3:ECDH-ECDSA-DES-CBC3-SHA TLSv1/SSLv3:DES-CBC3-SHA TLSv1/SSLv3:PSK-3DES-EDE-CBC-SHA TLSv1/SSLv3:ECDHE-RSA-AES128-GCM-SHA256 TLSv1/SSLv3:ECDHE-ECDSA-AES128-GCM-SHA256TLSv1/SSLv3:ECDHE-RSA-AES128-SHA256 TLSv1/SSLv3:ECDHE-ECDSA-AES128-SHA256TLSv1/SSLv3:ECDHE-RSA-AES128-SHA TLSv1/SSLv3:ECDHE-ECDSA-AES128-SHA TLSv1/SSLv3:SRP-DSS-AES-128-CBC-SHA TLSv1/SSLv3:SRP-RSA-AES-128-CBC-SHA TLSv1/SSLv3:DHE-DSS-AES128-GCM-SHA256 TLSv1/SSLv3:DHE-RSA-AES128-GCM-SHA256TLSv1/SSLv3:DHE-RSA-AES128-SHA256 TLSv1/SSLv3:DHE-DSS-AES128-SHA256 TLSv1/SSLv3:DHE-RSA-AES128-SHA TLSv1/SSLv3:DHE-DSS-AES128-SHA TLSv1/SSLv3:DHE-RSA-SEED-SHA TLSv1/SSLv3:DHE-DSS-SEED-SHA TLSv1/SSLv3:DHE-RSA-CAMELLIA128-SHA TLSv1/SSLv3:DHE-DSS-CAMELLIA128-SHA TLSv1/SSLv3:ECDH-RSA-AES128-GCM-SHA256 TLSv1/SSLv3:ECDH-ECDSA-AES128-GCM-SHA256TLSv1/SSLv3:ECDH-RSA-AES128-SHA256 TLSv1/SSLv3:ECDH-ECDSA-AES128-SHA256 TLSv1/SSLv3:ECDH-RSA-AES128-SHA TLSv1/SSLv3:ECDH-ECDSA-AES128-SHA TLSv1/SSLv3:AES128-GCM-SHA256 TLSv1/SSLv3:AES128-SHA256 TLSv1/SSLv3:AES128-SHA TLSv1/SSLv3:SEED-SHA TLSv1/SSLv3:CAMELLIA128-SHA TLSv1/SSLv3:PSK-AES128-CBC-SHA TLSv1/SSLv3:ECDHE-RSA-RC4-SHA TLSv1/SSLv3:ECDHE-ECDSA-RC4-SHA TLSv1/SSLv3:ECDH-RSA-RC4-SHA TLSv1/SSLv3:ECDH-ECDSA-RC4-SHA TLSv1/SSLv3:RC4-SHA TLSv1/SSLv3:RC4-MD5 TLSv1/SSLv3:PSK-RC4-SHA TLSv1/SSLv3:EDH-RSA-DES-CBC-SHA TLSv1/SSLv3:EDH-DSS-DES-CBC-SHA TLSv1/SSLv3:DES-CBC-SHA TLSv1/SSLv3:EXP-EDH-RSA-DES-CBC-SHA TLSv1/SSLv3:EXP-EDH-DSS-DES-CBC-SHA TLSv1/SSLv3:EXP-DES-CBC-SHA TLSv1/SSLv3:EXP-RC2-CBC-MD5 TLSv1/SSLv3:EXP-RC4-MD5 --- Ciphers common between both SSL end points: ECDHE-ECDSA-AES256-SHA ECDHE-RSA-AES256-SHA DHE-RSA-CAMELLIA256-SHA DHE-DSS-CAMELLIA256-SHA DHE-RSA-AES256-SHA DHE-DSS-AES256-SHA ECDH-RSA-AES256-SHA ECDH-ECDSA-AES256-SHA CAMELLIA256-SHA AES256-SHA ECDHE-ECDSA-RC4-SHA ECDHE-ECDSA-AES128-SHA ECDHE-RSA-RC4-SHA ECDHE-RSA-AES128-SHA DHE-RSA-CAMELLIA128-SHA DHE-DSS-CAMELLIA128-SHA DHE-RSA-AES128-SHA DHE-DSS-AES128-SHA ECDH-RSA-RC4-SHA ECDH-RSA-AES128-SHA ECDH-ECDSA-RC4-SHA ECDH-ECDSA-AES128-SHA SEED-SHA CAMELLIA128-SHA RC4-SHA RC4-MD5 AES128-SHA ECDHE-ECDSA-DES-CBC3-SHA ECDHE-RSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA EDH-DSS-DES-CBC3-SHA ECDH-RSA-DES-CBC3-SHA ECDH-ECDSA-DES-CBC3-SHA DES-CBC3-SHA --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA SSL-Session: Protocol : TLSv1.1 Cipher : ECDHE-RSA-AES256-SHA Session-ID: Session-ID-ctx: 01000000 Master-Key: 9D707F79ED6FCF06935AAEC6C52E1E42642B2900EC790BC3BBC602F6DD19619220C3C800E173D75313D83DA6053E6786 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1372085057 Timeout : 300 (sec) Verify return code: 0 (ok) --- 0 items in the session cache 0 client connects (SSL_connect()) 0 client renegotiates (SSL_connect()) 0 client connects that finished 1 server accepts (SSL_accept()) 0 server renegotiates (SSL_accept()) 1 server accepts that finished 0 session cache hits 1 session cache misses 0 session cache timeouts 0 callback cache hits 0 cache full overflows (128 allowed) --- no client certificate available On Mon, Jun 24, 2013 at 8:35 AM, Ingela Andin > wrote: Hi again, 2013/6/23 Andrew Shu > THANK YOU for posting this! This being my first time using Cowboy HTTPS and secure websockets, I was going crazy trying to figure out why SSL wasn't working via Chromium on Linux, while curl seemed to handle the self-signed certificates okay. It wouldn't have occurred to me that it could be an Erlang bug. After reverting to R16B, and removing all traces of R16B01, everything seems working. I wasted a lot of time swapping out SSL certificates to no avail. I think sticking with R16B is the best, or only, solution for now. I had been getting a Chromium gray error screen with "ERR_SSL_CLIENT_AUTH_SIGNATURE_FAILED". Not the usual "this certificate is not trusted" red screen. Firefox choked too. Curl seemed ok, strangely enough. Ok just make sure I run an openssl client against the erlang server too, with the cowboy example (as you try to connect to an erlang-server with other clients). This works too. So it seems openssl and curl can connect to the erlang-server but not chrom and firefox? And the connection fails due to that the client sends and alert. So atleas this problem seems not to be related to ECDSA certificates. (The missed TODO) . The other clients could also have issues with ecc-cipher suites, you could try to setting up an openssl server using openssl s_server -accept 4433 -CAfile ca.crt -cert server.crt -key server.key and trying the clients to see if they can connect with ecc-ciphers. The following command must return elliptic curve ciper cuites ECDH* ECDSA* > openssl ciphers Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From stas@REDACTED Tue Jul 2 19:23:57 2013 From: stas@REDACTED (Stanislav Sedov) Date: Tue, 2 Jul 2013 10:23:57 -0700 Subject: [erlang-questions] iPhone unable to connect over SSL after upgrading to R16B01 In-Reply-To: <9da2873ba4274c0e86bca0a24c36e645@BL2PR08MB002.namprd08.prod.outlook.com> References: <5f2a8a61-7cad-4896-a2b5-be02df1ad1ad@googlegroups.com> <9da2873ba4274c0e86bca0a24c36e645@BL2PR08MB002.namprd08.prod.outlook.com> Message-ID: On Jul 2, 2013, at 9:57 AM, Ransom Richardson wrote: > Ingela, thanks for looking at this. Do you still need additional information? The issue seems to be client specific, and I don't have an isolated repro. Based on what I'm seeing it looks like {ecdhe_rsa,aes_256_cbc,sha} works fine when I connect with a python client, but not when I connect with an iOS client. This is the case even if I also support {rsa,aes_256_cbc,sha256}, which does work with the iOS client. So the iOS client/Erlang R16B01 server are choosing an EC cipher that then doesn't work. I think you will see the issue if you implement just a basic ssl server and connect with an iOS client. Please let me know if I can provide more information. I wonder if this somehow may be related to an apparently broken ECDSA implementation in Safari. It can be seen that in the list of ciphers you posted DH/ECDHA comes first. Did the ciphers priority change for some reason in R16B01, or is it possible your R16B01 is linked agains a different version of OpenSSL? My apologies if you already considered this, but hope this might help. [1] http://openssl.6102.n7.nabble.com/Apple-are-apparently-dicks-td45512.html -- ST4096-RIPE -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Tue Jul 2 21:46:44 2013 From: erlang@REDACTED (Tim) Date: Wed, 3 Jul 2013 05:16:44 +0930 Subject: [erlang-questions] untyped records Message-ID: Greetings, I'm getting an untyped record from a web form submission. io_lib:format("~p", [MyRecord]) displays: {struct,[{code,1372793577767}, {desc,"e7iOSxy6PYQiMRk87r5cHK4DVAUj5SzI"}, {units,"g|Kg"}]} Now, I know if I define a record myself, I can access fields using the format: Name#type.field but in this case the type of record is unknown. MyRecord#struct.desc returns a 'record struct undefined' error at compile time. Can I access the fields of this record without deconstructing it? Cheers. From k.petrauskas@REDACTED Tue Jul 2 22:01:31 2013 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 2 Jul 2013 23:01:31 +0300 Subject: [erlang-questions] untyped records In-Reply-To: References: Message-ID: Hi, Its a proplist wrapped to a {struct, _}. You can access its fields as follows: {struct, X} = {struct,[ {code,1372793577767}, {desc,"e7iOSxy6PYQiMRk87r5cHK4DVAUj5SzI"}, {units,"g|Kg"} ]}. proplists:get_value(desc, X). Karolis On Tue, Jul 2, 2013 at 10:46 PM, Tim wrote: > Greetings, > > I'm getting an untyped record from a web form submission. io_lib:format("~p", [MyRecord]) displays: > > {struct,[{code,1372793577767}, > {desc,"e7iOSxy6PYQiMRk87r5cHK4DVAUj5SzI"}, > {units,"g|Kg"}]} > > Now, I know if I define a record myself, I can access fields using the format: Name#type.field > > but in this case the type of record is unknown. > > MyRecord#struct.desc returns a 'record struct undefined' error at compile time. > > Can I access the fields of this record without deconstructing it? > > Cheers. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From thomas@REDACTED Tue Jul 2 21:59:59 2013 From: thomas@REDACTED (Thomas Allen) Date: Tue, 02 Jul 2013 12:59:59 -0700 Subject: [erlang-questions] untyped records In-Reply-To: References: Message-ID: <1372795199.26608.140661251181573.26C40BA8@webmail.messagingengine.com> Is this a record or a tagged tuple? It looks like the latter. Perhaps this will work for you: {struct, Props} = R, V = proplists:get_value(k, Props), %% ... Thomas On Tue, Jul 2, 2013, at 12:46 PM, Tim wrote: > Greetings, > > I'm getting an untyped record from a web form submission. > io_lib:format("~p", [MyRecord]) displays: > > {struct,[{code,1372793577767}, > {desc,"e7iOSxy6PYQiMRk87r5cHK4DVAUj5SzI"}, > {units,"g|Kg"}]} > > Now, I know if I define a record myself, I can access fields using the > format: Name#type.field > > but in this case the type of record is unknown. > > MyRecord#struct.desc returns a 'record struct undefined' error at compile > time. > > Can I access the fields of this record without deconstructing it? > > Cheers. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From erlang@REDACTED Tue Jul 2 22:41:13 2013 From: erlang@REDACTED (Tim) Date: Wed, 3 Jul 2013 06:11:13 +0930 Subject: [erlang-questions] untyped records In-Reply-To: References: Message-ID: On 03/07/2013, at 5:31 AM, Karolis Petrauskas wrote: > Its a proplist wrapped to a {struct, _}. You can access its > fields as follows: > {struct, X} = {struct,[ > {code,1372793577767}, > {desc,"e7iOSxy6PYQiMRk87r5cHK4DVAUj5SzI"}, > {units,"g|Kg"} > ]}. > proplists:get_value(desc, X). Thanks! From olav@REDACTED Wed Jul 3 11:11:21 2013 From: olav@REDACTED (Olav Frengstad) Date: Wed, 3 Jul 2013 11:11:21 +0200 Subject: [erlang-questions] Shell output for Common Test Message-ID: Hi, I'm running common test cases in a shell session, but i'm struggling when i want more verbose output. Currently I have to navigate through the generated HTML to get stack traces and stdout. Is there a way to get all output printed in the shell? I'm running ct by calling `ct:run_test(Opts)`. -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Wed Jul 3 11:28:22 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Wed, 3 Jul 2013 10:28:22 +0100 Subject: [erlang-questions] The odd behaviour of net_ticktime Message-ID: I have observed - anecdotally via the m/l and google and through looking at the code in the kernel application - that net_kernel's tick mechanism appears to do nothing to prioritise tick messages over the dist port, so that when a connection between two nodes is very busy, it's possible for tick messages to lag behind other traffic leading to unnecessary disconnects. Firstly, have I understood this properly? I couldn't find anything in the vm code that indicates special handling of these messages, nor have there been any statements from the OTP team on the m/l that I can find, indicating otherwise. Secondly, if I'm right, then why is net_kernel doing things this way? If there is traffic on a dist port (i.e., socket), then clearly we've not actually experienced a netsplit, and reporting one seems like the wrong thing to do. What I'm contemplating at the moment, is setting the net_ticktime to a very high value and implementing user-level heartbeats, combined with out of band checks (using getstat) on the dist ports in use, to verify that traffic is (or isn't) actually flowing before firing a 'DOWN' event. I really want to put all the effort in if this isn't necessary though. Another question too, if I may: assuming I do decide to implement a user level heartbeat/keep-alive mechanism, I'm undecided about /how/ to indicate a detected netsplit to the processes in my application(s). Because most of these work on monitors, 'DOWN' messages and/or net_kernel:monitor_nodes already, I'm inclined to call net_kernel:disconnect_node/1 in response to seeing a netsplit in my user-level monitoring code - I would expect that to trigger all the relevant local monitors in a timely fashion. Does that sound like a reasonable way to go? Cheers, Tim -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP using GPGMail URL: From 249505968@REDACTED Wed Jul 3 12:03:07 2013 From: 249505968@REDACTED (=?gb18030?B?99L30Q==?=) Date: Wed, 3 Jul 2013 18:03:07 +0800 Subject: [erlang-questions] What's this mean:[1|1]?It's it a bug? Message-ID: One of my new teammate found that this expression can work like list:nth(1,[1|1]) is legal.And the is_list([1|1]) is true so what's this expression mean or it's just like some bug? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Wed Jul 3 12:09:18 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Wed, 03 Jul 2013 11:09:18 +0100 Subject: [erlang-questions] What's this mean:[1|1]?It's it a bug? In-Reply-To: References: Message-ID: <51D3F84E.6030201@llaisdy.com> [1|1] is an "improper list", in other words, a list that doesn't end with an empty list []. They're valid, but not terribly safe: 1> is_list([1|1]). true 2> length([1|1]). ** exception error: bad argument in function length/1 called as length([1|1]) I don't know if there's any use for them. Ivan On 03/07/2013 11:03, ?? wrote: > One of my new teammate found that this expression can work like > list:nth(1,[1|1]) is legal. > And the is_list([1|1]) is true > so what's this expression mean or it's just like some bug? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From f355@REDACTED Wed Jul 3 12:16:09 2013 From: f355@REDACTED (Konstantin Tcepliaev) Date: Wed, 03 Jul 2013 14:16:09 +0400 Subject: [erlang-questions] What's this mean:[1|1]?It's it a bug? In-Reply-To: References: Message-ID: <94901372846569@web14g.yandex.ru> An HTML attachment was scrubbed... URL: From 249505968@REDACTED Wed Jul 3 12:21:08 2013 From: 249505968@REDACTED (=?gb18030?B?99L30Q==?=) Date: Wed, 3 Jul 2013 18:21:08 +0800 Subject: [erlang-questions] =?gb18030?b?u9i4tKO6ICBXaGF0J3MgdGhpcyBtZWFu?= =?gb18030?b?OlsxfDFdP0l0J3MgaXQgYSBidWc/?= Message-ID: oh thank you! I will just try to keep it out of my door ------------------ ???? ------------------ ???: "Konstantin Tcepliaev"; ????: 2013?7?3?(???) ??6:16 ???: "??"<249505968@REDACTED>; "erlang-questions"; ??: Re: [erlang-questions] What's this mean:[1|1]?It's it a bug? Hi, That's what is called "improper list". Lists in Erlang are basically two pointers, to head and to tail, and both of them actually can be of any type. List of, say, two elements - [1, 2] - is actually just a syntactic sugar for [1 | [2 | []]], where [] is a special value meaning "empty list". Obviously, anything can be used in its place. Improper lists can lead to strange errors and definitely should be avoided. They probably can be used in place of 2-tuples when you need some crazy micro-optimization, because I think they're a bit faster. But still, do not try that at home. Cheers, -- Konstantin 03.07.2013, 14:03, "??" <249505968@REDACTED>: One of my new teammate found that this expression can work like list:nth(1,[1|1]) is legal.And the is_list([1|1]) is true so what's this expression mean or it's just like some bug? , _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From arjan@REDACTED Wed Jul 3 13:16:27 2013 From: arjan@REDACTED (Arjan Scherpenisse) Date: Wed, 03 Jul 2013 13:16:27 +0200 Subject: [erlang-questions] Compiler warnings on impossible constructs? Message-ID: <51D4080B.4000603@scherpenisse.net> Hi list, I today fixed a stupid bug in something I wrote last week, namely, an atom-to-atom comparison which should have been a variable-to-atom comparioon: It read: foo =:= bar while it should have read Foo =:= bar. So, my question is, can't Erlang detect this kind of situations on compile time? Obviously, the expression foo =:= bar will never be true, right? Or maybe I just trust flymake to much instead of doublechecking the stuff I write.. ;) Arjan -- Arjan Scherpenisse ----- New media art, research, development arjan@REDACTED +31641322599 - http://scherpenisse.net From kostis@REDACTED Wed Jul 3 13:25:07 2013 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 03 Jul 2013 13:25:07 +0200 Subject: [erlang-questions] Compiler warnings on impossible constructs? In-Reply-To: <51D4080B.4000603@scherpenisse.net> References: <51D4080B.4000603@scherpenisse.net> Message-ID: <51D40A13.7090508@cs.ntua.gr> On 07/03/2013 01:16 PM, Arjan Scherpenisse wrote: > Hi list, > > I today fixed a stupid bug in something I wrote last week, namely, an > atom-to-atom comparison which should have been a variable-to-atom > comparioon: > > It read: foo =:= bar while it should have read Foo =:= bar. > > So, my question is, can't Erlang detect this kind of situations on > compile time? Obviously, the expression foo =:= bar will never be true, > right? Have you run dialyzer on your code? Kostis From dmkolesnikov@REDACTED Wed Jul 3 13:25:51 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 3 Jul 2013 14:25:51 +0300 Subject: [erlang-questions] Compiler warnings on impossible constructs? In-Reply-To: <51D4080B.4000603@scherpenisse.net> References: <51D4080B.4000603@scherpenisse.net> Message-ID: Hello, You would get warning if pattern matching is used. - Dmitry On Jul 3, 2013, at 2:16 PM, Arjan Scherpenisse wrote: > Hi list, > > I today fixed a stupid bug in something I wrote last week, namely, an atom-to-atom comparison which should have been a variable-to-atom comparioon: > > It read: foo =:= bar while it should have read Foo =:= bar. > > So, my question is, can't Erlang detect this kind of situations on compile time? Obviously, the expression foo =:= bar will never be true, right? > > Or maybe I just trust flymake to much instead of doublechecking the stuff I write.. ;) > > Arjan > > > -- > Arjan Scherpenisse ----- New media art, research, development > arjan@REDACTED +31641322599 - http://scherpenisse.net > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From arjan@REDACTED Wed Jul 3 13:39:25 2013 From: arjan@REDACTED (Arjan Scherpenisse) Date: Wed, 03 Jul 2013 13:39:25 +0200 Subject: [erlang-questions] Compiler warnings on impossible constructs? In-Reply-To: <51D40A13.7090508@cs.ntua.gr> References: <51D4080B.4000603@scherpenisse.net> <51D40A13.7090508@cs.ntua.gr> Message-ID: <51D40D6D.4020406@scherpenisse.net> On 03-07-13 13:25, Kostis Sagonas wrote: > On 07/03/2013 01:16 PM, Arjan Scherpenisse wrote: >> Hi list, >> >> I today fixed a stupid bug in something I wrote last week, namely, an >> atom-to-atom comparison which should have been a variable-to-atom >> comparioon: >> >> It read: foo =:= bar while it should have read Foo =:= bar. >> >> So, my question is, can't Erlang detect this kind of situations on >> compile time? Obviously, the expression foo =:= bar will never be true, >> right? > > Have you run dialyzer on your code? Just after I hit "send" i thought the same... I guess I should find a way to hook dialyzer into flymake.. :) Arjan From lukas@REDACTED Wed Jul 3 14:34:47 2013 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 3 Jul 2013 14:34:47 +0200 Subject: [erlang-questions] What's this mean:[1|1]?It's it a bug? In-Reply-To: <51D3F84E.6030201@llaisdy.com> References: <51D3F84E.6030201@llaisdy.com> Message-ID: On Wed, Jul 3, 2013 at 12:09 PM, Ivan Uemlianin wrote: > [1|1] is an "improper list", in other words, a list that doesn't end with > an empty list []. They're valid, but not terribly safe: > > 1> is_list([1|1]). > true > 2> length([1|1]). > ** exception error: bad argument > in function length/1 > called as length([1|1]) > > I don't know if there's any use for them. > It can be used to save two words of memory as [1|1] is two words but [1,1] is four words. So if you are developing a highly optimized data structure (like dict for instance) it is useful. Most people will however not need this. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Wed Jul 3 15:20:19 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Wed, 3 Jul 2013 14:20:19 +0100 Subject: [erlang-questions] The odd behaviour of net_ticktime In-Reply-To: References: Message-ID: <7FD7ECDA-A202-4987-BEDC-DD1DE3E11D17@gmail.com> Just to clarify... On 3 Jul 2013, at 10:28, Tim Watson wrote: > Secondly, if I'm right, then why is net_kernel doing things this way? If there is traffic on a dist port (i.e., socket), then clearly we've not actually experienced a netsplit, and reporting one seems like the wrong thing to do. > I did notice that the timeout checks the socket for activity - this is (presumably) handling the "TickTime * 4" i.e., 60 seconds inactivity timeout that's mentioned in the documentation? > What I'm contemplating at the moment, is setting the net_ticktime to a very high value and implementing user-level heartbeats, combined with out of band checks (using getstat) on the dist ports in use, to verify that traffic is (or isn't) actually flowing before firing a 'DOWN' event. I really want to put all the effort in if this isn't necessary though. > I'm still contemplating doing this, since such a long potential delay (viz 30 - 60 seconds) isn't tolerable. And besides, pushing net_kernel's tick timer out of the way isn't really plausible if the following comments from dist_util.erl hold true: %% And then every 60 seconds we also check the connection and %% close it if we havn't received anything on it for the %% last 60 secs. If ticked == tick we havn't received anything %% on the connection the last 60 secs. That's frankly just confusing. It *sounds* like we cannot realistically extend net_ticktime to anything larger than 60 seconds, but in practise that "60 seconds" actually just means "TickTime seconds" right? I'm trying to work out the utility of having a very low ticktime (such as 2 seconds) versus implementing user-level heartbeats. The former scares me, since it's such a fundamental element of system configuration, but the latter is a bunch of work I'd rather not do. I can tolerate a 10 second delay, but not a 60 second one. Cheers, Tim -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP using GPGMail URL: From thomas@REDACTED Wed Jul 3 14:16:44 2013 From: thomas@REDACTED (Thomas Allen) Date: Wed, 03 Jul 2013 05:16:44 -0700 Subject: [erlang-questions] Shell output for Common Test In-Reply-To: References: Message-ID: <1372853804.15119.140661251444189.43CE8857@webmail.messagingengine.com> This doesn't answer your question about Common Test not outputting to stderr/stdout, but if you find navigating to your test logs tedious, you might find this snippet useful (on ct error, it finds the log and opens it in Firefox): https://github.com/oinksoft/dtl/blob/master/Makefile#L71-L81 Thomas On Wed, Jul 3, 2013, at 02:11 AM, Olav Frengstad wrote: > Hi, > > I'm running common test cases in a shell session, but i'm struggling when > i > want more verbose output. > > Currently I have to navigate through the generated HTML to get stack > traces > and stdout. Is there a way to get all output printed in the shell? > > I'm running ct by calling `ct:run_test(Opts)`. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From garry@REDACTED Wed Jul 3 20:11:50 2013 From: garry@REDACTED (Garry Hodgson) Date: Wed, 03 Jul 2013 14:11:50 -0400 Subject: [erlang-questions] low level packet access from erlang In-Reply-To: <20130605132308.GA32186@ioctl> References: <515C6E13.1050406@research.att.com> <20130403212407.GA25006@ioctl> <51ADE6E1.4030205@research.att.com> <20130604153054.GA31551@ioctl> <51AE7341.4090200@research.att.com> <51AE7D17.2050807@research.att.com> <20130605132308.GA32186@ioctl> Message-ID: <51D46966.8000107@research.att.com> On 06/05/2013 09:23 AM, Michael Santos wrote: > Ask away and if you have any comments/suggestions, please let me know! i'm happily using tun now to intercept and inspect packets, but any that i modify get dropped as malformed when i send them. wireshark tells me the checksums are wrong. and indeed, i don't get what i expect from pkt:checksum(): classify( <<4:4, _IHL:4, _TypeOfService:8, _TotalLength:16, _Identification:16, _FlagX:1, _FlagD:1, _FlagM:1, _FragmentOffset:13, _TTL:8, ?IPPROTO_TCP:8, _HeaderCheckSum:16, _SrcAddr:32, _DestAddr:32, _Rest/binary>> = Raw ) -> { IPv4, IpPayload } = pkt:ipv4( Raw ), { Tcp, TcpPayload } = pkt:tcp( IpPayload ), TestSum = pkt:checksum( [ IPv4, Tcp, TcpPayload ] ), alog:debug( "classify: computed = ~b, actual = ~b", [ TestSum, Tcp#tcp.sum ] ), ... =INFO REPORT==== 3-Jul-2013::14:04:02 === {log,debug,"classify: computed = 43987, actual = 52256",r3@REDACTED} any idea what i'm doing wrong? thanks -- Garry Hodgson AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From yoursurrogategod@REDACTED Wed Jul 3 20:53:32 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Wed, 3 Jul 2013 14:53:32 -0400 Subject: [erlang-questions] Question about some Erlang code in Handbook of Neuroevolution Through Erlang. Message-ID: Hello all, I recently got this book, listed here: http://www.amazon.com/Handbook-Neuroevolution-Through-Erlang-Gene/dp/1461444624 The topic intrigues me, because it makes so much sense to try to design artificial neural networks with Erlang (a point that the author makes very well in the book). I got to chapter 6 and this is the code that I have for constructor.erl. Now, I'm going through the create_NeuroLayers/4 and... I don't see the connection between what's written and artificial neural networks... The code: http://bin.cakephp.org/view/1946842426 I don't want to bash the author, but I'm really not "getting" what he's describing. For example, this description: The function create_NeuroLayers/3 prepares the initial step before starting the recursive cre- ate_NeuroLayers/7 function which will create all the Neuron records. We first generate the place holder Input Ids ?Plus?(Input_IdPs), which are tuples composed of Ids and the vector lengths of the incoming signals associated with them. The proper input_idps will have a weight [me: but we haven't assigned the weight, have we?] list in the tuple instead of the vector length. Because we are only building NNs each with only a single Sensor and Actuator, the IdP to the first layer is composed of the single Sensor Id with the vector length of its sensory signal, likewise in the case of the Actuator. We then generate unique ids for the neurons in the first layer, and drop into the recursive create_NeuroLayers/7 function. Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Thu Jul 4 02:20:31 2013 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 4 Jul 2013 01:20:31 +0100 (BST) Subject: [erlang-questions] Question about some Erlang code in Handbook of Neuroevolution Through Erlang. In-Reply-To: Message-ID: <413153326.21315415.1372897231095.JavaMail.root@erlang-solutions.com> I am working my way through the book, but I haven't got that far yet. Will let you know when I do. Robert ----- Original Message ----- > From: "Yves S. Garret" > To: erlang-questions@REDACTED > Sent: Wednesday, 3 July, 2013 8:53:32 PM > Subject: [erlang-questions] Question about some Erlang code in > Handbook of Neuroevolution Through Erlang. > Hello all, > I recently got this book, listed here: > http://www.amazon.com/Handbook-Neuroevolution-Through-Erlang-Gene/dp/1461444624 > The topic intrigues me, because it makes so much sense to try to > design artificial > neural networks with Erlang (a point that the author makes very well > in the book). > I got to chapter 6 and this is the code that I have for > constructor.erl. Now, I'm going > through the create_NeuroLayers/4 and... I don't see the connection > between what's > written and artificial neural networks... > The code: > http://bin.cakephp.org/view/1946842426 > I don't want to bash the author, but I'm really not "getting" what > he's describing. For example, this description: > The function create_NeuroLayers/3 prepares the initial step before > starting the recursive cre- > ate_NeuroLayers/7 function which will create all the Neuron records. > We first generate the > place holder Input Ids ?Plus?(Input_IdPs), which are tuples composed > of Ids and the vector > lengths of the incoming signals associated with them. The proper > input_idps will have a weight [me: but we haven't assigned the > weight, have we?] > list in the tuple instead of the vector length. Because we are only > building NNs each with only a > single Sensor and Actuator, the IdP to the first layer is composed of > the single Sensor Id with > the vector length of its sensory signal, likewise in the case of the > Actuator. We then generate > unique ids for the neurons in the first layer, and drop into the > recursive create_NeuroLayers/7 > function. > Thanks in advance. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Thu Jul 4 08:57:07 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 14:57:07 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Why do I get the following error when I tried to connect the DB? 2> application:start(crypto). ok 3> application:start(emysql). ok 4> emysql:add_pool(db_pool, 10, "client", "mypass", "localhost", 3306, "hello", utf8). ** exception exit: {failed_to_connect_to_database,econnrefused} in function emysql_conn:open_connections/1 (src/emysql_conn.erl, line 136) in call from emysql:add_pool/9 (src/emysql.erl, line 226) On Fri, Jun 28, 2013 at 3:45 AM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > Hi Mike, > > There is an `as_json/1` in the works as a pull request. Feel free to > provide the other convenience operations so we can add them to the util > module. They could be rather nice to have since everyone needs something > like that. > > > On Thu, Jun 27, 2013 at 9:08 PM, Mike Oxford wrote: > >> Can't look right now, as I'm about to run into a meeting, but this is >> good news. >> >> Did you happen to add an "as_dict" and/or "as_proplist" output settings >> in util? >> >> -mox >> >> On Fri, Jun 21, 2013 at 2:38 AM, Jesper Louis Andersen >> wrote: >> > data properly. On the other hand, this means that th >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 4 08:59:00 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Jul 2013 10:59:00 +0400 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: I've looked at Playback.app traffic, it sends about 30 different SSDP NOTIFY messages on start. From desired.mta@REDACTED Thu Jul 4 09:30:01 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Thu, 4 Jul 2013 09:30:01 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: On Thu, Jul 4, 2013 at 8:57 AM, Barco You wrote: > Why do I get the following error when I tried to connect the DB? Usually this error happens if MySQL server is not running/not listening on localhost:3306. Try this: $ mysql -hlocalhost -P3306 -uclient -Pmypass Does the above work? -- Motiejus Jak?tys From barcojie@REDACTED Thu Jul 4 09:32:22 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 15:32:22 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: sure, I tested already by directly connecting Mysql server from shell with the command you show and it works fine. On Thu, Jul 4, 2013 at 3:30 PM, Motiejus Jak?tys wrote: > On Thu, Jul 4, 2013 at 8:57 AM, Barco You wrote: > > Why do I get the following error when I tried to connect the DB? > > Usually this error happens if MySQL server is not running/not > listening on localhost:3306. Try this: > > $ mysql -hlocalhost -P3306 -uclient -Pmypass > > Does the above work? > > -- > Motiejus Jak?tys > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Thu Jul 4 09:33:12 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 15:33:12 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: > mysql -hlocalhost -P3306 -uclient -pmypass Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 60 Server version: 5.5.31-0ubuntu0.12.04.2 (Ubuntu) Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> On Thu, Jul 4, 2013 at 3:32 PM, Barco You wrote: > sure, I tested already by directly connecting Mysql server from shell with > the command you show and it works fine. > > > On Thu, Jul 4, 2013 at 3:30 PM, Motiejus Jak?tys wrote: > >> On Thu, Jul 4, 2013 at 8:57 AM, Barco You wrote: >> > Why do I get the following error when I tried to connect the DB? >> >> Usually this error happens if MySQL server is not running/not >> listening on localhost:3306. Try this: >> >> $ mysql -hlocalhost -P3306 -uclient -Pmypass >> >> Does the above work? >> >> -- >> Motiejus Jak?tys >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 4 09:41:32 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Jul 2013 11:41:32 +0400 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: localhost may be handled differently and mysql console client may use unix domain socket to connect. Check 127.0.0.1, not localhost From barcojie@REDACTED Thu Jul 4 09:43:35 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 15:43:35 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: The same result: Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false] [dtrace] Eshell V5.10.2 (abort with ^G) 1> application:start(crypto). ok 2> application:start(emysql). ok 3> emysql:add_pool(db_pool, 10, "client", "mypass", "127.0.0.1", 3306, "hello", utf8). ** exception exit: {failed_to_connect_to_database,econnrefused} in function emysql_conn:open_connections/1 (src/emysql_conn.erl, line 136) in call from emysql:add_pool/9 (src/emysql.erl, line 226) 4> On Thu, Jul 4, 2013 at 3:41 PM, Max Lapshin wrote: > localhost may be handled differently and mysql console client may use > unix domain socket to connect. > > Check 127.0.0.1, not localhost > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 4 09:55:12 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Jul 2013 11:55:12 +0400 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: No, you need to check your mysql: telnet 127.0.0.1 3306 From barcojie@REDACTED Thu Jul 4 09:57:09 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 15:57:09 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: what does this indicate? $ telnet 127.0.0.1 3306 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused On Thu, Jul 4, 2013 at 3:55 PM, Max Lapshin wrote: > No, you need to check your mysql: > > telnet 127.0.0.1 3306 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Thu Jul 4 10:16:49 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 16:16:49 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: And I can use Mysqlworkbench to connect the server from a remote Window desktop. On Thu, Jul 4, 2013 at 3:57 PM, Barco You wrote: > what does this indicate? > > $ telnet 127.0.0.1 3306 > Trying 127.0.0.1... > telnet: Unable to connect to remote host: Connection refused > > > > On Thu, Jul 4, 2013 at 3:55 PM, Max Lapshin wrote: > >> No, you need to check your mysql: >> >> telnet 127.0.0.1 3306 >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 4 10:21:28 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Jul 2013 12:21:28 +0400 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Barco, it means that you need to read manual for mysql server and enable socket listener. From barcojie@REDACTED Thu Jul 4 10:35:40 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 16:35:40 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Solved, the bind-address of mhysql should be changed to 0.0.0.0 But the result packet of emysql:execute looks very verbose and fuzzy. How can I get only the clean field value or a row, like {field, value} ? emysql:execute(db_pool, <<"select * from user where tel = 13761975289">>). {result_packet,11, [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], [[1, <<231,138,185,230,157,176>>, <<"237689098736789870">>,1,32,<<"13761975289">>, <<"you.jie@REDACTED">>]], <<>>} On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin wrote: > Barco, it means that you need to read manual for mysql server and > enable socket listener. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Thu Jul 4 10:37:58 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 16:37:58 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: What is this part as following for? [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], On Thu, Jul 4, 2013 at 4:35 PM, Barco You wrote: > Solved, the bind-address of mhysql should be changed to 0.0.0.0 > > But the result packet of emysql:execute looks very verbose and fuzzy. How > can I get only the clean field value or a row, like {field, value} ? > > emysql:execute(db_pool, <<"select * from user where tel = 13761975289">>). > {result_packet,11, > [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, > {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, > {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, > {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, > {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, > {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, > {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], > [[1, > <<231,138,185,230,157,176>>, > <<"237689098736789870">>,1,32,<<"13761975289">>, > <<"you.jie@REDACTED">>]], > <<>>} > > > > On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin wrote: > >> Barco, it means that you need to read manual for mysql server and >> enable socket listener. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Thu Jul 4 11:12:45 2013 From: barcojie@REDACTED (Barco You) Date: Thu, 4 Jul 2013 17:12:45 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Where can I find the detailed description about the fields for result_packet? The result_pakcet looks like {result_packet, Number, List1, List2, <<>>}, and seemingly only List2 is meaningful. Why is there Number, List1 and <<>> ? On Thu, Jul 4, 2013 at 4:37 PM, Barco You wrote: > What is this part as following for? > > [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, > {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, > {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, > {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, > {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, > {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, > {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], > > > On Thu, Jul 4, 2013 at 4:35 PM, Barco You wrote: > >> Solved, the bind-address of mhysql should be changed to 0.0.0.0 >> >> But the result packet of emysql:execute looks very verbose and fuzzy. How >> can I get only the clean field value or a row, like {field, value} ? >> >> emysql:execute(db_pool, <<"select * from user where tel = 13761975289">>). >> {result_packet,11, >> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >> [[1, >> <<231,138,185,230,157,176>>, >> <<"237689098736789870">>,1,32,<<"13761975289">>, >> <<"you.jie@REDACTED">>]], >> <<>>} >> >> >> >> On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin wrote: >> >>> Barco, it means that you need to read manual for mysql server and >>> enable socket listener. >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Thu Jul 4 11:55:26 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Thu, 04 Jul 2013 10:55:26 +0100 Subject: [erlang-questions] Question about some Erlang code in Handbook of Neuroevolution Through Erlang. In-Reply-To: References: Message-ID: <51D5468E.5070103@llaisdy.com> Dear Yves Sorry if this doesn't answer your question. The module constructor.erl creates a neural network Genotype which is a specification for a Phenotype (using the terminology of the book). The Phenotype is the actual neural network. Evolution is simulated by mutating Genotypes and so generating slightly different populations of Phenotypes. The output of constructor:construct_Genotype/4 is a list of records representing a kind of directed graph. The next section (6.6 Developing the Genotype to Phenotype Mapping Module) introduces an exoself.erl module which uses the Genotype specification to generate Phenotypes (and which will also generate a Genotype from a given Phenotype). The create_Neuro* functions (starting with create_NeuroLayers/4) use the given HiddenLayerDensities list to create a list of neuron records. Each neuron lists the ids of its input and output neurons (including sensors and actuators where appropriate). Does that help? Ivan On 03/07/2013 19:53, Yves S. Garret wrote: > Hello all, > > I recently got this book, listed here: > http://www.amazon.com/Handbook-Neuroevolution-Through-Erlang-Gene/dp/1461444624 > > The topic intrigues me, because it makes so much sense to try to design > artificial > neural networks with Erlang (a point that the author makes very well in > the book). > > I got to chapter 6 and this is the code that I have for > constructor.erl. Now, I'm going > through the create_NeuroLayers/4 and... I don't see the connection > between what's > written and artificial neural networks... > > The code: > http://bin.cakephp.org/view/1946842426 > > I don't want to bash the author, but I'm really not "getting" what he's > describing. For example, this description: > > The function create_NeuroLayers/3 prepares the initial step before > starting the recursive cre- > ate_NeuroLayers/7 function which will create all the Neuron records. We > first generate the > place holder Input Ids ?Plus?(Input_IdPs), which are tuples composed of > Ids and the vector > lengths of the incoming signals associated with them. The proper > input_idps will have a weight [me: but we haven't assigned the weight, > have we?] > list in the tuple instead of the vector length. Because we are only > building NNs each with only a > single Sensor and Actuator, the IdP to the first layer is composed of > the single Sensor Id with > the vector length of its sensory signal, likewise in the case of the > Actuator. We then generate > unique ids for the neurons in the first layer, and drop into the > recursive create_NeuroLayers/7 > function. > > Thanks in advance. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From jesper.louis.andersen@REDACTED Thu Jul 4 12:05:43 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 4 Jul 2013 12:05:43 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: A couple of hints: Mysql result packets are results from the underlying database. They are defined as records in mysql.hrl which you may want to include in your application. Furthermore, emysql_util contains some convenience-converter-functions which can turn a result into a more erlang-like result which you can then operate on. to_json conversion is in a pull request which I have yet to merge. On Thu, Jul 4, 2013 at 11:12 AM, Barco You wrote: > Where can I find the detailed description about the fields for > result_packet? > > The result_pakcet looks like {result_packet, Number, List1, List2, <<>>}, > and seemingly only List2 is meaningful. Why is there Number, List1 and <<>> > ? > > > On Thu, Jul 4, 2013 at 4:37 PM, Barco You wrote: > >> What is this part as following for? >> >> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >> >> >> On Thu, Jul 4, 2013 at 4:35 PM, Barco You wrote: >> >>> Solved, the bind-address of mhysql should be changed to 0.0.0.0 >>> >>> But the result packet of emysql:execute looks very verbose and fuzzy. >>> How can I get only the clean field value or a row, like {field, value} ? >>> >>> emysql:execute(db_pool, <<"select * from user where tel = >>> 13761975289">>). >>> {result_packet,11, >>> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >>> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >>> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >>> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >>> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >>> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >>> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >>> [[1, >>> <<231,138,185,230,157,176>>, >>> <<"237689098736789870">>,1,32,<<"13761975289">>, >>> <<"you.jie@REDACTED">>]], >>> <<>>} >>> >>> >>> >>> On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin wrote: >>> >>>> Barco, it means that you need to read manual for mysql server and >>>> enable socket listener. >>>> >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peerst@REDACTED Thu Jul 4 16:09:32 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Thu, 4 Jul 2013 16:09:32 +0200 Subject: [erlang-questions] Compiler list comprehension optimizations References: <51BA7407.3060307@comcast.net> <1371201405.33856.YahooMailNeo@web140103.mail.bf1.yahoo.com> <51BB5B04.4090408@comcast.net> <7c70418ea0bbaa919c3ed2befba9be70.squirrel@chasm.otago.ac.nz> Message-ID: On 2013-06-15 12:21:25 +0000, ok@REDACTED said: > We have seen [... where I = 0 then I+1 while I < 100] > proposed, but that's not actually as general as a functional > language should be. What's needed is a general 'unfold' > proposal, and we've had that for a while too. Somehow I have difficulty to find the "unfold" proposal, in the EEPS I only found tuple comprehensions EEP12 and multigenerators EEP19. Unfold sounds very interesting, I'd like to read the proposal on this. Cheers -- Peer From peerst@REDACTED Thu Jul 4 16:45:45 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Thu, 4 Jul 2013 16:45:45 +0200 Subject: [erlang-questions] erl_driver and select() usage References: Message-ID: On 2013-06-24 09:30:47 +0000, Alessandro Sivieri said: > Well, the asynchronous function of the library that I want to use > actually uses select() in its implementation... so in a way it is > already doing what I usually do with driver_select(). The driver_select() descriptor is necessary to notify the VM of async events. Simplified the VM uses the filedescriptor in a big select or poll call together with all other drivers its waiting on. If you can't use your library without doing its internal select() you need to run it in a separate thread like Max said and create a pipe to get a filedescriptor being passed in driver_select. You then can write a byte to the other end of the pipe from your asyncrously running library to signal that you want your ready_{input,output} callback called to process the event. AFAIK there is no other way to signal events to the VM from an separately running thread. Cheers, -- Peer > On Mon, Jun 24, 2013 at 5:43 AM, Max Lapshin wrote: > It is good if you can find some file descriptor that is responsible for > your hardware events. > If no then launch it in separate thread > > > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From coody1029@REDACTED Tue Jul 2 12:35:18 2013 From: coody1029@REDACTED (coody1029) Date: Tue, 02 Jul 2013 18:35:18 +0800 Subject: [erlang-questions] ]) aa. ~ Message-ID: ??????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alessandro.sivieri@REDACTED Thu Jul 4 18:11:30 2013 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Thu, 4 Jul 2013 18:11:30 +0200 Subject: [erlang-questions] erl_driver and select() usage In-Reply-To: References: Message-ID: On Thu, Jul 4, 2013 at 4:45 PM, Peer Stritzinger wrote: > AFAIK there is no other way to signal events to the VM from an separately > running thread. > Ok, thanks for the advice, I had no time to work on the code in the last few days but I will try it as soon as I can. -- Sivieri Alessandro alessandro.sivieri@REDACTED http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Jul 4 22:24:06 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 4 Jul 2013 22:24:06 +0200 Subject: [erlang-questions] iPhone unable to connect over SSL after upgrading to R16B01 Message-ID: For the list too as it is apparently is so easy to push the wrong button! ---------- Forwarded message ---------- From: Ingela Andin Date: 2013/7/3 Subject: Re: [erlang-questions] iPhone unable to connect over SSL after upgrading to R16B01 To: Ransom Richardson Hi! If you have a none Erlang client that is broken then there is not much I can do about it. Maybe the client will let you configure which chipher suites to use so that you can disable the ones it can not handle (but claims it can)? Maybe there is a patch for the client? The server will pick the first cipher suite that the client advertises and that it supports. Of course you can disable cipher suites in the server too but that will of course disable them for all client connections. Elliptic curve cipher suites where not supported before R16B01 by Erlang ssl application. Regards Ingela Erlang/OTP team - Ericsson AB 2013/7/2 Ransom Richardson > Ingela, thanks for looking at this. Do you still need additional > information? The issue seems to be client specific, and I don't have an > isolated repro. Based on what I'm seeing it looks like {ecdhe_rsa,aes_256_cbc,sha} > works fine when I connect with a python client, but not when I connect with > an iOS client. This is the case even if I also support {rsa,aes_256_cbc,sha256}, > which does work with the iOS client. So the iOS client/Erlang R16B01 server > are choosing an EC cipher that then doesn't work. I think you will see the > issue if you implement just a basic ssl server and connect with an iOS > client. Please let me know if I can provide more information. > > Ransom > > ------------------------------ > *From:* erlang-questions-bounces@REDACTED < > erlang-questions-bounces@REDACTED> on behalf of Andrew Shu < > talklittle@REDACTED> > *Sent:* Monday, June 24, 2013 10:49 AM > *To:* Ingela Andin > *Cc:* erlang-questions@REDACTED > > *Subject:* Re: [erlang-questions] iPhone unable to connect over SSL after > upgrading to R16B01 > > Hi Ingela thanks for troubleshooting this. I ran the openssl server and > pointed Chromium at https://localhost:4433, using the same certs from the > Cowboy example. > > It works, so I don't think the web browser client is the issue. > > Here's the long HTML printout by openssl server, when I hit > https://localhost:4433 using Chromium: > > s_server -accept 4433 -CAfile cowboy-ca.crt -cert server.crt -key > server.key -www > Ciphers supported in s_server binary > > TLSv1/SSLv3:ECDHE-RSA-AES256-GCM-SHA384TLSv1/SSLv3:ECDHE-ECDSA-AES256-GCM-SHA384 > TLSv1/SSLv3:ECDHE-RSA-AES256-SHA384 TLSv1/SSLv3:ECDHE-ECDSA-AES256-SHA384 > TLSv1/SSLv3:ECDHE-RSA-AES256-SHA TLSv1/SSLv3:ECDHE-ECDSA-AES256-SHA > TLSv1/SSLv3:SRP-DSS-AES-256-CBC-SHA TLSv1/SSLv3:SRP-RSA-AES-256-CBC-SHA > TLSv1/SSLv3:DHE-DSS-AES256-GCM-SHA384TLSv1/SSLv3:DHE-RSA-AES256-GCM-SHA384 > TLSv1/SSLv3:DHE-RSA-AES256-SHA256 TLSv1/SSLv3:DHE-DSS-AES256-SHA256 > TLSv1/SSLv3:DHE-RSA-AES256-SHA TLSv1/SSLv3:DHE-DSS-AES256-SHA > TLSv1/SSLv3:DHE-RSA-CAMELLIA256-SHA TLSv1/SSLv3:DHE-DSS-CAMELLIA256-SHA > > TLSv1/SSLv3:ECDH-RSA-AES256-GCM-SHA384TLSv1/SSLv3:ECDH-ECDSA-AES256-GCM-SHA384 > TLSv1/SSLv3:ECDH-RSA-AES256-SHA384 TLSv1/SSLv3:ECDH-ECDSA-AES256-SHA384 > TLSv1/SSLv3:ECDH-RSA-AES256-SHA TLSv1/SSLv3:ECDH-ECDSA-AES256-SHA > TLSv1/SSLv3:AES256-GCM-SHA384 TLSv1/SSLv3:AES256-SHA256 > TLSv1/SSLv3:AES256-SHA TLSv1/SSLv3:CAMELLIA256-SHA > TLSv1/SSLv3:PSK-AES256-CBC-SHA TLSv1/SSLv3:ECDHE-RSA-DES-CBC3-SHA > TLSv1/SSLv3:ECDHE-ECDSA-DES-CBC3-SHA TLSv1/SSLv3:SRP-DSS-3DES-EDE-CBC-SHA > TLSv1/SSLv3:SRP-RSA-3DES-EDE-CBC-SHA TLSv1/SSLv3:EDH-RSA-DES-CBC3-SHA > TLSv1/SSLv3:EDH-DSS-DES-CBC3-SHA TLSv1/SSLv3:ECDH-RSA-DES-CBC3-SHA > TLSv1/SSLv3:ECDH-ECDSA-DES-CBC3-SHA TLSv1/SSLv3:DES-CBC3-SHA > TLSv1/SSLv3:PSK-3DES-EDE-CBC-SHA > TLSv1/SSLv3:ECDHE-RSA-AES128-GCM-SHA256 > TLSv1/SSLv3:ECDHE-ECDSA-AES128-GCM-SHA256TLSv1/SSLv3:ECDHE-RSA-AES128-SHA256 > > TLSv1/SSLv3:ECDHE-ECDSA-AES128-SHA256TLSv1/SSLv3:ECDHE-RSA-AES128-SHA > TLSv1/SSLv3:ECDHE-ECDSA-AES128-SHA TLSv1/SSLv3:SRP-DSS-AES-128-CBC-SHA > TLSv1/SSLv3:SRP-RSA-AES-128-CBC-SHA TLSv1/SSLv3:DHE-DSS-AES128-GCM-SHA256 > TLSv1/SSLv3:DHE-RSA-AES128-GCM-SHA256TLSv1/SSLv3:DHE-RSA-AES128-SHA256 > TLSv1/SSLv3:DHE-DSS-AES128-SHA256 TLSv1/SSLv3:DHE-RSA-AES128-SHA > TLSv1/SSLv3:DHE-DSS-AES128-SHA TLSv1/SSLv3:DHE-RSA-SEED-SHA > TLSv1/SSLv3:DHE-DSS-SEED-SHA TLSv1/SSLv3:DHE-RSA-CAMELLIA128-SHA > TLSv1/SSLv3:DHE-DSS-CAMELLIA128-SHA TLSv1/SSLv3:ECDH-RSA-AES128-GCM-SHA256 > TLSv1/SSLv3:ECDH-ECDSA-AES128-GCM-SHA256TLSv1/SSLv3:ECDH-RSA-AES128-SHA256 > > TLSv1/SSLv3:ECDH-ECDSA-AES128-SHA256 TLSv1/SSLv3:ECDH-RSA-AES128-SHA > TLSv1/SSLv3:ECDH-ECDSA-AES128-SHA TLSv1/SSLv3:AES128-GCM-SHA256 > TLSv1/SSLv3:AES128-SHA256 TLSv1/SSLv3:AES128-SHA > TLSv1/SSLv3:SEED-SHA TLSv1/SSLv3:CAMELLIA128-SHA > TLSv1/SSLv3:PSK-AES128-CBC-SHA TLSv1/SSLv3:ECDHE-RSA-RC4-SHA > TLSv1/SSLv3:ECDHE-ECDSA-RC4-SHA TLSv1/SSLv3:ECDH-RSA-RC4-SHA > TLSv1/SSLv3:ECDH-ECDSA-RC4-SHA TLSv1/SSLv3:RC4-SHA > TLSv1/SSLv3:RC4-MD5 TLSv1/SSLv3:PSK-RC4-SHA > TLSv1/SSLv3:EDH-RSA-DES-CBC-SHA TLSv1/SSLv3:EDH-DSS-DES-CBC-SHA > TLSv1/SSLv3:DES-CBC-SHA TLSv1/SSLv3:EXP-EDH-RSA-DES-CBC-SHA > TLSv1/SSLv3:EXP-EDH-DSS-DES-CBC-SHA TLSv1/SSLv3:EXP-DES-CBC-SHA > TLSv1/SSLv3:EXP-RC2-CBC-MD5 TLSv1/SSLv3:EXP-RC4-MD5 > --- > Ciphers common between both SSL end points: > ECDHE-ECDSA-AES256-SHA ECDHE-RSA-AES256-SHA > DHE-RSA-CAMELLIA256-SHA > DHE-DSS-CAMELLIA256-SHA DHE-RSA-AES256-SHA DHE-DSS-AES256-SHA > > ECDH-RSA-AES256-SHA ECDH-ECDSA-AES256-SHA CAMELLIA256-SHA > > AES256-SHA ECDHE-ECDSA-RC4-SHA > ECDHE-ECDSA-AES128-SHA > ECDHE-RSA-RC4-SHA ECDHE-RSA-AES128-SHA > DHE-RSA-CAMELLIA128-SHA > DHE-DSS-CAMELLIA128-SHA DHE-RSA-AES128-SHA DHE-DSS-AES128-SHA > > ECDH-RSA-RC4-SHA ECDH-RSA-AES128-SHA ECDH-ECDSA-RC4-SHA > > ECDH-ECDSA-AES128-SHA SEED-SHA CAMELLIA128-SHA > > RC4-SHA RC4-MD5 AES128-SHA > > ECDHE-ECDSA-DES-CBC3-SHA ECDHE-RSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA > > EDH-DSS-DES-CBC3-SHA ECDH-RSA-DES-CBC3-SHA > ECDH-ECDSA-DES-CBC3-SHA > DES-CBC3-SHA > --- > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA > SSL-Session: > Protocol : TLSv1.1 > Cipher : ECDHE-RSA-AES256-SHA > Session-ID: > Session-ID-ctx: 01000000 > Master-Key: > 9D707F79ED6FCF06935AAEC6C52E1E42642B2900EC790BC3BBC602F6DD19619220C3C800E173D75313D83DA6053E6786 > Key-Arg : None > PSK identity: None > PSK identity hint: None > SRP username: None > Start Time: 1372085057 > Timeout : 300 (sec) > Verify return code: 0 (ok) > --- > 0 items in the session cache > 0 client connects (SSL_connect()) > 0 client renegotiates (SSL_connect()) > 0 client connects that finished > 1 server accepts (SSL_accept()) > 0 server renegotiates (SSL_accept()) > 1 server accepts that finished > 0 session cache hits > 1 session cache misses > 0 session cache timeouts > 0 callback cache hits > 0 cache full overflows (128 allowed) > --- > no client certificate available > > > > On Mon, Jun 24, 2013 at 8:35 AM, Ingela Andin wrote: > >> Hi again, >> >> 2013/6/23 Andrew Shu >> >>> THANK YOU for posting this! This being my first time using Cowboy HTTPS >>> and secure websockets, I was going crazy trying to figure out why SSL >>> wasn't working via Chromium on Linux, while curl seemed to handle the >>> self-signed certificates okay. It wouldn't have occurred to me that it >>> could be an Erlang bug. >>> >>> After reverting to R16B, and removing all traces of R16B01, everything >>> seems working. >>> I wasted a lot of time swapping out SSL certificates to no avail. I >>> think sticking with R16B is the best, or only, solution for now. >>> >>> I had been getting a Chromium gray error screen with >>> "ERR_SSL_CLIENT_AUTH_SIGNATURE_FAILED". Not the usual "this certificate is >>> not trusted" red screen. >>> Firefox choked too. Curl seemed ok, strangely enough. >>> >>> >> Ok just make sure I run an openssl client against the erlang server >> too, with the cowboy example (as you >> try to connect to an erlang-server with other clients). This works >> too. So it seems openssl and curl can connect to the erlang-server but not >> chrom and firefox? And the connection fails due to that the client sends >> and alert. So atleas this problem seems not to be related to ECDSA >> certificates. (The missed TODO) . >> >> The other clients could also have issues with ecc-cipher suites, you >> could try to setting up an openssl server using >> >> openssl s_server -accept 4433 -CAfile ca.crt -cert server.crt -key >> server.key >> >> and trying the clients to see if they can connect with ecc-ciphers. >> >> The following command must return elliptic curve ciper cuites ECDH* >> ECDSA* >> >> > openssl ciphers >> >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Jul 4 22:34:17 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 4 Jul 2013 22:34:17 +0200 Subject: [erlang-questions] Restarting supervision tree In-Reply-To: <51D1482F.5090800@gmail.com> References: <51D1482F.5090800@gmail.com> Message-ID: Hi! Stopping and starting your application again would accomplish a "restart" of the whole supervison tree. I agree that there are things missing from the supervisors that hopefully we will be able to adress in the future. Regards Ingela Erlang OTP/team - Ericsson AB 2013/7/1 Martin Dimitrov > Hello, > > In our app, I would like to reset (restart) a whole supervision tree. If > I return {stop, normal, []} from a permanent worker with supervisor's > restart strategy set to one_for_all, the supervisor will terminate and > restart the whole tree, right? But this will count towards the maximum > restart frequency and I have to avoid this. If I call > supervisor:terminate_child/2 and then supervisor:restart_child/2 on the > permanent worker, will this affect the maximum restart frequency? > > Thank you. > > Martin > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Jul 4 22:50:48 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 4 Jul 2013 22:50:48 +0200 Subject: [erlang-questions] Async ssl_accept, ssl:close, gen_tcp:send and ssl:send Message-ID: Again for the list, and now back to my vacation. ---------- Forwarded message ---------- From: Ingela Andin Date: 2013/7/3 Subject: Re: [erlang-questions] Async ssl_accept, ssl:close, gen_tcp:send and ssl:send To: Lo?c Hoguin Hi! 2013/6/29 Lo?c Hoguin > Hello, > > I'd like to start a discussion on making additions to the gen_tcp and ssl > API to allow it to handle more asynchronous operations. The reasons for > allowing them async are as follow: > > * async ssl_accept: This call can take a long time because there's a few > round trips between the two endpoints to perform the handshake. Therefore > there's a lot of waiting going on, not just busy operations. It could be > more interesting to not block a process for this because the process could > for example initialize the state for the connection while it waits for the > handshake to be completed, reducing the latency of applications. > > Is it not enough that ssl:transport_accept and ssl:ssl_accept can/should be called by different processes? > * async ssl:close: This call can also take a long time for similar > reasons. I think the same can apply to gen_tcp? Anyway if you have a long > running process that holds a connection at some point, and wants to close > it, it might interrupt its normal workflow. A temporary solution would be > to have a process dedicated to closing these sockets and pass it ownership, > but if that can be avoided... > > * async send: Send also blocks, I suppose because it waits for the TCP > ack before returning. Sometimes you don't have the luxury to wait, so it > would be good to have an alternative documented solution. An undocumented > solution can be found here: http://www.erlang-factory.com/** > conference/SFBay2013/speakers/**GeoffCant I will not go in to the other points now, as I am supposed to be on vacation, so I will go back to that! Regards Ingela Erlang/OTP team - Ericsson AB > > I'm mostly interested in the first point, as this is something that doesn't > really have solutions. I could help with implementing or testing it if > there is interest. > > There could also be an async TCP accept I suppose, except I already looked > into adding that and the current code doesn't allow it easily. And I don't > need it anymore, but perhaps someone does. > > Thanks. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jul 5 01:10:14 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 5 Jul 2013 11:10:14 +1200 Subject: [erlang-questions] Compiler list comprehension optimizations In-Reply-To: References: <51BA7407.3060307@comcast.net> <1371201405.33856.YahooMailNeo@web140103.mail.bf1.yahoo.com> <51BB5B04.4090408@comcast.net> <7c70418ea0bbaa919c3ed2befba9be70.squirrel@chasm.otago.ac.nz> Message-ID: <9ACE6ABA-B9EB-428B-BDE2-300DFB683684@cs.otago.ac.nz> On 5/07/2013, at 2:09 AM, Peer Stritzinger wrote: > On 2013-06-15 12:21:25 +0000, ok@REDACTED said: >> We have seen [... where I = 0 then I+1 while I < 100] >> proposed, but that's not actually as general as a functional >> language should be. What's needed is a general 'unfold' >> proposal, and we've had that for a while too. > > Somehow I have difficulty to find the "unfold" proposal, in the EEPS I only found tuple comprehensions EEP12 and multigenerators EEP19. I didn't say that there was an unfold EEP, I said that a certain syntax had been proposed (by me, in this mailing list) and that there had been an unfold proposal (also by me, also in this mailing list). The unfold approach I described used your_unfold(Your_State) -> [] or [Next|Your_State'] However, Haskell has led the way. What we really want is something closer to streams. A "stream" function maps a state to one of done {skip,New_State} {next,New_State,Item} to_list(Unfolder, State) -> case Unfolder(State) of done -> [] ; {skip,New_State} -> to_list(Unfolder, New_State) ; {next,New_State,Item} -> [Item | to_list(Unfolder, New_State)] end. The nice thing about streams is that they can handle mapping and filtering. s_map(F, Unfolder) -> fun (State) -> case R = Unfolder(State) of done -> R ; {skip,State1} -> R ; {next,State1,Item} -> {next,State1,F(Item)} end end. s_filter(P, Unfolder) -> fun (State) -> case R = Unfolder(State) of done -> R ; {skip,State1} -> R ; {next,State1,Item} -> case P(Item) of false -> {skip,State1} ; true -> R end end end. They can also handle a 'while': s_while(P, Unfolder) -> fun (State) -> case R = Unfolder(State) of done -> R ; {skip,State1} -> R ; {next,State1,Item} -> case P(Item) of false -> done ; true -> R end end end. So we can compose any chain of "map", "filter", and "takeWhile" steps into a single unfolder->unfolder function, which can then be applied to anything we can convert to a stream, which includes lists, integer ranges, tuples, and other things. With a little ingenuity, we can squeeze in "expanding the state" so that we can include "dropWhile", every Nth, and other wrinkles into our chains. From wbin00@REDACTED Fri Jul 5 07:47:48 2013 From: wbin00@REDACTED (Bin Wang) Date: Fri, 5 Jul 2013 13:47:48 +0800 Subject: [erlang-questions] How to handle deps while using rebar to release? Message-ID: I'm using "rebar generate" to handle release, but when I start the application, the deps I use will not be found. I'm wonder how to config rebar.config and reltool.config to handle dependencies? Thanks. From moxford@REDACTED Fri Jul 5 08:35:44 2013 From: moxford@REDACTED (Mike Oxford) Date: Thu, 4 Jul 2013 23:35:44 -0700 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: emysql_util also has some other methods ... as_record/1, as_proplist/1, as_dict/1 Those will clean up the raw output into something far more usable/accessible. -mox On Thu, Jul 4, 2013 at 2:12 AM, Barco You wrote: > Where can I find the detailed description about the fields for > result_packet? > > The result_pakcet looks like {result_packet, Number, List1, List2, <<>>}, > and seemingly only List2 is meaningful. Why is there Number, List1 and <<>> > ? > > > On Thu, Jul 4, 2013 at 4:37 PM, Barco You wrote: > >> What is this part as following for? >> >> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >> >> >> On Thu, Jul 4, 2013 at 4:35 PM, Barco You wrote: >> >>> Solved, the bind-address of mhysql should be changed to 0.0.0.0 >>> >>> But the result packet of emysql:execute looks very verbose and fuzzy. >>> How can I get only the clean field value or a row, like {field, value} ? >>> >>> emysql:execute(db_pool, <<"select * from user where tel = >>> 13761975289">>). >>> {result_packet,11, >>> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >>> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >>> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >>> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >>> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >>> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >>> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >>> [[1, >>> <<231,138,185,230,157,176>>, >>> <<"237689098736789870">>,1,32,<<"13761975289">>, >>> <<"you.jie@REDACTED">>]], >>> <<>>} >>> >>> >>> >>> On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin wrote: >>> >>>> Barco, it means that you need to read manual for mysql server and >>>> enable socket listener. >>>> >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Fri Jul 5 08:53:04 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 5 Jul 2013 08:53:04 +0200 Subject: [erlang-questions] [ANN] nat_upnp In-Reply-To: References: Message-ID: On Thu, Jul 4, 2013 at 8:59 AM, Max Lapshin wrote: > I've looked at Playback.app traffic, it sends about 30 different SSDP > NOTIFY messages on start. > mmm is there anything wrong ? - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre.graf@REDACTED Fri Jul 5 10:11:01 2013 From: andre.graf@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Fri, 5 Jul 2013 10:11:01 +0200 Subject: [erlang-questions] How to handle deps while using rebar to release? In-Reply-To: References: Message-ID: Hello I usually put {lib_dirs, ["../deps"]}, into the sys-configuration inside reltool.config. Cheers, Andre On 5 July 2013 07:47, Bin Wang wrote: > I'm using "rebar generate" to handle release, but when I start the > application, the deps I use will not be found. > > I'm wonder how to config rebar.config and reltool.config to handle > dependencies? Thanks. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jeti789@REDACTED Fri Jul 5 10:27:44 2013 From: jeti789@REDACTED (jeti789@REDACTED) Date: Fri, 5 Jul 2013 10:27:44 +0200 (CEST) Subject: [erlang-questions] What happens in Erlang if return receipt never arrives? Message-ID: An HTML attachment was scrubbed... URL: From paulperegud@REDACTED Fri Jul 5 12:07:07 2013 From: paulperegud@REDACTED (Paul Peregud) Date: Fri, 5 Jul 2013 12:07:07 +0200 Subject: [erlang-questions] What happens in Erlang if return receipt never arrives? In-Reply-To: References: Message-ID: In practice you just subscribe to notification of receivers death. It does not solve a case when your receiver is overflowed with work, but it is solved by careful planning of flow of messages in your application. On Fri, Jul 5, 2013 at 10:27 AM, wrote: > I just happened to read the thesis of Joe Armstrong and don't have much > prior knowledge of Erlang. I wonder what happens if a delivery receipt for > some message never arrives. What does the sending actor do? It sends the > message another time? This could confuse the recipient actor when it > receives the same message another time. It has to be able to tell that its > receipt was not received and therefore the second message is void. > > That kind of problems always kept me away from solutions where message > delivery is not transactional. I think I know the answer: the sending actor > tells its supervising actor that something must be wrong when it didn't > obtain a receipt in reasonable time causing the supervisor to take some > action (like restarting the involed actors or something). Is this correct? I > see no other solution that doesn't result in theroretically possible > infinite message sends. > > Thanks for any answer, > > Bienlein > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Best regards, Paul Peregud +48602112091 From jeti789@REDACTED Fri Jul 5 13:23:54 2013 From: jeti789@REDACTED (jeti789@REDACTED) Date: Fri, 5 Jul 2013 13:23:54 +0200 (CEST) Subject: [erlang-questions] Fw: Re: What happens in Erlang if return receipt never arrives? References: , Message-ID: An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Jul 5 15:27:21 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 14:27:21 +0100 Subject: [erlang-questions] What happens in Erlang if return receipt never arrives? In-Reply-To: References: , Message-ID: What we're talking about here are byzantine failures, and despite a huge body of research on these topics, there are no fool-proof answers or solutions. Erlang does not use ACKs implicitly - sending a message to a process is an asynchronous (fire and forget) action and no implicit reply channel exists. This is a deliberate design decision, since making various levels of guarantee about delivery is not a core requirement, but rather, an application specific need. You ask what "happens if the receiver is not able to notify the world about its death", and this is handled in two ways by erlang's runtime system. If the receiver is a local process (i.e., resides on the same physical emulator) then as soon as it exits, any monitors that have been set up (by other processes) will be triggered and a message sent to the subscribers. If the receiver resides on another node (whether physically on the same host or not) then monitors are managed at both ends and when the remote process dies, monitor notifications are sent across the wire to subscribers. If the receiver resides on another node and the network link dies, then the net_kernel's tick timer will detect the disconnect (after a default timeout) and once it considers the node to have been disconnected, will fire monitor notifications to all (local) subscribers. In terms of guarantees about delivery, in practise these are impossible to make without certain caveats. Guaranteed "exactly once" delivery, for example, is a promise that cannot be made without theoretically infinite storage capacity being available for all participants. In practise one can "get away with" finite storage capacity, but that means under certain circumstances, the promise/guarantee won't hold. The way to 'ensure' a message has arrived across a distribution link then, is to expect an ACK. You must wait a potentially infinite time until that ACK arrives before considering the message delivered, unless you see a monitor notification indicating the receiver's death, at which point you might re-send once the receiver comes back online. Even then, you cannot be sure if the receiver did in fact see the message *and* sent an ACK, but the network link died before the ACK was delivered. In this case, you'll have to re-send anyway - since you've no way of knowing message was actually delivered, having not seen the lost ACK - and the receiver will have to de-dup the message(s) at his end. Other configurations are possible. Introducing a ring overlay, you can achieve atomic broadcast with two round trips, one for delivery and another for ACK, though you'd have to either have each node take responsibility for its neighbours messages (and ACKs) during group membership changes, or make group membership changes atomic by using distributed transactions (e.g., paxos, etc). There is a definite cost to adding these kind of guarantees. Viz your observation about wanting to make delivery atomic, in practise many solutions go the asynchronous route in order to avoid the performance (and latency) overheads of distributed transactions. If you want to use ACKs in your erlang code, you can either implement this yourself or use gen_server:call to make a synchronous invocation that waits on a response (and monitors the server). This will not only throttle the producer, but will also make the server a bottleneck in the system, since it can only handle one message at once. Cheers, Tim On 5 Jul 2013, at 12:23, jeti789@REDACTED wrote: > I see. What happens if the receiver is not able to notify the world about its death? The sender after a timeout sends a notification out about a possible death? The supervisor detects that there is no more heart beat? Not wanting to be pedantic. Just trying to understand how this works. Is there some document/book where you can read about those things. I like that kind of problems ;-). > > Regards, Oliver > In practice you just subscribe to notification of receivers death. It > does not solve a case when your receiver is overflowed with work, but > it is solved by careful planning of flow of messages in your > application. > > > On Fri, Jul 5, 2013 at 10:27 AM, wrote: > > I just happened to read the thesis of Joe Armstrong and don't have much > > prior knowledge of Erlang. I wonder what happens if a delivery receipt for > > some message never arrives. What does the sending actor do? It sends the > > message another time? This could confuse the recipient actor when it > > receives the same message another time. It has to be able to tell that its > > receipt was not received and therefore the second message is void. > > > > That kind of problems always kept me away from solutions where message > > delivery is not transactional. I think I know the answer: the sending actor > > tells its supervising actor that something must be wrong when it didn't > > obtain a receipt in reasonable time causing the supervisor to take some > > action (like restarting the involed actors or something). Is this correct? I > > see no other solution that doesn't result in theroretically possible > > infinite message sends. > > > > Thanks for any answer, > > > > Bienlein > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -- > Best regards, > Paul Peregud > +48602112091 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Fri Jul 5 15:34:43 2013 From: barcojie@REDACTED (Barco You) Date: Fri, 5 Jul 2013 21:34:43 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Thank you all. Whats the maximum connections can it bear when add a pool? If I use Cowboy to provide a REST API for data query and I start up the http listener with 10000 acceptors and 10000 emysql connections, does that mean i can support 10000 concurrency? On 5 Jul 2013 14:35, "Mike Oxford" wrote: > emysql_util also has some other methods ... as_record/1, as_proplist/1, > as_dict/1 > Those will clean up the raw output into something far more > usable/accessible. > > -mox > > On Thu, Jul 4, 2013 at 2:12 AM, Barco You wrote: > >> Where can I find the detailed description about the fields for >> result_packet? >> >> The result_pakcet looks like {result_packet, Number, List1, List2, <<>>}, >> and seemingly only List2 is meaningful. Why is there Number, List1 and <<>> >> ? >> >> >> On Thu, Jul 4, 2013 at 4:37 PM, Barco You wrote: >> >>> What is this part as following for? >>> >>> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >>> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >>> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >>> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >>> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >>> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >>> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >>> >>> >>> On Thu, Jul 4, 2013 at 4:35 PM, Barco You wrote: >>> >>>> Solved, the bind-address of mhysql should be changed to 0.0.0.0 >>>> >>>> But the result packet of emysql:execute looks very verbose and fuzzy. >>>> How can I get only the clean field value or a row, like {field, value} ? >>>> >>>> emysql:execute(db_pool, <<"select * from user where tel = >>>> 13761975289">>). >>>> {result_packet,11, >>>> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >>>> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >>>> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >>>> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >>>> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >>>> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >>>> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>>> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >>>> [[1, >>>> <<231,138,185,230,157,176>>, >>>> <<"237689098736789870">>,1,32,<<"13761975289">>, >>>> <<"you.jie@REDACTED">>]], >>>> <<>>} >>>> >>>> >>>> >>>> On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin wrote: >>>> >>>>> Barco, it means that you need to read manual for mysql server and >>>>> enable socket listener. >>>>> >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmakarlsson@REDACTED Fri Jul 5 15:52:42 2013 From: jmakarlsson@REDACTED (J K) Date: Fri, 5 Jul 2013 06:52:42 -0700 (PDT) Subject: [erlang-questions] Cowboy static file cahce Message-ID: <1373032362.68046.YahooMailNeo@web162806.mail.bf1.yahoo.com> Hi, does anyone know if Cowboy can do static file caching? BR Johan -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Fri Jul 5 16:04:29 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 5 Jul 2013 09:04:29 -0500 Subject: [erlang-questions] OSCON 2013 Message-ID: Are there any Erlangers going to OSCON in Portland in a few weeks? From essen@REDACTED Fri Jul 5 16:08:05 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 05 Jul 2013 16:08:05 +0200 Subject: [erlang-questions] Cowboy static file cahce In-Reply-To: <1373032362.68046.YahooMailNeo@web162806.mail.bf1.yahoo.com> References: <1373032362.68046.YahooMailNeo@web162806.mail.bf1.yahoo.com> Message-ID: <51D6D345.5020005@ninenines.eu> On 07/05/2013 03:52 PM, J K wrote: > Hi, > does anyone know if Cowboy can do static file caching? It doesn't, you could do it, but you probably shouldn't do it. When using TCP the static handler will use sendfile to send the files. Sendfile does the write directly from the kernel, and if the file was in the kernel file cache (if it was requested before/enough) it'll be very fast. When using SSL you can't just write the file to the socket. So you might want something else. Generally the best solution is to use a CDN for files (on a cookie-less domain) and this is the only recommendation we make. The static handler is only there to help during development. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From g@REDACTED Fri Jul 5 16:27:14 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 5 Jul 2013 09:27:14 -0500 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: On Fri, Jun 21, 2013 at 4:38 AM, Jesper Louis Andersen wrote: > Hi Erlangers, > > Henning Diedrich and I have been working a bit on the Emysql driver, > > https://github.com/Eonblast/Emysql > > cleaning it up and removing old cruft from it. Over the last month or so we > have reached a point where 3-4 emysql branches were merged back into the > above path. It has been running in production at a customer site, Issuu, for > about 1 month without any problems to report and we believe it is fairly > stable. > > NOTE: There is a backwards-incompatible change in `master`. We removed the > encoding support of automatic conversion of binaries into utf-8. This means > that client *users* are now responsible for encoding data properly. On the > other hand, this means that the driver won't try to encode binary BLOB data > as utf-8 anymore :) > > Branches of interest: `master`. Try it out. It contains the merger. > Tags of interest: `0.2.16`. This tag is just *before* the merge, so if you > need time to adapt to the new setup, you can lock to this tag and obtain the > old behaviour. Sorry for the delay in looking at this. At first glance, it looks like I'd be losing the connect timeout enhancements from here: https://github.com/gar1t/Emysql I noticed there's a connect test change however -- is that a functional equivalent to handling connection timeouts? It's less important, but that repo also has a much saner app_pool/2 function that uses Options rather than the 8 - 9 [sick] arguments of the other two variants. Garrett From jmakarlsson@REDACTED Fri Jul 5 16:36:31 2013 From: jmakarlsson@REDACTED (J K) Date: Fri, 5 Jul 2013 07:36:31 -0700 (PDT) Subject: [erlang-questions] Cowboy static file cahce In-Reply-To: <51D6D345.5020005@ninenines.eu> References: <1373032362.68046.YahooMailNeo@web162806.mail.bf1.yahoo.com> <51D6D345.5020005@ninenines.eu> Message-ID: <1373034991.17594.YahooMailNeo@web162803.mail.bf1.yahoo.com> Ok, I've already written a very simple cache function??(based on ets)?for other usage, it just happens that you could probably use that to cache anything. We might want to use SSL in our solution. I guess I should do some measurements then. NOTE: this is just html and js files so it might be that the OS caching function is sufficient. Thanks, Johan >________________________________ > From: Lo?c Hoguin >To: J K >Cc: Erlang Questions >Sent: Friday, July 5, 2013 4:08 PM >Subject: Re: [erlang-questions] Cowboy static file cahce > > >On 07/05/2013 03:52 PM, J K wrote: >> Hi, >> does anyone know if Cowboy can do static file caching? > >It doesn't, you could do it, but you probably shouldn't do it. > >When using TCP the static handler will use sendfile to send the files. >Sendfile does the write directly from the kernel, and if the file was in >the kernel file cache (if it was requested before/enough) it'll be very >fast. > >When using SSL you can't just write the file to the socket. So you might >want something else. > >Generally the best solution is to use a CDN for files (on a cookie-less >domain) and this is the only recommendation we make. The static handler >is only there to help during development. > >-- >Lo?c Hoguin >Erlang Cowboy >Nine Nines >http://ninenines.eu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jleivent@REDACTED Fri Jul 5 16:49:14 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Fri, 05 Jul 2013 10:49:14 -0400 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? Message-ID: <51D6DCEA.7080705@comcast.net> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? Or, is there some kind of internal persistent state that prevents this? If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node. -- Jonathan From simonstl@REDACTED Fri Jul 5 17:05:04 2013 From: simonstl@REDACTED (Simon St.Laurent) Date: Fri, 05 Jul 2013 11:05:04 -0400 Subject: [erlang-questions] OSCON 2013 In-Reply-To: References: Message-ID: <51D6E0A0.9050809@simonstl.com> On 7/5/13 10:04 AM, Garrett Smith wrote: > Are there any Erlangers going to OSCON in Portland in a few weeks? I'll be there, speaking on Making Functional Programming Approachable. Francesco Cesarini will also be speaking, on Erlang 101 and OTP. Hopefully there will be more! Thanks, -- Simon St.Laurent http://simonstl.com/ From noah.schwartz1@REDACTED Fri Jul 5 17:07:35 2013 From: noah.schwartz1@REDACTED (Noah Schwartz) Date: Fri, 5 Jul 2013 11:07:35 -0400 Subject: [erlang-questions] inconsistent results with fragmented table Message-ID: I have a table of type set that worked fine as a non-fragmented table. We started seeing performance issues with the table and with the anticipation of more clients in the system, we were worried the size would exceed the 2 GB max. The table contains chat messages and we purge records older than 7 days. There seem to be a number of keys that don't show up in a read activity but, do show up when doing a table dump, a select, or using qlc to find records. As such, when this purge policy runs records are found that when I try to delete don't actually get deleted. It almost seems like read/delete are working off of one set while qlc is working off of another set. Code/outputs below. As you can see for a given key read returns nothing, qlc returns an object, delete says it worked ok. Running the code again gives the same results. Am I using the fragmentation API incorrectly somehow? Thanks in advance *Code:* -module(sel). -include_lib("stdlib/include/qlc.hrl"). -record(archive_message313, {owner_with_day_utc, owner, with, day_gregorian_seconds, messages = []}). -export([run/0]). run () -> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, DayUtc = {2013, 6, 28}, K = {Owner, With, DayUtc}, DelFun = fun () -> mnesia:delete({archive_message313, K}) end, ReadFun = fun () -> mnesia:read({archive_message313, K}) end, QlcFun = fun () -> Q = qlc:q([M || M <- mnesia:table(archive_message313), M#archive_message313.owner_with_day_utc == K]), qlc:eval(Q) end, Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), {read_result, Read, qlc_result, Qlc, del_result, Del}. *Output:* {read_result,[],qlc_result, [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, {2013,6,28}}, {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, 63539596800, [{{{2013,6,28},{13,36,34}}, {xmlelement,"forwarded", [{"xmlns","urn:xmpp:forward:0"}], [{xmlelement,"delay", [{"xmlns","urn:xmpp:delay"}, {"stamp","2013-06-28T13:36:34.000000Z"}], []}, {xmlelement,"message", [{"to", " 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, {"from", " d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, {"type","chat"}], [{xmlelement,"body",[],[...]}]}]}, 1372426594.889164}, {{{2013,6,28},{13,36,35}}, {xmlelement,"forwarded", [{"xmlns","urn:xmpp:forward:0"}], [{xmlelement,"delay", [{"xmlns","urn:xmpp:delay"}, {"stamp","2013-06-28T13:36:35.000000Z"}], []}, {xmlelement,"message", [{"to", " 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, {"from", " d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, {"type",[...]}], [{xmlelement,"body",[],...}]}]}, 1372426595.124266}, {{{2013,6,28},{13,36,35}}, {xmlelement,"forwarded", [{"xmlns","urn:xmpp:forward:0"}], [{xmlelement,"delay", [{"xmlns","urn:xmpp:delay"}, {"stamp","2013-06-28T13:36:35.000000Z"}], []}, {xmlelement,"message", [{"to", " 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, {"from",[...]}, {[...],...}], [{xmlelement,[...],...}]}]}, 1372426595.337535}]}], del_result,ok} -- Noah -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Jul 5 17:22:37 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 16:22:37 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <51D6DCEA.7080705@comcast.net> References: <51D6DCEA.7080705@comcast.net> Message-ID: <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. Cheers Tim On 5 Jul 2013, at 15:49, Jonathan Leivent wrote: > In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? Or, is there some kind of internal persistent state that prevents this? > > If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node. > > -- Jonathan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sergej.jurecko@REDACTED Fri Jul 5 17:24:32 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Fri, 5 Jul 2013 17:24:32 +0200 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> Message-ID: Well yes erlang does reconnect, but you still get a nodedown/nodeup message no? Sergej On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: > As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. > > Cheers > Tim > > On 5 Jul 2013, at 15:49, Jonathan Leivent wrote: > >> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? Or, is there some kind of internal persistent state that prevents this? >> >> If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node. >> >> -- Jonathan >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Fri Jul 5 17:26:15 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 05 Jul 2013 17:26:15 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: <51D6E597.8030903@ninenines.eu> On 07/05/2013 03:34 PM, Barco You wrote: > Thank you all. > > Whats the maximum connections can it bear when add a pool? If I use > Cowboy to provide a REST API for data query and I start up the http > listener with 10000 acceptors and 10000 emysql connections, does that > mean i can support 10000 concurrency? Not answering your question. Acceptors aren't the maximum number of connections, it's the processes doing the actual accept. They're only configurable to allow tweaking performance, and with 10000 you will accept slower, that's too much. The setting you probably meant is max_connections. > On 5 Jul 2013 14:35, "Mike Oxford" > wrote: > > emysql_util also has some other methods ... as_record/1, > as_proplist/1, as_dict/1 > Those will clean up the raw output into something far more > usable/accessible. > > -mox > > On Thu, Jul 4, 2013 at 2:12 AM, Barco You > wrote: > > Where can I find the detailed description about the fields for > result_packet? > > The result_pakcet looks like {result_packet, Number, List1, > List2, <<>>}, and seemingly only List2 is meaningful. Why is > there Number, List1 and <<>> ? > > > On Thu, Jul 4, 2013 at 4:37 PM, Barco You > wrote: > > What is this part as following for? > > [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, > > {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, > > {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, > > {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, > > {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, > > {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, > > {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], > > > On Thu, Jul 4, 2013 at 4:35 PM, Barco You > > wrote: > > Solved, the bind-address of mhysql should be changed to > 0.0.0.0 > > But the result packet of emysql:execute looks very > verbose and fuzzy. How can I get only the clean field > value or a row, like {field, value} ? > > emysql:execute(db_pool, <<"select * from user where tel > = 13761975289">>). > {result_packet,11, > > [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, > > {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, > > {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, > > {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, > > {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, > > {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, > > {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, > > <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], > [[1, > <<231,138,185,230,157,176>>, > > <<"237689098736789870">>,1,32,<<"13761975289">>, > <<"you.jie@REDACTED > ">>]], > <<>>} > > > > On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin > > > wrote: > > Barco, it means that you need to read manual for > mysql server and > enable socket listener. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From jesper.louis.andersen@REDACTED Fri Jul 5 17:30:59 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 5 Jul 2013 17:30:59 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 3:34 PM, Barco You wrote: > Thank you all. > > Whats the maximum connections can it bear when add a pool? If I use Cowboy > to provide a REST API for data query and I start up the http listener with > 10000 acceptors and 10000 emysql connections, does that mean i can support > 10000 concurrency? > MySQL will probably not handle 10000 simultaneous connections well. The idea is to keep the connection count lower, around 100 perhaps, and then use other methods if you need a higher concurrency count. Note that while you have an HTTP connection, you might only be communicating with the underlying database for a short amount of time. And then most of the time is spent sending a result back to the client and so on. In other words, it might not gain you much to have a high connection count to MySQL. Rather, you should tune. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Fri Jul 5 17:42:04 2013 From: barcojie@REDACTED (Barco You) Date: Fri, 5 Jul 2013 23:42:04 +0800 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: <51D6E597.8030903@ninenines.eu> References: <51D6E597.8030903@ninenines.eu> Message-ID: Then what should be a good number of acceptors and max_connections? depending on hardware? Can have 1 acceptor and thousands of max_connections? Is it only max_connections that determines the concurrency? On 5 Jul 2013 23:26, "Lo?c Hoguin" wrote: > On 07/05/2013 03:34 PM, Barco You wrote: > >> Thank you all. >> >> Whats the maximum connections can it bear when add a pool? If I use >> Cowboy to provide a REST API for data query and I start up the http >> listener with 10000 acceptors and 10000 emysql connections, does that >> mean i can support 10000 concurrency? >> > > Not answering your question. > > Acceptors aren't the maximum number of connections, it's the processes > doing the actual accept. They're only configurable to allow tweaking > performance, and with 10000 you will accept slower, that's too much. The > setting you probably meant is max_connections. > > On 5 Jul 2013 14:35, "Mike Oxford" > > wrote: >> >> emysql_util also has some other methods ... as_record/1, >> as_proplist/1, as_dict/1 >> Those will clean up the raw output into something far more >> usable/accessible. >> >> -mox >> >> On Thu, Jul 4, 2013 at 2:12 AM, Barco You > > wrote: >> >> Where can I find the detailed description about the fields for >> result_packet? >> >> The result_pakcet looks like {result_packet, Number, List1, >> List2, <<>>}, and seemingly only List2 is meaningful. Why is >> there Number, List1 and <<>> ? >> >> >> On Thu, Jul 4, 2013 at 4:37 PM, Barco You > > wrote: >> >> What is this part as following for? >> >> [{field,2,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"userid">>,<<"userid">>,8,<<**>>,63,20,16899,0}, >> >> {field,3,<<"def">>,<<"**fangchan">>,<<"user">>,<<"**user">>, >> >> <<"name">>,<<"name">>,254,<<>>**,33,36,20489,0}, >> >> {field,4,<<"def">>,<<"**fangchan">>,<<"user">>,<<"**user">>, >> >> <<"id_no">>,<<"id_no">>,254,<<**>>,33,54,20481,0}, >> >> {field,5,<<"def">>,<<"**fangchan">>,<<"user">>,<<"**user">>, >> <<"sex">>,<<"sex">>,1,<<>>,63,** >> 1,0,0}, >> >> {field,6,<<"def">>,<<"**fangchan">>,<<"user">>,<<"**user">>, >> <<"age">>,<<"age">>,2,<<>>,63,** >> 6,0,0}, >> >> {field,7,<<"def">>,<<"**fangchan">>,<<"user">>,<<"**user">>, >> <<"tel">>,<<"tel">>,254,<<>>,** >> 33,33,0,0}, >> >> {field,8,<<"def">>,<<"**fangchan">>,<<"user">>,<<"**user">>, >> >> <<"email">>,<<"email">>,253,<<**>>,33,132,0,0}], >> >> >> On Thu, Jul 4, 2013 at 4:35 PM, Barco You >> > wrote: >> >> Solved, the bind-address of mhysql should be changed to >> 0.0.0.0 >> >> But the result packet of emysql:execute looks very >> verbose and fuzzy. How can I get only the clean field >> value or a row, like {field, value} ? >> >> emysql:execute(db_pool, <<"select * from user where tel >> = 13761975289">>). >> {result_packet,11, >> >> [{field,2,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"userid">>,<<"userid">>,8,<<**>>,63,20,16899,0}, >> >> {field,3,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"name">>,<<"name">>,254,<<>>**,33,36,20489,0}, >> >> {field,4,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"id_no">>,<<"id_no">>,254,<<**>>,33,54,20481,0}, >> >> {field,5,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"sex">>,<<"sex">>,1,<<>>,63,**1,0,0}, >> >> {field,6,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"age">>,<<"age">>,2,<<>>,63,**6,0,0}, >> >> {field,7,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"tel">>,<<"tel">>,254,<<>>,**33,33,0,0}, >> >> {field,8,<<"def">>,<<"**fangchan">>,<<"user">>,<<"** >> user">>, >> >> <<"email">>,<<"email">>,253,<<**>>,33,132,0,0}], >> [[1, >> <<231,138,185,230,157,176>>, >> >> <<"237689098736789870">>,1,32,**<<"13761975289">>, >> <<"you.jie@REDACTED >> "**>>]], >> <<>>} >> >> >> >> On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin >> **> >> wrote: >> >> Barco, it means that you need to read manual for >> mysql server and >> enable socket listener. >> >> >> >> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> > >> http://erlang.org/mailman/**listinfo/erlang-questions >> >> >> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> >> > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Fri Jul 5 17:43:29 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 5 Jul 2013 17:43:29 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 4:27 PM, Garrett Smith wrote: > > At first glance, it looks like I'd be losing the connect timeout > enhancements from here: > > https://github.com/gar1t/Emysql > > Yes, we are currently missing those changes in the code base. I only took branches which were updated fairly recently on github, and yours were overlooked for some reason. I like most of the changes you have in there and most of them definitely should be part of the full release. I also agree that your functional change to the gen_tcp:recv timeout is something we want to fix. The only slight worry is that it may be a too big behavioral change for some people. The other changes are changes I want as well, so i'll probably pull them unless you want to prepare a PR yourself. I guess the initial reason for having the TCP timeout was to catch slow queries and get rid of them quickly, rather than waiting around. This makes a lot of sense in an environment where you have a large amount of small queries and are not allowed to wait around. But for larger queries and procedure calls, this is definitely not what one wants. The whole decoder loop looks weird as well. It should be possible to make it smarter than what is it currently, but this is a future change. Just drop me a note if you want to push it yourself or I should get to work. But it won't be this weekend, so I'll only be able to look at this Monday at the earliest, perhaps late Sunday. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Jul 5 17:43:21 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 05 Jul 2013 17:43:21 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: <51D6E597.8030903@ninenines.eu> Message-ID: <51D6E999.10907@ninenines.eu> It depends on the accept frequency you need. Measure. The examples and guide use 100 which work good enough in most cases but there's no optimal value for everyone. On 07/05/2013 05:42 PM, Barco You wrote: > Then what should be a good number of acceptors and max_connections? > depending on hardware? Can have 1 acceptor and thousands of > max_connections? Is it only max_connections that determines the concurrency? > > On 5 Jul 2013 23:26, "Lo?c Hoguin" > wrote: > > On 07/05/2013 03:34 PM, Barco You wrote: > > Thank you all. > > Whats the maximum connections can it bear when add a pool? If I use > Cowboy to provide a REST API for data query and I start up the http > listener with 10000 acceptors and 10000 emysql connections, does > that > mean i can support 10000 concurrency? > > > Not answering your question. > > Acceptors aren't the maximum number of connections, it's the > processes doing the actual accept. They're only configurable to > allow tweaking performance, and with 10000 you will accept slower, > that's too much. The setting you probably meant is max_connections. > > On 5 Jul 2013 14:35, "Mike Oxford" > >> wrote: > > emysql_util also has some other methods ... as_record/1, > as_proplist/1, as_dict/1 > Those will clean up the raw output into something far more > usable/accessible. > > -mox > > On Thu, Jul 4, 2013 at 2:12 AM, Barco You > > >> wrote: > > Where can I find the detailed description about the > fields for > result_packet? > > The result_pakcet looks like {result_packet, Number, List1, > List2, <<>>}, and seemingly only List2 is meaningful. > Why is > there Number, List1 and <<>> ? > > > On Thu, Jul 4, 2013 at 4:37 PM, Barco You > > >> wrote: > > What is this part as following for? > > > [{field,2,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"userid">>,<<"userid">>,8,<<__>>,63,20,16899,0}, > > > {field,3,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"name">>,<<"name">>,254,<<>>__,33,36,20489,0}, > > > {field,4,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"id_no">>,<<"id_no">>,254,<<__>>,33,54,20481,0}, > > > {field,5,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"sex">>,<<"sex">>,1,<<>>,63,__1,0,0}, > > > {field,6,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"age">>,<<"age">>,2,<<>>,63,__6,0,0}, > > > {field,7,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"tel">>,<<"tel">>,254,<<>>,__33,33,0,0}, > > > {field,8,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"email">>,<<"email">>,253,<<__>>,33,132,0,0}], > > > On Thu, Jul 4, 2013 at 4:35 PM, Barco You > > >> wrote: > > Solved, the bind-address of mhysql should be > changed to > 0.0.0.0 > > But the result packet of emysql:execute looks very > verbose and fuzzy. How can I get only the clean > field > value or a row, like {field, value} ? > > emysql:execute(db_pool, <<"select * from user > where tel > = 13761975289">>). > {result_packet,11, > > > [{field,2,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > > <<"userid">>,<<"userid">>,8,<<__>>,63,20,16899,0}, > > > {field,3,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"name">>,<<"name">>,254,<<>>__,33,36,20489,0}, > > > {field,4,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > > <<"id_no">>,<<"id_no">>,254,<<__>>,33,54,20481,0}, > > > {field,5,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"sex">>,<<"sex">>,1,<<>>,63,__1,0,0}, > > > {field,6,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"age">>,<<"age">>,2,<<>>,63,__6,0,0}, > > > {field,7,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"tel">>,<<"tel">>,254,<<>>,__33,33,0,0}, > > > {field,8,<<"def">>,<<"__fangchan">>,<<"user">>,<<"__user">>, > > <<"email">>,<<"email">>,253,<<__>>,33,132,0,0}], > [[1, > <<231,138,185,230,157,176>>, > > > <<"237689098736789870">>,1,32,__<<"13761975289">>, > <<"you.jie@REDACTED > > >"__>>]], > <<>>} > > > > On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin > >__> > wrote: > > Barco, it means that you need to read > manual for > mysql server and > enable socket listener. > > > > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From g@REDACTED Fri Jul 5 18:52:43 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 5 Jul 2013 11:52:43 -0500 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: <51D6E597.8030903@ninenines.eu> Message-ID: I offhand can't think of an example of where the answer "it depends" is more applicable :) Here's how I approach this problem: - Start by limiting the number of connections to the smallest reasonable number - this usually falls out as the number of client threads (sequential processing, so e.g. Erlang processes, or posix threads in other environments) - As your concurrency requirements increase -- and you start running into disk IO bottlenecks on MySQL -- look to add memory and modify your my.cnf to take advantage of it As Loic said, this requires measuring and understanding performance bottlenecks. In loaded scenarios this unfortunately gets quite complicated. There's a small industry around MySQL performance optimization. At the 30K foot level, you're dealing with a shared disk problem -- and all else held constant, throwing more concurrent access at a disk is bad idea. Memory/buffers will insulate you from the IO bottleneck, but with trade offs (CPU cost, cache warming, db crash recovery). Move down 15K feet and you're looking at the problems of sharing data -- databases, tables, rows, etc. and the complexity of transactions -- and that gets nasty complicated. On Fri, Jul 5, 2013 at 10:42 AM, Barco You wrote: > Then what should be a good number of acceptors and max_connections? > depending on hardware? Can have 1 acceptor and thousands of > max_connections? Is it only max_connections that determines the concurrency? > > On 5 Jul 2013 23:26, "Lo?c Hoguin" wrote: >> >> On 07/05/2013 03:34 PM, Barco You wrote: >>> >>> Thank you all. >>> >>> Whats the maximum connections can it bear when add a pool? If I use >>> Cowboy to provide a REST API for data query and I start up the http >>> listener with 10000 acceptors and 10000 emysql connections, does that >>> mean i can support 10000 concurrency? >> >> >> Not answering your question. >> >> Acceptors aren't the maximum number of connections, it's the processes >> doing the actual accept. They're only configurable to allow tweaking >> performance, and with 10000 you will accept slower, that's too much. The >> setting you probably meant is max_connections. >> >>> On 5 Jul 2013 14:35, "Mike Oxford" >> > wrote: >>> >>> emysql_util also has some other methods ... as_record/1, >>> as_proplist/1, as_dict/1 >>> Those will clean up the raw output into something far more >>> usable/accessible. >>> >>> -mox >>> >>> On Thu, Jul 4, 2013 at 2:12 AM, Barco You >> > wrote: >>> >>> Where can I find the detailed description about the fields for >>> result_packet? >>> >>> The result_pakcet looks like {result_packet, Number, List1, >>> List2, <<>>}, and seemingly only List2 is meaningful. Why is >>> there Number, List1 and <<>> ? >>> >>> >>> On Thu, Jul 4, 2013 at 4:37 PM, Barco You >> > wrote: >>> >>> What is this part as following for? >>> >>> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >>> >>> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >>> >>> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >>> >>> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >>> >>> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >>> >>> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >>> >>> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >>> >>> >>> On Thu, Jul 4, 2013 at 4:35 PM, Barco You >>> > wrote: >>> >>> Solved, the bind-address of mhysql should be changed to >>> 0.0.0.0 >>> >>> But the result packet of emysql:execute looks very >>> verbose and fuzzy. How can I get only the clean field >>> value or a row, like {field, value} ? >>> >>> emysql:execute(db_pool, <<"select * from user where tel >>> = 13761975289">>). >>> {result_packet,11, >>> >>> >>> [{field,2,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"userid">>,<<"userid">>,8,<<>>,63,20,16899,0}, >>> >>> {field,3,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"name">>,<<"name">>,254,<<>>,33,36,20489,0}, >>> >>> {field,4,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"id_no">>,<<"id_no">>,254,<<>>,33,54,20481,0}, >>> >>> {field,5,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"sex">>,<<"sex">>,1,<<>>,63,1,0,0}, >>> >>> {field,6,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"age">>,<<"age">>,2,<<>>,63,6,0,0}, >>> >>> {field,7,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"tel">>,<<"tel">>,254,<<>>,33,33,0,0}, >>> >>> {field,8,<<"def">>,<<"fangchan">>,<<"user">>,<<"user">>, >>> >>> <<"email">>,<<"email">>,253,<<>>,33,132,0,0}], >>> [[1, >>> <<231,138,185,230,157,176>>, >>> >>> <<"237689098736789870">>,1,32,<<"13761975289">>, >>> <<"you.jie@REDACTED >>> ">>]], >>> <<>>} >>> >>> >>> >>> On Thu, Jul 4, 2013 at 4:21 PM, Max Lapshin >>> > >>> wrote: >>> >>> Barco, it means that you need to read manual for >>> mysql server and >>> enable socket listener. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> -- >> Lo?c Hoguin >> >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From gleber.p@REDACTED Fri Jul 5 19:00:30 2013 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 5 Jul 2013 19:00:30 +0200 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> Message-ID: If it was an intermittent network issue, TCP can mask the problem and Erlang would never know about it. And I believe Erlang depends on TCP and explicit pings to detect dead nodes. But if remote failed node has been restarted in mean time, Erlang will detect it as, IIRC, it maintains some kind of "node version" in it's distribution protocol state. On Fri, Jul 5, 2013 at 5:24 PM, Sergej Jurecko wrote: > Well yes erlang does reconnect, but you still get a nodedown/nodeup message no? > > > Sergej > > On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: > >> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. >> >> Cheers >> Tim >> >> On 5 Jul 2013, at 15:49, Jonathan Leivent wrote: >> >>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? Or, is there some kind of internal persistent state that prevents this? >>> >>> If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node. >>> >>> -- Jonathan >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From g@REDACTED Fri Jul 5 19:03:59 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 5 Jul 2013 12:03:59 -0500 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: On Fri, Jul 5, 2013 at 10:43 AM, Jesper Louis Andersen wrote: > On Fri, Jul 5, 2013 at 4:27 PM, Garrett Smith wrote: >> >> >> At first glance, it looks like I'd be losing the connect timeout >> enhancements from here: >> >> https://github.com/gar1t/Emysql >> > > Yes, we are currently missing those changes in the code base. I only took > branches which were updated fairly recently on github, and yours were > overlooked for some reason. > > I like most of the changes you have in there and most of them definitely > should be part of the full release. I also agree that your functional change > to the gen_tcp:recv timeout is something we want to fix. The only slight > worry is that it may be a too big behavioral change for some people. The > other changes are changes I want as well, so i'll probably pull them unless > you want to prepare a PR yourself. I'll have to look again, but the intent was that default behavior is unchanged. > I guess the initial reason for having the TCP timeout was to catch slow > queries and get rid of them quickly, rather than waiting around. This makes > a lot of sense in an environment where you have a large amount of small > queries and are not allowed to wait around. But for larger queries and > procedure calls, this is definitely not what one wants. A default timeout other than infinity strikes me as a very bad idea. There are many reasons, but to your point in particular, a client close during a transaction will trigger a rollback, which can have profoundly negative effects on the system performance -- in many cases much worse then just waiting for the query to finish. Not that this should influence the library feature set, but I tend to deal with problematic long-running queries as an out-of-band problem. E.g. a separate monitor that looks for problem connections and kills them (could be a sys admin in the simplest case). > The whole decoder loop looks weird as well. It should be possible to make it > smarter than what is it currently, but this is a future change. > > Just drop me a note if you want to push it yourself or I should get to work. > But it won't be this weekend, so I'll only be able to look at this Monday at > the earliest, perhaps late Sunday. I can tackle this -- but it likely won't be within days :) From moxford@REDACTED Fri Jul 5 19:32:34 2013 From: moxford@REDACTED (Mike Oxford) Date: Fri, 5 Jul 2013 10:32:34 -0700 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> Message-ID: net_ticktime defines "how quickly" the nodes ping each other and, thus, notice that a node is down. http://www.erlang.org/doc/man/net_kernel.html#set_net_ticktime-1 -mox On Fri, Jul 5, 2013 at 10:00 AM, Gleb Peregud wrote: > If it was an intermittent network issue, TCP can mask the problem and > Erlang would never know about it. And I believe Erlang depends on TCP > and explicit pings to detect dead nodes. But if remote failed node has > been restarted in mean time, Erlang will detect it as, IIRC, it > maintains some kind of "node version" in it's distribution protocol > state. > > On Fri, Jul 5, 2013 at 5:24 PM, Sergej Jurecko > wrote: > > Well yes erlang does reconnect, but you still get a nodedown/nodeup > message no? > > > > > > Sergej > > > > On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: > > > >> As i understand it, this can and does happen, because erlang does > automatic reconnect in order to provide reliable communications. > >> > >> Cheers > >> Tim > >> > >> On 5 Jul 2013, at 15:49, Jonathan Leivent wrote: > >> > >>> In Erlang, is it possible for a monitored node to fail and recover so > quickly that nodes monitoring it won't detect the failure? Or, is there > some kind of internal persistent state that prevents this? > >>> > >>> If such quick fail/recover without detection cases are possible, I > think I have to roll my own monitoring scheme, and not rely on > erlang:monitor_node. > >>> > >>> -- Jonathan > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Jul 5 19:33:23 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 18:33:23 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> Message-ID: On 5 Jul 2013, at 18:00, Gleb Peregud wrote: > If it was an intermittent network issue, TCP can mask the problem and > Erlang would never know about it. And I believe Erlang depends on TCP > and explicit pings to detect dead nodes. But if remote failed node has > been restarted in mean time, Erlang will detect it as, IIRC, it > maintains some kind of "node version" in it's distribution protocol > state. > That's correct, so issues can arise if the node goes away and comes back within net_ticktime and TCP masks the fact. You can also run into problems with pid re-use in this way. But perhaps more importantly than this, if a node goes away and the runtime does notice, there's no guarantee that you'll see the 'DOWN' (or nodedown) message in any particular order with regards other communications flying around the system. This can make it difficult to identify which messages might potentially need re-sending. There are a couple of papers worth reading in this space: [1] Programming Distributed Erlang Applications: Pitfalls and Recipes (Hans Svensson, Lars-?ke Fredlund) [2] A Unified Semantics for Future Erlang (Hans Svensson, Lars-?ke Fredlund, Clara Benac Earle) Cheers, Tim From watson.timothy@REDACTED Fri Jul 5 19:35:43 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 18:35:43 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> Message-ID: <849D8475-1D69-42C5-B579-A8502F67BF31@gmail.com> On 5 Jul 2013, at 18:32, Mike Oxford wrote: > net_ticktime defines "how quickly" the nodes ping each other and, thus, notice that a node is down. > Sure, but you've got to be careful with that one. Setting the net_ticktime too short can lead to false positives, i.e., thinking nodes are down when they're simply busy, perhaps responding to some other party or struggling due to a busy_dist_port. Setting the net_ticktime too long might have the opposite effect. Another one worth a read in that space is "On Failure Detection Algorithms in Overlay Networks - Shelley Q. Zhuang, Dennis Geels, Ion Stoica, Randy H. Katz". Cheers, Tim From chandrashekhar.mullaparthi@REDACTED Fri Jul 5 20:58:43 2013 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Fri, 5 Jul 2013 19:58:43 +0100 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: How were these records written to the table in the first place? I've seen this happen when you write to the table without using mnesia_frag as the activity callback module. cheers Chandru On 5 July 2013 16:07, Noah Schwartz wrote: > I have a table of type set that worked fine as a non-fragmented table. We > started seeing performance issues with the table and with the anticipation > of more clients in the system, we were worried the size would exceed the 2 > GB max. > > The table contains chat messages and we purge records older than 7 days. > There seem to be a number of keys that don't show up in a read activity > but, do show up when doing a table dump, a select, or using qlc to find > records. As such, when this purge policy runs records are found that when I > try to delete don't actually get deleted. It almost seems like read/delete > are working off of one set while qlc is working off of another set. > > Code/outputs below. As you can see for a given key read returns nothing, > qlc returns an object, delete says it worked ok. Running the code again > gives the same results. Am I using the fragmentation API incorrectly > somehow? > > Thanks in advance > > *Code:* > -module(sel). > -include_lib("stdlib/include/qlc.hrl"). > -record(archive_message313, {owner_with_day_utc, owner, with, > day_gregorian_seconds, messages = []}). > -export([run/0]). > > run () -> > Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, > With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, > DayUtc = {2013, 6, 28}, > K = {Owner, With, DayUtc}, > > DelFun = fun () -> mnesia:delete({archive_message313, K}) end, > ReadFun = fun () -> mnesia:read({archive_message313, K}) end, > > QlcFun = fun () -> > Q = qlc:q([M || M <- mnesia:table(archive_message313), > M#archive_message313.owner_with_day_utc == K]), > qlc:eval(Q) > end, > > Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), > Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), > Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), > > {read_result, Read, qlc_result, Qlc, del_result, Del}. > > *Output:* > {read_result,[],qlc_result, > [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", > "dj.barker.xxx.com"}, > {"d735280e-d263-4e2b-beff-ed8ccaca5535", > "dj.barker.xxx.com"}, > {2013,6,28}}, > {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", > "dj.barker.xxx.com"}, > {"d735280e-d263-4e2b-beff-ed8ccaca5535", > "dj.barker.xxx.com"}, > 63539596800, > [{{{2013,6,28},{13,36,34}}, > {xmlelement,"forwarded", > > [{"xmlns","urn:xmpp:forward:0"}], > [{xmlelement,"delay", > > [{"xmlns","urn:xmpp:delay"}, > > {"stamp","2013-06-28T13:36:34.000000Z"}], > []}, > {xmlelement,"message", > [{"to", > " > 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, > {"from", > " > d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, > > {"type","chat"}], > > [{xmlelement,"body",[],[...]}]}]}, > 1372426594.889164}, > {{{2013,6,28},{13,36,35}}, > {xmlelement,"forwarded", > > [{"xmlns","urn:xmpp:forward:0"}], > [{xmlelement,"delay", > > [{"xmlns","urn:xmpp:delay"}, > > {"stamp","2013-06-28T13:36:35.000000Z"}], > []}, > {xmlelement,"message", > [{"to", > " > 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, > {"from", > " > d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, > > {"type",[...]}], > > [{xmlelement,"body",[],...}]}]}, > 1372426595.124266}, > {{{2013,6,28},{13,36,35}}, > {xmlelement,"forwarded", > > [{"xmlns","urn:xmpp:forward:0"}], > [{xmlelement,"delay", > > [{"xmlns","urn:xmpp:delay"}, > > {"stamp","2013-06-28T13:36:35.000000Z"}], > []}, > {xmlelement,"message", > [{"to", > " > 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, > > {"from",[...]}, > {[...],...}], > > [{xmlelement,[...],...}]}]}, > 1372426595.337535}]}], > del_result,ok} > -- > Noah > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jleivent@REDACTED Fri Jul 5 21:10:28 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Fri, 05 Jul 2013 15:10:28 -0400 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> Message-ID: <51D71A24.7090604@comcast.net> On 07/05/2013 01:33 PM, Tim Watson wrote: > On 5 Jul 2013, at 18:00, Gleb Peregud wrote: > >> If it was an intermittent network issue, TCP can mask the problem and >> Erlang would never know about it. And I believe Erlang depends on TCP >> and explicit pings to detect dead nodes. But if remote failed node has >> been restarted in mean time, Erlang will detect it as, IIRC, it >> maintains some kind of "node version" in it's distribution protocol >> state. >> > > That's correct, so issues can arise if the node goes away and comes back within net_ticktime and TCP masks the fact. You can also run into problems with pid re-use in this way. > > But perhaps more importantly than this, if a node goes away and the runtime does notice, there's no guarantee that you'll see the 'DOWN' (or nodedown) message in any particular order with regards other communications flying around the system. This can make it difficult to identify which messages might potentially need re-sending. Is this even true for net_kernel:monitor_nodes? The doc suggests that there is an ordering between messages and nodeup/nodedown notifications, at least in later releases. > There are a couple of papers worth reading in this space: > > [1] Programming Distributed Erlang Applications: Pitfalls and Recipes (Hans Svensson, Lars-?ke Fredlund) > [2] A Unified Semantics for Future Erlang (Hans Svensson, Lars-?ke Fredlund, Clara Benac Earle) > Thanks - I just skimmed the first paper. I don't have access to the second. -- Jonathan From chandrashekhar.mullaparthi@REDACTED Fri Jul 5 21:12:40 2013 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Fri, 5 Jul 2013 20:12:40 +0100 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: Sounds like the redistribution of data among fragments hasn't happened for whatever reason. One way to fix this is as follows: - Take a list of all the keys (assuming there aren't millions of them) - Figure out if it is in the correct fragment using functions in mnesia_frag_hash.erl - If not, read the record, delete it from the appropriate fragment, and re-insert If taking a list of all the keys isn't practical, I suggest: - take a backup - write a script which will traverses the backup and extracts all the keys which are in the wrong fragments - write another script to move said records from wrong fragment to the correct one cheers Chandru On 5 July 2013 20:03, Noah Schwartz wrote: > This was originally an un-fragmented table that I converted to a > fragmented table. I converted by calling change_table_frag with active and > then adding my frags. > > > On Fri, Jul 5, 2013 at 2:58 PM, Chandru < > chandrashekhar.mullaparthi@REDACTED> wrote: > >> How were these records written to the table in the first place? I've seen >> this happen when you write to the table without using mnesia_frag as the >> activity callback module. >> >> cheers >> Chandru >> >> On 5 July 2013 16:07, Noah Schwartz wrote: >> >>> I have a table of type set that worked fine as a non-fragmented table. >>> We started seeing performance issues with the table and with the >>> anticipation of more clients in the system, we were worried the size would >>> exceed the 2 GB max. >>> >>> The table contains chat messages and we purge records older than 7 days. >>> There seem to be a number of keys that don't show up in a read activity >>> but, do show up when doing a table dump, a select, or using qlc to find >>> records. As such, when this purge policy runs records are found that when I >>> try to delete don't actually get deleted. It almost seems like read/delete >>> are working off of one set while qlc is working off of another set. >>> >>> Code/outputs below. As you can see for a given key read returns nothing, >>> qlc returns an object, delete says it worked ok. Running the code again >>> gives the same results. Am I using the fragmentation API incorrectly >>> somehow? >>> >>> Thanks in advance >>> >>> *Code:* >>> -module(sel). >>> -include_lib("stdlib/include/qlc.hrl"). >>> -record(archive_message313, {owner_with_day_utc, owner, with, >>> day_gregorian_seconds, messages = []}). >>> -export([run/0]). >>> >>> run () -> >>> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, >>> With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, >>> DayUtc = {2013, 6, 28}, >>> K = {Owner, With, DayUtc}, >>> >>> DelFun = fun () -> mnesia:delete({archive_message313, K}) end, >>> ReadFun = fun () -> mnesia:read({archive_message313, K}) end, >>> >>> QlcFun = fun () -> >>> Q = qlc:q([M || M <- mnesia:table(archive_message313), >>> M#archive_message313.owner_with_day_utc == K]), >>> qlc:eval(Q) >>> end, >>> >>> Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), >>> Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), >>> Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), >>> >>> {read_result, Read, qlc_result, Qlc, del_result, Del}. >>> >>> *Output:* >>> {read_result,[],qlc_result, >>> >>> [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>> "dj.barker.xxx.com"}, >>> >>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>> "dj.barker.xxx.com"}, >>> {2013,6,28}}, >>> >>> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>> "dj.barker.xxx.com"}, >>> >>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>> "dj.barker.xxx.com"}, >>> 63539596800, >>> [{{{2013,6,28},{13,36,34}}, >>> {xmlelement,"forwarded", >>> >>> [{"xmlns","urn:xmpp:forward:0"}], >>> [{xmlelement,"delay", >>> >>> [{"xmlns","urn:xmpp:delay"}, >>> >>> {"stamp","2013-06-28T13:36:34.000000Z"}], >>> []}, >>> {xmlelement,"message", >>> [{"to", >>> " >>> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>> {"from", >>> " >>> d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>> >>> {"type","chat"}], >>> >>> [{xmlelement,"body",[],[...]}]}]}, >>> 1372426594.889164}, >>> {{{2013,6,28},{13,36,35}}, >>> {xmlelement,"forwarded", >>> >>> [{"xmlns","urn:xmpp:forward:0"}], >>> [{xmlelement,"delay", >>> >>> [{"xmlns","urn:xmpp:delay"}, >>> >>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>> []}, >>> {xmlelement,"message", >>> [{"to", >>> " >>> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>> {"from", >>> " >>> d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>> >>> {"type",[...]}], >>> >>> [{xmlelement,"body",[],...}]}]}, >>> 1372426595.124266}, >>> {{{2013,6,28},{13,36,35}}, >>> {xmlelement,"forwarded", >>> >>> [{"xmlns","urn:xmpp:forward:0"}], >>> [{xmlelement,"delay", >>> >>> [{"xmlns","urn:xmpp:delay"}, >>> >>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>> []}, >>> {xmlelement,"message", >>> [{"to", >>> " >>> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>> >>> {"from",[...]}, >>> >>> {[...],...}], >>> >>> [{xmlelement,[...],...}]}]}, >>> 1372426595.337535}]}], >>> del_result,ok} >>> -- >>> Noah >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > -- > Noah > -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Jul 5 21:21:06 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 20:21:06 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <51D71A24.7090604@comcast.net> References: <51D6DCEA.7080705@comcast.net> <40869834-0601-4294-B978-4B2FAECC81D9@gmail.com> <51D71A24.7090604@comcast.net> Message-ID: <0490FBA9-C4DB-4AA0-B867-74AEC1426CF7@gmail.com> On 5 Jul 2013, at 20:10, Jonathan Leivent wrote: > On 07/05/2013 01:33 PM, Tim Watson wrote: >> On 5 Jul 2013, at 18:00, Gleb Peregud wrote: >> >>> If it was an intermittent network issue, TCP can mask the problem and >>> Erlang would never know about it. And I believe Erlang depends on TCP >>> and explicit pings to detect dead nodes. But if remote failed node has >>> been restarted in mean time, Erlang will detect it as, IIRC, it >>> maintains some kind of "node version" in it's distribution protocol >>> state. >>> >> >> That's correct, so issues can arise if the node goes away and comes back within net_ticktime and TCP masks the fact. You can also run into problems with pid re-use in this way. >> >> But perhaps more importantly than this, if a node goes away and the runtime does notice, there's no guarantee that you'll see the 'DOWN' (or nodedown) message in any particular order with regards other communications flying around the system. This can make it difficult to identify which messages might potentially need re-sending. > > Is this even true for net_kernel:monitor_nodes? The doc suggests that there is an ordering between messages and nodeup/nodedown notifications, at least in later releases. > That ordering guarantee only applies to the process that called monitor_nodes, and afaict it won't apply unless you're using (what the runtime deems to be) a new connection. If you get hit by the situation where the runtime doesn't notice the node has gone down, that's a different matter - as Gleb mentioned, TCP can mask a disconnect/reconnect and net_ticktime isn't instantaneous (for obvious reasons). >> There are a couple of papers worth reading in this space: >> >> [1] Programming Distributed Erlang Applications: Pitfalls and Recipes (Hans Svensson, Lars-?ke Fredlund) >> [2] A Unified Semantics for Future Erlang (Hans Svensson, Lars-?ke Fredlund, Clara Benac Earle) >> > > Thanks - I just skimmed the first paper. I don't have access to the second. You'll need an ACM (or similar) account. The first one's more relevant anyway. Cheers, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From per@REDACTED Fri Jul 5 21:23:35 2013 From: per@REDACTED (Per Hedeland) Date: Fri, 5 Jul 2013 21:23:35 +0200 (CEST) Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: Message-ID: <201307051923.r65JNZYC009968@pluto.hedeland.org> Gleb Peregud wrote: > >If it was an intermittent network issue, TCP can mask the problem and >Erlang would never know about it. This is a completely different (non-)problem, the question was about a node failing and recovering. If both nodes keep running and there is an "intermittent network issue" that "TCP can mask", there *is* no problem - TCP deals with it, that's arguably its main purpose in life. >On Fri, Jul 5, 2013 at 5:24 PM, Sergej Jurecko wrote: >> Well yes erlang does reconnect, but you still get a nodedown/nodeup message no? Yes, and 'DOWN' for the process(es). But in both cases only if you monitor (or link => exit signal). >> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: >> >>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. No. >>> On 5 Jul 2013, at 15:49, Jonathan Leivent wrote: >>> >>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? No. The TCP connection to the old node instance cannot be used for communication with the new node instance, i.e. there is no way that communication with the new node instance can be established without the local VM generating node_down/'DOWN'/exit messages for the old instance. >>>> Or, is there some kind of internal persistent state that prevents this? This is where it potentially gets interesting - i.e. assuming *no* monitoring or linking - and that's where the "creation" part of a node identifier comes into play. If a distributed node restarts, it will get a new "creation" value courtesy of epmd, and any any pid() values referring to the old node instance will be invalid. However the "creation" is only 2 bits, so if a node restarts frequently, old pid() values may become "valid" once again, i.e. referring to a new, different process. Is it a problem in practice, outside academic research papers? I've never heard of such a case. If you really have this problem, you should probably look at why the node keeps restarting... --Per Hedeland From watson.timothy@REDACTED Fri Jul 5 21:33:45 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 20:33:45 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307051923.r65JNZYC009968@pluto.hedeland.org> References: <201307051923.r65JNZYC009968@pluto.hedeland.org> Message-ID: Thanks for the clarifications Per - that's cleared up a few things that I was unaware of. On 5 Jul 2013, at 20:23, Per Hedeland wrote: >>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: >>> >>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. > > No. So is the Svensson and Frelund paper (viz [2] from my earlier post) incorrect in its assertion that messages between nodes can be dropped in the face of rapid node reconnects? >>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? > > No. The TCP connection to the old node instance cannot be used for > communication with the new node instance, i.e. there is no way that > communication with the new node instance can be established without the > local VM generating node_down/'DOWN'/exit messages for the old instance. > Just out of interest, is this enforced by epmd or internally? Also, it would be worth making this explicit in the documentation somewhere, since this question comes up frequently. >>>>> Or, is there some kind of internal persistent state that prevents this? > > This is where it potentially gets interesting - i.e. assuming *no* > monitoring or linking - and that's where the "creation" part of a node > identifier comes into play. If a distributed node restarts, it will get > a new "creation" value courtesy of epmd, and any any pid() values > referring to the old node instance will be invalid. > Does this depend on epmd having stayed up and running the whole time, or does epmd now have some local persistent state? Cheers, Tim From jleivent@REDACTED Fri Jul 5 22:03:13 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Fri, 05 Jul 2013 16:03:13 -0400 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307051923.r65JNZYC009968@pluto.hedeland.org> References: <201307051923.r65JNZYC009968@pluto.hedeland.org> Message-ID: <51D72681.8010805@comcast.net> >>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? > > No. The TCP connection to the old node instance cannot be used for > communication with the new node instance, i.e. there is no way that > communication with the new node instance can be established without the > local VM generating node_down/'DOWN'/exit messages for the old instance. OK - so it is TCP that saves the day. I'm not sure I want to be that dependent on TCP, though. But, for now, it's an convenient solution. > >>>>> Or, is there some kind of internal persistent state that prevents this? > > This is where it potentially gets interesting - i.e. assuming *no* > monitoring or linking - and that's where the "creation" part of a node > identifier comes into play. If a distributed node restarts, it will get > a new "creation" value courtesy of epmd, and any any pid() values > referring to the old node instance will be invalid. > > However the "creation" is only 2 bits, so if a node restarts frequently, > old pid() values may become "valid" once again, i.e. referring to a new, > different process. Is it a problem in practice, outside academic > research papers? I've never heard of such a case. If you really have > this problem, you should probably look at why the node keeps > restarting... > > --Per Hedeland That's interesting. I'll make sure I monitor a node before relying on pid values from that node. Thanks, Jonathan From per@REDACTED Fri Jul 5 22:07:05 2013 From: per@REDACTED (Per Hedeland) Date: Fri, 5 Jul 2013 22:07:05 +0200 (CEST) Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: Message-ID: <201307052007.r65K75gk010746@pluto.hedeland.org> Tim Watson wrote: > >On 5 Jul 2013, at 20:23, Per Hedeland wrote: >>>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: >>>> >>>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. >> >> No. > >So is the Svensson and Frelund paper (viz [2] from my earlier post) incorrect in its assertion that messages between nodes can be dropped in the face of rapid node reconnects? No, of course they can be dropped - if the destination node goes down, it's impossible to know whether a given, sent message was a) received at the remote *host*, b) received by the remote Erlang node, c) received by the remote Erlang process, d) processed by the remote Erlang process, or e) none of the above. But if you monitor/link (e.g. use gen_server:call()), you will know that "badness happened", and can take corrective action. "Re-sending only messages that need to be re-sent" is not possible in general, and this is not specific to Erlang distribution. See also http://www.erlang.org/faq/academic.html#id58000, which Matthias Lang was kind enough to write up in a nice form based on some ramblings of mine in the distant past. It could probably use s/link/monitor/, but the general principle holds. >>>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure? >> >> No. The TCP connection to the old node instance cannot be used for >> communication with the new node instance, i.e. there is no way that >> communication with the new node instance can be established without the >> local VM generating node_down/'DOWN'/exit messages for the old instance. >> > >Just out of interest, is this enforced by epmd or internally? epmd has no role in inter-node communication once the connection has been established. TCP enforces "cannot be used for ...". The VM/net_kernel will not make a new connection until it has decided that the old one isn't working any more, and at that point it will generate the node_down/'DOWN'/exit messages. >>>>>> Or, is there some kind of internal persistent state that prevents this? >> >> This is where it potentially gets interesting - i.e. assuming *no* >> monitoring or linking - and that's where the "creation" part of a node >> identifier comes into play. If a distributed node restarts, it will get >> a new "creation" value courtesy of epmd, and any any pid() values >> referring to the old node instance will be invalid. >> > >Does this depend on epmd having stayed up and running the whole time, or does epmd now have some local persistent state? Good point - it depends on epmd having stayed up and running, i.e. if the *host* reboots, there is a 25% possibility of the new node instance getting the same "creation" value. However, see the FAQ again - if your communication is critical, you can't depend on "creation" - it won't tell you about failures anyway. It's just a way to try to prevent that messages get delivered to the wrong process. --Per From per@REDACTED Fri Jul 5 22:31:51 2013 From: per@REDACTED (Per Hedeland) Date: Fri, 5 Jul 2013 22:31:51 +0200 (CEST) Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <51D72681.8010805@comcast.net> Message-ID: <201307052031.r65KVpoX011131@pluto.hedeland.org> Jonathan Leivent wrote: > >OK - so it is TCP that saves the day. I'm not sure I want to be that >dependent on TCP, though. I'm not sure how to interpret that statement. If you mean - "I want to be able to use something other than TCP" - fine, as long as it provides the same guarantees, and you can figure out how to hook it into the Erlang distribution. - "I want more reliability than TCP can give me" - again fine, assuming that you understand precisely what it is that TCP gives you, i.e. that it either fulfills its goal of providing an ordered, reliable byte stream or tells you about its failure to do so. See the FAQ entry I pointed to - and remember that the lack of an application-level ack is no guarantee that the message wasn't processed *at all*. - "I don't think TCP is reliable enough to be dependant on" - in that case I would suggest that you forget about the Internet and networking in general. --Per From jleivent@REDACTED Fri Jul 5 23:16:54 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Fri, 05 Jul 2013 17:16:54 -0400 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307052031.r65KVpoX011131@pluto.hedeland.org> References: <201307052031.r65KVpoX011131@pluto.hedeland.org> Message-ID: <51D737C6.1030104@comcast.net> On 07/05/2013 04:31 PM, Per Hedeland wrote: > Jonathan Leivent wrote: >> >> OK - so it is TCP that saves the day. I'm not sure I want to be that >> dependent on TCP, though. > > I'm not sure how to interpret that statement. If you mean > > - "I want to be able to use something other than TCP" - fine, as long as > it provides the same guarantees, and you can figure out how to hook it > into the Erlang distribution. I might want to use UDP in the future. Or, something that is just slightly more than UDP - such as providing traffic control, but not reliability. My app is supposed to be able to handle lost/duplicate/unordered messages on is own - so much of what TCP provides is redundant. From fritchie@REDACTED Fri Jul 5 23:26:00 2013 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 05 Jul 2013 16:26:00 -0500 Subject: [erlang-questions] OSCON 2013 In-Reply-To: Message of "Fri, 05 Jul 2013 09:04:29 CDT." Message-ID: <25697.1373059560@snookles.snookles.com> Hi, Garrett. I won't be attending, alas, but there should be several Basho folks there: Eric Redmond at the very least. I've heard rumors that Ryan Zezeski, Brett Hazen, and Bryan Fink will also be in town that week. Adron Hall lives in Portland as does Chris Tilt. -Scott From per@REDACTED Fri Jul 5 23:44:39 2013 From: per@REDACTED (Per Hedeland) Date: Fri, 5 Jul 2013 23:44:39 +0200 (CEST) Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <51D737C6.1030104@comcast.net> Message-ID: <201307052144.r65Liduf012922@pluto.hedeland.org> Jonathan Leivent wrote: > >On 07/05/2013 04:31 PM, Per Hedeland wrote: >> Jonathan Leivent wrote: >>> >>> OK - so it is TCP that saves the day. I'm not sure I want to be that >>> dependent on TCP, though. >> >> I'm not sure how to interpret that statement. If you mean >> >> - "I want to be able to use something other than TCP" - fine, as long as >> it provides the same guarantees, and you can figure out how to hook it >> into the Erlang distribution. > >I might want to use UDP in the future. Or, something that is just >slightly more than UDP - such as providing traffic control, but not >reliability. My app is supposed to be able to handle >lost/duplicate/unordered messages on is own - so much of what TCP >provides is redundant. So use UDP to send Erlang terms as binaries over plain sockets, gen_udp is a fine module. But don't try to implement Erlang distribution on top of UDP - some number of years from now you will at best have re-implemented TCP with characteristics that are significantly inferior to the real thing. Or you may have something that *isn't* Erlang distribution. --Per From lambdadmitry@REDACTED Fri Jul 5 23:51:28 2013 From: lambdadmitry@REDACTED (Dmitry Groshev) Date: Sat, 6 Jul 2013 01:51:28 +0400 Subject: [erlang-questions] [ANN] ecirca - fast mutable circular arrays for Erlang Message-ID: The project: https://github.com/band115/ecirca ecirca is a library for working with fast mutable circular arrays of integer values. It's NIF-based, heavily tested (Proper is an outstanding tool for testing you NIF-related stuff!) and it's in production for more than two years now. Features include: -store integers of different sizes (int16, int32, int64 minus 4 bits for special values described below); -quickly set them; -"push" values, removing ones at the end (the size of array is provided at it's creation and can't be changed later); -update values using aggregate functions, such as "min", "max", "sum" and "avg"; -store special values, represented as atoms of 2 different types: "weak" ones that can be rewritten by integer if the value is updated ("empty" is such atom) and "strong" ones that can't; -export an entire structure as an Erlang binary and load it back. -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Jul 5 23:52:49 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 5 Jul 2013 22:52:49 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307052007.r65K75gk010746@pluto.hedeland.org> References: <201307052007.r65K75gk010746@pluto.hedeland.org> Message-ID: <47399570-E40A-4BB9-A39B-C34DA9DE93AA@gmail.com> On 5 Jul 2013, at 21:07, Per Hedeland wrote: > > No, of course they can be dropped - if the destination node goes down, > it's impossible to know whether a given, sent message was a) received at > the remote *host*, b) received by the remote Erlang node, c) received by > the remote Erlang process, d) processed by the remote Erlang process, or > e) none of the above. But if you monitor/link (e.g. use > gen_server:call()), you will know that "badness happened", and can take > corrective action. "Re-sending only messages that need to be re-sent" is > not possible in general, and this is not specific to Erlang distribution. > Yeah I know about those issues. We were discussing it on another thread earlier today. :) > See also http://www.erlang.org/faq/academic.html#id58000, which Matthias > Lang was kind enough to write up in a nice form based on some ramblings > of mine in the distant past. It could probably use s/link/monitor/, but > the general principle holds. > I'll take a look, thanks. >> >> Just out of interest, is this enforced by epmd or internally? > > epmd has no role in inter-node communication once the connection has > been established. Thats what I thought. > TCP enforces "cannot be used for ...". The > VM/net_kernel will not make a new connection until it has decided that > the old one isn't working any more, and at that point it will generate > the node_down/'DOWN'/exit messages. So given that intermediate kit could be proxying the connection, it is /possible/ that the node was reset but the network layer didn't notice? I mention that because if it is the case, people should bear it in mind when designing their infrastructure. > there is a 25% possibility of the new node instance > getting the same "creation" value. I wasn't fully aware of this node generation tracking stuff. We see a lot of stale pids (discarding messages from Old incarnations of the node) in some deployments, but then we actually store pids in mnesia which makes things a bit more complicated viz getting the timing right between node down messages, updating the db and actually using the pids. Cheers Tim From noah.schwartz1@REDACTED Fri Jul 5 21:03:45 2013 From: noah.schwartz1@REDACTED (Noah Schwartz) Date: Fri, 5 Jul 2013 15:03:45 -0400 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: This was originally an un-fragmented table that I converted to a fragmented table. I converted by calling change_table_frag with active and then adding my frags. On Fri, Jul 5, 2013 at 2:58 PM, Chandru < chandrashekhar.mullaparthi@REDACTED> wrote: > How were these records written to the table in the first place? I've seen > this happen when you write to the table without using mnesia_frag as the > activity callback module. > > cheers > Chandru > > On 5 July 2013 16:07, Noah Schwartz wrote: > >> I have a table of type set that worked fine as a non-fragmented table. >> We started seeing performance issues with the table and with the >> anticipation of more clients in the system, we were worried the size would >> exceed the 2 GB max. >> >> The table contains chat messages and we purge records older than 7 days. >> There seem to be a number of keys that don't show up in a read activity >> but, do show up when doing a table dump, a select, or using qlc to find >> records. As such, when this purge policy runs records are found that when I >> try to delete don't actually get deleted. It almost seems like read/delete >> are working off of one set while qlc is working off of another set. >> >> Code/outputs below. As you can see for a given key read returns nothing, >> qlc returns an object, delete says it worked ok. Running the code again >> gives the same results. Am I using the fragmentation API incorrectly >> somehow? >> >> Thanks in advance >> >> *Code:* >> -module(sel). >> -include_lib("stdlib/include/qlc.hrl"). >> -record(archive_message313, {owner_with_day_utc, owner, with, >> day_gregorian_seconds, messages = []}). >> -export([run/0]). >> >> run () -> >> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, >> With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, >> DayUtc = {2013, 6, 28}, >> K = {Owner, With, DayUtc}, >> >> DelFun = fun () -> mnesia:delete({archive_message313, K}) end, >> ReadFun = fun () -> mnesia:read({archive_message313, K}) end, >> >> QlcFun = fun () -> >> Q = qlc:q([M || M <- mnesia:table(archive_message313), >> M#archive_message313.owner_with_day_utc == K]), >> qlc:eval(Q) >> end, >> >> Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), >> Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), >> Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), >> >> {read_result, Read, qlc_result, Qlc, del_result, Del}. >> >> *Output:* >> {read_result,[],qlc_result, >> >> [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >> "dj.barker.xxx.com"}, >> >> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >> "dj.barker.xxx.com"}, >> {2013,6,28}}, >> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >> "dj.barker.xxx.com"}, >> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >> "dj.barker.xxx.com"}, >> 63539596800, >> [{{{2013,6,28},{13,36,34}}, >> {xmlelement,"forwarded", >> >> [{"xmlns","urn:xmpp:forward:0"}], >> [{xmlelement,"delay", >> >> [{"xmlns","urn:xmpp:delay"}, >> >> {"stamp","2013-06-28T13:36:34.000000Z"}], >> []}, >> {xmlelement,"message", >> [{"to", >> " >> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >> {"from", >> " >> d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >> >> {"type","chat"}], >> >> [{xmlelement,"body",[],[...]}]}]}, >> 1372426594.889164}, >> {{{2013,6,28},{13,36,35}}, >> {xmlelement,"forwarded", >> >> [{"xmlns","urn:xmpp:forward:0"}], >> [{xmlelement,"delay", >> >> [{"xmlns","urn:xmpp:delay"}, >> >> {"stamp","2013-06-28T13:36:35.000000Z"}], >> []}, >> {xmlelement,"message", >> [{"to", >> " >> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >> {"from", >> " >> d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >> >> {"type",[...]}], >> >> [{xmlelement,"body",[],...}]}]}, >> 1372426595.124266}, >> {{{2013,6,28},{13,36,35}}, >> {xmlelement,"forwarded", >> >> [{"xmlns","urn:xmpp:forward:0"}], >> [{xmlelement,"delay", >> >> [{"xmlns","urn:xmpp:delay"}, >> >> {"stamp","2013-06-28T13:36:35.000000Z"}], >> []}, >> {xmlelement,"message", >> [{"to", >> " >> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >> >> {"from",[...]}, >> >> {[...],...}], >> >> [{xmlelement,[...],...}]}]}, >> 1372426595.337535}]}], >> del_result,ok} >> -- >> Noah >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- Noah -------------- next part -------------- An HTML attachment was scrubbed... URL: From jleivent@REDACTED Sat Jul 6 00:32:31 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Fri, 05 Jul 2013 18:32:31 -0400 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307052144.r65Liduf012922@pluto.hedeland.org> References: <201307052144.r65Liduf012922@pluto.hedeland.org> Message-ID: <51D7497F.5010008@comcast.net> On 07/05/2013 05:44 PM, Per Hedeland wrote: > Jonathan Leivent wrote: >> >> On 07/05/2013 04:31 PM, Per Hedeland wrote: >>> Jonathan Leivent wrote: >>>> >>>> OK - so it is TCP that saves the day. I'm not sure I want to be that >>>> dependent on TCP, though. >>> >>> I'm not sure how to interpret that statement. If you mean >>> >>> - "I want to be able to use something other than TCP" - fine, as long as >>> it provides the same guarantees, and you can figure out how to hook it >>> into the Erlang distribution. >> >> I might want to use UDP in the future. Or, something that is just >> slightly more than UDP - such as providing traffic control, but not >> reliability. My app is supposed to be able to handle >> lost/duplicate/unordered messages on is own - so much of what TCP >> provides is redundant. > > So use UDP to send Erlang terms as binaries over plain sockets, gen_udp > is a fine module. But don't try to implement Erlang distribution on top > of UDP - some number of years from now you will at best have > re-implemented TCP with characteristics that are significantly inferior > to the real thing. Or you may have something that *isn't* Erlang > distribution. > > --Per > The Erlang implementation is a reference implementation - so it will be translated elsewhere, including to places where TCP has too much overhead. I'm not trying to re-invent Erlang's distributed infrastructure for Erlang. But, I might want to demonstrate that the reference implementation, although in Erlang, can be ported elsewhere without also having to port much of Erlang's infrastructure with it. -- Jonathan From cmeiklejohn@REDACTED Sat Jul 6 02:30:15 2013 From: cmeiklejohn@REDACTED (Christopher Meiklejohn) Date: Fri, 5 Jul 2013 20:30:15 -0400 Subject: [erlang-questions] OSCON 2013 In-Reply-To: <25697.1373059560@snookles.snookles.com> References: <25697.1373059560@snookles.snookles.com> Message-ID: On Friday, July 5, 2013 at 5:26 PM, Scott Lystig Fritchie wrote: > Hi, Garrett. I won't be attending, alas, but there should be several > Basho folks there: Eric Redmond at the very least. I've heard rumors > that Ryan Zezeski, Brett Hazen, and Bryan Fink will also be in town that > week. Adron Hall lives in Portland as does Chris Tilt. I'll also be around that weekend, as I'm in summer school in Eugene, but will be returning to Portland for the weekend following OSCON. - Chris -- Christopher Meiklejohn Software Engineer Basho Technologies, Inc. From felixgallo@REDACTED Sat Jul 6 03:54:07 2013 From: felixgallo@REDACTED (Felix Gallo) Date: Fri, 5 Jul 2013 18:54:07 -0700 Subject: [erlang-questions] idiomatic large data sets; shared and unshared Message-ID: In my continuing quest to learn erlang, I'm trying to implement a 1000x1000 grid into which items may be placed or removed, with perhaps a millon items in placement at once, spread fairly evenly across the grid. No grid cell will hold more than a thousand things. In the future, this grid may be shared across multiple processes, but for now I'm content to just try to get it working for one. In the mutable world, you might structure it as a hashtable of hashes, or hashtable of sets etc., and use the coordinate tuple {x,y} as a key into the hash table, then mutate the list. addtocell(Coordinate, Value) and removefromcell(Coordinate, Value) are very small, easy, and extremely fast (sub-second for a million iterations of addtocell()). In erlang world, the naive approach of attempting to use a list of lists (and then a list of sets), and lists:keytake() worked for small grid dimensions (10x10) but ran for several minutes before I killed it when I moved up in grid size and tried to insert a million things, with no indication when it would ever finish. The slightly less naive approach of using ets worked for larger grid dimensions, but was also highly nonperformant (60s for a million iterations of addtocell()). The problem sort of has me stumped. I don't have the natural intuition of what I can trade away for more speed in order to get to the same order of magnitude as the mutable solutions. It may be that I'm doing one of those fabled things you are not ever supposed to do in Erlang, but on the other hand, it seems like network switches have scenarios under which you might have to access randomly into a shared large matrix. And then there's Riak, which appears to be at least moderately performant and which appears to be built around the idea of permitting random write access into a shared datastore of indeterminate arity. So knowing that the mutable solution is essentially always going to win just because it doesn't have to do copies, is it possible to get erlang into the same ballpark here? Or is the idiomatic solution to write a NIF? F. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Sat Jul 6 11:17:30 2013 From: donpedrothird@REDACTED (Evgeny M) Date: Sat, 6 Jul 2013 02:17:30 -0700 (PDT) Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: This may happen if you used dirty_write functions, even if you did this in activity with mnesia_frag callback ???????, 5 ???? 2013 ?., 19:07:35 UTC+4 ???????????? Noah Schwartz ???????: > > I have a table of type set that worked fine as a non-fragmented table. We > started seeing performance issues with the table and with the anticipation > of more clients in the system, we were worried the size would exceed the 2 > GB max. > > The table contains chat messages and we purge records older than 7 days. > There seem to be a number of keys that don't show up in a read activity > but, do show up when doing a table dump, a select, or using qlc to find > records. As such, when this purge policy runs records are found that when I > try to delete don't actually get deleted. It almost seems like read/delete > are working off of one set while qlc is working off of another set. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Sat Jul 6 14:55:44 2013 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Sat, 6 Jul 2013 13:55:44 +0100 Subject: [erlang-questions] idiomatic large data sets; shared and unshared In-Reply-To: References: Message-ID: On 6 July 2013 02:54, Felix Gallo wrote: > > In my continuing quest to learn erlang, I'm trying to implement a > 1000x1000 grid into which items may be placed or removed, with perhaps a > millon items in placement at once, spread fairly evenly across the grid. > No grid cell will hold more than a thousand things. In the future, this > grid may be shared across multiple processes, but for now I'm content to > just try to get it working for one. > > In the mutable world, you might structure it as a hashtable of hashes, or > hashtable of sets etc., and use the coordinate tuple {x,y} as a key into > the hash table, then mutate the list. addtocell(Coordinate, Value) and > removefromcell(Coordinate, Value) are very small, easy, and extremely fast > (sub-second for a million iterations of addtocell()). > > In erlang world, the naive approach of attempting to use a list of lists > (and then a list of sets), and lists:keytake() worked for small grid > dimensions (10x10) but ran for several minutes before I killed it when I > moved up in grid size and tried to insert a million things, with no > indication when it would ever finish. > > The slightly less naive approach of using ets worked for larger grid > dimensions, but was also highly nonperformant (60s for a million iterations > of addtocell()). > > If I use the cell-id (Row-index, Col-index) as the key for each row in an ETS table, I am able to insert a million cells in about 1.25 seconds. The timing is the same whether the ets table is of type set or bag. Have I misunderstood your problem? 16> grid:init_grid(set). <0.81.0> 17> grid:add_cells(1000, 1000). 1252287 18> ets:info(grid, size). 1000000 19> 19> grid ! stop. stop 20> 20> 20> grid:init_grid(bag). <0.86.0> 21> grid:add_cells(1000, 1000). 1248606 22> grid:add_cells(1000, 1000). 1247905 23> grid:add_cells(1000, 1000). 1345457 24> ets:info(grid, size). 3000000 %%------------------------ grid.erl -------------------- -module(grid). -compile(export_all). init_grid(Type) -> proc_lib:spawn(?MODULE, grid_owner, [Type]). grid_owner(Type) -> ets:new(grid, [named_table, public, Type]), register(grid, self()), receive stop -> ok end. add_cells(Num_rows, Num_cols) -> Start = now(), add_to_cell(Num_rows, Num_cols), End = now(), timer:now_diff(End, Start). add_to_cell(0, _) -> ok; add_to_cell(Row_num, Num_cols) -> add_to_row(Row_num, Num_cols), add_to_cell(Row_num - 1, Num_cols). add_to_row(_, 0) -> ok; add_to_row(Row_num, Col_num) -> ets:insert(grid, {{Row_num, Col_num}, os:timestamp()}), add_to_row(Row_num, Col_num - 1). -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah.schwartz1@REDACTED Sat Jul 6 15:15:43 2013 From: noah.schwartz1@REDACTED (Noah Schwartz) Date: Sat, 6 Jul 2013 09:15:43 -0400 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: Well, I've never written to this table before or after converting it to a fragmented table. When you say "this" can happen are you referring to the problem that Chandru said? That I've somehow ended up with records in the wrong fragment? Sent from my iPhone On Jul 6, 2013, at 5:17 AM, Evgeny M wrote: > > This may happen if you used dirty_write functions, even if you did this in activity with mnesia_frag callback > > ???????, 5 ???? 2013 ?., 19:07:35 UTC+4 ???????????? Noah Schwartz ???????: >> >> I have a table of type set that worked fine as a non-fragmented table. We started seeing performance issues with the table and with the anticipation of more clients in the system, we were worried the size would exceed the 2 GB max. >> >> The table contains chat messages and we purge records older than 7 days. There seem to be a number of keys that don't show up in a read activity but, do show up when doing a table dump, a select, or using qlc to find records. As such, when this purge policy runs records are found that when I try to delete don't actually get deleted. It almost seems like read/delete are working off of one set while qlc is working off of another set. >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah.schwartz1@REDACTED Sat Jul 6 15:17:39 2013 From: noah.schwartz1@REDACTED (Noah Schwartz) Date: Sat, 6 Jul 2013 09:17:39 -0400 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: I can give this a shot. So you think that at some point while adding fragments items got put into the wrong fragment causing the deletes/reads to fail? Qlc works since its essentially scanning the whole table? How can I avoid this problem when adding fragments in the future? Sent from my iPhone On Jul 5, 2013, at 3:12 PM, Chandru wrote: > Sounds like the redistribution of data among fragments hasn't happened for whatever reason. One way to fix this is as follows: > > - Take a list of all the keys (assuming there aren't millions of them) > - Figure out if it is in the correct fragment using functions in mnesia_frag_hash.erl > - If not, read the record, delete it from the appropriate fragment, and re-insert > > If taking a list of all the keys isn't practical, I suggest: > > - take a backup > - write a script which will traverses the backup and extracts all the keys which are in the wrong fragments > - write another script to move said records from wrong fragment to the correct one > > cheers > Chandru > > On 5 July 2013 20:03, Noah Schwartz wrote: >> This was originally an un-fragmented table that I converted to a fragmented table. I converted by calling change_table_frag with active and then adding my frags. >> >> >> On Fri, Jul 5, 2013 at 2:58 PM, Chandru wrote: >>> How were these records written to the table in the first place? I've seen this happen when you write to the table without using mnesia_frag as the activity callback module. >>> >>> cheers >>> Chandru >>> >>> On 5 July 2013 16:07, Noah Schwartz wrote: >>>> I have a table of type set that worked fine as a non-fragmented table. We started seeing performance issues with the table and with the anticipation of more clients in the system, we were worried the size would exceed the 2 GB max. >>>> >>>> The table contains chat messages and we purge records older than 7 days. There seem to be a number of keys that don't show up in a read activity but, do show up when doing a table dump, a select, or using qlc to find records. As such, when this purge policy runs records are found that when I try to delete don't actually get deleted. It almost seems like read/delete are working off of one set while qlc is working off of another set. >>>> >>>> Code/outputs below. As you can see for a given key read returns nothing, qlc returns an object, delete says it worked ok. Running the code again gives the same results. Am I using the fragmentation API incorrectly somehow? >>>> >>>> Thanks in advance >>>> >>>> Code: >>>> -module(sel). >>>> -include_lib("stdlib/include/qlc.hrl"). >>>> -record(archive_message313, {owner_with_day_utc, owner, with, day_gregorian_seconds, messages = []}). >>>> -export([run/0]). >>>> >>>> run () -> >>>> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, >>>> With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, >>>> DayUtc = {2013, 6, 28}, >>>> K = {Owner, With, DayUtc}, >>>> >>>> DelFun = fun () -> mnesia:delete({archive_message313, K}) end, >>>> ReadFun = fun () -> mnesia:read({archive_message313, K}) end, >>>> >>>> QlcFun = fun () -> >>>> Q = qlc:q([M || M <- mnesia:table(archive_message313), M#archive_message313.owner_with_day_utc == K]), >>>> qlc:eval(Q) >>>> end, >>>> >>>> Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), >>>> Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), >>>> Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), >>>> >>>> {read_result, Read, qlc_result, Qlc, del_result, Del}. >>>> >>>> Output: >>>> {read_result,[],qlc_result, >>>> [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>>> "dj.barker.xxx.com"}, >>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>>> "dj.barker.xxx.com"}, >>>> {2013,6,28}}, >>>> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>>> "dj.barker.xxx.com"}, >>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>>> "dj.barker.xxx.com"}, >>>> 63539596800, >>>> [{{{2013,6,28},{13,36,34}}, >>>> {xmlelement,"forwarded", >>>> [{"xmlns","urn:xmpp:forward:0"}], >>>> [{xmlelement,"delay", >>>> [{"xmlns","urn:xmpp:delay"}, >>>> {"stamp","2013-06-28T13:36:34.000000Z"}], >>>> []}, >>>> {xmlelement,"message", >>>> [{"to", >>>> "9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>> {"from", >>>> "d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>>> {"type","chat"}], >>>> [{xmlelement,"body",[],[...]}]}]}, >>>> 1372426594.889164}, >>>> {{{2013,6,28},{13,36,35}}, >>>> {xmlelement,"forwarded", >>>> [{"xmlns","urn:xmpp:forward:0"}], >>>> [{xmlelement,"delay", >>>> [{"xmlns","urn:xmpp:delay"}, >>>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>>> []}, >>>> {xmlelement,"message", >>>> [{"to", >>>> "9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>> {"from", >>>> "d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>>> {"type",[...]}], >>>> [{xmlelement,"body",[],...}]}]}, >>>> 1372426595.124266}, >>>> {{{2013,6,28},{13,36,35}}, >>>> {xmlelement,"forwarded", >>>> [{"xmlns","urn:xmpp:forward:0"}], >>>> [{xmlelement,"delay", >>>> [{"xmlns","urn:xmpp:delay"}, >>>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>>> []}, >>>> {xmlelement,"message", >>>> [{"to", >>>> "9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>> {"from",[...]}, >>>> {[...],...}], >>>> [{xmlelement,[...],...}]}]}, >>>> 1372426595.337535}]}], >>>> del_result,ok} >>>> -- >>>> Noah >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -- >> Noah > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandrashekhar.mullaparthi@REDACTED Sat Jul 6 15:56:03 2013 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Sat, 6 Jul 2013 14:56:03 +0100 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: On 6 July 2013 14:17, Noah Schwartz wrote: > I can give this a shot. So you think that at some point while adding > fragments items got put into the wrong fragment causing the deletes/reads > to fail? Qlc works since its essentially scanning the whole table? > Yes. > How can I avoid this problem when adding fragments in the future? > Not entirely sure. I've never relied on this feature myself because I didn't trust it to work properly! I just create a fragmented table, and repopulate the DB. On Jul 5, 2013, at 3:12 PM, Chandru wrote: Sounds like the redistribution of data among fragments hasn't happened for > whatever reason. One way to fix this is as follows: > > - Take a list of all the keys (assuming there aren't millions of them) > - Figure out if it is in the correct fragment using functions in > mnesia_frag_hash.erl > - If not, read the record, delete it from the appropriate fragment, and > re-insert > > If taking a list of all the keys isn't practical, I suggest: > > - take a backup > - write a script which will traverses the backup and extracts all the keys > which are in the wrong fragments > - write another script to move said records from wrong fragment to the > correct one > > cheers > Chandru > > On 5 July 2013 20:03, Noah Schwartz wrote: > >> This was originally an un-fragmented table that I converted to a >> fragmented table. I converted by calling change_table_frag with active and >> then adding my frags. >> >> >> On Fri, Jul 5, 2013 at 2:58 PM, Chandru < >> chandrashekhar.mullaparthi@REDACTED> wrote: >> >>> How were these records written to the table in the first place? I've >>> seen this happen when you write to the table without using mnesia_frag as >>> the activity callback module. >>> >>> cheers >>> Chandru >>> >>> On 5 July 2013 16:07, Noah Schwartz wrote: >>> >>>> I have a table of type set that worked fine as a non-fragmented >>>> table. We started seeing performance issues with the table and with the >>>> anticipation of more clients in the system, we were worried the size would >>>> exceed the 2 GB max. >>>> >>>> The table contains chat messages and we purge records older than 7 >>>> days. There seem to be a number of keys that don't show up in a read >>>> activity but, do show up when doing a table dump, a select, or using qlc to >>>> find records. As such, when this purge policy runs records are found that >>>> when I try to delete don't actually get deleted. It almost seems like >>>> read/delete are working off of one set while qlc is working off of another >>>> set. >>>> >>>> Code/outputs below. As you can see for a given key read returns >>>> nothing, qlc returns an object, delete says it worked ok. Running the code >>>> again gives the same results. Am I using the fragmentation API incorrectly >>>> somehow? >>>> >>>> Thanks in advance >>>> >>>> *Code:* >>>> -module(sel). >>>> -include_lib("stdlib/include/qlc.hrl"). >>>> -record(archive_message313, {owner_with_day_utc, owner, with, >>>> day_gregorian_seconds, messages = []}). >>>> -export([run/0]). >>>> >>>> run () -> >>>> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com >>>> "}, >>>> With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com >>>> "}, >>>> DayUtc = {2013, 6, 28}, >>>> K = {Owner, With, DayUtc}, >>>> >>>> DelFun = fun () -> mnesia:delete({archive_message313, K}) end, >>>> ReadFun = fun () -> mnesia:read({archive_message313, K}) end, >>>> >>>> QlcFun = fun () -> >>>> Q = qlc:q([M || M <- mnesia:table(archive_message313), >>>> M#archive_message313.owner_with_day_utc == K]), >>>> qlc:eval(Q) >>>> end, >>>> >>>> Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), >>>> Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), >>>> Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), >>>> >>>> {read_result, Read, qlc_result, Qlc, del_result, Del}. >>>> >>>> *Output:* >>>> {read_result,[],qlc_result, >>>> >>>> [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>>> "dj.barker.xxx.com"}, >>>> >>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>>> "dj.barker.xxx.com"}, >>>> {2013,6,28}}, >>>> >>>> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>>> "dj.barker.xxx.com"}, >>>> >>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>>> "dj.barker.xxx.com"}, >>>> 63539596800, >>>> [{{{2013,6,28},{13,36,34}}, >>>> {xmlelement,"forwarded", >>>> >>>> [{"xmlns","urn:xmpp:forward:0"}], >>>> [{xmlelement,"delay", >>>> >>>> [{"xmlns","urn:xmpp:delay"}, >>>> >>>> {"stamp","2013-06-28T13:36:34.000000Z"}], >>>> []}, >>>> {xmlelement,"message", >>>> [{"to", >>>> " >>>> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>> {"from", >>>> " >>>> d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>>> >>>> {"type","chat"}], >>>> >>>> [{xmlelement,"body",[],[...]}]}]}, >>>> 1372426594.889164}, >>>> {{{2013,6,28},{13,36,35}}, >>>> {xmlelement,"forwarded", >>>> >>>> [{"xmlns","urn:xmpp:forward:0"}], >>>> [{xmlelement,"delay", >>>> >>>> [{"xmlns","urn:xmpp:delay"}, >>>> >>>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>>> []}, >>>> {xmlelement,"message", >>>> [{"to", >>>> " >>>> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>> {"from", >>>> " >>>> d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>>> >>>> {"type",[...]}], >>>> >>>> [{xmlelement,"body",[],...}]}]}, >>>> 1372426595.124266}, >>>> {{{2013,6,28},{13,36,35}}, >>>> {xmlelement,"forwarded", >>>> >>>> [{"xmlns","urn:xmpp:forward:0"}], >>>> [{xmlelement,"delay", >>>> >>>> [{"xmlns","urn:xmpp:delay"}, >>>> >>>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>>> []}, >>>> {xmlelement,"message", >>>> [{"to", >>>> " >>>> 9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>> >>>> {"from",[...]}, >>>> >>>> {[...],...}], >>>> >>>> [{xmlelement,[...],...}]}]}, >>>> 1372426595.337535}]}], >>>> del_result,ok} >>>> -- >>>> Noah >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> >> -- >> Noah >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah.schwartz1@REDACTED Sat Jul 6 16:10:19 2013 From: noah.schwartz1@REDACTED (Noah Schwartz) Date: Sat, 6 Jul 2013 10:10:19 -0400 Subject: [erlang-questions] inconsistent results with fragmented table In-Reply-To: References: Message-ID: Gotcha. That's kind of concerning given that its possible we will need to add fragments in the future... I've also noticed that there's a primary key violation. I have the same exact record in two fragments which should be impossible. I've never done an operation on the fragments directly after converting the table. Sent from my iPhone On Jul 6, 2013, at 9:56 AM, Chandru wrote: > On 6 July 2013 14:17, Noah Schwartz wrote: >> I can give this a shot. So you think that at some point while adding fragments items got put into the wrong fragment causing the deletes/reads to fail? Qlc works since its essentially scanning the whole table? > > Yes. > >> How can I avoid this problem when adding fragments in the future? > > Not entirely sure. I've never relied on this feature myself because I didn't trust it to work properly! I just create a fragmented table, and repopulate the DB. > > On Jul 5, 2013, at 3:12 PM, Chandru wrote: > >>> Sounds like the redistribution of data among fragments hasn't happened for whatever reason. One way to fix this is as follows: >>> >>> - Take a list of all the keys (assuming there aren't millions of them) >>> - Figure out if it is in the correct fragment using functions in mnesia_frag_hash.erl >>> - If not, read the record, delete it from the appropriate fragment, and re-insert >>> >>> If taking a list of all the keys isn't practical, I suggest: >>> >>> - take a backup >>> - write a script which will traverses the backup and extracts all the keys which are in the wrong fragments >>> - write another script to move said records from wrong fragment to the correct one >>> >>> cheers >>> Chandru >>> >>> On 5 July 2013 20:03, Noah Schwartz wrote: >>>> This was originally an un-fragmented table that I converted to a fragmented table. I converted by calling change_table_frag with active and then adding my frags. >>>> >>>> >>>> On Fri, Jul 5, 2013 at 2:58 PM, Chandru wrote: >>>>> How were these records written to the table in the first place? I've seen this happen when you write to the table without using mnesia_frag as the activity callback module. >>>>> >>>>> cheers >>>>> Chandru >>>>> >>>>> On 5 July 2013 16:07, Noah Schwartz wrote: >>>>>> I have a table of type set that worked fine as a non-fragmented table. We started seeing performance issues with the table and with the anticipation of more clients in the system, we were worried the size would exceed the 2 GB max. >>>>>> >>>>>> The table contains chat messages and we purge records older than 7 days. There seem to be a number of keys that don't show up in a read activity but, do show up when doing a table dump, a select, or using qlc to find records. As such, when this purge policy runs records are found that when I try to delete don't actually get deleted. It almost seems like read/delete are working off of one set while qlc is working off of another set. >>>>>> >>>>>> Code/outputs below. As you can see for a given key read returns nothing, qlc returns an object, delete says it worked ok. Running the code again gives the same results. Am I using the fragmentation API incorrectly somehow? >>>>>> >>>>>> Thanks in advance >>>>>> >>>>>> Code: >>>>>> -module(sel). >>>>>> -include_lib("stdlib/include/qlc.hrl"). >>>>>> -record(archive_message313, {owner_with_day_utc, owner, with, day_gregorian_seconds, messages = []}). >>>>>> -export([run/0]). >>>>>> >>>>>> run () -> >>>>>> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"}, >>>>>> With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"}, >>>>>> DayUtc = {2013, 6, 28}, >>>>>> K = {Owner, With, DayUtc}, >>>>>> >>>>>> DelFun = fun () -> mnesia:delete({archive_message313, K}) end, >>>>>> ReadFun = fun () -> mnesia:read({archive_message313, K}) end, >>>>>> >>>>>> QlcFun = fun () -> >>>>>> Q = qlc:q([M || M <- mnesia:table(archive_message313), M#archive_message313.owner_with_day_utc == K]), >>>>>> qlc:eval(Q) >>>>>> end, >>>>>> >>>>>> Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag), >>>>>> Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag), >>>>>> Del = mnesia:activity(transaction, DelFun, [], mnesia_frag), >>>>>> >>>>>> {read_result, Read, qlc_result, Qlc, del_result, Del}. >>>>>> >>>>>> Output: >>>>>> {read_result,[],qlc_result, >>>>>> [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>>>>> "dj.barker.xxx.com"}, >>>>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>>>>> "dj.barker.xxx.com"}, >>>>>> {2013,6,28}}, >>>>>> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", >>>>>> "dj.barker.xxx.com"}, >>>>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535", >>>>>> "dj.barker.xxx.com"}, >>>>>> 63539596800, >>>>>> [{{{2013,6,28},{13,36,34}}, >>>>>> {xmlelement,"forwarded", >>>>>> [{"xmlns","urn:xmpp:forward:0"}], >>>>>> [{xmlelement,"delay", >>>>>> [{"xmlns","urn:xmpp:delay"}, >>>>>> {"stamp","2013-06-28T13:36:34.000000Z"}], >>>>>> []}, >>>>>> {xmlelement,"message", >>>>>> [{"to", >>>>>> "9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>>>> {"from", >>>>>> "d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>>>>> {"type","chat"}], >>>>>> [{xmlelement,"body",[],[...]}]}]}, >>>>>> 1372426594.889164}, >>>>>> {{{2013,6,28},{13,36,35}}, >>>>>> {xmlelement,"forwarded", >>>>>> [{"xmlns","urn:xmpp:forward:0"}], >>>>>> [{xmlelement,"delay", >>>>>> [{"xmlns","urn:xmpp:delay"}, >>>>>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>>>>> []}, >>>>>> {xmlelement,"message", >>>>>> [{"to", >>>>>> "9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>>>> {"from", >>>>>> "d735280e-d263-4e2b-beff-ed8ccaca5535@REDACTED"}, >>>>>> {"type",[...]}], >>>>>> [{xmlelement,"body",[],...}]}]}, >>>>>> 1372426595.124266}, >>>>>> {{{2013,6,28},{13,36,35}}, >>>>>> {xmlelement,"forwarded", >>>>>> [{"xmlns","urn:xmpp:forward:0"}], >>>>>> [{xmlelement,"delay", >>>>>> [{"xmlns","urn:xmpp:delay"}, >>>>>> {"stamp","2013-06-28T13:36:35.000000Z"}], >>>>>> []}, >>>>>> {xmlelement,"message", >>>>>> [{"to", >>>>>> "9afa8671-7e88-436c-9ad4-4cb13ad45e1e@REDACTED"}, >>>>>> {"from",[...]}, >>>>>> {[...],...}], >>>>>> [{xmlelement,[...],...}]}]}, >>>>>> 1372426595.337535}]}], >>>>>> del_result,ok} >>>>>> -- >>>>>> Noah >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>>> >>>> -- >>>> Noah >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sun Jul 7 04:59:40 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 7 Jul 2013 04:59:40 +0200 Subject: [erlang-questions] STUN server In-Reply-To: <51A94A2A.2020303@gmail.com> References: <51A94A2A.2020303@gmail.com> Message-ID: On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov wrote: > On 01.06.2013 01:00, Igor Karymov wrote: > >> You may take a look here: >> https://github.com/processone/**ejabberd/tree/master/src/stun >> >> > It's a separate project now: https://github.com/processone/**stun > > -- > > I'm just back on that topic. Thanks for the code. Unfortunately the licensee is too restrictive for my project so I will have to find another way to do that. Anyway just wanted to say thanks it can be useful for some. - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From per@REDACTED Sun Jul 7 15:36:10 2013 From: per@REDACTED (Per Hedeland) Date: Sun, 7 Jul 2013 15:36:10 +0200 (CEST) Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <47399570-E40A-4BB9-A39B-C34DA9DE93AA@gmail.com> Message-ID: <201307071336.r67DaAZl061044@pluto.hedeland.org> Tim Watson wrote: > >On 5 Jul 2013, at 21:07, Per Hedeland wrote: > >> TCP enforces "cannot be used for ...". The >> VM/net_kernel will not make a new connection until it has decided that >> the old one isn't working any more, and at that point it will generate >> the node_down/'DOWN'/exit messages. > >So given that intermediate kit could be proxying the connection, it is /possible/ that the node was reset but the network layer didn't notice? I mention that because if it is the case, people should bear it in mind when designing their infrastructure. I'm actually not sure what you're talking about here - AFAIK it's not common or even easily doable to "proxy" Erlang distribution connections. But if you somehow manage to do it for some reason, you need to make sure that the resulting connection has the same end-to-end semantics as TCP at least in the reliability department. I.e. if your proxy "hides" the fact that a remote node restarted, it is basically broken. --Per From michael.santos@REDACTED Sun Jul 7 15:37:23 2013 From: michael.santos@REDACTED (Michael Santos) Date: Sun, 7 Jul 2013 09:37:23 -0400 Subject: [erlang-questions] low level packet access from erlang In-Reply-To: <51D46966.8000107@research.att.com> References: <515C6E13.1050406@research.att.com> <20130403212407.GA25006@ioctl> <51ADE6E1.4030205@research.att.com> <20130604153054.GA31551@ioctl> <51AE7341.4090200@research.att.com> <51AE7D17.2050807@research.att.com> <20130605132308.GA32186@ioctl> <51D46966.8000107@research.att.com> Message-ID: <20130707133723.GA26612@ioctl> On Wed, Jul 03, 2013 at 02:11:50PM -0400, Garry Hodgson wrote: > On 06/05/2013 09:23 AM, Michael Santos wrote: > >Ask away and if you have any comments/suggestions, please let me know! > i'm happily using tun now to intercept and inspect packets, but Glad to hear you are making some progress! > any that i modify get dropped as malformed when i send them. > wireshark tells me the checksums are wrong. and indeed, i > don't get what i expect from pkt:checksum(): > > classify( <<4:4, _IHL:4, _TypeOfService:8, _TotalLength:16, > _Identification:16, _FlagX:1, _FlagD:1, _FlagM:1, > _FragmentOffset:13, > _TTL:8, ?IPPROTO_TCP:8, _HeaderCheckSum:16, > _SrcAddr:32, _DestAddr:32, _Rest/binary>> = Raw ) -> > > { IPv4, IpPayload } = pkt:ipv4( Raw ), > { Tcp, TcpPayload } = pkt:tcp( IpPayload ), > > TestSum = pkt:checksum( [ IPv4, Tcp, TcpPayload ] ), > alog:debug( "classify: computed = ~b, actual = ~b", [ TestSum, > Tcp#tcp.sum ] ), > ... > > =INFO REPORT==== 3-Jul-2013::14:04:02 === > {log,debug,"classify: computed = 43987, actual = 52256",r3@REDACTED} > > any idea what i'm doing wrong? > > thanks There were a few bugs: TCP options were left out of the record to binary conversion and the length of the payload was left out of the checksum. Should be fixed now, thanks for letting me know! Also, use pkt:makesum/1 to calculate the checksum. This should work: TestSum = pkt:makesum( [ IPv4, Tcp, TcpPayload ] ), You may still see some checksum mismatches if your system uses TCP checksum offloading. > -- > Garry Hodgson > AT&T Chief Security Office (CSO) > > "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." > From watson.timothy@REDACTED Sun Jul 7 16:44:11 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Sun, 7 Jul 2013 15:44:11 +0100 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307052007.r65K75gk010746@pluto.hedeland.org> References: <201307052007.r65K75gk010746@pluto.hedeland.org> Message-ID: <0D10896F-134C-4253-B810-81E9631D5816@gmail.com> On 5 Jul 2013, at 21:07, Per Hedeland wrote: > Tim Watson wrote: >> >> On 5 Jul 2013, at 20:23, Per Hedeland wrote: >>>>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote: >>>>> >>>>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications. >>> >>> No. >> >> So is the Svensson and Frelund paper (viz [2] from my earlier post) incorrect in its assertion that messages between nodes can be dropped in the face of rapid node reconnects? > > No, of course they can be dropped - if the destination node goes down, > it's impossible to know whether a given, sent message was a) received at > the remote *host*, b) received by the remote Erlang node, c) received by > the remote Erlang process, d) processed by the remote Erlang process, or > e) none of the above. But if you monitor/link (e.g. use > gen_server:call()), you will know that "badness happened", and can take > corrective action. Right, that makes sense. The bit about "signal loss" that I was missing there is that monitoring (or linking) *is* guaranteed in the situation where a node disconnects and reconnects quickly. > "Re-sending only messages that need to be re-sent" is > not possible in general, and this is not specific to Erlang distribution. > That's not what I was talking about, since we know sending is meant to be asynchronous and never fail - I was thinking about both ends, not just sender or receiver. The point I was making is that even if monitor_nodes can guarantee that you'll see nodedown before any other traffic over a new connection, you can only guarantee that for a process that actually called monitor_nodes (or is monitoring the remote process), since if you rely on some *other* process to do monitoring, then there's no ordering guarantee to be had. In other words (as the paper puts it) you have to monitor each interaction to be sure. And this holds, more importantly, for receipt as well. So if you aren't monitoring a remote process (or node) are you're holding on to a remote pid, then you've got to have your wits about you. Anyway, it's good to know that the monitoring guarantees are absolute. Cheers, Tim From jleivent@REDACTED Sun Jul 7 17:44:00 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Sun, 07 Jul 2013 11:44:00 -0400 Subject: [erlang-questions] can nodes fail/recover too fast to be seen? In-Reply-To: <201307071336.r67DaAZl061044@pluto.hedeland.org> References: <201307071336.r67DaAZl061044@pluto.hedeland.org> Message-ID: <51D98CC0.5010707@comcast.net> I have another related question. When monitor(process, {Name, Node}) is called, and the Node is already down, how quickly will the corresponding 'DOWN' event show up? Will it show up synchronously with the monitor/2 function call? In other words, is a pattern like this: MRef = monitor(process, {Name, Node}), receive {'DOWN', MRef, process, {Name, Node}, _Info} -> ... after 0 -> ok end, useful as a way to determine if the {Name, Node} is currently down or disconnected? Or, must the monitoring process always wait patiently for 'DOWN' events on nodes it monitors? How about when the Node is up, but Name is not registered there. Or it is registered there, but the registered process is not up? -- Jonathan From n.oxyde@REDACTED Sun Jul 7 19:17:57 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sun, 7 Jul 2013 19:17:57 +0200 Subject: [erlang-questions] Compiler list comprehension optimizations In-Reply-To: <9ACE6ABA-B9EB-428B-BDE2-300DFB683684@cs.otago.ac.nz> References: <51BA7407.3060307@comcast.net> <1371201405.33856.YahooMailNeo@web140103.mail.bf1.yahoo.com> <51BB5B04.4090408@comcast.net> <7c70418ea0bbaa919c3ed2befba9be70.squirrel@chasm.otago.ac.nz> <9ACE6ABA-B9EB-428B-BDE2-300DFB683684@cs.otago.ac.nz> Message-ID: Hello Richard, What is the point of the skip constructor? Couldn't the unfolder call itself on its own with the updated state? Also, could you formalize that in an EEP? Regards, -- Anthony Ramine Le 5 juil. 2013 ? 01:10, Richard A. O'Keefe a ?crit : > > On 5/07/2013, at 2:09 AM, Peer Stritzinger wrote: > >> On 2013-06-15 12:21:25 +0000, ok@REDACTED said: >>> We have seen [... where I = 0 then I+1 while I < 100] >>> proposed, but that's not actually as general as a functional >>> language should be. What's needed is a general 'unfold' >>> proposal, and we've had that for a while too. >> >> Somehow I have difficulty to find the "unfold" proposal, in the EEPS I only found tuple comprehensions EEP12 and multigenerators EEP19. > > I didn't say that there was an unfold EEP, > I said that a certain syntax had been proposed (by me, > in this mailing list) and that there had been an > unfold proposal (also by me, also in this mailing list). > > The unfold approach I described used > > your_unfold(Your_State) -> [] or [Next|Your_State'] > > However, Haskell has led the way. What we really want is > something closer to streams. > > A "stream" function maps a state to one of > done > {skip,New_State} > {next,New_State,Item} > > to_list(Unfolder, State) -> > case Unfolder(State) > of done -> [] > ; {skip,New_State} -> to_list(Unfolder, New_State) > ; {next,New_State,Item} -> [Item | to_list(Unfolder, New_State)] > end. > > The nice thing about streams is that they can handle mapping and > filtering. > > s_map(F, Unfolder) -> > fun (State) -> > case R = Unfolder(State) > of done -> R > ; {skip,State1} -> R > ; {next,State1,Item} -> {next,State1,F(Item)} > end > end. > > s_filter(P, Unfolder) -> > fun (State) -> > case R = Unfolder(State) > of done -> R > ; {skip,State1} -> R > ; {next,State1,Item} -> > case P(Item) > of false -> {skip,State1} > ; true -> R > end > end > end. > > They can also handle a 'while': > > s_while(P, Unfolder) -> > fun (State) -> > case R = Unfolder(State) > of done -> R > ; {skip,State1} -> R > ; {next,State1,Item} -> > case P(Item) > of false -> done > ; true -> R > end > end > end. > > So we can compose any chain of "map", "filter", and > "takeWhile" steps into a single unfolder->unfolder > function, which can then be applied to anything we > can convert to a stream, which includes lists, > integer ranges, tuples, and other things. > > With a little ingenuity, we can squeeze in "expanding > the state" so that we can include "dropWhile", every Nth, > and other wrinkles into our chains. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From n.oxyde@REDACTED Sun Jul 7 19:21:49 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sun, 7 Jul 2013 19:21:49 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> Message-ID: <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> Hello, My implementation of cond is unfortunately still a work in progress [1]. I couldn't decide which error should be thrown when the cond test doesn't return a boolean but I have found some references to cond expressions in an old paper [2] (page 37) and decided to stick to their choices. Regards, [1] https://github.com/nox/otp/compare/erlang:master...cond [2] http://babel.ls.fi.upm.es/~fred/thesis.pdf -- Anthony Ramine Le 21 juin 2013 ? 10:22, Anthony Ramine a ?crit : > I'll make a proper implementation of it this week-end. I would have done it earlier if you had written an EEP :) > > -- > Anthony Ramine > > Le 21 juin 2013 ? 07:04, Richard A. O'Keefe a ?crit : > >> This is what I have by way of patches to erl_parse.yrl so far. > From ok@REDACTED Mon Jul 8 00:04:02 2013 From: ok@REDACTED (ok@REDACTED) Date: Mon, 8 Jul 2013 10:04:02 +1200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> Message-ID: <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> > I couldn't decide which error should be thrown when the cond test doesn't > return a boolean but I have found some references to cond expressions in > an old paper [2] (page 37) and decided to stick to their choices. To be honest, I couldn't see any good reason to make cond E1 -> B1 ; ... ; En -> Bn end any different from case E1 of true -> B1 ; false -> ... case En of true -> Bn end ... end I see great virtue in there being *no* difference whatsoever in the semantics of the two, so that the choice is *solely* stylistic and rewriting one into the other (either way) *cannot* break a working program. Having looked at that reference, I do not see any advantage for the more complicated version. As long as you can find out *where* the error was, it just isn't _useful_ to know 'it was a cond' rather than 'it was a case'; a program catching errors would never want to treat them differently, and a human who has been told _where_ can instantly see _what_. From ok@REDACTED Mon Jul 8 04:04:48 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 8 Jul 2013 14:04:48 +1200 Subject: [erlang-questions] Compiler list comprehension optimizations In-Reply-To: References: <51BA7407.3060307@comcast.net> <1371201405.33856.YahooMailNeo@web140103.mail.bf1.yahoo.com> <51BB5B04.4090408@comcast.net> <7c70418ea0bbaa919c3ed2befba9be70.squirrel@chasm.otago.ac.nz> <9ACE6ABA-B9EB-428B-BDE2-300DFB683684@cs.otago.ac.nz> Message-ID: On 8/07/2013, at 5:17 AM, Anthony Ramine wrote: > Hello Richard, > > What is the point of the skip constructor? To a first approximation: to handle filtering. > Couldn't the unfolder call itself on its own with the updated state? No. Part of the point of the 'stream' approach is that every step along the way is *non*-recursive code that's amenable to inlining, right up to the end where you "pull" the results in a tail-recursive loop (which would like to have the rest of the pipeline inlined into it, thank you very much). > > Also, could you formalize that in an EEP? There's more formalisation than you could shake a banana at in Stream Fusion: Practical shortcut fusion for coinductive sequence types Duncan Coutts PhD thesis, Worcester College, University of Oxford, 2010. I don't have an EEP, what I _do_ have is a module -- which has undergone **minimal** testing -- to try the idea out. There are three classes of functions: - producers turn lists, tuples, integer ranges, &c into streams - transformers turn streams (possibly more than one) into streams - consumers accept streams and pull results out of them It's pretty amazing what you can do with stream transformers. t_alternate(S1, S2) elements of S1 interleaved with elements of S2 t_drop(N, S) all but the first N elements of S t_drop_while(P, S) all the elements of S starting with the first to satisfy P t_fby(S1, S2) concatenation of streams t_filter(P, S) all the elements of S that satisfy P t_map(F, S) F(X) for each X in S t_merge(S1, S2) Merge two sorted streams producing sorted output. (Ordered-set union, intersection, difference, &c are obviously doable since merge is doable; I happen not to have done them.) t_repeat(F, S) each element X of S repeated F(X) times t_scan(F, A, S) -> where Y1 = F(A,X1), Y2 = F(Y1,X2), ... t_take(N, S) the first N elements of S t_take_while(P, S) the leading elements of S satisfying P t_zip(S1, S2) If S1 is and S2 is then <{X1,X2}, ...> t_zip_with(F, S1, S2) the same as t_zip but F(X,Y) instead of {X,Y}. That's all I have at the moment. Quite a few of these transformers would be somewhere between difficult and impossible to write _as_ transformers without 'skip'. One reason to care about this stuff, especially in the context of list comprehension, is that a consumer fed by a network of transformers from some producers can be compiled into a single block of code that assigns to hidden local stack variables but preserves over-all functional purity and would require no change to the garbage collector, so an astonishingly rich extension of list comprehension *could* be provided The current draft of the module can be found at http://www.cs.otago.ac.nz/staffpriv/ok/stream.erl It was thrown together in something of a hurry to explore the idea, and is rather short of comments. I must do something about that, but since the 2nd semester has just started, I've already spent too long on this. I must stress that if you take the code as it stands, it's going to do *lots* of short-lived allocations. Think of it as a prototype to see what a compiler *could* do and it makes a lot more sense. From garry@REDACTED Mon Jul 8 04:18:58 2013 From: garry@REDACTED (Garry Hodgson) Date: Sun, 07 Jul 2013 22:18:58 -0400 Subject: [erlang-questions] low level packet access from erlang In-Reply-To: <20130707133723.GA26612@ioctl> References: <515C6E13.1050406@research.att.com> <20130403212407.GA25006@ioctl> <51ADE6E1.4030205@research.att.com> <20130604153054.GA31551@ioctl> <51AE7341.4090200@research.att.com> <51AE7D17.2050807@research.att.com> <20130605132308.GA32186@ioctl> <51D46966.8000107@research.att.com> <20130707133723.GA26612@ioctl> Message-ID: <51DA2192.2050906@research.att.com> On 07/07/2013 09:37 AM, Michael Santos wrote: > There were a few bugs: TCP options were left out of the > record to binary conversion and the length of the payload > was left out of the checksum. after i posted i made my own version that included the opts and the wireshark errors went away, so i figured it was that. didn't know about the payload length issue, though, so mine might still have that problem. i wonder if that could be causing it to still not work despite (hopefully) well formed packets. > Also, use pkt:makesum/1 to calculate the checksum. > This should work: TestSum = pkt:makesum( [ IPv4, Tcp, TcpPayload ] ), yes, i was using that at first, then switched to call my own versions. i'll retrofit once i turn latest version into rpm form. > You may still see some checksum mismatches if your > system uses TCP checksum offloading. oh, goodie. more things to trip over. thanks a lot for your help. -- Garry Hodgson AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From sean@REDACTED Mon Jul 8 08:00:02 2013 From: sean@REDACTED (Functional Jobs) Date: Mon, 8 Jul 2013 02:00:02 -0400 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <51da55785ceec@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Lead Server Core Developer (Java/Erlang) at yuilop http://functionaljobs.com/jobs/3074-lead-server-core-developer-java-erlang-at-yuilop Cheers, Sean Murphy FunctionalJobs.com From n.oxyde@REDACTED Mon Jul 8 08:04:34 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 8 Jul 2013 08:04:34 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> Message-ID: With that logic, I see no reason to make if G -> B end different from case {} of {} when G -> B end. Expanding cond to nested cases earlier in the compilation would mean confusing and misguided warning, e.g. "cond foobar -> ok end" would trigger the warning "no clause will ever match"; instead of the clearer "this clause will crash". -- Anthony Ramine Le 8 juil. 2013 ? 00:04, ok@REDACTED a ?crit : > >> I couldn't decide which error should be thrown when the cond test doesn't >> return a boolean but I have found some references to cond expressions in >> an old paper [2] (page 37) and decided to stick to their choices. > > To be honest, I couldn't see any good reason to make > cond E1 -> B1 ; ... ; En -> Bn end > any different from > case E1 of true -> B1 ; false -> > ... > case En of true -> Bn > end > ... > end > I see great virtue in there being *no* difference whatsoever > in the semantics of the two, so that the choice is *solely* > stylistic and rewriting one into the other (either way) *cannot* > break a working program. > > Having looked at that reference, I do not see any advantage for > the more complicated version. As long as you can find out > *where* the error was, it just isn't _useful_ to know 'it was a > cond' rather than 'it was a case'; a program catching errors > would never want to treat them differently, and a human who has > been told _where_ can instantly see _what_. > > > From ok@REDACTED Mon Jul 8 08:22:02 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 8 Jul 2013 18:22:02 +1200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> Message-ID: <08EA433D-351D-4558-A882-2B1A18315D7C@cs.otago.ac.nz> On 8/07/2013, at 6:04 PM, Anthony Ramine wrote: > With that logic, I see no reason to make if G -> B end different from case {} of {} when G -> B end. Neither do I. It is excess complexity that we really don't need, and it inhibits a range of source-to-source transformations that ought not to be inhibited. If I am told which *function clause* a selection failure is in, better still, if I am told which *line* it is in, no distinction between if/case/cond in exceptions is of any real help. The *only* time it is helpful is when you have one 'if' and one 'case'. Have *two* 'if's or *two* 'case's, and it's useless. > > Expanding cond to nested cases earlier in the compilation would mean confusing and misguided warning, e.g. "cond foobar -> ok end" would trigger the warning "no clause will ever match"; instead of the clearer "this clause will crash". You have presented a good argument for rewording the error message. That's all. From n.oxyde@REDACTED Mon Jul 8 10:35:25 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 8 Jul 2013 10:35:25 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <08EA433D-351D-4558-A882-2B1A18315D7C@cs.otago.ac.nz> References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> <08EA433D-351D-4558-A882-2B1A18315D7C@cs.otago.ac.nz> Message-ID: <249F45EB-527A-4D11-907F-5C99C3BF242B@gmail.com> No this example was just confusing, what I meant is that if you need a warning "no clause will ever match" in the case of an explicit "case foobar of true -> ok; false -> ok end" and a warning "this clause will crash" in the case of "cond foobar -> ok; true -> ok end". There is no rewording that makes the warning fit in both situations, they need to be handled separately. In my implementation I decided to annotate the Core Erlang cases to which a cond is compiled with the atom "cond_case". -- Anthony Ramine Le 8 juil. 2013 ? 08:22, "Richard A. O'Keefe" a ?crit : > > On 8/07/2013, at 6:04 PM, Anthony Ramine wrote: > >> With that logic, I see no reason to make if G -> B end different from case {} of {} when G -> B end. > > Neither do I. It is excess complexity that we really don't need, > and it inhibits a range of source-to-source transformations that > ought not to be inhibited. > > If I am told which *function clause* a selection failure is in, > better still, if I am told which *line* it is in, no distinction > between if/case/cond in exceptions is of any real help. The > *only* time it is helpful is when you have one 'if' and one 'case'. > Have *two* 'if's or *two* 'case's, and it's useless. > >> >> Expanding cond to nested cases earlier in the compilation would mean confusing and misguided warning, e.g. "cond foobar -> ok end" would trigger the warning "no clause will ever match"; instead of the clearer "this clause will crash". > > You have presented a good argument for rewording the error message. > > That's all. > From k.petrauskas@REDACTED Mon Jul 8 13:16:41 2013 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Mon, 8 Jul 2013 14:16:41 +0300 Subject: [erlang-questions] Unable to start mnesia DB. Message-ID: Hello, I cannot load table after erlang node restart. Mnesia prints no errors, just is loading the database for very long time. The database is on single node, all tables are disc_copy. In total the DB is ~220 MB. On startup, the last thing the mnesia prints is: Getting table my_db_lnk (disc_copies) from disc: local_only Size of this table (my_db_lnk) on disc is is ~36 MB in the DCD file and ~7 MB in he DCL file. This table is not the last table to be loaded in the database. IO at that time is almost zero, but CPU is used heavily (200% on machine having 2 cores). I was waiting for 3 hours for this table and then aborted. Tried to restart the node with the following options: bin/app console_clean -mnesia debug trace -mnesia auto_repair false But situation has not been changed. mnesia:info() shows several of my tables as active, but not all. After printing "Getting table...", mnesia prints "Transaction log dump initiated by time_threshold: already_dumped" from time to time. How could i debug this situation? Karolis -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Mon Jul 8 15:06:35 2013 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Mon, 8 Jul 2013 16:06:35 +0300 Subject: [erlang-questions] Unable to start mnesia DB. In-Reply-To: References: Message-ID: Additional info. The processes with highest reductions are with the following "current functions": {current_function,{disk_log,monitor_request,2}} {current_function,{disk_log,loop,1}} I am not sure if its related, but all the tables that are loaded successfully, have no DCL files. Karolis On Mon, Jul 8, 2013 at 2:16 PM, Karolis Petrauskas wrote: > Hello, > > I cannot load table after erlang node restart. Mnesia prints no > errors, just is loading the database for very long time. The database is on > single node, all tables are disc_copy. In total the DB is ~220 MB. On > startup, the last thing the mnesia prints is: > > Getting table my_db_lnk (disc_copies) from disc: local_only > > Size of this table (my_db_lnk) on disc is is ~36 MB in the DCD file and ~7 > MB in he DCL file. This table is not the last table to be loaded in the > database. IO at that time is almost zero, but CPU is used heavily (200% on > machine having 2 cores). I was waiting for 3 hours for this table and then > aborted. Tried to restart the node with the following options: > > bin/app console_clean -mnesia debug trace -mnesia auto_repair false > > But situation has not been changed. mnesia:info() shows several of my > tables as active, but not all. After printing "Getting table...", mnesia > prints "Transaction log dump initiated by time_threshold: already_dumped" > from time to time. > > How could i debug this situation? > > Karolis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Jul 8 16:38:56 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 8 Jul 2013 16:38:56 +0200 Subject: [erlang-questions] idiomatic large data sets; shared and unshared In-Reply-To: References: Message-ID: On Sat, Jul 6, 2013 at 3:54 AM, Felix Gallo wrote: > In the mutable world, you might structure it as a hashtable of hashes, or > hashtable of sets etc., and use the coordinate tuple {x,y} as a key into > the hash table, then mutate the list. addtocell(Coordinate, Value) and > removefromcell(Coordinate, Value) are very small, easy, and extremely fast > (sub-second for a million iterations of addtocell()). In a game, you will get into trouble with this representation alone. You need to have locational information about what are in the cells. In other words, given a thing in a cell, you must know, quickly what are adjacent or close to that cell in order to quickly asses what to do and so on. The trick in a game is spatial separation. In other words, you break the world into regions and then you handle each region by its own, gaining locality. The same is true in Erlang. You need to break your big array into a spatial structure, probably controlled by a process tree which handles each area of the whole world. In turn, this enables parallel execution and it also simplifies local decisions which can be made process-local. Take a look into stuff like K,d-trees, octrees and so on for ideas on how to approach the problem in a more general fasion. A blind grid can be made in ETS: {{X, Y}, Content}. But this runs the risk of having to copy Content to and from a process on each access and it is also not safe from races on insertion. Think is, this is not a trivial problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From felixgallo@REDACTED Mon Jul 8 17:02:35 2013 From: felixgallo@REDACTED (Felix Gallo) Date: Mon, 8 Jul 2013 08:02:35 -0700 Subject: [erlang-questions] idiomatic large data sets; shared and unshared In-Reply-To: References: Message-ID: Thanks much to Chandru for the simplification and to Jesper for the helpful and thoughtful insights. It turned out that I tricked myself in three different ways when trying to set up the ets grid at first. For future readers here were the fundamental flaws in my reasoning: 1. My test ran a million iterations of ets:insert({random:uniform(N), random:uniform(M), P}). Although my target was eventually N = M = 1000, I ran it with N = M = 10 in order to be able to print out the grid and visually spot check it for uniformity. ets is good at many things but inserting into a set or a bag (and not a duplicate bag) of size 100000 is not one of them. 2. set and bag both prevent duplicates from being inserted, which obviously, in hindsight, is super bad when the data structure is a linked list, because the entire list has to be traversed every time. For the scenario in which you have control over whether duplication happens in the first place, going to duplicate_bag is a massive performance improvement. 3. random:uniform/1 is moderately fast but under profiling turned out to be a significant part of the performance. Since the locations of bucketed entities wouldn't be random in practice but rather client-provided, I switched to using 'N rem M' to better assess ets speed. F. On Mon, Jul 8, 2013 at 7:38 AM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > > On Sat, Jul 6, 2013 at 3:54 AM, Felix Gallo wrote: > >> In the mutable world, you might structure it as a hashtable of hashes, or >> hashtable of sets etc., and use the coordinate tuple {x,y} as a key into >> the hash table, then mutate the list. addtocell(Coordinate, Value) and >> removefromcell(Coordinate, Value) are very small, easy, and extremely fast >> (sub-second for a million iterations of addtocell()). > > > In a game, you will get into trouble with this representation alone. You > need to have locational information about what are in the cells. In other > words, given a thing in a cell, you must know, quickly what are adjacent or > close to that cell in order to quickly asses what to do and so on. > > The trick in a game is spatial separation. In other words, you break the > world into regions and then you handle each region by its own, gaining > locality. The same is true in Erlang. You need to break your big array into > a spatial structure, probably controlled by a process tree which handles > each area of the whole world. In turn, this enables parallel execution and > it also simplifies local decisions which can be made process-local. > > Take a look into stuff like K,d-trees, octrees and so on for ideas on how > to approach the problem in a more general fasion. > > A blind grid can be made in ETS: {{X, Y}, Content}. But this runs the risk > of having to copy Content to and from a process on each access and it is > also not safe from races on insertion. Think is, this is not a trivial > problem. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kuangyel2000@REDACTED Mon Jul 8 18:20:41 2013 From: kuangyel2000@REDACTED (kuangyel2000@REDACTED) Date: Mon, 8 Jul 2013 09:20:41 -0700 (PDT) Subject: [erlang-questions] error_loger after run "erl -name foo" Message-ID: <9efd6592-6948-4652-9a0f-b34354967f41@googlegroups.com> I run "erl -name foo" on cmd of windows, it show: {error_logger,{{2013,7,8},{15,51,6}},"inet_parse:~p:~p: erroneous line, SKIPPED~n",["c:/Windows/System32/drivers/etc/hosts",1]} =INFO REPORT==== 8-Jul-2013::23:51:06 === inet_parse:"c:/Windows/System32/drivers/etc/hosts":1: erroneous line, SKIPPED Eshell V5.10.2 (abort with ^G) foo@REDACTED> but I run "erl -sname foo" it is OK. it has no error_logger. my file: c:/Windows/System32/drivers/etc/hosts 127.0.0.1 localhost ::1 localhost 127.0.0.1 sgwin712.com 127.0.0.1 sgwin712 is it OK? What is the problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.arnon@REDACTED Mon Jul 8 19:11:50 2013 From: alex.arnon@REDACTED (Alex Arnon) Date: Mon, 8 Jul 2013 20:11:50 +0300 Subject: [erlang-questions] Needed: Great big ordered int set. Message-ID: Hi All, I need to implement a very large set of data, with the following requirements: - It will be populated EXCLUSIVELY by 64-bit integers. - The only operations will be: - add element, - get number of elements, and - fold/foreach over the SORTED dataset. - The invocation order will be strictly: - create data structure, - add elements sequentially, - run one or more iteration operations, - discard data structure. - The size of the dataset MUST scale to 500M elements, preferably billions should be possible too. - The data does not have to reside in memory - however, 32 to 64 GB of RAM may be allocated. (of course, these will be used by the OS buffer cache in case a file-based solution is chosen). In summary: Performance is not a must, but volume and the ability to iterate over the ordered values is. Thanks in advance!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Mon Jul 8 19:22:44 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Mon, 8 Jul 2013 19:22:44 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: Message-ID: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> The data structure is a sorted list of integers. That 500M dataset number, is that over a single list of integers, or is that the sum of all lists of integers? What are the reliability requirements? Do you need redundancy and/or backups? It is a very different problem if a single server solution is enough, or if it requires a network of computers. Sergej On Jul 8, 2013, at 7:11 PM, Alex Arnon wrote: > Hi All, > > I need to implement a very large set of data, with the following requirements: > - It will be populated EXCLUSIVELY by 64-bit integers. > - The only operations will be: > - add element, > - get number of elements, and > - fold/foreach over the SORTED dataset. > - The invocation order will be strictly: > - create data structure, > - add elements sequentially, > - run one or more iteration operations, > - discard data structure. > - The size of the dataset MUST scale to 500M elements, preferably billions should be possible too. > - The data does not have to reside in memory - however, 32 to 64 GB of RAM may be allocated. (of course, these will be used by the OS buffer cache in case a file-based solution is chosen). > > In summary: Performance is not a must, but volume and the ability to iterate over the ordered values is. > > Thanks in advance!!! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From alex.arnon@REDACTED Mon Jul 8 19:29:07 2013 From: alex.arnon@REDACTED (Alex Arnon) Date: Mon, 8 Jul 2013 20:29:07 +0300 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> References: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> Message-ID: - A single set of integers - 500M of them. - This is a throwaway piece of data - once I've added all the values and iterated over them a couple of times, it is of no further use. - Mutation (addition of an integer) speed is not very critical, however due to the size of the dataset, it should be "reasonable" - i.e. less than a millisecond per insertion on average. On Mon, Jul 8, 2013 at 8:22 PM, Sergej Jurecko wrote: > The data structure is a sorted list of integers. That 500M dataset number, > is that over a single list of integers, or is that the sum of all lists of > integers? > What are the reliability requirements? Do you need redundancy and/or > backups? It is a very different problem if a single server solution is > enough, or if it requires a network of computers. > > > Sergej > > On Jul 8, 2013, at 7:11 PM, Alex Arnon wrote: > > > Hi All, > > > > I need to implement a very large set of data, with the following > requirements: > > - It will be populated EXCLUSIVELY by 64-bit integers. > > - The only operations will be: > > - add element, > > - get number of elements, and > > - fold/foreach over the SORTED dataset. > > - The invocation order will be strictly: > > - create data structure, > > - add elements sequentially, > > - run one or more iteration operations, > > - discard data structure. > > - The size of the dataset MUST scale to 500M elements, preferably > billions should be possible too. > > - The data does not have to reside in memory - however, 32 to 64 GB of > RAM may be allocated. (of course, these will be used by the OS buffer cache > in case a file-based solution is chosen). > > > > In summary: Performance is not a must, but volume and the ability to > iterate over the ordered values is. > > > > Thanks in advance!!! > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Mon Jul 8 19:36:41 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Mon, 8 Jul 2013 19:36:41 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> Message-ID: <4681F1BE-8EC7-415A-BC62-64D1F957A89C@gmail.com> Why not just use mongodb, mysql or postgresql? Sergej On Jul 8, 2013, at 7:29 PM, Alex Arnon wrote: > - A single set of integers - 500M of them. > - This is a throwaway piece of data - once I've added all the values and iterated over them a couple of times, it is of no further use. > - Mutation (addition of an integer) speed is not very critical, however due to the size of the dataset, it should be "reasonable" - i.e. less than a millisecond per insertion on average. > > > On Mon, Jul 8, 2013 at 8:22 PM, Sergej Jurecko wrote: > The data structure is a sorted list of integers. That 500M dataset number, is that over a single list of integers, or is that the sum of all lists of integers? > What are the reliability requirements? Do you need redundancy and/or backups? It is a very different problem if a single server solution is enough, or if it requires a network of computers. > > > Sergej > > On Jul 8, 2013, at 7:11 PM, Alex Arnon wrote: > > > Hi All, > > > > I need to implement a very large set of data, with the following requirements: > > - It will be populated EXCLUSIVELY by 64-bit integers. > > - The only operations will be: > > - add element, > > - get number of elements, and > > - fold/foreach over the SORTED dataset. > > - The invocation order will be strictly: > > - create data structure, > > - add elements sequentially, > > - run one or more iteration operations, > > - discard data structure. > > - The size of the dataset MUST scale to 500M elements, preferably billions should be possible too. > > - The data does not have to reside in memory - however, 32 to 64 GB of RAM may be allocated. (of course, these will be used by the OS buffer cache in case a file-based solution is chosen). > > > > In summary: Performance is not a must, but volume and the ability to iterate over the ordered values is. > > > > Thanks in advance!!! > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.arnon@REDACTED Mon Jul 8 20:14:26 2013 From: alex.arnon@REDACTED (Alex Arnon) Date: Mon, 8 Jul 2013 21:14:26 +0300 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: <4681F1BE-8EC7-415A-BC62-64D1F957A89C@gmail.com> References: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> <4681F1BE-8EC7-415A-BC62-64D1F957A89C@gmail.com> Message-ID: I'll probably end up doing just that, but was hoping I could resolve the thing in-process. On 8 Jul 2013, at 20:36, Sergej Jurecko wrote: > Why not just use mongodb, mysql or postgresql? > > > Sergej > > On Jul 8, 2013, at 7:29 PM, Alex Arnon wrote: > >> - A single set of integers - 500M of them. >> - This is a throwaway piece of data - once I've added all the values and iterated over them a couple of times, it is of no further use. >> - Mutation (addition of an integer) speed is not very critical, however due to the size of the dataset, it should be "reasonable" - i.e. less than a millisecond per insertion on average. >> >> >> On Mon, Jul 8, 2013 at 8:22 PM, Sergej Jurecko wrote: >>> The data structure is a sorted list of integers. That 500M dataset number, is that over a single list of integers, or is that the sum of all lists of integers? >>> What are the reliability requirements? Do you need redundancy and/or backups? It is a very different problem if a single server solution is enough, or if it requires a network of computers. >>> >>> >>> Sergej >>> >>> On Jul 8, 2013, at 7:11 PM, Alex Arnon wrote: >>> >>> > Hi All, >>> > >>> > I need to implement a very large set of data, with the following requirements: >>> > - It will be populated EXCLUSIVELY by 64-bit integers. >>> > - The only operations will be: >>> > - add element, >>> > - get number of elements, and >>> > - fold/foreach over the SORTED dataset. >>> > - The invocation order will be strictly: >>> > - create data structure, >>> > - add elements sequentially, >>> > - run one or more iteration operations, >>> > - discard data structure. >>> > - The size of the dataset MUST scale to 500M elements, preferably billions should be possible too. >>> > - The data does not have to reside in memory - however, 32 to 64 GB of RAM may be allocated. (of course, these will be used by the OS buffer cache in case a file-based solution is chosen). >>> > >>> > In summary: Performance is not a must, but volume and the ability to iterate over the ordered values is. >>> > >>> > Thanks in advance!!! >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Mon Jul 8 20:31:30 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Mon, 8 Jul 2013 20:31:30 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> <4681F1BE-8EC7-415A-BC62-64D1F957A89C@gmail.com> Message-ID: <0A48589B-1E46-4D87-ADDD-0704DBA14A77@gmail.com> elevedb https://github.com/basho/eleveldb hanoidb https://github.com/krestenkrab/hanoidb Sergej On Jul 8, 2013, at 8:14 PM, Alex Arnon wrote: > I'll probably end up doing just that, but was hoping I could resolve the thing in-process. > > > On 8 Jul 2013, at 20:36, Sergej Jurecko wrote: > >> Why not just use mongodb, mysql or postgresql? >> >> >> Sergej >> >> On Jul 8, 2013, at 7:29 PM, Alex Arnon wrote: >> >>> - A single set of integers - 500M of them. >>> - This is a throwaway piece of data - once I've added all the values and iterated over them a couple of times, it is of no further use. >>> - Mutation (addition of an integer) speed is not very critical, however due to the size of the dataset, it should be "reasonable" - i.e. less than a millisecond per insertion on average. >>> >>> >>> On Mon, Jul 8, 2013 at 8:22 PM, Sergej Jurecko wrote: >>> The data structure is a sorted list of integers. That 500M dataset number, is that over a single list of integers, or is that the sum of all lists of integers? >>> What are the reliability requirements? Do you need redundancy and/or backups? It is a very different problem if a single server solution is enough, or if it requires a network of computers. >>> >>> >>> Sergej >>> >>> On Jul 8, 2013, at 7:11 PM, Alex Arnon wrote: >>> >>> > Hi All, >>> > >>> > I need to implement a very large set of data, with the following requirements: >>> > - It will be populated EXCLUSIVELY by 64-bit integers. >>> > - The only operations will be: >>> > - add element, >>> > - get number of elements, and >>> > - fold/foreach over the SORTED dataset. >>> > - The invocation order will be strictly: >>> > - create data structure, >>> > - add elements sequentially, >>> > - run one or more iteration operations, >>> > - discard data structure. >>> > - The size of the dataset MUST scale to 500M elements, preferably billions should be possible too. >>> > - The data does not have to reside in memory - however, 32 to 64 GB of RAM may be allocated. (of course, these will be used by the OS buffer cache in case a file-based solution is chosen). >>> > >>> > In summary: Performance is not a must, but volume and the ability to iterate over the ordered values is. >>> > >>> > Thanks in advance!!! >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Mon Jul 8 20:21:17 2013 From: freza@REDACTED (Jachym Holecek) Date: Mon, 8 Jul 2013 14:21:17 -0400 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> <4681F1BE-8EC7-415A-BC62-64D1F957A89C@gmail.com> Message-ID: <20130708182117.GA21257@circlewave.net> # Alex Arnon 2013-07-08: > I'll probably end up doing just that, but was hoping I could resolve the > thing in-process. You could experiment with packing your fixed-width integers into binaries. Keep integers sorted so you can bsearch on given binary chunk. Then build some index on top of these, say and ordered_set ETS. If you're concerned about memory you can also leverage in-chunk ordering for some simple but (probably) efficient compression scheme (this will complicate lookup a bit, but also means higher data density -- have a look at literature around column-oriented datastores). Insert accumulates suitable number of entries on-heap, then builds up another chunk and inserts into toplevel index -- ETS entry could be {Min_item, Bin}. Lookup finds chunk (see ets:next/X, ets:prev/X) and searches in it. Foreach and fold-merge are sweet because you can easily process N chunks in parallel if you like (note fold-merge as opposed to inherenly sequential fold!). Range queries are obvious. Random inserts would be pretty expensive I think. Note I am here assuming that binaries are kept as off-heap objects even when inserted in ETS. I believe that's the case, but haven't checked -- there are viable alternatives (trees of processes were mentioned recently) so no big deal even if I'm wrong. So no: there probably isn't a ready-made structure for this in Erlang, but it doesn't sound terribly hard to build something yourself -- you've mentioned so many handy assumptions you can make that it almost looks easy. It's not the kind of problem I would personally use an enterprise relational database for... :-) BR, -- Jachym From jleivent@REDACTED Mon Jul 8 21:31:45 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Mon, 08 Jul 2013 15:31:45 -0400 Subject: [erlang-questions] gen_server aggregating calls/casts Message-ID: <51DB13A1.5010001@comcast.net> I would like the server process I'm designing to be able to aggregate certain calls and/or casts - but I don't see any way to do this with gen_server. Specifically, when this server is within a particular handle_call, it should find all similar calls on its message queue and handle them together, responding to all of the callers at once after performing a single combined action on their behalf. It can't accumulate calls into its state because there are other unrelated calls/casts it can encounter prior to seeing all of the currently queued matching calls. Conceptually, it needs to do something like: drain_all_requests(List) -> receive R={request, _} -> drain_all_requests([R|List]) after 0 -> List end. handle_call(R={request, Info}, From, State) -> RequestList = drain_all_requests([R]), handle_aggregated_requests(RequestList, State), ... However, I can't write the necessary receive to drain the message queue of the matching calls I'd like to aggregate - because I don't know the internal format of message wrappers used by gen_server. And I don't have a way to respond to any but the one that put the server into the handle_call. I would be willing to use handle_cast instead (and roll my own response scheme), but I still don't have a way to receive the matching casts on the message queue. I guess I could examine the source code for gen_server and determine the call and/or cast message format, then use that. But, I'd rather play by the modularity rules. Is there any way I can do that? Thanks -- Jonathan From sergej.jurecko@REDACTED Mon Jul 8 21:40:23 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Mon, 8 Jul 2013 21:40:23 +0200 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: <51DB13A1.5010001@comcast.net> References: <51DB13A1.5010001@comcast.net> Message-ID: Easiest to explain with code: handle_call(Msg,From,State) -> {noreply,State#rec{calls = [From | State#rec.calls}}; handle_call(replyall,_,State) -> [gen_server:reply(From,SomeMessage) || From <- lists:reverse(State#rec.calls)], {reply,ok,State} Queue instead of a list is probably better. Sergej On Jul 8, 2013, at 9:31 PM, Jonathan Leivent wrote: > I would like the server process I'm designing to be able to aggregate certain calls and/or casts - but I don't see any way to do this with gen_server. > > Specifically, when this server is within a particular handle_call, it should find all similar calls on its message queue and handle them together, responding to all of the callers at once after performing a single combined action on their behalf. It can't accumulate calls into its state because there are other unrelated calls/casts it can encounter prior to seeing all of the currently queued matching calls. Conceptually, it needs to do something like: > > drain_all_requests(List) -> > receive > R={request, _} -> drain_all_requests([R|List]) > after 0 -> List > end. > > handle_call(R={request, Info}, From, State) -> > RequestList = drain_all_requests([R]), > handle_aggregated_requests(RequestList, State), > ... > > However, I can't write the necessary receive to drain the message queue of the matching calls I'd like to aggregate - because I don't know the internal format of message wrappers used by gen_server. And I don't have a way to respond to any but the one that put the server into the handle_call. I would be willing to use handle_cast instead (and roll my own response scheme), but I still don't have a way to receive the matching casts on the message queue. > > I guess I could examine the source code for gen_server and determine the call and/or cast message format, then use that. But, I'd rather play by the modularity rules. Is there any way I can do that? > > Thanks > -- Jonathan > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jleivent@REDACTED Mon Jul 8 21:59:47 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Mon, 08 Jul 2013 15:59:47 -0400 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: References: <51DB13A1.5010001@comcast.net> Message-ID: <51DB1A33.2020402@comcast.net> On 07/08/2013 03:40 PM, Sergej Jurecko wrote: > Easiest to explain with code: > > handle_call(Msg,From,State) -> > {noreply,State#rec{calls = [From | State#rec.calls}}; > > handle_call(replyall,_,State) -> > [gen_server:reply(From,SomeMessage) || From <- lists:reverse(State#rec.calls)], > {reply,ok,State} > > Queue instead of a list is probably better. > > > Sergej As I said: "It can't accumulate calls into its state because there are other unrelated calls/casts it can encounter prior to seeing all of the currently queued matching calls." Also, it will never get anything like your replyall call. Maybe I need two separate gen_server processes - one to just receive the requests and aggregate them into its state, then hand them off to the other, which handles both the aggregated requests and other calls/casts. -- Jonathan From rtrlists@REDACTED Mon Jul 8 22:30:18 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Mon, 8 Jul 2013 21:30:18 +0100 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: <51DB1A33.2020402@comcast.net> References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> Message-ID: A gen_server is there to handle a resource such that requests to that resource are handled sequentially. It sounds like that doesn't quite match your use case. What you describe appears a little bit odd. Say you receive a request, and now, before you handle it, you check if more of the same request type has come in, and if they have, you add them in your "send results to" set. Now what? Do you check again? Or are you satisfied you can actually do some work? Do you maybe want to memoize results? By adding results to your server state, you could reply to identical(?) requests via lookup, leaving the resource in peace. Or do you require some kind of batching, only bothering the resource if enough interest is present? Maybe augmented with some kind of time limit? This would probably be a bit harder to implement. Robby On Jul 8, 2013 8:59 PM, "Jonathan Leivent" wrote: > On 07/08/2013 03:40 PM, Sergej Jurecko wrote: > >> Easiest to explain with code: >> >> handle_call(Msg,From,State) -> >> {noreply,State#rec{calls = [From | State#rec.calls}}; >> >> handle_call(replyall,_,State) -> >> [gen_server:reply(From,**SomeMessage) || From <- >> lists:reverse(State#rec.calls)**], >> {reply,ok,State} >> >> Queue instead of a list is probably better. >> >> >> Sergej >> > > As I said: "It can't accumulate calls into its state because there are > other unrelated calls/casts it can encounter prior to seeing all of the > currently queued matching calls." Also, it will never get anything like > your replyall call. > > Maybe I need two separate gen_server processes - one to just receive the > requests and aggregate them into its state, then hand them off to the > other, which handles both the aggregated requests and other calls/casts. > > -- Jonathan > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Mon Jul 8 22:31:49 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Mon, 08 Jul 2013 22:31:49 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: <20130708182117.GA21257@circlewave.net> References: <7A25DFA4-FA79-436F-BEC0-8781F8FBDCC7@gmail.com> <4681F1BE-8EC7-415A-BC62-64D1F957A89C@gmail.com> <20130708182117.GA21257@circlewave.net> Message-ID: <51DB21B5.1030104@gmail.com> On 2013-07-08 20:21, Jachym Holecek wrote: > You could experiment with packing your fixed-width integers into binaries. Keep > integers sorted so you can bsearch on given binary chunk. There is no point in using binaries to "pack" numbers. On a 64-bit Erlang VM, the majority of those integers will be a fixnum anyway, represented by a single 64-bit word. The remaining ones will be encoded as bignums, which is just as efficient as using binaries (and both simpler and faster). I tried building an ordered_set ETS table of {N}-tuples, and it used 7 words per entry. For 500M entries, it would need 28 GB, so if you have 32 GB, it could work for you. Advantages: simple, and also fast. I assume that these numbers are something like hashes, so that the distribution is completely random, and since the keyspace is 64 bits, you can't use tricks like bitmaps. In fact, the array module can be a pretty space-efficient representation of this kind of data, and is even reasonably fast. The following (which populates the array with 'true' for all N in the list, but doesn't count the lists:seq/1 call as part of the execution time) took about 4 seconds on my machine for 1M entries: 1> ((fun(L) -> element(1, timer:tc(fun () -> lists:foldl(fun (X, A) -> array:set(X, true, A) end, array:new(), L) end)) end)(lists:seq(1,1000000))). 4008000 Taking the data structure size in words instead of the time: 2> ((fun(L) -> erts_debug:flat_size(timer:tc(fun () -> lists:foldl(fun (X, A) -> array:set(X, true, A) end, array:new(), L) end)) end)(lists:seq(1,1000000))). 1233424 it uses 9 MB for 1M entries. I upped the size to 100M entries and could comfortably populate the array in a few minutes on my home PC, resulting in a data structure of size 123M words (940 MB). My example of course uses a dense data set, and so it has the smallest possible representation. The array module handles sparse arrays pretty well, but if entries tend to be spaced 10 or more apart and are rarely clustered close to each other, it will use about 10 times as much space as a dense array, so it might or might not suite you; try it and see. Uninitialized entries will simply return 'undefined', and there are traversal functions that will skip uninitialized entries for you. /Richard From jleivent@REDACTED Tue Jul 9 00:16:37 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Mon, 08 Jul 2013 18:16:37 -0400 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> Message-ID: <51DB3A45.2070404@comcast.net> On 07/08/2013 04:30 PM, Robert Raschke wrote: > A gen_server is there to handle a resource such that requests to that > resource are handled sequentially. It sounds like that doesn't quite match > your use case. > > What you describe appears a little bit odd. Say you receive a request, and > now, before you handle it, you check if more of the same request type has > come in, and if they have, you add them in your "send results to" set. Now > what? Do you check again? Or are you satisfied you can actually do some > work? The server goes through longish periods when it is doing work (writing and syncing requests to disk, communicating with other servers, etc.). After these longish periods, it is very likely that many requests have accumulated on its queue, along with other messages that are not requests. I'd be satisfied if it could aggregate all requests that are currently on the server's queue, but do so without having to first go through the processing of the non-request messages that have also accumulated on its queue. Then, batch process those aggregated requests together - including writing/syncing them to disk together - with big performance savings due to amortizing the expensive write/sync operation (as well as reducing network overhead for the other parts of request processing). > > Do you maybe want to memoize results? By adding results to your server > state, you could reply to identical(?) requests via lookup, leaving the > resource in peace. It needs to service any requests it aggregates before servicing any non-request message. Or, rather, that's true of the first request. Said a better way: requests can be processed earlier than non-request messages that arrive before those requests, but the opposite is not true. So, if the server memoizes the requests, it has to process them as soon as it encounters the first non-request message - even though there may be many more requests later in its queue it could aggregate with the earlier ones. Because earlier requests must be processed before later non-requests. Unless I memoize ALL messages, requests and non-requests - so that the server delays handling any until it gets to a queue-empty state or times out or something. That's possible, but it sounds like I'll have to write my own gen_server-like infrastructure dispatch loop once I've accumulated all those non-requests. I guess that's possible... A separate aggregating gen_server might be easier and more modular. Like this: aggregator actual server ========== ============= doing busy work... request --> memoize and forward --> forwarded request enqueued request --> memoize ... request --> memoize ... done busy work see forwarded request handle call <-- call aggregator reply all and clear --> handle aggregated requests I'll try both... > Or do you require some kind of batching, only bothering the resource if > enough interest is present? Maybe augmented with some kind of time limit? > This would probably be a bit harder to implement. Yes - such processing might be nice. Knowing how many requests can be written/synced to disk in about the same time as a single request, for instance, and knowing approximately how long such a write/sync takes. Which will vary greatly based on available hardware and the underlying OS, external demands on disk IO, etc. I'll leave all that for version 2... -- Jonathan From ok@REDACTED Tue Jul 9 00:12:05 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 9 Jul 2013 10:12:05 +1200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <249F45EB-527A-4D11-907F-5C99C3BF242B@gmail.com> References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> <08EA433D-351D-4558-A882-2B1A18315D7C@cs.otago.ac.nz> <249F45EB-527A-4D11-907F-5C99C3BF242B@gmail.com> Message-ID: On 8/07/2013, at 8:35 PM, Anthony Ramine wrote: > No this example was just confusing, what I meant is that if you need a warning "no clause will ever match" in the case of an explicit "case foobar of true -> ok; false -> ok end" and a warning "this clause will crash" in the case of "cond foobar -> ok; true -> ok end". There is no rewording that makes the warning fit in both situations, they need to be handled separately. In the English text here you have an "if" with no consequent. case foobar of true -> ok ; false -> ok end Yes, it is appropriate to say "No matching clause" here. As for cond foobar -> {ok,1} ; true-> {ok,2} end this should *BE* case foobar of true -> {ok,1} ; false -> case true of true -> {ok,2} end end and no error message is appropriate here because there *is* a clause that will match and be selected. It is only true that the two situations need separate handling because one of them is not an error. Another not-entirely-unreasonable translation of cond C1 -> B1 ; ... ; Cn -> Bn end would be case (C1 andalso true) of true -> B1 ; false -> ... case (Cn andalso true) of true -> Bn end end and in this case you might just possibly get a warning from the compiler that might just possibly actually be useful (unlike 'this clause will crash'): you might be told "expression cannot be Boolean". It could *never* be appropriate to say "this clause will crash" for the simple reason that it's *not* the clause that would crash but the *condition* of the clause. "This condition cannot be Boolean" might make sense, but not the other. From bchesneau@REDACTED Tue Jul 9 00:55:23 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 9 Jul 2013 00:55:23 +0200 Subject: [erlang-questions] [ANN] barrel: a simple TCP Acceptor pool, Message-ID: I just released barrel a simple TCP Acceptor pool, https://github.com/benoitc/barrel/tree/master/src The design of barrel differs from ranch] [1] ). Instead of spawning w new process once a connectino and passing the control of the socket to it which can be slow, Barrel ois only spawning a new acceptors. The accepted socket will continue to be used in the same process that accepted it before. It can be used in the case you don't need a strong supervision (though acceptors are still supervised). A simple example of its usage can be found here: https://github.com/benoitc/barrel/tree/master/example/echo Also sieve that I just released is using it: https://github.com/benoitc/sieve Enjoy! [1] http://github.com/extend/ranch -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Tue Jul 9 01:27:52 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Tue, 9 Jul 2013 00:27:52 +0100 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: <51DB3A45.2070404@comcast.net> References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> <51DB3A45.2070404@comcast.net> Message-ID: Have you considered using a queuing framework, like RabbitMQ? By using topic exchanges and appropriately bound queues, it seems a bit more straightforward to implement what you desire. Alternatively, you'll probably want to double layer your server, as was previously suggested. Have the outer server deal with the problem of batching and the inner one serialise access to your resource. Judicious use of gen_server:call with noreply results in the outer server, whilst keeping track of the callers, and a combination of immediate calls to the lower layer and timeout handling depending on your use cases, would be my first attempt. Robby On Jul 8, 2013 11:16 PM, "Jonathan Leivent" wrote: > On 07/08/2013 04:30 PM, Robert Raschke wrote: > >> A gen_server is there to handle a resource such that requests to that >> resource are handled sequentially. It sounds like that doesn't quite match >> your use case. >> >> What you describe appears a little bit odd. Say you receive a request, and >> now, before you handle it, you check if more of the same request type has >> come in, and if they have, you add them in your "send results to" set. Now >> what? Do you check again? Or are you satisfied you can actually do some >> work? >> > > The server goes through longish periods when it is doing work (writing and > syncing requests to disk, communicating with other servers, etc.). After > these longish periods, it is very likely that many requests have > accumulated on its queue, along with other messages that are not requests. > I'd be satisfied if it could aggregate all requests that are currently on > the server's queue, but do so without having to first go through the > processing of the non-request messages that have also accumulated on its > queue. Then, batch process those aggregated requests together - including > writing/syncing them to disk together - with big performance savings due to > amortizing the expensive write/sync operation (as well as reducing network > overhead for the other parts of request processing). > > >> Do you maybe want to memoize results? By adding results to your server >> state, you could reply to identical(?) requests via lookup, leaving the >> resource in peace. >> > > It needs to service any requests it aggregates before servicing any > non-request message. Or, rather, that's true of the first request. Said a > better way: requests can be processed earlier than non-request messages > that arrive before those requests, but the opposite is not true. > > So, if the server memoizes the requests, it has to process them as soon as > it encounters the first non-request message - even though there may be many > more requests later in its queue it could aggregate with the earlier ones. > Because earlier requests must be processed before later non-requests. > > Unless I memoize ALL messages, requests and non-requests - so that the > server delays handling any until it gets to a queue-empty state or times > out or something. That's possible, but it sounds like I'll have to write > my own gen_server-like infrastructure dispatch loop once I've accumulated > all those non-requests. I guess that's possible... > > A separate aggregating gen_server might be easier and more modular. Like > this: > > aggregator actual server > ========== ============= > doing busy work... > request --> memoize and forward --> forwarded request enqueued > request --> memoize > ... > request --> memoize > ... done busy work > see forwarded request > handle call <-- call aggregator > reply all and clear --> handle aggregated requests > > I'll try both... > > Or do you require some kind of batching, only bothering the resource if >> enough interest is present? Maybe augmented with some kind of time limit? >> This would probably be a bit harder to implement. >> > > Yes - such processing might be nice. Knowing how many requests can be > written/synced to disk in about the same time as a single request, for > instance, and knowing approximately how long such a write/sync takes. Which > will vary greatly based on available hardware and the underlying OS, > external demands on disk IO, etc. I'll leave all that for version 2... > > -- Jonathan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moxford@REDACTED Tue Jul 9 05:35:25 2013 From: moxford@REDACTED (Mike Oxford) Date: Mon, 8 Jul 2013 20:35:25 -0700 Subject: [erlang-questions] Erlang-based groupware Message-ID: In looking at various groupware packages, I do not see any which are (surprisingly) Erlang-based. Am I missing an obvious one somewhere? -mox -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredrikelinder@REDACTED Tue Jul 9 03:50:51 2013 From: fredrikelinder@REDACTED (Fredrik Linder) Date: Mon, 8 Jul 2013 18:50:51 -0700 (PDT) Subject: [erlang-questions] EUnit and *timed out* test cases Message-ID: <6ca4eed5-04ff-4b50-8be6-98358b32a231@googlegroups.com> Hello Erlang Devs Has anyone an updated EUnit version that reports *timed out* tests as failed tests (or as timed out), so that *timed out* tests are not silently ignored by EUnit? If not, is this something Erlang/OTP would consider implement? /Fredrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From yash_ganthe@REDACTED Tue Jul 9 08:06:58 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Tue, 9 Jul 2013 06:06:58 +0000 Subject: [erlang-questions] Dialyzer warning about patterns Message-ID: Hi, I have the following code: Port = open_port({spawn,Args},[{packet,2}]), case is_port(Port) of true -> io:format("~n ~p port created successfully ~n",[Port]), ok; false -> %%Line 52 io:format("something went wrong ~n"), failed end. Dialyzer gives the following warning: mymod.erl:52: The pattern 'false' can never match the type 'true' The possible results of is_port are true and false considering is_port(Term) -> boolean() What do I need to do to get rid of such warnings? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Tue Jul 9 08:31:22 2013 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 09 Jul 2013 09:31:22 +0300 Subject: [erlang-questions] Dialyzer warning about patterns In-Reply-To: References: Message-ID: <51DBAE3A.5040600@cs.ntua.gr> On 07/09/2013 09:06 AM, Yash Ganthe wrote: > Hi, > > I have the following code: > > Port = open_port({spawn,Args},[{packet,2}]), > > case is_port(Port) of > > true -> > > io:format("~n ~p port created successfully ~n",[Port]), > > ok; > > false -> %%Line 52 > > io:format("something went wrong ~n"), > > failed > > end. > > Dialyzer gives the following warning: > > mymod.erl:52: The pattern 'false' can never match the type 'true' > > The possible results of is_port are true and false considering > > is_port(Term) -> boolean() > > What do I need to do to get rid of such warnings? You simplify your code by taking out the false branch of the case statement and the case statement itself. Sorry to point it out, but in its current state the above code is moronic. Yes, is_port/1 returns boolean() but that's if you pass it any term(), not something that is clearly a port(). Kostis From knutin@REDACTED Tue Jul 9 08:48:47 2013 From: knutin@REDACTED (Knut Nesheim) Date: Tue, 9 Jul 2013 08:48:47 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: Message-ID: <7180DF2E-D6D7-4B97-8CA9-908C6ADFA877@gmail.com> Hi Alex, I have had similar challenges in the past. I needed to efficiently store 250M 64 bit ints with a byte value and do 100k reads per second. Writes for me was less than 100 per second. I came up with https://github.com/knutin/bisect It uses a big binary, keeps the data sorted and does a binary search. On a decent machine it can do 8M random reads per second (reads can also be parallelized). Inserts get slow when the structure grows big (lots of memory to copy around). Using multiple of these structures might be an option to keep the write perfomance high. Regards Knut On 08.07.2013, at 19:11, Alex Arnon wrote: > Hi All, > > I need to implement a very large set of data, with the following requirements: > - It will be populated EXCLUSIVELY by 64-bit integers. > - The only operations will be: > - add element, > - get number of elements, and > - fold/foreach over the SORTED dataset. > - The invocation order will be strictly: > - create data structure, > - add elements sequentially, > - run one or more iteration operations, > - discard data structure. > - The size of the dataset MUST scale to 500M elements, preferably billions should be possible too. > - The data does not have to reside in memory - however, 32 to 64 GB of RAM may be allocated. (of course, these will be used by the OS buffer cache in case a file-based solution is chosen). > > In summary: Performance is not a must, but volume and the ability to iterate over the ordered values is. > > Thanks in advance!!! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Tue Jul 9 11:32:46 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 9 Jul 2013 11:32:46 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: Message-ID: On Mon, Jul 8, 2013 at 7:11 PM, Alex Arnon wrote: > Hi All, > > I need to implement a very large set of data, with the following > requirements: > - It will be populated EXCLUSIVELY by 64-bit integers. > - The only operations will be: > - add element, > - get number of elements, and > - fold/foreach over the SORTED dataset. > > Unless I am totally wrong here about what you want to get, why don't you use a digital representation? A 64 bit integer is a stream of bits. You can then share the common parts of the bit pattern in a tree-like structure. This in turn means that compression is "automatic" since the common bit patterns acts like the compression pattern. Add something like "everything below this node is present" and you can compress common parts of the tree. In general, the solution is RADIX-trees, or something along that pattern. They have, incidentally, also been used for quick Integer-key-based maps and so on. -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Tue Jul 9 12:00:10 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 9 Jul 2013 12:00:10 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> <08EA433D-351D-4558-A882-2B1A18315D7C@cs.otago.ac.nz> <249F45EB-527A-4D11-907F-5C99C3BF242B@gmail.com> Message-ID: <970D171A-05CC-411F-8257-621706B92005@gmail.com> Which Erlang clause matches here? -- Anthony Ramine Le 9 juil. 2013 ? 00:12, Richard A. O'Keefe a ?crit : > this should *BE* > > case foobar > of true -> {ok,1} > ; false -> case true of true -> {ok,2} end > end > > and no error message is appropriate here because there > *is* a clause that will match and be selected. From carlsson.richard@REDACTED Tue Jul 9 12:02:08 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 09 Jul 2013 12:02:08 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: References: Message-ID: <51DBDFA0.2080102@gmail.com> On 2013-07-09 11:32 , Jesper Louis Andersen wrote: > Unless I am totally wrong here about what you want to get, why don't you > use a digital representation? A 64 bit integer is a stream of bits. You > can then share the common parts of the bit pattern in a tree-like > structure. This in turn means that compression is "automatic" since the > common bit patterns acts like the compression pattern. Add something > like "everything below this node is present" and you can compress common > parts of the tree. > > In general, the solution is RADIX-trees, or something along that > pattern. They have, incidentally, also been used for quick > Integer-key-based maps and so on. Assuming a 64-bit Erlang VM, there's not much point in using a radix tree, since the smallest unit of representation will use at least 64 bits anyway, even if it's just for storing the last few unique bits of each entry. (On a 32-bit Erlang VM, a 2- or 3- level radix tree would be good though.) So you'll end up with something proportional to N*Log(N) words (times the overhead for the data structure), unless you start packing the radix trees into binaries. That could work, but would probably be quite slow and rather messy and error prone. /Richard From knutin@REDACTED Tue Jul 9 12:37:45 2013 From: knutin@REDACTED (Knut Nesheim) Date: Tue, 9 Jul 2013 12:37:45 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: <51DBDFA0.2080102@gmail.com> References: <51DBDFA0.2080102@gmail.com> Message-ID: <00D86B8D-6215-4B19-A35D-46B2D8509502@gmail.com> On 09.07.2013, at 12:02, Richard Carlsson wrote: > On 2013-07-09 11:32 , Jesper Louis Andersen wrote: >> Unless I am totally wrong here about what you want to get, why don't you >> use a digital representation? A 64 bit integer is a stream of bits. You >> can then share the common parts of the bit pattern in a tree-like >> structure. This in turn means that compression is "automatic" since the >> common bit patterns acts like the compression pattern. Add something >> like "everything below this node is present" and you can compress common >> parts of the tree. >> >> In general, the solution is RADIX-trees, or something along that >> pattern. They have, incidentally, also been used for quick >> Integer-key-based maps and so on. > > Assuming a 64-bit Erlang VM, there's not much point in using a radix tree, since the smallest unit of representation will use at least 64 bits anyway Yes. Being a bit naive, I implemented such a tree structure in Erlang. The memory overhead is way too big and the performance quite low. I even tried the approach of using a binary for storage, available at https://github.com/knutin/btrie Judy arrays and especially the C implementation from HP has all the tricks you can think of. As Richard points out, if the word size is 64 bits, just the overhead of the pointers in such a structure will trump the size of the data. Regards Knut From y.snaky@REDACTED Tue Jul 9 12:56:25 2013 From: y.snaky@REDACTED (Yuri Lukyanov) Date: Tue, 9 Jul 2013 14:56:25 +0400 Subject: [erlang-questions] cover & export_all Message-ID: It seems that there is no way to cover-compile modules with export_all. Here is a simple example: cover_test.erl: -module(cover_test). test() -> ok. $ erl Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) 1> c(cover_test, [debug_info,export_all]). {ok,cover_test} 2> cover_test:test(). ok 3> cover:compile_beam(cover_test). {ok,cover_test} 4> cover_test:test(). ** exception error: undefined function cover_test:test/0 5> Could someone explain why it is like this? Is it done on purpose? Maybe it's a bug? The reason I want modules to be cover-compiled with +export_all is that it is sometimes convinient to have unit tests outside of a module. Before unit tests are run the modules get compiled with +export_all for tests to be able to access private functions. But the situation is that it is not possibe in this case to enable coverage analysys. From heinz@REDACTED Tue Jul 9 13:06:35 2013 From: heinz@REDACTED (Heinz Nikolaus Gies) Date: Tue, 9 Jul 2013 13:06:35 +0200 Subject: [erlang-questions] Needed: Great big ordered int set. In-Reply-To: <00D86B8D-6215-4B19-A35D-46B2D8509502@gmail.com> References: <51DBDFA0.2080102@gmail.com> <00D86B8D-6215-4B19-A35D-46B2D8509502@gmail.com> Message-ID: <4E694A46-481D-41B3-99A4-44085201973C@licenser.net> If you know that data often appear in hunks you could so something like: [{start,end} | lone] so worst case you literally get a straight list, best case you can get a good compression by getting a lot of bunches. Just a thought ;) not sure if it'd work for you - I've thought of that to represent a list of IPs where I would likely get a good compression out of that. Cheers, Heinz On Jul 9, 2013, at 12:37, Knut Nesheim wrote: > > > > > On 09.07.2013, at 12:02, Richard Carlsson wrote: > >> On 2013-07-09 11:32 , Jesper Louis Andersen wrote: >>> Unless I am totally wrong here about what you want to get, why don't you >>> use a digital representation? A 64 bit integer is a stream of bits. You >>> can then share the common parts of the bit pattern in a tree-like >>> structure. This in turn means that compression is "automatic" since the >>> common bit patterns acts like the compression pattern. Add something >>> like "everything below this node is present" and you can compress common >>> parts of the tree. >>> >>> In general, the solution is RADIX-trees, or something along that >>> pattern. They have, incidentally, also been used for quick >>> Integer-key-based maps and so on. >> >> Assuming a 64-bit Erlang VM, there's not much point in using a radix tree, since the smallest unit of representation will use at least 64 bits anyway > > Yes. Being a bit naive, I implemented such a tree structure in Erlang. The memory overhead is way too big and the performance quite low. I even tried the approach of using a binary for storage, available at https://github.com/knutin/btrie > > Judy arrays and especially the C implementation from HP has all the tricks you can think of. As Richard points out, if the word size is 64 bits, just the overhead of the pointers in such a structure will trump the size of the data. > > Regards > Knut > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From yoursurrogategod@REDACTED Tue Jul 9 18:43:34 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Tue, 9 Jul 2013 12:43:34 -0400 Subject: [erlang-questions] What does ":gen" do in this case? Message-ID: I'm looking at this code: spawn_CerebralUnits(IdsNPIds, CerebralUnitType, [Id | Ids])-> PId = CerebralUnitType:gen(self(), node()), ets:insert(IdsNPIds, {Id, PId}), ets:insert(IdsNPIds, {PId, Id}), spawn_CerebralUnits(IdsNPIds, CerebralUnitType, Ids); spawn_CerebralUnits(_IdsNPIds, _CerebralUnitType, [])-> true. On the line where PId is bound, you have this "CerebralUnitType:gen(self(), node()),". CerebralUnitType is an atom "cortex". So what does :gen do in this case? -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas@REDACTED Tue Jul 9 18:45:13 2013 From: thomas@REDACTED (Thomas Allen) Date: Tue, 09 Jul 2013 09:45:13 -0700 Subject: [erlang-questions] What does ":gen" do in this case? In-Reply-To: References: Message-ID: <1373388313.30229.140661253671910.2F53B71C@webmail.messagingengine.com> It calls the function `gen/2' in the `cortex' module. Thomas On Tue, Jul 9, 2013, at 09:43 AM, Yves S. Garret wrote: > I'm looking at this code: > > spawn_CerebralUnits(IdsNPIds, CerebralUnitType, [Id | Ids])-> > PId = CerebralUnitType:gen(self(), node()), > ets:insert(IdsNPIds, {Id, PId}), > ets:insert(IdsNPIds, {PId, Id}), > spawn_CerebralUnits(IdsNPIds, CerebralUnitType, Ids); > spawn_CerebralUnits(_IdsNPIds, _CerebralUnitType, [])-> > true. > > On the line where PId is bound, you have this > "CerebralUnitType:gen(self(), node()),". CerebralUnitType is an atom > "cortex". So > what does :gen do in this case? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From yoursurrogategod@REDACTED Tue Jul 9 18:58:49 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Tue, 9 Jul 2013 12:58:49 -0400 Subject: [erlang-questions] Fwd: What does ":gen" do in this case? In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Yves S. Garret Date: Tue, Jul 9, 2013 at 12:43 PM Subject: What does ":gen" do in this case? To: "erlang-questions@REDACTED" I'm looking at this code: spawn_CerebralUnits(IdsNPIds, CerebralUnitType, [Id | Ids])-> PId = CerebralUnitType:gen(self(), node()), ets:insert(IdsNPIds, {Id, PId}), ets:insert(IdsNPIds, {PId, Id}), spawn_CerebralUnits(IdsNPIds, CerebralUnitType, Ids); spawn_CerebralUnits(_IdsNPIds, _CerebralUnitType, [])-> true. On the line where PId is bound, you have this "CerebralUnitType:gen(self(), node()),". CerebralUnitType is an atom "cortex". So what does :gen do in this case? Ah, just saw your response Thomas. Yes, you're correct. -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Tue Jul 9 19:16:09 2013 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Tue, 9 Jul 2013 20:16:09 +0300 Subject: [erlang-questions] Unable to start mnesia DB. In-Reply-To: References: Message-ID: Hi, The database eventually started. It took 8 hours and 15 minutes to start DB of size ~160 MB (measured using `du data/mnesia/db`). I have done this test in more clean environment: I started the node in the console_clean mode and invoked application:start(mnesia). After the DB become available, I stopped the node by invoking init:stop(). Then I repeated the test. It is still running, but timings look the same (2 cores were busy for 2.5 hours, and after that 1 core is loaded fully, expected for ~5.5 hours). For the second start up attempt the DB has no DCL files, only DCD are left after first load. The test is performed on 64 bit Debian GNU/Linux 6.0.6 (squeeze) running on a virtual server (vmware). I am using Erlang binaries downloaded from the ESL site, version: Erlang R16B01 (erts-5.10.2) [source-bdf5300] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false] Any hints are welcome. Karolis On Mon, Jul 8, 2013 at 4:06 PM, Karolis Petrauskas wrote: > Additional info. The processes with highest reductions are with the > following "current functions": > > {current_function,{disk_log,monitor_request,2}} > {current_function,{disk_log,loop,1}} > > I am not sure if its related, but all the tables that are loaded > successfully, have no DCL files. > > Karolis > > > On Mon, Jul 8, 2013 at 2:16 PM, Karolis Petrauskas > wrote: >> >> Hello, >> >> I cannot load table after erlang node restart. Mnesia prints no >> errors, just is loading the database for very long time. The database is on >> single node, all tables are disc_copy. In total the DB is ~220 MB. On >> startup, the last thing the mnesia prints is: >> >> Getting table my_db_lnk (disc_copies) from disc: local_only >> >> Size of this table (my_db_lnk) on disc is is ~36 MB in the DCD file and ~7 >> MB in he DCL file. This table is not the last table to be loaded in the >> database. IO at that time is almost zero, but CPU is used heavily (200% on >> machine having 2 cores). I was waiting for 3 hours for this table and then >> aborted. Tried to restart the node with the following options: >> >> bin/app console_clean -mnesia debug trace -mnesia auto_repair false >> >> But situation has not been changed. mnesia:info() shows several of my >> tables as active, but not all. After printing "Getting table...", mnesia >> prints "Transaction log dump initiated by time_threshold: already_dumped" >> from time to time. >> >> How could i debug this situation? >> >> Karolis >> > From jleivent@REDACTED Tue Jul 9 19:23:34 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Tue, 09 Jul 2013 13:23:34 -0400 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> <51DB3A45.2070404@comcast.net> Message-ID: <51DC4716.1050900@comcast.net> On 07/08/2013 07:27 PM, Robert Raschke wrote: > Have you considered using a queuing framework, like RabbitMQ? What I'm trying to implement is too low level to make use of RabbitMQ. Ideally, if this works, it would be used within such frameworks. > ... > > Alternatively, you'll probably want to double layer your server, as was > previously suggested. I think that will eventually be my best option - especially as I just learned that Erlang does no inter-node flow control on its own. So, the outer layer will have to both aggregate requests AND flow control them as well, as there may be a very large number of requests coming from a very large number of clients. It's surprising to me that Erlang provides no built-in flow control, since it is using TCP. A gen_server that just receives and aggregates messages on a list could end up using a lot of memory with no way to push back. My gen_server can actually tolerate having incoming requests dropped more than it can tolerate having aggregation use up lots of memory (potentially dying if it runs out, or at least slowing things down). Maybe that means I will need to use gen_UDP - I knew I would eventually switch to UDP to bypass the TCP overhead, but now that looks like a sooner-rather-than-later project. For now, I cheated and used gen_server's internal message format - just because doing that turns out to perturb the code I already have the least. All it takes is: aggregate_requests(L) -> receive {'$gen_call', From, R={request, _}} -> aggregate_requests([{From, R}|L]) after 0 -> L end. That's just too tidy and easy to pass up, even if it does break modularity. I know I'll have to replace this when I face the flow-control issue. -- Jonathan From jleivent@REDACTED Tue Jul 9 20:36:54 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Tue, 09 Jul 2013 14:36:54 -0400 Subject: [erlang-questions] Dialyzer warning about patterns In-Reply-To: <51DBAE3A.5040600@cs.ntua.gr> References: <51DBAE3A.5040600@cs.ntua.gr> Message-ID: <51DC5846.7080504@comcast.net> On 07/09/2013 02:31 AM, Kostis Sagonas wrote: > On 07/09/2013 09:06 AM, Yash Ganthe wrote: >> Hi, >> >> I have the following code: >> >> Port = open_port({spawn,Args},[{packet,2}]), >> >> case is_port(Port) of >> >> true -> >> >> io:format("~n ~p port created successfully ~n",[Port]), >> >> ok; >> >> false -> %%Line 52 >> >> io:format("something went wrong ~n"), >> >> failed >> >> end. >> >> Dialyzer gives the following warning: >> >> mymod.erl:52: The pattern 'false' can never match the type 'true' >> >> The possible results of is_port are true and false considering >> >> is_port(Term) -> boolean() >> >> What do I need to do to get rid of such warnings? > > You simplify your code by taking out the false branch of the case > statement and the case statement itself. Sorry to point it out, but in > its current state the above code is moronic. > > Yes, is_port/1 returns boolean() but that's if you pass it any term(), > not something that is clearly a port(). > > Kostis I have a related issue where the false branch of the case is only false because of the way a macro is defined. If the macro is defined another way, the other branch is false. So, I end up with this dialyzer warning just because I'm using a conditional that evaluates to a constant at preprocessing time. I would think this is pretty common with macros like 'debug'. Is there a way to tell dialyzer to ignore certain instances of warnings? A way to add a comment on that line to tell dialyzer to ignore a particular problem on that line, for instance? -- Jonathan From yoursurrogategod@REDACTED Tue Jul 9 21:02:14 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Tue, 9 Jul 2013 15:02:14 -0400 Subject: [erlang-questions] Why does this compile without warnings/errors? Message-ID: I'm looking at this code: loop(ExoSelf_PId) -> receive {ExoSelf_PId, {Id, Cx_PId, ActuatorName, Fanin_PIds}} -> loop(Id, Cx_PId, ActuatorName, {Fanin_PIds, Fanin_PIds}, []) end. Now, what I'm not quite 'getting' is why the compiler is not complaining about ExoSelf_PId. First of all, it's not used anywhere. Second of all, if it's used, it's used to bind (to an already bound) variable and that's the message that's received to this process. If I were to pass in Foo and then do Foo = 5, the compiler would tell me that that is impossible, so what is different in this case? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianfranco.alongi@REDACTED Tue Jul 9 21:07:06 2013 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Tue, 9 Jul 2013 21:07:06 +0200 Subject: [erlang-questions] Why does this compile without warnings/errors? In-Reply-To: References: Message-ID: 1. it is used - in a match It is used to match against the first bound instance (the one passed in the function head). if loop/1 is called as loop(<0.123.0>) then, this is the <0.123.0> that will be matched against in the receive. 2. The compiler can not know a priori the value of Foo when you pass as a function Argument. So, if you do Foo = 5 , it might be a valid match. On Tue, Jul 9, 2013 at 9:02 PM, Yves S. Garret wrote: > I'm looking at this code: > > loop(ExoSelf_PId) -> > receive > {ExoSelf_PId, {Id, Cx_PId, ActuatorName, Fanin_PIds}} -> > loop(Id, Cx_PId, ActuatorName, {Fanin_PIds, Fanin_PIds}, []) > end. > > Now, what I'm not quite 'getting' is why the compiler is not complaining > about > ExoSelf_PId. First of all, it's not used anywhere. Second of all, if it's > used, it's > used to bind (to an already bound) variable and that's the message that's > received > to this process. If I were to pass in Foo and then do Foo = 5, the compiler > would > tell me that that is impossible, so what is different in this case? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ivan@REDACTED Tue Jul 9 21:11:42 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Tue, 9 Jul 2013 20:11:42 +0100 Subject: [erlang-questions] Why does this compile without warnings/errors? In-Reply-To: References: Message-ID: <0939B611-897C-45F6-A7E0-FADDB215326B@llaisdy.com> This is called a "selective receive". -- festina lente On 9 Jul 2013, at 20:02, "Yves S. Garret" wrote: > I'm looking at this code: > > loop(ExoSelf_PId) -> > receive > {ExoSelf_PId, {Id, Cx_PId, ActuatorName, Fanin_PIds}} -> > loop(Id, Cx_PId, ActuatorName, {Fanin_PIds, Fanin_PIds}, []) > end. > > Now, what I'm not quite 'getting' is why the compiler is not complaining about > ExoSelf_PId. First of all, it's not used anywhere. Second of all, if it's used, it's > used to bind (to an already bound) variable and that's the message that's received > to this process. If I were to pass in Foo and then do Foo = 5, the compiler would > tell me that that is impossible, so what is different in this case? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From eriksoe@REDACTED Tue Jul 9 23:49:21 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 9 Jul 2013 23:49:21 +0200 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: <51DC4716.1050900@comcast.net> References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> <51DB3A45.2070404@comcast.net> <51DC4716.1050900@comcast.net> Message-ID: Don't feel too bad about breaking modularity like that - I must confess I've done that a couple of times. One of those times it even took the form of a "prioritizing_server" overlay on gen_server. Den 09/07/2013 19.23 skrev "Jonathan Leivent" : > On 07/08/2013 07:27 PM, Robert Raschke wrote: > >> Have you considered using a queuing framework, like RabbitMQ? >> > > What I'm trying to implement is too low level to make use of RabbitMQ. > Ideally, if this works, it would be used within such frameworks. > > ... >> >> Alternatively, you'll probably want to double layer your server, as was >> previously suggested. >> > > I think that will eventually be my best option - especially as I just > learned that Erlang does no inter-node flow control on its own. So, the > outer layer will have to both aggregate requests AND flow control them as > well, as there may be a very large number of requests coming from a very > large number of clients. > > It's surprising to me that Erlang provides no built-in flow control, since > it is using TCP. A gen_server that just receives and aggregates messages > on a list could end up using a lot of memory with no way to push back. > > My gen_server can actually tolerate having incoming requests dropped more > than it can tolerate having aggregation use up lots of memory (potentially > dying if it runs out, or at least slowing things down). Maybe that means I > will need to use gen_UDP - I knew I would eventually switch to UDP to > bypass the TCP overhead, but now that looks like a sooner-rather-than-later > project. > > For now, I cheated and used gen_server's internal message format - just > because doing that turns out to perturb the code I already have the least. > All it takes is: > > aggregate_requests(L) -> > receive > {'$gen_call', From, R={request, _}} -> > aggregate_requests([{From, R}|L]) > after 0 -> L > end. > > That's just too tidy and easy to pass up, even if it does break > modularity. I know I'll have to replace this when I face the flow-control > issue. > > -- Jonathan > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Jul 10 00:42:04 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 10 Jul 2013 10:42:04 +1200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <970D171A-05CC-411F-8257-621706B92005@gmail.com> References: <51C1F735.5040706@comcast.net> <7399173B-E72B-409C-BC0F-EDF737CAAB1A@cs.otago.ac.nz> <51C31FE4.4030100@comcast.net> <51C39FA3.5020907@comcast.net> <7B6C2A52-CBEA-4439-B886-788F780D913A@cs.otago.ac.nz> <51C3CB52.70906@comcast.net> <3ED3C4FC-2C69-45EC-9799-87D7F60AC094@cs.otago.ac.nz> <84CCF390-5221-4457-BB76-CB6FA9639DF7@gmail.com> <1a467e5a828a2d39bb1eeb2c2a25fcf7.squirrel@chasm.otago.ac.nz> <08EA433D-351D-4558-A882-2B1A18315D7C@cs.otago.ac.nz> <249F45EB-527A-4D11-907F-5C99C3BF242B@gmail.com> <970D171A-05CC-411F-8257-621706B92005@gmail.com> Message-ID: On 9/07/2013, at 10:00 PM, Anthony Ramine wrote: > Which Erlang clause matches here? None. I was completely out to lunch. > From gomoripeti@REDACTED Wed Jul 10 01:34:29 2013 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Wed, 10 Jul 2013 00:34:29 +0100 Subject: [erlang-questions] cover & export_all In-Reply-To: References: Message-ID: Hi Yuri, You are right, while cover compiling from source works (you can use this as a workaround): > cover:compile(cover_test, UserOptions = [export_all, debug_info]). {ok,cover_test} > cover_test:test(). ok (because UserOptions from the arguments is taken when cover recompiles the instrumented forms) when cover compiling from beam UserOptions = [] is taken. This could be fixed in cover by taking the compile options from the beam file as you assumed how it works. Actually the documentation still says that "Only options defining include file directories and macros are passed to compile:file/2, everything else is ignored." apparently a patch from Tobias Schlager added export_all to the allowed options of cover:compile for the exact same use case as you had. But this is missing from cover:compile_beam. May be I try to come up with a patch br Peter On Tue, Jul 9, 2013 at 11:56 AM, Yuri Lukyanov wrote: > It seems that there is no way to cover-compile modules with export_all. > Here is a simple example: > > cover_test.erl: > > -module(cover_test). > test() -> ok. > > > $ erl > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] > [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> c(cover_test, [debug_info,export_all]). > {ok,cover_test} > 2> cover_test:test(). > ok > 3> cover:compile_beam(cover_test). > {ok,cover_test} > 4> cover_test:test(). > ** exception error: undefined function cover_test:test/0 > 5> > > > Could someone explain why it is like this? Is it done on purpose? > Maybe it's a bug? > The reason I want modules to be cover-compiled with +export_all is > that it is sometimes convinient to have unit tests outside of a > module. Before unit tests are run the modules get compiled with > +export_all for tests to be able to access private functions. But the > situation is that it is not possibe in this case to enable coverage > analysys. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gomoripeti@REDACTED Wed Jul 10 03:19:16 2013 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Wed, 10 Jul 2013 02:19:16 +0100 Subject: [erlang-questions] cover & export_all In-Reply-To: References: Message-ID: Hi OTP team, here is a patch that addresses the problem Yuri described. (I haven't added any tests or docs though) is it true that now you also accept pull requests? git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all.patch br Peter On Wed, Jul 10, 2013 at 12:34 AM, Peti G?m?ri wrote: > Hi Yuri, > > You are right, while cover compiling from source works (you can use this > as a workaround): > > cover:compile(cover_test, UserOptions = [export_all, debug_info]). > {ok,cover_test} > > cover_test:test(). > ok > > (because UserOptions from the arguments is taken when cover recompiles the > instrumented forms) > when cover compiling from beam UserOptions = [] is taken. > This could be fixed in cover by taking the compile options from the beam > file as you assumed how it works. > > Actually the documentation still says that "Only options defining include > file directories and macros are passed to compile:file/2, everything else > is ignored." apparently a patch from Tobias Schlager added export_all to > the allowed options of cover:compile for the exact same use case as you > had. But this is missing from cover:compile_beam. > > May be I try to come up with a patch > > br > Peter > > > > On Tue, Jul 9, 2013 at 11:56 AM, Yuri Lukyanov wrote: > >> It seems that there is no way to cover-compile modules with export_all. >> Here is a simple example: >> >> cover_test.erl: >> >> -module(cover_test). >> test() -> ok. >> >> >> $ erl >> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] >> [async-threads:0] [hipe] [kernel-poll:false] >> >> Eshell V5.9.1 (abort with ^G) >> 1> c(cover_test, [debug_info,export_all]). >> {ok,cover_test} >> 2> cover_test:test(). >> ok >> 3> cover:compile_beam(cover_test). >> {ok,cover_test} >> 4> cover_test:test(). >> ** exception error: undefined function cover_test:test/0 >> 5> >> >> >> Could someone explain why it is like this? Is it done on purpose? >> Maybe it's a bug? >> The reason I want modules to be cover-compiled with +export_all is >> that it is sometimes convinient to have unit tests outside of a >> module. Before unit tests are run the modules get compiled with >> +export_all for tests to be able to access private functions. But the >> situation is that it is not possibe in this case to enable coverage >> analysys. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbin00@REDACTED Wed Jul 10 05:13:30 2013 From: wbin00@REDACTED (Bin Wang) Date: Wed, 10 Jul 2013 11:13:30 +0800 Subject: [erlang-questions] How to handle deps while using rebar to release? In-Reply-To: References: Message-ID: Hi, Andre It works, thanks for the answer! 2013/7/5 Andr? Graf : > Hello > > I usually put {lib_dirs, ["../deps"]}, into the sys-configuration > inside reltool.config. > > Cheers, > Andre > > On 5 July 2013 07:47, Bin Wang wrote: >> I'm using "rebar generate" to handle release, but when I start the >> application, the deps I use will not be found. >> >> I'm wonder how to config rebar.config and reltool.config to handle >> dependencies? Thanks. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From kuangyel2000@REDACTED Wed Jul 10 08:38:05 2013 From: kuangyel2000@REDACTED (kuangyel2000@REDACTED) Date: Tue, 9 Jul 2013 23:38:05 -0700 (PDT) Subject: [erlang-questions] error_loger after run "erl -name foo" In-Reply-To: <9efd6592-6948-4652-9a0f-b34354967f41@googlegroups.com> References: <9efd6592-6948-4652-9a0f-b34354967f41@googlegroups.com> Message-ID: Is it a bug? ? 2013?7?9????UTC+8??12?20?41??kuangy...@REDACTED??? > > I run "erl -name foo" on cmd of windows, it show: > > {error_logger,{{2013,7,8},{15,51,6}},"inet_parse:~p:~p: erroneous line, > SKIPPED~n",["c:/Windows/System32/drivers/etc/hosts",1]} > =INFO REPORT==== 8-Jul-2013::23:51:06 === > inet_parse:"c:/Windows/System32/drivers/etc/hosts":1: erroneous line, > SKIPPED > Eshell V5.10.2 (abort with ^G) > foo@REDACTED> > > but I run "erl -sname foo" > it is OK. it has no error_logger. > > my file: c:/Windows/System32/drivers/etc/hosts > 127.0.0.1 localhost > ::1 localhost > 127.0.0.1 sgwin712.com > 127.0.0.1 sgwin712 > > is it OK? > What is the problem? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredrik@REDACTED Wed Jul 10 10:11:09 2013 From: fredrik@REDACTED (Fredrik) Date: Wed, 10 Jul 2013 10:11:09 +0200 Subject: [erlang-questions] [erlang-patches] cover & export_all In-Reply-To: References: Message-ID: <51DD171D.2090802@erlang.org> On 07/10/2013 03:19 AM, Peti G?m?ri wrote: > Hi OTP team, > > here is a patch that addresses the problem Yuri described. > (I haven't added any tests or docs though) > > is it true that now you also accept pull requests? > > > > git fetch git://github.com/gomoripeti/otp.git > pg?-cover-export-all > > https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all > > https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all.patch > > > > br > Peter > > On Wed, Jul 10, 2013 at 12:34 AM, Peti G?m?ri > wrote: > > Hi Yuri, > > You are right, while cover compiling from source works (you can > use this as a workaround): > > cover:compile(cover_test, UserOptions = [export_all, debug_info]). > {ok,cover_test} > > cover_test:test(). > ok > > (because UserOptions from the arguments is taken when cover > recompiles the instrumented forms) > when cover compiling from beam UserOptions = [] is taken. > This could be fixed in cover by taking the compile options from > the beam file as you assumed how it works. > > Actually the documentation still says that "Only options defining > include file directories and macros are passed to compile:file/2, > everything else is ignored." apparently a patch from Tobias > Schlager added export_all to the allowed options of cover:compile > for the exact same use case as you had. But this is missing from > cover:compile_beam. > > May be I try to come up with a patch > > br > Peter > > > > On Tue, Jul 9, 2013 at 11:56 AM, Yuri Lukyanov > wrote: > > It seems that there is no way to cover-compile modules with > export_all. > Here is a simple example: > > cover_test.erl: > > -module(cover_test). > test() -> ok. > > > $ erl > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] > [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> c(cover_test, [debug_info,export_all]). > {ok,cover_test} > 2> cover_test:test(). > ok > 3> cover:compile_beam(cover_test). > {ok,cover_test} > 4> cover_test:test(). > ** exception error: undefined function cover_test:test/0 > 5> > > > Could someone explain why it is like this? Is it done on purpose? > Maybe it's a bug? > The reason I want modules to be cover-compiled with +export_all is > that it is sometimes convinient to have unit tests outside of a > module. Before unit tests are run the modules get compiled with > +export_all for tests to be able to access private functions. > But the > situation is that it is not possibe in this case to enable > coverage > analysys. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-patches mailing list > erlang-patches@REDACTED > http://erlang.org/mailman/listinfo/erlang-patches Hello Peti, I am getting, fatal: Couldn't find remote ref pg?-cover-export-all when I am trying to fetch. Yes pull requests are accepted. -- BR Fredrik Gustafsson Erlang OTP Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From gomoripeti@REDACTED Wed Jul 10 10:34:38 2013 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Wed, 10 Jul 2013 09:34:38 +0100 Subject: [erlang-questions] [erlang-patches] cover & export_all In-Reply-To: <51DD171D.2090802@erlang.org> References: <51DD171D.2090802@erlang.org> Message-ID: Hi Fredrik, sorry, what about this one (maybe it was a wrong ?) git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all On Wed, Jul 10, 2013 at 9:11 AM, Fredrik wrote: > On 07/10/2013 03:19 AM, Peti G?m?ri wrote: > > Hi OTP team, > > here is a patch that addresses the problem Yuri described. > (I haven't added any tests or docs though) > > is it true that now you also accept pull requests? > > > > git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all > > > https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all > > https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all.patch > > > br > Peter > > On Wed, Jul 10, 2013 at 12:34 AM, Peti G?m?ri wrote: > >> Hi Yuri, >> >> You are right, while cover compiling from source works (you can use >> this as a workaround): >> > cover:compile(cover_test, UserOptions = [export_all, debug_info]). >> {ok,cover_test} >> > cover_test:test(). >> ok >> >> (because UserOptions from the arguments is taken when cover recompiles >> the instrumented forms) >> when cover compiling from beam UserOptions = [] is taken. >> This could be fixed in cover by taking the compile options from the beam >> file as you assumed how it works. >> >> Actually the documentation still says that "Only options defining >> include file directories and macros are passed to compile:file/2, >> everything else is ignored." apparently a patch from Tobias Schlager added >> export_all to the allowed options of cover:compile for the exact same use >> case as you had. But this is missing from cover:compile_beam. >> >> May be I try to come up with a patch >> >> br >> Peter >> >> >> >> On Tue, Jul 9, 2013 at 11:56 AM, Yuri Lukyanov wrote: >> >>> It seems that there is no way to cover-compile modules with export_all. >>> Here is a simple example: >>> >>> cover_test.erl: >>> >>> -module(cover_test). >>> test() -> ok. >>> >>> >>> $ erl >>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] >>> [async-threads:0] [hipe] [kernel-poll:false] >>> >>> Eshell V5.9.1 (abort with ^G) >>> 1> c(cover_test, [debug_info,export_all]). >>> {ok,cover_test} >>> 2> cover_test:test(). >>> ok >>> 3> cover:compile_beam(cover_test). >>> {ok,cover_test} >>> 4> cover_test:test(). >>> ** exception error: undefined function cover_test:test/0 >>> 5> >>> >>> >>> Could someone explain why it is like this? Is it done on purpose? >>> Maybe it's a bug? >>> The reason I want modules to be cover-compiled with +export_all is >>> that it is sometimes convinient to have unit tests outside of a >>> module. Before unit tests are run the modules get compiled with >>> +export_all for tests to be able to access private functions. But the >>> situation is that it is not possibe in this case to enable coverage >>> analysys. >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> > > > _______________________________________________ > erlang-patches mailing listerlang-patches@REDACTED://erlang.org/mailman/listinfo/erlang-patches > > Hello Peti, > I am getting, > fatal: > Couldn't find remote ref pg?-cover-export-all when I am trying to fetch. > Yes pull requests are accepted. > > -- > > BR Fredrik Gustafsson > Erlang OTP Team > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peerst@REDACTED Wed Jul 10 11:17:51 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Wed, 10 Jul 2013 11:17:51 +0200 Subject: [erlang-questions] gen_server aggregating calls/casts References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> <51DB3A45.2070404@comcast.net> <51DC4716.1050900@comcast.net> Message-ID: On 2013-07-09 17:23:34 +0000, Jonathan Leivent said: > My gen_server can actually tolerate having incoming requests dropped > more than it can tolerate having aggregation use up lots of memory > (potentially dying if it runs out, or at least slowing things down). > Maybe that means I will need to use gen_UDP - I knew I would eventually > switch to UDP to bypass the TCP overhead, but now that looks like a > sooner-rather-than-later project. But with gen_tcp you'd use {active, once} anyway (yes?) and this would give you TCP's flow control. > For now, I cheated and used gen_server's internal message format - just > because doing that turns out to perturb the code I already have the > least. All it takes is: > > aggregate_requests(L) -> > receive > {'$gen_call', From, R={request, _}} -> > aggregate_requests([{From, R}|L]) > after 0 -> L > end. > > That's just too tidy and easy to pass up, even if it does break > modularity. I know I'll have to replace this when I face the > flow-control issue. I often just (mis)use the gen module, especially gen:call(), this gives you all the neat monitoring and everything from gen_server. But then I use a plain erlang server (started with proc_lib:spawn_link or even use something like Ulf's plain_fsm) to be able to do selective receive. Selective receive is a very powerful tool in Erlang that you can't use properly if using the gen_* servers. Those are for sequential requests, and often you want this. But if you need selective receive as in your case they just get into the way. I think gen:call should be documented and made official so it is exposed better and can be used with a more quiet state of mind. On the other hand I don't think its likely to change so I use it anyway. Having said this, your possibly unbounded growing mailbox is a bit scary and will at some point probably crash your node. You should combine your selective receives with some way of limiting requests. Jespers https://github.com/jlouis/safetyvalve comes into mind, I think it has pluggable queuing so you might be able to keep your mailbox selective recieve aggregation and just use Safetyvalves CoDel to control it as queue. @Jesper would that be possible with Safetyvalve (to use it with a Erlang mailbox as Queue)? > > -- Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yash_ganthe@REDACTED Wed Jul 10 13:02:11 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Wed, 10 Jul 2013 11:02:11 +0000 Subject: [erlang-questions] internal error in v3_life Message-ID: Hi, I am getting the following error in my Erlang OTP application running with Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] [kernel-poll:false] It is reporting an issue with a case_clause, but is not pointing out the precise case clause. Any pointers will be greatly appreciated. Thanks, Yash ------ (myapp@REDACTED)2> :none: internal error in v3_life;^M crash reason: {{case_clause,^M {'EXIT',^M {undef,^M [{v3_life,module,^M [{k_mdef,[],log_filter,^M [{cutoff_level,0},{module_info,0},{module_info,1}],^M [],^M [{k_fdef,^M {k,[],[],[5,{file,[]}]},^M cutoff_level,0,[],^M {k_return,{k,[],[],[]},[{k_atom,[5,{file,[]}],all}]}},^M {k_fdef,^M {k,[],[],[0,{file,[]}]},^M module_info,0,[],^M {k_enter,^M {k,[],[],[0,{file,[]}]},^M {k_remote,[],^M {k_atom,[0,{file,[]}],erlang},^M {k_atom,[0,{file,[]}],get_module_info},^M 1},^M [{k_atom,[0,{file,[]}],log_filter}]}},^M {k_fdef,^M {k,[],[],[0,{file,[]}]},^M module_info,1,^M [{k_var,[0,{file,[]}],cor0}],^M {k_enter,^M {k,[cor0],[],[0,{file,[]}]},^M {k_remote,[],^M {k_atom,[0,{file,[]}],erlang},^M {k_atom,[0,{file,[]}],get_module_info},^M 2},^M [{k_atom,[0,{file,[]}],log_filter},^M {k_var,[0,{file,[]}],cor0}]}}]},^M [binary,verbose,report_errors,report_warnings,debug_info,^M {i,^M "/home/yganthe/myproj/myproj_libs/erl/erlmq/include"},^M {i,"/opt/yaws/include"}]],^M []},^M {compile,'-select_passes/2-anonymous-2-',2,^M [{file,"compile.erl"},{line,469}]},^M {compile,'-internal_comp/4-anonymous-1-',2,^M [{file,"compile.erl"},{line,269}]},^M {compile,fold_comp,3,[{file,"compile.erl"},{line,287}]},^M {compile,internal_comp,4,[{file,"compile.erl"},{line,271}]},^M {compile,internal,3,[{file,"compile.erl"},{line,246}]}]}}},^M [{compile,'-select_passes/2-anonymous-2-',2,^M [{file,"compile.erl"},{line,469}]},^M {compile,'-internal_comp/4-anonymous-1-',2,^M [{file,"compile.erl"},{line,269}]},^M {compile,fold_comp,3,[{file,"compile.erl"},{line,287}]},^M {compile,internal_comp,4,[{file,"compile.erl"},{line,271}]},^M {compile,internal,3,[{file,"compile.erl"},{line,246}]}]}^M DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Wed Jul 10 13:45:07 2013 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 10 Jul 2013 14:45:07 +0300 Subject: [erlang-questions] internal error in v3_life In-Reply-To: References: Message-ID: <51DD4943.2000108@cs.ntua.gr> On 07/10/13 14:02, Yash Ganthe wrote: > Hi, > > I am getting the following error in my Erlang OTP application running with > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] > [kernel-poll:false] > > It is reporting an issue with a case_clause, but is not pointing out the > precise case clause. You are running a quite old version (current Erlang/OTP is R16B01) so it's quite likely that this problem is fixed by now. Try a newer OTP and if the problem persists, submit (a mimimized) version of your code to erlang-bugs. This needs to be fixed if it's still present. Kostis From mononcqc@REDACTED Wed Jul 10 14:07:01 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 10 Jul 2013 08:07:01 -0400 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> <51DB3A45.2070404@comcast.net> <51DC4716.1050900@comcast.net> Message-ID: <20130710120700.GB25134@ferdmbp.local> On 07/10, Peer Stritzinger wrote: > > Having said this, your possibly unbounded growing mailbox is a bit > scary and will at some point probably crash your node. You should > combine your selective receives with some way of limiting requests. > I have recently extracted a couple of patterns implemented in Heroku's Logplex to make them into a library for a case like that, and dubbed it PO Box (https://github.com/ferd/pobox) as an external mailbox. This model works on the idea of dropping messages that are going over a threshold, and simply reporting the number of losses for the cases where you simply can't tell producers to slow down. The process that uses the PO Box can just request to have all of its mail delivered as a single message, handle them individually, then get the next batch (if any). The pattern has proven itself to be fairly nice when dealing with blocking operations in a critical process -- decoupling the queue management from the blocking operation in two different processes, and it has worked well in production so far (though we've had issues when I attempted to deploy it to a very common component that ended up doubling the number of processes on the server and had the schedulers crapping themselves). It also works well when you can afford to batch operations, such as inserting logs in a DB or writing them to disk, behaving as a buffer. While the pattern has seen plenty of production use, the library hasn't seen itself tried in critical paths yet, but it has been in prod in smaller internal apps and appeared to work very well. Regards, Fred. From tuncer.ayaz@REDACTED Wed Jul 10 15:02:09 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 10 Jul 2013 15:02:09 +0200 Subject: [erlang-questions] internal error in v3_life In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 1:02 PM, Yash Ganthe wrote: > Hi, > > I am getting the following error in my Erlang OTP application > running with > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] > [kernel-poll:false] > > It is reporting an issue with a case_clause, but is not pointing out > the precise case clause. > > Any pointers will be greatly appreciated. > > Thanks, > > Yash > > ------ > > (myapp@REDACTED)2> :none: internal error in v3_life;^M > > crash reason: {{case_clause,^M > {'EXIT',^M > {undef,^M > [{v3_life,module,^M Have you checked that the v3_life module is available? There was a similar problem in a reltool generated release. The issue was that the nodetool escript had a compile directive and the release was missing modules in lib/compile/ebin. So, the compiler was incomplete and unable to function. BTW, another common case where the Erlang compiler is needed in a release are Yaws applications. Relevant discussion(s): https://github.com/rebar/rebar/issues/64 http://erlang.org/pipermail/erlang-questions/2013-April/073185.html http://erlang.org/pipermail/erlang-questions/2013-April/073323.html From yash_ganthe@REDACTED Wed Jul 10 14:11:34 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Wed, 10 Jul 2013 12:11:34 +0000 Subject: [erlang-questions] internal error in v3_life In-Reply-To: <51DD4943.2000108@cs.ntua.gr> References: <51DD4943.2000108@cs.ntua.gr> Message-ID: Hi all, If this is a known issue in previous versions, is there a workaround that I can use? Thanks, Yash -----Original Message----- From: Kostis Sagonas [mailto:kostis@REDACTED] Sent: Wednesday, July 10, 2013 5:15 PM To: erlang-questions@REDACTED; Yash Ganthe Subject: Re: [erlang-questions] internal error in v3_life On 07/10/13 14:02, Yash Ganthe wrote: > Hi, > > I am getting the following error in my Erlang OTP application running > with > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] > [kernel-poll:false] > > It is reporting an issue with a case_clause, but is not pointing out > the precise case clause. You are running a quite old version (current Erlang/OTP is R16B01) so it's quite likely that this problem is fixed by now. Try a newer OTP and if the problem persists, submit (a mimimized) version of your code to erlang-bugs. This needs to be fixed if it's still present. Kostis DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. From hm@REDACTED Wed Jul 10 16:05:51 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 10 Jul 2013 16:05:51 +0200 Subject: [erlang-questions] internal error in v3_life Message-ID: There is no explicit calls to any function in the 'v3_life' module. Its functions are only called via apply. This means that reltool may have problems to determine that the module is used or not. But as the default setting of module inclusion ('mod_cond' set to 'all') in reltool also includes all modules listed in the app file there should be no problem. I suspect that you have set 'mod_cond' to 'derived'. If that is the case, reltool would not include the module. /H?kan On Wed, Jul 10, 2013 at 3:02 PM, Tuncer Ayaz wrote: > On Wed, Jul 10, 2013 at 1:02 PM, Yash Ganthe wrote: >> Hi, >> >> I am getting the following error in my Erlang OTP application >> running with >> >> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] >> [kernel-poll:false] >> >> It is reporting an issue with a case_clause, but is not pointing out >> the precise case clause. >> >> Any pointers will be greatly appreciated. >> >> Thanks, >> >> Yash >> >> ------ >> >> (myapp@REDACTED)2> :none: internal error in v3_life;^M >> >> crash reason: {{case_clause,^M >> {'EXIT',^M >> {undef,^M >> [{v3_life,module,^M > > Have you checked that the v3_life module is available? > > There was a similar problem in a reltool generated release. The issue > was that the nodetool escript had a compile directive and the release > was missing modules in lib/compile/ebin. So, the compiler was > incomplete and unable to function. BTW, another common case where the > Erlang compiler is needed in a release are Yaws applications. > > Relevant discussion(s): > https://github.com/rebar/rebar/issues/64 > http://erlang.org/pipermail/erlang-questions/2013-April/073185.html > http://erlang.org/pipermail/erlang-questions/2013-April/073323.html > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From tuncer.ayaz@REDACTED Wed Jul 10 17:10:48 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 10 Jul 2013 17:10:48 +0200 Subject: [erlang-questions] internal error in v3_life In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 4:05 PM, Hakan Mattsson wrote: > There is no explicit calls to any function in the 'v3_life' module. > Its functions are only called via apply. This means that reltool may > have problems to determine that the module is used or not. > > But as the default setting of module inclusion ('mod_cond' set to > 'all') in reltool also includes all modules listed in the app file > there should be no problem. > > I suspect that you have set 'mod_cond' to 'derived'. If that is the > case, reltool would not include the module. Yes, that's why we've removed {mod_cond, derived} in the rebar template[1] to enable default behavior. After all, a broken a release is worse than one with potentially unused files. For anyone that needs a really lean release there's reltool's slim[2] release feature, but it's not yet integrated[3] into rebar. Hakan, if you take a look at the previously linked thread[4], you will see that with {incl_cond, derived} and without a custom mod_cond setting reltool pulls in 14 additional apps. Shouldn't the same apps already be included due to {incl_cond, derived} regardless of mod_cond? [1] https://github.com/rebar/rebar/blob/master/priv/templates/simplenode.reltool.config [2] https://github.com/rebar/rebar/issues/7 [3] https://github.com/rebar/rebar/pull/52 [4] http://erlang.org/pipermail/erlang-questions/2013-July/074717.html > On Wed, Jul 10, 2013 at 3:02 PM, Tuncer Ayaz wrote: >> On Wed, Jul 10, 2013 at 1:02 PM, Yash Ganthe wrote: >>> Hi, >>> >>> I am getting the following error in my Erlang OTP application >>> running with >>> >>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] >>> [kernel-poll:false] >>> >>> It is reporting an issue with a case_clause, but is not pointing out >>> the precise case clause. >>> >>> Any pointers will be greatly appreciated. >>> >>> Thanks, >>> >>> Yash >>> >>> ------ >>> >>> (myapp@REDACTED)2> :none: internal error in v3_life;^M >>> >>> crash reason: {{case_clause,^M >>> {'EXIT',^M >>> {undef,^M >>> [{v3_life,module,^M >> >> Have you checked that the v3_life module is available? >> >> There was a similar problem in a reltool generated release. The issue >> was that the nodetool escript had a compile directive and the release >> was missing modules in lib/compile/ebin. So, the compiler was >> incomplete and unable to function. BTW, another common case where the >> Erlang compiler is needed in a release are Yaws applications. >> >> Relevant discussion(s): >> https://github.com/rebar/rebar/issues/64 >> http://erlang.org/pipermail/erlang-questions/2013-April/073185.html >> http://erlang.org/pipermail/erlang-questions/2013-April/073323.html From carlosj.gf@REDACTED Wed Jul 10 18:49:21 2013 From: carlosj.gf@REDACTED (=?ISO-8859-1?Q?Carlos_Gonz=E1lez_Florido?=) Date: Wed, 10 Jul 2013 18:49:21 +0200 Subject: [erlang-questions] [ANN] NkSIP Erlang SIP Application Server Message-ID: Hello, I'm happy to announce the first open source release[1] of NkSIP, an Erlang framework to ease the building of SIP applications. SIP is a very powerful, flexible and complex protocol, and NkSIP tries to facilitate the development of any SIP application, including proxies and B2BUAs, allowing the developer to focus on the application logic. The main features are: - Full RFC3261 coverage, including SIP Registrar (RAM storage only). - A written from scratch, fully typed Erlang code easy to understand and extend, with more than 50 unit tests. - Hot core and application code upgrade. - Very few external dependencies: Lager for error logging and Cowboy as TCP/SSL acceptor and Websocket server. - UDP, TCP and TLS transports, capable of handling thousands of simultaneous sessions. - Stateful proxy servers with serial and parallel forking. - Stateless proxy servers, even using TCP/TLS. - Automatic registrations and timed pings. - Dialog and SDP media start and stop detection. - SDP processing utilities. - Simple STUN server (for future SIP Outbound support). - Robust and highly scalable, using all available processor cores. NkSIP is currently alpha quality and not suitable for production, but it is quite robust and has already a decent documentation, a tutorial and some sample applications. There is also some exciting stuff in the radar like SIP Websockets support and a fully distributed version based on Riak Core. Any feedback is welcomed. Thank you! [1] https://github.com/kalta/nksip Carlos Gonz?lez @carlosjgf -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Jul 10 21:10:42 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Wed, 10 Jul 2013 21:10:42 +0200 Subject: [erlang-questions] [ANN] NkSIP Erlang SIP Application Server In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 6:49 PM, Carlos Gonz?lez Florido wrote: > Hello, > > I'm happy to announce the first open source release[1] of NkSIP, an Erlang > framework to ease the building of SIP applications. SIP is a very powerful, > flexible and complex protocol, and NkSIP tries to facilitate the development > of any SIP application, including proxies and B2BUAs, allowing the developer > to focus on the application logic. > Hi Carlos, this is very interesting and certainly a lot of work. Can you tell us more where this project is going? Is there (will be there) a company backing this? In my perspective, VoIP world certainly needs something better than Asterisk... So this is a certainly a great job, but what next? Also, how exactly do you plan to add distribution using riak_core? Which parts of it are you planning to reuse and how? That aside, why is the main applicaiton behind "nksip" folder? Why not plainly "src", "test" in top-level folder? I did not look at the codebase/tests yet (I will!), but want to ask this early. :-) -- Motiejus Jak?tys From tom@REDACTED Wed Jul 10 23:23:47 2013 From: tom@REDACTED (Tom Samplonius) Date: Wed, 10 Jul 2013 14:23:47 -0700 Subject: [erlang-questions] [ANN] NkSIP Erlang SIP Application Server In-Reply-To: References: Message-ID: <6C0EEA62-51A8-4EA0-B4C5-75CEDB09F37C@samplonius.org> On 2013-07-10, at 12:10 PM, Motiejus Jak?tys wrote: > Hi Carlos, > > this is very interesting and certainly a lot of work. Can you tell us > more where this project is going? Is there (will be there) a company > backing this? In my perspective, VoIP world certainly needs something > better than Asterisk... So this is a certainly a great job, but what > next? ... All of bigger installations are using FreeSwitch (http://www.freeswitch.org/) instead of Asterisk. The README says: NkSIP is a pure SIP framework, so it does not support any real RTP media processing it can't record a call, host an audio conference or transcode. These type of tasks should be done with a SIP media server, like Freeswitch or Asterisk. So, it looks like more of a replacement for OpenSIPS, which is often used a SIP load balancer, or session border controller. If anything, NkSIP looks like a good component for Kazoo (http://2600hz.com/platform.html, https://github.com/2600hz/kazoo). The Kazoo platform uses OpenSIPS and FreeSwitch, but most of the other stuff is Erlang (RabbitMQ, BigCouch, etc.). Rather than using RabbitMQ to distribute SIP events from OpenSIPS/FreeSwitch between nodes and zones, using a SIP registrar on top of Riak seems a lot simpler. Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Jul 11 00:03:18 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 11 Jul 2013 00:03:18 +0200 Subject: [erlang-questions] [ann] Barrel 2.0 has been released Message-ID: Happy to announce that barrel 2.0 has been released. Barrel l is a generic TCP acceptor pool with low latency in #Erlang. See the release announcement here: https://github.com/benoitc/barrel/releases/2.0 Enjoy! - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From goofan.su@REDACTED Thu Jul 11 07:48:08 2013 From: goofan.su@REDACTED (goofansu) Date: Thu, 11 Jul 2013 13:48:08 +0800 Subject: [erlang-questions] ALIVE2_REQ from non local address Message-ID: <846D07BB2FDC4D0C933635E3A8F31B59@gmail.com> Hi, I encountered this message when starting nodes with "erl -name test@REDACTED" or "erl -sname test". $ epmd -d epmd: Thu Jul 11 04:01:32 2013: epmd running - daemon = 0 epmd: Thu Jul 11 04:01:37 2013: ** got ALIVE2_REQ epmd: Thu Jul 11 04:01:37 2013: ALIVE2_REQ from non local address $ cat /etc/hosts 127.0.0.1 localhost li498-7 ::1 localhost li498-7 $ hostname li498-7 There is no iptables rules. Here is the dump file: http://d.pr/f/ctn7 -- Kind Regards goofansu -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbin00@REDACTED Thu Jul 11 08:41:05 2013 From: wbin00@REDACTED (Bin Wang) Date: Thu, 11 Jul 2013 14:41:05 +0800 Subject: [erlang-questions] Will terminate of gen_server be called in this situation? Message-ID: Hi, I found my terminate in gen_server is not been called in this situation, did I miss something? Part of the code is: handle_info(stop, State) -> {stop, normal, State}. terminate(_, _) -> io:format(user, "termiate~n", []). And I have set process_flag(trap_exit, true) in "init", and this gen_server is under a supervisor tree. It is a simple_one_for_one supervisor and the spec is like this, "client" here is the module name: {ok, {{simple_one_for_one, 3, 10}, [{ client, {client, start, []}, transient, 10000, worker, [client] } ]}}. When I send some info with stop to the gen_server, it did stop, but the terminate function is not called. Anybody have some ideas? Thanks! From goofan.su@REDACTED Thu Jul 11 09:48:13 2013 From: goofan.su@REDACTED (goofansu) Date: Thu, 11 Jul 2013 15:48:13 +0800 Subject: [erlang-questions] =?utf-8?Q?=E5=9B=9E=E5=A4=8D=EF=BC=9A_?= Will terminate of gen_server be called in this situation? In-Reply-To: References: Message-ID: <5AFDE1E6930248D3827D9993D264D33C@gmail.com> Hi, I have the same question before and you can see my question here: http://stackoverflow.com/questions/16693922/simple-one-for-one-can-only-be-terminated-if-appointed-shutdown-strategy-to-brut. As you've described, I created an example on github: https://github.com/goofansu/sofo-example, and can print "terminate" as expected. My operations like following: $ erl -pa ebin -s myapp Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) 1> {ok, ChildPid} = myapp_sup:start_child(). {ok,<0.37.0>} 2> ChildPid ! stop. terminate stop 3> If you use myapp_sup:stop_child(ChildPid). It will terminate directly. -- Kind Regards goofansu ? 2013?7?11???????2:41?Bin Wang ??? > Hi, > > I found my terminate in gen_server is not been called in this > situation, did I miss something? > > Part of the code is: > > handle_info(stop, State) -> > {stop, normal, State}. > > terminate(_, _) -> > io:format(user, "termiate~n", []). > > And I have set process_flag(trap_exit, true) in "init", and this > gen_server is under a supervisor tree. It is a simple_one_for_one > supervisor and the spec is like this, "client" here is the module > name: > > {ok, {{simple_one_for_one, 3, 10}, [{ > client, > {client, start, []}, > transient, > 10000, > worker, > [client] > } > ]}}. > > When I send some info with stop to the gen_server, it did stop, but > the terminate function is not called. Anybody have some ideas? Thanks! > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbin00@REDACTED Thu Jul 11 09:48:50 2013 From: wbin00@REDACTED (Bin Wang) Date: Thu, 11 Jul 2013 15:48:50 +0800 Subject: [erlang-questions] Will terminate of gen_server be called in this situation? In-Reply-To: References: Message-ID: Hi, everyone Never mind this. I do some tests and it should work. I have write some wrong code elsewhere. 2013/7/11 Bin Wang : > Hi, > > I found my terminate in gen_server is not been called in this > situation, did I miss something? > > Part of the code is: > > handle_info(stop, State) -> > {stop, normal, State}. > > terminate(_, _) -> > io:format(user, "termiate~n", []). > > And I have set process_flag(trap_exit, true) in "init", and this > gen_server is under a supervisor tree. It is a simple_one_for_one > supervisor and the spec is like this, "client" here is the module > name: > > {ok, {{simple_one_for_one, 3, 10}, [{ > client, > {client, start, []}, > transient, > 10000, > worker, > [client] > } > ]}}. > > When I send some info with stop to the gen_server, it did stop, but > the terminate function is not called. Anybody have some ideas? Thanks! From yash_ganthe@REDACTED Thu Jul 11 11:50:12 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 11 Jul 2013 09:50:12 +0000 Subject: [erlang-questions] internal error in v3_life In-Reply-To: References: Message-ID: Thanks Tuncer. That was it. The module was not packaged by reltool. Got it working now. -----Original Message----- From: Tuncer Ayaz [mailto:tuncer.ayaz@REDACTED] Sent: Wednesday, July 10, 2013 6:32 PM To: Yash Ganthe Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] internal error in v3_life On Wed, Jul 10, 2013 at 1:02 PM, Yash Ganthe wrote: > Hi, > > I am getting the following error in my Erlang OTP application running > with > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] > [kernel-poll:false] > > It is reporting an issue with a case_clause, but is not pointing out > the precise case clause. > > Any pointers will be greatly appreciated. > > Thanks, > > Yash > > ------ > > (myapp@REDACTED)2> :none: internal error in v3_life;^M > > crash reason: {{case_clause,^M > {'EXIT',^M > {undef,^M > [{v3_life,module,^M Have you checked that the v3_life module is available? There was a similar problem in a reltool generated release. The issue was that the nodetool escript had a compile directive and the release was missing modules in lib/compile/ebin. So, the compiler was incomplete and unable to function. BTW, another common case where the Erlang compiler is needed in a release are Yaws applications. Relevant discussion(s): https://github.com/rebar/rebar/issues/64 http://erlang.org/pipermail/erlang-questions/2013-April/073185.html http://erlang.org/pipermail/erlang-questions/2013-April/073323.html DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. From carlosj.gf@REDACTED Thu Jul 11 12:08:54 2013 From: carlosj.gf@REDACTED (=?ISO-8859-1?Q?Carlos_Gonz=E1lez_Florido?=) Date: Thu, 11 Jul 2013 12:08:54 +0200 Subject: [erlang-questions] [ANN] NkSIP Erlang SIP Application Server In-Reply-To: References: Message-ID: Hi Motiejus Thank you! Of course there will be a company backing and supporting the project, but my priority right now is to see how useful it is in the real life, test it by as much people as possible, find and correct all important bugs and add the most demanded missing features (As you know, writing a reliable 100% compliant SIP server is quite a complex thing, even with Erlang). My idea with Riak Core (and a lot of code is already written) is to achieve something similar to what Riak database does: - You can start with a single node, it receives SIP requests, reply, etc. - You add a second node at any moment, and "magically" half of the work is now processed at each of the nodes. You add a third, node, and one third is processed at each one. - You can remove a node at any time. If one of the nodes fails, the rest of the nodes take care of its load while it is down, all of it automatically. In the current version, NkSIP starts by default 8 queues. Using a consistent hash over the Call-ID, each request is sent to one of these queues (the same Call-ID always goes to the same queue). Each queue decides to start a brand new Erlang process to manage the request or wait (if a previous request with the same Call-ID is active, think in INVITE+ACK+BYE). With the Riak Core version, I will use vnodes instead of the queues. Riak Core takes care of the distribution of vnodes among all active nodes, so every request will be sent to the same vnode, depending on its Call-ID. When nodes are added or removed, Riak Core moves the vnode to another physical node. In respect to the question about the directory structure, you are right. The reason was having a samples directory with sample applications, but it is probably a good idea to move the sample applications out into a different repository (much like any real application made with NkSIP) On Wed, Jul 10, 2013 at 9:10 PM, Motiejus Jak?tys wrote: > On Wed, Jul 10, 2013 at 6:49 PM, Carlos Gonz?lez Florido > wrote: > > Hello, > > > > I'm happy to announce the first open source release[1] of NkSIP, an > Erlang > > framework to ease the building of SIP applications. SIP is a very > powerful, > > flexible and complex protocol, and NkSIP tries to facilitate the > development > > of any SIP application, including proxies and B2BUAs, allowing the > developer > > to focus on the application logic. > > > > Hi Carlos, > > this is very interesting and certainly a lot of work. Can you tell us > more where this project is going? Is there (will be there) a company > backing this? In my perspective, VoIP world certainly needs something > better than Asterisk... So this is a certainly a great job, but what > next? > > Also, how exactly do you plan to add distribution using riak_core? > Which parts of it are you planning to reuse and how? > > That aside, why is the main applicaiton behind "nksip" folder? Why not > plainly "src", "test" in top-level folder? I did not look at the > codebase/tests yet (I will!), but want to ask this early. :-) > > -- > Motiejus Jak?tys > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosj.gf@REDACTED Thu Jul 11 12:39:32 2013 From: carlosj.gf@REDACTED (=?ISO-8859-1?Q?Carlos_Gonz=E1lez_Florido?=) Date: Thu, 11 Jul 2013 12:39:32 +0200 Subject: [erlang-questions] [ANN] NkSIP Erlang SIP Application Server In-Reply-To: <6C0EEA62-51A8-4EA0-B4C5-75CEDB09F37C@samplonius.org> References: <6C0EEA62-51A8-4EA0-B4C5-75CEDB09F37C@samplonius.org> Message-ID: Yes, you are right. NkSIP is clearly an OpenSIPS/Kamailio replacement (but NkSIP is not yet even near the maturity of those projects, and I'm not a SIP guru with the experience of the people backing them). The reason to develop NkSIP instead of using OpenSIPS/Kamailio/etc. is that (at least for me) it is quite complex and not "natural" to develop applications with these SIP servers. I don't really fully support their architecture with hundreds of modules... For me it should be more like an application flow. You don't need a Least Cost Route, Call Detail Record or distribution module, it is much easier to do it in your application code within 50 Erlang code lines. At least using Erlang, I think it makes no sense to have a lot of modules to connect to a lot of databases, just implement the callbacks you need and use whatever backend you want. I'm not saying that being modular is a bad thing (NkSIP will probably have some kind of modules or plugins in the future), but they should be much more horizontal. This approach is much simpler, now I have to prove it actually works ;-) Another reason is that SIP is evolving quickly, and it is much, much easier to add functionality using Erlang and a OTP structure that in low level C modules. Of course OpenSIPS and Kamailio should perform much better, but I suppose this advantage is not so clear when you start using many processor cores. Finally, NkSIP will have soon a fully distributed, highly available solution. While you can build such a solution with those projects, it is going to be much more difficult. NkSIP looks like a perfect fit for Kazoo! On Wed, Jul 10, 2013 at 11:23 PM, Tom Samplonius wrote: > > On 2013-07-10, at 12:10 PM, Motiejus Jak?tys > wrote: > > Hi Carlos, > > this is very interesting and certainly a lot of work. Can you tell us > more where this project is going? Is there (will be there) a company > backing this? In my perspective, VoIP world certainly needs something > better than Asterisk... So this is a certainly a great job, but what > next? > > ... > > All of bigger installations are using FreeSwitch ( > http://www.freeswitch.org/) instead of Asterisk. The README says: > > NkSIP is a pure SIP framework, so it *does not support any real RTP media > processing* it can't record a call, host an audio conference or > transcode. These type of tasks should be done with a SIP media server, like > Freeswitch or Asterisk > . > > > So, it looks like more of a replacement for OpenSIPS, which is often > used a SIP load balancer, or session border controller. > > If anything, NkSIP looks like a good component for Kazoo ( > http://2600hz.com/platform.html, https://github.com/2600hz/kazoo). The > Kazoo platform uses OpenSIPS and FreeSwitch, but most of the other stuff is > Erlang (RabbitMQ, BigCouch, etc.). Rather than using RabbitMQ to > distribute SIP events from OpenSIPS/FreeSwitch between nodes and zones, > using a SIP registrar on top of Riak seems a lot simpler. > > > Tom > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yash_ganthe@REDACTED Thu Jul 11 15:56:30 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 11 Jul 2013 13:56:30 +0000 Subject: [erlang-questions] Erlang terminating with dump Message-ID: Hi, My Erlang setup is: Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:1:1] [async-threads:0] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) 1> erlang:system_info(smp_support). true I have an application that uses an NIF to do certain things like read from Wepsphere MQ. The application runs smoothly for a while, reading messages from MQ and processing them, and all of a sudden comes to a halt. Using the following, I have ensured that erl creates a dump export RUN_ERL_LOG_MAXSIZE=10000000 export RUN_ERL_LOG_GENERATIONS=20 export ERL_CRASH_DUMP_SECONDS=-1 export ERL_CRASH_DUMP_NICE=1 The dump says: Kernel pid terminated (application_controller) ({application_terminated,log4erl,shutdown}) The part of the NIF which reads from MQ has exception handling that should print an error message. This does not get printed. I am not sure if the NIF is causing the issue. Is there any better way to identify the root cause of the problem? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From valery.meleshkin@REDACTED Thu Jul 11 17:25:37 2013 From: valery.meleshkin@REDACTED (Valery Meleshkin) Date: Thu, 11 Jul 2013 19:25:37 +0400 Subject: [erlang-questions] Interop between net_kernel and erts Message-ID: <86F6AD6C-C43B-4D9C-B309-A957D7B4C6DC@gmail.com> Hi, I am wondering how the net_kernel and its subordinated *_dist are wired with ERTS. My current findings are: 1) net_kernel and *_dist (which in turn uses dist_util and erl_epmd) do local naming setup (erlang:setnode/2), epmd registration/resolution and handshake by themselves. 2) everything is implemented on top of the gen_tcp which is a part of the kernel. gen_tcp is implemented on top of the prim_inet which is a part of so-called "preloaded". 3) I spotted call to the erlang:setnode/3 in dist_util and I suppose it sets a correspondence between a port and a node for the erlang:send. And here are my questions: 1) Are my findings correct? 2) Is there any protocol between net_kernel (with its neighbors) and ERTS or every known message comes from the "user-space"? 3) Although controlling process of the internode socket sets inet:setopts(S, [{active, true}, {deliver, port}, {packet, 4}, nodelay()]) after handshake and setnode/3 (see dist_util:connection and FPostNodeup call) there is no data-message handling code inside dist_util:con_loop. So, are those data-messages handled somehow magically within ERTS? Thanks, Valery From jleivent@REDACTED Thu Jul 11 20:47:18 2013 From: jleivent@REDACTED (Jonathan Leivent) Date: Thu, 11 Jul 2013 14:47:18 -0400 Subject: [erlang-questions] gen_server aggregating calls/casts In-Reply-To: <20130710120700.GB25134@ferdmbp.local> References: <51DB13A1.5010001@comcast.net> <51DB1A33.2020402@comcast.net> <51DB3A45.2070404@comcast.net> <51DC4716.1050900@comcast.net> <20130710120700.GB25134@ferdmbp.local> Message-ID: <51DEFDB6.9070205@comcast.net> On 07/10/2013 08:07 AM, Fred Hebert wrote: > On 07/10, Peer Stritzinger wrote: >> >> Having said this, your possibly unbounded growing mailbox is a bit >> scary and will at some point probably crash your node. You should >> combine your selective receives with some way of limiting requests. >> > > I have recently extracted a couple of patterns implemented in Heroku's > Logplex to make them into a library for a case like that, and dubbed it > PO Box (https://github.com/ferd/pobox) as an external mailbox. Well, this looks like it might be just what I need! Thanks muchly! I will check it out more when I'm up to that point. -- Jonathan From baliulia@REDACTED Fri Jul 12 09:12:52 2013 From: baliulia@REDACTED (=?UTF-8?B?SWduYXMgVnnFoW5pYXVza2Fz?=) Date: Fri, 12 Jul 2013 09:12:52 +0200 Subject: [erlang-questions] Unable to start mnesia DB. In-Reply-To: References: Message-ID: <51DFAC74.9060401@gmail.com> On 07/09/2013 07:16 PM, Karolis Petrauskas wrote: > After the DB become available, I stopped the node by invoking > init:stop(). Then I repeated the test. It is still running, but > timings look the same (2 cores were busy for 2.5 hours, and after > that 1 core is loaded fully, expected for ~5.5 hours). For the second > start up attempt the DB has no DCL files, only DCD are left after > first load. A poke in the dark: do you maybe have lots of secondary indices? These will not be stored on disk and so will be recreated each time the tables are loaded, I've experienced very long mnesia startup times in such cases. Otherwise -- no idea. -- Ignas From john@REDACTED Fri Jul 12 15:24:42 2013 From: john@REDACTED (John Koenig) Date: Fri, 12 Jul 2013 08:24:42 -0500 Subject: [erlang-questions] Announcing Ymir: A Distributed, 3D Game Engine Written in Erlang Message-ID: <51E0039A.5040602@calledshot.org> I wanted to bring to light, Ymir, which is currently the focus of my PhD pursuits. The engine is coming along nicely and I would like to share some early performance results with the erlang community. Below you will find links to a sample scene simulated in Ymir (rendered using Ymir's built-in support for Mitsuba) as well as a selection of performance results: Video - https://www.youtube.com/watch?v=uYH5dux3AfU Results - Generated while simulating the scene above on an Intel(R) Xeon(R) CPU E5-2670 (Sandy Bridge) @ 2.60GHz. * % Speedup vs # of Cores: http://goo.gl/y7gVz * Frame Rate (16 cores): http://goo.gl/lbxD6 * Simulation Time vs # of Cores: http://goo.gl/e1Vc7 I am excited by these early results and I hope some of you are too. If you have any feedback or constructive criticism, please don't be shy. We also recently launched out first game project to be hosted on Ymir. If you are curious, head on over to: http://igg.me/at/TEAoW-PD/x/2446696 Thanks! From barcojie@REDACTED Fri Jul 12 15:34:47 2013 From: barcojie@REDACTED (Barco You) Date: Fri, 12 Jul 2013 21:34:47 +0800 Subject: [erlang-questions] Announcing Ymir: A Distributed, 3D Game Engine Written in Erlang In-Reply-To: <51E0039A.5040602@calledshot.org> References: <51E0039A.5040602@calledshot.org> Message-ID: Not Found got when visiting goo.gl On 12 Jul 2013 21:28, "John Koenig" wrote: > I wanted to bring to light, Ymir, which is currently the focus of my PhD > pursuits. The engine is coming along nicely and I would like to share > some early performance results with the erlang community. Below you will > find links to a sample scene simulated in Ymir (rendered using Ymir's > built-in support for Mitsuba) as well as a selection of performance results: > > Video - https://www.youtube.com/watch?**v=uYH5dux3AfU > > Results - > > Generated while simulating the scene above on an Intel(R) Xeon(R) CPU > E5-2670 (Sandy Bridge) @ 2.60GHz. > > * % Speedup vs # of Cores: http://goo.gl/y7gVz > * Frame Rate (16 cores): http://goo.gl/lbxD6 > * Simulation Time vs # of Cores: http://goo.gl/e1Vc7 > > I am excited by these early results and I hope some of you are too. If > you have any feedback or constructive criticism, please don't be shy. > > We also recently launched out first game project to be hosted on Ymir. If > you are curious, head on over to: http://igg.me/at/TEAoW-PD/x/**2446696 > > Thanks! > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john@REDACTED Fri Jul 12 15:42:51 2013 From: john@REDACTED (John Koenig) Date: Fri, 12 Jul 2013 08:42:51 -0500 Subject: [erlang-questions] Announcing Ymir: A Distributed, 3D Game Engine Written in Erlang In-Reply-To: References: <51E0039A.5040602@calledshot.org> Message-ID: <51E007DB.6020907@calledshot.org> Oh, they seem to be working for me. Perhaps I had cached versions. The long versions follow: % Speedup vs # of Cores: speedup Frame Rate (16 cores): framerate Simulation Time vs # of Cores: simulation_time If those fail to work too, then the graphs are also present in the Gallery section of the linked indiegogo page. Sorry about that! On 07/12/2013 08:34 AM, Barco You wrote: > > Not Found got when visiting goo.gl > > On 12 Jul 2013 21:28, "John Koenig" > wrote: > > I wanted to bring to light, Ymir, which is currently the focus of > my PhD pursuits. The engine is coming along nicely and I would > like to share some early performance results with the erlang > community. Below you will find links to a sample scene simulated > in Ymir (rendered using Ymir's built-in support for Mitsuba) as > well as a selection of performance results: > > Video - https://www.youtube.com/watch?v=uYH5dux3AfU > > Results - > > Generated while simulating the scene above on an Intel(R) Xeon(R) > CPU E5-2670 (Sandy Bridge) @ 2.60GHz. > > * % Speedup vs # of Cores: http://goo.gl/y7gVz > * Frame Rate (16 cores): http://goo.gl/lbxD6 > * Simulation Time vs # of Cores: http://goo.gl/e1Vc7 > > I am excited by these early results and I hope some of you are > too. If you have any feedback or constructive criticism, please > don't be shy. > > We also recently launched out first game project to be hosted on > Ymir. If you are curious, head on over to: > http://igg.me/at/TEAoW-PD/x/2446696 > > Thanks! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Fri Jul 12 16:00:59 2013 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 12 Jul 2013 16:00:59 +0200 Subject: [erlang-questions] Announcing Ymir: A Distributed, 3D Game Engine Written in Erlang In-Reply-To: <51E007DB.6020907@calledshot.org> References: <51E0039A.5040602@calledshot.org> <51E007DB.6020907@calledshot.org> Message-ID: I just pledged 100$. You guys are doing amazing job in uncharted territories! Keep on! Graphs (and some other graphics) are not showing for me in the indiegogo page. On Fri, Jul 12, 2013 at 3:42 PM, John Koenig wrote: > Oh, they seem to be working for me. Perhaps I had cached versions. The > long versions follow: > > % Speedup vs # of Cores: speedup > Frame Rate (16 cores): framerate > Simulation Time vs # of Cores: simulation_time > > If those fail to work too, then the graphs are also present in the Gallery > section of the linked indiegogo page. > > Sorry about that! > > > On 07/12/2013 08:34 AM, Barco You wrote: > > Not Found got when visiting goo.gl > > On 12 Jul 2013 21:28, "John Koenig" wrote: >> >> I wanted to bring to light, Ymir, which is currently the focus of my PhD >> pursuits. The engine is coming along nicely and I would like to share some >> early performance results with the erlang community. Below you will find >> links to a sample scene simulated in Ymir (rendered using Ymir's built-in >> support for Mitsuba) as well as a selection of performance results: >> >> Video - https://www.youtube.com/watch?v=uYH5dux3AfU >> >> Results - >> >> Generated while simulating the scene above on an Intel(R) Xeon(R) CPU >> E5-2670 (Sandy Bridge) @ 2.60GHz. >> >> * % Speedup vs # of Cores: http://goo.gl/y7gVz >> * Frame Rate (16 cores): http://goo.gl/lbxD6 >> * Simulation Time vs # of Cores: http://goo.gl/e1Vc7 >> >> I am excited by these early results and I hope some of you are too. If >> you have any feedback or constructive criticism, please don't be shy. >> >> We also recently launched out first game project to be hosted on Ymir. If >> you are curious, head on over to: http://igg.me/at/TEAoW-PD/x/2446696 >> >> Thanks! >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From john@REDACTED Fri Jul 12 16:07:59 2013 From: john@REDACTED (John Koenig) Date: Fri, 12 Jul 2013 09:07:59 -0500 Subject: [erlang-questions] Announcing Ymir: A Distributed, 3D Game Engine Written in Erlang In-Reply-To: References: <51E0039A.5040602@calledshot.org> <51E007DB.6020907@calledshot.org> Message-ID: <51E00DBF.2080801@calledshot.org> Thanks for your support, Gleb! I will look into the issues on the indiegogo page. I find the indiegogo workflow has room for improvement. John On 07/12/2013 09:00 AM, Gleb Peregud wrote: > I just pledged 100$. You guys are doing amazing job in uncharted > territories! Keep on! > > Graphs (and some other graphics) are not showing for me in the indiegogo page. > > On Fri, Jul 12, 2013 at 3:42 PM, John Koenig wrote: >> Oh, they seem to be working for me. Perhaps I had cached versions. The >> long versions follow: >> >> % Speedup vs # of Cores: speedup >> Frame Rate (16 cores): framerate >> Simulation Time vs # of Cores: simulation_time >> >> If those fail to work too, then the graphs are also present in the Gallery >> section of the linked indiegogo page. >> >> Sorry about that! >> >> >> On 07/12/2013 08:34 AM, Barco You wrote: >> >> Not Found got when visiting goo.gl >> >> On 12 Jul 2013 21:28, "John Koenig" wrote: >>> I wanted to bring to light, Ymir, which is currently the focus of my PhD >>> pursuits. The engine is coming along nicely and I would like to share some >>> early performance results with the erlang community. Below you will find >>> links to a sample scene simulated in Ymir (rendered using Ymir's built-in >>> support for Mitsuba) as well as a selection of performance results: >>> >>> Video - https://www.youtube.com/watch?v=uYH5dux3AfU >>> >>> Results - >>> >>> Generated while simulating the scene above on an Intel(R) Xeon(R) CPU >>> E5-2670 (Sandy Bridge) @ 2.60GHz. >>> >>> * % Speedup vs # of Cores: http://goo.gl/y7gVz >>> * Frame Rate (16 cores): http://goo.gl/lbxD6 >>> * Simulation Time vs # of Cores: http://goo.gl/e1Vc7 >>> >>> I am excited by these early results and I hope some of you are too. If >>> you have any feedback or constructive criticism, please don't be shy. >>> >>> We also recently launched out first game project to be hosted on Ymir. If >>> you are curious, head on over to: http://igg.me/at/TEAoW-PD/x/2446696 >>> >>> Thanks! >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From moxford@REDACTED Sat Jul 13 02:58:07 2013 From: moxford@REDACTED (Mike Oxford) Date: Fri, 12 Jul 2013 17:58:07 -0700 Subject: [erlang-questions] jiffy encoding Message-ID: >From the docs... {[{foo, bar}]} -> {"foo": "bar"} -> {[{<<"foo">>, <<"bar">>}]} Why does jiffy need the extra "tuple list" syntax? Anyone happen to know? jiffy:encode({x,y}) fails. jiffy:encode({[{x,y}]}) => "{\"x\":\"y\"}" From moxford@REDACTED Sat Jul 13 06:33:05 2013 From: moxford@REDACTED (Mike Oxford) Date: Fri, 12 Jul 2013 21:33:05 -0700 Subject: [erlang-questions] Bug in erlang? Message-ID: Maybe idiosyncrasy? >From the command-line, everything is expected and looks correct... list_to_float(lists:flatten(io_lib:format("~.2f",[3.4]))). 3.4 list_to_float(lists:flatten(io_lib:format("~.2f",[1.000000e+00]))). 1.00 But inside of a .beam'd module... erlang:display(V), erlang:display(lists:flatten(io_lib:format("~.2f",[V]))), V2 = list_to_float(lists:flatten(io_lib:format("~.2f",[V]))), erlang:display(V2) gives... 1.000000e+00 "1.00" 1.000000e+00 So the interesting part is that it (list_to_float) works as expected in the shell, but not in a compiled .beam file. Note that float_to_list/2 will take options, but there is not list_to_float/2 with options to specify format. Curious. -mox From goofan.su@REDACTED Sat Jul 13 08:34:16 2013 From: goofan.su@REDACTED (goofansu) Date: Sat, 13 Jul 2013 14:34:16 +0800 Subject: [erlang-questions] ERL_MAX_PORTS has no effect Message-ID: <8293812F868F43A99270A2BE49389225@gmail.com> I encounted "fd=1024 is larger than the largest allowed fd=1023", then I set the ERL_MAX_PORTS variable when start the erlang vm. But it takes no effect. $ erl -env ERL_MAX_PORTS 4096 Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) 1> erlang:system_info(check_io). [{name,erts_poll}, {primary,select}, {fallback,false}, {kernel_poll,false}, {memory_size,6344}, {total_poll_set_size,2}, {lazy_updates,true}, {pending_updates,0}, {batch_updates,false}, {concurrent_updates,false}, {max_fds,1024}] 2> The max_fds is always 1024. -- Kind Regards goofansu -------------- next part -------------- An HTML attachment was scrubbed... URL: From aggelgian@REDACTED Sat Jul 13 09:05:00 2013 From: aggelgian@REDACTED (Aggelos Giantsios) Date: Sat, 13 Jul 2013 10:05:00 +0300 Subject: [erlang-questions] jiffy encoding In-Reply-To: References: Message-ID: On Sat, Jul 13, 2013 at 3:58 AM, Mike Oxford wrote: > From the docs... > {[{foo, bar}]} -> {"foo": "bar"} -> {[{<<"foo">>, <<"bar">>}]} > > Why does jiffy need the extra "tuple list" syntax? > Anyone happen to know? > Hello! I believe the notion is that an orddict is converted to a JSON dictionary. I guess in order to distinguish an orddict from a regular list, they wrap the orddict in a tuple. Aggelos -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Sat Jul 13 14:08:39 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sat, 13 Jul 2013 14:08:39 +0200 Subject: [erlang-questions] ERL_MAX_PORTS has no effect In-Reply-To: <8293812F868F43A99270A2BE49389225@gmail.com> References: <8293812F868F43A99270A2BE49389225@gmail.com> Message-ID: <8A30B7E3-77EC-47CF-B6C6-92AAC2F1D708@gmail.com> Hello, ERL_MAX_PORTS sets the maximum number of running ports. Not the maximum number of file descriptors. I guess you are running OS X which has a broken poll implementation and Erlang is forced to use select(), which is limited to FD_SETSIZE file descriptors. Fortunately, OS X's select() implementation can also be made to support an infinite number of file descriptors through a macro which I taught Erlang/OTP to use. Upgrade to R16B01 and you should be fine. Regards, -- Anthony Ramine Le 13 juil. 2013 ? 08:34, goofansu a ?crit : > I encounted "fd=1024 is larger than the largest allowed fd=1023", then I set the ERL_MAX_PORTS variable when start the erlang vm. But it takes no effect. > > $ erl -env ERL_MAX_PORTS 4096 > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> erlang:system_info(check_io). > [{name,erts_poll}, > {primary,select}, > {fallback,false}, > {kernel_poll,false}, > {memory_size,6344}, > {total_poll_set_size,2}, > {lazy_updates,true}, > {pending_updates,0}, > {batch_updates,false}, > {concurrent_updates,false}, > {max_fds,1024}] > 2> > > The max_fds is always 1024. > > -- > Kind Regards > goofansu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From carlsson.richard@REDACTED Sat Jul 13 14:33:14 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sat, 13 Jul 2013 14:33:14 +0200 Subject: [erlang-questions] Bug in erlang? In-Reply-To: References: Message-ID: <51E1490A.7020005@gmail.com> On 2013-07-13 06:33 , Mike Oxford wrote: > From the command-line, everything is expected and looks correct... > > list_to_float(lists:flatten(io_lib:format("~.2f",[1.000000e+00]))). > 1.00 > > But inside of a .beam'd module... > erlang:display(V), > > erlang:display(lists:flatten(io_lib:format("~.2f",[V]))), > V2 = > list_to_float(lists:flatten(io_lib:format("~.2f",[V]))), > erlang:display(V2) > > gives... > > 1.000000e+00 > "1.00" > 1.000000e+00 > > So the interesting part is that it (list_to_float) works as expected > in the shell, but not in a compiled .beam file. It's not a problem with list_to_float - I'm sure it gives you the same value in both cases. But in the first case you let the shell use its default formatting of values to print the float, and in the second case you use erlang:display(), which is a debugging function that always prints floats in a fixed format. Try using erlang:display() on the result on the command line and see, or use io:format(...) instead of erlang:display() to show the result in the module. To convince yourself further, you could use term_to_binary(V) on the floats in both cases and print the binary representation (e.g. use binary_to_list() and print the byte values). /Richard From goofan.su@REDACTED Sat Jul 13 15:04:35 2013 From: goofan.su@REDACTED (goofansu) Date: Sat, 13 Jul 2013 21:04:35 +0800 Subject: [erlang-questions] =?utf-8?Q?=E5=9B=9E=E5=A4=8D=EF=BC=9A_?= ERL_MAX_PORTS has no effect In-Reply-To: <8A30B7E3-77EC-47CF-B6C6-92AAC2F1D708@gmail.com> References: <8293812F868F43A99270A2BE49389225@gmail.com> <8A30B7E3-77EC-47CF-B6C6-92AAC2F1D708@gmail.com> Message-ID: <4BAB6CCD60AC4578B8AD23AF6FC2B5F7@gmail.com> Hi, Anthony Thank you for the explanation. I tested on the VPS again and set `ulimit -n` to a number like 4096 takes effects. I'm running Riak so cannot upgrade to R16B01 now. Kind Regards, -- goofansu -- Kind Regards goofansu ? 2013?7?13???????8:08?Anthony Ramine ??? > Hello, > > ERL_MAX_PORTS sets the maximum number of running ports. Not the maximum number of file descriptors. > > I guess you are running OS X which has a broken poll implementation and Erlang is forced to use select(), which is limited to FD_SETSIZE file descriptors. Fortunately, OS X's select() implementation can also be made to support an infinite number of file descriptors through a macro which I taught Erlang/OTP to use. Upgrade to R16B01 and you should be fine. > > Regards, > > -- > Anthony Ramine > > Le 13 juil. 2013 ? 08:34, goofansu a ?crit : > > > I encounted "fd=1024 is larger than the largest allowed fd=1023", then I set the ERL_MAX_PORTS variable when start the erlang vm. But it takes no effect. > > > > $ erl -env ERL_MAX_PORTS 4096 > > > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false] > > > > Eshell V5.9.1 (abort with ^G) > > 1> erlang:system_info(check_io). > > [{name,erts_poll}, > > {primary,select}, > > {fallback,false}, > > {kernel_poll,false}, > > {memory_size,6344}, > > {total_poll_set_size,2}, > > {lazy_updates,true}, > > {pending_updates,0}, > > {batch_updates,false}, > > {concurrent_updates,false}, > > {max_fds,1024}] > > 2> > > > > The max_fds is always 1024. > > > > -- > > Kind Regards > > goofansu > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Sat Jul 13 15:19:54 2013 From: tony@REDACTED (Tony Rogvall) Date: Sat, 13 Jul 2013 15:19:54 +0200 Subject: [erlang-questions] =?utf-8?b?5Zue5aSN77yaICBFUkxfTUFYX1BPUlRTIGhh?= =?utf-8?q?s_no_effect?= In-Reply-To: <4BAB6CCD60AC4578B8AD23AF6FC2B5F7@gmail.com> References: <8293812F868F43A99270A2BE49389225@gmail.com> <8A30B7E3-77EC-47CF-B6C6-92AAC2F1D708@gmail.com> <4BAB6CCD60AC4578B8AD23AF6FC2B5F7@gmail.com> Message-ID: Hi! Setting the ulimit works nicely with files, but not so nice with sockets. You wont notice until the driver start using driver_select and thereby, indirectly, using the system call select. I suspected that "erl +K true", enable kernel poll (kqueue for max os x), should do the trick. But, not. There is still a check for max_fds that triggers somewhere which is unfortunate. This is on R15, have not checked R16 yet. Rickard Green may now more about the status on max_fds ? Regards /Tony On 13 jul 2013, at 15:04, goofansu wrote: > Hi, Anthony > > Thank you for the explanation. > > I tested on the VPS again and set `ulimit -n` to a number like 4096 takes effects. > > I'm running Riak so cannot upgrade to R16B01 now. > > > Kind Regards, > > -- > goofansu > > > -- > Kind Regards > goofansu > > ? 2013?7?13???????8:08?Anthony Ramine ??? > >> Hello, >> >> ERL_MAX_PORTS sets the maximum number of running ports. Not the maximum number of file descriptors. >> >> I guess you are running OS X which has a broken poll implementation and Erlang is forced to use select(), which is limited to FD_SETSIZE file descriptors. Fortunately, OS X's select() implementation can also be made to support an infinite number of file descriptors through a macro which I taught Erlang/OTP to use. Upgrade to R16B01 and you should be fine. >> >> Regards, >> >> -- >> Anthony Ramine >> >> Le 13 juil. 2013 ? 08:34, goofansu a ?crit : >> >>> I encounted "fd=1024 is larger than the largest allowed fd=1023", then I set the ERL_MAX_PORTS variable when start the erlang vm. But it takes no effect. >>> >>> $ erl -env ERL_MAX_PORTS 4096 >>> >>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false] >>> >>> Eshell V5.9.1 (abort with ^G) >>> 1> erlang:system_info(check_io). >>> [{name,erts_poll}, >>> {primary,select}, >>> {fallback,false}, >>> {kernel_poll,false}, >>> {memory_size,6344}, >>> {total_poll_set_size,2}, >>> {lazy_updates,true}, >>> {pending_updates,0}, >>> {batch_updates,false}, >>> {concurrent_updates,false}, >>> {max_fds,1024}] >>> 2> >>> >>> The max_fds is always 1024. >>> >>> -- >>> Kind Regards >>> goofansu >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgisinger@REDACTED Sat Jul 13 15:40:15 2013 From: hgisinger@REDACTED (Hernando Gisinger) Date: Sat, 13 Jul 2013 10:40:15 -0300 Subject: [erlang-questions] Bug in erlang? In-Reply-To: <51E1490A.7020005@gmail.com> References: <51E1490A.7020005@gmail.com> Message-ID: Erlang R16B01 (erts-5.10.2) [async-threads:10] Eshell V5.10.2 (abort with ^G) 1> V = 1.000000e+00. 1.0 2> erlang:display(V). 1.000000e+00 true 3> erlang:display(lists:flatten(io_lib:format("~.2f",[V]))). "1.00" true 4> V2 = list_to_float(lists:flatten(io_lib:format("~.2f",[V]))). 1.0 5> erlang:display(V2). 1.000000e+00 true 6> 2013/7/13 Richard Carlsson > On 2013-07-13 06:33 , Mike Oxford wrote: > >> From the command-line, everything is expected and looks correct... >> >> list_to_float(lists:flatten(**io_lib:format("~.2f",[1.**000000e+00]))). >> 1.00 >> >> But inside of a .beam'd module... >> erlang:display(V), >> >> erlang:display(lists:flatten(**io_lib:format("~.2f",[V]))), >> V2 = >> list_to_float(lists:flatten(**io_lib:format("~.2f",[V]))), >> erlang:display(V2) >> >> gives... >> >> 1.000000e+00 >> "1.00" >> 1.000000e+00 >> >> So the interesting part is that it (list_to_float) works as expected >> in the shell, but not in a compiled .beam file. >> > > It's not a problem with list_to_float - I'm sure it gives you the same > value in both cases. But in the first case you let the shell use its > default formatting of values to print the float, and in the second case you > use erlang:display(), which is a debugging function that always prints > floats in a fixed format. Try using erlang:display() on the result on the > command line and see, or use io:format(...) instead of erlang:display() to > show the result in the module. > > To convince yourself further, you could use term_to_binary(V) on the > floats in both cases and print the binary representation (e.g. use > binary_to_list() and print the byte values). > > /Richard > > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wvvwwvw@REDACTED Sat Jul 13 18:07:41 2013 From: wvvwwvw@REDACTED (Andrew Ledvina) Date: Sat, 13 Jul 2013 09:07:41 -0700 Subject: [erlang-questions] jiffy encoding Message-ID: > > The reason for the extra tuple list is to distinguish ambiguous cases > involving empty lists and empty objects. Mochijson takes a different > approach but something must be done to deal with the Erlang types available. > > ------------------------------ > > Message: 9 > Date: Sat, 13 Jul 2013 10:05:00 +0300 > From: Aggelos Giantsios > > To: Mike Oxford > > Cc: Erlang Questions > > Subject: Re: [erlang-questions] jiffy encoding > Message-ID: > < > CAAyN70R1LjoHcjWbb8WPnCZKS5ez12Y_AaW0HKJYU+o3hzG_KQ@REDACTED > > > Content-Type: text/plain; charset="iso-8859-1" > > On Sat, Jul 13, 2013 at 3:58 AM, Mike Oxford > > wrote: > > > From the docs... > > {[{foo, bar}]} -> {"foo": "bar"} -> {[{<<"foo">>, > <<"bar">>}]} > > > > Why does jiffy need the extra "tuple list" syntax? > > Anyone happen to know? > > > > Hello! > > I believe the notion is that an orddict is converted to a JSON dictionary. > I guess in order to distinguish an orddict from a regular list, they wrap > the orddict in a tuple. > > Aggelos > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20130713/a890fa51/attachment-0001.html > > > > ------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > End of erlang-questions Digest, Vol 121, Issue 7 > ************************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.petrauskas@REDACTED Sun Jul 14 21:59:38 2013 From: k.petrauskas@REDACTED (Karolis Petrauskas) Date: Sun, 14 Jul 2013 22:59:38 +0300 Subject: [erlang-questions] Unable to start mnesia DB. In-Reply-To: <51DFAC74.9060401@gmail.com> References: <51DFAC74.9060401@gmail.com> Message-ID: Hi, Thanks for your response. I have 1 secondary index in the entire DB. I will try to load the DB with the index dropped. Karolis On Fri, Jul 12, 2013 at 10:12 AM, Ignas Vy?niauskas wrote: > On 07/09/2013 07:16 PM, Karolis Petrauskas wrote: >> After the DB become available, I stopped the node by invoking >> init:stop(). Then I repeated the test. It is still running, but >> timings look the same (2 cores were busy for 2.5 hours, and after >> that 1 core is loaded fully, expected for ~5.5 hours). For the second >> start up attempt the DB has no DCL files, only DCD are left after >> first load. > > A poke in the dark: do you maybe have lots of secondary indices? These > will not be stored on disk and so will be recreated each time the tables > are loaded, I've experienced very long mnesia startup times in such > cases. Otherwise -- no idea. > > -- > Ignas From donpedrothird@REDACTED Sun Jul 14 18:50:34 2013 From: donpedrothird@REDACTED (Evgeny M) Date: Sun, 14 Jul 2013 09:50:34 -0700 (PDT) Subject: [erlang-questions] serializing funs Message-ID: <09381e00-139c-442e-bb9a-645f5226bace@googlegroups.com> Hi, How can I serialize a fun with its'complete representation from a module? Seems like term_to_binary returns full representation only when it is called from shell, in module it returns reference or something like this. I need to create a fun in a module and pass it to another node that has different modules. Is this possible at all? Also, am i right that such serialization returns closure, including all variables bound outside of the fun body? -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Mon Jul 15 13:39:56 2013 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 15 Jul 2013 12:39:56 +0100 (BST) Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: Message-ID: <1383545459.32289304.1373888396067.JavaMail.root@erlang-solutions.com> Sorry for getting in this discussion late, on holidays. Just an historical curiosity. Before we had 'if' and we needed something if-ish this is exactly what we would do. We would use a case with a trivial expression and clause patterns which would always match and all the work was done in the guards. It looked so horrific that we added 'if'. Robert ----- Original Message ----- > From: "Anthony Ramine" > To: ok@REDACTED > Cc: erlang-questions@REDACTED > Sent: Monday, 8 July, 2013 8:04:34 AM > Subject: Re: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially > > With that logic, I see no reason to make if G -> B end different from > case {} of {} when G -> B end. > > Expanding cond to nested cases earlier in the compilation would mean > confusing and misguided warning, e.g. "cond foobar -> ok end" would > trigger the warning "no clause will ever match"; instead of the > clearer "this clause will crash". > > -- > Anthony Ramine > > Le 8 juil. 2013 ? 00:04, ok@REDACTED a ?crit : > > > > >> I couldn't decide which error should be thrown when the cond test > >> doesn't > >> return a boolean but I have found some references to cond > >> expressions in > >> an old paper [2] (page 37) and decided to stick to their choices. > > > > To be honest, I couldn't see any good reason to make > > cond E1 -> B1 ; ... ; En -> Bn end > > any different from > > case E1 of true -> B1 ; false -> > > ... > > case En of true -> Bn > > end > > ... > > end > > I see great virtue in there being *no* difference whatsoever > > in the semantics of the two, so that the choice is *solely* > > stylistic and rewriting one into the other (either way) *cannot* > > break a working program. > > > > Having looked at that reference, I do not see any advantage for > > the more complicated version. As long as you can find out > > *where* the error was, it just isn't _useful_ to know 'it was a > > cond' rather than 'it was a case'; a program catching errors > > would never want to treat them differently, and a human who has > > been told _where_ can instantly see _what_. > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From robert.virding@REDACTED Mon Jul 15 13:51:30 2013 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 15 Jul 2013 12:51:30 +0100 (BST) Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <249F45EB-527A-4D11-907F-5C99C3BF242B@gmail.com> Message-ID: <2015940198.32297057.1373889090322.JavaMail.root@erlang-solutions.com> If you are implementing a 'cond' then there is one extension which is quite practical though it breaks the purity of cond. It is to add the possibility of having an explicit match in the test, not just whether it returns 'true' of 'false'. Also having any variables in the match accessible in the body. I don't really have a good suggestion for a syntax except perhaps to extend 'cond' tests with a " = " option, or perhaps " ?= ". If the match fails then no error is generated and the next test is tried. I added this to LFE and it is quite useful. Something like this was discussed a while back as a general extension to guards but I can't remember the syntax. It would only allow approved guard expressions. Robert ----- Original Message ----- > From: "Anthony Ramine" > To: "Richard A. O'Keefe" > Cc: erlang-questions@REDACTED > Sent: Monday, 8 July, 2013 10:35:25 AM > Subject: Re: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially > > No this example was just confusing, what I meant is that if you need > a warning "no clause will ever match" in the case of an explicit > "case foobar of true -> ok; false -> ok end" and a warning "this > clause will crash" in the case of "cond foobar -> ok; true -> ok > end". There is no rewording that makes the warning fit in both > situations, they need to be handled separately. > > In my implementation I decided to annotate the Core Erlang cases to > which a cond is compiled with the atom "cond_case". > > -- > Anthony Ramine > > Le 8 juil. 2013 ? 08:22, "Richard A. O'Keefe" a > ?crit : > > > > > On 8/07/2013, at 6:04 PM, Anthony Ramine wrote: > > > >> With that logic, I see no reason to make if G -> B end different > >> from case {} of {} when G -> B end. > > > > Neither do I. It is excess complexity that we really don't need, > > and it inhibits a range of source-to-source transformations that > > ought not to be inhibited. > > > > If I am told which *function clause* a selection failure is in, > > better still, if I am told which *line* it is in, no distinction > > between if/case/cond in exceptions is of any real help. The > > *only* time it is helpful is when you have one 'if' and one 'case'. > > Have *two* 'if's or *two* 'case's, and it's useless. > > > >> > >> Expanding cond to nested cases earlier in the compilation would > >> mean confusing and misguided warning, e.g. "cond foobar -> ok > >> end" would trigger the warning "no clause will ever match"; > >> instead of the clearer "this clause will crash". > > > > You have presented a good argument for rewording the error message. > > > > That's all. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From devangana@REDACTED Mon Jul 15 21:08:12 2013 From: devangana@REDACTED (Devangana Tarafdar) Date: Mon, 15 Jul 2013 14:08:12 -0500 Subject: [erlang-questions] Setting compatibility mode via +R Message-ID: Hello, I am experimenting with the +R emulator flag as described in the erl man page ( http://www.erlang.org/doc/man/erl.html) and I am confused about when I should be using this and also what happens if I do not use this when communicating between 2 nodes from different releases. *" +R ReleaseNumber* Sets the compatibility mode. The distribution mechanism is not backwards compatible by default. This flags sets the emulator in compatibility mode with an earlier Erlang/OTP release ReleaseNumber. The release number must be in the range -2... This limits the emulator, making it possible for it to communicate with Erlang nodes (as well as C- and Java nodes) running that earlier release. Note: Make sure all nodes (Erlang-, C-, and Java nodes) of a distributed Erlang system is of the same Erlang/OTP release, or from two different Erlang/OTP releases X and Y, where *all* Y nodes have compatibility mode X." This is what I am doing when setting up 2 nodes from release 13B03 and 16B. The release numbers difference is greater than 2 so I cannot use the +R flag anyway but the nodes have no problems communicating without the flag. I was expecting that the nodes would not be able to connect at all but clearly I am wrong. #startup #first node erl -name node_a -setcookie x Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.7.4 (abort with ^G) (node_a@REDACTED)1> erlang:system_info(otp_release). "R13B03" #second node downloads/otp_src_R16B/bin/erl -name node_b -setcookie x Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false] Eshell V5.10.1 (abort with ^G) (node_b@REDACTED)1> erlang:system_info(otp_release). "R16B" # subsequent session (node_b@REDACTED)3> net_adm:ping('node_a@REDACTED'). pong (node_a@REDACTED)2> nodes(). ['node_b@REDACTED'] (node_a@REDACTED)5> net_kernel:disconnect('node_b@REDACTED'). true (node_a@REDACTED)6> nodes(). [] (node_a@REDACTED)7> net_adm:ping('node_b@REDACTED'). pong (node_a@REDACTED)8> Thank you for any help in interpreting this. Devangana -------------- next part -------------- An HTML attachment was scrubbed... URL: From devangana@REDACTED Mon Jul 15 21:26:09 2013 From: devangana@REDACTED (Devangana Tarafdar) Date: Mon, 15 Jul 2013 14:26:09 -0500 Subject: [erlang-questions] Setting compatibility mode via +R In-Reply-To: References: Message-ID: Thank you for the link, I will go through this. -Devangana On Mon, Jul 15, 2013 at 2:11 PM, Michael Loftis wrote: > I don't believe the distribution protocol has changed since like R6 or so. > > http://www.erlang.org/doc/apps/erts/erl_dist_protocol.html > > On Mon, Jul 15, 2013 at 12:08 PM, Devangana Tarafdar > wrote: > > Hello, > > > > I am experimenting with the +R emulator flag as > described > > in the erl man page ( http://www.erlang.org/doc/man/erl.html) and I am > > confused about when I should be using this and also what happens if I do > not > > use this when communicating between 2 nodes from different releases. > > > > " +R ReleaseNumber > > > > Sets the compatibility mode. > > > > The distribution mechanism is not backwards compatible by default. This > > flags sets the emulator in compatibility mode with an earlier Erlang/OTP > > release ReleaseNumber. The release number must be in the range > release>-2... This limits the emulator, making it > possible > > for it to communicate with Erlang nodes (as well as C- and Java nodes) > > running that earlier release. > > > > Note: Make sure all nodes (Erlang-, C-, and Java nodes) of a distributed > > Erlang system is of the same Erlang/OTP release, or from two different > > Erlang/OTP releases X and Y, where all Y nodes have compatibility mode > X." > > > > This is what I am doing when setting up 2 nodes from release 13B03 and > 16B. > > The release numbers difference is greater than 2 so I cannot use the +R > flag > > anyway but the nodes have no problems communicating without the flag. I > was > > expecting that the nodes would not be able to connect at all but clearly > I > > am wrong. > > > > #startup > > > > #first node > > erl -name node_a -setcookie x > > Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] > > [async-threads:0] [hipe] [kernel-poll:false] > > > > Eshell V5.7.4 (abort with ^G) > > (node_a@REDACTED)1> erlang:system_info(otp_release). > > "R13B03" > > > > #second node > > downloads/otp_src_R16B/bin/erl -name node_b -setcookie x > > Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:4:4] [async-threads:10] > > [kernel-poll:false] > > > > Eshell V5.10.1 (abort with ^G) > > (node_b@REDACTED)1> erlang:system_info(otp_release). > > "R16B" > > > > > > # subsequent session > > (node_b@REDACTED)3> net_adm:ping('node_a@REDACTED'). > > pong > > > > > > (node_a@REDACTED)2> nodes(). > > ['node_b@REDACTED'] > > (node_a@REDACTED)5> net_kernel:disconnect('node_b@REDACTED'). > > true > > (node_a@REDACTED)6> nodes(). > > [] > > (node_a@REDACTED)7> net_adm:ping('node_b@REDACTED'). > > pong > > (node_a@REDACTED)8> > > > > > > Thank you for any help in interpreting this. > > > > Devangana > > > > > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -- > > "Genius might be described as a supreme capacity for getting its possessors > into trouble of all kinds." > -- Samuel Butler > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mloftis@REDACTED Mon Jul 15 21:11:33 2013 From: mloftis@REDACTED (Michael Loftis) Date: Mon, 15 Jul 2013 12:11:33 -0700 Subject: [erlang-questions] Setting compatibility mode via +R In-Reply-To: References: Message-ID: I don't believe the distribution protocol has changed since like R6 or so. http://www.erlang.org/doc/apps/erts/erl_dist_protocol.html On Mon, Jul 15, 2013 at 12:08 PM, Devangana Tarafdar wrote: > Hello, > > I am experimenting with the +R emulator flag as described > in the erl man page ( http://www.erlang.org/doc/man/erl.html) and I am > confused about when I should be using this and also what happens if I do not > use this when communicating between 2 nodes from different releases. > > " +R ReleaseNumber > > Sets the compatibility mode. > > The distribution mechanism is not backwards compatible by default. This > flags sets the emulator in compatibility mode with an earlier Erlang/OTP > release ReleaseNumber. The release number must be in the range release>-2... This limits the emulator, making it possible > for it to communicate with Erlang nodes (as well as C- and Java nodes) > running that earlier release. > > Note: Make sure all nodes (Erlang-, C-, and Java nodes) of a distributed > Erlang system is of the same Erlang/OTP release, or from two different > Erlang/OTP releases X and Y, where all Y nodes have compatibility mode X." > > This is what I am doing when setting up 2 nodes from release 13B03 and 16B. > The release numbers difference is greater than 2 so I cannot use the +R flag > anyway but the nodes have no problems communicating without the flag. I was > expecting that the nodes would not be able to connect at all but clearly I > am wrong. > > #startup > > #first node > erl -name node_a -setcookie x > Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] > [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.7.4 (abort with ^G) > (node_a@REDACTED)1> erlang:system_info(otp_release). > "R13B03" > > #second node > downloads/otp_src_R16B/bin/erl -name node_b -setcookie x > Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:4:4] [async-threads:10] > [kernel-poll:false] > > Eshell V5.10.1 (abort with ^G) > (node_b@REDACTED)1> erlang:system_info(otp_release). > "R16B" > > > # subsequent session > (node_b@REDACTED)3> net_adm:ping('node_a@REDACTED'). > pong > > > (node_a@REDACTED)2> nodes(). > ['node_b@REDACTED'] > (node_a@REDACTED)5> net_kernel:disconnect('node_b@REDACTED'). > true > (node_a@REDACTED)6> nodes(). > [] > (node_a@REDACTED)7> net_adm:ping('node_b@REDACTED'). > pong > (node_a@REDACTED)8> > > > Thank you for any help in interpreting this. > > Devangana > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler From paul.joseph.davis@REDACTED Tue Jul 16 00:23:03 2013 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Mon, 15 Jul 2013 17:23:03 -0500 Subject: [erlang-questions] jiffy encoding In-Reply-To: References: Message-ID: Aggelos is correct that it's used to easily pattern match between an object and a list when writing JSON processing functions. Other suggestions required more creative means to make the distinction. The reason specifically for the 1-tuple are mostly historical. CouchDB originally used mochijson2 directly which uses the form {struct, proplist()}. It was decided that the struct atom was superfluous so it got dropped using mochijson2's optional callbacks. At one point there was a thread between Joe Armstrong, Damien Katz, and Bob Ippolito that settled on the 1-tuple version but that was before NIFs came around and let anyone implement their own efficient JSON parsing. Since NIFs there hasn't been much action around providing BIFs which has resulted in no single format being blessed as the "official" representation. AFAIK, the only other widely used format beyond the {proplist()} and {struct, proplist()} format is from jsx [1] which uses proplists exclusively. Alisdair (the author of JSX) and I have had many discussions on the pros and cons of both formats but have only managed to agree to disagree on which is better. [1] https://github.com/talentdeficit/jsx On Sat, Jul 13, 2013 at 11:07 AM, Andrew Ledvina wrote: > The reason for the extra tuple list is to distinguish ambiguous cases >> involving empty lists and empty objects. Mochijson takes a different >> approach but something must be done to deal with the Erlang types available. >> >> ------------------------------ >> >> Message: 9 >> Date: Sat, 13 Jul 2013 10:05:00 +0300 >> From: Aggelos Giantsios >> To: Mike Oxford >> Cc: Erlang Questions >> Subject: Re: [erlang-questions] jiffy encoding >> Message-ID: >> < >> CAAyN70R1LjoHcjWbb8WPnCZKS5ez12Y_AaW0HKJYU+o3hzG_KQ@REDACTED> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> On Sat, Jul 13, 2013 at 3:58 AM, Mike Oxford wrote: >> >> > From the docs... >> > {[{foo, bar}]} -> {"foo": "bar"} -> {[{<<"foo">>, >> <<"bar">>}]} >> > >> > Why does jiffy need the extra "tuple list" syntax? >> > Anyone happen to know? >> > >> >> Hello! >> >> I believe the notion is that an orddict is converted to a JSON dictionary. >> I guess in order to distinguish an orddict from a regular list, they wrap >> the orddict in a tuple. >> >> Aggelos >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> http://erlang.org/pipermail/erlang-questions/attachments/20130713/a890fa51/attachment-0001.html >> > >> >> ------------------------------ >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> End of erlang-questions Digest, Vol 121, Issue 7 >> ************************************************ >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Jul 16 00:30:04 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 16 Jul 2013 10:30:04 +1200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <1383545459.32289304.1373888396067.JavaMail.root@erlang-solutions.com> References: <1383545459.32289304.1373888396067.JavaMail.root@erlang-solutions.com> Message-ID: <89BEF379-9272-45C5-AF79-174ADBE8C383@cs.otago.ac.nz> When there is a choice between two constructs such as if G -> Bt ; true -> Bf end and case {} of {} when G -> Bt ; {} when true -> Bf end that *LOOKS* as though it is purely stylistic, it should *BE* purely stylistic. Consider if E -> Bt ; true -> Bf end and cond E -> Bt ; true -> Bf end where E is such that both forms are syntactically legal. It *looks* as though the choice is purely stylistic. So it should *be* purely stylistic, otherwise it would be gratuitously difficult to change from one to the other. And that means that cond foobar -> Bt ; true -> Bf end should execute Bf and NOT complain about foobar. (Yes, my prototype got this wrong. I just hadn't thought it through clearly.) From n.oxyde@REDACTED Tue Jul 16 03:16:52 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 16 Jul 2013 03:16:52 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <89BEF379-9272-45C5-AF79-174ADBE8C383@cs.otago.ac.nz> References: <1383545459.32289304.1373888396067.JavaMail.root@erlang-solutions.com> <89BEF379-9272-45C5-AF79-174ADBE8C383@cs.otago.ac.nz> Message-ID: <4BBC34A3-5F34-4773-8696-65EC2D9A740E@gmail.com> I disagree. In the case of 'if', the tests are disjunctions of conjunctions of guards; whereas in the case of 'cond' they should be boolean tests. So what is a valid 'if' clause may not be a 'cond' one. Furthermore, in the case of 'if', the fact that foobar just fails silently is because guard semantics are used: the very thing that is not used in 'cond' expressions. -- Anthony Ramine Le 16 juil. 2013 ? 00:30, Richard A. O'Keefe a ?crit : > When there is a choice between two constructs such as > if G -> Bt ; true -> Bf end > and case {} of {} when G -> Bt ; {} when true -> Bf end > that *LOOKS* as though it is purely stylistic, > it should *BE* purely stylistic. > > Consider > if E -> Bt ; true -> Bf end > and > cond E -> Bt ; true -> Bf end > > where E is such that both forms are syntactically legal. > It *looks* as though the choice is purely stylistic. > So it should *be* purely stylistic, otherwise it would > be gratuitously difficult to change from one to the other. > And that means that > > cond foobar -> Bt ; true -> Bf end > > should execute Bf and NOT complain about foobar. > (Yes, my prototype got this wrong. I just hadn't thought > it through clearly.) > > From n.oxyde@REDACTED Tue Jul 16 03:20:19 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 16 Jul 2013 03:20:19 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <4BBC34A3-5F34-4773-8696-65EC2D9A740E@gmail.com> References: <1383545459.32289304.1373888396067.JavaMail.root@erlang-solutions.com> <89BEF379-9272-45C5-AF79-174ADBE8C383@cs.otago.ac.nz> <4BBC34A3-5F34-4773-8696-65EC2D9A740E@gmail.com> Message-ID: <528E1F65-0DA4-45D2-84CE-BFFF0CA1AA54@gmail.com> And reading your mail again, are you sure you meant that Bf should be executed? Whether in if or cond, that definitely sounds wrong. -- Anthony Ramine Le 16 juil. 2013 ? 03:16, Anthony Ramine a ?crit : > I disagree. > > In the case of 'if', the tests are disjunctions of conjunctions of guards; whereas in the case of 'cond' they should be boolean tests. So what is a valid 'if' clause may not be a 'cond' one. > > Furthermore, in the case of 'if', the fact that foobar just fails silently is because guard semantics are used: the very thing that is not used in 'cond' expressions. > > -- > Anthony Ramine > > Le 16 juil. 2013 ? 00:30, Richard A. O'Keefe a ?crit : > >> When there is a choice between two constructs such as >> if G -> Bt ; true -> Bf end >> and case {} of {} when G -> Bt ; {} when true -> Bf end >> that *LOOKS* as though it is purely stylistic, >> it should *BE* purely stylistic. >> >> Consider >> if E -> Bt ; true -> Bf end >> and >> cond E -> Bt ; true -> Bf end >> >> where E is such that both forms are syntactically legal. >> It *looks* as though the choice is purely stylistic. >> So it should *be* purely stylistic, otherwise it would >> be gratuitously difficult to change from one to the other. >> And that means that >> >> cond foobar -> Bt ; true -> Bf end >> >> should execute Bf and NOT complain about foobar. >> (Yes, my prototype got this wrong. I just hadn't thought >> it through clearly.) >> >> > From n.oxyde@REDACTED Tue Jul 16 03:21:12 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 16 Jul 2013 03:21:12 +0200 Subject: [erlang-questions] style question - best way to test multiple non-guard conditions sequentially In-Reply-To: <528E1F65-0DA4-45D2-84CE-BFFF0CA1AA54@gmail.com> References: <1383545459.32289304.1373888396067.JavaMail.root@erlang-solutions.com> <89BEF379-9272-45C5-AF79-174ADBE8C383@cs.otago.ac.nz> <4BBC34A3-5F34-4773-8696-65EC2D9A740E@gmail.com> <528E1F65-0DA4-45D2-84CE-BFFF0CA1AA54@gmail.com> Message-ID: <14067903-6EEA-48E0-8D73-6D7E1C3F6BA5@gmail.com> Nevermind that part, mistook Bt for Bf. -- Anthony Ramine Le 16 juil. 2013 ? 03:20, Anthony Ramine a ?crit : > And reading your mail again, are you sure you meant that Bf should be executed? Whether in if or cond, that definitely sounds wrong. > > -- > Anthony Ramine > > Le 16 juil. 2013 ? 03:16, Anthony Ramine a ?crit : > >> I disagree. >> >> In the case of 'if', the tests are disjunctions of conjunctions of guards; whereas in the case of 'cond' they should be boolean tests. So what is a valid 'if' clause may not be a 'cond' one. >> >> Furthermore, in the case of 'if', the fact that foobar just fails silently is because guard semantics are used: the very thing that is not used in 'cond' expressions. >> >> -- >> Anthony Ramine >> >> Le 16 juil. 2013 ? 00:30, Richard A. O'Keefe a ?crit : >> >>> When there is a choice between two constructs such as >>> if G -> Bt ; true -> Bf end >>> and case {} of {} when G -> Bt ; {} when true -> Bf end >>> that *LOOKS* as though it is purely stylistic, >>> it should *BE* purely stylistic. >>> >>> Consider >>> if E -> Bt ; true -> Bf end >>> and >>> cond E -> Bt ; true -> Bf end >>> >>> where E is such that both forms are syntactically legal. >>> It *looks* as though the choice is purely stylistic. >>> So it should *be* purely stylistic, otherwise it would >>> be gratuitously difficult to change from one to the other. >>> And that means that >>> >>> cond foobar -> Bt ; true -> Bf end >>> >>> should execute Bf and NOT complain about foobar. >>> (Yes, my prototype got this wrong. I just hadn't thought >>> it through clearly.) >>> >>> >> > From wyt.daily@REDACTED Tue Jul 16 08:53:19 2013 From: wyt.daily@REDACTED (=?GB2312?B?zfXUy8zO?=) Date: Tue, 16 Jul 2013 14:53:19 +0800 Subject: [erlang-questions] mnesia bug? Message-ID: hello everyone: i got an error when using mnesia, but this error only occur on my archlinux (table type must be disc_only_copies), everything is ok when on centos or windows, is it a mnesia bug? below is the code and operation. %% code start %% -module(m). -compile(export_all). -record(kv, {k, v}). i() -> ok = mnesia:create_schema([node()]), ok = mnesia:start(), {atomic, ok} = mnesia:create_table(kv, [ {attributes, record_info(fields, kv)}, {disc_only_copies, [node()]} ]). f() -> dets:verbose(), % mnesia:transaction(fun() -> % mnesia:write(#kv{k=k1, v=v1}), % mnesia:write(#kv{k=k2, v=v2}) % end). mnesia:dirty_write(#kv{k=k1, v=v1}), mnesia:dirty_write(#kv{k=k2, v=v2}). e() -> mnesia:dirty_read(kv, k1). %% code end %% %% operation start %% Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> m:i(). {atomic,ok} 2> m:f(). ok 3> % below will occur in 5 seconds if set "dets:verbose()" =ERROR REPORT==== 16-Jul-2013::14:39:04 === ** dets: Corrupt table kv: {error, {bad_object_header, "/home/tux/Mnesia.nonode@REDACTED/kv.DAT"}} 3> m:e(). {error,{bad_object_header,"/home/tux/Mnesia.nonode@REDACTED/kv.DAT"}} 4> %% operation end %% -------------- next part -------------- An HTML attachment was scrubbed... URL: From xapwing@REDACTED Tue Jul 16 10:04:23 2013 From: xapwing@REDACTED (Arie van Wingerden) Date: Tue, 16 Jul 2013 10:04:23 +0200 Subject: [erlang-questions] Hard Real Time Erlang Message-ID: I have read this article http://www.erlang.se/workshop/2007/proceedings/05nicosi.pdf and I am wondering if there has been any progress in this field. TIA, Arie -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.vb80@REDACTED Tue Jul 16 10:39:16 2013 From: pablo.vb80@REDACTED (Pablo Vieytes) Date: Tue, 16 Jul 2013 10:39:16 +0200 Subject: [erlang-questions] rebar, ct and travis-ci Message-ID: Hi, I'm using common tests, rebar and travis-ci. It works fine but when some test falis in travis I can't see the log files. My Makefile all: ./rebar get-deps ./rebar compile bin/bootstrap clean: ./rebar clean test: all ./rebar ct skip_deps=true doc: all bin/create_edoc.sh Any suggestion? Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aveviort@REDACTED Tue Jul 16 10:49:28 2013 From: aveviort@REDACTED (?) Date: Tue, 16 Jul 2013 12:49:28 +0400 Subject: [erlang-questions] wxErlang Message-ID: Hello, I've got a question about creating applications using wxErlang. This is my first real Erlang project. My question is: which style of handling messages better? I do it as in "Erlang Programming", like this: settings_loop(SettingsWindow) -> receive #wx{id = ?SAVE_SETTINGS, event = #wxCommand{type = command_button_clicked}} -> ... end. In wxDemo I saw such method of handling: -behaviour(wx_object). ... handle_event(#wx{event = #wxAuiNotebook{type = command_auinotebook_page_close}}, State) -> ... What's the difference? And which method should I prefer? Thanks, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Tue Jul 16 16:00:49 2013 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 16 Jul 2013 16:00:49 +0200 Subject: [erlang-questions] how can I run emacs from erlang Message-ID: Is there a simple way to run emacs from erlang? This doesn't work > os:cmd("emacs"). "emacs: standard input is not a tty\n" I just want to pop up a window with emacs in it (and some text in a buffer) and get the text back when the user quits emacs Is there an easy way to do this? And no - I don't want to run erlang inside emacs - I want the other way around. Cheers /Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Jul 16 16:17:16 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 16 Jul 2013 16:17:16 +0200 Subject: [erlang-questions] how can I run emacs from erlang In-Reply-To: References: Message-ID: Hi Joe, emacsclient is exactly what you seek. What is usually done is that you arrange a temporary file to be written. Then you call $EDITOR on that file and wait until it finishes. Then you read in set file when the editor exits. Then you check if the file changed at all. emacsclient(1) is emacs driver for doing this in a console. It connects to a running emacs server instance and opens a frame in it. You exit with C-x #. For deeper communication to an emacs instance, you can look at what distel does/did. Perhaps EDTS (the name eludes me, but search for Thomas Jaervstrand). On Tue, Jul 16, 2013 at 4:00 PM, Joe Armstrong wrote: > Is there a simple way to run emacs from erlang? > > This doesn't work > > > os:cmd("emacs"). > "emacs: standard input is not a tty\n" > > I just want to pop up a window with emacs in it (and some text in a > buffer) > and get the text back when the user quits emacs > > Is there an easy way to do this? > > And no - I don't want to run erlang inside emacs - I want the other way > around. > > Cheers > > /Joe > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.l.france@REDACTED Tue Jul 16 16:18:00 2013 From: aaron.l.france@REDACTED (Aaron France) Date: Tue, 16 Jul 2013 15:18:00 +0100 Subject: [erlang-questions] how can I run emacs from erlang In-Reply-To: References: Message-ID: Hi, > os:cmd("emacs"). Works for me. Are you running in X? Aaron On Tue, Jul 16, 2013 at 3:00 PM, Joe Armstrong wrote: > Is there a simple way to run emacs from erlang? > > This doesn't work > > > os:cmd("emacs"). > "emacs: standard input is not a tty\n" > > I just want to pop up a window with emacs in it (and some text in a buffer) > and get the text back when the user quits emacs > > Is there an easy way to do this? > > And no - I don't want to run erlang inside emacs - I want the other way > around. > > Cheers > > /Joe > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus@REDACTED Tue Jul 16 16:26:43 2013 From: magnus@REDACTED (Magnus Henoch) Date: Tue, 16 Jul 2013 15:26:43 +0100 Subject: [erlang-questions] how can I run emacs from erlang In-Reply-To: (Joe Armstrong's message of "Tue, 16 Jul 2013 16:00:49 +0200") References: Message-ID: Joe Armstrong writes: > Is there a simple way to run emacs from erlang? > > This doesn't work > >> os:cmd("emacs"). > "emacs: standard input is not a tty\n" > > I just want to pop up a window with emacs in it (and some text in a buffer) > and get the text back when the user quits emacs > > Is there an easy way to do this? In a running Emacs, activate "server mode" with M-x server-start, and then run os:cmd("emacsclient myfile.txt") (the file name is required). That will open the file in your running Emacs instance, and block the emacsclient invocation until you hit C-x #. Regards, Magnus From tony@REDACTED Tue Jul 16 18:56:11 2013 From: tony@REDACTED (Tony Rogvall) Date: Tue, 16 Jul 2013 18:56:11 +0200 Subject: [erlang-questions] how can I run emacs from erlang In-Reply-To: References: Message-ID: <572689C1-9661-462F-849F-68243EC86BF4@rogvall.se> Probably not what you want, but I just remembered that I use this handy little tool daily :-) https://github.com/tonyrog/error_emacs (patch that can capture crashing gen_servers are welcome :-) /Tony On 16 jul 2013, at 16:00, Joe Armstrong wrote: > Is there a simple way to run emacs from erlang? > > This doesn't work > > > os:cmd("emacs"). > "emacs: standard input is not a tty\n" > > I just want to pop up a window with emacs in it (and some text in a buffer) > and get the text back when the user quits emacs > > Is there an easy way to do this? > > And no - I don't want to run erlang inside emacs - I want the other way around. > > Cheers > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From publicityifl@REDACTED Tue Jul 16 20:48:52 2013 From: publicityifl@REDACTED (publicityifl@REDACTED) Date: Tue, 16 Jul 2013 11:48:52 -0700 (PDT) Subject: [erlang-questions] Call for Papers IFL 2013 Message-ID: <51e59594.c2ce0e0a.3b45.7609@mx.google.com> Hello, Please, find below the third call for papers for IFL 2013. Please forward these to anyone you think may be interested. Apologies for any duplicates you may receive. best regards, Jurriaan Hage Publicity Chair of IFL CALL FOR PAPERS 25th SYMPOSIUM ON IMPLEMENTATION AND APPLICATION OF FUNCTIONAL LANGUAGES - IFL 2013 RADBOUD UNIVERSITY NIJMEGEN, THE NETHERLANDS ACM In-Cooperation / ACM SIGPLAN AUGUST 28 - 30 2013 "Landgoed Holthurnsche Hof" http://ifl2013.cs.ru.nl We are proud to announce that the 25th edition of the IFL series returns to its roots at the Radboud University Nijmegen in the Netherlands. The symposium is held from 28th to 30th of August 2013. Scope ----- The goal of the IFL symposia is to bring together researchers actively engaged in the implementation and application of functional and function-based programming languages. IFL 2013 will be a venue for researchers to present and discuss new ideas and concepts, work in progress, and publication-ripe results related to the implementation and application of functional languages and function-based programming. Following the IFL tradition, IFL 2013 will use a post-symposium review process to produce the formal proceedings which will be published in the ACM Digital Library. All participants of IFL 2013 are invited to submit either a draft paper or an extended abstract describing work to be presented at the symposium. At no time may work submitted to IFL be simultaneously submitted to other venues; submissions must adhere to ACM SIGPLAN's republication policy: http://www.sigplan.org/Resources/Policies/Republication The submissions will be screened by the program committee chair to make sure they are within the scope of IFL, and will appear in the draft proceedings distributed at the symposium. Submissions appearing in the draft proceedings are not peer-reviewed publications. Hence, publications that appear only in the draft proceedings do not count as publication for the ACM SIGPLAN republication policy. After the symposium, authors will be given the opportunity to incorporate the feedback from discussions at the symposium and will be invited to submit a revised full article for the formal review process. From the revised submissions, the program committee will select papers for the formal proceedings considering their correctness, novelty, originality, relevance, significance, and clarity. Invited Speaker --------------- Lennart Augustsson, currently employed by the Standard Chartered Bank, well-known for his work on Haskell, parallel Haskell, Cayenne, and Bluespec, is the invited speaker of IFL 2013. He will be talking about practical applications of functional programming. Submission Details ------------------ Submission deadline draft papers: July 31 Notification of acceptance for presentation: August 2 Early registration deadline: August 7 Late registration deadline: August 14 Submission deadline for pre-symposium proceedings: August 21 25th IFL Symposium: August 28-30 Submission deadline for post-symposium proceedings: November 11 Notification of acceptance for post-symposium proceedings: December 18 Camera-ready version for post-symposium proceedings: February 3 2014 Prospective authors are encouraged to submit papers or extended abstracts to be published in the draft proceedings and to present them at the symposium. All contributions must be written in English. Papers must adhere to the standard ACM two columns conference format. For the pre-symposium proceedings we adopt a 'weak' page limit of 12 pages. For the post-symposium proceedings the page limit of 12 pages is firm. A suitable document template for LaTeX can be found at: http://www.acm.org/sigs/sigplan/authorInformation.htm Papers are to be submitted via the conference's EasyChair submission page: https://www.easychair.org/conferences/?conf=ifl2013 Topics ------ IFL welcomes submissions describing practical and theoretical work as well as submissions describing applications and tools in the context of functional programming. If you are not sure whether your work is appropriate for IFL 2013, please contact the PC chair at rinus@REDACTED Topics of interest include, but are not limited to: - language concepts - type systems, type checking, type inferencing - compilation techniques - staged compilation - run-time function specialization - run-time code generation - partial evaluation - (abstract) interpretation - metaprogramming - generic programming - automatic program generation - array processing - concurrent/parallel programming - concurrent/parallel program execution - embedded systems - web applications - (embedded) domain specific languages - security - novel memory management techniques - run-time profiling performance measurements - debugging and tracing - virtual/abstract machine architectures - validation, verification of functional programs - tools and programming techniques - (industrial) applications Peter Landin Prize ------------------ The Peter Landin Prize is awarded to the best paper presented at the symposium every year. The honoured article is selected by the program committee based on the submissions received for the formal review process. The prize carries a cash award equivalent to 150 Euros. Programme committee ------------------- - Thomas Arts, Quviq, Gothenburg, Sweden - Andrew Butterfield, Trinity College, Dublin, Ireland - Edwin Brady, University of St. Andrews, UK - Clemens Grelck, University of Amsterdam, Netherlands - Adam Granicz, IntelliFactory, Budapest, Hungary - Jeremy Gibbons, University of Oxford, UK - Fritz Henglein, University of Copenhagen, Denmark - Stephan Herhut, Intel Labs, Santa Clara, US - Ralf Hinze (co-chair), University of Oxford, UK - Zoltan Horvath, Eotvos Lorand University, Budapest, Hungary - Zhenjiang Hu, University of Tokyo, Japan - Mauro Jaskelioff, Universidad Nacional de Rosario, Argentina - Johan Jeuring, University of Utrecht, Netherlands - Rita Loogen, University of Marburg, Germany - Marco T. Morazan, Seton Hall University, New Jersey, US - Dominic Orchard, University of Cambridge, UK - Rinus Plasmeijer (chair), Radboud University Nijmegen, Netherlands - Tim Sheard, Portland State University, US - Sam Tobin-Hochstadt, Northeastern University / Indiana University, US - Peter Thiemann, University of Freiburg, Germany - Simon Thompson, University of Kent, UK Venue ----- The 25th IFL is organized by the Radboud University Nijmegen, Model Based Software Development Department at the Nijmegen Institute for Computing and Information Sciences. The event is held in the Landgoed ?Holthurnsche Hof?, a rural estate in the woodlands surrounding Nijmegen. It can be reached quickly and easily by public transport. From ok@REDACTED Wed Jul 17 00:50:21 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 17 Jul 2013 10:50:21 +1200 Subject: [erlang-questions] how can I run emacs from erlang In-Reply-To: References: Message-ID: <3E7B363F-3F67-4171-9CCB-84DF4DD10EFC@cs.otago.ac.nz> I have a little C program called ttyname, whose source code is #include #include int main(void) { char *s = ttyname(0); return s == 0 || puts(s) == EOF; } > os:cmd("emacs"). answers "emacs: standard input is not a tty\n". > os:cmd("emacs -nw /dev/tty"). answers "/bin/sh: line 1: /dev/tty: Device not configured\n" So what I sometimes do is % export TTYNAME=`ttyname` % erl ... > TTY = os:getenv("TTYNAME"). ... > os:cmd("emacs -nw <" ++ TTY ++ " >" ++ TTY). and that works fine. From francesco@REDACTED Wed Jul 17 02:15:35 2013 From: francesco@REDACTED (Francesco Cesarini) Date: Tue, 16 Jul 2013 17:15:35 -0700 Subject: [erlang-questions] De Facto event and alarm handlers Message-ID: <51E5E227.4070508@erlang-solutions.com> Hi all, Any suggestions for open source alarm and event handlers for Erlang systems? Having written about half a dozen in the last ten years, none of which we were allowed to release as open source, I was looking around for an open source one which might be gaining traction. We used to have EVA as part of OTP, but it was dropped early on. Functionality I am looking for includes the possibility for alarms to be be listed and categorized, be given different severities, have configurable dependencies, etc etc etc. The usual. Basically, a folsom or lager for alarming. Any ideas? Thanks, Francesco -- Erlang Solutions Ltd. http://www.erlang-solutions.com From zjfhappy@REDACTED Wed Jul 17 04:46:02 2013 From: zjfhappy@REDACTED (=?GBK?B?1cW/obfl?=) Date: Wed, 17 Jul 2013 10:46:02 +0800 (CST) Subject: [erlang-questions] is it a mensia bug?mnesia:load_textfile("/root/db4"). Error in line:61 of:"/root/db4" syntax error before: Ref Message-ID: <27263f6d.3735.13fea8526eb.Coremail.zjfhappy@126.com> 1, i get a error here when load_textfile: (ccmanager@REDACTED)1> mnesia:load_textfile("/root/db-20130716173944"). Error in line:61 of:"/root/db-20130716173944" syntax error before: Ref {error,read} (ccmanager@REDACTED)2> 2, and the db file is follow here with line number: 57 {vpool,"c7be3db1-0176-4efb-8676-47286ef2629d",null,<<"asfdf_?">>,0,0, 58 1373967539,null,0,0,0, 59 ["192.168.1.104","192.168.1.105","192.168.1.106"], 60 []}. 61 {task,"6c6ecf49-20b6-438c-82e5-daca138fe645",#Ref<0.0.0.238759>,image_to_vm, 62 'ccagent@REDACTED',<10230.1586.0>,1373967277,1373967278, 63 edog_task_h, 64 [{diskid,"ec46d38a-679a-41e3-9a75-f113f93eb33c"}], 65 [{ok,ok}], 66 true}. 67 {task,"79f136d5-53e3-4b70-ac80-0ac1943fb4fa",#Ref<0.0.0.253168>,image_to_vm, 68 'ccagent@REDACTED',<10231.877.0>,1373967323,1373967324,edog_task_h, 69 [{diskid,"f9f474c2-3277-4b93-87fe-bcd71661f763"}], 70 [{ok,ok}], 71 true}. 3, the erl version is : [root@REDACTED ~]#erl Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) > is this a mensia bug? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hm@REDACTED Wed Jul 17 09:50:52 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 17 Jul 2013 09:50:52 +0200 Subject: [erlang-questions] is it a mensia bug?mnesia:load_textfile("/root/db4"). Error in line:61 of:"/root/db4" syntax error before: Ref Message-ID: The format of the text file is plain Erlang terms. It cannot cope with references and pids. Your text file contains a reference. Do not use mnesia:dump_to_textfile/ and mnesia:load_textfile/1. They should have been deprecated years ago. /H?kan On Wed, Jul 17, 2013 at 4:46 AM, ??? wrote: > 1, i get a error here when load_textfile: > (ccmanager@REDACTED)1> > mnesia:load_textfile("/root/db-20130716173944"). > Error in line:61 of:"/root/db-20130716173944" syntax error before: Ref > {error,read} > (ccmanager@REDACTED)2> > > 2, and the db file is follow here with line number: > 57 {vpool,"c7be3db1-0176-4efb-8676-47286ef2629d",null,<<"asfdf_?">>,0,0, > 58 1373967539,null,0,0,0, > 59 ["192.168.1.104","192.168.1.105","192.168.1.106"], > 60 []}. > 61 > {task,"6c6ecf49-20b6-438c-82e5-daca138fe645",#Ref<0.0.0.238759>,image_to_vm, > 62 'ccagent@REDACTED',<10230.1586.0>,1373967277,1373967278, > 63 edog_task_h, > 64 [{diskid,"ec46d38a-679a-41e3-9a75-f113f93eb33c"}], > 65 [{ok,ok}], > 66 true}. > 67 > {task,"79f136d5-53e3-4b70-ac80-0ac1943fb4fa",#Ref<0.0.0.253168>,image_to_vm, > 68 > 'ccagent@REDACTED',<10231.877.0>,1373967323,1373967324,edog_task_h, > 69 [{diskid,"f9f474c2-3277-4b93-87fe-bcd71661f763"}], > 70 [{ok,ok}], > 71 true}. > > 3, > the erl version is : > [root@REDACTED ~]#erl > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] > [hipe] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) >> > > is this a mensia bug? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From hm@REDACTED Wed Jul 17 10:01:10 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 17 Jul 2013 10:01:10 +0200 Subject: [erlang-questions] how can I run emacs from erlang Message-ID: It looks like you are trying to use Emacs without proper X settings. Erlang R16B01 (erts-5.10.2) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> os:cmd("emacs"). [] 2> os:cmd("emacs -nw"). "emacs: standard input is not a tty\n" 3> On Tue, Jul 16, 2013 at 4:18 PM, Aaron France wrote: > Hi, > >> os:cmd("emacs"). > > Works for me. Are you running in X? > > Aaron > > > On Tue, Jul 16, 2013 at 3:00 PM, Joe Armstrong wrote: >> >> Is there a simple way to run emacs from erlang? >> >> This doesn't work >> >> > os:cmd("emacs"). >> "emacs: standard input is not a tty\n" >> >> I just want to pop up a window with emacs in it (and some text in a >> buffer) >> and get the text back when the user quits emacs >> >> Is there an easy way to do this? >> >> And no - I don't want to run erlang inside emacs - I want the other way >> around. >> >> Cheers >> >> /Joe >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From jinni.park@REDACTED Wed Jul 17 11:06:10 2013 From: jinni.park@REDACTED (Park, Sungjin) Date: Wed, 17 Jul 2013 18:06:10 +0900 Subject: [erlang-questions] {aborted,nomore} from mnesia Message-ID: Following is my code to use mnesia and I get {aborted,nomore} sometimes. What would be the reason? up(Name, Module) -> Pid = self(), Route = #?MODULE{name=Name, addr=Pid, module=Module}, F = fun() -> case mnesia:wread({?MODULE, Name}) of [#?MODULE{name=Name, addr=Pid, module=Module}] -> ok; [#?MODULE{name=Name, addr=undefined, module=Module}] -> mnesia:write(Route); [#?MODULE{name=Name, addr=Addr, module=Module}] -> exit(Addr, kill), mnesia:write(Route); [#?MODULE{name=Name}] -> {error, collision}; [] -> mnesia:write(Route); Error -> {error, Error} end end, case mnesia:transaction(F, ?MNESIA_TRANSACTION) of {atomic, Result} -> Result; Aborted -> Aborted end. -- Park, Sungjin ------------------------------------------------------------------------------------------------------------------- Peculiar travel suggestions are dancing lessons from god. -- The Books of Bokonon ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From publicityifl@REDACTED Wed Jul 17 11:29:28 2013 From: publicityifl@REDACTED (publicityifl@REDACTED) Date: Wed, 17 Jul 2013 02:29:28 -0700 (PDT) Subject: [erlang-questions] Call for Papers PEPM 2014 Message-ID: <51e663f8.c5330f0a.0952.ffffaf88@mx.google.com> Hello, Please, find below the first call for papers for PEPM 2014. Please forward these to anyone you think may be interested. Apologies for any duplicates you may receive. best regards, Jurriaan Hage Co-chair of PEPM 2014 ----------------------------- C A L L F O R P A P E R S ----------------------------- ======= PEPM 2014 =========== ACM SIGPLAN 2014 WORKSHOP ON PARTIAL EVALUATION AND PROGRAM MANIPULATION Mon-Tue, January 20-21, 2014 San Diego, California, USA co-located with POPL'14 Sponsored by ACM SIGPLAN http://www.program-transformation.org/PEPM14 SCOPE The PEPM Symposium/Workshop series aims at bringing together researchers and practitioners working in the areas of program manipulation, partial evaluation, and program generation. PEPM focuses on techniques, theory, tools, and applications of analysis and manipulation of programs. The 2014 PEPM workshop will be based on a broad interpretation of semantics-based program manipulation and continue last years' successful effort to expand the scope of PEPM significantly beyond the traditionally covered areas of partial evaluation and specialization and include practical applications of program transformations such as refactoring tools, and practical implementation techniques such as rule-based transformation systems. In addition, the scope of PEPM covers manipulation and transformations of program and system representations such as structural and semantic models that occur in the context of model-driven development. In order to reach out to practitioners, a separate category of tool demonstration papers will be solicited. Topics of interest for PEPM'14 include, but are not limited to: Program and model manipulation techniques such as: supercompilation, partial evaluation, fusion, on-the-fly program adaptation, active libraries, program inversion, slicing, symbolic execution, refactoring, decompilation, and obfuscation. Program analysis techniques that are used to drive program/model manipulation such as: abstract interpretation, termination checking, binding-time analysis, constraint solving, type systems, automated testing and test case generation. Techniques that treat programs/models as data objects including metaprogramming, generative programming, embedded domain-specific languages, program synthesis by sketching and inductive programming, staged computation, and model-driven program generation and transformation. Application of the above techniques including case studies of program manipulation in real-world (industrial, open-source) projects and software development processes, descriptions of robust tools capable of effectively handling realistic applications, benchmarking. Examples of application domains include legacy program understanding and transformation, DSL implementations, visual languages and end-user programming, scientific computing, middleware frameworks and infrastructure needed for distributed and web-based applications, resource-limited computation, and security. To maintain the dynamic and interactive nature of PEPM, we will continue the category of `short papers' for tool demonstrations and for presentations of exciting if not fully polished research, and of interesting academic, industrial and open-source applications that are new or unfamiliar. Student attendants with accepted papers can apply for a SIGPLAN PAC grant to help cover travel expenses and other support. PAC also offers other support, such as for child-care expenses during the meeting or for travel costs for companions of SIGPLAN members with physical disabilities, as well as for travel from locations outside of North America and Europe. For details on the PAC programme, see its web page. All accepted papers, short papers included, will appear in formal proceedings published by ACM Press. In addition to printed proceedings, accepted papers will be included in the ACM Digital Library. A special issue for Science of Computer Programming is planned with recommended papers from PEPM'14. PEPM has also established a Best Paper award. The winner will be announced at the workshop. SUBMISSION CATEGORIES AND GUIDELINES Regular Research Papers must not exceed 12 pages in ACM Proceedings style (including appendix). Tool demonstration papers and short papers must not exceed 6 pages in ACM Proceedings style (including appendix). At least one author of each accepted contribution must attend the workshop and present the work. In the case of tool demonstration papers, a live demonstration of the described tool is expected. Suggested topics, evaluation criteria, and writing guidelines for both research tool demonstration papers will be made available on the PEPM'14 Web-site. Papers should be submitted electronically via the workshop web site. Authors using LaTeX to prepare their submissions should use the new improved SIGPLAN proceedings style (sigplanconf.cls, 9pt template). IMPORTANT DATES Abstract due: Sat, September 28, 2013 Paper submission: Sat, October 5, 2013, 23:59, GMT Author notification: Mon, November 11, 2013 Camera-ready papers due: * to be announced * INVITED SPEAKERS to be announced PROGRAM CHAIRS Wei Ngan Chin (National University of Singapore, Singapore) Jurriaan Hage (Utrecht University, Netherlands) PROGRAM COMMITTEE ?velyne Contejean (LRI, CNRS, Universit? Paris-Sud, France) Cristina David (University of Oxford, UK) Alain Frisch (LexiFi, France) Ronald Garcia (University of British Columbia, Canada) Zhenjiang Hu (National Institute of Informatics, Japan) Paul H J Kelly (Imperial College, UK) Oleg Kiselyov (Monterey, USA) Naoki Kobayashi (University of Tokyo, Japan) Jens Krinke (University College London, UK) Ryan Newton (University of Indiana, USA) Alberto Pardo (Universidad de la Rep?blica, Uruguay) Sungwoo Park (Pohang University of Science and Technology, South Korea) Tiark Rompf (Oracle Labs & EPFL, Switzerland) Sukyoung Ryu (KAIST, South Korea) Kostis Sagonas (Uppsala University, Sweden) Max Schaefer (Nanyang Technological University, Singapore) Harald S?ndergaard (The University of Melbourne, Australia) Eijiro Sumii (Tohoku University, Japan) Eric Van Wyk (University of Minnesota, USA) Jeremy Yallop (University of Cambridge, UK) From hm@REDACTED Wed Jul 17 12:36:44 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 17 Jul 2013 12:36:44 +0200 Subject: [erlang-questions] {aborted,nomore} from mnesia In-Reply-To: References: Message-ID: When Mnesia detects that your transaction have run into a potential deadlock, it may restart your transaction. By default there is no restart limit. But you have explicitly configured Mnesia to only allow a certain number of restarts. When that limit is reached, the transaction will be aborted with 'nomore' as reason. /H?kan On Wed, Jul 17, 2013 at 11:06 AM, Park, Sungjin wrote: > Following is my code to use mnesia and I get {aborted,nomore} sometimes. > What would be the reason? > > up(Name, Module) -> > > Pid = self(), > > Route = #?MODULE{name=Name, addr=Pid, module=Module}, > > F = fun() -> > > case mnesia:wread({?MODULE, Name}) of > > [#?MODULE{name=Name, addr=Pid, module=Module}] -> > > ok; > > [#?MODULE{name=Name, addr=undefined, module=Module}] -> > > mnesia:write(Route); > > [#?MODULE{name=Name, addr=Addr, module=Module}] -> > > exit(Addr, kill), > > mnesia:write(Route); > > [#?MODULE{name=Name}] -> > > {error, collision}; > > [] -> > > mnesia:write(Route); > > Error -> > > {error, Error} > > end > > end, > > case mnesia:transaction(F, ?MNESIA_TRANSACTION) of > > {atomic, Result} -> Result; > > Aborted -> Aborted > > end. > > > -- > Park, Sungjin > ------------------------------------------------------------------------------------------------------------------- > Peculiar travel suggestions are dancing lessons from god. > -- The Books of Bokonon > ------------------------------------------------------------------------------------------------------------------- > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From jinni.park@REDACTED Wed Jul 17 12:58:43 2013 From: jinni.park@REDACTED (Park, Sungjin) Date: Wed, 17 Jul 2013 19:58:43 +0900 Subject: [erlang-questions] {aborted,nomore} from mnesia In-Reply-To: References: Message-ID: Great thanks! On Wed, Jul 17, 2013 at 7:36 PM, H?kan Mattsson wrote: > When Mnesia detects that your transaction have run into a potential > deadlock, it may restart your transaction. By default there is no > restart limit. But you have explicitly configured Mnesia to only allow > a certain number of restarts. When that limit is reached, the > transaction will be aborted with 'nomore' as reason. > > /H?kan > > On Wed, Jul 17, 2013 at 11:06 AM, Park, Sungjin > wrote: > > Following is my code to use mnesia and I get {aborted,nomore} sometimes. > > What would be the reason? > > > > up(Name, Module) -> > > > > Pid = self(), > > > > Route = #?MODULE{name=Name, addr=Pid, module=Module}, > > > > F = fun() -> > > > > case mnesia:wread({?MODULE, Name}) of > > > > [#?MODULE{name=Name, addr=Pid, module=Module}] -> > > > > ok; > > > > [#?MODULE{name=Name, addr=undefined, module=Module}] -> > > > > mnesia:write(Route); > > > > [#?MODULE{name=Name, addr=Addr, module=Module}] -> > > > > exit(Addr, kill), > > > > mnesia:write(Route); > > > > [#?MODULE{name=Name}] -> > > > > {error, collision}; > > > > [] -> > > > > mnesia:write(Route); > > > > Error -> > > > > {error, Error} > > > > end > > > > end, > > > > case mnesia:transaction(F, ?MNESIA_TRANSACTION) of > > > > {atomic, Result} -> Result; > > > > Aborted -> Aborted > > > > end. > > > > > > -- > > Park, Sungjin > > > ------------------------------------------------------------------------------------------------------------------- > > Peculiar travel suggestions are dancing lessons from god. > > -- The Books of Bokonon > > > ------------------------------------------------------------------------------------------------------------------- > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- Park, Sungjin ------------------------------------------------------------------------------------------------------------------- Peculiar travel suggestions are dancing lessons from god. -- The Books of Bokonon ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From wyt.daily@REDACTED Wed Jul 17 12:48:37 2013 From: wyt.daily@REDACTED (yt w) Date: Wed, 17 Jul 2013 18:48:37 +0800 Subject: [erlang-questions] mnesia bug? Message-ID: hello everyone: i got an error when using mnesia, but this error only occur on my archlinux, everything is ok when on centos or windows, is it a mnesia bug? below is the code and operation. %% code start %% -module(m). -compile(export_all). -record(kv, {k, v}). i() -> ok = mnesia:create_schema([node()]), ok = mnesia:start(), {atomic, ok} = mnesia:create_table(kv, [ {attributes, record_info(fields, kv)}, {disc_only_copies, [node()]} ]). f() -> dets:verbose(), % mnesia:transaction(fun() -> % mnesia:write(#kv{k=k1, v=v1}), % mnesia:write(#kv{k=k2, v=v2}) % end). mnesia:dirty_write(#kv{k=k1, v=v1}), mnesia:dirty_write(#kv{k=k2, v=v2}). e() -> mnesia:dirty_read(kv, k1). %% code end %% %% operation start %% Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> m:i(). {atomic,ok} 2> m:f(). ok 3> % below will occur in 5 seconds if set "dets:verbose()" =ERROR REPORT==== 16-Jul-2013::14:39:04 === ** dets: Corrupt table kv: {error, {bad_object_header, "/home/tux/Mnesia.nonode@REDACTED/kv.DAT"}} 3> m:e(). {error,{bad_object_header,"/home/tux/Mnesia.nonode@REDACTED/kv.DAT"}} 4> %% operation end %% -------------- next part -------------- An HTML attachment was scrubbed... URL: From js@REDACTED Wed Jul 17 16:02:53 2013 From: js@REDACTED (Schneider) Date: Wed, 17 Jul 2013 16:02:53 +0200 Subject: [erlang-questions] undefined function erlang:binary_to_float/1 Message-ID: <51E6A40D.1000908@globe.de> Hi List, I cannot find the function binary_to_float/1. using Erlang R14B04 (erts-5.8.5) on ubuntu precise. What do I miss? bg, Johannes -- GLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nsterGLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nster 0251/5205 390 From ates@REDACTED Wed Jul 17 16:26:29 2013 From: ates@REDACTED (Artem Teslenko) Date: Wed, 17 Jul 2013 16:26:29 +0200 Subject: [erlang-questions] undefined function erlang:binary_to_float/1 In-Reply-To: <51E6A40D.1000908@globe.de> References: <51E6A40D.1000908@globe.de> Message-ID: <20130717142628.GA12561@ipv6.dp.ua> Looks like that function was introduced in R16 On Wed, 17 Jul 2013, Schneider wrote: > Hi List, > > I cannot find the function binary_to_float/1. > > using Erlang R14B04 (erts-5.8.5) on ubuntu precise. > What do I miss? > > bg, > Johannes > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From comptekki@REDACTED Wed Jul 17 16:30:45 2013 From: comptekki@REDACTED (Wes James) Date: Wed, 17 Jul 2013 08:30:45 -0600 Subject: [erlang-questions] undefined function erlang:binary_to_float/1 In-Reply-To: <51E6A40D.1000908@globe.de> References: <51E6A40D.1000908@globe.de> Message-ID: Alternative for now?? list_to_float(binary:bin_to_list(<<"10.0">>)). 10.0 On Wed, Jul 17, 2013 at 8:02 AM, Schneider wrote: > Hi List, > > I cannot find the function binary_to_float/1. > > using Erlang R14B04 (erts-5.8.5) on ubuntu precise. > What do I miss? > > bg, > Johannes > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hm@REDACTED Wed Jul 17 16:46:37 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Wed, 17 Jul 2013 16:46:37 +0200 Subject: [erlang-questions] internal error in v3_life In-Reply-To: References: Message-ID: On Wed, Jul 10, 2013 at 5:10 PM, Tuncer Ayaz wrote: > On Wed, Jul 10, 2013 at 4:05 PM, Hakan Mattsson wrote: >> There is no explicit calls to any function in the 'v3_life' module. >> Its functions are only called via apply. This means that reltool may >> have problems to determine that the module is used or not. >> >> But as the default setting of module inclusion ('mod_cond' set to >> 'all') in reltool also includes all modules listed in the app file >> there should be no problem. >> >> I suspect that you have set 'mod_cond' to 'derived'. If that is the >> case, reltool would not include the module. > > Hakan, if you take a look at the previously linked thread[4], you will > see that with {incl_cond, derived} and without a custom mod_cond > setting reltool pulls in 14 additional apps. Shouldn't the same apps > already be included due to {incl_cond, derived} regardless of > mod_cond? No. If you for example take the tools app. It is included because sasl uses xref. But tools does also contain more rarely used modules such as cover_web. And cover_web uses webtool. webtool uses inets. inets uses mnesia and ssl. etc. etc. This means that changing mod_cond from 'all' to 'derived' will have a big impact of how many applications/modules that will be included in the target system. If you have deep system knowledge you may of course do lots of tweaks to customize your system. Such as changing mod_cond for the tools app or explicitly excluding the cover_web module. I recommend using the graphical version of reltool (erl -s reltool) to explore system dependencies and play around with different settings of mod_cond, incl_cond etc. It is interesting (and a bit scary) to see all these unexpected dependencies... /H?kan From bombadil@REDACTED Wed Jul 17 19:42:39 2013 From: bombadil@REDACTED (Manuel A. Rubio "Bombadil") Date: Wed, 17 Jul 2013 19:42:39 +0200 Subject: [erlang-questions] [ANN] myproto 0.2.0 Message-ID: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Hi, new release of the MySQL server side protocol. I only add a new feature, a PEG for MySQL parse and a new (optional) config param to activate or deactivate it. By default this feature is off. The main idea is you can receive a query as: SELECT password FROM users WHERE username = 'user42'; And the 'gen_myproto' module developed, receives: #select{ params = [ #key{name = <<"password">>} ], tables= [ #table{name = <<"users">>} ], conditions = #condition{ nexo=eq, op1=#key{name = <<"username">>}, op2=#value{value = <<"user42">>} } } I'm adding eunit tests too, in this moment I have tests only for PEG. More in the next releases :-) Regards. Manuel Rubio. From peerst@REDACTED Wed Jul 17 19:56:26 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Wed, 17 Jul 2013 19:56:26 +0200 Subject: [erlang-questions] Hard Real Time Erlang References: Message-ID: On 2013-07-16 08:04:23 +0000, Arie van Wingerden said: > I have read this article > http://www.erlang.se/workshop/2007/proceedings/05nicosi.pdf?and I am > wondering if there has been any progress in this field. Not that I heard of. I work around this problem by porting Erlang to RTEMS which is a hard realtime OS. So one can do all hard realtime stuff in C, C++ or fwiw even Ada and run it at a higher prio than the Erlang VM For more info: http://www.grisp.org Even more info will be up soon, also with adding RSS to the site to make tracking it easier. I could imagine some subset of Erlang along FRP lines being compiled specially being able to also write hard realtime tasks without C but thats just pipedreams at the moment. But often the hard realtime stuff in systems is quite small so keeping just the hard realtime stuff in C and do as much as possible in Erlang on a Grisp system is very feasible. Cheers, -- Peer -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Jul 17 20:35:57 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 17 Jul 2013 22:35:57 +0400 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: Wow, it may be a very useful thing. For example, sphinx search engine allows users to connect via mysql protocol and make queries via usual API. Your project may be useful. For example, ejabberd server can show list of currently connected clients via mysql protocol. From maruthavanan_s@REDACTED Wed Jul 17 20:46:04 2013 From: maruthavanan_s@REDACTED (Maruthavanan Subbarayan) Date: Wed, 17 Jul 2013 14:46:04 -0400 Subject: [erlang-questions] An approach to common test Message-ID: HI All, I am new to common test in erlang, I just wanted to know how to approach the below scenario in common test. I have an application that would send group of multiple messages to RMQ, and expects response in back and I have to validate in testcase, so I thought of doing the below. all() -> [{group,general_test_group}, {group,rmq_test_group}] groups() ->[{general_group_test, [sequence], GeneralTestList},{rmq_test_group, [sequence], RmqTestList}]. so I thought I can have init_per_group function doing all initialization stuff like queue/exchange creation and bindings and subscribings. But I would not be getting the RMQ messages to my test case process, or I would endup in duplication by writing the above in all testcases. Thanks in advance,Marutha -------------- next part -------------- An HTML attachment was scrubbed... URL: From gumm@REDACTED Wed Jul 17 22:31:31 2013 From: gumm@REDACTED (Jesse Gumm) Date: Wed, 17 Jul 2013 15:31:31 -0500 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: Wow! This looks really fantastic! Thanks for sharing! On Jul 17, 2013 12:42 PM, "Manuel A. Rubio "Bombadil < bombadil@REDACTED> wrote: > Hi, > > new release of the MySQL server side protocol. I only add a new feature, a > PEG for MySQL parse and a new (optional) config param to activate or > deactivate it. By default this feature is off. > > The main idea is you can receive a query as: > > SELECT password FROM users WHERE username = 'user42'; > > And the 'gen_myproto' module developed, receives: > > #select{ > params = [ #key{name = <<"password">>} ], > tables= [ #table{name = <<"users">>} ], > conditions = #condition{ > nexo=eq, > op1=#key{name = <<"username">>}, > op2=#value{value = <<"user42">>} > } > } > > I'm adding eunit tests too, in this moment I have tests only for PEG. > > More in the next releases :-) > > Regards. > Manuel Rubio. > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay@REDACTED Thu Jul 18 09:04:11 2013 From: jay@REDACTED (Jay Nelson) Date: Thu, 18 Jul 2013 00:04:11 -0700 Subject: [erlang-questions] Enhancing the ETS API Message-ID: Recently I have been working on an ets_buffer module that allows for concurrent updates with multiple readers and writers to a central queue or stack data structure. Through careful combination of {write_concurrency, true} and atomic operations with update_counter it is possible to dramatically improve performance over a strictly serial gen_server wrapper around normal erlang queues, but the limited number of atomic mutable operations on ets tables make it difficult to architect useful data structures safely in ets. The most glaring missing features are to read and remove an entry in one operation, and to increment data elements which may be thresholded by another data element in the same record. The following commands could be useful with a write_concurrency ets table: 1) ets:match_delete(Tab, MatchSpec, Return_Objects::true) -> [Deleted_Object] ets:select_delete(Tab, MatchSpec, Return_Objects::true) -> [Deleted_Object] Currently, delete does not return any objects. 2) ets:update_counter(Tab, Key, Cmd) where Cmd also allows: {Field_Pos, Inc, Thresh, rollback} Do not change value, but return existing value if Thresh violated {Field_Pos, Inc, {Thresh_Pos}, New_Value | rollback} Threshold_Pos refers to another value stored in same record Anyone else interested in enhancements to the ets interface to allow more variants on update_counter? Are there other API calls that would make concurrent mutable data structure operations easier? jay From yash_ganthe@REDACTED Thu Jul 18 09:15:27 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 18 Jul 2013 07:15:27 +0000 Subject: [erlang-questions] Defining records using macros Message-ID: -define(REC_DEFINE(Type), -record(Type, { id ,val }). %% The . is an essential part of record definition ). %% Complains with syntax error: syntax error before: ')' My intention is to use a macro for defining records. ?REC_DEFINE(my_rec). Is there an escape character that the preprocessor can be give for the dot? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Jul 18 09:21:04 2013 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 18 Jul 2013 09:21:04 +0200 Subject: [erlang-questions] Enhancing the ETS API In-Reply-To: References: Message-ID: <21EBA329-7EBC-476A-A639-20ADDECC320F@feuerlabs.com> 18 jul 2013 kl. 09:04 skrev Jay Nelson : > 2) ets:update_counter(Tab, Key, Cmd) > where Cmd also allows: > > {Field_Pos, Inc, Thresh, rollback} > > Do not change value, but return existing value if Thresh violated What should it return otherwise? BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://feuerlabs.com From mabrek@REDACTED Thu Jul 18 09:25:33 2013 From: mabrek@REDACTED (Anton Lebedevich) Date: Thu, 18 Jul 2013 11:25:33 +0400 Subject: [erlang-questions] Enhancing the ETS API In-Reply-To: References: Message-ID: <51E7986D.80703@gmail.com> On 07/18/2013 11:04 AM, Jay Nelson wrote: > Recently I have been working on an ets_buffer module that allows > for concurrent updates with multiple readers and writers to a > central queue or stack data structure. Through careful combination > of {write_concurrency, true} and atomic operations with update_counter > it is possible to dramatically improve performance over a strictly serial > gen_server wrapper around normal erlang queues, but the limited > number of atomic mutable operations on ets tables make it difficult > to architect useful data structures safely in ets. > > The most glaring missing features are to read and remove an entry > in one operation, and to increment data elements which may be > thresholded by another data element in the same record. > > The following commands could be useful with a write_concurrency ets table: > > 1) ets:match_delete(Tab, MatchSpec, Return_Objects::true) -> [Deleted_Object] > ets:select_delete(Tab, MatchSpec, Return_Objects::true) -> [Deleted_Object] > > Currently, delete does not return any objects. > > 2) ets:update_counter(Tab, Key, Cmd) > where Cmd also allows: > > {Field_Pos, Inc, Thresh, rollback} > > Do not change value, but return existing value if Thresh violated > > {Field_Pos, Inc, {Thresh_Pos}, New_Value | rollback} > > Threshold_Pos refers to another value stored in same record > > > Anyone else interested in enhancements to the ets interface to allow > more variants on update_counter? Are there other API calls that would > make concurrent mutable data structure operations easier? +1 for atomic match_delete Another usable addition would be an atomic CAS (compare-and-set) operation on ETS. There was a discussion about it some time ago https://groups.google.com/d/msg/erlang-programming/Lsoo68dXONA/mwq7AXatiewJ Regards, Anton Lebedevich. From jay@REDACTED Thu Jul 18 09:25:59 2013 From: jay@REDACTED (Jay Nelson) Date: Thu, 18 Jul 2013 00:25:59 -0700 Subject: [erlang-questions] Enhancing the ETS API In-Reply-To: <21EBA329-7EBC-476A-A639-20ADDECC320F@feuerlabs.com> References: <21EBA329-7EBC-476A-A639-20ADDECC320F@feuerlabs.com> Message-ID: <6B79A17B-DB0B-485D-A8A8-C64E31E6021D@duomark.com> If increment does not violate threshold, it operates as currently, returning the newly incremented value. But rather than warping the value if the threshold is violated, just leave it as is and return the current value. I wanted to increment a pointer, but not beyond a certain value, so that it always pointed to the top of the stack. This feature was most useful when the threshold was stored in a different field of the same record and I did not know what that value was. On Jul 18, 2013, at 12:21 AM, Ulf Wiger wrote: > 18 jul 2013 kl. 09:04 skrev Jay Nelson : > >> 2) ets:update_counter(Tab, Key, Cmd) >> where Cmd also allows: >> >> {Field_Pos, Inc, Thresh, rollback} >> >> Do not change value, but return existing value if Thresh violated > > What should it return otherwise? > > BR, > Ulf W > > Ulf Wiger, Feuerlabs, Inc. > http://feuerlabs.com From carlsson.richard@REDACTED Thu Jul 18 09:33:59 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 18 Jul 2013 09:33:59 +0200 Subject: [erlang-questions] Defining records using macros In-Reply-To: References: Message-ID: <51E79A67.2000604@gmail.com> On 2013-07-18 09:15 , Yash Ganthe wrote: > -define(REC_DEFINE(Type), > > -record(Type, { > > id > > ,val > > }). %% The . is an essential part of record definition > > ). %% Complains with syntax error: syntax error before: ')' > > My intention is to use a macro for defining records. > > ?REC_DEFINE(my_rec). > > Is there an escape character that the preprocessor can be give for the dot? The preprocessor (just like the compiler) works on units of whole "forms", i.e., dot-terminated token sequences. Since the define must also be a whole form, it will end at the first dot token and there is no way to escape it. But you need to write your macro instantiations with a dot terminator anyway (just as you did above, in fact), otherwise the preprocessor won't have a unit to work on. Thus, you don't need to supply the dot in the definition, and the following works: -module(foo). -export([f/0]). -define(R(T), -record(T, {a, b})). ?R(foo). ?R(bar). f() -> #foo{a=A1,b=B1} = #foo{}, #bar{a=A2,b=B2} = #bar{}. /Richard From bombadil@REDACTED Thu Jul 18 09:37:10 2013 From: bombadil@REDACTED (Manuel A. Rubio "Bombadil") Date: Thu, 18 Jul 2013 09:37:10 +0200 Subject: [erlang-questions] Defining records using macros In-Reply-To: References: Message-ID: <8947259326bb36af2c9a6bdce4b8e70e@bosqueviejo.net> Hi Yash, El 2013-07-18 09:15, Yash Ganthe escribi?: > -define(REC_DEFINE(Type), > > -record(Type, { > > id > > ,val > > }). %% The . is an essential part of record definition > > ). %% Complains with syntax error: syntax error before: ')' > > My intention is to use a macro for defining records. > > ?REC_DEFINE(my_rec). > > Is there an escape character that the preprocessor can be give for > the > dot? I wrote a test: ------- -module(test). -compile([export_all]). -define(REC_DEFINE(Type), -record(Type, {id, val}) ). ?REC_DEFINE(one). ?REC_DEFINE(two). ?REC_DEFINE(three). testing() -> #one{}, #two{}, #three{}, ok. -------- Works well... if you want to include record definition inside a function, is not possible with or without a macro. A macro is only a "rewritter", by example: -------- -define('->', :). main() -> test ?'->' testing(). --------- If you want to do something with the erlang syntax, I think you should review the parse_transform way to do it :-) Regards. Manuel Rubio. From jay@REDACTED Thu Jul 18 09:49:45 2013 From: jay@REDACTED (Jay Nelson) Date: Thu, 18 Jul 2013 00:49:45 -0700 Subject: [erlang-questions] Enhancing the ETS API In-Reply-To: <21EBA329-7EBC-476A-A639-20ADDECC320F@feuerlabs.com> References: <21EBA329-7EBC-476A-A639-20ADDECC320F@feuerlabs.com> Message-ID: <5CD20C6C-2157-4480-A6DA-2F93DA2D4673@duomark.com> For context, here is a sneak preview of ets_buffer as part of the soon to be announced dk_cxy project: https://github.com/duomark/dk_cxy/tree/release_0_9_2 I am still reconsidering the LIFO approach, but FIFO and Ring buffers I believe are complete. I welcome any tips, corrections or contributions. This code was generalized from some specific code developed in production. While it passes the included minimal common_test suites, it has not yet been rolled back into production. Concurrency control is working in production but has not been updated to the latest ets_buffer calls and generational caching is running (rather nicely I might say) in production but has not been ported to the Open Source repository yet. jay From ulf@REDACTED Thu Jul 18 11:01:24 2013 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 18 Jul 2013 11:01:24 +0200 Subject: [erlang-questions] Enhancing the ETS API In-Reply-To: <5CD20C6C-2157-4480-A6DA-2F93DA2D4673@duomark.com> References: <21EBA329-7EBC-476A-A639-20ADDECC320F@feuerlabs.com> <5CD20C6C-2157-4480-A6DA-2F93DA2D4673@duomark.com> Message-ID: <781943E4-D2A2-4305-8F60-30D02B10CD3B@feuerlabs.com> I remembered that I once doodled on an extended ets update function. Here's what I found. A bit over the top perhaps? (: BR, Ulf -------------- next part -------------- A non-text attachment was scrubbed... Name: eets.erl Type: application/octet-stream Size: 3976 bytes Desc: not available URL: -------------- next part -------------- Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From tony@REDACTED Thu Jul 18 11:55:26 2013 From: tony@REDACTED (Tony Rogvall) Date: Thu, 18 Jul 2013 11:55:26 +0200 Subject: [erlang-questions] Hard Real Time Erlang In-Reply-To: References: Message-ID: On 17 jul 2013, at 19:56, Peer Stritzinger wrote: > > On 2013-07-16 08:04:23 +0000, Arie van Wingerden said: > > I have read this article http://www.erlang.se/workshop/2007/proceedings/05nicosi.pdf and I am wondering if there has been any progress in this field. > > Not that I heard of. > > I work around this problem by porting Erlang to RTEMS which is a hard realtime OS. > So one can do all hard realtime stuff in C, C++ or fwiw even Ada and run it at a higher prio than the Erlang VM > > For more info: http://www.grisp.org > > Even more info will be up soon, also with adding RSS to the site to make tracking it easier. > > I could imagine some subset of Erlang along FRP lines being compiled specially being able to also write hard realtime tasks without C but thats just pipedreams at the moment. Static Erlang ? I like it :-) > > But often the hard realtime stuff in systems is quite small so keeping just the hard realtime stuff in C and do as much as possible in Erlang on a Grisp system is very feasible. > +1. Even when I write interrupt handlers in C the code fragments tend to be very tiny, like 10-50 linux code. Mostly they update some registers, increment a counter, put some data into a queue for later processing. Just think about the amount of cpu required to switch to the Erlang VM and dispatch some Erlang code in an interrupt handler. Holy cow :-) /Tony > Cheers, > -- Peer > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 18 12:08:27 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 18 Jul 2013 14:08:27 +0400 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: Sorry, but you haven't added any links to the source code. From paperless@REDACTED Thu Jul 18 12:28:54 2013 From: paperless@REDACTED (Tim McNamara) Date: Thu, 18 Jul 2013 22:28:54 +1200 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: https://github.com/bosqueviejo/myproto (Thanks Google) On 18 July 2013 22:08, Max Lapshin wrote: > Sorry, but you haven't added any links to the source code. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xramtsov@REDACTED Thu Jul 18 12:33:51 2013 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Thu, 18 Jul 2013 20:33:51 +1000 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: <51E7C48F.6010405@gmail.com> On 18.07.2013 20:08, Max Lapshin wrote: > Sorry, but you haven't added any links to the source code. > https://github.com/bosqueviejo/myproto/ -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From bombadil@REDACTED Thu Jul 18 12:41:05 2013 From: bombadil@REDACTED (Manuel A. Rubio "Bombadil") Date: Thu, 18 Jul 2013 12:41:05 +0200 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: <109774b3905a3b34fc1e2ee158d14a5f@bosqueviejo.net> Hi, El 2013-07-18 12:08, Max Lapshin escribi?: > Sorry, but you haven't added any links to the source code. sorry, I forget it: https://github.com/bosqueviejo/myproto Thanks. Manuel Rubio. From js@REDACTED Thu Jul 18 13:19:50 2013 From: js@REDACTED (Schneider) Date: Thu, 18 Jul 2013 13:19:50 +0200 Subject: [erlang-questions] undefined function erlang:binary_to_float/1 In-Reply-To: References: <51E6A40D.1000908@globe.de> Message-ID: <51E7CF56.4080005@globe.de> yea, this should work. But It's looks like my Problem is a little bit more complicated: I have a C client sending some Data via UDP. And I want to write a Erlang program receiving this data. the Data is of the form {unsigned long, float} (both stored as 4Byte values) but the binary I receive is of the form {int, int}. How can i convert the second one to an erlang float? bg, Johannes On Wed 17 Jul 2013 04:30:45 PM CEST, Wes James wrote: > Alternative for now?? > > list_to_float(binary:bin_to_list(<<"10.0">>)). > 10.0 > > > On Wed, Jul 17, 2013 at 8:02 AM, Schneider > wrote: > > Hi List, > > I cannot find the function binary_to_float/1. > > using Erlang R14B04 (erts-5.8.5) on ubuntu precise. > What do I miss? > > bg, > Johannes > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > -- GLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nsterGLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nster 0251/5205 390 From tony@REDACTED Thu Jul 18 13:31:58 2013 From: tony@REDACTED (Tony Rogvall) Date: Thu, 18 Jul 2013 13:31:58 +0200 Subject: [erlang-questions] undefined function erlang:binary_to_float/1 In-Reply-To: <51E7CF56.4080005@globe.de> References: <51E6A40D.1000908@globe.de> <51E7CF56.4080005@globe.de> Message-ID: On 18 jul 2013, at 13:19, Schneider wrote: > yea, this should work. But It's looks like my Problem is a little bit more complicated: > > I have a C client sending some Data via UDP. And I want to write a Erlang program receiving this data. > > the Data is of the form {unsigned long, float} (both stored as 4Byte values) but the binary I receive is of the form {int, int}. > How can i convert the second one to an erlang float? > If the data is sent on the same machine without endian handling (byte order) then you can match (binary) data like: <> You can change native to 'big' or 'little' if that byte order used. Files formats and network protocol mostly select a byte order or flag it some how in the protocol specification. Hint. For intel based machines little endian is used. PowerPC / M680xx are big endian. /Tony > bg, > Johannes > > On Wed 17 Jul 2013 04:30:45 PM CEST, Wes James wrote: >> Alternative for now?? >> >> list_to_float(binary:bin_to_list(<<"10.0">>)). >> 10.0 >> >> >> On Wed, Jul 17, 2013 at 8:02 AM, Schneider > > wrote: >> >> Hi List, >> >> I cannot find the function binary_to_float/1. >> >> using Erlang R14B04 (erts-5.8.5) on ubuntu precise. >> What do I miss? >> >> bg, >> Johannes >> >> -- >> GLOBE Development GmbH >> K?nigsberger Strasse 260 >> 48157 M?nsterGLOBE Development GmbH >> K?nigsberger Strasse 260 >> 48157 M?nster >> 0251/5205 390 >> >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> > > > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Thu Jul 18 13:34:59 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Thu, 18 Jul 2013 13:34:59 +0200 Subject: [erlang-questions] undefined function erlang:binary_to_float/1 In-Reply-To: <51E7CF56.4080005@globe.de> References: <51E6A40D.1000908@globe.de> <51E7CF56.4080005@globe.de> Message-ID: I don't know what you mean by "the binary I receive is of the form {int, int}", but decoding binaries directly is done using the bitstring syntax: 5> Data = <<123,0,0,0,205,204,204,62>>. 6> <> = Data. % In case the numbers are encoded in little-endian. 7> A. 123 8> B. 0.4000000059604645 2013/7/18 Schneider > yea, this should work. But It's looks like my Problem is a little bit more > complicated: > > I have a C client sending some Data via UDP. And I want to write a Erlang > program receiving this data. > > the Data is of the form {unsigned long, float} (both stored as 4Byte > values) but the binary I receive is of the form {int, int}. > How can i convert the second one to an erlang float? > > bg, > Johannes > > > On Wed 17 Jul 2013 04:30:45 PM CEST, Wes James wrote: > >> Alternative for now?? >> >> list_to_float(binary:bin_to_**list(<<"10.0">>)). >> 10.0 >> >> >> On Wed, Jul 17, 2013 at 8:02 AM, Schneider > > wrote: >> >> Hi List, >> >> I cannot find the function binary_to_float/1. >> >> using Erlang R14B04 (erts-5.8.5) on ubuntu precise. >> What do I miss? >> >> bg, >> Johannes >> >> -- >> GLOBE Development GmbH >> K?nigsberger Strasse 260 >> 48157 M?nsterGLOBE Development GmbH >> K?nigsberger Strasse 260 >> 48157 M?nster >> 0251/5205 390 >> >> ______________________________**___________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> > >> http://erlang.org/mailman/__**listinfo/erlang-questions >> >> > >> >> >> > > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Jul 18 18:25:40 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 18 Jul 2013 12:25:40 -0400 Subject: [erlang-questions] Are crash dumps collecting my garbage? Message-ID: <20130718162539.GD3229@ferdair.local> Hi everyone. I was investigating a refc binary leak in a couple of production nodes, and had a few questions regarding Erlang crash dump following some discoveries I made. tl;dr: is the Erlang VM garbage-collecting refc binary references when generating the crash dump or am I not reading the dumps right? -- First of all, I diagnosed the binary refc leak problem by using a function a bit like follows: f(MostLeaky). MostLeaky = fun(N) -> lists:sublist( lists:usort( fun({K1,V1},{K2,V2}) -> {V1,K1} =< {V2,K2} end, [try {_,Pre} = erlang:process_info(Pid, binary), erlang:garbage_collect(Pid), {_,Post} = erlang:process_info(Pid, binary), {Pid, length(Post)-length(Pre)} catch _:_ -> {Pid, 0} end || Pid <- processes()]), N) end. %% Pairs = MostLeaky(25). I ran this function on a production node, using the 'binary' option of process_info/2 to get the list of binaries referenced by the process. I quickly found I had processes leaking dozens to hundreds of thousands of them. I took a similar node, running the same code at the same time in the same cluster and under a similar load and let it crash dump once it got out of memory. I noticed that the fields `=proc_heap:Pid` in the crash dump often referred to binaries (in `=binary:Id` fields) by using `:Yc:Hex` and decided to write a script[1] to find what processes were hogging refc binary references when the node died. However, I found out that this script gave me output that made it look ,ike there was no actual refc leak (highest counts were fully reasonable), and there was a huge discrepancy in the values returned by the `=memory` field and the individual binary sizes (calculated by using the prefix in `HexPrefix:HexBinary` in each `=binary` field). The results I had, for example, were: binary: 3360361264 binary-memory-counted: 138502020 I expect a difference to be had due to how they're stored with the refcounts, fragmentation and everything, but I do not expect this to cause an overhead of 24:1 between the two data sets I counted, which would be mighty scary. Have I made some kind of mistake in my script or understanding of the crash dump when counting for refs, or is the VM really omitting the garbage-collectable references to refc binaries when dumping? Regards, Fred. [1]: https://gist.github.com/ferd/6030174 (awk to show biggest processes hogging references, and refc binaries with their respective refc count) From wbin00@REDACTED Thu Jul 18 19:06:46 2013 From: wbin00@REDACTED (Bin Wang) Date: Fri, 19 Jul 2013 01:06:46 +0800 Subject: [erlang-questions] What is "key" in rb:filter Message-ID: In the doc of rb:filter, it says: filter(Filters) Filters = [filter()] filter() = {Key, Value} But what is the "key" means? From mahesh@REDACTED Thu Jul 18 19:23:14 2013 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Thu, 18 Jul 2013 10:23:14 -0700 Subject: [erlang-questions] An approach to common test In-Reply-To: References: Message-ID: <6E51C0D8-3682-4898-B0BE-798A813570D1@dieswaytoofast.com> That would actually work quite well, but you need to be sure to tear down your connections etc. in end_per_group (otherwise you'll end up w/ duplicated connections, errors, etc.) In general, we tend to do this somewhat hierarchically, i.e. * init_per_suite : "Global" setup/initializations, that are done just once for the entire test run * init_per_group : Any additional setup/initializations necessary for the specific group of testcases being run * init_per_testcase : Any setup/initialization necessary for this (and only this!) testcase Do remember to tear down pretty much everything you setup/initialized in the init_per_*. Yes, the temptation is always there to say "yeah, the process is going to get killed so what do I care", but trust me, it *will* come back to bite you. Cheers Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Google+ | Blog | Twitter | LinkedIn On Jul 17, 2013, at 11:46 AM, Maruthavanan Subbarayan wrote: > HI All, > > I am new to common test in erlang, I just wanted to know how to approach the below scenario in common test. > > I have an application that would send group of multiple messages to RMQ, and expects response in back and I have to validate in testcase, so I thought of doing the below. > > all() -> > [{group,general_test_group}, {group,rmq_test_group}] > > groups() -> > [{general_group_test, [sequence], GeneralTestList} > ,{rmq_test_group, [sequence], RmqTestList}]. > > so I thought I can have init_per_group function doing all initialization stuff like queue/exchange creation and bindings and subscribings. > > But I would not be getting the RMQ messages to my test case process, or I would endup in duplication by writing the above in all testcases. > > Thanks in advance, > Marutha > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Jul 18 19:50:52 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 18 Jul 2013 21:50:52 +0400 Subject: [erlang-questions] An approach to common test In-Reply-To: <6E51C0D8-3682-4898-B0BE-798A813570D1@dieswaytoofast.com> References: <6E51C0D8-3682-4898-B0BE-798A813570D1@dieswaytoofast.com> Message-ID: I also advise you to run tests with [parallel] option. It will quickly teach you to make them independent and run much faster =) From sylvain.benner@REDACTED Fri Jul 19 04:56:06 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Thu, 18 Jul 2013 22:56:06 -0400 Subject: [erlang-questions] Struggling with ct_master and ssh Message-ID: Dear community, I am a developer working in the gaming industry and we have chosen Erlang to develop an internal tool running on our private cloud platform. I'm currently learning Common Test via the great LYSE book and I am not able to make the distributed tests work. The sample can be found in the source code archive of the book. It can be downloaded on the cover page: http://learnyousomeerlang.com/ The sample is in the *ct* directory. First I had to update the dist.spec file to make it understable on my system (R16B01 esl distribution, Ubuntu 12.04 or 13.04, zsh) like this: %% ------------------------------------------------------------------------------ {node, a, 'a@REDACTED'}. {node, b, 'b@REDACTED'}. {init, [a,b], [{node_start, [{monitor_master, true}]}]}. {alias, demo, "./demo/"}. {alias, meeting, "./meeting/"}. {logdir, master, "./logs/"}. {logdir, all_nodes, "./logs/"}. {suites, [b], meeting, all}. {suites, [a], demo, all}. {skip_cases, [a], demo, basic_SUITE, test2, "This test fails on purpose"}. %% ------------------------------------------------------------------------------ 1) If I create manually the nodes a and b, the spec runs flawlessly. 2) If I don't than I've got boot_timeout errors: %% ------------------------------------------------------------------------------ (ct@REDACTED)13> ct_master:run("dist.spec"). === Master Logdir === /home/sylvain/Downloads/learn-you-some-erlang/ct/logs === Master Logger process started === <0.284.0> ssh password: Failed to start node 'a@REDACTED' with callback ct_slave! Reason: boot_timeout ssh password: Failed to start node 'b@REDACTED' with callback ct_slave! Reason: boot_timeout WARNING: Node 'a@REDACTED' is not alive but has eval option WARNING: Node 'b@REDACTED' is not alive but has eval option The following nodes are inaccessible: ['a@REDACTED', 'b@REDACTED'] Proceed(p), Rescan(r) or Abort(a)? [p/r/a]>a ** exception exit: aborted %% ------------------------------------------------------------------------------ I dug into ct_slave.erl and replicated the ssh commands of spawn_remote_node/3 in a shell: %% ------------------------------------------------------------------------------ (testssh@REDACTED)5> {ok, SSHConnRef} = ssh:connect("localhost.local", 22, []). ssh password: {ok,<0.60.0>} (testssh@REDACTED)6> {ok, SSHChannelId} = ssh_connection:session_channel(SSHConnRef, infinity). {ok,0} (testssh@REDACTED)7> net_adm:ping('node_from_ssh@REDACTED'). pang (testssh@REDACTED)8> ssh_connection:exec(SSHConnRef, SSHChannelId, "erl -detached -noinput -name node_from_ssh@REDACTED", infinity). success (testssh@REDACTED)9> net_adm:ping('node_from_ssh@REDACTED'). pong %% ------------------------------------------------------------------------------ It works. Then I tried ssh_connection:setenv/5 and all I get is a failure so it may be causing some trouble if ct_slave tries to set some environment variable in ssh_setenv/3. I started to walk in OTP sources to find the message sent to set an environment variable and at the same time I though it was time to join the ML. So here are the questions: 1) Do you have any idea about the ssh_connection:setenv failure ? 2) Do I need ct_master to test a protocol ? What is the erlang way for testing protocols between 2 nodes ? I have the feeling that ct_master is too much for what I need but I was seduced by its set of possibilities and would like to use it to run my suites. Cheers, syl20bnr -------------- next part -------------- An HTML attachment was scrubbed... URL: From zjfhappy@REDACTED Fri Jul 19 14:29:04 2013 From: zjfhappy@REDACTED (=?UTF-8?B?5byg5L+K5bOw?=) Date: Fri, 19 Jul 2013 20:29:04 +0800 (CST) Subject: [erlang-questions] is it a mensia bug?mnesia:load_textfile("/root/db4"). Error in line:61 of:"/root/db4" syntax error before: Ref In-Reply-To: References: Message-ID: <510c38ee.1272a.13ff6e7a868.Coremail.zjfhappy@126.com> thank you very much, At 2013-07-17 15:50:52,"H?kan Mattsson" wrote: >The format of the text file is plain Erlang terms. It cannot cope with >references and pids. Your text file contains a reference. > >Do not use mnesia:dump_to_textfile/ and mnesia:load_textfile/1. They >should have been deprecated years ago. > >/H?kan > >On Wed, Jul 17, 2013 at 4:46 AM, ??? wrote: >> 1, i get a error here when load_textfile: >> (ccmanager@REDACTED)1> >> mnesia:load_textfile("/root/db-20130716173944"). >> Error in line:61 of:"/root/db-20130716173944" syntax error before: Ref >> {error,read} >> (ccmanager@REDACTED)2> >> >> 2, and the db file is follow here with line number: >> 57 {vpool,"c7be3db1-0176-4efb-8676-47286ef2629d",null,<<"asfdf_?">>,0,0, >> 58 1373967539,null,0,0,0, >> 59 ["192.168.1.104","192.168.1.105","192.168.1.106"], >> 60 []}. >> 61 >> {task,"6c6ecf49-20b6-438c-82e5-daca138fe645",#Ref<0.0.0.238759>,image_to_vm, >> 62 'ccagent@REDACTED',<10230.1586.0>,1373967277,1373967278, >> 63 edog_task_h, >> 64 [{diskid,"ec46d38a-679a-41e3-9a75-f113f93eb33c"}], >> 65 [{ok,ok}], >> 66 true}. >> 67 >> {task,"79f136d5-53e3-4b70-ac80-0ac1943fb4fa",#Ref<0.0.0.253168>,image_to_vm, >> 68 >> 'ccagent@REDACTED',<10231.877.0>,1373967323,1373967324,edog_task_h, >> 69 [{diskid,"f9f474c2-3277-4b93-87fe-bcd71661f763"}], >> 70 [{ok,ok}], >> 71 true}. >> >> 3, >> the erl version is : >> [root@REDACTED ~]#erl >> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] >> [hipe] [kernel-poll:false] >> >> Eshell V5.9.1 (abort with ^G) >>> >> >> is this a mensia bug? >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Jul 19 14:31:12 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 19 Jul 2013 08:31:12 -0400 Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: References: Message-ID: <20130719123111.GD9952@ferdmbp.local> Answers inline. On 07/18, Sylvain Benner wrote: > I'm currently learning Common Test via the great LYSE book and I am not > able to make the distributed tests work. > Glad you like it! Let's see for the CT stuff. > %% > ------------------------------------------------------------------------------ > {node, a, 'a@REDACTED'}. > {node, b, 'b@REDACTED'}. < [...] > ------------------------------------------------------------------------------ > > 1) If I create manually the nodes a and b, the spec runs flawlessly. > 2) If I don't than I've got boot_timeout errors: > > %% > ------------------------------------------------------------------------------ > (ct@REDACTED)13> ct_master:run("dist.spec"). > === Master Logdir === First problem is right here! You started a node with a short name (-sname) but the spec mentions nodes with long names (-name, using a '.' in their name elevates them to FQDN). Start the master node with erl -name ct@REDACTED ... And it should probably fix it. It's not possible for a short name node to discuss with a long-name node. > > I dug into ct_slave.erl and replicated the ssh commands of > spawn_remote_node/3 in a shell: > > %% > ------------------------------------------------------------------------------ > (testssh@REDACTED)5> {ok, SSHConnRef} = > ssh:connect("localhost.local", 22, []). > ssh password: > ... > (testssh@REDACTED)9> net_adm:ping('node_from_ssh@REDACTED'). > pong > %% > ------------------------------------------------------------------------------ > > It works. > And I would expect so given your node name tells me this one is actually using a long name to connect! That sounds like a confirmation for the soliution. > So here are the questions: > > 1) Do you have any idea about the ssh_connection:setenv failure ? No specific idea, but I'm guessing it's the name of the nodes that is your first issue. Once that's cleared, you can check and try over again. > 2) Do I need ct_master to test a protocol ? What is the erlang way for > testing protocols between 2 nodes ? To test between two nodes it can make sense to have two nodes. I like to try and find ways to avoid that, although it's not necessarily easy. For most TCP-based protocols, I try to just have the test code have a 'trusted' side and spawn a server, my test behaving as the client, or the other way around. It helps to test some behaivour, but not all of it. If you've got more time and want to make sure the protocol stands on its own, I would look into Quickcheck or PropEr as a way to model everything and do fancier testing. It's considerably trickier to get things right in there if you haven't done it before, though. Regards, Fred. From sylvain.benner@REDACTED Fri Jul 19 15:03:04 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Fri, 19 Jul 2013 06:03:04 -0700 (PDT) Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: <20130719123111.GD9952@ferdmbp.local> References: <20130719123111.GD9952@ferdmbp.local> Message-ID: <85eadf73-bade-43c4-8d4b-f1b06f73d880@googlegroups.com> Hello Fred, Thanks for the reply! I tried within a node@REDACTED and this is the same behavior. This time on work computer: Ubuntu 12.04, Erlang R16B01 so it seems I have the same issue on a 12.04 box and 13.04 box, although I have to check with a long name at home to confirm it happens on the 2 computers. I created a "ct_user" account for the purpose and copied the .erlang.cookie from my account to be sure they share the same. I verified this by remotely spawning a node over ssh from a node x@REDACTED and then ping it from ct0@REDACTED: it works. If I run the spec right after this, still boot_timeout. %% -------------------------------------------------------------------------------------------- (ct0@REDACTED)21> ct_master:run("dist.spec"). === Master Logdir === /home/sbenner/Downloads/learn-you-some-erlang/ct/logs === Master Logger process started === <0.190.0> Failed to start node 'a@REDACTED' with callback ct_slave! Reason: boot_timeout Failed to start node 'b@REDACTED' with callback ct_slave! Reason: boot_timeout WARNING: Node 'a@REDACTED' is not alive but has eval option WARNING: Node 'b@REDACTED' is not alive but has eval option The following nodes are inaccessible: ['a@REDACTED', 'b@REDACTED'] Proceed(p), Rescan(r) or Abort(a)? [p/r/a]>a ** exception exit: aborted (ct0@REDACTED)22> %% -------------------------------------------------------------------------------------------- Cheers, syl20bnr -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Fri Jul 19 15:21:31 2013 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Fri, 19 Jul 2013 15:21:31 +0200 Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: <85eadf73-bade-43c4-8d4b-f1b06f73d880@googlegroups.com> References: <20130719123111.GD9952@ferdmbp.local> <85eadf73-bade-43c4-8d4b-f1b06f73d880@googlegroups.com> Message-ID: How about 127.0.0.1? On 19 July 2013 15:03, Sylvain Benner wrote: > > Hello Fred, > > Thanks for the reply! > > I tried within a node@REDACTED and this is the same behavior. This > time on work computer: Ubuntu 12.04, Erlang R16B01 so it seems I have the > same issue on a 12.04 box and 13.04 box, although I have to check with a > long name at home to confirm it happens on the 2 computers. > > I created a "ct_user" account for the purpose and copied the > .erlang.cookie from my account to be sure they share the same. I verified > this by remotely spawning a node over ssh from a node x@REDACTED then ping it from ct0@REDACTED: > it works. > If I run the spec right after this, still boot_timeout. > > %% > -------------------------------------------------------------------------------------------- > (ct0@REDACTED)21> ct_master:run("dist.spec"). > === Master Logdir === > /home/sbenner/Downloads/learn-you-some-erlang/ct/logs > === Master Logger process started === > <0.190.0> > Failed to start node 'a@REDACTED' with callback ct_slave! Reason: > boot_timeout > Failed to start node 'b@REDACTED' with callback ct_slave! Reason: > boot_timeout > WARNING: Node 'a@REDACTED' is not alive but has eval option > WARNING: Node 'b@REDACTED' is not alive but has eval option > > The following nodes are inaccessible: ['a@REDACTED', > 'b@REDACTED'] > > Proceed(p), Rescan(r) or Abort(a)? [p/r/a]>a > ** exception exit: aborted > (ct0@REDACTED)22> > %% > -------------------------------------------------------------------------------------------- > > Cheers, > syl20bnr > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahesh@REDACTED Fri Jul 19 15:23:57 2013 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Fri, 19 Jul 2013 09:23:57 -0400 Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: <20130719123111.GD9952@ferdmbp.local> References: <20130719123111.GD9952@ferdmbp.local> Message-ID: This isn't really a solution - but I'd strongly recommend getting this working on the same node first (multiple vms, but the same node), and then moving on to multiple nodes? cheers Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Google+ | Blog | Twitter | LinkedIn On Jul 19, 2013, at 8:31 AM, Fred Hebert wrote: > Answers inline. > > On 07/18, Sylvain Benner wrote: >> I'm currently learning Common Test via the great LYSE book and I am not >> able to make the distributed tests work. >> > > Glad you like it! Let's see for the CT stuff. > >> %% >> ------------------------------------------------------------------------------ >> {node, a, 'a@REDACTED'}. >> {node, b, 'b@REDACTED'}. > < [...] >> ------------------------------------------------------------------------------ >> >> 1) If I create manually the nodes a and b, the spec runs flawlessly. >> 2) If I don't than I've got boot_timeout errors: >> >> %% >> ------------------------------------------------------------------------------ >> (ct@REDACTED)13> ct_master:run("dist.spec"). >> === Master Logdir === > > First problem is right here! You started a node with a short name > (-sname) but the spec mentions nodes with long names (-name, using a '.' > in their name elevates them to FQDN). Start the master node with > > erl -name ct@REDACTED ... > > And it should probably fix it. It's not possible for a short name node > to discuss with a long-name node. > >> >> I dug into ct_slave.erl and replicated the ssh commands of >> spawn_remote_node/3 in a shell: >> >> %% >> ------------------------------------------------------------------------------ >> (testssh@REDACTED)5> {ok, SSHConnRef} = >> ssh:connect("localhost.local", 22, []). >> ssh password: >> ... >> (testssh@REDACTED)9> net_adm:ping('node_from_ssh@REDACTED'). >> pong >> %% >> ------------------------------------------------------------------------------ >> >> It works. >> > > And I would expect so given your node name tells me this one is actually > using a long name to connect! That sounds like a confirmation for the > soliution. > >> So here are the questions: >> >> 1) Do you have any idea about the ssh_connection:setenv failure ? > > No specific idea, but I'm guessing it's the name of the nodes that is > your first issue. Once that's cleared, you can check and try over again. > >> 2) Do I need ct_master to test a protocol ? What is the erlang way for >> testing protocols between 2 nodes ? > > To test between two nodes it can make sense to have two nodes. I like to > try and find ways to avoid that, although it's not necessarily easy. For > most TCP-based protocols, I try to just have the test code have a > 'trusted' side and spawn a server, my test behaving as the client, or > the other way around. It helps to test some behaivour, but not all of > it. > > If you've got more time and want to make sure the protocol stands on its > own, I would look into Quickcheck or PropEr as a way to model everything > and do fancier testing. It's considerably trickier to get things right > in there if you haven't done it before, though. > > Regards, > Fred. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.benner@REDACTED Fri Jul 19 15:30:48 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Fri, 19 Jul 2013 06:30:48 -0700 (PDT) Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: References: <20130719123111.GD9952@ferdmbp.local> <85eadf73-bade-43c4-8d4b-f1b06f73d880@googlegroups.com> Message-ID: <6ec86d1a-62d5-418f-9de9-79fb1f558535@googlegroups.com> Not working either. Thanks for the proposition. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Fri Jul 19 18:02:08 2013 From: dmercer@REDACTED (David Mercer) Date: Fri, 19 Jul 2013 11:02:08 -0500 Subject: [erlang-questions] Short Names vs. Long Names Message-ID: <01cb01ce8499$52a0b690$f7e223b0$@gmail.com> Just wondering about the distinction between short-name nodes and long-name nodes. You must use one or the other, but never the twain should mix. Is this distinction ? having the two different ways to name nodes ? by design or an accident of history. If the former, can someone explain? Thanks. Cheers, DBM -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Fri Jul 19 18:23:31 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 19 Jul 2013 18:23:31 +0200 Subject: [erlang-questions] rebar, ct and travis-ci In-Reply-To: References: Message-ID: On Tue, Jul 16, 2013 at 10:39 AM, Pablo Vieytes wrote: > Hi, > I'm using common tests, rebar and travis-ci. It works fine but when some > test falis in travis I can't see the log files. > > My Makefile > > all: > ./rebar get-deps > ./rebar compile > bin/bootstrap > > clean: > ./rebar clean > > test: all > ./rebar ct skip_deps=true > > doc: all > bin/create_edoc.sh > > > > Any suggestion? Hi Pablo, have you tried to extract the info from ct's logs? $ rebar help ct === rebar_ct:ct === Run common_test suites. Valid rebar.config options: {ct_dir,"itest"} {ct_log_dir,"test/logs"} {ct_extra_params,"-boot start_sasl -s myapp"} {ct_use_short_names,true} Valid command line options: suites=foo,bar - run /foo_SUITE and /bar_SUITE case="mycase" - run individual test case foo_SUITE:mycase From matthew.simarmata@REDACTED Fri Jul 19 20:54:15 2013 From: matthew.simarmata@REDACTED (Matthew Simarmata) Date: Fri, 19 Jul 2013 14:54:15 -0400 Subject: [erlang-questions] Question about eldap - substring filter Message-ID: Hi, I made following eldap search eldap:search(S, [Base, Scope, {filter, eldap:equalityMatch("uid", "matthew")}]). and got below result: {ok,{eldap_search_result, [{eldap_entry, "uid=matthew,ou=Support,ou=People,dc=maxcrc,dc=com", [{"objectClass",["posixAccount","top","inetOrgPerson"]}, {"gidNumber",["0"]}, {"givenName",["matthew"]}, {"sn",["Simarmata"]}, {"displayName",["matthew Simarmata"]}, {"uid",["matthew"]}, {"homeDirectory",["/home/matthew"]}, {"loginShell",["matthew"]}, {"cn",["matthew Simarmata"]}, {"uidNumber",["64152"]}]}], []}} I want to filter based on substring on 'homeDirectory', and submit following eldap search: eldap:search(S, [Base, Scope, {filter, eldap:substrings("homeDirectory", [{any,"matthew"}])}, {attributes, ["homeDirectory"]}]). However I got empty result. I need help please? Thanks, MS -------------- next part -------------- An HTML attachment was scrubbed... URL: From mloftis@REDACTED Fri Jul 19 22:13:50 2013 From: mloftis@REDACTED (Michael Loftis) Date: Fri, 19 Jul 2013 13:13:50 -0700 Subject: [erlang-questions] Question about eldap - substring filter In-Reply-To: References: Message-ID: Can't offer any specific eldap advice, but, it's very likely that your LDAP Schema won't even allow a substring match there as the normal posixAccount schema will usually only specify equality matches. You'll need to change your LDAP server schema in order to allow a substring match in addition to equality matches. On Fri, Jul 19, 2013 at 11:54 AM, Matthew Simarmata wrote: > Hi, > > I made following eldap search > eldap:search(S, [Base, Scope, {filter, eldap:equalityMatch("uid", > "matthew")}]). > > and got below result: > > {ok,{eldap_search_result, > [{eldap_entry, > "uid=matthew,ou=Support,ou=People,dc=maxcrc,dc=com", > [{"objectClass",["posixAccount","top","inetOrgPerson"]}, > {"gidNumber",["0"]}, > {"givenName",["matthew"]}, > {"sn",["Simarmata"]}, > {"displayName",["matthew Simarmata"]}, > {"uid",["matthew"]}, > {"homeDirectory",["/home/matthew"]}, > {"loginShell",["matthew"]}, > {"cn",["matthew Simarmata"]}, > {"uidNumber",["64152"]}]}], > []}} > > > I want to filter based on substring on 'homeDirectory', and submit following > eldap search: > > eldap:search(S, [Base, Scope, {filter, eldap:substrings("homeDirectory", > [{any,"matthew"}])}, {attributes, ["homeDirectory"]}]). > > However I got empty result. > > I need help please? > > Thanks, > MS > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler From mloftis@REDACTED Fri Jul 19 22:14:18 2013 From: mloftis@REDACTED (Michael Loftis) Date: Fri, 19 Jul 2013 13:14:18 -0700 Subject: [erlang-questions] Question about eldap - substring filter In-Reply-To: References: Message-ID: On Fri, Jul 19, 2013 at 1:13 PM, Michael Loftis wrote: > Can't offer any specific eldap advice, but, it's very likely that your > LDAP Schema won't even allow a substring match there as the normal > posixAccount schema will usually only specify equality matches. > You'll need to change your LDAP server schema in order to allow a > substring match in addition to equality matches. That is to say an equality match on the homeDirectory attribute. From paul@REDACTED Fri Jul 19 23:31:58 2013 From: paul@REDACTED (Paul Rubin) Date: Fri, 19 Jul 2013 14:31:58 -0700 Subject: [erlang-questions] advice needed: cleanup after a timeout (using gproc?) Message-ID: I'm running a server that does something sort of similar to inetd. It sits there quietly until a client connection arrives on a certain socket, at which point it starts up a service which handles client requests. Starting the service involves spinning up external database connections and some other slightly expensive things. While the service is running, more client connections (potentially a lot of them arriving very fast) can come and go. When the last client connection closes, I'd like to keep the service running for another minute or so (to avoid having to restart if new connections arrive soon afterwards), then shut it off if things have stayed quiet. The 1 minute doesn't have to be precise: let's say anything between 1 and 5 minutes is fine. There will actually be a lot of similar but separate instances of these services, i.e. there will be maybe a dozen active and a few thousand idle at any given moment. Expecting clients to update some global state before exiting isn't good, since per Erlang tradition, they might crash ungracefully. My thought is to use a gproc aggregated counter to track the number of client connections. Each new connection spawns a gen_server whose init increments the counter by 1, and the counter automatically decrements when the process terminates (through a monitor in the gproc server, I guess). I don't see an obvious way for my application to detect when the counter reaches zero. I can have a separate process waking up once a minute and checking the counts, but that doesn't say that a count has stayed zero for a full minute. E.g. the count can reach zero at time T, then some connections arrive and quickly exit, so the count is again zero at time T+60. I suppose I could update a timestamp in some ETS table whenever a connection arrives, but that doesn't seem in Erlang's mutation-avoiding spirit. Another idea is to add "zero-crossing detection" to gproc itself, but before I ask Ulf whether he'd accept such a patch, I thought I'd ask here if the idea is repulsive and if there are easier ways (it seems to me like a pattern potentially common enough to be useful). The notion is to pass gproc a callback ({M,F,A}) to associate with the counter key, that would get called whenever the aggregated count changes from zero to nonzero or the reverse. Then I'd supply a callback that starts a timer when the counter reaches zero, and cancels it if the counter becomes nonzero again before the timer expires. I guess it would be ok for zero crossing to use gproc:send despite the message passing overhead. It looks like I can *almost* do this with the conditional publication stuff in gproc_ps. But I'd need a way to generate events when an aggregated counter changes value, and the publish condition would have to know the internal ETS table representation of the aggregated counter, which doesn't seem good. Thanks for any thoughts. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihai@REDACTED Sat Jul 20 00:30:05 2013 From: mihai@REDACTED (Mihai Balea) Date: Fri, 19 Jul 2013 18:30:05 -0400 Subject: [erlang-questions] advice needed: cleanup after a timeout (using gproc?) In-Reply-To: References: Message-ID: The "traditional" way of doing this would be to keep track of connections yourself by monitoring connection processes and keeping count of them in your main service process. The gproc approach is elegant and would save you some effort but, as you already realized, you will have problems trying to figure out when the counter goes down to 0. Polling for 0 in a process would expose you to race conditions, in addition to potentially not being correct. Personally, I would probably go with the traditional approach. There might be 3rd party libraries that already implement this pattern, but I am not familiar with any. Anyways, no big deal to implement. Hope this helps, Mihai On Jul 19, 2013, at 5:31 PM, Paul Rubin wrote: > I'm running a server that does something sort of similar to inetd. It sits there quietly until a client connection arrives on a certain socket, at which point it starts up a service which handles client requests. Starting the service involves spinning up external database connections and some other slightly expensive things. While the service is running, more client connections (potentially a lot of them arriving very fast) can come and go. When the last client connection closes, I'd like to keep the service running for another minute or so (to avoid having to restart if new connections arrive soon afterwards), then shut it off if things have stayed quiet. The 1 minute doesn't have to be precise: let's say anything between 1 and 5 minutes is fine. There will actually be a lot of similar but separate instances of these services, i.e. there will be maybe a dozen active and a few thousand idle at any given moment. Expecting clients to update some global state before exiting isn't good, since per Erlang tradition, they might crash ungracefully. > > My thought is to use a gproc aggregated counter to track the number of client connections. Each new connection spawns a gen_server whose init increments the counter by 1, and the counter automatically decrements when the process terminates (through a monitor in the gproc server, I guess). I don't see an obvious way for my application to detect when the counter reaches zero. I can have a separate process waking up once a minute and checking the counts, but that doesn't say that a count has stayed zero for a full minute. E.g. the count can reach zero at time T, then some connections arrive and quickly exit, so the count is again zero at time T+60. I suppose I could update a timestamp in some ETS table whenever a connection arrives, but that doesn't seem in Erlang's mutation-avoiding spirit. > > Another idea is to add "zero-crossing detection" to gproc itself, but before I ask Ulf whether he'd accept such a patch, I thought I'd ask here if the idea is repulsive and if there are easier ways (it seems to me like a pattern potentially common enough to be useful). The notion is to pass gproc a callback ({M,F,A}) to associate with the counter key, that would get called whenever the aggregated count changes from zero to nonzero or the reverse. Then I'd supply a callback that starts a timer when the counter reaches zero, and cancels it if the counter becomes nonzero again before the timer expires. I guess it would be ok for zero crossing to use gproc:send despite the message passing overhead. > > It looks like I can *almost* do this with the conditional publication stuff in gproc_ps. But I'd need a way to generate events when an aggregated counter changes value, and the publish condition would have to know the internal ETS table representation of the aggregated counter, which doesn't seem good. > > Thanks for any thoughts. > > Paul > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From paul@REDACTED Sat Jul 20 01:13:33 2013 From: paul@REDACTED (Paul Rubin) Date: Fri, 19 Jul 2013 16:13:33 -0700 Subject: [erlang-questions] advice needed: cleanup after a timeout (using gproc?) In-Reply-To: References: Message-ID: Thanks. I'm already using gproc heavily in the application, to track what the different connection processes are doing, so I figured it was the natural way to aggregate the connection counts. Monitoring the counts (and I guess maintaining an ETS table) myself would amount to duplicating existing gproc functionality with slight changes. But I guess I can do that. I'm starting to understand the machinery of this stuff a bit better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.benner@REDACTED Sat Jul 20 02:36:50 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Fri, 19 Jul 2013 17:36:50 -0700 (PDT) Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: <85eadf73-bade-43c4-8d4b-f1b06f73d880@googlegroups.com> References: <20130719123111.GD9952@ferdmbp.local> <85eadf73-bade-43c4-8d4b-f1b06f73d880@googlegroups.com> Message-ID: ... although I have to check with a long name at home to confirm it happens > on the 2 computers. > Just verified and same error on my home computer. Does anybody encounter the same issue ? Should I post on the bug ML ? Cheers, syl20bnt -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.benner@REDACTED Sat Jul 20 02:40:30 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Fri, 19 Jul 2013 17:40:30 -0700 (PDT) Subject: [erlang-questions] Struggling with ct_master and ssh In-Reply-To: References: <20130719123111.GD9952@ferdmbp.local> Message-ID: <6818df68-1413-4f8a-8c92-169102a119b8@googlegroups.com> Thanks for the advice, this is what I'm doing for now, something like Fred mentioned. Still I want to be able to run ct_master with node_start :-) This isn't really a solution - but I'd strongly recommend getting this > working on the same node first (multiple vms, but the same node), and then > moving on to multiple nodes? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Jul 20 08:39:03 2013 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 20 Jul 2013 08:39:03 +0200 Subject: [erlang-questions] advice needed: cleanup after a timeout (using gproc?) In-Reply-To: References: Message-ID: <509F9B9E-7918-4FCA-A9BC-1FDE7F824F14@feuerlabs.com> 19 jul 2013 kl. 23:31 skrev Paul Rubin : > Another idea is to add "zero-crossing detection" to gproc itself, but before I ask Ulf whether he'd accept such a patch I don't think I would accept such a change to aggregated counters, since that would require an extra lookup for each call to update_counter() *and* a race condition, since the check could trigger in any process. One thing that would make sense is to complement gproc with measurement facilities that could e.g. trigger on thresholds. I've thought about that for quite a while, but nowadays, perhaps some alignment with folsom would be the way to go... *That* would be an area where I would welcome patches and suggestions. :) BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://feuerlabs.com From me@REDACTED Sat Jul 20 01:16:29 2013 From: me@REDACTED (Anton Nikishaev) Date: Sat, 20 Jul 2013 03:16:29 +0400 Subject: [erlang-questions] how can I run emacs from erlang References: <3E7B363F-3F67-4171-9CCB-84DF4DD10EFC@cs.otago.ac.nz> Message-ID: "Richard A. O'Keefe" writes: > I have a little C program called ttyname, whose source code is > #include > #include > > int main(void) { > char *s = ttyname(0); > return s == 0 || puts(s) == EOF; > } /usr/bin/tty >> os:cmd("emacs"). > answers > "emacs: standard input is not a tty\n". > >> os:cmd("emacs -nw /dev/tty"). > answers > "/bin/sh: line 1: /dev/tty: Device not configured\n" > > So what I sometimes do is > > % export TTYNAME=`ttyname` > % erl > ... >> TTY = os:getenv("TTYNAME"). > ... >> os:cmd("emacs -nw <" ++ TTY ++ " >" ++ TTY). > > and that works fine. emacs -t DEVICE -- lelf From pierrefenoll@REDACTED Sat Jul 20 16:12:41 2013 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Sat, 20 Jul 2013 16:12:41 +0200 Subject: [erlang-questions] Why is there an atom table? Message-ID: Hi, So, atoms are stored in a shared table, using locks to read & write, and never garbage-collected. Well we know the issues that emerge from this. What I can't find on the Web is the purpose of this table. Why does Erlang needs to assign a unique hash to a <255-characters string and put it in memory? I thought atoms were just like enums, thus the only actions needed on them were ==, =:= and the various conversions to string or binary. An atom (approximately) corresponds to this regexp: [a-z'][a-zA-Z0-9_']{,254}. Its syntactic representation **already** supports the actions I cited. Why not get rid of the atom table and and just use strncmp(3)? Cheers, Pierre -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Sat Jul 20 17:12:10 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Sat, 20 Jul 2013 17:12:10 +0200 Subject: [erlang-questions] Why is there an atom table? In-Reply-To: References: Message-ID: The key thing with atoms is that they are immediates. This means that the atom term it self takes zero heap memory in an Erlang process. Atoms are extremely efficient. A matching '=:=' operation, the normal thing to do with atoms, only require a one word compare. Atoms need to store the textual representation somewhere and this is where the table comes in. The immediate term is an reference to this representation. It is system wide shared to save memory of course. When developer uses atoms as strings and not as "enums" they are still space efficient but will require lookups to the atom table. Not a huge deal but annoying. To answer your question: Getting rid of the table and storing textual representation in heap memory will increase memory consumption dramatically. There are suggested comprises, one is https://github.com/erlang/eep/blob/master/eeps/eep-0020.md for instance. The one thing to remember: we want atoms to be *immediates* - it is an extremely important feature. // Bj?rn-Egil 2013/7/20 Pierre Fenoll > Hi, > > So, atoms are stored in a shared table, using locks to read & write, and > never garbage-collected. > Well we know the issues that emerge from this. > > What I can't find on the Web is the purpose of this table. > Why does Erlang needs to assign a unique hash to a <255-characters string > and put it in memory? > > I thought atoms were just like enums, thus the only actions needed on them > were ==, =:= and the various conversions to string or binary. > An atom (approximately) corresponds to this regexp: > [a-z'][a-zA-Z0-9_']{,254}. > Its syntactic representation **already** supports the actions I cited. > > Why not get rid of the atom table and and just use strncmp(3)? > > Cheers, > Pierre > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Sun Jul 21 00:00:07 2013 From: serge@REDACTED (Serge Aleynikov) Date: Sat, 20 Jul 2013 18:00:07 -0400 Subject: [erlang-questions] Arrow keys in erl started by cygwin mintty terminal Message-ID: <51EB0867.9050202@aleynikov.org> Hi, Does anyone have an idea how to configure a cygwin terminal running erl on Windows so that it would recognize the up/down arrow keys? While the up/down keys work fine in werl and in erl that was started from Windows command prompt, I have a trouble with them working in erl started by cygwin's mintty - the cursor moves up/down screen beyond the current line of the shell, and my attempts to address the issue by editing ~/.inputrc haven't succeeded. Serge From esumbar@REDACTED Mon Jul 22 02:50:51 2013 From: esumbar@REDACTED (Edmund Sumbar) Date: Sun, 21 Jul 2013 18:50:51 -0600 Subject: [erlang-questions] propagation of exit signals among linked processes Message-ID: Hi all, I'm still new to Erlang and am having difficulty understanding why the system (or shell) hangs when a process at the end of a chain of linked processes crashes. Here is the parent module (exits are being trapped). -module (parent). -export ([start/0, init/0]). start() -> spawn_link(parent, init, []). init() -> process_flag(trap_exit, true), apply(child, start, []), loop(). loop() -> receive {'EXIT', Pid, _Reason} -> io:format("Trapped exit pid ~p~n", [Pid]); %%loop(); Other -> io:format("Other ~p~n", [Other]) end. And here is the child. The child is started by the parent. -module (child). -export ([start/0, calc/1, loop/0]). start() -> register(child, spawn_link(child, loop, [])). calc(Msg) -> child ! {Msg, self()}, receive Result -> Result end. loop() -> receive {Msg, From} -> >From ! Msg + 2, loop() end. When the child crashes, the system hangs. I have to interrupt the process to restore the shell. 263> parent:start(). <0.620.0> 264> child:calc(a). Trapped exit pid <0.621.0> =ERROR REPORT==== 21-Jul-2013::15:11:03 === Error in process <0.621.0> with exit value: {badarith,[{child,loop,0,[{file,"child.erl"},{line,14}]}]} User switch command --> i --> c ** exception exit: killed 265> i(). On the other hand, if the parent code is altered to not trap exits, the crash takes down the child, parent, and shell, but the system does not hang. I immediately end up with a new shell process. 260> child:calc(a). =ERROR REPORT==== 21-Jul-2013::15:09:52 === Error in process <0.607.0> with exit value: {badarith,[{child,loop,0,[{file,"child.erl"},{line,14}]}]} ** exception exit: badarith in function child:loop/0 (child.erl, line 14) 261> Sorry for the rookie question, but I just can not figure it out. Can someone explain this behaviour or point me to some documentation. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jul 22 06:48:10 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 22 Jul 2013 16:48:10 +1200 Subject: [erlang-questions] Why is there an atom table? In-Reply-To: References: Message-ID: <1DDC734F-165F-4B34-BB33-501171E80B6B@cs.otago.ac.nz> On 21/07/2013, at 2:12 AM, Pierre Fenoll wrote: > Hi, > > So, atoms are stored in a shared table, using locks to read & write, and never garbage-collected. > Well we know the issues that emerge from this. We do talk and have been talking about alternatives for a L O N G time. I suspect that it would be possible to use a lock-free hash table but those things make my head hurt. > > What I can't find on the Web is the purpose of this table. You will find it in the better books. Actually, there _are_ places on the web that explain it, and you _can_ find them with straightforward Googling. > Why does Erlang needs to assign a unique hash to a <255-characters string and put it in memory? The limit on the length of an atom is highly problematic. Back in the 1980s, Quintus found themselves forced to raise the length limit on atoms to 1024. Modern Prolog and Lisp systems typically have no length limit other than available memory. Erlang is just about the only programming language I use where (1) the language *has* atoms (2) they are not usable to represent full file names. > I thought atoms were just like enums, thus the only actions needed on them were ==, =:= and the various conversions to string or binary. Atoms can be compared using < as well. I'm not sure what language you have in mind when you talk about 'enums', but even in Java they are *unique*. It is precisely *because* atoms can be used like enums that they need to be stored uniquely > An atom (approximately) corresponds to this regexp: [a-z'][a-zA-Z0-9_']{,254}. No, an atom corresponds to /^.{0,255}$/. There is no restriction on the characters that can be in an atom name. Even \0 is allowed, anywhere. > Its syntactic representation **already** supports the actions I cited. Yes, but it doesn't support them *FAST*. Storing atoms uniquely means that - testing for identity takes constant time, only a couple of instructions - when converting an atom to binary, perhaps for sending to another node, repeated occurrences of the same atom are recognised *FAST* and sent *compactly*. Lisp, Prolog, Erlang, Smalltalk, and Ruby all have symbols for basically the same reasons: - things that can be compared fast - and so make great keys for hash tables - which the language implementation takes private advantage of - safe tokens that cannot be changed (Lisp, Smalltalk, and Ruby strings are mutable and the construction of Prolog strings can be unwound) > > Why not get rid of the atom table and and just use strncmp(3)? strncmp(3) would not work because \0 is legal _inside_ an atom and does not terminate the name. In any case, strncmp() is much slower than a simple pointer comparison. It's a typical computing tradeoff: - if creating atoms were extremely common and comparing them rare, it would not make sense to have an atom table, - but since comparing atoms is *extremely* common and creating atoms *much* rarer, it makes sense to slow down creation in order to speed up comparison. From esumbar@REDACTED Mon Jul 22 07:18:32 2013 From: esumbar@REDACTED (Edmund Sumbar) Date: Sun, 21 Jul 2013 23:18:32 -0600 Subject: [erlang-questions] propagation of exit signals among linked processes In-Reply-To: References: Message-ID: Oops, forgot to mention my Erlang version. Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace] Eshell V5.10.1 (abort with ^G) On Sun, Jul 21, 2013 at 6:50 PM, Edmund Sumbar wrote: > Hi all, > > I'm still new to Erlang and am having difficulty understanding why the > system (or shell) hangs when a process at the end of a chain of linked > processes crashes. > > Here is the parent module (exits are being trapped). > > -module (parent). > -export ([start/0, init/0]). > > start() -> > spawn_link(parent, init, []). > > init() -> > process_flag(trap_exit, true), > apply(child, start, []), > loop(). > > loop() -> > receive > {'EXIT', Pid, _Reason} -> > io:format("Trapped exit pid ~p~n", [Pid]); > %%loop(); > Other -> > io:format("Other ~p~n", [Other]) > end. > > And here is the child. The child is started by the parent. > > -module (child). > -export ([start/0, calc/1, loop/0]). > > start() -> > register(child, spawn_link(child, loop, [])). > > calc(Msg) -> > child ! {Msg, self()}, > receive Result -> Result end. > > loop() -> > receive > {Msg, From} -> > From ! Msg + 2, > loop() > end. > > When the child crashes, the system hangs. I have to interrupt the process > to restore the shell. > > 263> parent:start(). > <0.620.0> > 264> child:calc(a). > Trapped exit pid <0.621.0> > > =ERROR REPORT==== 21-Jul-2013::15:11:03 === > Error in process <0.621.0> with exit value: > {badarith,[{child,loop,0,[{file,"child.erl"},{line,14}]}]} > > > User switch command > --> i > --> c > ** exception exit: killed > 265> i(). > > On the other hand, if the parent code is altered to not trap exits, the > crash takes down the child, parent, and shell, but the system does not > hang. I immediately end up with a new shell process. > > 260> child:calc(a). > > =ERROR REPORT==== 21-Jul-2013::15:09:52 === > Error in process <0.607.0> with exit value: > {badarith,[{child,loop,0,[{file,"child.erl"},{line,14}]}]} > > ** exception exit: badarith > in function child:loop/0 (child.erl, line 14) > 261> > > Sorry for the rookie question, but I just can not figure it out. Can > someone explain this behaviour or point me to some documentation. Thanks. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Mon Jul 22 08:36:01 2013 From: gleber.p@REDACTED (Gleb Peregud) Date: Mon, 22 Jul 2013 08:36:01 +0200 Subject: [erlang-questions] propagation of exit signals among linked processes In-Reply-To: References: Message-ID: When you run child:calc/1 it sends a message to the child, and immediately after that shell process executes second line of child:calc/1, which is message receive. Shell hangs, since it waits for a message which never arrives. If you are doing a request to a process which might crash before responding , monitor that process temporarily starting before sending the message and demonitoring after response is received - in case child dies you will get 'DOWN' message. HTH, Gleb Peregud Sent from mobile, sorry for typos. 22 lip 2013 08:21, "Edmund Sumbar" napisa?(a): > Oops, forgot to mention my Erlang version. > > Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8] [async-threads:10] > [hipe] [kernel-poll:false] [dtrace] > > Eshell V5.10.1 (abort with ^G) > > > On Sun, Jul 21, 2013 at 6:50 PM, Edmund Sumbar wrote: > >> Hi all, >> >> I'm still new to Erlang and am having difficulty understanding why the >> system (or shell) hangs when a process at the end of a chain of linked >> processes crashes. >> >> Here is the parent module (exits are being trapped). >> >> -module (parent). >> -export ([start/0, init/0]). >> >> start() -> >> spawn_link(parent, init, []). >> >> init() -> >> process_flag(trap_exit, true), >> apply(child, start, []), >> loop(). >> >> loop() -> >> receive >> {'EXIT', Pid, _Reason} -> >> io:format("Trapped exit pid ~p~n", [Pid]); >> %%loop(); >> Other -> >> io:format("Other ~p~n", [Other]) >> end. >> >> And here is the child. The child is started by the parent. >> >> -module (child). >> -export ([start/0, calc/1, loop/0]). >> >> start() -> >> register(child, spawn_link(child, loop, [])). >> >> calc(Msg) -> >> child ! {Msg, self()}, >> receive Result -> Result end. >> >> loop() -> >> receive >> {Msg, From} -> >> From ! Msg + 2, >> loop() >> end. >> >> When the child crashes, the system hangs. I have to interrupt the process >> to restore the shell. >> >> 263> parent:start(). >> <0.620.0> >> 264> child:calc(a). >> Trapped exit pid <0.621.0> >> >> =ERROR REPORT==== 21-Jul-2013::15:11:03 === >> Error in process <0.621.0> with exit value: >> {badarith,[{child,loop,0,[{file,"child.erl"},{line,14}]}]} >> >> >> User switch command >> --> i >> --> c >> ** exception exit: killed >> 265> i(). >> >> On the other hand, if the parent code is altered to not trap exits, the >> crash takes down the child, parent, and shell, but the system does not >> hang. I immediately end up with a new shell process. >> >> 260> child:calc(a). >> >> =ERROR REPORT==== 21-Jul-2013::15:09:52 === >> Error in process <0.607.0> with exit value: >> {badarith,[{child,loop,0,[{file,"child.erl"},{line,14}]}]} >> >> ** exception exit: badarith >> in function child:loop/0 (child.erl, line 14) >> 261> >> >> Sorry for the rookie question, but I just can not figure it out. Can >> someone explain this behaviour or point me to some documentation. Thanks. >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aveviort@REDACTED Mon Jul 22 10:19:06 2013 From: aveviort@REDACTED (?) Date: Mon, 22 Jul 2013 12:19:06 +0400 Subject: [erlang-questions] Short Names vs. Long Names Message-ID: Hi David, only nodes with the same type of names can cooperate. IMHO full names and short names are just a handy tools for different types of systems: e.g, if I make nodes in local network, I prefer using short names, but if I use computer in global network, I'll use long name. Long name contains full IP of computer, but IP can be converted to name. Regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From js@REDACTED Mon Jul 22 15:28:23 2013 From: js@REDACTED (Schneider) Date: Mon, 22 Jul 2013 15:28:23 +0200 Subject: [erlang-questions] SNMP Agent in erlang Message-ID: <51ED3377.7060204@globe.de> Hi List, I'm going to write a small SNMP agent and I'm wondering if its the best to use the Libs coming with Erlang (http://www.erlang.org/doc/apps/snmp) or if there is another way. The App in the link above looks maybe a little bit oversized for my project. Do I have to send the MIB Data in the form it's done there? Do I have to compile them every time I want to send some data? Or just once in the beginning to specify what kind of data can be send? bg, Johannes -- GLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nsterGLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nster 0251/5205 390 From roe.adrian@REDACTED Mon Jul 22 15:48:54 2013 From: roe.adrian@REDACTED (Adrian Roe) Date: Mon, 22 Jul 2013 14:48:54 +0100 Subject: [erlang-questions] SNMP Agent in erlang In-Reply-To: <51ED3377.7060204@globe.de> References: <51ED3377.7060204@globe.de> Message-ID: <1E5C371266B04CB4A29876323399FF7C@gmail.com> I found getting to grips with SNMP in Erlang challenging at first - not least because SNMP itself is big and complicated so it is inevitable that any implementation also be somewhat complicated. The most helpful starting points I came across were the guide at http://www.trapexit.org/SNMP_Quick_Start and the test cases in .../opt_source_root_dir/lib/snmp/test/ Good luck! Adrian -- Adrian Roe Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Monday, 22 July 2013 at 14:28, Schneider wrote: > Hi List, > > I'm going to write a small SNMP agent and I'm wondering if its the best > to use the > Libs coming with Erlang (http://www.erlang.org/doc/apps/snmp) or if > there is another way. > > The App in the link above looks maybe a little bit oversized for my > project. Do I have to > send the MIB Data in the form it's done there? Do I have to compile > them every time I > want to send some data? Or just once in the beginning to specify what > kind of data can be send? > > bg, > Johannes > > -- > GLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nsterGLOBE Development GmbH > K?nigsberger Strasse 260 > 48157 M?nster > 0251/5205 390 > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Mon Jul 22 16:57:14 2013 From: v@REDACTED (Valentin Micic) Date: Mon, 22 Jul 2013 16:57:14 +0200 Subject: [erlang-questions] Cost of hibernation Message-ID: <478AD72D-4A99-4DDF-9B98-424F9AA1C418@pharos-avantgard.com> Dear all, Erlang reference manual makes a following statement regarding erlang:hibernate/3 "erlang:hibernate(Module, Function, Args) Types: Module = Function = atom() Args = [term()] Puts the calling process into a wait state where its memory allocation has been reduced as much as possible, which is useful if the process does not expect to receive any messages in the near future. ?" I have never used the hibernate function fearing that performance penalty to be paid may be to high. In other words, I never really had to use it ;-) But now that I do, may I kindly ask if there is anyone out there that did successfully used this feature and has a clear idea regarding performance penalty. Would it be cheaper to store and retrieve a context using ETS (which would certainly complicate programming), or would a call to hibernate do the trick? Maybe another way to ask this question could be: what is the cost of calling a garbage collection frequently? Also, would anyone care to define a meaning of "the near future" within this context? Should one measure "the near future" in milliseconds, seconds or minutes? Kind regards V/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From esumbar@REDACTED Mon Jul 22 17:04:09 2013 From: esumbar@REDACTED (Edmund Sumbar) Date: Mon, 22 Jul 2013 09:04:09 -0600 Subject: [erlang-questions] propagation of exit signals among linked processes In-Reply-To: References: Message-ID: Very embarrassing. Your comments were very helpful. Thank you. On Mon, Jul 22, 2013 at 12:36 AM, Gleb Peregud wrote: > When you run child:calc/1 it sends a message to the child, and immediately > after that shell process executes second line of child:calc/1, which is > message receive. Shell hangs, since it waits for a message which never > arrives. If you are doing a request to a process which might crash before > responding , monitor that process temporarily starting before sending the > message and demonitoring after response is received - in case child dies > you will get 'DOWN' message. > > HTH, > Gleb Peregud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.polvorin@REDACTED Mon Jul 22 17:26:31 2013 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Mon, 22 Jul 2013 12:26:31 -0300 Subject: [erlang-questions] Cost of hibernation In-Reply-To: <478AD72D-4A99-4DDF-9B98-424F9AA1C418@pharos-avantgard.com> References: <478AD72D-4A99-4DDF-9B98-424F9AA1C418@pharos-avantgard.com> Message-ID: Hello, On 22 July 2013 11:57, Valentin Micic wrote: > Dear all, > > Erlang reference manual makes a following statement regarding > erlang:hibernate/3 > > "erlang:hibernate(Module, Function, Args) > > Types: > > Module = Function = atom() > Args = [term()] > > Puts the calling process into a wait state where its memory allocation has > been reduced as much as possible, which is useful if the process does not > expect to receive any messages in the near future. > > ?" > > I have never used the hibernate function fearing that performance penalty to > be paid may be to high. In other words, I never really had to use it ;-) > > But now that I do, may I kindly ask if there is anyone out there that did > successfully used this feature and has a clear idea regarding performance > penalty. > Would it be cheaper to store and retrieve a context using ETS (which would > certainly complicate programming), or would a call to hibernate do the > trick? > > Maybe another way to ask this question could be: what is the cost of calling > a garbage collection frequently? > > Also, would anyone care to define a meaning of "the near future" within this > context? Should one measure "the near future" in milliseconds, seconds or > minutes? As vague as it is, maybe this helps: I had used it in a few systems. In tsung for example (with hundreds of thousands of processes simulating user behavior), there was a big memory saving, and the time the processes where hibernated was in the range of seconds, not minutes. The difference in cpu usage in that case was barely noticeable > > Kind regards > > V/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Pablo Polvorin ProcessOne From v@REDACTED Mon Jul 22 18:18:30 2013 From: v@REDACTED (Valentin Micic) Date: Mon, 22 Jul 2013 18:18:30 +0200 Subject: [erlang-questions] Cost of hibernation In-Reply-To: References: <478AD72D-4A99-4DDF-9B98-424F9AA1C418@pharos-avantgard.com> Message-ID: <21729778-C076-4E66-847E-C18F3076648D@pharos-avantgard.com> Thanks Pablo, Just want to confirm that when you say: "The time the processes where hibernated was in the range of seconds?" you mean that an average time between two messages addressing the same process can be measured in seconds? Kind reagards V/ On 22 Jul 2013, at 5:26 PM, Pablo Polvorin wrote: > Hello, > On 22 July 2013 11:57, Valentin Micic wrote: >> Dear all, >> >> Erlang reference manual makes a following statement regarding >> erlang:hibernate/3 >> >> "erlang:hibernate(Module, Function, Args) >> >> Types: >> >> Module = Function = atom() >> Args = [term()] >> >> Puts the calling process into a wait state where its memory allocation has >> been reduced as much as possible, which is useful if the process does not >> expect to receive any messages in the near future. >> >> ?" >> >> I have never used the hibernate function fearing that performance penalty to >> be paid may be to high. In other words, I never really had to use it ;-) >> >> But now that I do, may I kindly ask if there is anyone out there that did >> successfully used this feature and has a clear idea regarding performance >> penalty. >> Would it be cheaper to store and retrieve a context using ETS (which would >> certainly complicate programming), or would a call to hibernate do the >> trick? >> >> Maybe another way to ask this question could be: what is the cost of calling >> a garbage collection frequently? >> >> Also, would anyone care to define a meaning of "the near future" within this >> context? Should one measure "the near future" in milliseconds, seconds or >> minutes? > As vague as it is, maybe this helps: > I had used it in a few systems. In tsung for example (with hundreds > of thousands of processes simulating user behavior), there was a big > memory saving, and the time the processes where hibernated was in the > range of seconds, not minutes. The difference in cpu usage in that > case was barely noticeable > >> >> Kind regards >> >> V/ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Pablo Polvorin > ProcessOne > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From pablo.polvorin@REDACTED Mon Jul 22 19:52:33 2013 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Mon, 22 Jul 2013 14:52:33 -0300 Subject: [erlang-questions] Cost of hibernation In-Reply-To: <21729778-C076-4E66-847E-C18F3076648D@pharos-avantgard.com> References: <478AD72D-4A99-4DDF-9B98-424F9AA1C418@pharos-avantgard.com> <21729778-C076-4E66-847E-C18F3076648D@pharos-avantgard.com> Message-ID: Hello On 22 July 2013 13:18, Valentin Micic wrote: > Thanks Pablo, > > Just want to confirm that when you say: > > "The time the processes where hibernated was in the range of seconds?" > > you mean that an average time between two messages addressing the same process can be measured in seconds? yes, exactly. > > Kind reagards > > V/ > > > On 22 Jul 2013, at 5:26 PM, Pablo Polvorin wrote: > >> Hello, >> On 22 July 2013 11:57, Valentin Micic wrote: >>> Dear all, >>> >>> Erlang reference manual makes a following statement regarding >>> erlang:hibernate/3 >>> >>> "erlang:hibernate(Module, Function, Args) >>> >>> Types: >>> >>> Module = Function = atom() >>> Args = [term()] >>> >>> Puts the calling process into a wait state where its memory allocation has >>> been reduced as much as possible, which is useful if the process does not >>> expect to receive any messages in the near future. >>> >>> ?" >>> >>> I have never used the hibernate function fearing that performance penalty to >>> be paid may be to high. In other words, I never really had to use it ;-) >>> >>> But now that I do, may I kindly ask if there is anyone out there that did >>> successfully used this feature and has a clear idea regarding performance >>> penalty. >>> Would it be cheaper to store and retrieve a context using ETS (which would >>> certainly complicate programming), or would a call to hibernate do the >>> trick? >>> >>> Maybe another way to ask this question could be: what is the cost of calling >>> a garbage collection frequently? >>> >>> Also, would anyone care to define a meaning of "the near future" within this >>> context? Should one measure "the near future" in milliseconds, seconds or >>> minutes? >> As vague as it is, maybe this helps: >> I had used it in a few systems. In tsung for example (with hundreds >> of thousands of processes simulating user behavior), there was a big >> memory saving, and the time the processes where hibernated was in the >> range of seconds, not minutes. The difference in cpu usage in that >> case was barely noticeable >> >>> >>> Kind regards >>> >>> V/ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> >> -- >> Pablo Polvorin >> ProcessOne >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -- Pablo Polvorin ProcessOne From sylvain.benner@REDACTED Mon Jul 22 23:14:54 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Mon, 22 Jul 2013 14:14:54 -0700 (PDT) Subject: [erlang-questions] CTH with code:add_paths/1 Message-ID: <8ebc6b4f-9ad3-455c-a2e8-27e3cc90eb4f@googlegroups.com> I have a CTH which starts and ends an OTP app in pre_init_per_suite and post_end_per_suite respectively. In the documentation there is mention of interactive and embedded mode, then in ct_run there is the -shell option but it ignores the -spec option and I have to start it manually. T start the tests more quickly, I would prefere to only type ct_run -spec my.spec on the command line and be able to add some code paths to the code server to be able to start my app. Is it possible to use code:add_paths/1 in a CTH when using ct_run -spec ? Cheers, syl20bnr -------------- next part -------------- An HTML attachment was scrubbed... URL: From seancribbs@REDACTED Tue Jul 23 04:09:50 2013 From: seancribbs@REDACTED (Sean Cribbs) Date: Mon, 22 Jul 2013 21:09:50 -0500 Subject: [erlang-questions] rebar, ct and travis-ci In-Reply-To: References: Message-ID: You should also follow the directions on the Travis CI blog for uploading build artifacts. http://about.travis-ci.org/blog/2012-12-18-travis-artifacts/ If you have an S3 account, you can upload the entire output of the ct run easily. On Fri, Jul 19, 2013 at 11:23 AM, Tuncer Ayaz wrote: > On Tue, Jul 16, 2013 at 10:39 AM, Pablo Vieytes > wrote: > > Hi, > > I'm using common tests, rebar and travis-ci. It works fine but when some > > test falis in travis I can't see the log files. > > > > My Makefile > > > > all: > > ./rebar get-deps > > ./rebar compile > > bin/bootstrap > > > > clean: > > ./rebar clean > > > > test: all > > ./rebar ct skip_deps=true > > > > doc: all > > bin/create_edoc.sh > > > > > > > > Any suggestion? > > Hi Pablo, > > have you tried to extract the info from ct's logs? > > $ rebar help ct > === rebar_ct:ct === > Run common_test suites. > > Valid rebar.config options: > {ct_dir,"itest"} > {ct_log_dir,"test/logs"} > {ct_extra_params,"-boot start_sasl -s myapp"} > {ct_use_short_names,true} > Valid command line options: > suites=foo,bar - run /foo_SUITE and /bar_SUITE > case="mycase" - run individual test case foo_SUITE:mycase > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yann.secq@REDACTED Tue Jul 23 12:56:56 2013 From: yann.secq@REDACTED (Yann SECQ) Date: Tue, 23 Jul 2013 12:56:56 +0200 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) Message-ID: <51EE6178.9060205@univ-lille1.fr> Hi all, for those interested in nice new hardware that could fit wisely with erlang, you'll be happy to know that you can pre-order this new Raspberry Pi sized parallel computer: - http://shop.adapteva.com/collections/parallella/products/parallella-16 It seems that some folks within Erlang Solutions are already hacking this little beast: - http://www.parallella.org/2013/02/04/parallella-and-erlang-an-introduction/ - http://www.parallella.org/2013/05/25/explorations-in-erlang-with-the-parallela-a-prelude/ I tried to find if a specific erlang bundle was available for parallela there: - https://www.erlang-solutions.com/downloads/download-erlang-otp - or, http://www.erlang-embedded.com but have not been successful :( Do you know if some VM and tools (cf. OpenCL in the demo) will be packaged and released by Erlang Solutions in the near future ? Said otherwise, should I buy my parallela right now or wait some months ? ;) Thank you, yann. -- Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq www.ouverture-independance.fr | www.sauvonsluniversite.com -- "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux From karol.urbanski@REDACTED Tue Jul 23 13:19:58 2013 From: karol.urbanski@REDACTED (Karol Urbanski) Date: Tue, 23 Jul 2013 13:19:58 +0200 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: <51EE6178.9060205@univ-lille1.fr> References: <51EE6178.9060205@univ-lille1.fr> Message-ID: <20130723111958.GA9162@dex.pl.erlangsystems.com> Hi Yann, we're working on releasing the erlang virtual machine for Parallella, and we're pretty close to completing the packaging process. If all goes well, we should have something up on the Erlang Solutions page by the end of this week. :) Best regards, Karol Urbanski On Tue, Jul 23, 2013 at 12:56:56PM +0200, Yann SECQ wrote: > Hi all, > > for those interested in nice new hardware that could fit wisely > with erlang, you'll be happy to know that you can pre-order this > new Raspberry Pi sized parallel computer: > - http://shop.adapteva.com/collections/parallella/products/parallella-16 > > It seems that some folks within Erlang Solutions are already > hacking this little beast: > - http://www.parallella.org/2013/02/04/parallella-and-erlang-an-introduction/ > - http://www.parallella.org/2013/05/25/explorations-in-erlang-with-the-parallela-a-prelude/ > > I tried to find if a specific erlang bundle was available for > parallela there: > - https://www.erlang-solutions.com/downloads/download-erlang-otp > - or, http://www.erlang-embedded.com > but have not been successful :( > > Do you know if some VM and tools (cf. OpenCL in the demo) will > be packaged and released by Erlang Solutions in the near future ? > > Said otherwise, should I buy my parallela right now or wait some > months ? ;) > > Thank you, yann. > > -- > Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq > www.ouverture-independance.fr | www.sauvonsluniversite.com > -- > "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From francesco@REDACTED Tue Jul 23 14:14:06 2013 From: francesco@REDACTED (Francesco Cesarini) Date: Tue, 23 Jul 2013 05:14:06 -0700 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: <20130723111958.GA9162@dex.pl.erlangsystems.com> References: <51EE6178.9060205@univ-lille1.fr> <20130723111958.GA9162@dex.pl.erlangsystems.com> Message-ID: And if you believe in rumors, the parallela board should come preinstalled with Erlang as part of the standard image.... F Karol Urbanski wrote: >Hi Yann, > >we're working on releasing the erlang virtual machine for Parallella, >and we're pretty close to completing the packaging process. If all goes >well, we should have something up on the Erlang Solutions page by the >end of this week. :) > >Best regards, >Karol Urbanski >On Tue, Jul 23, 2013 at 12:56:56PM +0200, Yann SECQ wrote: >> Hi all, >> >> for those interested in nice new hardware that could fit wisely >> with erlang, you'll be happy to know that you can pre-order this >> new Raspberry Pi sized parallel computer: >> - >http://shop.adapteva.com/collections/parallella/products/parallella-16 >> >> It seems that some folks within Erlang Solutions are already >> hacking this little beast: >> - >http://www.parallella.org/2013/02/04/parallella-and-erlang-an-introduction/ >> - >http://www.parallella.org/2013/05/25/explorations-in-erlang-with-the-parallela-a-prelude/ >> >> I tried to find if a specific erlang bundle was available for >> parallela there: >> - https://www.erlang-solutions.com/downloads/download-erlang-otp >> - or, http://www.erlang-embedded.com >> but have not been successful :( >> >> Do you know if some VM and tools (cf. OpenCL in the demo) will >> be packaged and released by Erlang Solutions in the near future ? >> >> Said otherwise, should I buy my parallela right now or wait some >> months ? ;) >> >> Thank you, yann. >> >> -- >> Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq >> www.ouverture-independance.fr | www.sauvonsluniversite.com >> -- >> "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -- Erlang Solutions Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.benner@REDACTED Tue Jul 23 14:12:18 2013 From: sylvain.benner@REDACTED (Sylvain Benner) Date: Tue, 23 Jul 2013 05:12:18 -0700 (PDT) Subject: [erlang-questions] CTH with code:add_paths/1 In-Reply-To: <8ebc6b4f-9ad3-455c-a2e8-27e3cc90eb4f@googlegroups.com> References: <8ebc6b4f-9ad3-455c-a2e8-27e3cc90eb4f@googlegroups.com> Message-ID: <5aab9d82-26ef-4b4b-9cb8-ca522a3bed32@googlegroups.com> Nevermind, ct_run changes the working directory so the passed relative code paths were incorrect. Le lundi 22 juillet 2013 17:14:54 UTC-4, Sylvain Benner a ?crit : > > I have a CTH which starts and ends an OTP app in pre_init_per_suite and > post_end_per_suite respectively. > In the documentation there is mention of interactive and embedded mode, > then in ct_run there is the -shell option but it ignores the -spec option > and I have to start it manually. T start the tests more quickly, I would > prefere to only type ct_run -spec my.spec on the command line and be able > to add some code paths to the code server to be able to start my app. > > Is it possible to use code:add_paths/1 in a CTH when using ct_run -spec ? > > Cheers, > syl20bnr > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lee.sylvester@REDACTED Tue Jul 23 16:03:22 2013 From: lee.sylvester@REDACTED (Lee Sylvester) Date: Tue, 23 Jul 2013 15:03:22 +0100 Subject: [erlang-questions] SSL Issue Message-ID: Hi guys, So, I have a Cowboy server running and I want to start using SSL certs to secure my services. Things compile fine and everything runs, but when I access a service, I get this output in the console: =ERROR REPORT==== 23-Jul-2013::09:56:29 === SSL: 1159: error:[<<48,130,6,220,48,130,5,196,160,3,2,1,2,2,16,15,199,72,40,33, 126,49,13, [snip] 45,193>>, <<48,130,6 [snip] 118,247,97>>] /usr/certs/cert.pem [{ssl_connection,init_certificates,8, [{file,"ssl_connection.erl"},{line,1155}]}, {ssl_connection,ssl_init,2,[{file,"ssl_connection.erl"},{line,1110}]}, {ssl_connection,init,1,[{file,"ssl_connection.erl"},{line,303}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}] Does anyone here have an idea as to what the issue might be? It's not a very helpful error :-) Thanks, Lee From lee.sylvester@REDACTED Tue Jul 23 19:48:32 2013 From: lee.sylvester@REDACTED (Lee Sylvester) Date: Tue, 23 Jul 2013 18:48:32 +0100 Subject: [erlang-questions] SSL Issue In-Reply-To: References: Message-ID: So, everything works if I use a self certified cert and key, but if I try to use an official cert, I get the error below. Does anyone have an idea? Thanks, Lee On 23 Jul 2013, at 15:03, Lee Sylvester wrote: > Hi guys, > > So, I have a Cowboy server running and I want to start using SSL certs to secure my services. Things compile fine and everything runs, but when I access a service, I get this output in the console: > > =ERROR REPORT==== 23-Jul-2013::09:56:29 === > SSL: 1159: error:[<<48,130,6,220,48,130,5,196,160,3,2,1,2,2,16,15,199,72,40,33, > 126,49,13, [snip] 45,193>>, > <<48,130,6 [snip] 118,247,97>>] /usr/certs/cert.pem > [{ssl_connection,init_certificates,8, > [{file,"ssl_connection.erl"},{line,1155}]}, > {ssl_connection,ssl_init,2,[{file,"ssl_connection.erl"},{line,1110}]}, > {ssl_connection,init,1,[{file,"ssl_connection.erl"},{line,303}]}, > {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}] > > > Does anyone here have an idea as to what the issue might be? It's not a very helpful error :-) > > Thanks, > Lee > > From koesters.damian@REDACTED Tue Jul 23 16:47:52 2013 From: koesters.damian@REDACTED (=?ISO-8859-1?Q?Damian_K=F6sters?=) Date: Tue, 23 Jul 2013 16:47:52 +0200 Subject: [erlang-questions] Prime numbers exercise from Project Euler Message-ID: <51EE9798.2020708@gmail.com> Hi everyone, I am new to Erlang and exercise by solving project-euler problems. The current one is this *The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17.**Find the sum of all the primes below two million.* I tried two approaches which work for small numbers but take way too long for larger inputs like 2 million. Is there anything to improve performance or is Erlang simply not made for problems like this? I would be happy to get some feedback on my code. For example I am not sure if it is a developer's task to keep track of variables like Max in the second example. (I've only included this because it is not allowed to use functions inside guard expressions). %1st approach: %returns a lists of prime numbers smaller than N %this works by eliminating non-prime numbers from the sequence(1,N) in N steps -module(primes). -export([primelist/1]). primelist(N) -> primelist(lists:seq(1,N),N,2). primelist(Filterlist,Max,I) when I primelist([X||X<-Filterlist,(X==I) or (X rem I /= 0)],Max,I+1); primelist([],Max,I) -> []; primelist(Filterlist,Max,I) when I == Max -> Filterlist. %2nd approach %returns the sum of the primes up to N %the function successively builds up a list of primes by testing candidates %and sums it afterwards -module(prime9). -export([sumprimes/1]). sumprimes(N) -> sumprimes(N,[2],[2],2,3). sumprimes(N,[H|T],L,Max,Try) when (Max sumprimes(N, T,L, Max, Try); sumprimes(N,[H|T],L,Max,Try) when (Max sumprimes(N, L,L, Max, Try+1); sumprimes(N,[2],L,Max,Try) when (Try rem 2 /= 0) -> sumprimes(N,[],L,Max,Try); sumprimes(N,[2],L,Max,Try) when (Try rem 2 == 0) -> sumprimes(N,L,L,Max,Try+1); sumprimes(N,[],L,Max,Try) when (Max sumprimes(N,lists:append([Try],L),lists:append([Try],L),Try,Try+1); sumprimes(N,L,[H|T],Max,Try) when (Max >= N) -> lists:sum(T). Thank you very much& best regards Damian -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Tue Jul 23 22:48:10 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Tue, 23 Jul 2013 13:48:10 -0700 Subject: [erlang-questions] Prime numbers exercise from Project Euler In-Reply-To: <51EE9798.2020708@gmail.com> References: <51EE9798.2020708@gmail.com> Message-ID: http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes I did a very cursory look at your code and it seems your approach to generating the primes is not very efficient. Also, I'd recommend adding it as you go to save memory (rather than storing all 2 million primes in memory and adding them at the end). On Tue, Jul 23, 2013 at 7:47 AM, Damian K?sters wrote: > Hi everyone, > > I am new to Erlang and exercise by solving project-euler problems. The > current one is this > > The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17. Find the sum of all > the primes below two million. > > I tried two approaches which work for small numbers but take way too long > for larger inputs like 2 million. Is there anything to improve performance > or is Erlang simply not made for problems like this? > I would be happy to get some feedback on my code. For example I am not sure > if it is a developer's task to keep track of variables like Max in the > second example. (I've only included this because it is not allowed to use > functions inside guard expressions). > > > %1st approach: > %returns a lists of prime numbers smaller than N > %this works by eliminating non-prime numbers from the sequence(1,N) in N > steps > > -module(primes). > -export([primelist/1]). > > primelist(N) -> primelist(lists:seq(1,N),N,2). > > primelist(Filterlist,Max,I) when I -> primelist([X||X<-Filterlist,(X==I) or (X rem I /= 0)],Max,I+1); > primelist([],Max,I) > -> []; > primelist(Filterlist,Max,I) when I == Max > -> Filterlist. > > > %2nd approach > %returns the sum of the primes up to N > %the function successively builds up a list of primes by testing candidates > %and sums it afterwards > > -module(prime9). > -export([sumprimes/1]). > > sumprimes(N) -> sumprimes(N,[2],[2],2,3). > > sumprimes(N,[H|T],L,Max,Try) when (Max -> sumprimes(N, T,L, Max, Try); > sumprimes(N,[H|T],L,Max,Try) when (Max -> sumprimes(N, L,L, Max, Try+1); > sumprimes(N,[2],L,Max,Try) when (Try rem 2 /= 0) > -> sumprimes(N,[],L,Max,Try); > sumprimes(N,[2],L,Max,Try) when (Try rem 2 == 0) > -> sumprimes(N,L,L,Max,Try+1); > sumprimes(N,[],L,Max,Try) when (Max -> sumprimes(N,lists:append([Try],L),lists:append([Try],L),Try,Try+1); > sumprimes(N,L,[H|T],Max,Try) when (Max >= N) > -> lists:sum(T). > > > Thank you very much& best regards > Damian > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From stas@REDACTED Tue Jul 23 23:22:38 2013 From: stas@REDACTED (Stanislav Sedov) Date: Tue, 23 Jul 2013 14:22:38 -0700 Subject: [erlang-questions] Prime numbers exercise from Project Euler In-Reply-To: References: <51EE9798.2020708@gmail.com> Message-ID: <20130723142238.8873b684e8c18bd2511bb5b4@FreeBSD.org> On Tue, 23 Jul 2013 13:48:10 -0700 Jeremy Ong mentioned: > http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes > > I did a very cursory look at your code and it seems your approach to > generating the primes is not very efficient. Also, I'd recommend > adding it as you go to save memory (rather than storing all 2 million > primes in memory and adding them at the end). > Yes, the sieve of Eratosthenes will most likely be the best method here. There are also a lot of probablistic primality tests available (e.g. Fermat test) that can speedup the detection significally: http://en.wikipedia.org/wiki/Primality_test . -- Stanislav Sedov ST4096-RIPE () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments From yann.secq@REDACTED Wed Jul 24 09:33:24 2013 From: yann.secq@REDACTED (Yann SECQ) Date: Wed, 24 Jul 2013 09:33:24 +0200 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: References: <51EE6178.9060205@univ-lille1.fr> <20130723111958.GA9162@dex.pl.erlangsystems.com> Message-ID: <51EF8344.6040202@univ-lille1.fr> Le 23/07/13 14:14, Francesco Cesarini a ?crit : > And if you believe in rumors, the parallela board should come > preinstalled with Erlang as part of the standard image.... Thank you Karol and Francesco for these great news ! Adapteva has sold one more chip ;) Best regards, yann. -- Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq www.ouverture-independance.fr | www.sauvonsluniversite.com -- "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux From Hal.Snyder@REDACTED Wed Jul 24 11:32:10 2013 From: Hal.Snyder@REDACTED (Snyder, Hal) Date: Wed, 24 Jul 2013 04:32:10 -0500 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: Message-ID: Big data, move over for big concurrency. I could see some apps shifting from passive NoSQL stores to function spaces where data elements are light-weight processes that manage their own interconnections. From: Francesco Cesarini > Date: Tue, 23 Jul 2013 07:14:06 -0500 To: Karol Urbanski >, Yann SECQ > Cc: Erlang-Questions List > Subject: Re: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) And if you believe in rumors, the parallela board should come preinstalled with Erlang as part of the standard image.... F Karol Urbanski > wrote: Hi Yann, we're working on releasing the erlang virtual machine for Parallella, and we're pretty close to completing the packaging process. If all goes well, we should have something up on the Erlang Solutions page by the end of this week. :) Best regards, Karol Urbanski On Tue, Jul 23, 2013 at 12:56:56PM +0200, Yann SECQ wrote: Hi all, for those interested in nice new hardware that could fit wisely with erlang, you'll be happy to know that you can pre-order this new Raspberry Pi sized parallel computer: - http://shop.adapteva.com/collections/parallella/products/parallella-16 It seems that so me folks within Erlang Solutions are already hacking this little beast: - http://www.parallella.org/2013/02/04/parallella-and-erlang-an-introduction/ - http://www.parallella.org/2013/05/25/explorations-in-erlang-with-the-parallela-a-prelude/ I tried to find if a specific erlang bundle was available for parallela there: - https://www.erlang-solutions.com/downloads/download-erlang-otp - or, http://www.erlang-embedded.com but have not been successful :( Do you know if some VM and tools (cf. OpenCL in the demo) will be packaged and released by Erlang Solutions in the near future ? Said otherwise, should I buy my parallela right now or wait some months ? ;) Thank you, yann. -- Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq www.ouverture-independance.fr | www.sauvonsluniversite.com -- "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux ________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions ________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -- Erlang Solutions Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: From yann.secq@REDACTED Wed Jul 24 15:01:30 2013 From: yann.secq@REDACTED (Yann SECQ) Date: Wed, 24 Jul 2013 15:01:30 +0200 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: References: Message-ID: <51EFD02A.2030100@univ-lille1.fr> Le 24/07/13 11:32, Snyder, Hal a ?crit : > Big data, move over for big concurrency. I could see some apps shifting > from passive NoSQL stores to function spaces where data elements are > light-weight processes that manage their own interconnections. I was also tinkering this idea by wondering if someone had already thought of an actor-based rdf store and what it could mean in terms of distributed querying and ontology evolution ... Just dreaming :) -- Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq www.ouverture-independance.fr | www.sauvonsluniversite.com -- "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux From antoine.koener@REDACTED Wed Jul 24 17:47:37 2013 From: antoine.koener@REDACTED (Antoine Koener) Date: Wed, 24 Jul 2013 17:47:37 +0200 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: <51EFD02A.2030100@univ-lille1.fr> References: <51EFD02A.2030100@univ-lille1.fr> Message-ID: <04E813B0-E810-490C-AE6D-B7F4B14AD50A@gmail.com> -- Take this message with special care because I was driving :-) On 24 juil. 2013, at 15:01, Yann SECQ wrote: > Le 24/07/13 11:32, Snyder, Hal a ?crit : >> Big data, move over for big concurrency. I could see some apps shifting >> from passive NoSQL stores to function spaces where data elements are >> light-weight processes that manage their own interconnections. > > I was also tinkering this idea by wondering if someone had already > thought of an actor-based rdf store and what it could mean in terms > of distributed querying and ontology evolution ... > > Just dreaming :) > No need to dream, see: Fis-gtm aka mumps. Replication aware column store is the answer. Search for Erlang egtm binding... > -- > Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq > www.ouverture-independance.fr | www.sauvonsluniversite.com > -- > "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From peerst@REDACTED Wed Jul 24 17:56:23 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Wed, 24 Jul 2013 17:56:23 +0200 Subject: [erlang-questions] Prime numbers exercise from Project Euler References: <51EE9798.2020708@gmail.com> Message-ID: On 2013-07-23 14:47:52 +0000, Damian K?sters said: > primelist(Filterlist,Max,I) when I ??? -> primelist([X||X<-Filterlist,(X==I) or (X rem I /= 0)],Max,I+1); > primelist([],Max,I) > ??? -> []; This is no Sieve of Erasthotenes but rather trial division. That is one reason why this is much slower than it needs to be. See this answer for an explanation why it is no Sieve: http://stackoverflow.com/a/389740/364327 (make sure to read the linked paper) Also one speedup in your code would already be to only check until a Max of sqrt(N). Generally Sieve of Erasthotenes is not very fast as is to be expected of a algorithm thats over 2000 years old ;.) If you really looking for a fast sieve try http://en.wikipedia.org/wiki/Sieve_of_Atkin But I think Erasthotenes Sieve would be sufficiently fast if implemented correctly. From n.oxyde@REDACTED Wed Jul 24 20:50:46 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Wed, 24 Jul 2013 20:50:46 +0200 Subject: [erlang-questions] Prime numbers exercise from Project Euler In-Reply-To: References: <51EE9798.2020708@gmail.com> Message-ID: <06485140-E470-484F-861C-146C434AC532@gmail.com> Hello Peer, I was bored so I implemented it with a mutable HiPE bitarray. https://gist.github.com/nox/6073337 Regards, -- Anthony Ramine Le 24 juil. 2013 ? 17:56, Peer Stritzinger a ?crit : > If you really looking for a fast sieve try http://en.wikipedia.org/wiki/Sieve_of_Atkin From devangana@REDACTED Wed Jul 24 20:58:42 2013 From: devangana@REDACTED (Devangana Tarafdar) Date: Wed, 24 Jul 2013 13:58:42 -0500 Subject: [erlang-questions] Memory occupied by an individual row in a mnesia table Message-ID: Hello, I am attempting to find the size occupied in memory by each row of a mnesia table. The table is of types ram_copies. I am doing the following: [A] = mnesia:dirty_read(Table_name, Key) Size_A = erts_debug:flat_size(A) ( I read about the erts_debug function from http://www.erlang.org/doc/efficiency_guide/processes.html#id68532 ) I was wondering if this is the correct approach or is there another method that I should be applying here ? Thank you for any suggestions/corrections, Devangana -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Jul 24 22:12:50 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Wed, 24 Jul 2013 23:12:50 +0300 Subject: [erlang-questions] Prime numbers exercise from Project Euler In-Reply-To: References: <51EE9798.2020708@gmail.com> <06485140-E470-484F-861C-146C434AC532@gmail.com> Message-ID: Forwarding to list. On Wed, Jul 24, 2013 at 10:29 PM, Motiejus Jak?tys wrote: > On Wed, Jul 24, 2013 at 9:50 PM, Anthony Ramine wrote: >> Hello Peer, >> >> I was bored so I implemented it with a mutable HiPE bitarray. >> >> https://gist.github.com/nox/6073337 >> > > Any way to familiarize ourselves with HiPE bitarrays better than > reading erts/emulator/hipe/hipe_bif0.c? > > Motiejus -- Motiejus Jak?tys From carlsson.richard@REDACTED Wed Jul 24 22:26:19 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 24 Jul 2013 22:26:19 +0200 Subject: [erlang-questions] Prime numbers exercise from Project Euler In-Reply-To: References: <51EE9798.2020708@gmail.com> <06485140-E470-484F-861C-146C434AC532@gmail.com> Message-ID: <51F0386B.1090208@gmail.com> The traditional way is to seek out a couple of the old HiPE team members and buy them beer until they talk. /Richard On 2013-07-24 22:12 , Motiejus Jak?tys wrote: > Forwarding to list. > > On Wed, Jul 24, 2013 at 10:29 PM, Motiejus Jak?tys > wrote: >> On Wed, Jul 24, 2013 at 9:50 PM, Anthony Ramine wrote: >>> Hello Peer, >>> >>> I was bored so I implemented it with a mutable HiPE bitarray. >>> >>> https://gist.github.com/nox/6073337 >>> >> >> Any way to familiarize ourselves with HiPE bitarrays better than >> reading erts/emulator/hipe/hipe_bif0.c? >> >> Motiejus From bezirg@REDACTED Wed Jul 24 22:24:45 2013 From: bezirg@REDACTED (Nikolaos Bezirgiannis) Date: Wed, 24 Jul 2013 22:24:45 +0200 Subject: [erlang-questions] Arrow keys in erl started by cygwin mintty terminal References: <51EB0867.9050202@aleynikov.org> Message-ID: <86bo5ryacy.fsf@gmail.com> Serge Aleynikov writes: > Hi, > > Does anyone have an idea how to configure a cygwin terminal running erl > on Windows so that it would recognize the up/down arrow keys? While the > up/down keys work fine in werl and in erl that was started from Windows > command prompt, I have a trouble with them working in erl started by > cygwin's mintty - the cursor moves up/down screen beyond the current > line of the shell, and my attempts to address the issue by editing > ~/.inputrc haven't succeeded. > > Serge Suggestion: Install the utility rlwrap from the Cygwin repository, and after that run from the console: rlwrap erl Regards, Nikolaos From serge@REDACTED Wed Jul 24 23:48:33 2013 From: serge@REDACTED (Serge Aleynikov) Date: Wed, 24 Jul 2013 17:48:33 -0400 Subject: [erlang-questions] Arrow keys in erl started by cygwin mintty terminal In-Reply-To: <86bo5ryacy.fsf@gmail.com> References: <51EB0867.9050202@aleynikov.org> <86bo5ryacy.fsf@gmail.com> Message-ID: <51F04BB1.8090605@aleynikov.org> Thanks. Looks like it solves the problem with up/down keys but tab command completion still doesn't work. Looks like there's another alternative to rlwrap called conin that addresses the same thing (http://code.google.com/p/mintty/source/browse/utils/conin.c?r=662), but it also doesn't heal the tab issue. Maybe I'll play with conin later to add that support, as it seems quite simple... On 7/24/2013 4:24 PM, Nikolaos Bezirgiannis wrote: > > Serge Aleynikov writes: > >> Hi, >> >> Does anyone have an idea how to configure a cygwin terminal running erl >> on Windows so that it would recognize the up/down arrow keys? While the >> up/down keys work fine in werl and in erl that was started from Windows >> command prompt, I have a trouble with them working in erl started by >> cygwin's mintty - the cursor moves up/down screen beyond the current >> line of the shell, and my attempts to address the issue by editing >> ~/.inputrc haven't succeeded. >> >> Serge > > Suggestion: > > Install the utility rlwrap from the Cygwin repository, > and after that run from the console: rlwrap erl > > Regards, > Nikolaos > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From rickp@REDACTED Thu Jul 25 11:52:27 2013 From: rickp@REDACTED (Rick Payne) Date: Thu, 25 Jul 2013 10:52:27 +0100 Subject: [erlang-questions] Noddy question on hashing Message-ID: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> Hi, I've been using erlang for some simple projects, and really enjoying it. However, when thinking about some of the things I'd like to do - I'm struggling a bit with hashing and how to handle that in a concurrent way in erlang. This is probably 101 stuff, sorry... Imagine we have a set of data streams giving us objects and attributes. Attributes can be a few kB but typically are less than 1kB. The objects are unique but some set of them share the attributes. Typically in C, I'm used to hashing the attributes and pointing at them from the objects. We hash to save memory and refcount so we know when we can delete from the hash. The good news is that the attributes are never updated in place, apart from the refcount. However, I'm struggling to get my head around how I'd do this in erlang efficiently. Any suggestions? Cheers, Rick From solomon.wzs@REDACTED Thu Jul 25 12:51:56 2013 From: solomon.wzs@REDACTED (Solomon) Date: Thu, 25 Jul 2013 18:51:56 +0800 Subject: [erlang-questions] Can not delete a dead node Message-ID: My cluster has 2 nodes:a@REDACTED and b@REDACTED, I call mnesia:del_table_copy(schema, b@REDACTED) to delete b@REDACTED, which was at a dead node, it was fail and the error message was {aborted,{no_exists,mytable}}, but mytable was exist at a@REDACTED, both a@REDACTED and b@REDACTED had a disc copy of mytable. what is the problem of mytable and how should I delete b@REDACTED? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hm@REDACTED Thu Jul 25 14:41:28 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Thu, 25 Jul 2013 14:41:28 +0200 Subject: [erlang-questions] Can not delete a dead node Message-ID: In order to remove node b@REDACTED from the Mnesia schema you must: 1 stop Mnesia on node b@REDACTED 2 run mnesia:del_table_copy(schema, b@REDACTED) on node a@REDACTED 3 never try to start Mnesia on node b@REDACTED again If that does not work, you need to investigate how Mnesia looks at these tables. Use mnesia:info(), mnesia:schema(schema) and mnesia:schema(mytable) to get more info about the Mnesia configuration. /H?kan On Thu, Jul 25, 2013 at 12:51 PM, Solomon wrote: > My cluster has 2 nodes:a@REDACTED and b@REDACTED, I call mnesia:del_table_copy(schema, > b@REDACTED) to delete b@REDACTED, which was at a dead node, it was fail and the error > message was {aborted,{no_exists,mytable}}, but mytable was exist at a@REDACTED, > both a@REDACTED and b@REDACTED had a disc copy of mytable. what is the problem of > mytable and how should I delete b@REDACTED? From davidnwelton@REDACTED Thu Jul 25 16:46:19 2013 From: davidnwelton@REDACTED (David Welton) Date: Thu, 25 Jul 2013 16:46:19 +0200 Subject: [erlang-questions] Testing a large, heterogeneous system Message-ID: Hi, I'm looking for ideas on testing a system we're developing. Currently we have nothing, so anything is better than that. We're looking to get the most 'bang for our buck', so to speak, so tests that cover, even if roughly, more code are going to be better in some ways that something that's too specific to a small part of the system. Erlang, in our project, sits in the middle: it offers a web interface that uses lots of Javascript via Chicago Boss, manages some web sockets, talks to a database (Postgres), interacts with several C nodes, and also manages various external programs via erlexec and open_port. To add some complexity, specialized hardware is involved and dealt with via the C programs/nodes. I read the chapter on Common Test in Learn You Some Erlang, and that looks like a pretty good platform. What I'm struggling with is what sort of strategy to use to test the whole thing. Things I'm dubious about: * How to handle the external bits? It'd be nice if the tests could be run on a developer's machine without requiring the hardware. But I could also see something rigged up to interact with the hardware being valuable as well. Forced to choose, I'd go with the former, I think. * The interaction with the hardware stuff is kind of complex and involves a lot of state, and messages going back and forth and input from the user via the web, storing stuff in the database and so on. The web portions of things... maybe could be tested in isolation to a certain point, but the sequence of messages going back and forth is tricky and probably the place where tests will give us the most benefits. I'm used to tests that try and test fairly small chunks of a system, but I'm not sure how feasible that is in this case - the setup and teardown might involve a lot of stuff just for a small test... Sorry if this is all a bit muddled - any advice is appreciated! Thank you, -- David N. Welton http://www.welton.it/davidw/ http://www.dedasys.com/ From g@REDACTED Thu Jul 25 17:57:51 2013 From: g@REDACTED (Garrett Smith) Date: Thu, 25 Jul 2013 09:57:51 -0600 Subject: [erlang-questions] Noddy question on hashing In-Reply-To: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> References: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> Message-ID: Hi Rick, On Thu, Jul 25, 2013 at 3:52 AM, Rick Payne wrote: > Hi, > > I've been using erlang for some simple projects, and really enjoying it. > However, when thinking about some of the things I'd like to do - I'm > struggling a bit with hashing and how to handle that in a concurrent way in > erlang. This is probably 101 stuff, sorry... > > Imagine we have a set of data streams giving us objects and attributes. > Attributes can be a few kB but typically are less than 1kB. The objects are > unique but some set of them share the attributes. Typically in C, I'm used > to hashing the attributes and pointing at them from the objects. We hash to > save memory and refcount so we know when we can delete from the hash. The > good news is that the attributes are never updated in place, apart from the > refcount. > > However, I'm struggling to get my head around how I'd do this in erlang > efficiently. Any suggestions? What specifically would you be doing in C? With more specific operations in mind, it'll be easier to point you toward something in Erlang I think. Without knowing much here, I'd list two facilities in Erlang. The first is somewhat blunt and might not be appropriate but it's an easy way to hash something largish like a 1K attribute: http://www.erlang.org/doc/man/erlang.html#md5-1 And this is a way to maintain your reference counts associated with hashes: http://erlang.org/doc/man/ets.html Garrett From siraaj@REDACTED Thu Jul 25 18:38:29 2013 From: siraaj@REDACTED (Siraaj Khandkar) Date: Thu, 25 Jul 2013 12:38:29 -0400 Subject: [erlang-questions] Noddy question on hashing In-Reply-To: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> References: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> Message-ID: <51F15485.3080102@khandkar.net> On 07/25/2013 05:52 AM, Rick Payne wrote:> Hi, > > I've been using erlang for some simple projects, and really enjoying > it. However, when thinking about some of the things I'd like to do - > I'm struggling a bit with hashing and how to handle that in a > concurrent way in erlang. This is probably 101 stuff, sorry... > > Imagine we have a set of data streams giving us objects and > attributes. Attributes can be a few kB but typically are less than > 1kB. The objects are unique but some set of them share the attributes. > Typically in C, I'm used to hashing the attributes and pointing at > them from the objects. We hash to save memory and refcount so we know > when we can delete from the hash. The good news is that the attributes > are never updated in place, apart from the refcount. > > However, I'm struggling to get my head around how I'd do this in > erlang efficiently. Any suggestions? Erlang joke: "To have fault-tolerant objects, you must have at least two copies of each object." :) Generally one does not approach things this was in Erlang. Space efficiency was not its motivation, fault-tolerance was, and thus redundancy is the law of the land. That said, if you have some large blob that you want several objects or even processes to point to, instead of copying, you could use ETS, which is basically a shared hash table. http://www.erlang.org/doc/man/ets.html http://learnyousomeerlang.com/ets If you're interested in correctness and want those attributes to be enums - you could use atoms as values and tell Dialyzer what each is expected to be, like so: -type attribute_a() :: a | b | c . -type attribute_b() :: d | e | f . -record(object, { a :: attribute_a() , b :: attribute_b() }) -type object() :: #object{}. This also works if those attributes are actually keys to the values stored in ETS. From bob@REDACTED Thu Jul 25 18:50:47 2013 From: bob@REDACTED (Bob Ippolito) Date: Thu, 25 Jul 2013 09:50:47 -0700 Subject: [erlang-questions] Noddy question on hashing In-Reply-To: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> References: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> Message-ID: Larger binaries (refc binaries) in the Erlang VM are sent by reference and not copied when sent from one process to another (the threshold is 64 bytes, and they use a reference counting strategy). If you have a process that can de-dupe these binaries then you don't use hashes at all, you just store the binaries and know that they're shared. I'm not sure if an ets table (instead of a process) will suffice to do the de-duping, but you could do some experiments to figure that out. If the attributes also have names (and there aren't too many possible names), then atoms are generally the right type for that. See also: http://www.erlang.org/doc/efficiency_guide/binaryhandling.html On Thu, Jul 25, 2013 at 2:52 AM, Rick Payne wrote: > Hi, > > I've been using erlang for some simple projects, and really enjoying it. > However, when thinking about some of the things I'd like to do - I'm > struggling a bit with hashing and how to handle that in a concurrent way in > erlang. This is probably 101 stuff, sorry... > > Imagine we have a set of data streams giving us objects and attributes. > Attributes can be a few kB but typically are less than 1kB. The objects are > unique but some set of them share the attributes. Typically in C, I'm used > to hashing the attributes and pointing at them from the objects. We hash to > save memory and refcount so we know when we can delete from the hash. The > good news is that the attributes are never updated in place, apart from the > refcount. > > However, I'm struggling to get my head around how I'd do this in erlang > efficiently. Any suggestions? > > Cheers, > Rick > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickp@REDACTED Thu Jul 25 18:58:16 2013 From: rickp@REDACTED (Rick Payne) Date: Thu, 25 Jul 2013 17:58:16 +0100 Subject: [erlang-questions] Noddy question on hashing In-Reply-To: <51F15485.3080102@khandkar.net> References: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> <51F15485.3080102@khandkar.net> Message-ID: <75368337011F863E8CCF4B93@scampi.rossfell.co.uk> --On 25 July 2013 12:38:29 -0400 Siraaj Khandkar wrote: > Erlang joke: "To have fault-tolerant objects, you must have at least > two > copies of each object." :) :) > Generally one does not approach things this was in Erlang. Space > efficiency was not its motivation, fault-tolerance was, and thus > redundancy is the law of the land. Right - for sure part of my problem is not being able to see things with the right mindset yet. > That said, if you have some large blob that you want several objects > or > even processes to point to, instead of copying, you could use ETS, > which > is basically a shared hash table. Ok, I think I need to spend some more time thinking about the problem. Maybe its a case of mapping attributes to a process id, and its the process which holds the state and processes it. Hmm - you've got me thinking... > If you're interested in correctness and want those attributes to be > enums - you could use atoms as values and tell Dialyzer what each is > expected to be, like so: Yup, already breaking the attributes up into atoms and lists of values in a somewhat sane way. Thanks for the pointers. Cheers, Rick From desired.mta@REDACTED Thu Jul 25 22:06:12 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Thu, 25 Jul 2013 23:06:12 +0300 Subject: [erlang-questions] Noddy question on hashing In-Reply-To: <51F15485.3080102@khandkar.net> References: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> <51F15485.3080102@khandkar.net> Message-ID: On Thu, Jul 25, 2013 at 7:38 PM, Siraaj Khandkar wrote: > On 07/25/2013 05:52 AM, Rick Payne wrote:> Hi, > > If you're interested in correctness and want those attributes to be > enums - you could use atoms as values and tell Dialyzer what each is > expected to be, like so: > > -type attribute_a() :: a > | b > | c > . > > -type attribute_b() :: d > | e > | f > . > > -record(object, > { a :: attribute_a() > , b :: attribute_b() > }) > > -type object() :: #object{}. > Note that once you have 13 or 14 of them they will just be reduced to 'atom()': http://erlang.org/pipermail/erlang-questions/2012-August/068851.html Motiejus From siraaj@REDACTED Fri Jul 26 02:33:36 2013 From: siraaj@REDACTED (Siraaj Khandkar) Date: Thu, 25 Jul 2013 20:33:36 -0400 Subject: [erlang-questions] Noddy question on hashing In-Reply-To: References: <139C623DFD56F659B24FFE1B@scampi.rossfell.co.uk> <51F15485.3080102@khandkar.net> Message-ID: <51F1C3E0.6030600@khandkar.net> On 07/25/2013 04:06 PM, Motiejus Jak?tys wrote: > On Thu, Jul 25, 2013 at 7:38 PM, Siraaj Khandkar wrote: >> On 07/25/2013 05:52 AM, Rick Payne wrote:> Hi, >> >> If you're interested in correctness and want those attributes to be >> enums - you could use atoms as values and tell Dialyzer what each is >> expected to be, like so: >> >> -type attribute_a() :: a >> | b >> | c >> . >> >> -type attribute_b() :: d >> | e >> | f >> . >> >> -record(object, >> { a :: attribute_a() >> , b :: attribute_b() >> }) >> >> -type object() :: #object{}. >> > > Note that once you have 13 or 14 of them they will just be reduced to 'atom()': > http://erlang.org/pipermail/erlang-questions/2012-August/068851.html > I did not know about this limit. Very, very important to know. Thank you for bringing it up! From solomon.wzs@REDACTED Fri Jul 26 03:33:45 2013 From: solomon.wzs@REDACTED (Solomon) Date: Fri, 26 Jul 2013 09:33:45 +0800 Subject: [erlang-questions] Can not delete a dead node In-Reply-To: References: Message-ID: I found the problem, mytable run at read_only mode. Because I can not restart b@REDACTED, I can not call mnesia:change_table_access_mode(mytable, read_write), the error message was "All replicas on diskfull nodes are not active yet". So How should I solve this problem? 2013/7/25 H?kan Mattsson > In order to remove node b@REDACTED from the Mnesia schema you must: > > 1 stop Mnesia on node b@REDACTED > 2 run mnesia:del_table_copy(schema, b@REDACTED) on node a@REDACTED > 3 never try to start Mnesia on node b@REDACTED again > > If that does not work, you need to investigate how Mnesia looks at > these tables. Use mnesia:info(), mnesia:schema(schema) and > mnesia:schema(mytable) to get more info about the Mnesia > configuration. > > /H?kan > > On Thu, Jul 25, 2013 at 12:51 PM, Solomon wrote: > > My cluster has 2 nodes:a@REDACTED and b@REDACTED, I call > mnesia:del_table_copy(schema, > > b@REDACTED) to delete b@REDACTED, which was at a dead node, it was fail and the > error > > message was {aborted,{no_exists,mytable}}, but mytable was exist at a@REDACTED > , > > both a@REDACTED and b@REDACTED had a disc copy of mytable. what is the problem of > > mytable and how should I delete b@REDACTED? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Fri Jul 26 07:04:37 2013 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Fri, 26 Jul 2013 08:04:37 +0300 Subject: [erlang-questions] binary typed schema-less protocol Message-ID: <20130726050437.GA2817@precise.local> Dear erlangers, About a year(?) ago I was pointed to a binary protocol which can encode strings, numbers, lists (maybe some more data types like map), and cannot rember its name nor find a reference. I remember is it encodes a string by prefixing it with its length. Every data type has its number. The protocol is designed to be as simple as possible to implement with as small area as possible for implementation mistakes. I also remember protocol's web page was very clutter-free (black & white, with only basic HTML tags in use). It was pointed by some Erlang person in either IRC or erlang-questions as a good way to ad-hoc communicate with other, written-not-by-you components. I believe I found a use case for it. Does it ring a bell to anyone? Thanks! Motiejus From bob@REDACTED Fri Jul 26 07:25:17 2013 From: bob@REDACTED (Bob Ippolito) Date: Thu, 25 Jul 2013 22:25:17 -0700 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <20130726050437.GA2817@precise.local> References: <20130726050437.GA2817@precise.local> Message-ID: BERT? msgpack? On Thursday, July 25, 2013, Motiejus Jak?tys wrote: > Dear erlangers, > > About a year(?) ago I was pointed to a binary protocol which can encode > strings, numbers, lists (maybe some more data types like map), and > cannot rember its name nor find a reference. > > I remember is it encodes a string by prefixing it with its length. Every > data type has its number. The protocol is designed to be as simple as > possible to implement with as small area as possible for implementation > mistakes. > > I also remember protocol's web page was very clutter-free (black & > white, with only basic HTML tags in use). > > It was pointed by some Erlang person in either IRC or erlang-questions > as a good way to ad-hoc communicate with other, written-not-by-you > components. I believe I found a use case for it. > > Does it ring a bell to anyone? > > Thanks! > Motiejus > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Jul 26 07:26:36 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 26 Jul 2013 17:26:36 +1200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <20130726050437.GA2817@precise.local> References: <20130726050437.GA2817@precise.local> Message-ID: <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> On 26/07/2013, at 5:04 PM, Motiejus Jak?tys wrote: > About a year(?) ago I was pointed to a binary protocol which can encode > strings, numbers, lists (maybe some more data types like map), and > cannot rember its name nor find a reference. Could this have been Joe Armstrong's UBF? Ad-hoc communication with other not-written-by-you components sounds a bit more like JSON or BSON, or _possibly_ BERT. UBF's strength is being able to slot a "contract checker" in between two components From moxford@REDACTED Fri Jul 26 09:49:02 2013 From: moxford@REDACTED (Mike Oxford) Date: Fri, 26 Jul 2013 00:49:02 -0700 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> Message-ID: Google protobuf On Thu, Jul 25, 2013 at 10:26 PM, Richard A. O'Keefe wrote: > > On 26/07/2013, at 5:04 PM, Motiejus Jak?tys wrote: > > About a year(?) ago I was pointed to a binary protocol which can encode > > strings, numbers, lists (maybe some more data types like map), and > > cannot rember its name nor find a reference. > > Could this have been Joe Armstrong's UBF? > > Ad-hoc communication with other not-written-by-you components > sounds a bit more like JSON or BSON, or _possibly_ BERT. > > UBF's strength is being able to slot a "contract checker" > in between two components > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Fri Jul 26 11:20:56 2013 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 26 Jul 2013 11:20:56 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> Message-ID: <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> On 26 Jul 2013, at 07:26, "Richard A. O'Keefe" wrote: > Ad-hoc communication with other not-written-by-you components > sounds a bit more like JSON or BSON, or _possibly_ BERT. > > UBF's strength is being able to slot a "contract checker" > in between two components At Feuerlabs, we use (among other things) BERT [1] and Yang [2] (RFC 6020) to accomplish RPCs with contract checking. In addition, our BERT implementation has access control lists and is pretty flexible about transport and authentication (with the option of an authentication plugin), and our Yang compiler (developed in cooperation with Tail-f) can generate Erlang code from a spec and has a facility to validate a JSON-style RPC against the 'contract'. Documentation is a bit sparse (hrm?) but our support staff can be very helpful. [1] https://github.com/Feuerlabs/bert [2] https://github.com/Feuerlabs/yang BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From desired.mta@REDACTED Fri Jul 26 12:40:47 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Fri, 26 Jul 2013 13:40:47 +0300 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> Message-ID: On Fri, Jul 26, 2013 at 8:25 AM, Bob Ippolito wrote: > BERT? msgpack? Thanks for the replies, I learnt something new today. The protocol I was looking for was not mentioned here. However, msgpack is closest to what I am looking for. I will know if it is once I check better how it is written. I should have said that I am going to use this from C (client-side, my job) and Something (server-side, not yet known whose job). An important goal of this is to be really portable and multiplatform. I would like to introduce schema, but want to keep the dependencies as simple and as small as possible (ideally, none, thus asked for that simple format). I and am not aware of any IDLs that are small and clean (say a shared library in clean C or C++98). -- Motiejus Jak?tys From desired.mta@REDACTED Fri Jul 26 12:55:31 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Fri, 26 Jul 2013 13:55:31 +0300 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> Message-ID: On Fri, Jul 26, 2013 at 1:40 PM, Motiejus Jak?tys wrote: > On Fri, Jul 26, 2013 at 8:25 AM, Bob Ippolito wrote: >> BERT? msgpack? > > The protocol I was looking for was not mentioned here. Found it. http://tnetstrings.org/ -- Motiejus Jak?tys From yash_ganthe@REDACTED Fri Jul 26 16:22:27 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Fri, 26 Jul 2013 14:22:27 +0000 Subject: [erlang-questions] How to rebar-package an app but not start it? Message-ID: Hi, In reltool.config I have: . . . {rel, "myrel", "0.0.1", [ kernel, stdlib, sasl, myapp ,myapp2 When I start the node, myapp starts on it followed by myapp2. How do I prevent myapp2 from starting? I would like to start it later. Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Fri Jul 26 19:01:51 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Fri, 26 Jul 2013 18:01:51 +0100 Subject: [erlang-questions] How to rebar-package an app but not start it? In-Reply-To: References: Message-ID: Hi Yash, you then want to make myapp2 into a library application. That is you don't provide the startup function as part of your myapp2.app definition. Hope this helps, Robby On Jul 26, 2013 3:22 PM, "Yash Ganthe" wrote: > Hi,**** > > ** ** > > In reltool.config I have:**** > > . . . {rel, "myrel", "0.0.1",**** > > [**** > > kernel,**** > > stdlib,**** > > sasl,**** > > myapp**** > > ,myapp2**** > > **** > > When I start the node, myapp starts on it followed by myapp2.**** > > ** ** > > How do I prevent myapp2 from starting? I would like to start it later.**** > > ** ** > > Thanks,**** > > Yash**** > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Ltd. It is intended > only for the use of the individual or entity to which it is addressed. If > you are not the intended recipient, you are not authorized to read, retain, > copy, print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Persistent Systems Ltd. does not accept any liability for > virus infected mails. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Sat Jul 27 00:15:24 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 26 Jul 2013 18:15:24 -0400 Subject: [erlang-questions] How to rebar-package an app but not start it? In-Reply-To: References: Message-ID: Defining the app as {Name, load} should do it if I recall correctly. It will be loaded in the boot sequence, but not started. On Jul 26, 2013 10:22 AM, "Yash Ganthe" wrote: > Hi,**** > > ** ** > > In reltool.config I have:**** > > . . . {rel, "myrel", "0.0.1",**** > > [**** > > kernel,**** > > stdlib,**** > > sasl,**** > > myapp**** > > ,myapp2**** > > **** > > When I start the node, myapp starts on it followed by myapp2.**** > > ** ** > > How do I prevent myapp2 from starting? I would like to start it later.**** > > ** ** > > Thanks,**** > > Yash**** > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Ltd. It is intended > only for the use of the individual or entity to which it is addressed. If > you are not the intended recipient, you are not authorized to read, retain, > copy, print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Persistent Systems Ltd. does not accept any liability for > virus infected mails. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562@REDACTED Fri Jul 26 20:38:29 2013 From: 0x6e6562@REDACTED (Ben Hood) Date: Fri, 26 Jul 2013 19:38:29 +0100 Subject: [erlang-questions] ECC public key example Message-ID: Hi, I'd like to verify signatures signed with ec keys generated by openssl in Erlang. Using R16B01 I've been able to convince the public_key module to sign some data with an openssl generated private key thusly: {ok, Pem} = file:read_file("ec_private_key.pem"), [ECParameters,ECPrivateKey] = public_key:pem_decode(Pem), PrivateKey = public_key:pem_entry_decode(ECPrivateKey), Curve = public_key:pem_entry_decode({'EcpkParameters', Params, Enc}), Sig = public_key:sign(Message, sha512, PrivateKey). But when it comes to processing the public key, it is not entirely clear how to decode it. This is where I got to: {ok, Pem} = file:read_file("ec_public_key.pem"), [{_,Der,_}] = public_key:pem_decode(Pem), {_, _, {0, KeyDer}} = public_key:der_decode('SubjectPublicKeyInfo', Der), [179,52,127,201] = public_key:der_decode('ECPoint', KeyDer) %% This is where the story ends I've looked through the ASN.1 types available and there doesn't seem to be anything that resembles an ECPublicKey. Could anybody point me to working example? Cheers, Ben From erlang@REDACTED Sat Jul 27 12:05:37 2013 From: erlang@REDACTED (=?UTF-8?Q?Micha=C5=82_Ptaszek?=) Date: Sat, 27 Jul 2013 12:05:37 +0200 Subject: [erlang-questions] Monitor option in spawn_opt/5 Message-ID: Hey, What is the reason for not allowing to pass 'monitor' option to processes spawned by erlang:spawn_opt/5? The code inside of erlang.erl states: case lists:member(monitor, O) of false -> ok; true -> erlang:error(badarg, [N, M, F, A, O]) end, However the documentation does not clarify anything. Setting up a monitor on the returned pid right after spawn works fine though. Thanks, Michal -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.simonsson@REDACTED Sat Jul 27 12:15:03 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Sat, 27 Jul 2013 12:15:03 +0200 Subject: [erlang-questions] Testing a large, heterogeneous system In-Reply-To: References: Message-ID: Hi, If you want to limit setup/teardown of the external parts (like your custom hardware), meck [1] is a great tool for mocking all or a subset of external dependencies. Without knowing the internals of your system it's hard to give specific advice, but if you have a bunch of state-full stuff proper [2] has some very powerful tools for testing states and reasoning about what should be valid before and after state transitions. Checkout proper_statem [3] and this [4] example of a very useful test implementing the proper_statem callbacks. Cheers, Gustav Simonsson [1] https://github.com/eproxus/meck [2] http://proper.softlab.ntua.gr/ [3] http://proper.softlab.ntua.gr/doc/proper_statem.html [4] https://github.com/wooga/locker/blob/master/src/locker_proper.erl On Thu, Jul 25, 2013 at 4:46 PM, David Welton wrote: > Hi, > > I'm looking for ideas on testing a system we're developing. > > Currently we have nothing, so anything is better than that. We're > looking to get the most 'bang for our buck', so to speak, so tests > that cover, even if roughly, more code are going to be better in some > ways that something that's too specific to a small part of the system. > > Erlang, in our project, sits in the middle: it offers a web interface > that uses lots of Javascript via Chicago Boss, manages some web > sockets, talks to a database (Postgres), interacts with several C > nodes, and also manages various external programs via erlexec and > open_port. To add some complexity, specialized hardware is involved > and dealt with via the C programs/nodes. > > I read the chapter on Common Test in Learn You Some Erlang, and that > looks like a pretty good platform. What I'm struggling with is what > sort of strategy to use to test the whole thing. > > Things I'm dubious about: > > * How to handle the external bits? It'd be nice if the tests could be > run on a developer's machine without requiring the hardware. But I > could also see something rigged up to interact with the hardware being > valuable as well. Forced to choose, I'd go with the former, I think. > > * The interaction with the hardware stuff is kind of complex and > involves a lot of state, and messages going back and forth and input > from the user via the web, storing stuff in the database and so on. > The web portions of things... maybe could be tested in isolation to a > certain point, but the sequence of messages going back and forth is > tricky and probably the place where tests will give us the most > benefits. I'm used to tests that try and test fairly small chunks of > a system, but I'm not sure how feasible that is in this case - the > setup and teardown might involve a lot of stuff just for a small > test... > > Sorry if this is all a bit muddled - any advice is appreciated! > > Thank you, > -- > David N. Welton > > http://www.welton.it/davidw/ > > http://www.dedasys.com/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Sat Jul 27 13:03:44 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Sat, 27 Jul 2013 14:03:44 +0300 Subject: [erlang-questions] Monitor option in spawn_opt/5 In-Reply-To: References: Message-ID: On Sat, Jul 27, 2013 at 1:05 PM, Micha? Ptaszek wrote: > Hey, > > What is the reason for not allowing to pass 'monitor' option to processes > spawned by erlang:spawn_opt/5? > > The code inside of erlang.erl states: > > case lists:member(monitor, O) of > false -> ok; > true -> erlang:error(badarg, [N, M, F, A, O]) > end, > > However the documentation does not clarify anything. > Setting up a monitor on the returned pid right after spawn works fine > though. I guess the reason is that there is no way in Erlang to atomically spawn and monitor the pid on another node. Though it's a bit strange that [link] works on remote nodes, but [monitor] doesn't. Anyone care to explain why? -- Motiejus Jak?tys From gustav.simonsson@REDACTED Sat Jul 27 16:10:32 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Sat, 27 Jul 2013 15:10:32 +0100 Subject: [erlang-questions] ECC public key example In-Reply-To: References: Message-ID: Hi Ben, I've replicated your console commands (see end of mail) and continued with steps which end with a successful verification of the signature with the EC Public Key. Some duplicated commands is removed from the log but it should still contain all the necessary steps. Some things I noticed: * There is one SubjectPublicKeyInfo which contains another SubjectPublicKeyInfo. This doesn't seem correct and I'm not really sure where the error causing this is. * The EC Public Key type in public_key is defined as: ec_public_key() = {#'ECPoint'{}, #'EcpkParameters'{} | {namedCurve, oid()}} But the EC Public Key Parameters are actually returned as a 'OTPEcpkParameters' PEM entry (record) whose record definition is missing. I suspect this could be a bug in public_key. * I didn't find a way to get a #'ECPoint' record back from decoding, so I manually created it by matching out the ECPoint octets from the SPKI record. * As the EC parameters was in the private key file, they had to be manually combined with the ECPoint to form the EC Public Key. Maybe it's possible to tell openssl to include the EC parameters in the public key file with some option? Test keys where generated with: openssl ecparam -out ec_key_priv.pem -name prime192v1 -genkey openssl ec -pubout -in ec_key_priv.pem -out ec_key_pub.pem Cheers, Gustav Simonsson Erlang console log: ================== Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> rr(public_key). ['AAControls','ACClearAttrs','AccessDescription', 'Algorithm','AlgorithmIdentifier', 'AlgorithmIdentifierPKCS-10','AlgorithmIdentifierPKCS-8', 'AlgorithmIdentifierPKCS5v2-0','AlgorithmIdentifierPKSC-7', 'AlgorithmNull','AnotherName','AttCertValidityPeriod', 'Attribute','AttributeCertificate', 'AttributeCertificateInfo','AttributePKCS-10', 'AttributePKCS-7','AttributeTypeAndValue', 'Attributes_SETOF', 'Attributes_SETOF_valuesWithContext_SETOF', 'AuthorityKeyIdentifier','BasicConstraints', 'BuiltInDomainDefinedAttribute','BuiltInStandardAttributes', 'Certificate','CertificateList','CertificationRequest', 'CertificationRequestInfo', 'CertificationRequestInfo_attributes_SETOF'|...] 2> {ok, Bin} = file:read_file("ec_key_priv.pem"). {ok,<<"-----BEGIN EC PARAMETERS-----\nBggqhkjOPQMBAQ==\n-----END EC PARAMETERS-----\n-----BEGIN EC PRIVATE KEY-----\nMF"...>>} 3> {ok, Bin2} = file:read_file("ec_key_pub.pem"). {ok,<<"-----BEGIN PUBLIC KEY-----\nMEkwEwYHKoZIzj0CAQYIKoZIzj0DAQEDMgAEUzzXw5xYnyZKIzSIo1xKN3enUe3M\nWSURKrWKOUB+Of+e"...>>} 4> [OTPEcpkParamsPem, ECPrivateKeyPem] = public_key:pem_decode(Bin). [{'OTPEcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, not_encrypted}, {'ECPrivateKey',<<48,95,2,1,1,4,24,233,53,27,191,96,145, 18,240,229,16,151,232,48,95,167,203,176, 145,...>>, not_encrypted}] 5> ECPrivateKey = public_key:pem_entry_decode(ECPrivateKeyPem). #'ECPrivateKey'{version = 1, privateKey = [233,53,27,191,96,145,18,240,229,16,151,232,48, 95,167,203,176,145,117,205,108,210,242,204], parameters = {namedCurve,{1,2,840,10045,3,1,1}}, publicKey = {0, <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92, 74,55,119,167,81,237,204,...>>}} 6> {_, ParamsBin, ParamsEnc} = OTPEcpkParamsPem. {'OTPEcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, not_encrypted} 7> EcpkParametersPem = {'EcpkParameters', ParamsBin, ParamsEnc}. {'EcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, not_encrypted} 8> ECParams = public_key:pem_entry_decode(EcpkParametersPem). {namedCurve,{1,2,840,10045,3,1,1}} 9> Msg = <<"The superhamster strikes again at 7PM">>. <<"The superhamster strikes again at 7PM">> 10> Sig = public_key:sign(Msg, sha256, ECPrivateKey). <<48,53,2,25,0,233,72,76,239,158,251,181,114,254,166,85, 195,11,179,248,53,150,58,138,93,86,76,171,41,...>> 11> [SPKI] = public_key:pem_decode(Bin2). [#'SubjectPublicKeyInfo'{algorithm = <<48,73,48,19,6,7,42, 134,72,206,61,2,1, 6,8,42,134,72,206, 61,3,1,1,3,50,0,...>>, subjectPublicKey = not_encrypted}] 12> #'SubjectPublicKeyInfo'{algorithm = Der} = SPKI. #'SubjectPublicKeyInfo'{algorithm = <<48,73,48,19,6,7,42, 134,72,206,61,2,1,6, 8,42,134,72,206,61, 3,1,1,3,50,0,4,...>>, subjectPublicKey = not_encrypted} 14> RealSPKI = public_key:der_decode('SubjectPublicKeyInfo', Der). #'SubjectPublicKeyInfo'{ algorithm = #'AlgorithmIdentifier'{ algorithm = {1,2,840,10045,2,1}, parameters = <<6,8,42,134,72,206,61,3,1,1>>}, subjectPublicKey = {0, <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92,74, 55,119,167,81,237,204,89,37,...>>}} 15> #'SubjectPublicKeyInfo'{subjectPublicKey = {_, Octets}} = RealSPKI. #'SubjectPublicKeyInfo'{ algorithm = #'AlgorithmIdentifier'{ algorithm = {1,2,840,10045,2,1}, parameters = <<6,8,42,134,72,206,61,3,1,1>>}, subjectPublicKey = {0, <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92,74, 55,119,167,81,237,204,89,37,...>>}} 16> ECPoint = #'ECPoint'{point = Octets}. #'ECPoint'{point = <<4,83,60,215,195,156,88,159,38,74,35, 52,136,163,92,74,55,119,167,81,237, 204,89,37,17,42,181,...>>} 18> ECPublicKey = {ECPoint, ECParams}. {#'ECPoint'{point = <<4,83,60,215,195,156,88,159,38,74, 35,52,136,163,92,74,55,119,167,81, 237,204,89,37,17,42,...>>}, {namedCurve,{1,2,840,10045,3,1,1}}} 19> public_key:verify(Msg, sha256, Sig, ECPublicKey). true ============================= On Fri, Jul 26, 2013 at 7:38 PM, Ben Hood <0x6e6562@REDACTED> wrote: > Hi, > > I'd like to verify signatures signed with ec keys generated by openssl > in Erlang. > > Using R16B01 I've been able to convince the public_key module to sign > some data with an openssl generated private key thusly: > > {ok, Pem} = file:read_file("ec_private_key.pem"), > [ECParameters,ECPrivateKey] = public_key:pem_decode(Pem), > PrivateKey = public_key:pem_entry_decode(ECPrivateKey), > Curve = public_key:pem_entry_decode({'EcpkParameters', Params, Enc}), > Sig = public_key:sign(Message, sha512, PrivateKey). > > But when it comes to processing the public key, it is not entirely > clear how to decode it. This is where I got to: > > {ok, Pem} = file:read_file("ec_public_key.pem"), > [{_,Der,_}] = public_key:pem_decode(Pem), > {_, _, {0, KeyDer}} = public_key:der_decode('SubjectPublicKeyInfo', Der), > [179,52,127,201] = public_key:der_decode('ECPoint', KeyDer) > %% This is where the story ends > > I've looked through the ASN.1 types available and there doesn't seem > to be anything that resembles an ECPublicKey. > > Could anybody point me to working example? > > Cheers, > > Ben > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562@REDACTED Sat Jul 27 16:39:53 2013 From: 0x6e6562@REDACTED (Ben Hood) Date: Sat, 27 Jul 2013 15:39:53 +0100 Subject: [erlang-questions] ECC public key example In-Reply-To: References: Message-ID: Hi Gustav, Thank you very much for deciphering the API for me :-) Using your example as inspiration, I was able to sign and verify some data with the key pair generated using openssl: $ openssl ecparam -out ecdsa.pem -name sect571r1 -genkey $ openssl ec -in ecdsa.pem -pubout -out ecdsa.pub I take your point about how you pulled the ECParams out of the private key, although this is actually necessary to construct the public key. However, I was able to extract this out of the SPKI stanza: #'SubjectPublicKeyInfo'{algorithm = #'AlgorithmIdentifier'{algorithm = {1,2,840,10045,2,1}, parameters = <<6,5,43,129,4,0,39>>}, subjectPublicKey = {0, <<4,4,179,52,127,201,246,179,111,174,210,158,176,114,47, 61,131,42,95,74,96,132,1,196,...>>}}, EcpkParametersPem = {'EcpkParameters', <<6,5,43,129,4,0,39>>, not_encrypted}, ECParams = public_key:pem_entry_decode(EcpkParametersPem). As for the issue with the OTPEcpkParameters pattern mismatch, judging from the ASN.1 definitions for the public_key module in R16B01, I think that the entity has somehow been prefixed with OTP by mistake. Previously to parse the private key file I had taken the same approach - see my original post. I too would question whether the ASN.1 schema is correct in this regard. Anyway, although you have to jump through a number of hoops, ECC signature verification in Erlang is now working for me. Thanks very much for your help. Cheers, Ben On Sat, Jul 27, 2013 at 3:10 PM, Gustav Simonsson wrote: > Hi Ben, > > I've replicated your console commands (see end of mail) and continued with > steps which end with a successful verification of the signature with the EC > Public Key. > Some duplicated commands is removed from the log but it should still contain > all the necessary steps. > > Some things I noticed: > > * There is one SubjectPublicKeyInfo which contains another > SubjectPublicKeyInfo. > This doesn't seem correct and I'm not really sure where the error causing > this is. > > * The EC Public Key type in public_key is defined as: > > ec_public_key() = {#'ECPoint'{}, #'EcpkParameters'{} | {namedCurve, oid()}} > > But the EC Public Key Parameters are actually returned as a > 'OTPEcpkParameters' PEM entry (record) > whose record definition is missing. I suspect this could be a bug in > public_key. > > * I didn't find a way to get a #'ECPoint' record back from decoding, so I > manually created it by > matching out the ECPoint octets from the SPKI record. > > * As the EC parameters was in the private key file, they had to be manually > combined with the ECPoint > to form the EC Public Key. Maybe it's possible to tell openssl to include > the EC parameters in the public > key file with some option? > > Test keys where generated with: > > openssl ecparam -out ec_key_priv.pem -name prime192v1 -genkey > openssl ec -pubout -in ec_key_priv.pem -out ec_key_pub.pem > > Cheers, > Gustav Simonsson > > Erlang console log: > ================== > > Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] > [hipe] [kernel-poll:false] > > Eshell V5.10.2 (abort with ^G) > 1> rr(public_key). > ['AAControls','ACClearAttrs','AccessDescription', > 'Algorithm','AlgorithmIdentifier', > 'AlgorithmIdentifierPKCS-10','AlgorithmIdentifierPKCS-8', > 'AlgorithmIdentifierPKCS5v2-0','AlgorithmIdentifierPKSC-7', > 'AlgorithmNull','AnotherName','AttCertValidityPeriod', > 'Attribute','AttributeCertificate', > 'AttributeCertificateInfo','AttributePKCS-10', > 'AttributePKCS-7','AttributeTypeAndValue', > 'Attributes_SETOF', > 'Attributes_SETOF_valuesWithContext_SETOF', > 'AuthorityKeyIdentifier','BasicConstraints', > 'BuiltInDomainDefinedAttribute','BuiltInStandardAttributes', > 'Certificate','CertificateList','CertificationRequest', > 'CertificationRequestInfo', > 'CertificationRequestInfo_attributes_SETOF'|...] > 2> {ok, Bin} = file:read_file("ec_key_priv.pem"). > {ok,<<"-----BEGIN EC PARAMETERS-----\nBggqhkjOPQMBAQ==\n-----END EC > PARAMETERS-----\n-----BEGIN EC PRIVATE KEY-----\nMF"...>>} > 3> {ok, Bin2} = file:read_file("ec_key_pub.pem"). > {ok,<<"-----BEGIN PUBLIC > KEY-----\nMEkwEwYHKoZIzj0CAQYIKoZIzj0DAQEDMgAEUzzXw5xYnyZKIzSIo1xKN3enUe3M\nWSURKrWKOUB+Of+e"...>>} > 4> [OTPEcpkParamsPem, ECPrivateKeyPem] = public_key:pem_decode(Bin). > [{'OTPEcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, > not_encrypted}, > {'ECPrivateKey',<<48,95,2,1,1,4,24,233,53,27,191,96,145, > 18,240,229,16,151,232,48,95,167,203,176, > 145,...>>, > not_encrypted}] > 5> ECPrivateKey = public_key:pem_entry_decode(ECPrivateKeyPem). > #'ECPrivateKey'{version = 1, > privateKey = [233,53,27,191,96,145,18,240,229,16,151,232,48, > 95,167,203,176,145,117,205,108,210,242,204], > parameters = {namedCurve,{1,2,840,10045,3,1,1}}, > publicKey = {0, > > <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92, > 74,55,119,167,81,237,204,...>>}} > 6> {_, ParamsBin, ParamsEnc} = OTPEcpkParamsPem. > {'OTPEcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, > not_encrypted} > 7> EcpkParametersPem = {'EcpkParameters', ParamsBin, ParamsEnc}. > {'EcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, > not_encrypted} > 8> ECParams = public_key:pem_entry_decode(EcpkParametersPem). > {namedCurve,{1,2,840,10045,3,1,1}} > 9> Msg = <<"The superhamster strikes again at 7PM">>. > <<"The superhamster strikes again at 7PM">> > 10> Sig = public_key:sign(Msg, sha256, ECPrivateKey). > <<48,53,2,25,0,233,72,76,239,158,251,181,114,254,166,85, > 195,11,179,248,53,150,58,138,93,86,76,171,41,...>> > 11> [SPKI] = public_key:pem_decode(Bin2). > [#'SubjectPublicKeyInfo'{algorithm = <<48,73,48,19,6,7,42, > 134,72,206,61,2,1, > 6,8,42,134,72,206, > 61,3,1,1,3,50,0,...>>, > subjectPublicKey = not_encrypted}] > 12> #'SubjectPublicKeyInfo'{algorithm = Der} = SPKI. > #'SubjectPublicKeyInfo'{algorithm = <<48,73,48,19,6,7,42, > 134,72,206,61,2,1,6, > 8,42,134,72,206,61, > 3,1,1,3,50,0,4,...>>, > subjectPublicKey = not_encrypted} > 14> RealSPKI = public_key:der_decode('SubjectPublicKeyInfo', Der). > #'SubjectPublicKeyInfo'{ > algorithm = > #'AlgorithmIdentifier'{ > algorithm = {1,2,840,10045,2,1}, > parameters = <<6,8,42,134,72,206,61,3,1,1>>}, > subjectPublicKey = > {0, > <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92,74, > 55,119,167,81,237,204,89,37,...>>}} > 15> #'SubjectPublicKeyInfo'{subjectPublicKey = {_, Octets}} = RealSPKI. > #'SubjectPublicKeyInfo'{ > algorithm = > #'AlgorithmIdentifier'{ > algorithm = {1,2,840,10045,2,1}, > parameters = <<6,8,42,134,72,206,61,3,1,1>>}, > subjectPublicKey = > {0, > <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92,74, > 55,119,167,81,237,204,89,37,...>>}} > 16> ECPoint = #'ECPoint'{point = Octets}. > #'ECPoint'{point = <<4,83,60,215,195,156,88,159,38,74,35, > 52,136,163,92,74,55,119,167,81,237, > 204,89,37,17,42,181,...>>} > 18> ECPublicKey = {ECPoint, ECParams}. > {#'ECPoint'{point = <<4,83,60,215,195,156,88,159,38,74, > 35,52,136,163,92,74,55,119,167,81, > 237,204,89,37,17,42,...>>}, > {namedCurve,{1,2,840,10045,3,1,1}}} > 19> public_key:verify(Msg, sha256, Sig, ECPublicKey). > true > > ============================= > > > On Fri, Jul 26, 2013 at 7:38 PM, Ben Hood <0x6e6562@REDACTED> wrote: >> >> Hi, >> >> I'd like to verify signatures signed with ec keys generated by openssl >> in Erlang. >> >> Using R16B01 I've been able to convince the public_key module to sign >> some data with an openssl generated private key thusly: >> >> {ok, Pem} = file:read_file("ec_private_key.pem"), >> [ECParameters,ECPrivateKey] = public_key:pem_decode(Pem), >> PrivateKey = public_key:pem_entry_decode(ECPrivateKey), >> Curve = public_key:pem_entry_decode({'EcpkParameters', Params, Enc}), >> Sig = public_key:sign(Message, sha512, PrivateKey). >> >> But when it comes to processing the public key, it is not entirely >> clear how to decode it. This is where I got to: >> >> {ok, Pem} = file:read_file("ec_public_key.pem"), >> [{_,Der,_}] = public_key:pem_decode(Pem), >> {_, _, {0, KeyDer}} = public_key:der_decode('SubjectPublicKeyInfo', Der), >> [179,52,127,201] = public_key:der_decode('ECPoint', KeyDer) >> %% This is where the story ends >> >> I've looked through the ASN.1 types available and there doesn't seem >> to be anything that resembles an ECPublicKey. >> >> Could anybody point me to working example? >> >> Cheers, >> >> Ben >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From 0x6e6562@REDACTED Sat Jul 27 17:31:22 2013 From: 0x6e6562@REDACTED (Ben Hood) Date: Sat, 27 Jul 2013 16:31:22 +0100 Subject: [erlang-questions] ECC public key example In-Reply-To: References: Message-ID: Hey Gustav, Just in case anybody googles this and needs a quick solution, I've put a working example onto Github: https://github.com/0x6e6562/otp_ecc Cheers, Ben On Sat, Jul 27, 2013 at 3:39 PM, Ben Hood <0x6e6562@REDACTED> wrote: > Hi Gustav, > > Thank you very much for deciphering the API for me :-) > > Using your example as inspiration, I was able to sign and verify some > data with the key pair generated using openssl: > > $ openssl ecparam -out ecdsa.pem -name sect571r1 -genkey > $ openssl ec -in ecdsa.pem -pubout -out ecdsa.pub > > I take your point about how you pulled the ECParams out of the private > key, although this is actually necessary to construct the public key. > However, I was able to extract this out of the SPKI stanza: > > #'SubjectPublicKeyInfo'{algorithm = #'AlgorithmIdentifier'{algorithm = > {1,2,840,10045,2,1}, > > parameters = <<6,5,43,129,4,0,39>>}, > subjectPublicKey = {0, > > <<4,4,179,52,127,201,246,179,111,174,210,158,176,114,47, > > 61,131,42,95,74,96,132,1,196,...>>}}, > EcpkParametersPem = {'EcpkParameters', <<6,5,43,129,4,0,39>>, not_encrypted}, > ECParams = public_key:pem_entry_decode(EcpkParametersPem). > > As for the issue with the OTPEcpkParameters pattern mismatch, judging > from the ASN.1 definitions for the public_key module in R16B01, I > think that the entity has somehow been prefixed with OTP by mistake. > Previously to parse the private key file I had taken the same approach > - see my original post. I too would question whether the ASN.1 schema > is correct in this regard. > > Anyway, although you have to jump through a number of hoops, ECC > signature verification in Erlang is now working for me. > > Thanks very much for your help. > > Cheers, > > Ben > > On Sat, Jul 27, 2013 at 3:10 PM, Gustav Simonsson > wrote: >> Hi Ben, >> >> I've replicated your console commands (see end of mail) and continued with >> steps which end with a successful verification of the signature with the EC >> Public Key. >> Some duplicated commands is removed from the log but it should still contain >> all the necessary steps. >> >> Some things I noticed: >> >> * There is one SubjectPublicKeyInfo which contains another >> SubjectPublicKeyInfo. >> This doesn't seem correct and I'm not really sure where the error causing >> this is. >> >> * The EC Public Key type in public_key is defined as: >> >> ec_public_key() = {#'ECPoint'{}, #'EcpkParameters'{} | {namedCurve, oid()}} >> >> But the EC Public Key Parameters are actually returned as a >> 'OTPEcpkParameters' PEM entry (record) >> whose record definition is missing. I suspect this could be a bug in >> public_key. >> >> * I didn't find a way to get a #'ECPoint' record back from decoding, so I >> manually created it by >> matching out the ECPoint octets from the SPKI record. >> >> * As the EC parameters was in the private key file, they had to be manually >> combined with the ECPoint >> to form the EC Public Key. Maybe it's possible to tell openssl to include >> the EC parameters in the public >> key file with some option? >> >> Test keys where generated with: >> >> openssl ecparam -out ec_key_priv.pem -name prime192v1 -genkey >> openssl ec -pubout -in ec_key_priv.pem -out ec_key_pub.pem >> >> Cheers, >> Gustav Simonsson >> >> Erlang console log: >> ================== >> >> Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] >> [hipe] [kernel-poll:false] >> >> Eshell V5.10.2 (abort with ^G) >> 1> rr(public_key). >> ['AAControls','ACClearAttrs','AccessDescription', >> 'Algorithm','AlgorithmIdentifier', >> 'AlgorithmIdentifierPKCS-10','AlgorithmIdentifierPKCS-8', >> 'AlgorithmIdentifierPKCS5v2-0','AlgorithmIdentifierPKSC-7', >> 'AlgorithmNull','AnotherName','AttCertValidityPeriod', >> 'Attribute','AttributeCertificate', >> 'AttributeCertificateInfo','AttributePKCS-10', >> 'AttributePKCS-7','AttributeTypeAndValue', >> 'Attributes_SETOF', >> 'Attributes_SETOF_valuesWithContext_SETOF', >> 'AuthorityKeyIdentifier','BasicConstraints', >> 'BuiltInDomainDefinedAttribute','BuiltInStandardAttributes', >> 'Certificate','CertificateList','CertificationRequest', >> 'CertificationRequestInfo', >> 'CertificationRequestInfo_attributes_SETOF'|...] >> 2> {ok, Bin} = file:read_file("ec_key_priv.pem"). >> {ok,<<"-----BEGIN EC PARAMETERS-----\nBggqhkjOPQMBAQ==\n-----END EC >> PARAMETERS-----\n-----BEGIN EC PRIVATE KEY-----\nMF"...>>} >> 3> {ok, Bin2} = file:read_file("ec_key_pub.pem"). >> {ok,<<"-----BEGIN PUBLIC >> KEY-----\nMEkwEwYHKoZIzj0CAQYIKoZIzj0DAQEDMgAEUzzXw5xYnyZKIzSIo1xKN3enUe3M\nWSURKrWKOUB+Of+e"...>>} >> 4> [OTPEcpkParamsPem, ECPrivateKeyPem] = public_key:pem_decode(Bin). >> [{'OTPEcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, >> not_encrypted}, >> {'ECPrivateKey',<<48,95,2,1,1,4,24,233,53,27,191,96,145, >> 18,240,229,16,151,232,48,95,167,203,176, >> 145,...>>, >> not_encrypted}] >> 5> ECPrivateKey = public_key:pem_entry_decode(ECPrivateKeyPem). >> #'ECPrivateKey'{version = 1, >> privateKey = [233,53,27,191,96,145,18,240,229,16,151,232,48, >> 95,167,203,176,145,117,205,108,210,242,204], >> parameters = {namedCurve,{1,2,840,10045,3,1,1}}, >> publicKey = {0, >> >> <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92, >> 74,55,119,167,81,237,204,...>>}} >> 6> {_, ParamsBin, ParamsEnc} = OTPEcpkParamsPem. >> {'OTPEcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, >> not_encrypted} >> 7> EcpkParametersPem = {'EcpkParameters', ParamsBin, ParamsEnc}. >> {'EcpkParameters',<<6,8,42,134,72,206,61,3,1,1>>, >> not_encrypted} >> 8> ECParams = public_key:pem_entry_decode(EcpkParametersPem). >> {namedCurve,{1,2,840,10045,3,1,1}} >> 9> Msg = <<"The superhamster strikes again at 7PM">>. >> <<"The superhamster strikes again at 7PM">> >> 10> Sig = public_key:sign(Msg, sha256, ECPrivateKey). >> <<48,53,2,25,0,233,72,76,239,158,251,181,114,254,166,85, >> 195,11,179,248,53,150,58,138,93,86,76,171,41,...>> >> 11> [SPKI] = public_key:pem_decode(Bin2). >> [#'SubjectPublicKeyInfo'{algorithm = <<48,73,48,19,6,7,42, >> 134,72,206,61,2,1, >> 6,8,42,134,72,206, >> 61,3,1,1,3,50,0,...>>, >> subjectPublicKey = not_encrypted}] >> 12> #'SubjectPublicKeyInfo'{algorithm = Der} = SPKI. >> #'SubjectPublicKeyInfo'{algorithm = <<48,73,48,19,6,7,42, >> 134,72,206,61,2,1,6, >> 8,42,134,72,206,61, >> 3,1,1,3,50,0,4,...>>, >> subjectPublicKey = not_encrypted} >> 14> RealSPKI = public_key:der_decode('SubjectPublicKeyInfo', Der). >> #'SubjectPublicKeyInfo'{ >> algorithm = >> #'AlgorithmIdentifier'{ >> algorithm = {1,2,840,10045,2,1}, >> parameters = <<6,8,42,134,72,206,61,3,1,1>>}, >> subjectPublicKey = >> {0, >> <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92,74, >> 55,119,167,81,237,204,89,37,...>>}} >> 15> #'SubjectPublicKeyInfo'{subjectPublicKey = {_, Octets}} = RealSPKI. >> #'SubjectPublicKeyInfo'{ >> algorithm = >> #'AlgorithmIdentifier'{ >> algorithm = {1,2,840,10045,2,1}, >> parameters = <<6,8,42,134,72,206,61,3,1,1>>}, >> subjectPublicKey = >> {0, >> <<4,83,60,215,195,156,88,159,38,74,35,52,136,163,92,74, >> 55,119,167,81,237,204,89,37,...>>}} >> 16> ECPoint = #'ECPoint'{point = Octets}. >> #'ECPoint'{point = <<4,83,60,215,195,156,88,159,38,74,35, >> 52,136,163,92,74,55,119,167,81,237, >> 204,89,37,17,42,181,...>>} >> 18> ECPublicKey = {ECPoint, ECParams}. >> {#'ECPoint'{point = <<4,83,60,215,195,156,88,159,38,74, >> 35,52,136,163,92,74,55,119,167,81, >> 237,204,89,37,17,42,...>>}, >> {namedCurve,{1,2,840,10045,3,1,1}}} >> 19> public_key:verify(Msg, sha256, Sig, ECPublicKey). >> true >> >> ============================= >> >> >> On Fri, Jul 26, 2013 at 7:38 PM, Ben Hood <0x6e6562@REDACTED> wrote: >>> >>> Hi, >>> >>> I'd like to verify signatures signed with ec keys generated by openssl >>> in Erlang. >>> >>> Using R16B01 I've been able to convince the public_key module to sign >>> some data with an openssl generated private key thusly: >>> >>> {ok, Pem} = file:read_file("ec_private_key.pem"), >>> [ECParameters,ECPrivateKey] = public_key:pem_decode(Pem), >>> PrivateKey = public_key:pem_entry_decode(ECPrivateKey), >>> Curve = public_key:pem_entry_decode({'EcpkParameters', Params, Enc}), >>> Sig = public_key:sign(Message, sha512, PrivateKey). >>> >>> But when it comes to processing the public key, it is not entirely >>> clear how to decode it. This is where I got to: >>> >>> {ok, Pem} = file:read_file("ec_public_key.pem"), >>> [{_,Der,_}] = public_key:pem_decode(Pem), >>> {_, _, {0, KeyDer}} = public_key:der_decode('SubjectPublicKeyInfo', Der), >>> [179,52,127,201] = public_key:der_decode('ECPoint', KeyDer) >>> %% This is where the story ends >>> >>> I've looked through the ASN.1 types available and there doesn't seem >>> to be anything that resembles an ECPublicKey. >>> >>> Could anybody point me to working example? >>> >>> Cheers, >>> >>> Ben >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> From yoursurrogategod@REDACTED Sat Jul 27 20:22:26 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Sat, 27 Jul 2013 14:22:26 -0400 Subject: [erlang-questions] Why can't I use array:size in a guard in a method? Message-ID: The code: printArray(Array, Size, IndexToStartWith) when array:size(Array) > IndexToStartWith -> io:format("~w~n", [array:get(IndexToStartWith, Array)]), printArray(Array, Size, IndexToStartWith + 1). The error: > c(arrays_example). arrays_example.erl:23: illegal guard expression error Why not? I could pass the size of the array along (hence the Size variable), but why not procure the length in the guard on the fly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Sun Jul 28 00:02:53 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Sun, 28 Jul 2013 00:02:53 +0200 Subject: [erlang-questions] Why can't I use array:size in a guard in a method? In-Reply-To: References: Message-ID: <51F4438D.8080000@gmail.com> On 2013-07-27 20:22, Yves S. Garret wrote: > The code: > > printArray(Array, Size, IndexToStartWith) > when array:size(Array) > IndexToStartWith -> > io:format("~w~n", [array:get(IndexToStartWith, Array)]), > printArray(Array, Size, IndexToStartWith + 1). > > The error: > > > c(arrays_example). > arrays_example.erl:23: illegal guard expression > error > > Why not? I could pass the size of the array along (hence the Size > variable), but why not procure > the length in the guard on the fly? You can't use any remote calls (m:f(...)) in a guard. Guards must be side effect free, and if the function is in another module, the compiler can't guarantee that (in particular considering that the other module could be replaced at runtime with code that does something completely different). /Richard From yoursurrogategod@REDACTED Sun Jul 28 06:16:23 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Sun, 28 Jul 2013 00:16:23 -0400 Subject: [erlang-questions] How to best parse XML in erlang? Message-ID: Hello. I've recently found out about erlsom and xmerl on erlang. What would you recommend for a way to parse XML files in erlang? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Sun Jul 28 06:45:29 2013 From: rapsey@REDACTED (Rapsey) Date: Sun, 28 Jul 2013 06:45:29 +0200 Subject: [erlang-questions] How to best parse XML in erlang? In-Reply-To: References: Message-ID: This is what I use. It turnes the XML into: [{ElementName,Properties,SubitemsOrValue},..] parsexml(InputXml,Opt) -> F = fun(B) -> {Xml,_} = xmerl_scan:string(B, [{space,normalize},{encoding,"utf-8"},{validation,off}|Opt]), strip_whitespace(xmerl_lib:simplify_element(Xml)) end, try F(InputXml) of Res -> Res catch error:_X -> false end. strip_whitespace({El,Attr,Children}) -> NChild = lists:filter(fun(X) -> case X of " " -> false; _ -> true end end,Children), Ch = lists:map(fun(X) -> strip_whitespace(X) end,NChild), {El,Attr,Ch}; strip_whitespace(String) -> String. On Sun, Jul 28, 2013 at 6:16 AM, Yves S. Garret wrote: > Hello. I've recently found out about erlsom and xmerl on erlang. What > would you recommend for a > way to parse XML files in erlang? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From n.oxyde@REDACTED Sun Jul 28 10:39:12 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sun, 28 Jul 2013 10:39:12 +0200 Subject: [erlang-questions] How to best parse XML in erlang? In-Reply-To: References: Message-ID: <6DB9F8CF-CE15-476C-9836-A9C6E38D2478@gmail.com> Hello, If the XML file is from a third-party source, xmerl is a no-no because it will make the atom table blow up. Regards, -- Anthony Ramine Le 28 juil. 2013 ? 06:16, Yves S. Garret a ?crit : > Hello. I've recently found out about erlsom and xmerl on erlang. What would you recommend for a > way to parse XML files in erlang? From baliulia@REDACTED Sun Jul 28 21:51:46 2013 From: baliulia@REDACTED (=?UTF-8?B?SWduYXMgVnnFoW5pYXVza2Fz?=) Date: Sun, 28 Jul 2013 21:51:46 +0200 Subject: [erlang-questions] How to best parse XML in erlang? In-Reply-To: References: Message-ID: <51F57652.3020103@gmail.com> Hi, I was actually researching available XML parsers last week, so I'm just going to list all the options I found: * xmerl (the standard) * erlsom (the "better") * exml ( https://github.com/paulgray/exml) (expat based) * parsexml (KISS) ( https://github.com/maxlapshin/parsexml) * (shameless plug) my even more simplified fork of parsexml https://github.com/yfyf/parsexml It all depends on what kind of XML do you want to parse/support, i.e. answering these questions to yourself: * how complex is your XML? * can you keep holding the XMLs in memory? * how much do you care about speed? -- Ignas From ulf@REDACTED Mon Jul 29 00:07:23 2013 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 29 Jul 2013 00:07:23 +0200 Subject: [erlang-questions] How to best parse XML in erlang? In-Reply-To: <51F57652.3020103@gmail.com> References: <51F57652.3020103@gmail.com> Message-ID: <261AB668-478F-47AC-9586-BBC29890064E@feuerlabs.com> Note that xmerl nowadays has a number of token scanners and SAX parser variants, that don't blow up the atom table, and are as fast as erlsom. However, they aren't wired into the main xmerl utilities, such as validation and XPATH. Here's one very basic parser based on xmerl_sax_parser: -module(xmerl_simple). -export([file/1, stream/1]). file(F) -> xmerl_sax_parser:file(F, options()). stream(S) -> xmerl_sax_parser:stream(S, options()). options() -> [{event_state, []}, {event_fun, fun event/3}]. event({startElement, _, _LocalName, QName, Attrs}, _, Acc) -> Attrs1 = [{qname({Pfx,N}), list_to_binary(V)} || {_, Pfx, N, V} <- Attrs], [{qname(QName), Attrs1, []}|Acc]; event({endElement, _, _LocalName, QName}, _, S) -> Name = qname(QName), case S of {Cs, [{Name, As, C}|Acc]} -> end_element(Name, As, iolist_to_binary([C,Cs]), Acc); [{Name, Attrs, C}|Acc] -> end_element(Name, Attrs, lists:reverse(C), Acc) end; event({ignorableWhitespace, _}, _, {C, Acc}) -> {[C, " "], Acc}; event({characters, Cs}, _, {C, Acc}) -> {[C,Cs], Acc}; event({characters, Cs}, _, Acc) -> {Cs, Acc}; event(_, _, S) -> S. end_element(Name, As, C, []) -> {Name, As, C}; end_element(Name, As, C, [{PName, PAs, PC}|Acc]) -> [{PName, PAs, [{Name, As, C}|PC]}|Acc]. qname({NS,Name}) -> {list_to_atom(NS),list_to_atom(Name)}. * * * Simple XML file: * * * * parsing: Eshell V5.9.2 (abort with ^G) 1> xmerl_simple:file("examples/ncrfc_rpc_example_1.xml"). {ok,{{'',example}, [], [{{'',rpc}, [{{'','message-id'},<<"101">>},{{ex,'user-id'},<<"fred">>}], [{{'',get},[],[]}]}, {{'','rpc-reply'}, [{{'','message-id'},<<"101">>},{{ex,'user-id'},<<"fred">>}], [{{'',data},[],[]}]}]}, <<>>} (Note that while this parse emitted atoms, that was *my* doing - not the parser's). BR, Ulf On 28 Jul 2013, at 21:51, Ignas Vy?niauskas wrote: > Hi, > > I was actually researching available XML parsers last week, so I'm just > going to list all the options I found: > > * xmerl (the standard) > * erlsom (the "better") > * exml ( https://github.com/paulgray/exml) (expat based) > * parsexml (KISS) ( https://github.com/maxlapshin/parsexml) > * (shameless plug) my even more simplified fork of parsexml > https://github.com/yfyf/parsexml > > It all depends on what kind of XML do you want to parse/support, i.e. > answering these questions to yourself: > * how complex is your XML? > * can you keep holding the XMLs in memory? > * how much do you care about speed? > > -- > Ignas > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From ok@REDACTED Mon Jul 29 03:10:48 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 29 Jul 2013 13:10:48 +1200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> Message-ID: <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> On 26/07/2013, at 10:55 PM, Motiejus Jak?tys wrote: > Found it. > > http://tnetstrings.org/ May I suggest *not* using that approach? Let's start with size. I have a small JSON test collection. Here's the result of reading each element of that collection and json - writing as JSON with a newline after every comma and a space after every colon, but no indentation xml - using , , , number, string, e1 ... en or e1 ... en, with the key strings appearing as key='..' attributes on the children of elements cson - described below, using base 10 for integers c64 - same as cson, but using base 64 for integers tns - using the encoding described at tnetstrings.org json xml cson c64 tns 9 31 8 8 16 385 470 341 336 387 201 273 167 165 204 428 545 362 350 422 2864 3262 2676 2550 2903 680 905 543 534 659 25 57 22 17 34 258 356 204 204 249 33 48 27 27 33 529 715 421 410 515 1192 1621 939 911 1161 192 255 160 154 191 2425 3289 1853 1807 2378 23 30 20 12 26 23 30 20 12 26 32 52 13 12 39 50 82 37 37 50 42 109 35 26 61 7 10 5 5 6 I'm rather pleased that an encoding (cson) that I threw together in a couple of minutes handily beats the rest, but not surprised. There is a simple blunder in the TNetStrings design that causes serious inefficiency if you try to transport nontrivial data that way: the "type" code is at the wrong end. You have to read an entire object before you can start decoding it, which is just plain silly. _And_ it is hard to transmit floating-point numbers accurately. Not only that, you cannot stream the output. There is a JSONGenerator class recently added to Java so that you can stream large amounts of data out without actually having to hold much in memory; this is a need people genuinely have. Let's just look at the output code for three techniques, taken from my Smalltalk library. To keep it simple, let's just look at arrays. printJsonOn: aStream aStream nextPut: $[. self do: [:each | each printJsonOn: aStream] separatedBy: [aStream nextPut: $,; cr]. aStream nextPut: $]. Output goes directly to the output stream with NO intermediate objects created. You can stream this without knowing the size of the virtual array until the end. printCsonOn: aStream self size printOn: aStream. aStream nextPut: $[. self do: [:each | each printCsonOn: aStream]. Output goes directly to the output stream with NO intermediate objects created. You can stream this as long as you know the size of the virtual array at the beginning. printTNetStringOn: aStream |s| s := StringBuffer new: self size * 6. self do: [:each | each printTNetStringOn: s]. s size printOn: aStream. aStream nextPut: $:; nextPutAll: s; nextPut: $]. OUCH! You have to convert every element to a string, concatenate them, then write the size of the *string* (not the *array*), and then the string. The whole thing has to be held in memory as a string. You cannot stream this. And having paid the heavy cost of building the output, you get no special benefit from the input. Yes, you can preallocate *strings*, but since you are never told the size of *arrays* or *objects*, you cannot preallocate them. What then is this "cson"? It is the TNetString approach with three simple fixes: (1) Type information goes at the BEGINNING, not the end. (2) Sizes for arrays and objects are the element counts of the arrays and objects, NOT the character counts (still less the byte counts) of the strings that represent them, so you can preallocate and stream input. For example, given a path to an item, you could decode just that item without allocating *any* space for unwanted stuff -- though you would still have to decode it. (3) Floats are represented as integers times a power of 2 so that they can easily be transported without rounding. The output is a byte sequence; where characters appear they are to be encoded using UTF-8. "+" a positive integer "-" a negative integer "*""+" a positive float with positive exponent "*""-" a negative float with positive exponent "/""+" a positive float with negative exponent "/""-" a negative float with negative exponent ">" positive infinity or NaN "<" negative infinity or NaN "#" false "=" true "!" null " a Unicode string "["()* a sequence "{"()* a dictionary where is a possibly empty sequence of decimal digits with no leading zeros. (In particular, 0 -> "+", not "0+".) This means that null, false, true come out as "!", "#", "=". For extra compactness, numbers could be encoded in base 64, using the digits 0-9A-Za-z$@, but experimentally, that only saves a couple of percent. The numbers reported above use decimal encoding. There is no restriction here that the keys of an "object" can be only strings; that's for a higher level protocol to decide. If sender and receiver can both handle more general dictionaries, why not? * stands for *(2**) and / stands for /(2**). The representation is unique: either is empty and is empty (using "*+" for +0.0 and "*-" for -0.0) or is an odd integer. ">" and "<" are +infinity and -infinity respectively; other leading numbers indicate NaNs. TNetStrings claims the following advantages: 1. Trivial to parse in every language without making errors. FALSE. To parse an array, you have to first read in the whole string, and then recursively decode it. You can't decode as you go because you don't find out what it _is_ until you read the end. 2. Resistant to buffer overflows and other problems. MISLEADING. Whether there can be buffer overflows depends on the implementation. My JSON parser is not subject to buffer overflows, and I can't think why any competent programmer's JSON parser would be. 3. Fast and low resource intensive. FALSE. 4. Makes no assumptions about string contents and can store binary data without escaping or encoding them. UNCLEAR. Dan Bernstein's netstrings proposal was specific to 8-bit characters. It was unsuitable for transmitting text between 8-bit systems using different code pages. TNetStrings says that all counts are *byte* counts and all data are *byte* sequences, which leaves the handling of Unicode text -- essential if this is to serve where JSON serves -- totally unclear. If Unicode text is transmitted in UTF-8, then it *cannot* store binary data without escaping or encoding. This may even be FALSE, because floating point numbers count as binary data in my book, and certainly have to be encoded in this format. The idea of representing 1.23e-20 as 0.0000000000000000000123 strikes me as gratuitously odd; you don't want to see 1.2345e300 ! 5. Backward compatible with original netstrings. TRUE, but so what? 6. Transport agnostic, so it works with streams, messages, files, anything that's 8-bit clean. TRUEish. There is an unstated assumption that we are dealing with *byte* streams, not *text* streams, which is what makes claim 4 almost certainly false. In any case, JSON also has this property, and so does CSON. CSON claims the following advantages: 1. Easy to generate, including streaming output as long as you know array/object element counts when you start to write, creating *no* intermediate data structures. 2. Easy to read, creating *no* intermediate data structures. 3. Handles Unicode. 4. Handles floating point numbers precisely as long as the receiver is able to hold the numbers. 5. Encoded data are byte streams, just like JSON. and the following disadvantage: 6. You can skip an unwanted item without *allocating* it but not without *decoding* it. From ok@REDACTED Mon Jul 29 03:15:14 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 29 Jul 2013 13:15:14 +1200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> Message-ID: <81688198-1A13-4525-9D73-098664E33415@cs.otago.ac.nz> On 29/07/2013, at 1:10 PM, Richard A. O'Keefe wrote: > printTNetStringOn: aStream > |s| > s := StringBuffer new: self size * 6. > self do: [:each | each printTNetStringOn: s]. > s size printOn: aStream. > aStream nextPut: $:; nextPutAll: s; nextPut: $]. > > OUCH! You have to convert every element to a string, > concatenate them, then write the size of the *string* > (not the *array*), and then the string. The whole > thing has to be held in memory as a string. You cannot > stream this. I should have mentioned that Erlang can do this quite handily using iolists. The amount of memory allocated for holding all these intermediate strings is linear in the size of the result. That's in marked contrast to using Java, C++, C#, or Javascript strings. So TNetStrings almost make sense from an Erlang point of view. And Erlang's treatment of binaries makes chopping them up relatively efficient too. Even so, "linear space overhead" is still worse than "zero space overhead". From co7eb@REDACTED Mon Jul 29 05:46:19 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Sun, 28 Jul 2013 23:46:19 -0400 Subject: [erlang-questions] Controlling LPT port from Erlang Message-ID: <000001ce8c0e$34a78e30$9df6aa90$@frcuba.co.cu> Hi all, Kindest regards, I have a little deal with my father we are trying to program the DVD players memories, my father did already the programming interface hardware from a datasheet but we miss still the software part, I?m wondering if there is a way to control the LPT port using Erlang to send bytes through there, since the hardware interface work also through LPT printer port. Regards, Ivan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From i@REDACTED Mon Jul 29 06:14:04 2013 From: i@REDACTED (Anthony Grimes) Date: Sun, 28 Jul 2013 21:14:04 -0700 Subject: [erlang-questions] Issues with stdin on ports Message-ID: <51F5EC0C.2040203@raynes.me> Howdy folks. I unfortunately have not been able to use Erlang for most of what I've been doing lately because of a long standing issue with Erlang ports that I'd like to start a discussion about here. As far as I am aware, ports are generally the only option for creating and communicating with external processes in Erlang. They seem to have at least one particular fatal flaw which prevents them from being very useful to me, and that is that there is no way to close stdin (and send EOF) and then also read from the process's stdout. For example, I cannot use a port to start the 'cat' program which listens on stdin for data and waits for EOF and then echos that data back to you. I can do the first part, which is send it data on stdin, but the only way for me to close it is to call port_close and close the entire process. This issue prevents Erlang users from doing any even slightly more than trivial communication with external processes without having some kind of middleman program that handles the creation of the actual process you need to talk to and looks for a specific byte sequence to indicate 'EOF'. I could totally be wrong, but it seems like we need something other than just port_close. Something like http://www.erlang.org/doc/man/gen_tcp.html#shutdown-2 which lets you say "Hey, I want to close the stdin of this process but still read from its stdout." or something similar. I could be totally off track on what a good solution would be. So I'm wondering if people are aware of this problem, and I'd like to make sure that people think it is an actual problem that should be fixed. I'm also curious what people think a good solution to the problem would be. I'm not sure I have the time/particular skill set to fix it given that the port code is some pretty obscure (to me) C code, but starting conversation seems like a good way to begin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jul 29 09:15:52 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 29 Jul 2013 19:15:52 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F5EC0C.2040203@raynes.me> References: <51F5EC0C.2040203@raynes.me> Message-ID: <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> On 29/07/2013, at 4:14 PM, Anthony Grimes wrote: > and communicating with external processes in Erlang. They seem to have > at least one particular fatal flaw which prevents them from being very > useful to me, and that is that there is no way to close stdin (and send > EOF) and then also read from the process's stdout. For example, I cannot > use a port to start the 'cat' program which listens on stdin for data > and waits for EOF and then echos that data back to you. I can do the > first part, which is send it data on stdin, but the only way for me to > close it is to call port_close and close the entire process. Note that "only send data to a command" and "only receive data from a command" are the traditional ways for a UNIX program to communicate with another over a pipe. popen(, "r") reads the output of the command and popen(, "w") writes to the input of the command. There isn't even any standard _term_ for talking about connecting to both stdin and stdout of a command in UNIX, and that's because it's an incredibly easy way to deadlock. > This issue prevents Erlang users from doing any even slightly more than > trivial communication with external processes without having some kind > of middleman program that handles the creation of the actual process you > need to talk to and looks for a specific byte sequence to indicate 'EOF'. Just like it prevents C users from doing the same thing. Unless they fake something up using named pipes or UNIX-domain sockets. (Or message queues. I do wish Mac OS X implemented rather more of POSIX...) Unix anonymous pipes are simply the wrong tool for the job in _any_ programming language. The historic way to do "slightly more than trivial communication with external processes" has been to set the external processes up as C nodes or to use sockets. From dmkolesnikov@REDACTED Mon Jul 29 10:01:39 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 29 Jul 2013 11:01:39 +0300 Subject: [erlang-questions] Controlling LPT port from Erlang In-Reply-To: <000001ce8c0e$34a78e30$9df6aa90$@frcuba.co.cu> References: <000001ce8c0e$34a78e30$9df6aa90$@frcuba.co.cu> Message-ID: <8CF95AA7-1572-4694-9782-B083385A6D3E@gmail.com> Hello, I am not aware of any LPT support or existed libraries at Erlang. The solution would be OS dependent. BTW, what is OS you are doing development? All-in-all, I am afraid that you have to procedure with NIF development. Please see the following link about NIF development: http://www.erlang.org/doc/man/erl_nif.html http://www.erlang.org/doc/tutorial/nif.html If you happened to do Linux then I would recommend to look into following project. I use direct memory I/O (no linux kernel drivers are needed) http://lptdriver.googlecode.com/svn/trunk Best Regards, Dmitry On Jul 29, 2013, at 6:46 AM, Ivan Carmenates Garc?a wrote: > Hi all, > Kindest regards, > > I have a little deal with my father we are trying to program the DVD players memories, my father did already the programming interface hardware from a datasheet but we miss still the software part, I?m wondering if there is a way to control the LPT port using Erlang to send bytes through there, since the hardware interface work also through LPT printer port. > > Regards, > Ivan. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From i@REDACTED Mon Jul 29 10:46:31 2013 From: i@REDACTED (Anthony Grimes) Date: Mon, 29 Jul 2013 01:46:31 -0700 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> Message-ID: <51F62BE7.3020508@raynes.me> I'm not sure I follow. I think you may have misunderstood what I am trying to accomplish. Here is a simple low-level example I just whipped up in a Clojure repl using the standard built-in Java Process library. user=> (def proc (.exec (Runtime/getRuntime) (into-array String ["cat"]))) #'user/proc user=> (def stdin (.getOutputStream proc)) #'user/stdin user=> (def stdout (.getInputStream proc)) #'user/stdout user=> (.write stdin (.getBytes "Hi!")) nil user=> (.close stdin) nil user=> (let [arr (byte-array 3)] (.read stdout arr) (String. arr)) "Hi!" Lots of unix programs work like this. We have cat in this example, but grep, wc, and various others work like that as well, in that when you run them they listen for data and wait for you to send EOF and then they start producing output. It's a very common thing. Every language I can think of lets you work with programs like this. Ports are an abstraction for both reading and writing. The important missing piece of functionality is being able to send EOF and tell the program that you are done writing so that it can produce output. I'm not following why this is a bad thing. Perhaps this example and clarification will clear things up, and I apologize in advance if I am missing something crucial! -Anthony > Richard A. O'Keefe > July 29, 2013 12:15 AM > On 29/07/2013, at 4:14 PM, Anthony Grimes wrote: > >> and communicating with external processes in Erlang. They seem to have >> at least one particular fatal flaw which prevents them from being very >> useful to me, and that is that there is no way to close stdin (and send >> EOF) and then also read from the process's stdout. For example, I cannot >> use a port to start the 'cat' program which listens on stdin for data >> and waits for EOF and then echos that data back to you. I can do the >> first part, which is send it data on stdin, but the only way for me to >> close it is to call port_close and close the entire process. > > Note that "only send data to a command" and "only receive data from a > command" are the traditional ways for a UNIX program to communicate > with another over a pipe. popen(, "r") reads the output of > the command and popen(, "w") writes to the input of the command. > There isn't even any standard _term_ for talking about connecting to both > stdin and stdout of a command in UNIX, and that's because it's an > incredibly easy way to deadlock. > >> This issue prevents Erlang users from doing any even slightly more than >> trivial communication with external processes without having some kind >> of middleman program that handles the creation of the actual process you >> need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > Just like it prevents C users from doing the same thing. > Unless they fake something up using named pipes or UNIX-domain sockets. > (Or message queues. I do wish Mac OS X implemented rather more of POSIX...) > > Unix anonymous pipes are simply the wrong tool for the job in _any_ > programming language. > > The historic way to do "slightly more than trivial communication with > external processes" has been to set the external processes up as C nodes > or to use sockets. > > > > > Anthony Grimes > July 28, 2013 9:14 PM > Howdy folks. > > I unfortunately have not been able to use Erlang for most of what I've > been doing lately because of a long standing issue with Erlang ports > that I'd like to start a discussion about here. > > As far as I am aware, ports are generally the only option for creating > and communicating with external processes in Erlang. They seem to have > at least one particular fatal flaw which prevents them from being very > useful to me, and that is that there is no way to close stdin (and send > EOF) and then also read from the process's stdout. For example, I cannot > use a port to start the 'cat' program which listens on stdin for data > and waits for EOF and then echos that data back to you. I can do the > first part, which is send it data on stdin, but the only way for me to > close it is to call port_close and close the entire process. > > This issue prevents Erlang users from doing any even slightly more than > trivial communication with external processes without having some kind > of middleman program that handles the creation of the actual process you > need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > I could totally be wrong, but it seems like we need something other than > just port_close. Something like > http://www.erlang.org/doc/man/gen_tcp.html#shutdown-2 > which lets you say > "Hey, I want to close the stdin of this process but still read from its > stdout." or something similar. I could be totally off track on what a > good solution would be. > > So I'm wondering if people are aware of this problem, and I'd like to > make sure that people think it is an actual problem that should be > fixed. I'm also curious what people think a good solution to the problem > would be. I'm not sure I have the time/particular skill set to fix it > given that the port code is some pretty obscure (to me) C code, but > starting conversation seems like a good way to begin. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1188 bytes Desc: not available URL: From rtrlists@REDACTED Mon Jul 29 11:12:31 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Mon, 29 Jul 2013 10:12:31 +0100 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F5EC0C.2040203@raynes.me> References: <51F5EC0C.2040203@raynes.me> Message-ID: Hi Anthony, In the past, I've tended to use the port mechanism to simply kick off a C node, which then allows you to have full control over whatever communications needs you have. This obviously only works if you are interfacing with a technology that will allow you to create C node and use the EI libs in some way. Not sure if that is the case from what you wrote. Regards, Robby On Jul 29, 2013 8:04 AM, "Anthony Grimes" wrote: > Howdy folks. > > I unfortunately have not been able to use Erlang for most of what I've > been doing lately because of a long standing issue with Erlang ports > that I'd like to start a discussion about here. > > As far as I am aware, ports are generally the only option for creating > and communicating with external processes in Erlang. They seem to have > at least one particular fatal flaw which prevents them from being very > useful to me, and that is that there is no way to close stdin (and send > EOF) and then also read from the process's stdout. For example, I cannot > use a port to start the 'cat' program which listens on stdin for data > and waits for EOF and then echos that data back to you. I can do the > first part, which is send it data on stdin, but the only way for me to > close it is to call port_close and close the entire process. > > This issue prevents Erlang users from doing any even slightly more than > trivial communication with external processes without having some kind > of middleman program that handles the creation of the actual process you > need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > I could totally be wrong, but it seems like we need something other than > just port_close. Something like > http://www.erlang.org/doc/man/**gen_tcp.html#shutdown-2 > which lets you say > "Hey, I want to close the stdin of this process but still read from its > stdout." or something similar. I could be totally off track on what a > good solution would be. > > So I'm wondering if people are aware of this problem, and I'd like to > make sure that people think it is an actual problem that should be > fixed. I'm also curious what people think a good solution to the problem > would be. I'm not sure I have the time/particular skill set to fix it > given that the port code is some pretty obscure (to me) C code, but > starting conversation seems like a good way to begin. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From i@REDACTED Mon Jul 29 11:19:25 2013 From: i@REDACTED (Anthony Grimes) Date: Mon, 29 Jul 2013 02:19:25 -0700 (PDT) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: References: Message-ID: <1375089565068.a42008e@Nodemailer> Hey Robert. I don't think C nodes help in this case, and they don't solve the general problem. One of my use cases is talking to pygmentize, which is a Python program. If I want to do that, I have to write a middleman program that does the actual communication with this program and talks to my Erlang program over a socket, or via ports if I make the middleman program look for some specific sequence of bytes to treat as EOF since I can't send actual EOF. On Mon, Jul 29, 2013 at 2:12 AM, Robert Raschke wrote: > Hi Anthony, > In the past, I've tended to use the port mechanism to simply kick off a C > node, which then allows you to have full control over whatever > communications needs you have. > This obviously only works if you are interfacing with a technology that > will allow you to create C node and use the EI libs in some way. Not sure > if that is the case from what you wrote. > Regards, > Robby > On Jul 29, 2013 8:04 AM, "Anthony Grimes" wrote: >> Howdy folks. >> >> I unfortunately have not been able to use Erlang for most of what I've >> been doing lately because of a long standing issue with Erlang ports >> that I'd like to start a discussion about here. >> >> As far as I am aware, ports are generally the only option for creating >> and communicating with external processes in Erlang. They seem to have >> at least one particular fatal flaw which prevents them from being very >> useful to me, and that is that there is no way to close stdin (and send >> EOF) and then also read from the process's stdout. For example, I cannot >> use a port to start the 'cat' program which listens on stdin for data >> and waits for EOF and then echos that data back to you. I can do the >> first part, which is send it data on stdin, but the only way for me to >> close it is to call port_close and close the entire process. >> >> This issue prevents Erlang users from doing any even slightly more than >> trivial communication with external processes without having some kind >> of middleman program that handles the creation of the actual process you >> need to talk to and looks for a specific byte sequence to indicate 'EOF'. >> >> I could totally be wrong, but it seems like we need something other than >> just port_close. Something like >> http://www.erlang.org/doc/man/**gen_tcp.html#shutdown-2 >> which lets you say >> "Hey, I want to close the stdin of this process but still read from its >> stdout." or something similar. I could be totally off track on what a >> good solution would be. >> >> So I'm wondering if people are aware of this problem, and I'd like to >> make sure that people think it is an actual problem that should be >> fixed. I'm also curious what people think a good solution to the problem >> would be. I'm not sure I have the time/particular skill set to fix it >> given that the port code is some pretty obscure (to me) C code, but >> starting conversation seems like a good way to begin. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtrlists@REDACTED Mon Jul 29 13:16:54 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Mon, 29 Jul 2013 12:16:54 +0100 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <1375089565068.a42008e@Nodemailer> References: <1375089565068.a42008e@Nodemailer> Message-ID: Hi Anthony, could Py-interface http://www.lysator.liu.se/~tab/erlang/py_interface/ help with that? Note, I've not used it, so don't know much at all beyond reading that page. It would probably mean using the Pygments library, rather than going via the command line pygmentize. You can probably take the pygmentize source and replace the command line handling with setting up a Python Node. Alternatively, and not necessarily very nice, instead of streaming to pygmentize, write to a file and invoke on that. You wouldn't even need a port then, you could get away with os:cmd/1 (if you aren't interested in return codes). But you already know this, I think. Robby PS Something like expect for Erlang ports would be pretty cool, though. Not that I'm not volunteering ;-) On Mon, Jul 29, 2013 at 10:19 AM, Anthony Grimes wrote: > Hey Robert. > > I don't think C nodes help in this case, and they don't solve the general > problem. One of my use cases is talking to pygmentize, which is a Python > program. If I want to do that, I have to write a middleman program that > does the actual communication with this program and talks to my Erlang > program over a socket, or via ports if I make the middleman program look > for some specific sequence of bytes to treat as EOF since I can't send > actual EOF. > > > On Mon, Jul 29, 2013 at 2:12 AM, Robert Raschke wrote: > >> Hi Anthony, >> >> In the past, I've tended to use the port mechanism to simply kick off a C >> node, which then allows you to have full control over whatever >> communications needs you have. >> >> This obviously only works if you are interfacing with a technology that >> will allow you to create C node and use the EI libs in some way. Not sure >> if that is the case from what you wrote. >> >> Regards, >> Robby >> On Jul 29, 2013 8:04 AM, "Anthony Grimes" wrote: >> >>> Howdy folks. >>> >>> I unfortunately have not been able to use Erlang for most of what I've >>> been doing lately because of a long standing issue with Erlang ports >>> that I'd like to start a discussion about here. >>> >>> As far as I am aware, ports are generally the only option for creating >>> and communicating with external processes in Erlang. They seem to have >>> at least one particular fatal flaw which prevents them from being very >>> useful to me, and that is that there is no way to close stdin (and send >>> EOF) and then also read from the process's stdout. For example, I cannot >>> use a port to start the 'cat' program which listens on stdin for data >>> and waits for EOF and then echos that data back to you. I can do the >>> first part, which is send it data on stdin, but the only way for me to >>> close it is to call port_close and close the entire process. >>> >>> This issue prevents Erlang users from doing any even slightly more than >>> trivial communication with external processes without having some kind >>> of middleman program that handles the creation of the actual process you >>> need to talk to and looks for a specific byte sequence to indicate 'EOF'. >>> >>> I could totally be wrong, but it seems like we need something other than >>> just port_close. Something like >>> http://www.erlang.org/doc/man/**gen_tcp.html#shutdown-2 >>> which lets you say >>> "Hey, I want to close the stdin of this process but still read from its >>> stdout." or something similar. I could be totally off track on what a >>> good solution would be. >>> >>> So I'm wondering if people are aware of this problem, and I'd like to >>> make sure that people think it is an actual problem that should be >>> fixed. I'm also curious what people think a good solution to the problem >>> would be. I'm not sure I have the time/particular skill set to fix it >>> given that the port code is some pretty obscure (to me) C code, but >>> starting conversation seems like a good way to begin. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Jul 29 13:54:54 2013 From: erlang@REDACTED (=?UTF-8?Q?Micha=C5=82_Ptaszek?=) Date: Mon, 29 Jul 2013 13:54:54 +0200 Subject: [erlang-questions] Monitor option in spawn_opt/5 In-Reply-To: References: Message-ID: I'm not sure if spawn and monitor have to be atomic though. If we set up a monitor on non-existing/exiting process 'DOWN' message is going to be delivered to the caller anyway ("A 'DOWN' message will be sent to the monitoring process if Item dies, if Item does not exist, or if the connection is lost to the node which Item resides on.", http://www.erlang.org/doc/man/erlang.html#monitor-2). It's not a big deal for me to spin up a monitor on my own, however it would be good to know if I miss something or should code/documentation be fixed. On Sat, Jul 27, 2013 at 1:03 PM, Motiejus Jak?tys wrote: > On Sat, Jul 27, 2013 at 1:05 PM, Micha? Ptaszek > wrote: > > Hey, > > > > What is the reason for not allowing to pass 'monitor' option to processes > > spawned by erlang:spawn_opt/5? > > > > The code inside of erlang.erl states: > > > > case lists:member(monitor, O) of > > false -> ok; > > true -> erlang:error(badarg, [N, M, F, A, O]) > > end, > > > > However the documentation does not clarify anything. > > Setting up a monitor on the returned pid right after spawn works fine > > though. > > I guess the reason is that there is no way in Erlang to atomically > spawn and monitor the pid on another node. Though it's a bit strange > that [link] works on remote nodes, but [monitor] doesn't. Anyone care > to explain why? > > -- > Motiejus Jak?tys > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Mon Jul 29 13:55:37 2013 From: tony@REDACTED (Tony Rogvall) Date: Mon, 29 Jul 2013 13:55:37 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> Message-ID: I am also pretty pleased with cson :-) Looks very nice. A small question. How are strings encoded? a) " b) "* c) "* d) Other? Thanks /Tony On 29 jul 2013, at 03:10, Richard A. O'Keefe wrote: > > On 26/07/2013, at 10:55 PM, Motiejus Jak?tys wrote: >> Found it. >> >> http://tnetstrings.org/ > > May I suggest *not* using that approach? > > Let's start with size. I have a small JSON test collection. > Here's the result of reading each element of that collection > and > json - writing as JSON with a newline after every comma and > a space after every colon, but no indentation > xml - using , , , number, string, > e1 ... en or e1 ... en, with the key > strings appearing as key='..' attributes on the children > of elements > cson - described below, using base 10 for integers > c64 - same as cson, but using base 64 for integers > tns - using the encoding described at tnetstrings.org > > > json xml cson c64 tns > 9 31 8 8 16 > 385 470 341 336 387 > 201 273 167 165 204 > 428 545 362 350 422 > 2864 3262 2676 2550 2903 > 680 905 543 534 659 > 25 57 22 17 34 > 258 356 204 204 249 > 33 48 27 27 33 > 529 715 421 410 515 > 1192 1621 939 911 1161 > 192 255 160 154 191 > 2425 3289 1853 1807 2378 > 23 30 20 12 26 > 23 30 20 12 26 > 32 52 13 12 39 > 50 82 37 37 50 > 42 109 35 26 61 > 7 10 5 5 6 > > > I'm rather pleased that an encoding (cson) that I threw together > in a couple of minutes handily beats the rest, but not surprised. > > There is a simple blunder in the TNetStrings design that causes > serious inefficiency if you try to transport nontrivial > data that way: the "type" code is at the wrong end. > > You have to read an entire object before you can start > decoding it, which is just plain silly. _And_ it is hard > to transmit floating-point numbers accurately. > > Not only that, you cannot stream the output. There is a > JSONGenerator class recently added to Java so that you can > stream large amounts of data out without actually having to > hold much in memory; this is a need people genuinely have. > > Let's just look at the output code for three techniques, taken > from my Smalltalk library. To keep it simple, let's just look > at arrays. > > printJsonOn: aStream > aStream nextPut: $[. > self do: [:each | each printJsonOn: aStream] > separatedBy: [aStream nextPut: $,; cr]. > aStream nextPut: $]. > > > Output goes directly to the output stream with NO > intermediate objects created. You can stream this > without knowing the size of the virtual array until > the end. > > printCsonOn: aStream > self size printOn: aStream. > aStream nextPut: $[. > self do: [:each | each printCsonOn: aStream]. > > Output goes directly to the output stream with NO > intermediate objects created. You can stream this > as long as you know the size of the virtual array > at the beginning. > > printTNetStringOn: aStream > |s| > s := StringBuffer new: self size * 6. > self do: [:each | each printTNetStringOn: s]. > s size printOn: aStream. > aStream nextPut: $:; nextPutAll: s; nextPut: $]. > > OUCH! You have to convert every element to a string, > concatenate them, then write the size of the *string* > (not the *array*), and then the string. The whole > thing has to be held in memory as a string. You cannot > stream this. > > And having paid the heavy cost of building the output, you get > no special benefit from the input. Yes, you can preallocate > *strings*, but since you are never told the size of *arrays* or > *objects*, you cannot preallocate them. > > What then is this "cson"? > > It is the TNetString approach with three simple fixes: > (1) Type information goes at the BEGINNING, not the end. > (2) Sizes for arrays and objects are the element counts of the > arrays and objects, NOT the character counts (still less > the byte counts) of the strings that represent them, > so you can preallocate and stream input. For example, > given a path to an item, you could decode just that item > without allocating *any* space for unwanted stuff -- though > you would still have to decode it. > (3) Floats are represented as integers times a power of 2 > so that they can easily be transported without rounding. > The output is a byte sequence; where characters appear they > are to be encoded using UTF-8. > > "+" a positive integer > "-" a negative integer > "*""+" a positive float with positive exponent > "*""-" a negative float with positive exponent > "/""+" a positive float with negative exponent > "/""-" a negative float with negative exponent > ">" positive infinity or NaN > "<" negative infinity or NaN > "#" false > "=" true > "!" null > " a Unicode string > "["()* a sequence > "{"()* a dictionary > > where is a possibly empty sequence of decimal digits > with no leading zeros. (In particular, 0 -> "+", not "0+".) > This means that null, false, true come out as "!", "#", "=". > For extra compactness, numbers could be encoded in base 64, > using the digits 0-9A-Za-z$@, > but experimentally, that only saves a couple of percent. > The numbers reported above use decimal encoding. > > There is no restriction here that the keys of an "object" can be only > strings; that's for a higher level protocol to decide. If sender and > receiver can both handle more general dictionaries, why not? > > * stands for *(2**) and / stands for /(2**). > The representation is unique: either is empty and is > empty (using "*+" for +0.0 and "*-" for -0.0) or is an odd > integer. ">" and "<" are +infinity and -infinity respectively; > other leading numbers indicate NaNs. > > > TNetStrings claims the following advantages: > > 1. Trivial to parse in every language without making errors. > > FALSE. To parse an array, you have to first read in the > whole string, and then recursively decode it. You can't > decode as you go because you don't find out what it _is_ > until you read the end. > > 2. Resistant to buffer overflows and other problems. > > MISLEADING. Whether there can be buffer overflows depends > on the implementation. My JSON parser is not subject to > buffer overflows, and I can't think why any competent programmer's > JSON parser would be. > > 3. Fast and low resource intensive. > > FALSE. > > 4. Makes no assumptions about string contents and can store binary data > without escaping or encoding them. > > UNCLEAR. Dan Bernstein's netstrings proposal was specific to 8-bit > characters. It was unsuitable for transmitting text between 8-bit > systems using different code pages. TNetStrings says that all > counts are *byte* counts and all data are *byte* sequences, which > leaves the handling of Unicode text -- essential if this is to serve > where JSON serves -- totally unclear. If Unicode text is transmitted > in UTF-8, then it *cannot* store binary data without escaping or > encoding. This may even be FALSE, because floating point numbers > count as binary data in my book, and certainly have to be encoded in > this format. The idea of representing 1.23e-20 as > 0.0000000000000000000123 strikes me as gratuitously odd; you don't > want to see 1.2345e300 ! > > 5. Backward compatible with original netstrings. > > TRUE, but so what? > > 6. Transport agnostic, so it works with streams, messages, files, > anything that's 8-bit clean. > > TRUEish. There is an unstated assumption that we are dealing > with *byte* streams, not *text* streams, which is what makes > claim 4 almost certainly false. In any case, JSON also has > this property, and so does CSON. > > CSON claims the following advantages: > > 1. Easy to generate, including streaming output as long as you > know array/object element counts when you start to write, > creating *no* intermediate data structures. > > 2. Easy to read, creating *no* intermediate data structures. > > 3. Handles Unicode. > > 4. Handles floating point numbers precisely as long as the > receiver is able to hold the numbers. > > 5. Encoded data are byte streams, just like JSON. > > and the following disadvantage: > > 6. You can skip an unwanted item without *allocating* it but > not without *decoding* it. > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Jul 29 14:08:07 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 29 Jul 2013 14:08:07 +0200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F5EC0C.2040203@raynes.me> References: <51F5EC0C.2040203@raynes.me> Message-ID: On Mon, Jul 29, 2013 at 6:14 AM, Anthony Grimes wrote: > So I'm wondering if people are aware of this problem, and I'd like to > make sure that people think it is an actual problem that should be > fixed. I'm also curious what people think a good solution to the problem > would be. I'm not sure I have the time/particular skill set to fix it > given that the port code is some pretty obscure (to me) C code, but > starting conversation seems like a good way to begin. I don't think ports were created with that use case in mind. Usually when connecting Erlang to foreign programs, you have the advantage of being able to specify the interface fully and then you can pick a better protocol than "send message and then EOF". The deeper trouble here is that such an interface means it is use-once. It will just blindly fork()/exec() programs all the time and that hardly seems productive and efficient. It probably will not scale very well and then you are looking at a framing solution on the stream in any case. Note the same problem is present with gen_tcp:shutdown/2. It seems counterproductive to pay the handshake every time you want to communicate. Another problem is that with a fork()/exec() solution you ctx switch to the pygmentize process (or whatever process) for one processing only and then you switch away again. You can't pipeline stuff to pygmentize and hence you pay ctx switch overhead all the time. In other words, I have a hard time seeing this to be useful, but you may correct me on that if you have a better use case or can shoot down my stream of thought. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Jul 29 15:49:41 2013 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 29 Jul 2013 15:49:41 +0200 Subject: [erlang-questions] Monitor option in spawn_opt/5 In-Reply-To: References: Message-ID: It shouldn't be the need for atomicity as much as a convenience and - in the case of remote spawn - an optimization. On R15B, it works as you'd expect: Erlang R15B (erts-5.9) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9 (abort with ^G) 1> spawn_opt(fun() -> io:fwrite("hi~n",[]) end, [monitor]). hi {<0.33.0>,#Ref<0.0.0.33>} 2> flush(). Shell got {'DOWN',#Ref<0.0.0.33>,process,<0.33.0>,normal} ok Same on R16B, as far as I can tell. BR, Ulf W On 29 Jul 2013, at 13:54, Micha? Ptaszek wrote: > I'm not sure if spawn and monitor have to be atomic though. If we set up a monitor on non-existing/exiting process 'DOWN' message is going to be delivered to the caller anyway ("A 'DOWN' message will be sent to the monitoring process if Item dies, if Item does not exist, or if the connection is lost to the node which Item resides on.", http://www.erlang.org/doc/man/erlang.html#monitor-2). > > It's not a big deal for me to spin up a monitor on my own, however it would be good to know if I miss something or should code/documentation be fixed. > > > On Sat, Jul 27, 2013 at 1:03 PM, Motiejus Jak?tys wrote: > On Sat, Jul 27, 2013 at 1:05 PM, Micha? Ptaszek wrote: > > Hey, > > > > What is the reason for not allowing to pass 'monitor' option to processes > > spawned by erlang:spawn_opt/5? > > > > The code inside of erlang.erl states: > > > > case lists:member(monitor, O) of > > false -> ok; > > true -> erlang:error(badarg, [N, M, F, A, O]) > > end, > > > > However the documentation does not clarify anything. > > Setting up a monitor on the returned pid right after spawn works fine > > though. > > I guess the reason is that there is no way in Erlang to atomically > spawn and monitor the pid on another node. Though it's a bit strange > that [link] works on remote nodes, but [monitor] doesn't. Anyone care > to explain why? > > -- > Motiejus Jak?tys > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From per@REDACTED Mon Jul 29 16:13:15 2013 From: per@REDACTED (Per Hedeland) Date: Mon, 29 Jul 2013 16:13:15 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> Message-ID: <201307291413.r6TEDFaX063329@pluto.hedeland.org> "Richard A. O'Keefe" wrote: > >On 29/07/2013, at 4:14 PM, Anthony Grimes wrote: > >> and communicating with external processes in Erlang. They seem to have >> at least one particular fatal flaw which prevents them from being very >> useful to me, and that is that there is no way to close stdin (and send >> EOF) and then also read from the process's stdout. For example, I cannot >> use a port to start the 'cat' program which listens on stdin for data >> and waits for EOF and then echos that data back to you. I can do the >> first part, which is send it data on stdin, but the only way for me to >> close it is to call port_close and close the entire process. FWIW, I definitely agree that this is a missing piece of functionality. I'm not sure how useful/important it is in the grand scheme of things, but personally I could have used it on a couple of occasions. As you mentioned, it's basically the equivalent of TCP shutdown() that is needed, although shutdown() is perhaps a bit over-engineered - I've never seen anyone use SHUT_RD or SHUT_RDWR... Also the "opposite" functionality is already available for ports via the 'eof' option - i.e. you get informed that the other end has closed its write side, but can still write data in the other direction. >Note that "only send data to a command" and "only receive data from a >command" are the traditional ways for a UNIX program to communicate >with another over a pipe. Well, it's basically the definition of the traditional pipeline concept of the Unix shells, and pipes are obviously what you need to implement it - but that doesn't preclude other uses of pipes. The zsh shell even allows you to set up "bidirectional pipes" on the commandline. > popen(, "r") reads the output of >the command and popen(, "w") writes to the input of the command. popen() is effectively a convenience function to abstract away the somewhat non-trivial application of pipe(), fork(), close(), and execve() that is required to set things up correctly for two particular and common usages of pipes in application code. (It is not used by common shells to implement pipelines though.) >There isn't even any standard _term_ for talking about connecting to both >stdin and stdout of a command in UNIX, and that's because it's an >incredibly easy way to deadlock. There is no need to have a term for it, since all you need is two pipes, one for each direction - and it's probably uncommon enough to not warrant its own convenience function. And you can indeed easily deadlock if you don't think about what you're doing, but I really doubt that this is the reason for any absence of terminology or functions. But anyway, I don't see how any of this is relevant to the question at hand. Opening a bi-directional connection between two processes by means of a pair of pipes is exactly what erlang:open_port/2 *already does* when you use 'spawn' (or 'spawn_executable' these days) to start an external process. And it has been doing this since day one, and I can't recall anyone complaining how this is hopelessly dangerous due to the risk of deadlock (the risk is of course reduced due to the fact that the VM does non-blocking I/O). >> This issue prevents Erlang users from doing any even slightly more than >> trivial communication with external processes without having some kind >> of middleman program that handles the creation of the actual process you >> need to talk to and looks for a specific byte sequence to indicate 'EOF'. > >Just like it prevents C users from doing the same thing. No, there is nothing that prevents C users from doing the same thing. And even if they have to go to some effort to do it, it just means having to write a bit more C - whereas the Erlang user can't write a bit more Erlang to do just the small addition of "close the write side of one of the pipes", even though the pipe pair is already there... >Unix anonymous pipes are simply the wrong tool for the job in _any_ >programming language. > >The historic way to do "slightly more than trivial communication with >external processes" has been to set the external processes up as C nodes >or to use sockets. Using (TCP) sockets instead of pipes doesn't really change the "risk of deadlock". In the case of the Erlang VM (i.e. open_port vs gen_tcp), it may actually increase it, due to the existence of the passive and {active, once} modes for sockets - another piece of functionality that is "missing" from ports. --Per Hedeland From wojtek@REDACTED Mon Jul 29 16:25:31 2013 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Mon, 29 Jul 2013 16:25:31 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> Message-ID: <51F67B5B.4020503@power.com.pl> W dniu 2013-07-26 11:20, Ulf Wiger pisze: > At Feuerlabs, we use (among other things) BERT [1] and > Yang [2] (RFC 6020) to accomplish RPCs with contract checking. > > In addition, our BERT implementation has access control lists and > is pretty flexible about transport and authentication (with the option > of an authentication plugin), and our Yang compiler (developed in > cooperation with Tail-f) can generate Erlang code from a spec and > has a facility to validate a JSON-style RPC against the 'contract'. > > Documentation is a bit sparse (hrm?) but our support staff can be very helpful. > > [1] https://github.com/Feuerlabs/bert > [2] https://github.com/Feuerlabs/yang > > It makes me wonder, what is wrong with ASN.1? I have never used it, but on paper it looks superior to any more recent schema description language. Sorry for being off-subject. --Kind regards, Wojtek Narczynski From erlang@REDACTED Mon Jul 29 16:31:12 2013 From: erlang@REDACTED (=?UTF-8?Q?Micha=C5=82_Ptaszek?=) Date: Mon, 29 Jul 2013 16:31:12 +0200 Subject: [erlang-questions] os:cmd performance Message-ID: Hey, I am trying to run some shell commands from Erlang, using os:cmd/1: os:cmd("cat /tmp/huge_file | grep 'some_regexp' | sort -u > /tmp/output_file"). Where huge_file is ~4GB big, the output_file is expected to occupy ~150MB on disk. When running this command from Erlang it takes ~45-50 minutes to create the output file, however when ran directly from the shell (bash) everything happens within 10 minutes. It's not a matter of FS cache (I "echo 3 > /proc/sys/vm/drop_caches" right before each test), or external load (it's a physical machine, completely idle during the test). Erlang version: R16B. My question here is not "how to do it?", but rather "why os:cmd/1 is so inefficient?", even though the output returned to the caller is empty (stdout is redirected to the disk file, there are no stderr printouts). Or maybe I'm doing something entirely stupid that should be avoided... Any clues/hints? Thanks, Michal -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Mon Jul 29 16:41:15 2013 From: jon@REDACTED (Jon Schneider) Date: Mon, 29 Jul 2013 15:41:15 +0100 Subject: [erlang-questions] os:cmd performance In-Reply-To: References: Message-ID: <039b88537897118f459aa4992bc68a19.squirrel@mail.jschneider.net> > os:cmd("cat /tmp/huge_file | grep 'some_regexp' | sort -u > > /tmp/output_file"). Never understood why so many people do cat file | something instead of something References: Message-ID: Could it be because of a different environment? In particular, a different PATH could make a difference of that scale. Try running "env" in both ways, and compare. Den 29/07/2013 16.31 skrev "Micha? Ptaszek" : > Hey, > > I am trying to run some shell commands from Erlang, using os:cmd/1: > > os:cmd("cat /tmp/huge_file | grep 'some_regexp' | sort -u > > /tmp/output_file"). > > Where huge_file is ~4GB big, the output_file is expected to occupy ~150MB > on disk. When running this command from Erlang it takes ~45-50 minutes to > create the output file, however when ran directly from the shell (bash) > everything happens within 10 minutes. > It's not a matter of FS cache (I "echo 3 > /proc/sys/vm/drop_caches" right > before each test), or external load (it's a physical machine, completely > idle during the test). Erlang version: R16B. > > My question here is not "how to do it?", but rather "why os:cmd/1 is so > inefficient?", even though the output returned to the caller is empty > (stdout is redirected to the disk file, there are no stderr printouts). Or > maybe I'm doing something entirely stupid that should be avoided... > > Any clues/hints? > Thanks, > Michal > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthonym@REDACTED Mon Jul 29 19:28:40 2013 From: anthonym@REDACTED (ANTHONY MOLINARO) Date: Mon, 29 Jul 2013 10:28:40 -0700 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <51F67B5B.4020503@power.com.pl> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> Message-ID: Also, slightly off topic, but not totally, is a serialization system I wrote about 15 years ago and which has been in continuous production since (it logged all traffic Goto.com/Overture.com/Yahoo did, as well as traffic at AT&T and OpenX, so I'd guess somewhere in the 100s of trillions of messages sent). http://www.lwes.org/ Unlike some of the others mentioned (but similar to ASN.1 IIRC), is that it is self describing, so doesn't require compiling bindings. It's original purpose was to pass dictionaries between language's (so send a Java HashMap receive a perl hash, or an erlang dict). The values in the dictionaries have evolved somewhat, but we've kept things backwards compatible. It's serialization format is length and type prefixed, so doesn't suffer from the issues Richard pointed out in tnetstrings, however since it's meant to send somewhat smallish bits of data over UDP there's not usually a memory concern. Anyway, just throwing it out there as its worked well for particular problems for many years for me. -Anthony On Jul 29, 2013, at 7:25 AM, Wojtek Narczy?ski wrote: > W dniu 2013-07-26 11:20, Ulf Wiger pisze: >> At Feuerlabs, we use (among other things) BERT [1] and >> Yang [2] (RFC 6020) to accomplish RPCs with contract checking. >> >> In addition, our BERT implementation has access control lists and >> is pretty flexible about transport and authentication (with the option >> of an authentication plugin), and our Yang compiler (developed in >> cooperation with Tail-f) can generate Erlang code from a spec and >> has a facility to validate a JSON-style RPC against the 'contract'. >> >> Documentation is a bit sparse (hrm?) but our support staff can be very helpful. >> >> [1] https://github.com/Feuerlabs/bert >> [2] https://github.com/Feuerlabs/yang >> >> > It makes me wonder, what is wrong with ASN.1? > > I have never used it, but on paper it looks superior to any more recent schema description language. > > Sorry for being off-subject. > > --Kind regards, > Wojtek Narczynski > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From i@REDACTED Mon Jul 29 19:54:07 2013 From: i@REDACTED (Anthony Grimes) Date: Mon, 29 Jul 2013 10:54:07 -0700 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: References: <1375089565068.a42008e@Nodemailer> Message-ID: <51F6AC3F.4090108@raynes.me> Right, there are certainly ways around it, but the point of this discussion is that it seems to be a missing piece of functionality that Erlangers cannot do unless they write C code. It seems like ports are 90% of the way there. I think Mr. Hedeland represented my points nicely. -Anthony > Robert Raschke > July 29, 2013 4:16 AM > Hi Anthony, > > could Py-interface http://www.lysator.liu.se/~tab/erlang/py_interface/ > help with that? > > Note, I've not used it, so don't know much at all beyond reading that > page. It would probably mean using the Pygments library, rather than > going via the command line pygmentize. You can probably take the > pygmentize source and replace the command line handling with setting > up a Python Node. > > Alternatively, and not necessarily very nice, instead of streaming to > pygmentize, write to a file and invoke on that. You wouldn't even need > a port then, you could get away with os:cmd/1 (if you aren't > interested in return codes). But you already know this, I think. > > Robby > > PS Something like expect for Erlang ports would be pretty cool, > though. Not that I'm not volunteering ;-) > > > > > Anthony Grimes > July 29, 2013 2:19 AM > Hey Robert. > > I don't think C nodes help in this case, and they don't solve the > general problem. One of my use cases is talking to pygmentize, which > is a Python program. If I want to do that, I have to write a middleman > program that does the actual communication with this program and talks > to my Erlang program over a socket, or via ports if I make the > middleman program look for some specific sequence of bytes to treat as > EOF since I can't send actual EOF. > > > > Robert Raschke > July 29, 2013 2:12 AM > > Hi Anthony, > > In the past, I've tended to use the port mechanism to simply kick off > a C node, which then allows you to have full control over whatever > communications needs you have. > > This obviously only works if you are interfacing with a technology > that will allow you to create C node and use the EI libs in some way. > Not sure if that is the case from what you wrote. > > Regards, > Robby > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1188 bytes Desc: not available URL: From fredrik@REDACTED Mon Jul 29 23:43:22 2013 From: fredrik@REDACTED (Fredrik Linder) Date: Mon, 29 Jul 2013 14:43:22 -0700 Subject: [erlang-questions] Segfault in NIF call In-Reply-To: References: <3da9e2e4-e1fc-4187-9573-faed5e1c249d@googlegroups.com> <517E4441.2040202@erix.ericsson.se> Message-ID: I just realized I never got back on this. Thank you for your help :-) I ended up re-writing it using a TDD approach, only took a few hours to get everything working again /Fredrik On Tue, Apr 30, 2013 at 6:13 AM, Greg Burd wrote: > Are you trying to build a NIF where the function is run on a > non-Erlang scheduler thread (async, via some thread pool)? If so, > here's a method for doing it. > > https://gist.github.com/gburd/4121795 > > It took a while to get this just right, reuse and enjoy. I'll update > the nifwait (https://github.com/basho/nifwait) repo with example code. > The best example of how to use it is in the wterl > (https://github.com/basho-labs/wterl) repo. > > -greg > > @gregburd | Basho Technologies | Riak | http://basho.com | @basho > > > On Mon, Apr 29, 2013 at 2:57 PM, Fredrik Linder > wrote: > > Found one error (yay!) -- enif_open_resource_type should have NULL as the > > module argument (missed this one in the docs) > > > > Running with cerl does not generate a segfault, nor any (extra) printouts > > Running with erl still generate segfault > > > > Now I get either of the following (using erl), does it reveal what my > error > > is? > > > >> USE_GDB=1 rebar eunit > > [cut] > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 0x2aaaad6c0700 (LWP 6384)] > > 0x000000000052b28a in process_main () at > > x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:979 > > 979 MoveDeallocateReturn(xb(tmp_packed1&BEAM_LOOSE_MASK), r(0), > > Qb((tmp_packed1>>BEAM_LOOSE_SHIFT))); > > (gdb) backtrace > > #0 0x000000000052b28a in process_main () at > > x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:979 > > #1 0x0000000000491463 in sched_thread_func (vesdp=0x2aaaac343ac0) at > > beam/erl_process.c:5632 > > #2 0x0000000000590440 in thr_wrapper (vtwd=0x7fffffffd850) at > > pthread/ethread.c:106 > > #3 0x00002aaaab3ffe9a in start_thread (arg=0x2aaaad6c0700) at > > pthread_create.c:308 > > #4 0x00002aaaab911cbd in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > > #5 0x0000000000000000 in ?? () > > > > or > > > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 0x2aaaad6c0700 (LWP 6914)] > > unlink_free_block (allctr=0x85e5c0, block=0x0, flags=0) at > > beam/erl_goodfit_alloc.c:458 > > 458 Uint sz = MBC_FBLK_SZ(&blk->block_head); > > (gdb) backtrace > > #0 unlink_free_block (allctr=0x85e5c0, block=0x0, flags=0) at > > beam/erl_goodfit_alloc.c:458 > > #1 0x0000000000444ee1 in get_free_block (allctr=0x85e5c0, > size= > out>, cand_blk=0x0, cand_size=0, flags=0) at beam/erl_goodfit_alloc.c:426 > > #2 0x000000000043b6da in mbc_alloc_block (alcu_flgsp= pointer>, > > blk_szp=, size=, allctr=0x85e5c0) at > > beam/erl_alloc_util.c:1309 > > #3 mbc_alloc (allctr=0x85e5c0, size=) at > > beam/erl_alloc_util.c:1451 > > #4 0x0000000000440d7b in do_erts_alcu_alloc (size=32, extra=0x85e5c0, > > type=148) at beam/erl_alloc_util.c:3530 > > #5 erts_alcu_alloc_thr_pref (type=148, extra=, > > size=) at beam/erl_alloc_util.c:3607 > > #6 0x0000000000490595 in erts_alloc (size=32, type=18967) at > > beam/erl_alloc.h:208 > > #7 new_message_buffer (size=0) at beam/erl_message.c:72 > > #8 erts_alloc_message_heap_state (statep=0x2aaaad6bfc4c, > > receiver_locks=0x2aaaad6bfcb0, receiver=0x2aaaac9c1db8, ohpp= > pointer>, bpp=, size=0) at beam/global.h:1017 > > #9 erts_send_message (sender=0x2aaaac9c2470, receiver=0x2aaaac9c1db8, > > receiver_locks=0x2aaaad6bfcb0, message=564171, flags=) at > > beam/erl_message.c:1039 > > #10 0x0000000000476350 in do_send (p=0x2aaaac9c2470, to=793099, > msg=564171, > > suspend=1, refp=) at beam/bif.c:2025 > > #11 0x0000000000476bf0 in erl_send (p=0x2aaaac9c2470, to=793099, > msg=564171) > > at beam/bif.c:2138 > > #12 0x000000000052d1d8 in process_main () at beam/beam_emu.c:2558 > > #13 0x0000000000491463 in sched_thread_func (vesdp=0x2aaaac343ac0) at > > beam/erl_process.c:5632 > > #14 0x0000000000590440 in thr_wrapper (vtwd=0x7fffffffd850) at > > pthread/ethread.c:106 > > #15 0x00002aaaab3ffe9a in start_thread (arg=0x2aaaad6c0700) at > > pthread_create.c:308 > > #16 0x00002aaaab911cbd in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > > #17 0x0000000000000000 in ?? () > > > > > > /Fredrik > > > > > > > > > > > > > > On Mon, Apr 29, 2013 at 2:58 AM, Sverker Eriksson > > wrote: > >> > >> Looks like some sort of memory corruption. > >> > >> Run on debug emulator and hope for a better (earlier) crash. > >> > >> # cd $ERL_TOP/erts/emulator > >> # make TYPE=debug smp plain > >> # $ERL_TOP/bin/cerl -debug > >> > >> /Sverker, Erlang/OTP Ericsson > >> > >> fredrik@REDACTED wrote: > >>> > >>> Hello folks, > >>> > >>> I'm having difficulties locating the cause of a segfault I'm getting > when > >>> running tests with a NIF implementation I have. > >>> > >>> Anything that would shed light on what's wrong is appreciated. > >>> > >>> The backtrace does not seem to have anything to do with my NIF. The > >>> NIF:ed function is called from a few different processes, and it > forwards > >>> the call to a worker thread which sends a message back to the caller > >>> process. > >>> > >>> > >>>> > >>>> strings -a > >>> > >>> > >>> > ../../otp/R16B/pre-5.10.1-mz-0.2/lib/erlang/erts-5.10.1/bin/beam.smp|fgrep > >>> GCC|sort -u > >>> GCC: (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 > >>> > >>> > >>>> > >>>> USE_GDB=1 rebar eunit > >>>> > >>> > >>> [cut] > >>> Program received signal SIGSEGV, Segmentation fault. > >>> [Switching to Thread 0x2aaaef381700 (LWP 19162)] > >>> unlink_free_block (allctr=0x85f240, block=0x0, flags=0) at > >>> beam/erl_goodfit_alloc.c:458 > >>> 458 Uint sz = MBC_FBLK_SZ(&blk->block_head); > >>> (gdb) backtrace > >>> #0 unlink_free_block (allctr=0x85f240, block=0x0, flags=0) at > >>> beam/erl_goodfit_alloc.c:458 > >>> #1 0x0000000000444ee1 in get_free_block (allctr=0x85f240, > >>> size=, cand_blk=0x0, cand_size=0, flags=0) at > >>> beam/erl_goodfit_alloc.c:426 > >>> #2 0x000000000043b6da in mbc_alloc_block (alcu_flgsp= >>> pointer>, blk_szp=, size=, > >>> allctr=0x85f240) at beam/erl_alloc_util.c:1309 > >>> #3 mbc_alloc (allctr=0x85f240, size=) at > >>> beam/erl_alloc_util.c:1451 > >>> #4 0x0000000000440d7b in do_erts_alcu_alloc (size=32, extra=0x85f240, > >>> type=148) at beam/erl_alloc_util.c:3530 > >>> #5 erts_alcu_alloc_thr_pref (type=148, extra=, > >>> size=) at beam/erl_alloc_util.c:3607 > >>> #6 0x0000000000490595 in erts_alloc (size=32, type=18967) at > >>> beam/erl_alloc.h:208 > >>> #7 new_message_buffer (size=0) at beam/erl_message.c:72 > >>> #8 erts_alloc_message_heap_state (statep=0x2aaaef380c4c, > >>> receiver_locks=0x2aaaef380cb0, receiver=0x2aaaac9ccac8, ohpp= >>> pointer>, bpp=, size=0) at beam/global.h:1017 > >>> #9 erts_send_message (sender=0x2aaaac9cd118, receiver=0x2aaaac9ccac8, > >>> receiver_locks=0x2aaaef380cb0, message=513931, flags=) > at > >>> beam/erl_message.c:1039 > >>> #10 0x0000000000476350 in do_send (p=0x2aaaac9cd118, to=550027, > >>> msg=513931, suspend=1, refp=) at beam/bif.c:2025 > >>> #11 0x0000000000476bf0 in erl_send (p=0x2aaaac9cd118, to=550027, > >>> msg=513931) at beam/bif.c:2138 > >>> #12 0x000000000052d1d8 in process_main () at beam/beam_emu.c:2558 > >>> #13 0x0000000000491463 in sched_thread_func (vesdp=0x2aaaac34de40) at > >>> beam/erl_process.c:5632 > >>> #14 0x0000000000590440 in thr_wrapper (vtwd=0x7fffffffd7e0) at > >>> pthread/ethread.c:106 > >>> #15 0x00002aaaab3ffe9a in start_thread (arg=0x2aaaef381700) at > >>> pthread_create.c:308 > >>> #16 0x00002aaaab911cbd in clone () at > >>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > >>> #17 0x0000000000000000 in ?? () > >>> (gdb) > >>> > >>> > >>> > ------------------------------------------------------------------------ > >>> > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >> > >> > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -- http://whosawesome.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Jul 29 23:52:12 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 30 Jul 2013 09:52:12 +1200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> Message-ID: On 29/07/2013, at 11:55 PM, Tony Rogvall wrote: > I am also pretty pleased with cson :-) > Looks very nice. > > A small question. > How are strings encoded? > > a) " > b) "* > c) "* > d) Other? Well, I was doing this in my Smalltalk system, where Strings are mutable arrays of Unicode, and where byte streams support a #nextUtf8 method. So having read a and noticed the " the next step is str := String new: n. 1 to: n do: [:i | str at: i put: stream nextUtf8]. ^str So it's (c). From tony@REDACTED Tue Jul 30 00:24:06 2013 From: tony@REDACTED (Tony Rogvall) Date: Tue, 30 Jul 2013 00:24:06 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> <6BF2D2AF-7DBE-4666-85F5-3B7FFDD227FC@cs.otago.ac.nz> Message-ID: <94A09A23-5CD2-4219-B91F-3A02699DBF19@rogvall.se> On 29 jul 2013, at 23:52, Richard A. O'Keefe wrote: > > On 29/07/2013, at 11:55 PM, Tony Rogvall wrote: > >> I am also pretty pleased with cson :-) >> Looks very nice. >> >> A small question. >> How are strings encoded? >> >> a) " >> b) "* >> c) "* >> d) Other? > > Well, I was doing this in my Smalltalk system, where Strings > are mutable arrays of Unicode, and where byte streams support > a #nextUtf8 method. So having read a chars> and noticed the " the next step is > > str := String new: n. > 1 to: n do: [:i | str at: i put: stream nextUtf8]. > ^str > > So it's (c). > Thanks. That was what I was afraid of. Luckily I remembered that there is now support for utf8 in Erlangs binary syntax: decode_string(0, Cs, Acc) -> {lists:reverse(Acc), Cs}; decode_string(I, <>, Acc) -> decode_string(I-1,Cs,[Char|Acc]). So all is well. /Tony From barcojie@REDACTED Tue Jul 30 04:53:49 2013 From: barcojie@REDACTED (Barco You) Date: Tue, 30 Jul 2013 10:53:49 +0800 Subject: [erlang-questions] Can not start many children with simple_one_for_one Message-ID: Dear Forks, I have a supervisor who will start children with simple_one_for_one strategy. First of all I have server process who starts the mentioned supervisor by using {ok, Pid} = supervisor:start_child(Supsup, ?SPEC), and then I add the children under the mentioned supervisor one by one also with supervisor:start_child(Pid, Args), but only the first child can be started and the rest children just got "process already started" error message. What's wrong I made? the snippets of the codes as following: -module(spider_sup). -behaviour(supervisor). -export([start_link/0]). %% API. -export([init/1]). %% supervisor. -define(SUPERVISOR, ?MODULE). %% API. start_link() -> supervisor:start_link({local, ?SUPERVISOR}, ?MODULE, []). %% supervisor. init([]) -> MaxRestart = 5, MaxTime = 60, {ok, {{simple_one_for_one, MaxRestart, MaxTime}, [{spider, {spider, start_link, []}, permanent, 120, worker, [spider]}]}}. ---------------------------------------------------------------------------------------------------------- -module(spider). -behavior(gen_fsm). -export([start_link/1]). -export([init/1, event1/2, handle_event/3, handle_sync_event/4, code_change/4, terminate/3, handle_info/3]). start_link(Code) -> gen_fsm:start_link({local, ?MODULE}, ?MODULE, Code, []). init(Code) -> io:format("~p Starting~n",[Code]), {ok, event1, Code, 10000}. --------------------------------------------------------------------------------------------------- %snippet from the server process start_spiders(_Pid, []) -> ok; start_spiders(Pid, [HCode | TCode]) -> supervisor:start_child(Pid, [HCode]), start_spiders(Pid, TCode). -------------- next part -------------- An HTML attachment was scrubbed... URL: From james@REDACTED Tue Jul 30 05:00:31 2013 From: james@REDACTED (James Aimonetti) Date: Mon, 29 Jul 2013 20:00:31 -0700 Subject: [erlang-questions] Can not start many children with simple_one_for_one In-Reply-To: References: Message-ID: <51F72C4F.8030605@2600hz.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 07/29/2013 07:53 PM, Barco You wrote: > Dear Forks, > > I have a supervisor who will start children with > simple_one_for_one strategy. First of all I have server process who > starts the mentioned supervisor by using {ok, Pid} = > supervisor:start_child(Supsup, ?SPEC), and then I add the children > under the mentioned supervisor one by one also with > supervisor:start_child(Pid, Args), but only the first child can be > started and the rest children just got "process already started" > error message. What's wrong I made? the snippets of the codes as > following: > > -module(spider_sup). -behaviour(supervisor). > > -export([start_link/0]). %% API. -export([init/1]). %% supervisor. > > -define(SUPERVISOR, ?MODULE). > > %% API. > > start_link() -> supervisor:start_link({local, ?SUPERVISOR}, > ?MODULE, []). > > %% supervisor. > > init([]) -> MaxRestart = 5, MaxTime = 60, {ok, > {{simple_one_for_one, MaxRestart, MaxTime}, [{spider, {spider, > start_link, []}, permanent, 120, worker, [spider]}]}}. > ---------------------------------------------------------------------------------------------------------- > > - -module(spider). > > -behavior(gen_fsm). > > -export([start_link/1]). > > -export([init/1, event1/2, handle_event/3, handle_sync_event/4, > code_change/4, terminate/3, handle_info/3]). > > start_link(Code) -> gen_fsm:start_link({local, ?MODULE}, ?MODULE, > Code, []). > > init(Code) -> io:format("~p Starting~n",[Code]), {ok, event1, Code, > 10000}. > --------------------------------------------------------------------------------------------------- > > %snippet from the server process > start_spiders(_Pid, []) -> ok; start_spiders(Pid, [HCode | TCode]) > -> supervisor:start_child(Pid, [HCode]), start_spiders(Pid, > TCode). > > > > _______________________________________________ erlang-questions > mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > It appears your gen_fsm is registering itself as a named process. So the first succeeds because the name isn't registered; subsequent attempts will not start additional children as the name is no longer available. Try using gen_fsm:start_link/3 [0] instead. [0] http://erldocs.com/R15B/stdlib/gen_fsm.html?i=2&search=gen_fsm%20start#start_link/3 - -- James Aimonetti Distributed Systems Engineer / DJ MC_ 2600hz | http://2600hz.com sip:james@REDACTED tel: 415.886.7905 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlH3LE8ACgkQ54NxaUq7OmBHqwD/Vu+If3oUpqgmHlooOW6ZwZ3z 34YJKrncaBJDOxRINloBAIx25ZL3migY3y29jZ8hS4Oho4NY1n4uwjDCe8QFeu8k =u5w5 -----END PGP SIGNATURE----- From barcojie@REDACTED Tue Jul 30 05:26:13 2013 From: barcojie@REDACTED (Barco You) Date: Tue, 30 Jul 2013 11:26:13 +0800 Subject: [erlang-questions] Can not start many children with simple_one_for_one In-Reply-To: <51F72C4F.8030605@2600hz.com> References: <51F72C4F.8030605@2600hz.com> Message-ID: Got it! Thank you very much! On Tue, Jul 30, 2013 at 11:00 AM, James Aimonetti wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 07/29/2013 07:53 PM, Barco You wrote: > > Dear Forks, > > > > I have a supervisor who will start children with > > simple_one_for_one strategy. First of all I have server process who > > starts the mentioned supervisor by using {ok, Pid} = > > supervisor:start_child(Supsup, ?SPEC), and then I add the children > > under the mentioned supervisor one by one also with > > supervisor:start_child(Pid, Args), but only the first child can be > > started and the rest children just got "process already started" > > error message. What's wrong I made? the snippets of the codes as > > following: > > > > -module(spider_sup). -behaviour(supervisor). > > > > -export([start_link/0]). %% API. -export([init/1]). %% supervisor. > > > > -define(SUPERVISOR, ?MODULE). > > > > %% API. > > > > start_link() -> supervisor:start_link({local, ?SUPERVISOR}, > > ?MODULE, []). > > > > %% supervisor. > > > > init([]) -> MaxRestart = 5, MaxTime = 60, {ok, > > {{simple_one_for_one, MaxRestart, MaxTime}, [{spider, {spider, > > start_link, []}, permanent, 120, worker, [spider]}]}}. > > > ---------------------------------------------------------------------------------------------------------- > > > > > - -module(spider). > > > > -behavior(gen_fsm). > > > > -export([start_link/1]). > > > > -export([init/1, event1/2, handle_event/3, handle_sync_event/4, > > code_change/4, terminate/3, handle_info/3]). > > > > start_link(Code) -> gen_fsm:start_link({local, ?MODULE}, ?MODULE, > > Code, []). > > > > init(Code) -> io:format("~p Starting~n",[Code]), {ok, event1, Code, > > 10000}. > > > --------------------------------------------------------------------------------------------------- > > > > > %snippet from the server process > > start_spiders(_Pid, []) -> ok; start_spiders(Pid, [HCode | TCode]) > > -> supervisor:start_child(Pid, [HCode]), start_spiders(Pid, > > TCode). > > > > > > > > _______________________________________________ erlang-questions > > mailing list erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > It appears your gen_fsm is registering itself as a named process. So > the first succeeds because the name isn't registered; subsequent > attempts will not start additional children as the name is no longer > available. Try using gen_fsm:start_link/3 [0] instead. > > [0] > > http://erldocs.com/R15B/stdlib/gen_fsm.html?i=2&search=gen_fsm%20start#start_link/3 > > - -- > James Aimonetti > Distributed Systems Engineer / DJ MC_ > > 2600hz | http://2600hz.com > sip:james@REDACTED > tel: 415.886.7905 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iF4EAREIAAYFAlH3LE8ACgkQ54NxaUq7OmBHqwD/Vu+If3oUpqgmHlooOW6ZwZ3z > 34YJKrncaBJDOxRINloBAIx25ZL3migY3y29jZ8hS4Oho4NY1n4uwjDCe8QFeu8k > =u5w5 > -----END PGP SIGNATURE----- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Tue Jul 30 06:44:29 2013 From: barcojie@REDACTED (Barco You) Date: Tue, 30 Jul 2013 12:44:29 +0800 Subject: [erlang-questions] {"init terminating in do_boot", {'cannot load', error_handler, get_files}} Message-ID: Hi, I've finished making an application and can successfully start it from erl shell by using application:start(myapp). But, after I use rebar to release it I got a error as the subject when I start the App from the rel/myapp/bin/myapp. Any clues for that? Thank you! Barco -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Tue Jul 30 06:45:11 2013 From: barcojie@REDACTED (Barco You) Date: Tue, 30 Jul 2013 12:45:11 +0800 Subject: [erlang-questions] {"init terminating in do_boot", {'cannot load', error_handler, get_files}} In-Reply-To: References: Message-ID: The error like this: {"init terminating in do_boot",{'cannot load',error_handler,get_files}} Crash dump was written to: erl_crash.dump init terminating in do_boot () On Tue, Jul 30, 2013 at 12:44 PM, Barco You wrote: > Hi, > > I've finished making an application and can successfully start it from erl > shell by using application:start(myapp). But, after I use rebar to release > it I got a error as the subject when I start the App from the > rel/myapp/bin/myapp. Any clues for that? > > Thank you! > Barco > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Jul 30 07:29:53 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 30 Jul 2013 08:29:53 +0300 Subject: [erlang-questions] {"init terminating in do_boot", {'cannot load', error_handler, get_files}} In-Reply-To: References: Message-ID: <637BFB44-2AAF-435D-8519-8116A390DEE1@gmail.com> Hello, It looks like some component is missing from release. Hard to say, reltool.config and apps deps are needed. Try to check that all application deps are part of release Best Regards, Dmitry >-|-|-*> On 30.7.2013, at 7.45, Barco You wrote: > The error like this: > > {"init terminating in do_boot",{'cannot load',error_handler,get_files}} > > Crash dump was written to: erl_crash.dump > init terminating in do_boot () > > > > On Tue, Jul 30, 2013 at 12:44 PM, Barco You wrote: >> Hi, >> >> I've finished making an application and can successfully start it from erl shell by using application:start(myapp). But, after I use rebar to release it I got a error as the subject when I start the App from the rel/myapp/bin/myapp. Any clues for that? >> >> Thank you! >> Barco > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Jul 30 07:44:34 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 30 Jul 2013 17:44:34 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F625B9.5040506@raynes.me> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> Message-ID: <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> On 29/07/2013, at 8:20 PM, Anthony Grimes wrote: > Yeah, re-reading your post a couple of times, I think we might be on the wrong page or something. Here is a low level example of what I'd like to be able to do in Erlang. This is a Clojure repl session where I interact with the 'cat' program via the built in Java Process library: > > user=> (def proc (.exec (Runtime/getRuntime) (into-array String ["cat"]))) > #'user/proc > user=> (def stdin (.getOutputStream proc)) > #'user/stdin > user=> (def stdout (.getInputStream proc)) I have some trouble reading Clojure. I don't know what the dots are. Hazarding a guess, This is *PRECISELY* the "Hello, deadlock!" kind of buggy stuff that the C interface was designed to *not* let you write. > Lots of unix programs work like this. > We have cat in this example, but grep, wc, and various others work like that as well. It is this easy or easier to do the same thing in every other language I can think of. Actually, NO. You are talking about "filters" here, and filters are designed to be connected into ***ACYCLIC*** networks. > If it's fundamentally a bad thing, I'm surprised these programs work like that in the first place and that these languages support this. The programs do NOT work the way you think they do. A filter reads from its standard input. It writes to its standard output. If it could have emotions, it would view the prospect of those two being the *same* thing with shuddering dread. (Except of course, when the thing is the terminal. The user is assumed to be capable of infinite buffering.) Erlang is perfectly happy to be connected to an ACYCLIC network of pipe-linked processes too. > It seems to be an entirely common place, basic feature any remotely high level programming language. Actually, no. The ability to connect to the standard input *AND* the standard output of the *same* process is *not* a commonplace feature of high level programming languages (some do, some don't) because unless you code with extreme (and to a certain extent, non-portable) care, you end up in deadlock land. Only if one of the programs is absolutely guaranteed to write a tiny amount of information -- at most one PIPE_BUF worth, do you have any shadow of a trace of a right to expect it to work. If you don't believe me, believe the Java documentation, where the page for java.lang.Process says All [the new process's] standard io (i.e. stdin, stdout, stderr) operations will be redirected to the parent process through three streams (getOutputStream(), getInputStream(), getErrorStream()). The parent process uses these streams to feed input to and get output from the subprocess. >>>>>> Because some native platforms only provide limited buffer size >>>>>> for standard input and output streams, failure to promptly >>>>>> write the input stream or read the output stream of the >>>>>> subprocess may cause the subprocess to block, and even deadlock. The POSIX guarantee for PIPE_BUF is just 512 bytes. That is, should the parent process write 513 bytes to the child, and the child write 513 bytes to the parent, hello deadlock! Like I said, connecting to *both* ends of a command through pipes is something to anticipate with shuddering dread. It is *not* a standard feature to be used lightly. I can't find anything about external processes in the Haskell 2010 report. System.Process http://www.haskell.org/ghc/docs/7.4-latest/html/libraries/process-1.1.0.1/System-Process.html isn't mentioned in Haskell 2010. I am actually pretty shocked that the documentation doesn't mention the deadlock problem. From ok@REDACTED Tue Jul 30 07:55:41 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 30 Jul 2013 17:55:41 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <201307291413.r6TEDFaX063329@pluto.hedeland.org> References: <201307291413.r6TEDFaX063329@pluto.hedeland.org> Message-ID: On 30/07/2013, at 2:13 AM, Per Hedeland wrote: > > FWIW, I definitely agree that this is a missing piece of functionality. Oh it's present in several languages. You could argue that it's missing. But is it missing like a car radio in a car that lacks one, or is it missing like an ejector seat in a car that lacks one? >> Note that "only send data to a command" and "only receive data from a >> command" are the traditional ways for a UNIX program to communicate >> with another over a pipe. > > Well, it's basically the definition of the traditional pipeline concept > of the Unix shells, and pipes are obviously what you need to implement > it - but that doesn't preclude other uses of pipes. The zsh shell even > allows you to set up "bidirectional pipes" on the commandline. There are reasons why I don't use zsh. That's one of them. > >> popen(, "r") reads the output of >> the command and popen(, "w") writes to the input of the command. > > popen() is effectively a convenience function to abstract away the > somewhat non-trivial application of pipe(), fork(), close(), and > execve() that is required to set things up correctly for two particular > and common usages of pipes in application code. (It is not used by > common shells to implement pipelines though.) Having implemented popen() for two other high level languages, I know that. The fact that common shells do not use it is irrelevant. > >> There isn't even any standard _term_ for talking about connecting to both >> stdin and stdout of a command in UNIX, and that's because it's an >> incredibly easy way to deadlock. > > There is no need to have a term for it, since all you need is two pipes, > one for each direction Non-sequitur. If it's a thing you need to do often, it's a thing you need to be able to talk about. When I've thought of doing it, I've used the word David Bacon used in his SETL2 system, "pump". And then I've used the phrase "looming disaster" and done something else. > - and it's probably uncommon enough to not > warrant its own convenience function. And you can indeed easily deadlock > if you don't think about what you're doing, but I really doubt that this > is the reason for any absence of terminology or functions. It's not that you can easily deadlock, it's that it's hard *NOT* to deadlock. > (the risk is of course reduced due to the fact that the > VM does non-blocking I/O). And *that* is the thing that saves Erlang. Of course, avoiding the coding complexity of dealing with nonblocking I/O is one of the reasons for using a multithreading language like Erlang. >> >> Just like it prevents C users from doing the same thing. > > No, there is nothing that prevents C users from doing the same thing. You may have misunderstood me. >>>> THE POPEN INTERFACE <<<< prevents C users doing this. Yes, all the other functions are there, and yes, if you desperately want to program it, you can. But it is enough work that nobody ever does this lightly. For that matter, it's not beyond the abilities of, say, the glibc authors, to extend their implementation of popen to support "r+" or "w+" modes, if there were much demand for it. (Oddly enough, Mac OS X 10.7.5 _does_ support "r+" mode, but the Linux I checked does not. Weird.) I have _never_ been able to understand the differences between Mac OS X and POSIX. From ok@REDACTED Tue Jul 30 07:58:10 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 30 Jul 2013 17:58:10 +1200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <51F67B5B.4020503@power.com.pl> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> Message-ID: <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> On 30/07/2013, at 2:25 AM, Wojtek Narczy?ski wrote: > It makes me wonder, what is wrong with ASN.1? The original poster explicitly asked for a >> schema-less << protocol. ASN.1 is very much schema-based. There are good reasons why Erlang does support ASN.1. From i@REDACTED Tue Jul 30 08:02:37 2013 From: i@REDACTED (Anthony Grimes) Date: Mon, 29 Jul 2013 23:02:37 -0700 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> Message-ID: <51F756FD.1090209@raynes.me> I apologize for getting terminology wrong, I guess. I've never heard these terms before in reference to working with external processes. I'm still not really getting it though. The problem here is that Erlang ports let me read from stdout and write to stdin, but not let me flush stdin and tell the program that I'm done writing, which is important for certain programs that wait for this EOF to start producing output. What am I saying that's wrong? I'm not saying these are the same things. In fact, I'm not trying to make any assertions about how unix processes and pipes works, and if it sounds like that then I apologize and retract any assertions I've made. I'm simply asking how I'm supposed to deal with programs that I have *no* control over that wait for a EOF on their input to start producing data. I just want to send a ^D.I have examples of programs that do that and a thousand ways to do it in other languages, and I'm being told it is disastrous to do. If the solution is to just keep doing what everybody has been doing in Erlang for years which is write giant hack middleman programs to do it, then I'll concede defeat. I've got to say though, I'm pretty blown away by your response to this. Anyways, thanks for the responses everyone! > Richard A. O'Keefe > July 29, 2013 10:44 PM > On 29/07/2013, at 8:20 PM, Anthony Grimes wrote: > >> Yeah, re-reading your post a couple of times, I think we might be on the wrong page or something. Here is a low level example of what I'd like to be able to do in Erlang. This is a Clojure repl session where I interact with the 'cat' program via the built in Java Process library: >> >> user=> (def proc (.exec (Runtime/getRuntime) (into-array String ["cat"]))) >> #'user/proc >> user=> (def stdin (.getOutputStream proc)) >> #'user/stdin >> user=> (def stdout (.getInputStream proc)) > > I have some trouble reading Clojure. I don't know what the dots are. > Hazarding a guess, > > This is *PRECISELY* the "Hello, deadlock!" > kind of buggy stuff that the C interface was designed > to *not* let you write. > >> Lots of unix programs work like this. >> We have cat in this example, but grep, wc, and various others work like that as well. It is this easy or easier to do the same thing in every other language I can think of. > > Actually, NO. You are talking about "filters" here, > and filters are designed to be connected into ***ACYCLIC*** networks. > >> If it's fundamentally a bad thing, I'm surprised these programs work like that in the first place and that these languages support this. > > The programs do NOT work the way you think they do. > A filter reads from its standard input. > It writes to its standard output. > If it could have emotions, it would view the prospect > of those two being the *same* thing with shuddering dread. > (Except of course, when the thing is the terminal. The > user is assumed to be capable of infinite buffering.) > > Erlang is perfectly happy to be connected to an ACYCLIC network > of pipe-linked processes too. > >> It seems to be an entirely common place, basic feature any remotely high level programming language. > > Actually, no. The ability to connect to the standard input *AND* the standard > output of the *same* process is *not* a commonplace feature of high level > programming languages (some do, some don't) because unless you code with > extreme (and to a certain extent, non-portable) care, you end up in deadlock land. > > Only if one of the programs is absolutely guaranteed to write a tiny > amount of information -- at most one PIPE_BUF worth, do you have > any shadow of a trace of a right to expect it to work. > > If you don't believe me, believe the Java documentation, > where the page for java.lang.Process says > > All [the new process's] standard io (i.e. stdin, stdout, stderr) > operations will be redirected to the parent process through > three streams (getOutputStream(), getInputStream(), > getErrorStream()). The parent process uses these streams to feed input to and get output from the subprocess. >>>>>>> Because some native platforms only provide limited buffer size >>>>>>> for standard input and output streams, failure to promptly >>>>>>> write the input stream or read the output stream of the >>>>>>> subprocess may cause the subprocess to block, and even deadlock. > > The POSIX guarantee for PIPE_BUF is just 512 bytes. > That is, should the parent process write 513 bytes to the child, > and the child write 513 bytes to the parent, > hello deadlock! > > Like I said, connecting to *both* ends of a command through pipes > is something to anticipate with shuddering dread. It is *not* a > standard feature to be used lightly. > > I can't find anything about external processes in the Haskell 2010 > report. System.Process > http://www.haskell.org/ghc/docs/7.4-latest/html/libraries/process-1.1.0.1/System-Process.html > isn't mentioned in Haskell 2010. I am actually pretty shocked that > the documentation doesn't mention the deadlock problem. > > > > Anthony Grimes > July 29, 2013 1:20 AM > Yeah, re-reading your post a couple of times, I think we might be on > the wrong page or something. Here is a low level example of what I'd > like to be able to do in Erlang. This is a Clojure repl session where > I interact with the 'cat' program via the built in Java Process library: > > user=> (def proc (.exec (Runtime/getRuntime) (into-array String ["cat"]))) > #'user/proc > user=> (def stdin (.getOutputStream proc)) > #'user/stdin > user=> (def stdout (.getInputStream proc)) > #'user/stdout > user=> (.write stdin (.getBytes "Hi!")) > nil > user=> (.close stdin) > nil > user=> (let [arr (byte-array 3)] (.read stdout arr) (String. arr)) > "Hi!" > > Lots of unix programs work like this. We have cat in this example, but > grep, wc, and various others work like that as well. It is this easy > or easier to do the same thing in every other language I can think of. > If it's fundamentally a bad thing, I'm surprised these programs work > like that in the first place and that these languages support this. It > seems to be an entirely common place, basic feature any remotely high > level programming language. > > Perhaps this example and clarification will clear things up! > > -Anthony > Richard A. O'Keefe > July 29, 2013 12:15 AM > On 29/07/2013, at 4:14 PM, Anthony Grimes wrote: > >> and communicating with external processes in Erlang. They seem to have >> at least one particular fatal flaw which prevents them from being very >> useful to me, and that is that there is no way to close stdin (and send >> EOF) and then also read from the process's stdout. For example, I cannot >> use a port to start the 'cat' program which listens on stdin for data >> and waits for EOF and then echos that data back to you. I can do the >> first part, which is send it data on stdin, but the only way for me to >> close it is to call port_close and close the entire process. > > Note that "only send data to a command" and "only receive data from a > command" are the traditional ways for a UNIX program to communicate > with another over a pipe. popen(, "r") reads the output of > the command and popen(, "w") writes to the input of the command. > There isn't even any standard _term_ for talking about connecting to both > stdin and stdout of a command in UNIX, and that's because it's an > incredibly easy way to deadlock. > >> This issue prevents Erlang users from doing any even slightly more than >> trivial communication with external processes without having some kind >> of middleman program that handles the creation of the actual process you >> need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > Just like it prevents C users from doing the same thing. > Unless they fake something up using named pipes or UNIX-domain sockets. > (Or message queues. I do wish Mac OS X implemented rather more of POSIX...) > > Unix anonymous pipes are simply the wrong tool for the job in _any_ > programming language. > > The historic way to do "slightly more than trivial communication with > external processes" has been to set the external processes up as C nodes > or to use sockets. > > > > > Anthony Grimes > July 28, 2013 9:14 PM > Howdy folks. > > I unfortunately have not been able to use Erlang for most of what I've > been doing lately because of a long standing issue with Erlang ports > that I'd like to start a discussion about here. > > As far as I am aware, ports are generally the only option for creating > and communicating with external processes in Erlang. They seem to have > at least one particular fatal flaw which prevents them from being very > useful to me, and that is that there is no way to close stdin (and send > EOF) and then also read from the process's stdout. For example, I cannot > use a port to start the 'cat' program which listens on stdin for data > and waits for EOF and then echos that data back to you. I can do the > first part, which is send it data on stdin, but the only way for me to > close it is to call port_close and close the entire process. > > This issue prevents Erlang users from doing any even slightly more than > trivial communication with external processes without having some kind > of middleman program that handles the creation of the actual process you > need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > I could totally be wrong, but it seems like we need something other than > just port_close. Something like > http://www.erlang.org/doc/man/gen_tcp.html#shutdown-2 > which lets you say > "Hey, I want to close the stdin of this process but still read from its > stdout." or something similar. I could be totally off track on what a > good solution would be. > > So I'm wondering if people are aware of this problem, and I'd like to > make sure that people think it is an actual problem that should be > fixed. I'm also curious what people think a good solution to the problem > would be. I'm not sure I have the time/particular skill set to fix it > given that the port code is some pretty obscure (to me) C code, but > starting conversation seems like a good way to begin. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1188 bytes Desc: not available URL: From i@REDACTED Tue Jul 30 08:23:25 2013 From: i@REDACTED (Anthony Grimes) Date: Mon, 29 Jul 2013 23:23:25 -0700 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: References: <201307291413.r6TEDFaX063329@pluto.hedeland.org> Message-ID: <51F75BDD.3080805@raynes.me> Also, fwiw, here are links to other people with the same issue over the years: http://erlang.org/pipermail/erlang-questions/2010-November/054330.html http://erlang.org/pipermail/erlang-questions/2010-October/053944.html http://erlang.org/pipermail/erlang-questions/2009-March/042123.html http://stackoverflow.com/questions/8792376/erlang-ports-interfacing-with-a-wc-like-program And here are hacks that have been written to get around this limitation: https://github.com/mattsta/erlang-stdinout-pool#why-is-this-special I assume this does as well: https://github.com/saleyn/erlexec Cheers! -Anthony > Richard A. O'Keefe > July 29, 2013 10:55 PM > On 30/07/2013, at 2:13 AM, Per Hedeland wrote: >> FWIW, I definitely agree that this is a missing piece of functionality. > > Oh it's present in several languages. You could argue that it's missing. > But is it missing like a car radio in a car that lacks one, > or is it missing like an ejector seat in a car that lacks one? > >>> Note that "only send data to a command" and "only receive data from a >>> command" are the traditional ways for a UNIX program to communicate >>> with another over a pipe. >> Well, it's basically the definition of the traditional pipeline concept >> of the Unix shells, and pipes are obviously what you need to implement >> it - but that doesn't preclude other uses of pipes. The zsh shell even >> allows you to set up "bidirectional pipes" on the commandline. > > There are reasons why I don't use zsh. That's one of them. >>> popen(, "r") reads the output of >>> the command and popen(, "w") writes to the input of the command. >> popen() is effectively a convenience function to abstract away the >> somewhat non-trivial application of pipe(), fork(), close(), and >> execve() that is required to set things up correctly for two particular >> and common usages of pipes in application code. (It is not used by >> common shells to implement pipelines though.) > > Having implemented popen() for two other high level languages, I know that. > The fact that common shells do not use it is irrelevant. >>> There isn't even any standard _term_ for talking about connecting to both >>> stdin and stdout of a command in UNIX, and that's because it's an >>> incredibly easy way to deadlock. >> There is no need to have a term for it, since all you need is two pipes, >> one for each direction > > Non-sequitur. If it's a thing you need to do often, it's a thing you > need to be able to talk about. When I've thought of doing it, I've > used the word David Bacon used in his SETL2 system, "pump". And then > I've used the phrase "looming disaster" and done something else. > >> - and it's probably uncommon enough to not >> warrant its own convenience function. And you can indeed easily deadlock >> if you don't think about what you're doing, but I really doubt that this >> is the reason for any absence of terminology or functions. > > It's not that you can easily deadlock, > it's that it's hard *NOT* to deadlock. > >> (the risk is of course reduced due to the fact that the >> VM does non-blocking I/O). > > And *that* is the thing that saves Erlang. Of course, avoiding the > coding complexity of dealing with nonblocking I/O is one of the reasons > for using a multithreading language like Erlang. >>> Just like it prevents C users from doing the same thing. >> No, there is nothing that prevents C users from doing the same thing. > > You may have misunderstood me. >>>>> THE POPEN INTERFACE <<<< prevents C users doing this. > Yes, all the other functions are there, and yes, if you desperately > want to program it, you can. But it is enough work that nobody > ever does this lightly. > > For that matter, it's not beyond the abilities of, say, the glibc > authors, to extend their implementation of popen to support "r+" > or "w+" modes, if there were much demand for it. (Oddly enough, > Mac OS X 10.7.5 _does_ support "r+" mode, but the Linux I checked > does not. Weird.) I have _never_ been able to understand the > differences between Mac OS X and POSIX. > > Per Hedeland > July 29, 2013 7:13 AM > "Richard A. O'Keefe" wrote: >> On 29/07/2013, at 4:14 PM, Anthony Grimes wrote: >> >>> and communicating with external processes in Erlang. They seem to have >>> at least one particular fatal flaw which prevents them from being very >>> useful to me, and that is that there is no way to close stdin (and send >>> EOF) and then also read from the process's stdout. For example, I cannot >>> use a port to start the 'cat' program which listens on stdin for data >>> and waits for EOF and then echos that data back to you. I can do the >>> first part, which is send it data on stdin, but the only way for me to >>> close it is to call port_close and close the entire process. > > FWIW, I definitely agree that this is a missing piece of functionality. > I'm not sure how useful/important it is in the grand scheme of things, > but personally I could have used it on a couple of occasions. As you > mentioned, it's basically the equivalent of TCP shutdown() that is > needed, although shutdown() is perhaps a bit over-engineered - I've > never seen anyone use SHUT_RD or SHUT_RDWR... > > Also the "opposite" functionality is already available for ports via the > 'eof' option - i.e. you get informed that the other end has closed its > write side, but can still write data in the other direction. > >> Note that "only send data to a command" and "only receive data from a >> command" are the traditional ways for a UNIX program to communicate >> with another over a pipe. > > Well, it's basically the definition of the traditional pipeline concept > of the Unix shells, and pipes are obviously what you need to implement > it - but that doesn't preclude other uses of pipes. The zsh shell even > allows you to set up "bidirectional pipes" on the commandline. > >> popen(, "r") reads the output of >> the command and popen(, "w") writes to the input of the command. > > popen() is effectively a convenience function to abstract away the > somewhat non-trivial application of pipe(), fork(), close(), and > execve() that is required to set things up correctly for two particular > and common usages of pipes in application code. (It is not used by > common shells to implement pipelines though.) > >> There isn't even any standard _term_ for talking about connecting to both >> stdin and stdout of a command in UNIX, and that's because it's an >> incredibly easy way to deadlock. > > There is no need to have a term for it, since all you need is two pipes, > one for each direction - and it's probably uncommon enough to not > warrant its own convenience function. And you can indeed easily deadlock > if you don't think about what you're doing, but I really doubt that this > is the reason for any absence of terminology or functions. > > But anyway, I don't see how any of this is relevant to the question at > hand. Opening a bi-directional connection between two processes by means > of a pair of pipes is exactly what erlang:open_port/2 *already does* > when you use 'spawn' (or 'spawn_executable' these days) to start an > external process. And it has been doing this since day one, and I can't > recall anyone complaining how this is hopelessly dangerous due to the > risk of deadlock (the risk is of course reduced due to the fact that the > VM does non-blocking I/O). > >>> This issue prevents Erlang users from doing any even slightly more than >>> trivial communication with external processes without having some kind >>> of middleman program that handles the creation of the actual process you >>> need to talk to and looks for a specific byte sequence to indicate 'EOF'. >> Just like it prevents C users from doing the same thing. > > No, there is nothing that prevents C users from doing the same thing. > And even if they have to go to some effort to do it, it just means > having to write a bit more C - whereas the Erlang user can't write a bit > more Erlang to do just the small addition of "close the write side of > one of the pipes", even though the pipe pair is already there... > >> Unix anonymous pipes are simply the wrong tool for the job in _any_ >> programming language. >> >> The historic way to do "slightly more than trivial communication with >> external processes" has been to set the external processes up as C nodes >> or to use sockets. > > Using (TCP) sockets instead of pipes doesn't really change the "risk of > deadlock". In the case of the Erlang VM (i.e. open_port vs gen_tcp), it > may actually increase it, due to the existence of the passive and > {active, once} modes for sockets - another piece of functionality that > is "missing" from ports. > > --Per Hedeland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: From ok@REDACTED Tue Jul 30 08:23:59 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Tue, 30 Jul 2013 18:23:59 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F756FD.1090209@raynes.me> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> <51F756FD.1090209@raynes.me> Message-ID: <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> On 30/07/2013, at 6:02 PM, Anthony Grimes wrote: > I'm simply asking how I'm supposed to deal with programs that I have *no* control over that wait for a EOF on their input to start producing data. I just want to send a ^D. End of file and Control-D are two different things. UNIX technically doesn't have a character that signifies end of file. There is an "EOF" character, but what it does is "send the current line NOW", and sending a line with *no* characters, not even a newline, causes a read() to return 0, and _that_ is the end of file indication. Since 1979, when I started using UNIX, my end-of-file character has been Control-Z, to be compatible with the other computer systems I was using. I am rather fed up with readline-infested programs that insist on Control-D instead of accepting the EOF character I set up. And a program that waits for all of its input before producing any data is by definition not a filter. There is a very very simple technique. (1) Create a temporary file. (2) Create a pipe, telling that pipe to write to the temporary file. (3) Send your data to the pipe and close the pipe. (4) Now read the temporary file. I still don't understand why you can't do this/ > I have examples of programs that do that and a thousand ways to do it in other languages, and I'm being told it is disastrous to do. If the solution is to just keep doing what everybody has been doing in Erlang for years which is write giant hack middleman programs to do it, No, that is NOT what everybody has been doing in Erlang. Creating a temporary file is simple and safe and easy to program. From i@REDACTED Tue Jul 30 08:30:24 2013 From: i@REDACTED (Anthony Grimes) Date: Mon, 29 Jul 2013 23:30:24 -0700 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> <51F756FD.1090209@raynes.me> <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> Message-ID: <51F75D80.3050809@raynes.me> Once again, I'm sorry I haven't got the terminology right. Sure, that's an option. I'm still unclear why I can do this in every other language I use but not Erlang. I find it hard to believe that all other languages with this capability are doing something utterly stupid, but Erlang, being the best language in the known universe, has gotten it utterly correct. But once again, I'll concede. Thanks for the responses and teaching me a thing or two, Mr. O'Keefe! > Richard A. O'Keefe > July 29, 2013 11:23 PM > On 30/07/2013, at 6:02 PM, Anthony Grimes wrote: >> I'm simply asking how I'm supposed to deal with programs that I have *no* control over that wait for a EOF on their input to start producing data. I just want to send a ^D. > > End of file and Control-D are two different things. > UNIX technically doesn't have a character that signifies end of file. > There is an "EOF" character, but what it does is "send the current line NOW", > and sending a line with *no* characters, not even a newline, causes a read() > to return 0, and _that_ is the end of file indication. > Since 1979, when I started using UNIX, my end-of-file character has been > Control-Z, to be compatible with the other computer systems I was using. > I am rather fed up with readline-infested programs that insist on Control-D > instead of accepting the EOF character I set up. > > And a program that waits for all of its input before producing any data > is by definition not a filter. > > There is a very very simple technique. > (1) Create a temporary file. > (2) Create a pipe, telling that pipe to write to the temporary file. > (3) Send your data to the pipe and close the pipe. > (4) Now read the temporary file. > > I still don't understand why you can't do this/ > >> I have examples of programs that do that and a thousand ways to do it in other languages, and I'm being told it is disastrous to do. If the solution is to just keep doing what everybody has been doing in Erlang for years which is write giant hack middleman programs to do it, > > No, that is NOT what everybody has been doing in Erlang. > Creating a temporary file is simple and safe and easy to program. > > Anthony Grimes > July 29, 2013 1:20 AM > Yeah, re-reading your post a couple of times, I think we might be on > the wrong page or something. Here is a low level example of what I'd > like to be able to do in Erlang. This is a Clojure repl session where > I interact with the 'cat' program via the built in Java Process library: > > user=> (def proc (.exec (Runtime/getRuntime) (into-array String ["cat"]))) > #'user/proc > user=> (def stdin (.getOutputStream proc)) > #'user/stdin > user=> (def stdout (.getInputStream proc)) > #'user/stdout > user=> (.write stdin (.getBytes "Hi!")) > nil > user=> (.close stdin) > nil > user=> (let [arr (byte-array 3)] (.read stdout arr) (String. arr)) > "Hi!" > > Lots of unix programs work like this. We have cat in this example, but > grep, wc, and various others work like that as well. It is this easy > or easier to do the same thing in every other language I can think of. > If it's fundamentally a bad thing, I'm surprised these programs work > like that in the first place and that these languages support this. It > seems to be an entirely common place, basic feature any remotely high > level programming language. > > Perhaps this example and clarification will clear things up! > > -Anthony > Richard A. O'Keefe > July 29, 2013 12:15 AM > On 29/07/2013, at 4:14 PM, Anthony Grimes wrote: > >> and communicating with external processes in Erlang. They seem to have >> at least one particular fatal flaw which prevents them from being very >> useful to me, and that is that there is no way to close stdin (and send >> EOF) and then also read from the process's stdout. For example, I cannot >> use a port to start the 'cat' program which listens on stdin for data >> and waits for EOF and then echos that data back to you. I can do the >> first part, which is send it data on stdin, but the only way for me to >> close it is to call port_close and close the entire process. > > Note that "only send data to a command" and "only receive data from a > command" are the traditional ways for a UNIX program to communicate > with another over a pipe. popen(, "r") reads the output of > the command and popen(, "w") writes to the input of the command. > There isn't even any standard _term_ for talking about connecting to both > stdin and stdout of a command in UNIX, and that's because it's an > incredibly easy way to deadlock. > >> This issue prevents Erlang users from doing any even slightly more than >> trivial communication with external processes without having some kind >> of middleman program that handles the creation of the actual process you >> need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > Just like it prevents C users from doing the same thing. > Unless they fake something up using named pipes or UNIX-domain sockets. > (Or message queues. I do wish Mac OS X implemented rather more of POSIX...) > > Unix anonymous pipes are simply the wrong tool for the job in _any_ > programming language. > > The historic way to do "slightly more than trivial communication with > external processes" has been to set the external processes up as C nodes > or to use sockets. > > > > > Anthony Grimes > July 28, 2013 9:14 PM > Howdy folks. > > I unfortunately have not been able to use Erlang for most of what I've > been doing lately because of a long standing issue with Erlang ports > that I'd like to start a discussion about here. > > As far as I am aware, ports are generally the only option for creating > and communicating with external processes in Erlang. They seem to have > at least one particular fatal flaw which prevents them from being very > useful to me, and that is that there is no way to close stdin (and send > EOF) and then also read from the process's stdout. For example, I cannot > use a port to start the 'cat' program which listens on stdin for data > and waits for EOF and then echos that data back to you. I can do the > first part, which is send it data on stdin, but the only way for me to > close it is to call port_close and close the entire process. > > This issue prevents Erlang users from doing any even slightly more than > trivial communication with external processes without having some kind > of middleman program that handles the creation of the actual process you > need to talk to and looks for a specific byte sequence to indicate 'EOF'. > > I could totally be wrong, but it seems like we need something other than > just port_close. Something like > http://www.erlang.org/doc/man/gen_tcp.html#shutdown-2 > which lets you say > "Hey, I want to close the stdin of this process but still read from its > stdout." or something similar. I could be totally off track on what a > good solution would be. > > So I'm wondering if people are aware of this problem, and I'd like to > make sure that people think it is an actual problem that should be > fixed. I'm also curious what people think a good solution to the problem > would be. I'm not sure I have the time/particular skill set to fix it > given that the port code is some pretty obscure (to me) C code, but > starting conversation seems like a good way to begin. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1188 bytes Desc: not available URL: From barcojie@REDACTED Tue Jul 30 08:46:04 2013 From: barcojie@REDACTED (Barco You) Date: Tue, 30 Jul 2013 14:46:04 +0800 Subject: [erlang-questions] {"init terminating in do_boot", {'cannot load', error_handler, get_files}} In-Reply-To: <637BFB44-2AAF-435D-8519-8116A390DEE1@gmail.com> References: <637BFB44-2AAF-435D-8519-8116A390DEE1@gmail.com> Message-ID: Here is my configurations. My application is called rlstock which depends on only jsx and gen_smtp. reltool.config : {sys, [ {lib_dirs, ["../deps", "../apps"]}, {erts, [{mod_cond, derived}, {app_file, strip}]}, {app_file, strip}, {rel, "rlstock", "0.1", [ kernel, stdlib, sasl, inets, jsx, gen_smtp, rlstock ]}, {rel, "start_clean", "", [ kernel, stdlib ]}, {boot_rel, "rlstock"}, {profile, embedded}, {incl_cond, derived}, {mod_cond, derived}, {excl_archive_filters, [".*"]}, %% Do not archive built libs {excl_sys_filters, ["^bin/.*", "^erts.*/bin/(dialyzer|typer)", "^erts.*/(doc|info|include|lib|man|src)"]}, {excl_app_filters, ["\.gitignore"]}, {app, sasl, [{incl_cond, include}]}, {app, stdlib, [{incl_cond, include}]}, {app, kernel, [{incl_cond, include}]}, {app, crypto, [{incl_cond, include}]}, {app, ssl, [{incl_cond, include}]}, {app, public_key, [{incl_cond, include}]}, {app, jsx, [{incl_cond, include}]}, {app, rlstock, [{mod_cond, app}, {incl_cond, include}]}, {app, hipe, [{incl_cond, exclude}]}, {app, wx, [{incl_cond, exclude}]}, {app, webtool, [{incl_cond, exclude}]} ]}. {target_dir, "rlstock"}. {overlay, [ {mkdir, "log/sasl"}, {copy, "files/erl", "\{\{erts_vsn\}\}/bin/erl"}, {copy, "files/nodetool", "\{\{erts_vsn\}\}/bin/nodetool"}, {copy, "files/rlstock", "bin/rlstock"}, {copy, "files/rlstock.cmd", "bin/rlstock.cmd"}, {copy, "files/start_erl.cmd", "bin/start_erl.cmd"}, {copy, "files/install_upgrade.escript", "bin/install_upgrade.escript"}, {copy, "files/sys.config", "releases/\{\{rel_vsn\}\}/sys.config"}, {copy, "files/vm.args", "releases/\{\{rel_vsn\}\}/vm.args"} ]}. ------------------------------------------------------------------------------------------------------------------ rlstock.app {application,rlstock, [{description,"Test Project."}, {vsn,"0.1"}, {modules,[agent,convert,dispatcher,rlstock,rlstock_app, rlstock_sup,spider,spider_sup,utils]}, {registered,[]}, {applications,[kernel,stdlib,inets,gen_smtp,jsx]}, {mod,{rlstock_app,[]}}, {env,[]}]}. ----------------------------------------------------------------------------------------------------------------------- rebar.config {lib_dirs, ["apps"]}. {sub_dirs, [ "rel", "apps/rlstock" ]}. {erl_opts, [debug_info, {i,"include"}]}. {deps_dir, ["deps"]}. {deps, [ {gen_smtp, ".*", {git, "git://github.com/Vagabond/gen_smtp.git", "master"}}, {jsx, ".*", {git, "git://github.com/talentdeficit/jsx.git", "master"}} ]}. On Tue, Jul 30, 2013 at 1:29 PM, Dmitry Kolesnikov wrote: > Hello, > > It looks like some component is missing from release. Hard to say, > reltool.config and apps deps are needed. > > Try to check that all application deps are part of release > > Best Regards, > Dmitry >-|-|-*> > > > On 30.7.2013, at 7.45, Barco You wrote: > > The error like this: > > {"init terminating in do_boot",{'cannot load',error_handler,get_files}} > > Crash dump was written to: erl_crash.dump > init terminating in do_boot () > > > > On Tue, Jul 30, 2013 at 12:44 PM, Barco You wrote: > >> Hi, >> >> I've finished making an application and can successfully start it from >> erl shell by using application:start(myapp). But, after I use rebar to >> release it I got a error as the subject when I start the App from the >> rel/myapp/bin/myapp. Any clues for that? >> >> Thank you! >> Barco >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Jul 30 09:29:22 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 30 Jul 2013 10:29:22 +0300 Subject: [erlang-questions] {"init terminating in do_boot", {'cannot load', error_handler, get_files}} In-Reply-To: References: <637BFB44-2AAF-435D-8519-8116A390DEE1@gmail.com> Message-ID: <46D885D3-DEC1-437A-B187-53752F23D19D@gmail.com> Hello, Here is you problem: {mod_cond, derived}, reltool tries to include only modules that are used by other modules. all other modules are excluded. I would advise you to drop this option. BTW, You can get more info about retool options here: http://learnyousomeerlang.com/release-is-the-word - Dmitry On Jul 30, 2013, at 9:46 AM, Barco You wrote: > Here is my configurations. My application is called rlstock which depends on only jsx and gen_smtp. > > reltool.config : > {sys, [ > {lib_dirs, ["../deps", "../apps"]}, > {erts, [{mod_cond, derived}, {app_file, strip}]}, > {app_file, strip}, > {rel, "rlstock", "0.1", > [ > kernel, > stdlib, > sasl, > inets, > jsx, > gen_smtp, > rlstock > ]}, > {rel, "start_clean", "", > [ > kernel, > stdlib > ]}, > {boot_rel, "rlstock"}, > {profile, embedded}, > {incl_cond, derived}, > {mod_cond, derived}, > {excl_archive_filters, [".*"]}, %% Do not archive built libs > {excl_sys_filters, ["^bin/.*", "^erts.*/bin/(dialyzer|typer)", > "^erts.*/(doc|info|include|lib|man|src)"]}, > {excl_app_filters, ["\.gitignore"]}, > {app, sasl, [{incl_cond, include}]}, > {app, stdlib, [{incl_cond, include}]}, > {app, kernel, [{incl_cond, include}]}, > {app, crypto, [{incl_cond, include}]}, > {app, ssl, [{incl_cond, include}]}, > {app, public_key, [{incl_cond, include}]}, > {app, jsx, [{incl_cond, include}]}, > {app, rlstock, [{mod_cond, app}, {incl_cond, include}]}, > {app, hipe, [{incl_cond, exclude}]}, > {app, wx, [{incl_cond, exclude}]}, > {app, webtool, [{incl_cond, exclude}]} > ]}. > > {target_dir, "rlstock"}. > > {overlay, [ > {mkdir, "log/sasl"}, > {copy, "files/erl", "\{\{erts_vsn\}\}/bin/erl"}, > {copy, "files/nodetool", "\{\{erts_vsn\}\}/bin/nodetool"}, > {copy, "files/rlstock", "bin/rlstock"}, > {copy, "files/rlstock.cmd", "bin/rlstock.cmd"}, > {copy, "files/start_erl.cmd", "bin/start_erl.cmd"}, > {copy, "files/install_upgrade.escript", "bin/install_upgrade.escript"}, > {copy, "files/sys.config", "releases/\{\{rel_vsn\}\}/sys.config"}, > {copy, "files/vm.args", "releases/\{\{rel_vsn\}\}/vm.args"} > ]}. > ------------------------------------------------------------------------------------------------------------------ > rlstock.app > {application,rlstock, > [{description,"Test Project."}, > {vsn,"0.1"}, > {modules,[agent,convert,dispatcher,rlstock,rlstock_app, > rlstock_sup,spider,spider_sup,utils]}, > {registered,[]}, > {applications,[kernel,stdlib,inets,gen_smtp,jsx]}, > {mod,{rlstock_app,[]}}, > {env,[]}]}. > ----------------------------------------------------------------------------------------------------------------------- > rebar.config > {lib_dirs, ["apps"]}. > > {sub_dirs, [ > "rel", > "apps/rlstock" > ]}. > > {erl_opts, [debug_info, {i,"include"}]}. > > {deps_dir, ["deps"]}. > > > {deps, [ > {gen_smtp, ".*", {git, "git://github.com/Vagabond/gen_smtp.git", "master"}}, > {jsx, ".*", {git, "git://github.com/talentdeficit/jsx.git", "master"}} > ]}. > > > > On Tue, Jul 30, 2013 at 1:29 PM, Dmitry Kolesnikov wrote: > Hello, > > It looks like some component is missing from release. Hard to say, reltool.config and apps deps are needed. > > Try to check that all application deps are part of release > > Best Regards, > Dmitry >-|-|-*> > > > On 30.7.2013, at 7.45, Barco You wrote: > >> The error like this: >> >> {"init terminating in do_boot",{'cannot load',error_handler,get_files}} >> >> Crash dump was written to: erl_crash.dump >> init terminating in do_boot () >> >> >> >> On Tue, Jul 30, 2013 at 12:44 PM, Barco You wrote: >> Hi, >> >> I've finished making an application and can successfully start it from erl shell by using application:start(myapp). But, after I use rebar to release it I got a error as the subject when I start the App from the rel/myapp/bin/myapp. Any clues for that? >> >> Thank you! >> Barco >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Tue Jul 30 10:27:09 2013 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 30 Jul 2013 10:27:09 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> Message-ID: <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> On 30 Jul 2013, at 07:58, Richard A. O'Keefe wrote: > On 30/07/2013, at 2:25 AM, Wojtek Narczy?ski wrote: >> It makes me wonder, what is wrong with ASN.1? > > The original poster explicitly asked for a >> schema-less << protocol. > > ASN.1 is very much schema-based. True. Our approach does not require a schema, but enables schema-based validation. Regarding Yang vs ASN.1, there are several diskussion threads not least in the Yang diskussion forums on why a new language would be better for NETCONF than using e.g. ASN.1. I'm sure Martin Bj?rklund* could give a dissertation on the topic, since he is the main author of the Yang RFC. Whether he wants to is another matter. :) Personally, I really prefer Yang over ASN.1. BR, Ulf W * For those of you who don't know, Martin was one of the architects of the first OTP releases, as well as the author of OTP's SNMP application. Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From jon@REDACTED Tue Jul 30 10:28:54 2013 From: jon@REDACTED (Jon Schneider) Date: Tue, 30 Jul 2013 09:28:54 +0100 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> <51F756FD.1090209@raynes.me> <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> Message-ID: > End of file and Control-D are two different things. > UNIX technically doesn't have a character that signifies end of file. What's more the connection between ^D (or other programmed characters) and things they cause such as signals exists in the terminal layer and does not exist in pipes at all. Or rather _only_ exists in tty devices (and virtual ones). Jon From jesper.louis.andersen@REDACTED Tue Jul 30 10:43:30 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 30 Jul 2013 10:43:30 +0200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F75D80.3050809@raynes.me> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> <51F756FD.1090209@raynes.me> <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> <51F75D80.3050809@raynes.me> Message-ID: On Tue, Jul 30, 2013 at 8:30 AM, Anthony Grimes wrote: > Sure, that's an option. I'm still unclear why I can do this in every other > language I use but not Erlang. I find it hard to believe that all other > languages with this capability are doing something utterly stupid, but > Erlang, being the best language in the known universe, has gotten it > utterly correct. But once again, I'll concede. One thing to muse about is how useful an unimplemented feature is, if said language is more than 20 years old :) Yes, you could add it, and it may make sense to add, but I find it way easier to hack pygments to be able to read data in {packet, 4} format. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barcojie@REDACTED Tue Jul 30 11:08:43 2013 From: barcojie@REDACTED (Barco You) Date: Tue, 30 Jul 2013 17:08:43 +0800 Subject: [erlang-questions] {"init terminating in do_boot", {'cannot load', error_handler, get_files}} In-Reply-To: <46D885D3-DEC1-437A-B187-53752F23D19D@gmail.com> References: <637BFB44-2AAF-435D-8519-8116A390DEE1@gmail.com> <46D885D3-DEC1-437A-B187-53752F23D19D@gmail.com> Message-ID: It really works. Thank you very much, Dmitry! On Tue, Jul 30, 2013 at 3:29 PM, Dmitry Kolesnikov wrote: > Hello, > > Here is you problem: *{mod_cond, derived},* > > reltool tries to include only modules that are used by other modules. all > other modules are excluded. > I would advise you to drop this option. > > BTW, You can get more info about retool options here: > http://learnyousomeerlang.com/release-is-the-word > > - Dmitry > > > > On Jul 30, 2013, at 9:46 AM, Barco You wrote: > > Here is my configurations. My application is called rlstock which depends > on only jsx and gen_smtp. > > reltool.config : > {sys, [ > {lib_dirs, ["../deps", "../apps"]}, > {erts, [{mod_cond, derived}, {app_file, strip}]}, > {app_file, strip}, > {rel, "rlstock", "0.1", > [ > kernel, > stdlib, > sasl, > inets, > jsx, > gen_smtp, > rlstock > ]}, > {rel, "start_clean", "", > [ > kernel, > stdlib > ]}, > {boot_rel, "rlstock"}, > {profile, embedded}, > {incl_cond, derived}, > {mod_cond, derived}, > {excl_archive_filters, [".*"]}, %% Do not archive built libs > {excl_sys_filters, ["^bin/.*", "^erts.*/bin/(dialyzer|typer)", > "^erts.*/(doc|info|include|lib|man|src)"]}, > {excl_app_filters, ["\.gitignore"]}, > {app, sasl, [{incl_cond, include}]}, > {app, stdlib, [{incl_cond, include}]}, > {app, kernel, [{incl_cond, include}]}, > {app, crypto, [{incl_cond, include}]}, > {app, ssl, [{incl_cond, include}]}, > {app, public_key, [{incl_cond, include}]}, > {app, jsx, [{incl_cond, include}]}, > {app, rlstock, [{mod_cond, app}, {incl_cond, include}]}, > {app, hipe, [{incl_cond, exclude}]}, > {app, wx, [{incl_cond, exclude}]}, > {app, webtool, [{incl_cond, exclude}]} > ]}. > > {target_dir, "rlstock"}. > > {overlay, [ > {mkdir, "log/sasl"}, > {copy, "files/erl", "\{\{erts_vsn\}\}/bin/erl"}, > {copy, "files/nodetool", "\{\{erts_vsn\}\}/bin/nodetool"}, > {copy, "files/rlstock", "bin/rlstock"}, > {copy, "files/rlstock.cmd", "bin/rlstock.cmd"}, > {copy, "files/start_erl.cmd", "bin/start_erl.cmd"}, > {copy, "files/install_upgrade.escript", > "bin/install_upgrade.escript"}, > {copy, "files/sys.config", > "releases/\{\{rel_vsn\}\}/sys.config"}, > {copy, "files/vm.args", "releases/\{\{rel_vsn\}\}/vm.args"} > ]}. > > ------------------------------------------------------------------------------------------------------------------ > rlstock.app > {application,rlstock, > [{description,"Test Project."}, > {vsn,"0.1"}, > {modules,[agent,convert,dispatcher,rlstock,rlstock_app, > rlstock_sup,spider,spider_sup,utils]}, > {registered,[]}, > {applications,[kernel,stdlib,inets,gen_smtp,jsx]}, > {mod,{rlstock_app,[]}}, > {env,[]}]}. > > ----------------------------------------------------------------------------------------------------------------------- > rebar.config > {lib_dirs, ["apps"]}. > > {sub_dirs, [ > "rel", > "apps/rlstock" > ]}. > > {erl_opts, [debug_info, {i,"include"}]}. > > {deps_dir, ["deps"]}. > > > {deps, [ > {gen_smtp, ".*", {git, "git://github.com/Vagabond/gen_smtp.git", > "master"}}, > {jsx, ".*", {git, "git://github.com/talentdeficit/jsx.git", > "master"}} > ]}. > > > > On Tue, Jul 30, 2013 at 1:29 PM, Dmitry Kolesnikov > wrote: > >> Hello, >> >> It looks like some component is missing from release. Hard to say, >> reltool.config and apps deps are needed. >> >> Try to check that all application deps are part of release >> >> Best Regards, >> Dmitry >-|-|-*> >> >> >> On 30.7.2013, at 7.45, Barco You wrote: >> >> The error like this: >> >> {"init terminating in do_boot",{'cannot load',error_handler,get_files}} >> >> Crash dump was written to: erl_crash.dump >> init terminating in do_boot () >> >> >> >> On Tue, Jul 30, 2013 at 12:44 PM, Barco You wrote: >> >>> Hi, >>> >>> I've finished making an application and can successfully start it from >>> erl shell by using application:start(myapp). But, after I use rebar to >>> release it I got a error as the subject when I start the App from the >>> rel/myapp/bin/myapp. Any clues for that? >>> >>> Thank you! >>> Barco >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From heinz@REDACTED Tue Jul 30 12:01:36 2013 From: heinz@REDACTED (Heinz Nikolaus Gies) Date: Tue, 30 Jul 2013 12:01:36 +0200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> <51F756FD.1090209@raynes.me> <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> <51F75D80.3050809@raynes.me> Message-ID: <54B73AC3-1BDC-41ED-BE30-A7A4E38EF5A2@licenser.net> I am facing the same issue as Anthony, it is rather annoying there are more issues that are involved with Erlangs handling of the STD* pipes, i.e. missing a way to terminate a program that does not terminate on STDin close. I would not argue 'because it's not in Erlang it is not important'. I greatly enjoy that Erlang is not jumping any change to change but that argument means nothing would ever get get done :P On Jul 30, 2013, at 10:43, Jesper Louis Andersen wrote: > > On Tue, Jul 30, 2013 at 8:30 AM, Anthony Grimes wrote: > Sure, that's an option. I'm still unclear why I can do this in every other language I use but not Erlang. I find it hard to believe that all other languages with this capability are doing something utterly stupid, but Erlang, being the best language in the known universe, has gotten it utterly correct. But once again, I'll concede. > > One thing to muse about is how useful an unimplemented feature is, if said language is more than 20 years old :) > > Yes, you could add it, and it may make sense to add, but I find it way easier to hack pygments to be able to read data in {packet, 4} format. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rtrlists@REDACTED Tue Jul 30 12:20:13 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Tue, 30 Jul 2013 11:20:13 +0100 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <54B73AC3-1BDC-41ED-BE30-A7A4E38EF5A2@licenser.net> References: <51F5EC0C.2040203@raynes.me> <84B285D8-E332-4ECD-8B77-C444D9C0BCE6@cs.otago.ac.nz> <51F625B9.5040506@raynes.me> <1D920410-A1C9-4680-AA25-66555A761057@cs.otago.ac.nz> <51F756FD.1090209@raynes.me> <2FDEF85A-8B1D-497D-8154-AD926E784D5F@cs.otago.ac.nz> <51F75D80.3050809@raynes.me> <54B73AC3-1BDC-41ED-BE30-A7A4E38EF5A2@licenser.net> Message-ID: Having thought about this for a couple of minutes now, and having looked around for some ideas from other languages, if I had the problem of interacting with some random piece of software from within Erlang, I would write a small Jinterface wrapper around ExpectJ or expect4j and take it from there. (I suspect that coding a native Erlang based expect clone is non-trivial, but maybe ripping off some existing code could get you started down that road.) Robby On Tue, Jul 30, 2013 at 11:01 AM, Heinz Nikolaus Gies wrote: > I am facing the same issue as Anthony, > it is rather annoying there are more issues that are involved with Erlangs > handling of the STD* pipes, i.e. missing a way to terminate a program that > does not terminate on STDin close. I would not argue 'because it's not in > Erlang it is not important'. I greatly enjoy that Erlang is not jumping any > change to change but that argument means nothing would ever get get done :P > > > On Jul 30, 2013, at 10:43, Jesper Louis Andersen < > jesper.louis.andersen@REDACTED> wrote: > > > On Tue, Jul 30, 2013 at 8:30 AM, Anthony Grimes wrote: > >> Sure, that's an option. I'm still unclear why I can do this in every >> other language I use but not Erlang. I find it hard to believe that all >> other languages with this capability are doing something utterly stupid, >> but Erlang, being the best language in the known universe, has gotten it >> utterly correct. But once again, I'll concede. > > > One thing to muse about is how useful an unimplemented feature is, if said > language is more than 20 years old :) > > Yes, you could add it, and it may make sense to add, but I find it way > easier to hack pygments to be able to read data in {packet, 4} format. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karol.urbanski@REDACTED Tue Jul 30 13:54:59 2013 From: karol.urbanski@REDACTED (Karol Urbanski) Date: Tue, 30 Jul 2013 13:54:59 +0200 Subject: [erlang-questions] Parallella is here ... anyone has some erlang VM running on it ? :) In-Reply-To: <51EE6178.9060205@univ-lille1.fr> References: <51EE6178.9060205@univ-lille1.fr> Message-ID: <20130730115459.GA30859@dex.pl.erlangsystems.com> Hey Yann, you (and others reading this) might be interested to know we have begun testing our packages for Parallella and Pi. Anyone who wants to play around is welcome to add this line to their /etc/apt/sources.list: deb http://packages.erlang-solutions.com/debian oneiric contrib and then run $ sudo apt-get update $ sudo apt-get install erlang which will install the R16B01 virtual machine and the entire OTP suite. If you want to install a smaller package with just the bare essentials, try $ sudo apt-get install erlang-mini We're eagerly awaiting your feedback. :) Best regards, Karol Urbanski On Tue, Jul 23, 2013 at 12:56:56PM +0200, Yann SECQ wrote: > Hi all, > > for those interested in nice new hardware that could fit wisely > with erlang, you'll be happy to know that you can pre-order this > new Raspberry Pi sized parallel computer: > - http://shop.adapteva.com/collections/parallella/products/parallella-16 > > It seems that some folks within Erlang Solutions are already > hacking this little beast: > - http://www.parallella.org/2013/02/04/parallella-and-erlang-an-introduction/ > - http://www.parallella.org/2013/05/25/explorations-in-erlang-with-the-parallela-a-prelude/ > > I tried to find if a specific erlang bundle was available for > parallela there: > - https://www.erlang-solutions.com/downloads/download-erlang-otp > - or, http://www.erlang-embedded.com > but have not been successful :( > > Do you know if some VM and tools (cf. OpenCL in the demo) will > be packaged and released by Erlang Solutions in the near future ? > > Said otherwise, should I buy my parallela right now or wait some > months ? ;) > > Thank you, yann. > > -- > Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq > www.ouverture-independance.fr | www.sauvonsluniversite.com > -- > "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From per@REDACTED Tue Jul 30 15:04:02 2013 From: per@REDACTED (Per Hedeland) Date: Tue, 30 Jul 2013 15:04:02 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: Message-ID: <201307301304.r6UD42Fd091678@pluto.hedeland.org> "Richard A. O'Keefe" wrote: > >On 30/07/2013, at 2:13 AM, Per Hedeland wrote: >> >> FWIW, I definitely agree that this is a missing piece of functionality. > >Oh it's present in several languages. You could argue that it's missing. >But is it missing like a car radio in a car that lacks one, >or is it missing like an ejector seat in a car that lacks one? I think it's more like a parking brake - not strictly needed, but quite useful, and definitely not dangerous in any way. >>> popen(, "r") reads the output of >>> the command and popen(, "w") writes to the input of the command. >> >> popen() is effectively a convenience function to abstract away the >> somewhat non-trivial application of pipe(), fork(), close(), and >> execve() that is required to set things up correctly for two particular >> and common usages of pipes in application code. (It is not used by >> common shells to implement pipelines though.) > >Having implemented popen() for two other high level languages, I know that. >The fact that common shells do not use it is irrelevant. No, it is quite relevant, since it's one example of how popen() doesn't define the "pipe API" in C, anymore than stdio defines the "I/O API" (in fact considerably less, since popen() is far more limited relative to the underlying basic API, and unlike stdio lacks the portability promise of being ANSI standard). >>> There isn't even any standard _term_ for talking about connecting to both >>> stdin and stdout of a command in UNIX, and that's because it's an >>> incredibly easy way to deadlock. >> >> There is no need to have a term for it, since all you need is two pipes, >> one for each direction > >Non-sequitur. If it's a thing you need to do often, it's a thing you >need to be able to talk about. Ditto - I didn't claim that it's something you need to do often, only responded to your claim that the absence of a term is due to the dangers of using it. >> (the risk is of course reduced due to the fact that the >> VM does non-blocking I/O). > >And *that* is the thing that saves Erlang. Yes, I was actually overly cautious with "reduced" - it completely eliminates the risk of deadlock (at lest as long as there is no way to block input from a port). >You may have misunderstood me. >>>>> THE POPEN INTERFACE <<<< prevents C users doing this. Well, that's not what you said, so I guess the problem wasn't a case of misunderstanding. >For that matter, it's not beyond the abilities of, say, the glibc >authors, to extend their implementation of popen to support "r+" >or "w+" modes, if there were much demand for it. (Oddly enough, >Mac OS X 10.7.5 _does_ support "r+" mode, but the Linux I checked >does not. Weird.) I have _never_ been able to understand the >differences between Mac OS X and POSIX. This is probably due to its relationship with FreeBSD, and the explanation seems to be present in the popen(3) man page of both: Because popen() is now implemented using a bidirectional pipe, the mode argument may request a bidirectional data flow. I.e. *pipes* are bidirectional on FreeBSD (and other *BSD AFAIK) - I'm not sure whether they actually are on MacOS X, the pipe(2) man page certainly doesn't say so (which it does on FreeBSD). It may well also be the case that MacOS X just copied the BSD popen(3) man page without bothering to make sure that it worked as advertised with their underlying pipe(2). (Side note, FreeBSD pipe(2) mentions that SysVR4 was actually first with bidirectional pipes, and indeed Solaris has them too - but according to its popen(3C), doesn't support "r+".) Anyway, interesting as it may be, I'm afraid that this will have to conclude my participation in a discussion about pipes in C and the potential dangers of using them in other ways than popen() allows for - it is after all completely off-topic in this thread at least. To summarize: Erlang open_port/2 *by default* creates a bi-directional, deadlock-free communication channel to the external process. The question at hand is just whether adding a way to close only the "output" side of this channel has merit. --Per From bengt.kleberg@REDACTED Tue Jul 30 15:15:58 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Tue, 30 Jul 2013 15:15:58 +0200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <201307301304.r6UD42Fd091678@pluto.hedeland.org> References: <201307301304.r6UD42Fd091678@pluto.hedeland.org> Message-ID: <1375190158.4789.46.camel@sekic1152.rnd.ki.sw.ericsson.se> Greetings, The summary (at the end) is very good, but perhaps it is missing something? The documentation seems to say that it is possible to close stdin by adding the option 'out' to open_port/2 bengt On Tue, 2013-07-30 at 15:04 +0200, Per Hedeland wrote: > "Richard A. O'Keefe" wrote: > > > >On 30/07/2013, at 2:13 AM, Per Hedeland wrote: > >> > >> FWIW, I definitely agree that this is a missing piece of functionality. > > > >Oh it's present in several languages. You could argue that it's missing. > >But is it missing like a car radio in a car that lacks one, > >or is it missing like an ejector seat in a car that lacks one? > > I think it's more like a parking brake - not strictly needed, but quite > useful, and definitely not dangerous in any way. > > >>> popen(, "r") reads the output of > >>> the command and popen(, "w") writes to the input of the command. > >> > >> popen() is effectively a convenience function to abstract away the > >> somewhat non-trivial application of pipe(), fork(), close(), and > >> execve() that is required to set things up correctly for two particular > >> and common usages of pipes in application code. (It is not used by > >> common shells to implement pipelines though.) > > > >Having implemented popen() for two other high level languages, I know that. > >The fact that common shells do not use it is irrelevant. > > No, it is quite relevant, since it's one example of how popen() doesn't > define the "pipe API" in C, anymore than stdio defines the "I/O API" (in > fact considerably less, since popen() is far more limited relative to > the underlying basic API, and unlike stdio lacks the portability promise > of being ANSI standard). > > >>> There isn't even any standard _term_ for talking about connecting to both > >>> stdin and stdout of a command in UNIX, and that's because it's an > >>> incredibly easy way to deadlock. > >> > >> There is no need to have a term for it, since all you need is two pipes, > >> one for each direction > > > >Non-sequitur. If it's a thing you need to do often, it's a thing you > >need to be able to talk about. > > Ditto - I didn't claim that it's something you need to do often, only > responded to your claim that the absence of a term is due to the dangers > of using it. > > >> (the risk is of course reduced due to the fact that the > >> VM does non-blocking I/O). > > > >And *that* is the thing that saves Erlang. > > Yes, I was actually overly cautious with "reduced" - it completely > eliminates the risk of deadlock (at lest as long as there is no way to > block input from a port). > > >You may have misunderstood me. > >>>>> THE POPEN INTERFACE <<<< prevents C users doing this. > > Well, that's not what you said, so I guess the problem wasn't a case of > misunderstanding. > > >For that matter, it's not beyond the abilities of, say, the glibc > >authors, to extend their implementation of popen to support "r+" > >or "w+" modes, if there were much demand for it. (Oddly enough, > >Mac OS X 10.7.5 _does_ support "r+" mode, but the Linux I checked > >does not. Weird.) I have _never_ been able to understand the > >differences between Mac OS X and POSIX. > > This is probably due to its relationship with FreeBSD, and the > explanation seems to be present in the popen(3) man page of both: > > Because popen() is now implemented using a > bidirectional pipe, the mode argument may request a bidirectional data > flow. > > I.e. *pipes* are bidirectional on FreeBSD (and other *BSD AFAIK) - I'm > not sure whether they actually are on MacOS X, the pipe(2) man page > certainly doesn't say so (which it does on FreeBSD). It may well also be > the case that MacOS X just copied the BSD popen(3) man page without > bothering to make sure that it worked as advertised with their > underlying pipe(2). (Side note, FreeBSD pipe(2) mentions that SysVR4 was > actually first with bidirectional pipes, and indeed Solaris has them too > - but according to its popen(3C), doesn't support "r+".) > > Anyway, interesting as it may be, I'm afraid that this will have to > conclude my participation in a discussion about pipes in C and the > potential dangers of using them in other ways than popen() allows for - > it is after all completely off-topic in this thread at least. > > To summarize: Erlang open_port/2 *by default* creates a bi-directional, > deadlock-free communication channel to the external process. The > question at hand is just whether adding a way to close only the "output" > side of this channel has merit. > > --Per > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From wojtek@REDACTED Tue Jul 30 15:19:25 2013 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Tue, 30 Jul 2013 15:19:25 +0200 Subject: [erlang-questions] Yang vs. ASN.1 (WAS: binary typed schema-less protocol) In-Reply-To: <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> Message-ID: <51F7BD5D.2050703@power.com.pl> W dniu 2013-07-30 10:27, Ulf Wiger pisze: > On 30 Jul 2013, at 07:58, Richard A. O'Keefe wrote: > >> On 30/07/2013, at 2:25 AM, Wojtek Narczy?ski wrote: >>> It makes me wonder, what is wrong with ASN.1? >> The original poster explicitly asked for a >> schema-less << protocol. >> >> ASN.1 is very much schema-based. I have a priori apologized for being off-subject in my original post. I have, however, seen worse on this list. > True. Our approach does not require a schema, but enables schema-based validation. > > Regarding Yang vs ASN.1, there are several diskussion threads not least in the Yang diskussion forums on why a new language would be better for NETCONF than using e.g. ASN.1. I'm sure Martin Bj?rklund* could give a dissertation on the topic, since he is the main author of the Yang RFC. Whether he wants to is another matter. :) > > Personally, I really prefer Yang over ASN.1. > Would your preference hold also for a business system, mean Orders, Invoices, and the like? --Kind regards, Wojtek Narczynski From per@REDACTED Tue Jul 30 15:30:49 2013 From: per@REDACTED (Per Hedeland) Date: Tue, 30 Jul 2013 15:30:49 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <1375190158.4789.46.camel@sekic1152.rnd.ki.sw.ericsson.se> Message-ID: <201307301330.r6UDUnpl092104@pluto.hedeland.org> [I'm ignoring your "Reply-To: bengt.kleberg@REDACTED" header - it seems inappropriate for a message sent to the mailing list that doesn't concern private/personal matters.] Bengt Kleberg wrote: > >The summary (at the end) is very good, but perhaps it is missing >something? The documentation seems to say that it is possible to close >stdin by adding the option 'out' to open_port/2 Well, 'out' actually says to close the *other* direction, from external process to Erlang - 'in' is more like what the OP is after. But both of them take effect from the outset, i.e. a *uni*-directional channel is created. What is asked for is a way to achieve the same effect later on, specifically after having written some data in the ('out') direction that is to be closed. --Per From ulf@REDACTED Tue Jul 30 15:36:34 2013 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 30 Jul 2013 15:36:34 +0200 Subject: [erlang-questions] Yang vs. ASN.1 (WAS: binary typed schema-less protocol) In-Reply-To: <51F7BD5D.2050703@power.com.pl> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> <51F7BD5D.2050703@power.com.pl> Message-ID: <121C5F48-B95E-4015-BF22-3602CD882691@feuerlabs.com> On 30 Jul 2013, at 15:19, Wojtek Narczy?ski wrote: >> Personally, I really prefer Yang over ASN.1. >> > Would your preference hold also for a business system, mean Orders, Invoices, and the like? Ehm, yes, but of course the above should have included "all things being equal?" I don't prefer Yang in places where ASN.1 is entrenched and expected, e.g. where ASN.1 *encoding* is in fact mandated. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From wojtek@REDACTED Tue Jul 30 16:02:04 2013 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Tue, 30 Jul 2013 16:02:04 +0200 Subject: [erlang-questions] Yang vs. ASN.1 (WAS: binary typed schema-less protocol) In-Reply-To: <121C5F48-B95E-4015-BF22-3602CD882691@feuerlabs.com> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> <51F7BD5D.2050703@power.com.pl> <121C5F48-B95E-4015-BF22-3602CD882691@feuerlabs.com> Message-ID: <51F7C75C.40102@power.com.pl> W dniu 2013-07-30 15:36, Ulf Wiger pisze: > On 30 Jul 2013, at 15:19, Wojtek Narczy?ski wrote: > >>> Personally, I really prefer Yang over ASN.1. >>> >> Would your preference hold also for a business system, mean Orders, Invoices, and the like? > Ehm, yes, but of course the above should have included "all things being equal?" > > I don't prefer Yang in places where ASN.1 is entrenched and expected, e.g. where ASN.1 *encoding* is in fact mandated. > > My little dream is something like ASN.1 with readable textual encoding, syntactically related to the specification language. RER (Readable Encoding Rules), so to speak. Personally I find XML extremely unreadable, so XER looks like a nightmare, rather than the fullfillment of this dream. Plus binary encoding when everything works, and when performance matters, but this is reality for a few decades now. Anyway, thanks for the Yang hint, we'll definitely look into it. --Kind regards, Wojtek Narczynski From colanderman@REDACTED Tue Jul 30 16:29:40 2013 From: colanderman@REDACTED (Chris King) Date: Tue, 30 Jul 2013 10:29:40 -0400 Subject: [erlang-questions] Yang vs. ASN.1 (WAS: binary typed schema-less protocol) In-Reply-To: <51F7C75C.40102@power.com.pl> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> <51F7BD5D.2050703@power.com.pl> <121C5F48-B95E-4015-BF22-3602CD882691@feuerlabs.com> <51F7C75C.40102@power.com.pl> Message-ID: On Tue, Jul 30, 2013 at 10:02 AM, Wojtek Narczy?ski wrote: > My little dream is something like ASN.1 with readable textual encoding, > syntactically related to the specification language. RER (Readable Encoding > Rules), so to speak. I think GSER is what you're looking for: http://tools.ietf.org/html/rfc3641 Fabrice Bellard has a tool to convert to/from it: http://bellard.org/ffasn1/ffasn1dump.html From colanderman@REDACTED Tue Jul 30 16:31:04 2013 From: colanderman@REDACTED (Chris King) Date: Tue, 30 Jul 2013 10:31:04 -0400 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> Message-ID: (Gah, missed the Reply All. Sorry Richard.) On Tue, Jul 30, 2013 at 1:58 AM, Richard A. O'Keefe wrote: > The original poster explicitly asked for a >> schema-less << protocol. > > ASN.1 is very much schema-based. Color me confused, but doesn't only PER require a schema? BER/DER/CER are all self-describing. From wojtek@REDACTED Tue Jul 30 17:15:25 2013 From: wojtek@REDACTED (=?UTF-8?B?V29qdGVrIE5hcmN6ecWEc2tp?=) Date: Tue, 30 Jul 2013 17:15:25 +0200 Subject: [erlang-questions] Yang vs. ASN.1 (WAS: binary typed schema-less protocol) In-Reply-To: References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> <3A68741D-139D-4795-80B4-B0A112CFB2E6@feuerlabs.com> <51F67B5B.4020503@power.com.pl> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> <97E77C41-8C1F-4224-838C-9C0BF11CD581@feuerlabs.com> <51F7BD5D.2050703@power.com.pl> <121C5F48-B95E-4015-BF22-3602CD882691@feuerlabs.com> <51F7C75C.40102@power.com.pl> Message-ID: <51F7D88D.8070108@power.com.pl> W dniu 2013-07-30 16:29, Chris King pisze: > On Tue, Jul 30, 2013 at 10:02 AM, Wojtek Narczy?ski wrote: >> My little dream is something like ASN.1 with readable textual encoding, >> syntactically related to the specification language. RER (Readable Encoding >> Rules), so to speak. > I think GSER is what you're looking for: http://tools.ietf.org/html/rfc3641 > > Fabrice Bellard has a tool to convert to/from it: > http://bellard.org/ffasn1/ffasn1dump.html Oh, indeed, looks like this is it, thank you! --Kind regards, Wojtek Narczynski From pablo.platt@REDACTED Tue Jul 30 18:39:52 2013 From: pablo.platt@REDACTED (pablo.platt@REDACTED) Date: Tue, 30 Jul 2013 09:39:52 -0700 (PDT) Subject: [erlang-questions] STUN server In-Reply-To: References: <51A94A2A.2020303@gmail.com> Message-ID: <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> benoit, did you make any progress with your ICE server? I'm testing the processone server but I need ICE (ice-lite) and there are few things missing like the priority, use-candidate, ice-controlled and ice-controlling attributes. I've added the missing attributes but still can't respond to ICE connectivity checks. I've found two more erlang implementations but they seems incomplete: https://github.com/kevinlynx/icerl https://github.com/kalta/nksip/blob/master/nksip/src/nksip_stun.erl On Sunday, July 7, 2013 5:59:40 AM UTC+3, Benoit Chesneau wrote: > > > > > On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov > > wrote: > >> On 01.06.2013 01:00, Igor Karymov wrote: >> >>> You may take a look here: >>> https://github.com/processone/**ejabberd/tree/master/src/stun >>> >>> >> It's a separate project now: https://github.com/processone/**stun >> >> -- >> >> > I'm just back on that topic. > > Thanks for the code. Unfortunately the licensee is too restrictive for my > project so I will have to find another way to do that. Anyway just wanted > to say thanks it can be useful for some. > > - benoit > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vincent.dephily@REDACTED Tue Jul 30 19:18:02 2013 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Tue, 30 Jul 2013 19:18:02 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> Message-ID: <3876988.xutA9s8QfZ@moltowork> On Tuesday 30 July 2013 10:31:04 Chris King wrote: > (Gah, missed the Reply All. Sorry Richard.) > > On Tue, Jul 30, 2013 at 1:58 AM, Richard A. O'Keefe wrote: > > The original poster explicitly asked for a >> schema-less << protocol. > > > > ASN.1 is very much schema-based. > > Color me confused, but doesn't only PER require a schema? BER/DER/CER > are all self-describing. Only up to a point. When decoding, you'll know that you're dealing with, say, an integer, but you won't know which field of the enclosing sequence it corresponds to, if the fields are optional (that's just one example of confusion, but there are more). So you can decode a squeleton of your data, but it'll lack meaning. Contrast this to json or msgpack, which very much encourage you to include the meaning inside the data: {"latitude":42.84}. You could play the same game by including json-like data description inside your BER payload, but what's the point ? You've lost some efficiency and all the automatic data validation. Schema-less decoding of BER can be interesting for debugging and for partial decoding (for example if you're a proxy and only look at metadata before forwarding the rest). But most protocols I've seen that need partial decoding just put an OCTET STRING in the grammar instead. -- Vincent de Phily From vincent.dephily@REDACTED Tue Jul 30 19:20:49 2013 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Tue, 30 Jul 2013 19:20:49 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> Message-ID: <2775509.lsLRk5Dhd5@moltowork> On Friday 26 July 2013 00:49:02 Mike Oxford wrote: > Google protobuf Which is great, but not schema-less. > On Thu, Jul 25, 2013 at 10:26 PM, Richard A. O'Keefe wrote: > > On 26/07/2013, at 5:04 PM, Motiejus Jak?tys wrote: > > > About a year(?) ago I was pointed to a binary protocol which can encode > > > strings, numbers, lists (maybe some more data types like map), and > > > cannot rember its name nor find a reference. > > > > Could this have been Joe Armstrong's UBF? > > > > Ad-hoc communication with other not-written-by-you components > > sounds a bit more like JSON or BSON, or _possibly_ BERT. > > > > UBF's strength is being able to slot a "contract checker" > > in between two components -- Vincent de Phily From vincent.dephily@REDACTED Tue Jul 30 19:28:44 2013 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Tue, 30 Jul 2013 19:28:44 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> References: <20130726050437.GA2817@precise.local> <30B06083-DB90-4F18-BCE1-3093A3F63DF5@cs.otago.ac.nz> Message-ID: <1470839.TDH6uEvoQv@moltowork> On Friday 26 July 2013 17:26:36 Richard A. O'Keefe wrote: > On 26/07/2013, at 5:04 PM, Motiejus Jak?tys wrote: > > About a year(?) ago I was pointed to a binary protocol which can encode > > strings, numbers, lists (maybe some more data types like map), and > > cannot rember its name nor find a reference. > > Could this have been Joe Armstrong's UBF? > > Ad-hoc communication with other not-written-by-you components > sounds a bit more like JSON or BSON, or _possibly_ BERT. I encourage you to look at msgpack too. It has I think a bit more traction that bson/bert, and often wins in time/size benchmarks. -- Vincent de Phily From colanderman@REDACTED Tue Jul 30 19:44:37 2013 From: colanderman@REDACTED (Chris King) Date: Tue, 30 Jul 2013 13:44:37 -0400 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: <3876988.xutA9s8QfZ@moltowork> References: <20130726050437.GA2817@precise.local> <2628FF90-8164-4B34-AB1A-5B64B5770100@cs.otago.ac.nz> <3876988.xutA9s8QfZ@moltowork> Message-ID: On Tue, Jul 30, 2013 at 1:18 PM, Vincent de Phily wrote: > On Tuesday 30 July 2013 10:31:04 Chris King wrote: >> Color me confused, but doesn't only PER require a schema? BER/DER/CER >> are all self-describing. > > Only up to a point. When decoding, you'll know that you're dealing with, say, > an integer, but you won't know which field of the enclosing sequence it > corresponds to, if the fields are optional (that's just one example of > confusion, but there are more). So you can decode a squeleton of your data, > but it'll lack meaning. This particular problem I thought was addressed by SET types with explicitly tagged members? The shortcoming I *do* see is that the member tags must be integers; the advantage JSON has is that the tags may be strings. Even then, string tags only aid humans, not machines. This is a limitation to be sure, but I wouldn't say this makes the difference between a schema-ful and schema-less encoding. (P.S. Sorry if you get two copies Vincent; I'm on an unfamiliar mail client today and keep hitting "Reply" instead of "Reply All".) From vincent.dephily@REDACTED Tue Jul 30 20:24:32 2013 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Tue, 30 Jul 2013 20:24:32 +0200 Subject: [erlang-questions] binary typed schema-less protocol In-Reply-To: References: <20130726050437.GA2817@precise.local> <3876988.xutA9s8QfZ@moltowork> Message-ID: <1823539.76WRjcSg0G@moltowork> On Tuesday 30 July 2013 13:44:37 Chris King wrote: > On Tue, Jul 30, 2013 at 1:18 PM, Vincent de Phily > > wrote: > > On Tuesday 30 July 2013 10:31:04 Chris King wrote: > >> Color me confused, but doesn't only PER require a schema? BER/DER/CER > >> are all self-describing. > > > > Only up to a point. When decoding, you'll know that you're dealing with, > > say, an integer, but you won't know which field of the enclosing sequence > > it corresponds to, if the fields are optional (that's just one example of > > confusion, but there are more). So you can decode a squeleton of your > > data, but it'll lack meaning. > > This particular problem I thought was addressed by SET types with > explicitly tagged members? > > The shortcoming I *do* see is that the member tags must be integers; > the advantage JSON has is that the tags may be strings. Even then, > string tags only aid humans, not machines. > > This is a limitation to be sure, but I wouldn't say this makes the > difference between a schema-ful and schema-less encoding. You can do that, it'll feel like protobuf/thrift. But unless your data is actually properly described as a set of integer->value pairs, you've introduced a schema without noticing it. The decoder will have to know what the key 1 stands for, either hardcoded or by parsing the asn1 grammar. It *is* possible to use asn1 BER as a schema-less format, if you follow some conventions. But that's fitting a square peg in a round hole. You can philosophize about what it means to be schema-free, but you'll still feel the pain when you use the wrong tool for the job. ASN1 is meant to be used with a schema. -- Vincent de Phily From ok@REDACTED Wed Jul 31 03:51:06 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Wed, 31 Jul 2013 13:51:06 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <201307301304.r6UD42Fd091678@pluto.hedeland.org> References: <201307301304.r6UD42Fd091678@pluto.hedeland.org> Message-ID: I think we may be approaching clarity here. Whether the pipes in your underlying UNIX are bidirectional or not, whether your popen() supports a "r+" argument or not, bidirectional pipe-based communication between a parent process and a child process is safe *IF* (1) the child process first reads all the data without writing and some time after receiving EOF writes all its results without reading, *OR* (2) both programs have been specially written to communicate in this way, possibly using asynchronous I/O, non-blocking I/O, or multiple threads in some way, *AND* they are aware of the value of PIPE_BUF that applies to the system they are running on. (3) pipes let you report end-of-file by closing them, but no other signal. In order to signal a program at the other end of a pipe, you need to know the process ID of the process, which for the equivalent of popen("foobar",...) is not trivially found. (4) To send attention signals of some sort, you need to use pty(4)s. Per Hedeland wrote: "Erlang open_port/2 *by default* creates a bi-directional, deadlock-free communication channel to the external process." That's still not quite right. The thing that governs whether a deadlock is possible or not is the *protocol* the communicating processes use. And it is still the case that an Erlang process may be deadlocked this way; it's just that the whole Erlang system won't be tied up if it happens. From barcojie@REDACTED Wed Jul 31 08:41:10 2013 From: barcojie@REDACTED (Barco You) Date: Wed, 31 Jul 2013 14:41:10 +0800 Subject: [erlang-questions] Can gen_fsm be restarted by a supervisor with state data kept? Message-ID: Dear All, I have a gen_fsm instance which is supervised by a supervisor. The gen_fsm instance sometimes crashed and then restarted by the supervisor, but the state data was reinitialized. How can I restart this instance with the state data it has at crash point? Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Wed Jul 31 09:20:07 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Wed, 31 Jul 2013 09:20:07 +0200 Subject: [erlang-questions] Can gen_fsm be restarted by a supervisor with state data kept? In-Reply-To: References: Message-ID: <1375255207.4788.12.camel@sekic1152.release> Greetings, The FSM has crashed. Is it really a good idea to start it with the same state data? bengt On Wed, 2013-07-31 at 14:41 +0800, Barco You wrote: > Dear All, > > > I have a gen_fsm instance which is supervised by a supervisor. The > gen_fsm instance sometimes crashed and then restarted by the > supervisor, but the state data was reinitialized. How can I restart > this instance with the state data it has at crash point? > > > Thank you. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From Tobias.Schlager@REDACTED Wed Jul 31 09:38:18 2013 From: Tobias.Schlager@REDACTED (Tobias Schlager) Date: Wed, 31 Jul 2013 07:38:18 +0000 Subject: [erlang-questions] Can gen_fsm be restarted by a supervisor with state data kept? In-Reply-To: <1375255207.4788.12.camel@sekic1152.release> References: , <1375255207.4788.12.camel@sekic1152.release> Message-ID: <12F2115FD1CCEE4294943B2608A18FA361B09BBC@MAIL01.win.lbaum.eu> What I usually do in the case a server (or FSM) keeps important data in its state is passing in an ets table id owned by the respective supervisor (e.g. created in its init function). In case of an error or exception, the important parts of the state can be saved in the ets table using the terminate callback of the server. When resurrecting, the server can retrieve the data from the table in its init callback function. However, as mentioned before, restarting with the crash state of a server can be dangerous. You should be extremely careful when selecting the parts of the state to preserve, otherwise you might inject a crash loop which in turn can crash the whole supervision tree. Regards Tobias ________________________________________ Von: erlang-questions-bounces@REDACTED [erlang-questions-bounces@REDACTED]" im Auftrag von "Bengt Kleberg [bengt.kleberg@REDACTED] Gesendet: Mittwoch, 31. Juli 2013 09:20 Cc: erlang-questions Betreff: Re: [erlang-questions] Can gen_fsm be restarted by a supervisor with state data kept? Greetings, The FSM has crashed. Is it really a good idea to start it with the same state data? bengt On Wed, 2013-07-31 at 14:41 +0800, Barco You wrote: > Dear All, > > > I have a gen_fsm instance which is supervised by a supervisor. The > gen_fsm instance sometimes crashed and then restarted by the > supervisor, but the state data was reinitialized. How can I restart > this instance with the state data it has at crash point? > > > Thank you. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From davidnwelton@REDACTED Wed Jul 31 10:10:53 2013 From: davidnwelton@REDACTED (David Welton) Date: Wed, 31 Jul 2013 10:10:53 +0200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F5EC0C.2040203@raynes.me> References: <51F5EC0C.2040203@raynes.me> Message-ID: > I unfortunately have not been able to use Erlang for most of what I've > been doing lately because of a long standing issue with Erlang ports > that I'd like to start a discussion about here. > > As far as I am aware, ports are generally the only option for creating > and communicating with external processes in Erlang. You might look into erlexec: https://github.com/saleyn/erlexec It currently does not handle the communication part of things that you need, but on the other hand, it's very hackable, the author is friendly, and responds quickly to new ideas, so it probably would not be that hard to add what you need to it. And it is an external program itself, so even if it's compiled code, you don't run the risk of crashing your Erlang process by using it. Erlang's handling of external programs leaves something to be desired compared to other languages: * You can't kill them. * You can't treat stdin and stderr as separate streams. Are two important ones that come to mind. -- David N. Welton http://www.welton.it/davidw/ http://www.dedasys.com/ From davidnwelton@REDACTED Wed Jul 31 10:55:04 2013 From: davidnwelton@REDACTED (David Welton) Date: Wed, 31 Jul 2013 10:55:04 +0200 Subject: [erlang-questions] Testing a large, heterogeneous system In-Reply-To: References: Message-ID: Hi, > If you want to limit setup/teardown of the external parts (like your custom > hardware), meck [1] is a great tool for mocking > all or a subset of external dependencies. Cool, that looks very useful indeed - thanks! > Without knowing the internals of your system it's hard to give specific > advice, but if you have a bunch of state-full stuff proper [2] > has some very powerful tools for testing states and reasoning about what > should be valid before and after state transitions. Checkout proper_statem > [3] and this [4] example of a very useful test implementing the > proper_statem callbacks. Looks interesting - are there any examples? It looks a bit ... abstract. With limited time and Erlang abilities, we need to aim for stuff that gets the most coverage with the least amount of work. Thank you -- David N. Welton http://www.welton.it/davidw/ http://www.dedasys.com/ From acautin@REDACTED Wed Jul 31 11:13:47 2013 From: acautin@REDACTED (Agustin Cautin) Date: Wed, 31 Jul 2013 11:13:47 +0200 Subject: [erlang-questions] pretty-printing keeping the comments. Message-ID: Hello, I am trying to pretty print some code using the erl_prettypr module, but in the process of scanning and parsing the string the comments are lost, using the return_comments options from the scanner leads to errors later in the parser. It is possible to keep the comments using this module ?. If not, what would be the preferred ways of achieving this ?. Thanks. Regards. Agustin Cautin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Jul 31 11:20:13 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 31 Jul 2013 02:20:13 -0700 (PDT) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <51F5EC0C.2040203@raynes.me> References: <51F5EC0C.2040203@raynes.me> Message-ID: <45af8ff7-aa07-4ade-b8d1-e0624cf35989@googlegroups.com> I have a ffmpeg-like program that is launched by erlang server and it is very busy doing its job. Sometimes erlang server can decide to kill it. I have implemented it so: there is a second thread that is reading stdin. If stdin is closed, this thread immediately calls _exit() Such approach works good but it requires modification of program. -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Jul 31 12:40:36 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Wed, 31 Jul 2013 13:40:36 +0300 Subject: [erlang-questions] Testing a large, heterogeneous system In-Reply-To: References: Message-ID: On Wed, Jul 31, 2013 at 11:55 AM, David Welton wrote: >> Without knowing the internals of your system it's hard to give specific >> advice, but if you have a bunch of state-full stuff proper [2] >> has some very powerful tools for testing states and reasoning about what >> should be valid before and after state transitions. Checkout proper_statem >> [3] and this [4] example of a very useful test implementing the >> proper_statem callbacks. > > Looks interesting - are there any examples? It looks a bit ... > abstract. With limited time and Erlang abilities, we need to aim for > stuff that gets the most coverage with the least amount of work. PropEr (and quickcheck, and all the property based testing) is certainly not easy, but really, really pays off. Check out the paper[1] (it's an easy read) and tutorials[2]. I've been working on putting all I can find projects that use PropEr to a single place for kind-of CI tests. Still in progress.. For instance, we tested our distributed CRUD system. PropEr interacts with it (mess with the database, add/remove nodes), and checks whether our postconditions hold (for instance, you get back what you have written). We have found numerous cache coherency issues, race conditions. The fun really started when we started turning on/off the worker nodes during the test. Hope that helps. Motiejus [1]: http://proper.softlab.ntua.gr/papers/proper_types.pdf [2]: http://proper.softlab.ntua.gr/Tutorials -- Motiejus Jak?tys From tech@REDACTED Wed Jul 31 13:24:53 2013 From: tech@REDACTED (Erlang Tech) Date: Wed, 31 Jul 2013 13:24:53 +0200 Subject: [erlang-questions] EPMD question Message-ID: <1375269893.10059.0.camel@greyowl.thebluegroup.co.za> Good day everyone we are noting and interesting situation on one of our servers. We have an erlang cluster with the application distributed over multiple servers. When we do a netstat we see a lot of the following on one specific server: tcp 0 0 10.27.103.12:epmd 10.27.103.80:52873 TIME_WAIT tcp 0 0 10.27.103.12:epmd 10.27.103.80:48273 TIME_WAIT tcp 0 0 10.27.103.12:epmd 10.27.103.80:38921 TIME_WAIT tcp 0 0 10.27.103.12:epmd 10.27.103.80:59294 TIME_WAIT tcp 0 0 10.27.103.12:epmd 10.27.103.80:58333 TIME_WAIT my question is what could be causing this multitude of connections to epmd from that one specific server. My understanding was that epmd would only require a single connection, to communicate between the two epmd server instances. The one server has 4 erlang nodes running, and the other server has a single erlang node running (mostly yaws). Erlang version R14B03. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Wed Jul 31 17:38:34 2013 From: serge@REDACTED (Serge Aleynikov) Date: Wed, 31 Jul 2013 11:38:34 -0400 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: References: <51F5EC0C.2040203@raynes.me> Message-ID: <51F92F7A.8070703@aleynikov.org> Speaking of erlexec - I just added experimental support to erlexec to read STDOUT and STDERR from spawned processes and redirect them to either a custom function or an Erlang Pid: See: http://saleyn.github.io/erlexec/ 10> exec:run("echo Hello World!; echo ERR!! 1>&2", [{stdout, self()}, {stderr, self()}]). {ok,<0.247.0>,18585} 11> flush(). Shell got {stdout,18585,<<"Hello World!\n">>} Shell got {stderr,18585,<<"ERR!!\n">>} ok On 7/31/2013 4:10 AM, David Welton wrote: >> I unfortunately have not been able to use Erlang for most of what I've >> been doing lately because of a long standing issue with Erlang ports >> that I'd like to start a discussion about here. >> >> As far as I am aware, ports are generally the only option for creating >> and communicating with external processes in Erlang. > > You might look into erlexec: https://github.com/saleyn/erlexec > > It currently does not handle the communication part of things that you > need, but on the other hand, it's very hackable, the author is > friendly, and responds quickly to new ideas, so it probably would not > be that hard to add what you need to it. And it is an external > program itself, so even if it's compiled code, you don't run the risk > of crashing your Erlang process by using it. > > Erlang's handling of external programs leaves something to be desired > compared to other languages: > > * You can't kill them. > > * You can't treat stdin and stderr as separate streams. > > Are two important ones that come to mind. > From pablo.platt@REDACTED Wed Jul 31 17:52:08 2013 From: pablo.platt@REDACTED (pablo platt) Date: Wed, 31 Jul 2013 18:52:08 +0300 Subject: [erlang-questions] DTLS client/server Message-ID: Hi, Is there a library that implements DTLS client and server in Erlang? DTLS is the UDP version of TLS Datagram Transport Layer Security http://tools.ietf.org/html/rfc4347 Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter@REDACTED Wed Jul 31 17:57:15 2013 From: peter@REDACTED (Peter Membrey) Date: Wed, 31 Jul 2013 23:57:15 +0800 Subject: [erlang-questions] Encoding empty strings using ei Message-ID: <51F933DB.9000904@membrey.hk> Hi guys, I've got a C program that encodes a list of tuples to send to an Erlang server, something like this: {mylist, [{1,2,3} , {2,3,4} , {3,4,5}]} To do this I encode a list header with: ei_x_encode_list_header(ei_x_buff* x, int arity); and all is well. The question I have though, is what should I do if I encode this tuple? {mylist, [] } Should I encode the tuple, followed by an empty list (ei_x_encode_empty_list) or do I need to put in a list header with an arity of 0 and then add the empty list? In short, what is the correct way to encode an empty list using the ei interface. Thanks in advance! Cheers, Pete From per@REDACTED Wed Jul 31 18:55:41 2013 From: per@REDACTED (Per Hedeland) Date: Wed, 31 Jul 2013 18:55:41 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: Message-ID: <201307311655.r6VGtfU2023039@pluto.hedeland.org> "Richard A. O'Keefe" wrote: > >Whether the pipes in your underlying UNIX are bidirectional >or not, whether your popen() supports a "r+" argument or not, >bidirectional pipe-based communication between a parent >process and a child process is safe *IF* > >(1) the child process first reads all the data without writing > and some time after receiving EOF writes all its results > without reading, *OR* Agreed - this is sufficient but not necessary. >(2) both programs have been specially written to communicate > in this way, possibly using asynchronous I/O, non-blocking > I/O, or multiple threads in some way, *AND* > they are aware of the value of PIPE_BUF that applies to > the system they are running on. No. It is sufficient that one of the programs (i.e. in this case the Erlang VM) uses async/non-blocking I/O or threads. If this is the case, the only requirement on the other program (i.e. in this case the one started by open_port(spawn)) is that it reads its stdin (or whatever file descriptor represents its end of the "out" pipe), and handles EOF when doing so. Neither program needs to know anything about PIPE_BUF. >(3) pipes let you report end-of-file by closing them, but no > other signal. Yes! And *this* alone is what the original request in this thread is about. > In order to signal a program at the other > end of a pipe, you need to know the process ID of the > process, which for the equivalent of popen("foobar",...) > is not trivially found. True, but not relevant to the original request. However since the issue has been mentioned in some other comments: open_port(spawn) obviously doesn't use popen(), and it carefully records the pid of the process that it forks. In modern OTP versions (as of R15B02 I believe, based on a patch from Matthias Lang) this pid is also available to Erlang code through erlang:port_info(Port, os_pid) - i.e. sending it an arbitrary POSIX signal is just an os:cmd/1 away. >(4) To send attention signals of some sort, you need to use > pty(4)s. I assume that by "attention signals" (not a term I have seen in this context) you mean "signals generated by sending a specific character on the communication channel". If so, true, for this and many other things you need a pty - but irrelevant. >Per Hedeland wrote: "Erlang open_port/2 *by default* creates >a bi-directional, deadlock-free communication channel to the >external process." That's still not quite right. The thing >that governs whether a deadlock is possible or not is the >*protocol* the communicating processes use. And it is still >the case that an Erlang process may be deadlocked this way; >it's just that the whole Erlang system won't be tied up if it >happens. No, the statement is quite correct, see above. The worst thing that can happen is that the external process doesn't fulfill its obligation to read its stdin, in which case the Erlang process may *block*. This is not a deadlock, in fact it is a feature in normal usage, where the external process *intermittently* doesn't read its stdin due to being busy processing the data it has received. But you're still re-iterating issues that, if they were genuine, would be with the open_port(spawn) functionality *that already exists*. No-one is suggesting that *it* be implemented, since it already is. And you can take advantage of that fact to refute your own assertions by simply using it, instead of posting them here. Here's a little something to get you started: 1> P = open_port({spawn, "/bin/sh"}, []). #Port<0.504> 2> P ! {self(), {command, "echo " ++ lists:duplicate(100000, $a) ++ "\n"}}, ok. ok 3> Rec = fun (F, Port, Acc) -> receive {Port, {data, Data}} -> F(F, Port, Acc ++ Data) after 0 -> Acc end end. #Fun 4> Got = Rec(Rec, P, []), ok. ok 5> length(Got). 100001 This was run on a Linux system where (according to its documentation) PIPE_BUF is 4096 bytes, and the capacity of a pipe (which is something else) is 65536 bytes. --Per From per@REDACTED Wed Jul 31 20:36:45 2013 From: per@REDACTED (Per Hedeland) Date: Wed, 31 Jul 2013 20:36:45 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <45af8ff7-aa07-4ade-b8d1-e0624cf35989@googlegroups.com> Message-ID: <201307311836.r6VIaj1r024897@pluto.hedeland.org> Max Lapshin wrote: > >I have a ffmpeg-like program that is launched by erlang server and it is >very busy doing its job. > >Sometimes erlang server can decide to kill it. > >I have implemented it so: there is a second thread that is reading stdin. >If stdin is closed, this thread immediately calls _exit() > >Such approach works good but it requires modification of program. Just in case you (understandably) can't be bothered to read my exchange with ROK, as of R15B02 you can get hold of the Unix pid of the external process with erlang:port_info(Port, os_pid), and thus be able to kill it (using os:cmd/1) without modification of the program. I believe David Welton posted with a similar issue, i.e. "You can't kill them" - not true anymore. --Per From max.lapshin@REDACTED Wed Jul 31 20:46:13 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 31 Jul 2013 22:46:13 +0400 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <201307311836.r6VIaj1r024897@pluto.hedeland.org> References: <45af8ff7-aa07-4ade-b8d1-e0624cf35989@googlegroups.com> <201307311836.r6VIaj1r024897@pluto.hedeland.org> Message-ID: Your solution is less obtrusive but less stable. What will happen with your system if you leave some processes because your code failed to do os:cmd? Or what will happen if your erlang is killed? On Wed, Jul 31, 2013 at 10:36 PM, Per Hedeland wrote: > Max Lapshin wrote: >> >>I have a ffmpeg-like program that is launched by erlang server and it is >>very busy doing its job. >> >>Sometimes erlang server can decide to kill it. >> >>I have implemented it so: there is a second thread that is reading stdin. >>If stdin is closed, this thread immediately calls _exit() >> >>Such approach works good but it requires modification of program. > > Just in case you (understandably) can't be bothered to read my exchange > with ROK, as of R15B02 you can get hold of the Unix pid of the external > process with erlang:port_info(Port, os_pid), and thus be able to kill it > (using os:cmd/1) without modification of the program. I believe David > Welton posted with a similar issue, i.e. "You can't kill them" - not > true anymore. > > --Per From per@REDACTED Wed Jul 31 21:54:44 2013 From: per@REDACTED (Per Hedeland) Date: Wed, 31 Jul 2013 21:54:44 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: Message-ID: <201307311954.r6VJsiNI026353@pluto.hedeland.org> Max Lapshin wrote: > >Your solution is less obtrusive but less stable. What will happen with >your system if you leave some processes because your code failed to do >os:cmd? Or what will happen if your erlang is killed? Good points - your solution is better at handling those cases. I just wanted to point out that it is now *possible* to terminate the external process even if it isn't reading stdin, without the need for a "wrapper" or modifications to the program. --Per From per@REDACTED Wed Jul 31 23:11:00 2013 From: per@REDACTED (Per Hedeland) Date: Wed, 31 Jul 2013 23:11:00 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <201307311954.r6VJsiNI026353@pluto.hedeland.org> Message-ID: <201307312111.r6VLB0xY028033@pluto.hedeland.org> Per Hedeland wrote: > >Max Lapshin wrote: >> >>Your solution is less obtrusive but less stable. What will happen with >>your system if you leave some processes because your code failed to do >>os:cmd? Or what will happen if your erlang is killed? > >Good points - your solution is better at handling those cases. Actually, that's quite an understatement. Let me try again (as a general statement): The open_port(spawn) design expects an external process that terminates when it sees EOF on stdin. This is the only way to ensure proper termination in all failure cases. In the case of a program that doesn't have this behaviour by default, it can be achieved by modifying the program as described by Max, or by using some kind of wrapper. If this isn't feasible, the Unix pid of the external process can be obtained with erlang:port_info(Port, os_pid), and used to terminate the process by sending it a signal with os:cmd/1. However this is a primitive workaround, and in particular if the Erlang VM is terminated, it doesn't apply, and the external process will continue to run indefinitely. --Per From ludovic@REDACTED Wed Jul 31 21:02:13 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Wed, 31 Jul 2013 21:02:13 +0200 Subject: [erlang-questions] Core erlang wildcards and sequences Message-ID: Hi everybody, I'm building a small DSL which compiles to Erlang VM. I have two questions about how to achieve this. First, if i want compile this pseudo code a = something b = do_something_with(a) %% involves a ... rest of code ... I end to build the following (pseudo) core erlang code: let A = something in let B = apply do_something_with(A) in ... rest of code I printed dummy erlang code to core erlang representation and it resulted in the same forms, namely a stack of 'let expressions'. Is that ok ? Are there better ways to achieve this in certain situations ? Then, i want to implement wildcards ( _ ), so in core erlang i create a regular variable (which is never used) to match any value. Again, is there a better way to do this, does one more variable involves more overhead (i assume the compiler will optimize it anyway) ? One more question !! I figured out that operators like + are calls to erlang:'+'. I would like to play with the cons opearator but neither erlang:'|' nor erlang:'cons' seems to exist. Is there another name ? This is not important, just for fun. Thank you ludovic