From pablo.platt@REDACTED Thu Aug 1 00:03:46 2013 From: pablo.platt@REDACTED (pablo platt) Date: Thu, 1 Aug 2013 01:03:46 +0300 Subject: [erlang-questions] DTLS client/server In-Reply-To: <-2812677860635758685@unknownmsgid> References: <-2812677860635758685@unknownmsgid> Message-ID: I see some DTLS files in the source: https://github.com/erlang/otp/tree/maint/lib/ssl/src Is it unusable or just unstable? Do you have a rough estimate when we can expect it? Next minor release? Next major release? Thanks On Thu, Aug 1, 2013 at 12:57 AM, Ingela Andin wrote: > Hi, OTP is working on that but it is not ready yet. > Regards Ingela - OTP team Ericsson AB > > Skickat fr?n min iPhone > > 31 jul 2013 kl. 17:52 skrev pablo platt : > > Hi, > > Is there a library that implements DTLS client and server in Erlang? > DTLS is the UDP version of TLS > Datagram Transport Layer Security > http://tools.ietf.org/html/rfc4347 > > Thanks > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Aug 1 00:13:57 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 1 Aug 2013 00:13:57 +0200 Subject: [erlang-questions] STUN server In-Reply-To: <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> References: <51A94A2A.2020303@gmail.com> <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> Message-ID: On Tue, Jul 30, 2013 at 6:39 PM, wrote: > benoit, did you make any progress with your ICE server? > > I'm testing the processone server but I need ICE (ice-lite) and there are > few things missing like the priority, use-candidate, ice-controlled and > ice-controlling attributes. > I've added the missing attributes but still can't respond to ICE > connectivity checks. > > I've found two more erlang implementations but they seems incomplete: > https://github.com/kevinlynx/icerl > https://github.com/kalta/nksip/blob/master/nksip/src/nksip_stun.erl Hi pablo, Yes I will have an ice client and stun/turn server available next week. The stun/turn server is working over TCP/TLS and udp. for tcp/tls it is using barrel. the stun/turn client is able to provide a socket linked to the public port. The ice client/server is able to decde ice messages and eventually use them. - benoit > > > On Sunday, July 7, 2013 5:59:40 AM UTC+3, Benoit Chesneau wrote: > >> >> >> >> On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov wrote: >> >>> On 01.06.2013 01:00, Igor Karymov wrote: >>> >>>> You may take a look here: >>>> https://github.com/processone/****ejabberd/tree/master/src/stun >>>> >>>> >>> It's a separate project now: https://github.com/processone/****stun >>> >>> -- >>> >>> >> I'm just back on that topic. >> >> Thanks for the code. Unfortunately the licensee is too restrictive for my >> project so I will have to find another way to do that. Anyway just wanted >> to say thanks it can be useful for some. >> >> - benoit >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Thu Aug 1 00:15:57 2013 From: pablo.platt@REDACTED (pablo platt) Date: Thu, 1 Aug 2013 01:15:57 +0300 Subject: [erlang-questions] STUN server In-Reply-To: References: <51A94A2A.2020303@gmail.com> <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> Message-ID: Great news Benoit. Thanks On Thu, Aug 1, 2013 at 1:13 AM, Benoit Chesneau wrote: > > > > On Tue, Jul 30, 2013 at 6:39 PM, wrote: > >> benoit, did you make any progress with your ICE server? >> >> I'm testing the processone server but I need ICE (ice-lite) and there are >> few things missing like the priority, use-candidate, ice-controlled and >> ice-controlling attributes. >> I've added the missing attributes but still can't respond to ICE >> connectivity checks. >> >> I've found two more erlang implementations but they seems incomplete: >> https://github.com/kevinlynx/icerl >> https://github.com/kalta/nksip/blob/master/nksip/src/nksip_stun.erl > > > Hi pablo, > > Yes I will have an ice client and stun/turn server available next week. > > The stun/turn server is working over TCP/TLS and udp. for tcp/tls it is > using barrel. the stun/turn client is able to provide a socket linked to > the public port. > > The ice client/server is able to decde ice messages and eventually use > them. > > - benoit > > > >> >> >> On Sunday, July 7, 2013 5:59:40 AM UTC+3, Benoit Chesneau wrote: >> >>> >>> >>> >>> On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov wrote: >>> >>>> On 01.06.2013 01:00, Igor Karymov wrote: >>>> >>>>> You may take a look here: >>>>> https://github.com/processone/****ejabberd/tree/master/src/stun >>>>> >>>>> >>>> It's a separate project now: https://github.com/processone/****stun >>>> >>>> -- >>>> >>>> >>> I'm just back on that topic. >>> >>> Thanks for the code. Unfortunately the licensee is too restrictive for >>> my project so I will have to find another way to do that. Anyway just >>> wanted to say thanks it can be useful for some. >>> >>> - benoit >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Aug 1 01:38:31 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 1 Aug 2013 11:38:31 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: References: <51F5EC0C.2040203@raynes.me> Message-ID: <212C2F17-A586-4314-BA1E-30D1B6A5A1B9@cs.otago.ac.nz> On 31/07/2013, at 8:10 PM, David Welton wrote: > Erlang's handling of external programs leaves something to be desired > compared to other languages: > > * You can't kill them. You really can't _always_ kill them (or not _simply_) in other languages either. Consider the following. % cat foobar.c #include #include int main(void) { int i; (void)setsid(); for (i = 0; i < 10; i++) { fprintf(stderr, "Iteration %d\n", i); sleep(30); } fprintf(stderr, "Done\n"); return 0; } % cc -o foobar foobar.c % csh -c "foobar" Set foobar running your favourite programming language. If you get a process ID back, it will be the process ID of the shell, _not_ the process ID of the foobar process. Kill the process ID you got back, and you just kill the shell. foobar keeps on running. Even if you are using a PTY to connect to the the script, a process that calls setsid() has no controlling terminal, so won't be interrupted by the equivalent of ^C. If the scheme that you use doesn't go through a shell, just change foobar to start with i = fork(); if (i != 0) return i < 0; setsid(); In the MAJORITY of cases, killing the first process that got started will in fact make the others drop dead too. Just be aware that it won't ALWAYS work. From ok@REDACTED Thu Aug 1 02:18:59 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 1 Aug 2013 12:18:59 +1200 Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: <201307311655.r6VGtfU2023039@pluto.hedeland.org> References: <201307311655.r6VGtfU2023039@pluto.hedeland.org> Message-ID: On 1/08/2013, at 4:55 AM, Per Hedeland wrote: > > True, but not relevant to the original request. However since the issue > has been mentioned in some other comments: open_port(spawn) obviously > doesn't use popen(), and it carefully records the pid of the process > that it forks. Yes, but that may well be the wrong process. >> (4) To send attention signals of some sort, you need to use >> pty(4)s. > > I assume that by "attention signals" (not a term I have seen in this > context) you mean "signals generated by sending a specific character on > the communication channel". If so, true, for this and many other things > you need a pty - but irrelevant. The term (an old mainframe term) was not meant to be precise. >> Per Hedeland wrote: "Erlang open_port/2 *by default* creates >> a bi-directional, deadlock-free communication channel to the >> external process." That's still not quite right. The thing >> that governs whether a deadlock is possible or not is the >> *protocol* the communicating processes use. And it is still >> the case that an Erlang process may be deadlocked this way; >> it's just that the whole Erlang system won't be tied up if it >> happens. > > No, the statement is quite correct, see above. No, it really really does depend on the protocol. It doesn't matter how the communication is implemented, if both processes start by trying to read, that's a deadlock. You appear to be saying that an Erlang process (but not the Erlang node containing it) can be blocked forever, but somehow that's not a deadlock. I don't understand. > > But you're still re-iterating issues that, if they were genuine, would > be with the open_port(spawn) functionality *that already exists*. They are genuine, and they do apply to the current functionality, as I demonstrate below. > No-one > is suggesting that *it* be implemented, since it already is. And you can > take advantage of that fact to refute your own assertions by simply > using it, instead of posting them here. Here's a little something to get > you started: > > 1> P = open_port({spawn, "/bin/sh"}, []). > #Port<0.504> > 2> P ! {self(), {command, "echo " ++ lists:duplicate(100000, $a) ++ "\n"}}, ok. > ok > 3> Rec = fun (F, Port, Acc) -> receive {Port, {data, Data}} -> F(F, Port, Acc ++ Data) after 0 -> Acc end end. > #Fun > 4> Got = Rec(Rec, P, []), ok. > ok > 5> length(Got). > 100001 This is *NOT* the situation I was talking about. I was talking about *cyclic* scenarios involving *two* processes. +----+ +----+ +--->| P1 | ---> (port) ---> (pipe) ---> | P2 |---+ | +----+ / ^ +----+ | +----------------- \----- (pipe) <-------------+ where the specially written port process is just a part of the communication machinery; how many concurrent activities there are inside it is pretty much irrelevant. _Functionally_, it's as if we had +----+ +----+ +--->| P1 | ---------------> (pipe) ---> | P2 |---+ | +----+ +----+ | +--------------------------- (pipe) <-------------+ The issue is that we have two processes connected by bounded buffers. The fact that one of them is an Erlang process and one of them a Unix process doesn't affect the logic. The fact that the bounded buffers are UNIX pipes rather than some sort of internal data structure is also irrelevant to the logic. In general, this won't work UNLESS the two processes communicate through an appropriate protocol. And your example is a perfect instance of such a protocol, the case I labelled (1). Let me repeat what I wrote: > > (1) the child process first reads all the data without writing > and some time after receiving EOF writes all its results > without reading, FIRST, P1 sends "echo " ++ lists:duplicate(100000, $a) ++ "\n" to P2, which reads *all* of it before writing anything. THEN P2 sends output back to P1. I should have been explicit that this complete-send-then- complete-receive can be embedded in a cycle: loop P1 sends a complete request to P2 without reading anything. This request is unambiguously terminated. P2 reads the complete request, processes it, and sends a complete response without reading anything. This response is unambiguously terminated. P1 reads the complete response without writing anything. end loop The unambiguous termination could be closing the connection. In this case, it's the "\n" that ends the command to the shell and the "\n" that ends the output of "echo". Now let's change your example ever so slightly. 1> P = open_port({spawn, "/bin/cat"}, []). 2> P ! {self(), {command, lists:duplicate(100000, $a) ++ "\n"}}. 3> Rec = fun (F, Port, Acc) -> receive {Port, {data, Data}} -> F(F, Port, Acc ++ Data) end end. 4> Got = Rec(Rec, P, []), ok. I didn't get as far as step 5, because this deadlocked. The big difference here is that cat reads a chunk and echoes it back (but P1 isn't interested in reading yet), reads another chunk, tries to echo it back, and is blocked because the pipe is full. Then P1 tries to send some more, and gets blocked, because the pipe is full. My assertions are not refuted, but proven. Your example works because it does what I said works. My example deadlocks because it does what I said deadlocks. Cyclic communications through bounded buffers *do* require careful design and it *is* easy to get deadlock that way using natural code, and the hard work that went into Erlang's port processes is no panacea. From serge@REDACTED Thu Aug 1 07:00:19 2013 From: serge@REDACTED (Serge Aleynikov) Date: Thu, 01 Aug 2013 01:00:19 -0400 Subject: [erlang-questions] Encoding empty strings using ei In-Reply-To: <51F933DB.9000904@membrey.hk> References: <51F933DB.9000904@membrey.hk> Message-ID: <51F9EB63.4070002@aleynikov.org> term_to_binary/1 is your friend: 1> term_to_binary({mylist, []}). <<131,104,2,100,0,6,109,121,108,105,115,116,106>> As you see at the end of that binary, the empty list gets encoded with a special 106 ('j') tag, which is what ei_x_encode_empty_list() call does under the hood. $ grep ERL_NIL_EXT lib/erl_interface/include/ei.h #define ERL_NIL_EXT 'j' So the correct way to encode the {mylist, []} tuple in ei is to call: ei_x_new_with_version(pbuf); ei_x_encode_tuple_header(pbuf, 2); ei_x_encode_atom(pbuf, "mylist"); ei_x_encode_empty_list(pbuf); BR, Serge On 7/31/2013 11:57 AM, Peter Membrey wrote: > Hi guys, > > I've got a C program that encodes a list of tuples to send to an Erlang > server, something like this: > > {mylist, [{1,2,3} , {2,3,4} , {3,4,5}]} > > To do this I encode a list header with: > > ei_x_encode_list_header(ei_x_buff* x, int arity); > > and all is well. > > The question I have though, is what should I do if I encode this tuple? > > {mylist, [] } > > Should I encode the tuple, followed by an empty list > (ei_x_encode_empty_list) or do I need to put in a list header with an > arity of 0 and then add the empty list? > > In short, what is the correct way to encode an empty list using the ei > interface. > > Thanks in advance! > > Cheers, > > Pete > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From peter@REDACTED Thu Aug 1 08:52:14 2013 From: peter@REDACTED (Peter Membrey) Date: Thu, 1 Aug 2013 08:52:14 +0200 (CEST) Subject: [erlang-questions] Encoding empty strings using ei In-Reply-To: <51F9EB63.4070002@aleynikov.org> References: <51F933DB.9000904@membrey.hk> <51F9EB63.4070002@aleynikov.org> Message-ID: <1290595210.17151.1375339934804.JavaMail.zimbra@membrey.hk> Hi Serge, Thanks for taking the time to go through that, much appreciated! Cheers, Pete ----- Original Message ----- From: "Serge Aleynikov" To: "Peter Membrey" Cc: "Erlang Questions" Sent: Thursday, 1 August, 2013 1:00:19 PM Subject: Re: [erlang-questions] Encoding empty strings using ei term_to_binary/1 is your friend: 1> term_to_binary({mylist, []}). <<131,104,2,100,0,6,109,121,108,105,115,116,106>> As you see at the end of that binary, the empty list gets encoded with a special 106 ('j') tag, which is what ei_x_encode_empty_list() call does under the hood. $ grep ERL_NIL_EXT lib/erl_interface/include/ei.h #define ERL_NIL_EXT 'j' So the correct way to encode the {mylist, []} tuple in ei is to call: ei_x_new_with_version(pbuf); ei_x_encode_tuple_header(pbuf, 2); ei_x_encode_atom(pbuf, "mylist"); ei_x_encode_empty_list(pbuf); BR, Serge On 7/31/2013 11:57 AM, Peter Membrey wrote: > Hi guys, > > I've got a C program that encodes a list of tuples to send to an Erlang > server, something like this: > > {mylist, [{1,2,3} , {2,3,4} , {3,4,5}]} > > To do this I encode a list header with: > > ei_x_encode_list_header(ei_x_buff* x, int arity); > > and all is well. > > The question I have though, is what should I do if I encode this tuple? > > {mylist, [] } > > Should I encode the tuple, followed by an empty list > (ei_x_encode_empty_list) or do I need to put in a list header with an > arity of 0 and then add the empty list? > > In short, what is the correct way to encode an empty list using the ei > interface. > > Thanks in advance! > > Cheers, > > Pete > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From samuelrivas@REDACTED Thu Aug 1 10:42:10 2013 From: samuelrivas@REDACTED (Samuel) Date: Thu, 1 Aug 2013 10:42:10 +0200 Subject: [erlang-questions] Testing a large, heterogeneous system In-Reply-To: References: Message-ID: >> If you want to limit setup/teardown of the external parts (like your custom >> hardware), meck [1] is a great tool for mocking >> all or a subset of external dependencies. > > Cool, that looks very useful indeed - thanks! When working with specially evil legacy software I often run into problems even with meck so I wrote another "anti-legacy" mock using a different approach to complement it, in case you find it useful: https://github.com/samuelrivas/moka >> Without knowing the internals of your system it's hard to give specific >> advice, but if you have a bunch of state-full stuff proper [2] >> has some very powerful tools for testing states and reasoning about what >> should be valid before and after state transitions. Checkout proper_statem >> [3] and this [4] example of a very useful test implementing the >> proper_statem callbacks. > > Looks interesting - are there any examples? It looks a bit ... > abstract. With limited time and Erlang abilities, we need to aim for > stuff that gets the most coverage with the least amount of work. Proper and QuickCheck can definitely add coverage in a very cost effective way, but they require some time investment to master them first. Some (maybe) useful references: http://www.erlang.se/euc/03/proceedings/1430John.pdf (old, there is much more in quickcheck today) http://erlcode.wordpress.com/tag/quickcheck/ http://www.quviq.com/demos.html http://www.erlang.org/workshop/2008/Sess31.pdf http://www.erlang-factory.com/upload/presentations/397/proper_talk.pdf http://basho.com/quickchecking-poolboy-for-fun-and-profit/ http://techmeshconf.com/dl/techmesh-london-2012/slides/JohnHughes_ThePropertiesOfRiak.pdf http://www.madsgroup.org/staff/samuel/files/articles/ast10.pdf http://www.erlang-factory.com/conference/ErlangUserConference2010/speakers/SamuelRivas Best -- Samuel From jesper.louis.andersen@REDACTED Thu Aug 1 11:25:58 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 1 Aug 2013 11:25:58 +0200 Subject: [erlang-questions] Testing a large, heterogeneous system In-Reply-To: References: Message-ID: On Thu, Jul 25, 2013 at 4:46 PM, David Welton wrote: > Erlang, in our project, sits in the middle: it offers a web interface > that uses lots of Javascript via Chicago Boss, manages some web > sockets, talks to a database (Postgres), interacts with several C > nodes, and also manages various external programs via erlexec and > open_port. To add some complexity, specialized hardware is involved > and dealt with via the C programs/nodes. > I am a fan of building virtualized environments rather than mocking. That is, you take something like http://www.vagrantup.com/ and then you build up the parts which your system consists of and make sure you can deploy code inside your virtual environment. For Postgres, look at pg_virtuelenv. Most of the C-nodes can perhaps be pushed into the virtual environment. The specialized hardware is usually nasty and you may have to mock that in order to get correct operation. But I usually hate mocking with a passion and much prefer to run virtualized setups. It also forces you to think about ease-of-development in general as it forces you to have an environment on a box. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pan@REDACTED Thu Aug 1 11:57:24 2013 From: pan@REDACTED (Patrik Nyblom) Date: Thu, 1 Aug 2013 11:57:24 +0200 Subject: [erlang-questions] EPMD question In-Reply-To: <1375269893.10059.0.camel@greyowl.thebluegroup.co.za> References: <1375269893.10059.0.camel@greyowl.thebluegroup.co.za> Message-ID: <51FA3104.5070506@erlang.org> Hi! On 07/31/2013 01:24 PM, Erlang Tech wrote: > > Good day everyone > > we are noting and interesting situation on one of our servers. We > have an erlang cluster with the application distributed over multiple > servers. > > When we do a netstat we see a lot of the following on one specific server: > > tcp 0 0 10.27.103.12:epmd 10.27.103.80:52873 > TIME_WAIT > tcp 0 0 10.27.103.12:epmd 10.27.103.80:48273 > TIME_WAIT > tcp 0 0 10.27.103.12:epmd 10.27.103.80:38921 > TIME_WAIT > tcp 0 0 10.27.103.12:epmd 10.27.103.80:59294 > TIME_WAIT > tcp 0 0 10.27.103.12:epmd 10.27.103.80:58333 > TIME_WAIT > > my question is what could be causing this multitude of connections to > epmd from that one specific server. My understanding was that epmd > would only require a single connection, > to communicate between the two epmd server instances. No, each Erlang node connects to a remote epmd on a machine where a remote Erlang node resides in the distribution connection phase. Like this: Host a.com: 1 EPMD instance (a) Erlang node foo@REDACTED Erlang node bar@REDACTED Host b.com: 1 EPMD instance (b) Erlang node foo@REDACTED EPMD (a), at a.com knows about 'foo' and 'bar' on a.com EPMD (b), at b.com knows about 'foo' at b.com When node foo@REDACTED wants to connect to foo@REDACTED, it splits the Erlang nodename (foo@REDACTED) into the name and host parts (foo and a.com), looks up the ip address for a.com, connects to the EPMD port on a.com and asks for 'foo'. EPMD gives it a port number and it connects to that port on a.com. If it then wants to connect to bar@REDACTED, the process is repeated, but it asks EPMD (a) for 'bar' and gets another port number. The questions to EPMD uses temporary connections, so for each question, a tcp TIME_WAIT connection will linger (as always in TCP). Note that the peer (the one asking) has used ephemeral ports, so as long as the frequency is not high enough to exhaust the ephemeral ports on 10.27.103.80, this is no problem. The TIME_WAIT connections will disappear in approx 4 minutes (2*MSL). The Erlang nodes however have persistent connections to their *local* EPMD (i.e. foo@REDACTED has a connection to EPMD (b) for the whole lifetime of the node), but that is a trick for EPMD to know when the Erlang node dies. That connection is of course also used to communicate the nodename (i.e. 'foo') to EPMD when registering, but after that it's silent and only used for supervision. So, in short, what you see is normal. > > The one server has 4 erlang nodes running, and the other server has a > single erlang node running (mostly yaws). > > Erlang version R14B03. > > Thank you > > Cheers, Patrik > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Thu Aug 1 13:19:10 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 1 Aug 2013 12:19:10 +0100 Subject: [erlang-questions] Cover Import/Export Noise Message-ID: <792AB798-3BC5-43A6-BAC7-84976280CBD8@gmail.com> Is there a compelling reason why cover decides to spew out lots of this (below) to stdio when importing is in use? Analysis includes data from imported files ["/tmp/systest/cover/data/cover.export"] I do a lot of coverage merging during test runs, and I can barely find the actual test case reports on the console for the 100s of lines repeating that message. Could the message not be selectively disabled? I'll happily submit a patch if necessary. Cheers, Tim -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP using GPGMail URL: From per@REDACTED Thu Aug 1 14:17:51 2013 From: per@REDACTED (Per Hedeland) Date: Thu, 1 Aug 2013 14:17:51 +0200 (CEST) Subject: [erlang-questions] Issues with stdin on ports In-Reply-To: Message-ID: <201308011217.r71CHpOt044905@pluto.hedeland.org> "Richard A. O'Keefe" wrote: > >On 1/08/2013, at 4:55 AM, Per Hedeland wrote: > > [Richard A. O'Keefe wrote:] > >>> Per Hedeland wrote: "Erlang open_port/2 *by default* creates >>> a bi-directional, deadlock-free communication channel to the >>> external process." That's still not quite right. The thing >>> that governs whether a deadlock is possible or not is the >>> *protocol* the communicating processes use. And it is still >>> the case that an Erlang process may be deadlocked this way; >>> it's just that the whole Erlang system won't be tied up if it >>> happens. >> >> No, the statement is quite correct, see above. > >No, it really really does depend on the protocol. >It doesn't matter how the communication is implemented, >if both processes start by trying to read, that's a >deadlock. Wow. I sincerely hope that this astute observation isn't the reason you have been dragging out this discussion. But I admit it, you are right, in *any* form of communication between two parties, including human-to- human, if both parties start out with an unbounded wait for a message from the other, there will be a deadlock. With that out of the way, I will try to make amends for my sin, which amounts to forgetting that you keep wandering away from the subject at hand. Thus, in the context of a discussion of alleged problems with bi-directional communication over the channel created by open_port(spawn), and given your assertion that in case (2) it required that "both programs have been specially written to communicate in this way", I incorrectly assumed that you were saying that a protocol specific to this kind of communication was required to avoid deadlock. So, to generalize, the channel created by open_port(spawn) doesn't introduce any risk of deadlock that wasn't already there - but it can't *remove* a risk of deadlock that exists *without* using this channel. True, but should be obvious, and is completely irrelevant. >You appear to be saying that an Erlang process (but not >the Erlang node containing it) can be blocked forever, >but somehow that's not a deadlock. I don't understand. Yes, of course - a deadlock requires that two parties are both waiting (perhaps indirectly) for some action from the other. There are obviously any number of scenarios where an Erlang (or any other) process may block forever without this constituting a deadlock. >The issue is that we have two processes connected by >bounded buffers. But we don't, since the buffer in the direction from the external process to the Erlang process is effectively *un*bounded. This is due to the fact that the VM, making use of non-blocking I/O, will always read what the external process writes, convert it to an Erlang message, and place it in the mailbox of the Erlang process that is the controlling process of the port. > And your >example is a perfect instance of such a protocol, the >case I labelled (1). True - as I wrote, it was just something to get you started. And observing this flaw, you could make the minimal change to make it no longer fit this case - just repeat the sending line: 2> P ! {self(), {command, "echo " ++ lists:duplicate(100000, $a) ++ "\n"}}, ok. And lo and behold, it still works without a deadlock, no matter how many times you repeat that line before doing the receive. (I guess I have to add that it stops working when the VM runs out of memory...) >Now let's change your example ever so slightly. > >1> P = open_port({spawn, "/bin/cat"}, []). >2> P ! {self(), {command, lists:duplicate(100000, $a) ++ "\n"}}. >3> Rec = fun (F, Port, Acc) -> > receive {Port, {data, Data}} -> F(F, Port, Acc ++ Data) end > end. >4> Got = Rec(Rec, P, []), ok. > >I didn't get as far as step 5, because this deadlocked. The problem in this example is just that you omitted the 'after 0' clause of the 'receive'. Thus Rec/3 can never return, and any call of it will block forever, regardless of what is going on with the communication. It *is* indeed a deadlock, since both 'cat' and the Erlang shell are waiting for data from the other - but this is due to the shell expecting 'cat' to produce an infinite amount of output, nothing to do with open_port/2. If you include the 'after 0' clause, the example works just fine: 4> Got = Rec(Rec, P, []), ok. ok 5> length(Got). 100001 >The big difference here is that cat reads a chunk and echoes >it back (but P1 isn't interested in reading yet), reads another >chunk, tries to echo it back, and is blocked because the pipe >is full. No, 'cat' is blocked in a read(0, ...) call - easily verified e.g. with 'strace' on Linux. >Cyclic communications through bounded buffers *do* require >careful design and it *is* easy to get deadlock that way >using natural code, True, but irrelevant. > and the hard work that went into Erlang's >port processes is no panacea. Probably not, but they do not have the problems you think they do. And in any case, this is irrelevant to the question of whether it would be useful to add a possibility to close the "out" direction of the channel. At this point, I have to conclude that you have nothing to contribute regarding this question. --Per From amindfv@REDACTED Thu Aug 1 15:51:57 2013 From: amindfv@REDACTED (amindfv@REDACTED) Date: Thu, 1 Aug 2013 09:51:57 -0400 Subject: [erlang-questions] Core erlang wildcards and sequences In-Reply-To: References: Message-ID: El Jul 31, 2013, a las 15:02, "Ludovic Demblans" escribi?: > Hi everybody, > > I'm building a small DSL which compiles to Erlang VM. I have two > questions about how to achieve this. > > First, if i want compile this pseudo code > > a = something > b = do_something_with(a) %% involves a > ... rest of code ... > > I end to build the following (pseudo) core erlang code: > > let A = something > in let B = apply do_something_with(A) > in ... rest of code > > I printed dummy erlang code to core erlang representation and it > resulted in the same forms, namely a stack of 'let expressions'. > > Is that ok ? Are there better ways to achieve this in certain situations ? > Core erlang has "letrec", so you shouldn't have to nest those let statements. > > Then, i want to implement wildcards ( _ ), so in core erlang i create > a regular variable (which is never used) to match any value. Again, is > there a better way to do this, does one more variable involves more > overhead (i assume the compiler will optimize it anyway) ? > > One more question !! I figured out that operators like + are calls to > erlang:'+'. I would like to play with the cons opearator but neither > erlang:'|' nor erlang:'cons' seems to exist. Is there another name ? > This is not important, just for fun. > Afaik, [ foo | bar ] is as desugared as it gets -- weird, I know. > Thank you > > ludovic > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Tom From n.oxyde@REDACTED Thu Aug 1 15:57:30 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Thu, 1 Aug 2013 15:57:30 +0200 Subject: [erlang-questions] Core erlang wildcards and sequences In-Reply-To: References: Message-ID: Hello, Replied inline. -- Anthony Ramine Le 1 ao?t 2013 ? 15:51, amindfv@REDACTED a ?crit : > > > El Jul 31, 2013, a las 15:02, "Ludovic Demblans" escribi?: > >> Hi everybody, >> >> I'm building a small DSL which compiles to Erlang VM. I have two >> questions about how to achieve this. >> >> First, if i want compile this pseudo code >> >> a = something >> b = do_something_with(a) %% involves a >> ... rest of code ... >> >> I end to build the following (pseudo) core erlang code: >> >> let A = something >> in let B = apply do_something_with(A) >> in ... rest of code >> >> I printed dummy erlang code to core erlang representation and it >> resulted in the same forms, namely a stack of 'let expressions'. >> >> Is that ok ? Are there better ways to achieve this in certain situations ? Erlang does it that way. > Core erlang has "letrec", so you shouldn't have to nest those let statements. Irrelevant, letrec is only for functional values. >> Then, i want to implement wildcards ( _ ), so in core erlang i create >> a regular variable (which is never used) to match any value. Again, is >> there a better way to do this, does one more variable involves more >> overhead (i assume the compiler will optimize it anyway) ? >> >> One more question !! I figured out that operators like + are calls to >> erlang:'+'. I would like to play with the cons opearator but neither >> erlang:'|' nor erlang:'cons' seems to exist. Is there another name ? >> This is not important, just for fun. There is no cons operator. > Afaik, [ foo | bar ] is as desugared as it gets -- weird, I know. > >> Thank you >> >> ludovic From thomas.elsgaard@REDACTED Sat Aug 3 02:20:18 2013 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Fri, 2 Aug 2013 22:20:18 -0200 Subject: [erlang-questions] Json as binary RabbitMQ Payload Message-ID: Hi all With mochijson2 i am encoding to json: D = mochijson2:encode({struct, [{"id", Id}]}) Now i would like to pass D to rabbitMQ as a payload , but in order to do that, it should be binary, how i can do that ? Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Sat Aug 3 02:49:02 2013 From: bob@REDACTED (Bob Ippolito) Date: Fri, 2 Aug 2013 17:49:02 -0700 Subject: [erlang-questions] Json as binary RabbitMQ Payload In-Reply-To: References: Message-ID: iolist_to_binary(D) On Fri, Aug 2, 2013 at 5:20 PM, Thomas Elsgaard wrote: > Hi all > > With mochijson2 i am encoding to json: > > D = mochijson2:encode({struct, [{"id", Id}]}) > > Now i would like to pass D to rabbitMQ as a payload , but in order to do > that, it should be binary, how i can do that ? > > Thomas > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juanjo@REDACTED Sat Aug 3 03:13:33 2013 From: juanjo@REDACTED (Juan Jose Comellas) Date: Fri, 2 Aug 2013 22:13:33 -0300 Subject: [erlang-questions] Json as binary RabbitMQ Payload In-Reply-To: References: Message-ID: You could use JSX [1], which encodes/decodes binaries by default. Bear in mind that the decoded format is not the same one mochiweb uses. [1] https://github.com/talentdeficit/jsx On Fri, Aug 2, 2013 at 9:20 PM, Thomas Elsgaard wrote: > Hi all > > With mochijson2 i am encoding to json: > > D = mochijson2:encode({struct, [{"id", Id}]}) > > Now i would like to pass D to rabbitMQ as a payload , but in order to do > that, it should be binary, how i can do that ? > > Thomas > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ludovic@REDACTED Sat Aug 3 15:50:36 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Sat, 03 Aug 2013 15:50:36 +0200 Subject: [erlang-questions] Core erlang wildcards and sequences In-Reply-To: References: Message-ID: Ok so i'll continue to stack let expressions, it seems simple. Thanks for the replies, Ludovic Le Thu, 01 Aug 2013 15:57:30 +0200, Anthony Ramine a ?crit: > Hello, > > Replied inline. > -- Utilisant le logiciel de courrier r?volutionnaire d'Opera : http://www.opera.com/mail/ From gordon@REDACTED Mon Aug 5 09:37:26 2013 From: gordon@REDACTED (Gordon Guthrie) Date: Mon, 5 Aug 2013 08:37:26 +0100 Subject: [erlang-questions] ANN: Mostly Functional Conference In Edinburgh @ Turing during the Festival Message-ID: Folks We have a Mostly Functional conference on as a fringe event to the Turing Festival during the Edinburgh Festival on the 22nd August. Strong Erlang representation naturally with Richard Carlsson speaking about OTP and Eric Merritt talking about Joxa and Jose Valim talking about Elixier, plus lots more (including some still to be confirmed). Details at: http://mostlyfunctional.com Gordon -- --- Gordon Guthrie CEO vixo.com @gordonguthrie +44 (0) 7776 251669 (in Bonnie Scotland!) vixo is made in Scotland from electrons -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Mon Aug 5 18:05:56 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Mon, 5 Aug 2013 18:05:56 +0200 Subject: [erlang-questions] gen_fsm + handle_info = spaghetti code? Message-ID: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> All, I'm working on an optical archiving application. I wrote a NIF wrapper around a burning SDK. Other relevant parts are a task queue, which holds burning or reading tasks, and a web based gui which shows the drives' state to the user. In the middle sits an Erlang module controlling a physical drive, and of cause the application should handle multiple drives. Because the drives' state diagram is fairly complex (reading, burning, handling different media capacities and types, hw events, errors, user interactions on GUI or pressing the eject button, etc.) I started to implement it as a gen_fsm. Now, my problem: most of the events triggering a state change are coming in plain Erlang messages (from the NIF, or from gproc what I use to connect to the GUI module). I can capture these events with handle_info but I lose the major advantage of the gen_fsm. If two subsequent events are coming through handle_info, I vainly specify the new state, because the state's callback won't be ever reached. My code is a bunch of handle_infos and they are just handling the specific events in any state. What is an advisabel practice to code such an FSM? Thanks for any ideas in advance. Janos -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Aug 5 18:12:05 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 5 Aug 2013 12:12:05 -0400 Subject: [erlang-questions] gen_fsm + handle_info = spaghetti code? In-Reply-To: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> References: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> Message-ID: <20130805161203.GA5514@ferdair.local> I would consider having a middle-man process whose role is to deal with the NIF code, and turns messages received into gen_fsm events. Alternatively, the NIF could cheat and use the internal gen_fsm wrapping of messages ({'gen_event', Event}, where Event is the original message). the gen_fsm process will receive that message and think of it as if it were sent as an asynchronous event through the regular gen_fsm functions, and call your own custom states. This is a bit messier given you break the abstraction, but should be faster at run time than having a middle-man just forward stuff. Regards, Fred. On 08/05, Janos Hary wrote: > All, > > > > I'm working on an optical archiving application. I wrote a NIF wrapper > around a burning SDK. Other relevant parts are a task queue, which holds > burning or reading tasks, and a web based gui which shows the drives' state > to the user. In the middle sits an Erlang module controlling a physical > drive, and of cause the application should handle multiple drives. > > Because the drives' state diagram is fairly complex (reading, burning, > handling different media capacities and types, hw events, errors, user > interactions on GUI or pressing the eject button, etc.) I started to > implement it as a gen_fsm. > > Now, my problem: most of the events triggering a state change are coming in > plain Erlang messages (from the NIF, or from gproc what I use to connect to > the GUI module). I can capture these events with handle_info but I lose the > major advantage of the gen_fsm. If two subsequent events are coming through > handle_info, I vainly specify the new state, because the state's callback > won't be ever reached. My code is a bunch of handle_infos and they are just > handling the specific events in any state. > > What is an advisabel practice to code such an FSM? > > > > Thanks for any ideas in advance. > > Janos > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kolorahl@REDACTED Mon Aug 5 19:58:59 2013 From: kolorahl@REDACTED (Kolo Rahl) Date: Mon, 5 Aug 2013 10:58:59 -0700 Subject: [erlang-questions] gen_fsm + handle_info = spaghetti code? In-Reply-To: <20130805161203.GA5514@ferdair.local> References: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> <20130805161203.GA5514@ferdair.local> Message-ID: I'm not sure what your NIF messages look like versus what your state handlers take, but if you have a single gen_fsm and you store it in a global registry, like with gproc, couldn't you have a single handle_info call that retrieves your Pid and passes along the data in a gen_fsm:send_event call? Example: handle_info(Info, _StateName, _StateData) -> Pid = get_pid(), gen_fsm:send_event(Pid, Info). Then you just need to make sure you register the pid and that your handlers are set up correctly for whatever data is being piped through. I have not actually done something like this before, so I can't conclusively say it works or is the right way to go. Perhaps someone else with more experience than I could comment if this is a reasonable solution. On Mon, Aug 5, 2013 at 9:12 AM, Fred Hebert wrote: > I would consider having a middle-man process whose role is to deal with > the NIF code, and turns messages received into gen_fsm events. > > Alternatively, the NIF could cheat and use the internal gen_fsm wrapping > of messages ({'gen_event', Event}, where Event is the original message). > the gen_fsm process will receive that message and think of it as if it > were sent as an asynchronous event through the regular gen_fsm > functions, and call your own custom states. > > This is a bit messier given you break the abstraction, but should be > faster at run time than having a middle-man just forward stuff. > > Regards, > Fred. > > On 08/05, Janos Hary wrote: > > All, > > > > > > > > I'm working on an optical archiving application. I wrote a NIF wrapper > > around a burning SDK. Other relevant parts are a task queue, which holds > > burning or reading tasks, and a web based gui which shows the drives' > state > > to the user. In the middle sits an Erlang module controlling a physical > > drive, and of cause the application should handle multiple drives. > > > > Because the drives' state diagram is fairly complex (reading, burning, > > handling different media capacities and types, hw events, errors, user > > interactions on GUI or pressing the eject button, etc.) I started to > > implement it as a gen_fsm. > > > > Now, my problem: most of the events triggering a state change are coming > in > > plain Erlang messages (from the NIF, or from gproc what I use to connect > to > > the GUI module). I can capture these events with handle_info but I lose > the > > major advantage of the gen_fsm. If two subsequent events are coming > through > > handle_info, I vainly specify the new state, because the state's callback > > won't be ever reached. My code is a bunch of handle_infos and they are > just > > handling the specific events in any state. > > > > What is an advisabel practice to code such an FSM? > > > > > > > > Thanks for any ideas in advance. > > > > Janos > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Mon Aug 5 20:32:53 2013 From: freza@REDACTED (Jachym Holecek) Date: Mon, 5 Aug 2013 14:32:53 -0400 Subject: [erlang-questions] gen_fsm + handle_info = spaghetti code? In-Reply-To: <20130805161203.GA5514@ferdair.local> References: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> <20130805161203.GA5514@ferdair.local> Message-ID: <20130805183252.GA22781@circlewave.net> # Fred Hebert 2013-08-05: > I would consider having a middle-man process whose role is to deal with > the NIF code, and turns messages received into gen_fsm events. Notice that the return convention of handle_info/3 and State_name/2 is the same, so you can just do: handle_info(Info, State_name, State_data) -> ?MODULE:State_name(Info, State_data); No middleman process needed, and you still get your full-on state machine. That said, I find gen_fsm's event wrapping habit annoying in maybe half of the cases where I do want a FSM, so I wrote a 'fsm' behaviour which is very close to gen_fsm but avoids the wrapping. Can publish that after a little bit of polishing if there's interest (it has some other extra niceties, but nothing major). BTW you'll sometimes see people doing this to "solve" the same problem: handle_info(Info, State_name, State_data) -> gen_fsm:send_event(self(), Info); Which is obviously incredibly broken -- for one it reorders messages and for other it does so unpredictably! > Alternatively, the NIF could cheat and use the internal gen_fsm wrapping > of messages ({'gen_event', Event}, where Event is the original message). > the gen_fsm process will receive that message and think of it as if it > were sent as an asynchronous event through the regular gen_fsm > functions, and call your own custom states. > > This is a bit messier given you break the abstraction, but should be > faster at run time than having a middle-man just forward stuff. That strikes me as a clean approach also: it solves the problem in a very simple and straightforward way, it doesn't introduce any undue (and, more importantly, any realistic) risks, and the question of "breaking the abs- traction" could be argued to be largely a matter of perception in this case and also to be somewhat overrated in general. Overall, what's there not to like? [Here I'm speaking as someone who's doing gen_server:reply/2 shamelessly from a NIF thread in one project. :-) It was the best way to get the flow control I needed in given circumstances, abstractions broken or not...] BR, -- Jachym From janos.n.hary@REDACTED Mon Aug 5 22:41:44 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Mon, 5 Aug 2013 22:41:44 +0200 Subject: [erlang-questions] gen_fsm + handle_info = spaghetti code? In-Reply-To: <20130805183252.GA22781@circlewave.net> References: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> <20130805161203.GA5514@ferdair.local> <20130805183252.GA22781@circlewave.net> Message-ID: <003901ce921c$33dec4a0$9b9c4de0$@gmail.com> Thanks for all the suggestions. I tried Fred's cheating solutions and its working for me very well (after I figured the missing $ sign in '$gen_event', but at least I had to look up the gen_fsm source). I'd be interested in the alternative implementation, especially if you could show those half of the cases where the library version gets annoying. Thanks again, Janos -----Original Message----- From: Jachym Holecek [mailto:freza@REDACTED] Sent: Monday, August 05, 2013 8:33 PM To: Fred Hebert Cc: Janos Hary; erlang-questions@REDACTED Subject: Re: [erlang-questions] gen_fsm + handle_info = spaghetti code? # Fred Hebert 2013-08-05: > I would consider having a middle-man process whose role is to deal > with the NIF code, and turns messages received into gen_fsm events. Notice that the return convention of handle_info/3 and State_name/2 is the same, so you can just do: handle_info(Info, State_name, State_data) -> ?MODULE:State_name(Info, State_data); No middleman process needed, and you still get your full-on state machine. That said, I find gen_fsm's event wrapping habit annoying in maybe half of the cases where I do want a FSM, so I wrote a 'fsm' behaviour which is very close to gen_fsm but avoids the wrapping. Can publish that after a little bit of polishing if there's interest (it has some other extra niceties, but nothing major). BTW you'll sometimes see people doing this to "solve" the same problem: handle_info(Info, State_name, State_data) -> gen_fsm:send_event(self(), Info); Which is obviously incredibly broken -- for one it reorders messages and for other it does so unpredictably! > Alternatively, the NIF could cheat and use the internal gen_fsm > wrapping of messages ({'gen_event', Event}, where Event is the original message). > the gen_fsm process will receive that message and think of it as if it > were sent as an asynchronous event through the regular gen_fsm > functions, and call your own custom states. > > This is a bit messier given you break the abstraction, but should be > faster at run time than having a middle-man just forward stuff. That strikes me as a clean approach also: it solves the problem in a very simple and straightforward way, it doesn't introduce any undue (and, more importantly, any realistic) risks, and the question of "breaking the abs- traction" could be argued to be largely a matter of perception in this case and also to be somewhat overrated in general. Overall, what's there not to like? [Here I'm speaking as someone who's doing gen_server:reply/2 shamelessly from a NIF thread in one project. :-) It was the best way to get the flow control I needed in given circumstances, abstractions broken or not...] BR, -- Jachym From bourinov@REDACTED Tue Aug 6 08:44:17 2013 From: bourinov@REDACTED (Max Bourinov) Date: Tue, 6 Aug 2013 08:44:17 +0200 Subject: [erlang-questions] Json as binary RabbitMQ Payload In-Reply-To: References: Message-ID: We use Jiffy [1] for this purpose. Works very well on production. 1. https://github.com/davisp/jiffy Best regards, Max On Sat, Aug 3, 2013 at 3:13 AM, Juan Jose Comellas wrote: > You could use JSX [1], which encodes/decodes binaries by default. Bear in > mind that the decoded format is not the same one mochiweb uses. > > [1] https://github.com/talentdeficit/jsx > > > On Fri, Aug 2, 2013 at 9:20 PM, Thomas Elsgaard > wrote: > >> Hi all >> >> With mochijson2 i am encoding to json: >> >> D = mochijson2:encode({struct, [{"id", Id}]}) >> >> Now i would like to pass D to rabbitMQ as a payload , but in order to do >> that, it should be binary, how i can do that ? >> >> Thomas >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From overminddl1@REDACTED Tue Aug 6 12:22:49 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Tue, 6 Aug 2013 04:22:49 -0600 Subject: [erlang-questions] ErlExec stdin and other questions Message-ID: I have been attempting to use ErlExec as I noticed that it had a few new features added six days ago to forward the stdout/stderr to an Erlang PID, however I am having some issue. The process that erlexec runs occasionally does not die when the Erlang VM dies, thus keeping it and its children running. Also I need a way to send commands back to the children, but I am either not seeing a command for that or have not figured out the format to send a message, which is difficult because of the main issue I have, when I have it launch a fairly large server software (that is not a daemon for whatever reason, hence why I am wrapping it), the Erlang shell seems to lose a great deal of input that I type in, only catching from every 1 in 4 I type to 1 in 400 characters I type, and it does not seem to matter based on the speed I hit keys or holding a key or anything of the sort, just purely on keypress down, and it remains like that until I kill the child process that I started through erlexec. Using the normal Erlang Ports works fine and has no such issues (other than the child not coming down with the VM either and no direct kill command when the child hangs, which it occasionally does, but os:cmd("kill ...") works around that well enough. Is ErlExec stable enough for high use? Or am I doing something wrong? -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.elsgaard@REDACTED Tue Aug 6 12:45:26 2013 From: thomas.elsgaard@REDACTED (Thomas Elsgaard) Date: Tue, 6 Aug 2013 08:45:26 -0200 Subject: [erlang-questions] Json as binary RabbitMQ Payload In-Reply-To: References: Message-ID: Hi everybody I got it to work very nice with Jiffy!! ///Thomas On Tue, Aug 6, 2013 at 4:44 AM, Max Bourinov wrote: > We use Jiffy [1] for this purpose. Works very well on production. > > 1. https://github.com/davisp/jiffy > > Best regards, > Max > > > > On Sat, Aug 3, 2013 at 3:13 AM, Juan Jose Comellas wrote: > >> You could use JSX [1], which encodes/decodes binaries by default. Bear in >> mind that the decoded format is not the same one mochiweb uses. >> >> [1] https://github.com/talentdeficit/jsx >> >> >> On Fri, Aug 2, 2013 at 9:20 PM, Thomas Elsgaard < >> thomas.elsgaard@REDACTED> wrote: >> >>> Hi all >>> >>> With mochijson2 i am encoding to json: >>> >>> D = mochijson2:encode({struct, [{"id", Id}]}) >>> >>> Now i would like to pass D to rabbitMQ as a payload , but in order to do >>> that, it should be binary, how i can do that ? >>> >>> Thomas >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlangsiri@REDACTED Tue Aug 6 14:42:53 2013 From: erlangsiri@REDACTED (Siri Hansen) Date: Tue, 6 Aug 2013 14:42:53 +0200 Subject: [erlang-questions] What is "key" in rb:filter In-Reply-To: References: Message-ID: The Key and Value corresponds to Tag and Data, respectively, in an error|info|warning_report - see http://www.erlang.org/doc/man/error_logger.html#type-report. It could definitely have been better explained in the documentation! /siri 2013/7/18 Bin Wang > In the doc of rb:filter, it says: > > filter(Filters) > Filters = [filter()] > filter() = {Key, Value} > > But what is the "key" means? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Tue Aug 6 15:35:09 2013 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 6 Aug 2013 15:35:09 +0200 Subject: [erlang-questions] process dictionary and get_keys/1 Message-ID: I pride myself of having a pretty good grasp of what's in OTP, but I confess that I had missed the introduction of the BIF get_keys(Value) in R15B. What it does is pretty simple: it returns all keys in the process dictionary that correspond to Value - what amounts to a linear search. Example: 1> [put(K,V) || {K,V} <- [{a,1}, {b,1}, {c,2}, {d,2}, {e,3}]]. [undefined,undefined,undefined,undefined,undefined] 2> [get_keys(V) || V <- [1, 2, 3]]. [[b,a], [c,d], [e]] I benchmarked it against a few other ways to do a linear search on Value in a {Key, Value} set. Here are some results for extracting 5 matching keys out of 100,000: - process_dictionary, get_keys: 431 us - LC on a KV-list: 2,740 us - gb_trees: 9,700 us - dict: 14,600 us - ets (set): 12,000 us I know the process dictionary has its disadvantages, but when it fits the problem, it can be pretty amazing. :) Exactly how to use this, I'm not sure yet, but I thought I'd share the info. Can anyone shed some light on why it was introduced in the first place? BR, Ulf Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From lukas@REDACTED Tue Aug 6 15:47:41 2013 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 6 Aug 2013 15:47:41 +0200 Subject: [erlang-questions] process dictionary and get_keys/1 In-Reply-To: References: Message-ID: On Tue, Aug 6, 2013 at 3:35 PM, Ulf Wiger wrote: > > > Can anyone shed some light on why it was introduced in the first place? > Not sure from where you got that it was added in R15B, from what I can gather it has always been there, at least since R4B[1]. It has not been overly used though it seems :) [1]: http://www.erlang.org/documentation/doc-4.7.3/lib/kernel-2.1.1/doc/html/erlang.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From peerst@REDACTED Tue Aug 6 17:02:31 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Tue, 6 Aug 2013 17:02:31 +0200 Subject: [erlang-questions] gen_fsm + handle_info = spaghetti code? References: <002601ce91f5$ac49dd20$04dd9760$@gmail.com> Message-ID: Whenever gen_server or gen_fsm gets in my way and the code gets unnecessarily convoluted just to deal with their respective way of doing it I just don't use them. Instead I just write plain erlang servers with fsm's coded as one function per state calling each other as state transitions. In the functions I just use receive statements to handle my messages. Adiitional benefits is that I get the additional power of selective receive on my messages that keep my state count and transitions lower. This works especially well when talking to real world hardware or for implementing protocol layers. For more info on why selective receive is an extra power: http://www.infoq.com/presentations/Death-by-Accidental-Complexity When you start your process with proc_lib:spawn_link instead of the normal spawn_link BIF the process can be run in a supervisor. If you also handle system messages you get a full OTP compliant process (aka Special Processes). System messages can be handled manually or better by using sys:handle_system_msg Example in the slides here: https://www.erlang-factory.com/conference/ErlangUserConference2013/speakers/LoicHoguin Also there would be https://github.com/uwiger/plain_fsm to help you write a Special Process. One additional trick: I use gen:call to get all the nice client features of gen_{server,fsm}, unfortunately it isn't documented except here https://github.com/erlang/otp/blob/maint/lib/stdlib/src/gen.erl#L134 (worth a read anyway as learning experience). If you find yourself repeating certain patterns repeatedly: write your own behaviours to factor out the common things. Cheers, -- Peer On 2013-08-05 16:05:56 +0000, Janos Hary said: > All, > ? > I?m working on an optical archiving application. I wrote a NIF wrapper > around a burning SDK. Other relevant parts are a task queue, which > holds burning or reading tasks, and a web based gui which shows the > drives? state to the user. In the middle sits an Erlang module > controlling a physical drive, and of cause the application should > handle multiple drives. > Because the drives? state diagram is fairly complex (reading, burning, > handling different media capacities and types, hw events, errors, user > interactions on GUI or pressing the eject button, etc.) I started to > implement it as a gen_fsm. > Now, my problem: most of the events triggering a state change are > coming in plain Erlang messages (from the NIF, or from gproc what I use > to connect to the GUI module). I can capture these events with > handle_info but I lose the major advantage of the gen_fsm. If two > subsequent events are coming through handle_info, I vainly specify the > new state, because the state?s callback won?t be ever reached. My code > is a bunch of handle_infos and they are just handling the specific > events in any state. > What is an advisabel practice to code such an FSM? > ? > Thanks for any ideas in advance. > Janos From ulf@REDACTED Tue Aug 6 19:11:28 2013 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 6 Aug 2013 19:11:28 +0200 Subject: [erlang-questions] process dictionary and get_keys/1 In-Reply-To: References: Message-ID: <84BE09FE-8299-4CF6-8BD2-034AB885F609@feuerlabs.com> True, I guess I was a bit sloppy when I tried it in different OTP versions. It even exists in R13B? :) A slightly slower linear search on process dictionaries, but more flexible, and still faster than searching a gb_trees or dict structure, is: select_keys(V) -> MS = ets:match_spec_compile([{{'$1',V}, [], ['$1']}]), ets:match_spec_run(get(), MS). About the same speed as doing a list comprehension using get() as generator, and faster than putting the same data in ets and using ets:select/2. BR, Ulf On 6 Aug 2013, at 15:47, Lukas Larsson wrote: > > > On Tue, Aug 6, 2013 at 3:35 PM, Ulf Wiger wrote: > > > Can anyone shed some light on why it was introduced in the first place? > > Not sure from where you got that it was added in R15B, from what I can gather it has always been there, at least since R4B[1]. It has not been overly used though it seems :) > > [1]: http://www.erlang.org/documentation/doc-4.7.3/lib/kernel-2.1.1/doc/html/erlang.html Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From overminddl1@REDACTED Wed Aug 7 06:49:01 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Tue, 6 Aug 2013 22:49:01 -0600 Subject: [erlang-questions] ErlExec stdin and other questions In-Reply-To: References: Message-ID: Excepting this, is there any other better way to launch a program as another user? I might just use ssh certs and ssh to localhost if not unless any other ideas? On Tue, Aug 6, 2013 at 4:22 AM, OvermindDL1 wrote: > I have been attempting to use ErlExec as I noticed that it had a few new > features added six days ago to forward the stdout/stderr to an Erlang PID, > however I am having some issue. The process that erlexec runs occasionally > does not die when the Erlang VM dies, thus keeping it and its children > running. > > Also I need a way to send commands back to the children, but I am either > not seeing a command for that or have not figured out the format to send a > message, which is difficult because of the main issue I have, when I have > it launch a fairly large server software (that is not a daemon for whatever > reason, hence why I am wrapping it), the Erlang shell seems to lose a great > deal of input that I type in, only catching from every 1 in 4 I type to 1 > in 400 characters I type, and it does not seem to matter based on the speed > I hit keys or holding a key or anything of the sort, just purely on > keypress down, and it remains like that until I kill the child process that > I started through erlexec. > > Using the normal Erlang Ports works fine and has no such issues (other > than the child not coming down with the VM either and no direct kill > command when the child hangs, which it occasionally does, but os:cmd("kill > ...") works around that well enough. > > Is ErlExec stable enough for high use? Or am I doing something wrong? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw@REDACTED Wed Aug 7 08:15:37 2013 From: flw@REDACTED (Florian Waas) Date: Tue, 6 Aug 2013 23:15:37 -0700 Subject: [erlang-questions] SSL/TLS renegotiation crash triggered by setopts Message-ID: Hi, Reading from a Postgres server tls_connection's gen_fsm crashes on me during TLS renegotiation with below error report (small snippet of most relevant parts only due to size). I'm setting the socket into {active, once} after each packet; the server kicks off renegotiation every 512MB. Looks like the problem occurs when setopts is called during renegotiation. When hammering the system I hit it pretty much every time the server renegotiates -- happens in both R15 and R16. Quick hack in tls_connection.erl:995 to allow the transition from 'connection' to 'hello' in handle_sync_event instead of insisting that setopts doesn't change states gets me out of this pickle... Anybody seen this problem? Clean(er) workarounds available? Thanks, -fl. =ERROR REPORT==== 6-Aug-2013::21:26:38 === ** State machine <0.368.0> terminating ** Last message in was {'$gen_sync_all_state_event', {<0.364.0>,#Ref<0.0.0.67905>}, {set_opts,[{active,once}]}} ** When State == connection ** Data == {state,client, {#Ref<0.0.0.247>,<0.364.0>}, [...] undefined, {[],[]}, false,true,false,undefined,undefined} ** Reason for termination = ** {{case_clause, {next_state,hello, {state,client, [...] [{tls_connection,handle_sync_event,4, [{file,"tls_connection.erl"},{line,994}]}, {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,505}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} From overminddl1@REDACTED Wed Aug 7 09:36:21 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Wed, 7 Aug 2013 01:36:21 -0600 Subject: [erlang-questions] ErlExec stdin and other questions In-Reply-To: References: Message-ID: Created a complete test-case example of my issue with erlexec, and I ran this same script on 3 different computers running variations of debian versions. Feel free to toss in application:start(sasl), at the top of the erlang script area if you wish, and yes the (minecraft) server software uses stderr instead of stdout, no I do not know why, and yes as you can see it does not return anything to the erlang system until it is killed on all the systems I tested. Using Erlang's ports you get each line as a message as that is how the server flushes it and it is instant. Yes it is ugly, but it is pared down from what it grew from and no need to re-do the receive's better in such a test. test_exec_mc.sh """ #!/bin/sh [ -d erlexec ] || git clone https://github.com/saleyn/erlexec.git cd erlexec rebar compile [ -f 'minecraft_server.1.6.2.jar' ] || wget ' https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar ' erl -pa $PWD/ebin -eval ' application:start(exec), {_, P, _} = exec:run_link("java -Xms1G -Xmx1G -jar minecraft_server.1.6.2.jar nogui", [{stdout, self()}, {stderr, self()}]), timer:sleep(1000), receive A0 -> io:format("Message ~p: ~p~n", [now(), A0]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A1 -> io:format("Message ~p: ~p~n", [now(), A1]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A2 -> io:format("Message ~p: ~p~n", [now(), A2]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A3 -> io:format("Message ~p: ~p~n", [now(), A3]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A4 -> io:format("Message ~p: ~p~n", [now(), A4]) after 1000 -> io:format("timeout ~p~n", [now()]) end, timer:sleep(1000), receive A5 -> io:format("Message ~p: ~p~n", [now(), A5]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A6 -> io:format("Message ~p: ~p~n", [now(), A6]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A7 -> io:format("Message ~p: ~p~n", [now(), A7]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A8 -> io:format("Message ~p: ~p~n", [now(), A8]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A9 -> io:format("Message ~p: ~p~n", [now(), A9]) after 1000 -> io:format("timeout ~p~n", [now()]) end, timer:sleep(10000), receive A10 -> io:format("Message ~p: ~p~n", [now(), A10]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A11 -> io:format("Message ~p: ~p~n", [now(), A11]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A12 -> io:format("Message ~p: ~p~n", [now(), A12]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A13 -> io:format("Message ~p: ~p~n", [now(), A13]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A14 -> io:format("Message ~p: ~p~n", [now(), A14]) after 1000 -> io:format("timeout ~p~n", [now()]) end, exec:kill(P, 9), receive A15 -> io:format("Message ~p: ~p~n", [now(), A15]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A16 -> io:format("Message ~p: ~p~n", [now(), A16]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A17 -> io:format("Message ~p: ~p~n", [now(), A17]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A18 -> io:format("Message ~p: ~p~n", [now(), A18]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A19 -> io:format("Message ~p: ~p~n", [now(), A19]) after 1000 -> io:format("timeout ~p~n", [now()]) end, timer:sleep(1000), receive A20 -> io:format("Message ~p: ~p~n", [now(), A20]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A21 -> io:format("Message ~p: ~p~n", [now(), A21]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A22 -> io:format("Message ~p: ~p~n", [now(), A22]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A23 -> io:format("Message ~p: ~p~n", [now(), A23]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A24 -> io:format("Message ~p: ~p~n", [now(), A24]) after 1000 -> io:format("timeout ~p~n", [now()]) end, q().' """ On Tue, Aug 6, 2013 at 10:49 PM, OvermindDL1 wrote: > Excepting this, is there any other better way to launch a program as > another user? I might just use ssh certs and ssh to localhost if not > unless any other ideas? > > > On Tue, Aug 6, 2013 at 4:22 AM, OvermindDL1 wrote: > >> I have been attempting to use ErlExec as I noticed that it had a few new >> features added six days ago to forward the stdout/stderr to an Erlang PID, >> however I am having some issue. The process that erlexec runs occasionally >> does not die when the Erlang VM dies, thus keeping it and its children >> running. >> >> Also I need a way to send commands back to the children, but I am either >> not seeing a command for that or have not figured out the format to send a >> message, which is difficult because of the main issue I have, when I have >> it launch a fairly large server software (that is not a daemon for whatever >> reason, hence why I am wrapping it), the Erlang shell seems to lose a great >> deal of input that I type in, only catching from every 1 in 4 I type to 1 >> in 400 characters I type, and it does not seem to matter based on the speed >> I hit keys or holding a key or anything of the sort, just purely on >> keypress down, and it remains like that until I kill the child process that >> I started through erlexec. >> >> Using the normal Erlang Ports works fine and has no such issues (other >> than the child not coming down with the VM either and no direct kill >> command when the child hangs, which it occasionally does, but os:cmd("kill >> ...") works around that well enough. >> >> Is ErlExec stable enough for high use? Or am I doing something wrong? >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peerst@REDACTED Wed Aug 7 12:49:53 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Wed, 7 Aug 2013 12:49:53 +0200 Subject: [erlang-questions] Testing a large, heterogeneous system References: Message-ID: On 2013-08-01 09:25:58 +0000, Jesper Louis Andersen said: > > I am a fan of building virtualized environments rather than mocking. > That is, you take something like > > http://www.vagrantup.com/ > > and then you build up the parts which your system consists of and make > sure you can deploy code inside your virtual environment. For Postgres, > look at pg_virtuelenv. Most of the C-nodes can perhaps be pushed into > the virtual environment. The specialized hardware is usually nasty and > you may have to mock that in order to get correct operation. > > But I usually hate mocking with a passion and much prefer to run > virtualized setups. It also forces you to think about > ease-of-development in general as it forces you to have an environment > on a box. For embedded stuff its often hard to emulate everything (certainly nice to have if its not too much work to set up). In these case I test the drivers that actually talk to the hardware devices on the actual hardware. Also using QuickCheck either with its C interface or if the driver is talked to from Erlang anyway through rpc from a non embedded node. For having the whole system in a box I write models for the hardware side and outside world in Erlang. Usually hooking in just above the driver I tested like mentioned above. These models have double use: I can use them to run tests without the special hardware just making it talk to the "world model" often on a separate node. When using QC I can also let it interrogate the state of the "world model" against the expected state. The world model can also be used as directly as state based QC model for testing the system from the outside (wherever this is). A extra use of these "world model" is that I can assemble complicated simulations of several system e.g. a whole factory plant using multiple stations each with its Erlang running embedded system. This is nice to have for larger system tests and also for building demos, measurement, and as development enviroment to fiddle with the higher level aspects. They can be considered Mocking but I don't see these models as such. They are not 100% of course so that they have in common with mocking. But when mocking one usually tries not to implement too much details. Weras with the simulation approach one tries to capture most of the essential system behaviour. BTW: Erlang is a great language for building simulations a neat convenience. From ulf@REDACTED Wed Aug 7 13:39:11 2013 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 7 Aug 2013 13:39:11 +0200 Subject: [erlang-questions] update_counter on steroids Message-ID: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> There was some discussion recently on teaching update_counter some new tricks. Here is a suggestion: Right now, the function takes (Tab, Key, UpOp | [UpOp]), where UpOp ::== {Pos, Incr} | {Pos, Incr, Threshold, WrapTo} I'd like to save the result of an operation in a temporary variable. An atomic reset, for example. (let's say counter c1 has the value 17): ets:update_counter(counters, c1, [{2, 0, '$1'}, {2, {'-', '$1'}}]) -> [17, 0] (Increment by 0 means we read the existing value, just like today). Why use this instead of simply ets:insert/2? Well, for one thing, we get the old value back, so it's an atomic read-reset. Sum counter: ets:update_counter(counters, c2, [{2,0,'$1'}, {3,0,'$2'}, {4, {'+', '$1', '$2'}}]) Wrap with parameterized threshold and reset value: ets:update_counter(counters, c3, [{3,0,'$1'}, {4,0,'$2'}, {2, Incr, '$1', '$2'}]) This assumes the following changes: - A number of temp variables, like in match specs. - A 3-tuple {Pos, Incr, SaveTo}, where SaveTo is a variable name (e.g. '$1') - A 5-tuple {Pos, Incr, Threshold, WrapTo, SaveTo} - The possibility to use, wherever an integer() is expected VariableName | {UnaryOp, VariableName} | {BinaryOp, Var1, Var2} We could also use this to e.g. store a value together with an increment operation. A reset with timestamp: TS = timestamp(), % in milliseconds ets:update_counter(counters, c4, [{2, 0, '$1'}, {2, {'-', '$1'}, {3, 0, '$2'}, {3, {'-', TS, '$2'}}]) ?a bit bizarre perhaps, but still. Comments? BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From thomasl_erlang@REDACTED Wed Aug 7 14:45:31 2013 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 7 Aug 2013 05:45:31 -0700 (PDT) Subject: [erlang-questions] update_counter on steroids In-Reply-To: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> References: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> Message-ID: <1375879531.53760.YahooMailNeo@web140104.mail.bf1.yahoo.com> 1. Conditionals and select (cond ? true : false) look like the next level.? 2. Use cases always come in handy. Here is one: It would be interesting if we could define realistic random number generators entirely in one update_counter operation. Linear congruential ones should fit with your proposal. 3. An old favorite of mine: instead of crashing when a counter does not exist, initialize it to the submitted value. (Implement as an option or a new BIF.) Usage: profiling, in my case. Best, Thomas >________________________________ > From: Ulf Wiger >To: erlang questions >Sent: Wednesday, August 7, 2013 1:39 PM >Subject: [erlang-questions] update_counter on steroids > > > >There was some discussion recently on teaching update_counter some new tricks. > >Here is a suggestion: > >Right now, the function takes (Tab, Key, UpOp | [UpOp]), where >? UpOp ::== {Pos, Incr} | {Pos, Incr, Threshold, WrapTo} > >I'd like to save the result of an operation in a temporary variable. > >An atomic reset, for example. (let's say counter c1 has the value 17): > >ets:update_counter(counters, c1, [{2, 0, '$1'}, {2, {'-', '$1'}}]) -> >? [17, 0] > >(Increment by 0 means we read the existing value, just like today). > >Why use this instead of simply ets:insert/2? Well, for one thing, we get the old value back, so it's an atomic read-reset. > >Sum counter: > >ets:update_counter(counters, c2, [{2,0,'$1'}, {3,0,'$2'}, {4, {'+', '$1', '$2'}}]) > >Wrap with parameterized threshold and reset value: > >ets:update_counter(counters, c3, [{3,0,'$1'}, {4,0,'$2'}, {2, Incr, '$1', '$2'}]) > >This assumes the following changes: > >- A number of temp variables, like in match specs. >- A 3-tuple? {Pos, Incr, SaveTo}, where SaveTo is a variable name (e.g. '$1') >- A 5-tuple? {Pos, Incr, Threshold, WrapTo, SaveTo} >- The possibility to use, wherever an integer() is expected >? VariableName | {UnaryOp, VariableName} | {BinaryOp, Var1, Var2} > >We could also use this to e.g. store a value together with an increment operation. A reset with timestamp: > >TS = timestamp(),? % in milliseconds >ets:update_counter(counters, c4, [{2, 0, '$1'}, {2, {'-', '$1'}, {3, 0, '$2'}, {3, {'-', TS, '$2'}}]) > >?a bit bizarre perhaps, but still. > >Comments? > >BR, >Ulf W > > >Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. >http://feuerlabs.com > > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Wed Aug 7 14:58:35 2013 From: donpedrothird@REDACTED (Evgeny M) Date: Wed, 7 Aug 2013 05:58:35 -0700 (PDT) Subject: [erlang-questions] update_counter on steroids In-Reply-To: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> References: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> Message-ID: <9a814973-7aa9-4914-bc40-d7647fc4a632@googlegroups.com> I miss an ability to increment more than one counter for a given key. Would be very useful for statistical aggregate data. Second thing I miss is a non-dirty update_counter. It doesn't look very useful, but would allow to use non-dirty update_counter in, for example, mnesia_frag context. ?????, 7 ??????? 2013 ?., 15:39:11 UTC+4 ???????????? Ulf Wiger ???????: > > > There was some discussion recently on teaching update_counter some new > tricks. > > Here is a suggestion: > > Right now, the function takes (Tab, Key, UpOp | [UpOp]), where > UpOp ::== {Pos, Incr} | {Pos, Incr, Threshold, WrapTo} > > I'd like to save the result of an operation in a temporary variable. > > An atomic reset, for example. (let's say counter c1 has the value 17): > > ets:update_counter(counters, c1, [{2, 0, '$1'}, {2, {'-', '$1'}}]) -> > [17, 0] > > (Increment by 0 means we read the existing value, just like today). > > Why use this instead of simply ets:insert/2? Well, for one thing, we get > the old value back, so it's an atomic read-reset. > > Sum counter: > > ets:update_counter(counters, c2, [{2,0,'$1'}, {3,0,'$2'}, {4, {'+', '$1', > '$2'}}]) > > Wrap with parameterized threshold and reset value: > > ets:update_counter(counters, c3, [{3,0,'$1'}, {4,0,'$2'}, {2, Incr, '$1', > '$2'}]) > > This assumes the following changes: > > - A number of temp variables, like in match specs. > - A 3-tuple {Pos, Incr, SaveTo}, where SaveTo is a variable name (e.g. > '$1') > - A 5-tuple {Pos, Incr, Threshold, WrapTo, SaveTo} > - The possibility to use, wherever an integer() is expected > VariableName | {UnaryOp, VariableName} | {BinaryOp, Var1, Var2} > > We could also use this to e.g. store a value together with an increment > operation. A reset with timestamp: > > TS = timestamp(), % in milliseconds > ets:update_counter(counters, c4, [{2, 0, '$1'}, {2, {'-', '$1'}, {3, 0, > '$2'}, {3, {'-', TS, '$2'}}]) > > ?a bit bizarre perhaps, but still. > > Comments? > > BR, > Ulf W > > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Wed Aug 7 15:03:23 2013 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 7 Aug 2013 15:03:23 +0200 Subject: [erlang-questions] update_counter on steroids In-Reply-To: <9a814973-7aa9-4914-bc40-d7647fc4a632@googlegroups.com> References: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> <9a814973-7aa9-4914-bc40-d7647fc4a632@googlegroups.com> Message-ID: <221F0C9A-D783-4E78-9130-0EACAF12CF0F@feuerlabs.com> On 7 Aug 2013, at 14:58, Evgeny M wrote: > I miss an ability to increment more than one counter for a given key. Actually, you *can* do that already. Eshell V5.10.2 (abort with ^G) 1> T = ets:new(t, [set]). 16400 2> ets:insert(T, {c, 0, 0, 0}). true 3> ets:update_counter(T, c, [{2,1},{3,10},{4,100}]). [1,10,100] 4> ets:lookup(T, c). [{c,1,10,100}] What I wanted to add was to capture values in previous operations and use them for derived operations later in the same atomic update. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Wed Aug 7 15:08:27 2013 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 7 Aug 2013 15:08:27 +0200 Subject: [erlang-questions] update_counter on steroids In-Reply-To: <1375879531.53760.YahooMailNeo@web140104.mail.bf1.yahoo.com> References: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> <1375879531.53760.YahooMailNeo@web140104.mail.bf1.yahoo.com> Message-ID: <66D79DCC-BE65-4779-BA08-47FD35F9B5FB@feuerlabs.com> On 7 Aug 2013, at 14:45, Thomas Lindgren wrote: > 1. Conditionals and select (cond ? true : false) look like the next level. Syntax suggestion? Would it be best to approach that using match spec syntax, using the action body to produce a list of update ops? I can easily visualize the logic in an Erlang program, but it might get a bit messy in C. :) (Here I reveal my own preferences, of course; I judge it based on how hard it would be for *me* to implement it in C.) BR, Ulf Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Wed Aug 7 17:56:26 2013 From: donpedrothird@REDACTED (Evgeny M) Date: Wed, 7 Aug 2013 08:56:26 -0700 (PDT) Subject: [erlang-questions] update_counter on steroids In-Reply-To: <221F0C9A-D783-4E78-9130-0EACAF12CF0F@feuerlabs.com> References: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> <9a814973-7aa9-4914-bc40-d7647fc4a632@googlegroups.com> <221F0C9A-D783-4E78-9130-0EACAF12CF0F@feuerlabs.com> Message-ID: Sorry, I thought this post is about mnesia dirty_update_counters My bad ?????, 7 ??????? 2013 ?., 17:03:23 UTC+4 ???????????? Ulf Wiger ???????: > > > On 7 Aug 2013, at 14:58, Evgeny M > > wrote: > > I miss an ability to increment more than one counter for a given key. > > > Actually, you *can* do that already. > > Eshell V5.10.2 (abort with ^G) > 1> T = ets:new(t, [set]). > 16400 > 2> ets:insert(T, {c, 0, 0, 0}). > true > 3> ets:update_counter(T, c, [{2,1},{3,10},{4,100}]). > [1,10,100] > 4> ets:lookup(T, c). > [{c,1,10,100}] > > What I wanted to add was to capture values in previous operations > and use them for derived operations later in the same atomic update. > > BR, > Ulf W > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw@REDACTED Wed Aug 7 18:34:16 2013 From: flw@REDACTED (Florian Waas) Date: Wed, 7 Aug 2013 09:34:16 -0700 Subject: [erlang-questions] SSL/TLS renegotiation crash triggered by setopts In-Reply-To: References: Message-ID: Ingela, Thanks for the quick response -- will use my "provisional" fix for the time being! -fl. On Wed, Aug 7, 2013 at 3:04 AM, Ingela Andin wrote: > Hi! > > I think you found a bug setopts should indeed not change the state, but if > you happen to be in a renegotiation you could get a handshake packet that > changes the state, as active is an emulated option and the fsm-process > internally uses active once. We will fix this for the next release. > > > Regards Ingela Erlang/OTP team - Ericsson AB > > > On Wed, Aug 7, 2013 at 8:15 AM, Florian Waas wrote: >> >> Hi, >> Reading from a Postgres server tls_connection's gen_fsm crashes on me >> during TLS renegotiation with below error report (small snippet of >> most relevant parts only due to size). >> >> I'm setting the socket into {active, once} after each packet; the >> server kicks off renegotiation every 512MB. Looks like the problem >> occurs when setopts is called during renegotiation. When hammering the >> system I hit it pretty much every time the server renegotiates -- >> happens in both R15 and R16. >> >> Quick hack in tls_connection.erl:995 to allow the transition from >> 'connection' to 'hello' in handle_sync_event instead of insisting that >> setopts doesn't change states gets me out of this pickle... >> >> >> Anybody seen this problem? Clean(er) workarounds available? >> >> Thanks, >> -fl. >> >> >> >> >> =ERROR REPORT==== 6-Aug-2013::21:26:38 === >> ** State machine <0.368.0> terminating >> ** Last message in was {'$gen_sync_all_state_event', >> {<0.364.0>,#Ref<0.0.0.67905>}, >> {set_opts,[{active,once}]}} >> ** When State == connection >> ** Data == {state,client, >> {#Ref<0.0.0.247>,<0.364.0>}, >> [...] >> undefined, >> {[],[]}, >> false,true,false,undefined,undefined} >> ** Reason for termination = >> ** {{case_clause, >> {next_state,hello, >> {state,client, >> [...] >> >> [{tls_connection,handle_sync_event,4, >> [{file,"tls_connection.erl"},{line,994}]}, >> {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,505}]}, >> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From 0x6e6562@REDACTED Thu Aug 8 00:39:36 2013 From: 0x6e6562@REDACTED (Ben Hood) Date: Wed, 7 Aug 2013 23:39:36 +0100 Subject: [erlang-questions] ECC public key example In-Reply-To: References: Message-ID: Hey Ingela, On Wed, Aug 7, 2013 at 2:57 PM, Ingela Andin wrote: > Thank you for the example I will use it to extend public_key tests. Also the > OTP prefix shall > not be there. > > Regards Ingela Erlang/OTP team - Ericsson AB Thanks for the heads up. Cheers, Ben From serge@REDACTED Thu Aug 8 05:15:01 2013 From: serge@REDACTED (Serge Aleynikov) Date: Wed, 7 Aug 2013 23:15:01 -0400 Subject: [erlang-questions] ErlExec stdin and other questions In-Reply-To: References: Message-ID: I am currently on vacation but will try to look into this some time later next week when I return. The stdout/stderr redirection to Erlang is not yet "production" quality - rather an experimental feature introduced last week. I am also working on adding stdin support, so that you could send messages to an OS process's stdin. This should also be available in a week after I return. As far as I know erlexec is used in several production systems, and except for the stdout/stderr redirection to Erlang is stable. Upon exiting the VM, the erl-exec port process started by exec application receives an end-of-file on the stdin pipe, and sets off the alarm, which guarantees that it'll be killed after a timeout if it hasn't finished the cleanup (termination) of managed processes. If you don't observe that behavior, please provide more information regarding OS you are running, and ideally some simple test cases. On Wed, Aug 7, 2013 at 3:36 AM, OvermindDL1 wrote: > Created a complete test-case example of my issue with erlexec, and I ran > this same script on 3 different computers running variations of debian > versions. Feel free to toss in application:start(sasl), at the top of the > erlang script area if you wish, and yes the (minecraft) server software > uses stderr instead of stdout, no I do not know why, and yes as you can see > it does not return anything to the erlang system until it is killed on all > the systems I tested. Using Erlang's ports you get each line as a message > as that is how the server flushes it and it is instant. > > > Yes it is ugly, but it is pared down from what it grew from and no need to > re-do the receive's better in such a test. > test_exec_mc.sh > """ > #!/bin/sh > > [ -d erlexec ] || git clone https://github.com/saleyn/erlexec.git > > cd erlexec > > rebar compile > > [ -f 'minecraft_server.1.6.2.jar' ] || wget ' > https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar > ' > > erl -pa $PWD/ebin -eval ' > application:start(exec), > {_, P, _} = exec:run_link("java -Xms1G -Xmx1G -jar > minecraft_server.1.6.2.jar nogui", [{stdout, self()}, {stderr, self()}]), > timer:sleep(1000), > receive A0 -> io:format("Message ~p: ~p~n", [now(), A0]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A1 -> io:format("Message ~p: ~p~n", [now(), A1]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A2 -> io:format("Message ~p: ~p~n", [now(), A2]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A3 -> io:format("Message ~p: ~p~n", [now(), A3]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A4 -> io:format("Message ~p: ~p~n", [now(), A4]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > timer:sleep(1000), > receive A5 -> io:format("Message ~p: ~p~n", [now(), A5]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A6 -> io:format("Message ~p: ~p~n", [now(), A6]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A7 -> io:format("Message ~p: ~p~n", [now(), A7]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A8 -> io:format("Message ~p: ~p~n", [now(), A8]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A9 -> io:format("Message ~p: ~p~n", [now(), A9]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > timer:sleep(10000), > receive A10 -> io:format("Message ~p: ~p~n", [now(), A10]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A11 -> io:format("Message ~p: ~p~n", [now(), A11]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A12 -> io:format("Message ~p: ~p~n", [now(), A12]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A13 -> io:format("Message ~p: ~p~n", [now(), A13]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A14 -> io:format("Message ~p: ~p~n", [now(), A14]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > exec:kill(P, 9), > receive A15 -> io:format("Message ~p: ~p~n", [now(), A15]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A16 -> io:format("Message ~p: ~p~n", [now(), A16]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A17 -> io:format("Message ~p: ~p~n", [now(), A17]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A18 -> io:format("Message ~p: ~p~n", [now(), A18]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A19 -> io:format("Message ~p: ~p~n", [now(), A19]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > timer:sleep(1000), > receive A20 -> io:format("Message ~p: ~p~n", [now(), A20]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A21 -> io:format("Message ~p: ~p~n", [now(), A21]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A22 -> io:format("Message ~p: ~p~n", [now(), A22]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A23 -> io:format("Message ~p: ~p~n", [now(), A23]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > receive A24 -> io:format("Message ~p: ~p~n", [now(), A24]) after 1000 -> > io:format("timeout ~p~n", [now()]) end, > q().' > """ > > > > > > On Tue, Aug 6, 2013 at 10:49 PM, OvermindDL1 wrote: > >> Excepting this, is there any other better way to launch a program as >> another user? I might just use ssh certs and ssh to localhost if not >> unless any other ideas? >> >> >> On Tue, Aug 6, 2013 at 4:22 AM, OvermindDL1 wrote: >> >>> I have been attempting to use ErlExec as I noticed that it had a few new >>> features added six days ago to forward the stdout/stderr to an Erlang PID, >>> however I am having some issue. The process that erlexec runs occasionally >>> does not die when the Erlang VM dies, thus keeping it and its children >>> running. >>> >>> Also I need a way to send commands back to the children, but I am either >>> not seeing a command for that or have not figured out the format to send a >>> message, which is difficult because of the main issue I have, when I have >>> it launch a fairly large server software (that is not a daemon for whatever >>> reason, hence why I am wrapping it), the Erlang shell seems to lose a great >>> deal of input that I type in, only catching from every 1 in 4 I type to 1 >>> in 400 characters I type, and it does not seem to matter based on the speed >>> I hit keys or holding a key or anything of the sort, just purely on >>> keypress down, and it remains like that until I kill the child process that >>> I started through erlexec. >>> >>> Using the normal Erlang Ports works fine and has no such issues (other >>> than the child not coming down with the VM either and no direct kill >>> command when the child hangs, which it occasionally does, but os:cmd("kill >>> ...") works around that well enough. >>> >>> Is ErlExec stable enough for high use? Or am I doing something wrong? >>> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomasl_erlang@REDACTED Thu Aug 8 07:27:17 2013 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Wed, 7 Aug 2013 22:27:17 -0700 (PDT) Subject: [erlang-questions] update_counter on steroids In-Reply-To: <66D79DCC-BE65-4779-BA08-47FD35F9B5FB@feuerlabs.com> References: <64245B5A-B63D-4C0C-A62F-80884C696524@feuerlabs.com> <1375879531.53760.YahooMailNeo@web140104.mail.bf1.yahoo.com> <66D79DCC-BE65-4779-BA08-47FD35F9B5FB@feuerlabs.com> Message-ID: <1375939637.75887.YahooMailNeo@web140103.mail.bf1.yahoo.com> Specifying multiple match spec clauses sounds pleasingly erlangish, so I'm for it.? Best, Thomas >________________________________ > From: Ulf Wiger > > > >On 7 Aug 2013, at 14:45, Thomas Lindgren wrote: > >1. Conditionals and select (cond ? true : false) look like the next level.? > > >Syntax suggestion? Would it be best to approach that using match spec syntax, >using the action body to produce a list of update ops? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From overminddl1@REDACTED Thu Aug 8 09:16:42 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Thu, 8 Aug 2013 01:16:42 -0600 Subject: [erlang-questions] ErlExec stdin and other questions In-Reply-To: References: Message-ID: I actually already left an issue on Github with such an example script yesterday, and system information. :-) On Aug 7, 2013 9:15 PM, "Serge Aleynikov" wrote: > I > am currently on vacation but will try to look into this some time later > next week when I return. > > The stdout/stderr redirection to Erlang is not yet "production" quality - > rather an experimental feature introduced last week. I am also working on > adding stdin support, so that you could send messages to an OS process's > stdin. This should also be available in a week after I return. > > As far as I know erlexec is used in several production systems, and except > for the stdout/stderr redirection to Erlang is stable. > > Upon exiting the VM, the erl-exec port process started by exec application > receives an end-of-file on the stdin pipe, and sets off the alarm, which > guarantees that it'll be killed after a timeout if it hasn't finished the > cleanup (termination) of managed processes. If you don't observe that > behavior, please provide more information regarding OS you are running, and > ideally some simple test cases. > > > On Wed, Aug 7, 2013 at 3:36 AM, OvermindDL1 wrote: > >> Created a complete test-case example of my issue with erlexec, and I ran >> this same script on 3 different computers running variations of debian >> versions. Feel free to toss in application:start(sasl), at the top of the >> erlang script area if you wish, and yes the (minecraft) server software >> uses stderr instead of stdout, no I do not know why, and yes as you can see >> it does not return anything to the erlang system until it is killed on all >> the systems I tested. Using Erlang's ports you get each line as a message >> as that is how the server flushes it and it is instant. >> >> >> Yes it is ugly, but it is pared down from what it grew from and no need >> to re-do the receive's better in such a test. >> test_exec_mc.sh >> """ >> #!/bin/sh >> >> [ -d erlexec ] || git clone https://github.com/saleyn/erlexec.git >> >> cd erlexec >> >> rebar compile >> >> [ -f 'minecraft_server.1.6.2.jar' ] || wget ' >> https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar >> ' >> >> erl -pa $PWD/ebin -eval ' >> application:start(exec), >> {_, P, _} = exec:run_link("java -Xms1G -Xmx1G -jar >> minecraft_server.1.6.2.jar nogui", [{stdout, self()}, {stderr, self()}]), >> timer:sleep(1000), >> receive A0 -> io:format("Message ~p: ~p~n", [now(), A0]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A1 -> io:format("Message ~p: ~p~n", [now(), A1]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A2 -> io:format("Message ~p: ~p~n", [now(), A2]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A3 -> io:format("Message ~p: ~p~n", [now(), A3]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A4 -> io:format("Message ~p: ~p~n", [now(), A4]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> timer:sleep(1000), >> receive A5 -> io:format("Message ~p: ~p~n", [now(), A5]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A6 -> io:format("Message ~p: ~p~n", [now(), A6]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A7 -> io:format("Message ~p: ~p~n", [now(), A7]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A8 -> io:format("Message ~p: ~p~n", [now(), A8]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A9 -> io:format("Message ~p: ~p~n", [now(), A9]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> timer:sleep(10000), >> receive A10 -> io:format("Message ~p: ~p~n", [now(), A10]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A11 -> io:format("Message ~p: ~p~n", [now(), A11]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A12 -> io:format("Message ~p: ~p~n", [now(), A12]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A13 -> io:format("Message ~p: ~p~n", [now(), A13]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A14 -> io:format("Message ~p: ~p~n", [now(), A14]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> exec:kill(P, 9), >> receive A15 -> io:format("Message ~p: ~p~n", [now(), A15]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A16 -> io:format("Message ~p: ~p~n", [now(), A16]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A17 -> io:format("Message ~p: ~p~n", [now(), A17]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A18 -> io:format("Message ~p: ~p~n", [now(), A18]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A19 -> io:format("Message ~p: ~p~n", [now(), A19]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> timer:sleep(1000), >> receive A20 -> io:format("Message ~p: ~p~n", [now(), A20]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A21 -> io:format("Message ~p: ~p~n", [now(), A21]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A22 -> io:format("Message ~p: ~p~n", [now(), A22]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A23 -> io:format("Message ~p: ~p~n", [now(), A23]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> receive A24 -> io:format("Message ~p: ~p~n", [now(), A24]) after 1000 -> >> io:format("timeout ~p~n", [now()]) end, >> q().' >> """ >> >> >> >> >> >> On Tue, Aug 6, 2013 at 10:49 PM, OvermindDL1 wrote: >> >>> Excepting this, is there any other better way to launch a program as >>> another user? I might just use ssh certs and ssh to localhost if not >>> unless any other ideas? >>> >>> >>> On Tue, Aug 6, 2013 at 4:22 AM, OvermindDL1 wrote: >>> >>>> I have been attempting to use ErlExec as I noticed that it had a few >>>> new features added six days ago to forward the stdout/stderr to an Erlang >>>> PID, however I am having some issue. The process that erlexec runs >>>> occasionally does not die when the Erlang VM dies, thus keeping it and its >>>> children running. >>>> >>>> Also I need a way to send commands back to the children, but I am >>>> either not seeing a command for that or have not figured out the format to >>>> send a message, which is difficult because of the main issue I have, when I >>>> have it launch a fairly large server software (that is not a daemon for >>>> whatever reason, hence why I am wrapping it), the Erlang shell seems to >>>> lose a great deal of input that I type in, only catching from every 1 in 4 >>>> I type to 1 in 400 characters I type, and it does not seem to matter based >>>> on the speed I hit keys or holding a key or anything of the sort, just >>>> purely on keypress down, and it remains like that until I kill the child >>>> process that I started through erlexec. >>>> >>>> Using the normal Erlang Ports works fine and has no such issues (other >>>> than the child not coming down with the VM either and no direct kill >>>> command when the child hangs, which it occasionally does, but os:cmd("kill >>>> ...") works around that well enough. >>>> >>>> Is ErlExec stable enough for high use? Or am I doing something wrong? >>>> >>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lcastro@REDACTED Thu Aug 8 09:59:09 2013 From: lcastro@REDACTED (Laura M. Castro) Date: Thu, 8 Aug 2013 09:59:09 +0200 Subject: [erlang-questions] ACM SIGPLAN Erlang Workshop Call for Participation Message-ID: ---------------------------------------------------------------------- CALL FOR PARTICIPATION Erlang Workshop 2013 12th ACM SIGPLAN Erlang Workshop September 28, 2013 Boston, Massachusetts (the day after ICFP 2013) http://www.erlang.org/workshop/2013/ ---------------------------------------------------------------------- This workshop brings together the open source, academic, and industrial programming communities of Erlang. It aims to enable participants to familiarize themselves with recent developments on new techniques and tools tailored to Erlang, novel applications, draw lessons from users' experiences and identify research problems and common areas relevant to the practice of Erlang and functional programming. ------------ Registration ------------ Deadline for early registration: 22 August 2013 Web site: https://regmaster3.com/2013conf/ICFP13/register.php This is the registration site for ICFP 2013 and all the affiliated workshops including Erlang Workshop 2013. ------------------------------------------ Keynote by Justin Sheehy (CTO Basho Technologies) ------------------------------------------ At this year's Erlang Workshop, we have the pleasure of presenting Justin Sheehy as our keynote speaker. Justin is a prominent member of the Erlang community not only due to his leadership in Erlang development efforts to create and maintain Webmachine and Riak (both are very well known and highly respected Erlang systems); but also for his efforts to promote Erlang through conference speaking, social media, and as a member of the Industrial Erlang Users Group. He has a very high level of knowledge and wisdom regarding distributed systems, and his CTO position affords him a view of both industry and academia that few others have. ---------------------- List of Accepted Talks ---------------------- Session: Scalability And Performance Using Many-core Coprocessor to Boost up Erlang VM (by Siyao Zheng, Xiang Long, Jingwei Yang) On the Scalability of the Erlang Term Storage (by David Klaftenegger, Konstantinos Sagonas, Kjell Winblad) Riak PG: Distributed Process Groups on Dynamo-style Distributed Storage (by Christopher Meiklejohn) Multicore Profiling for Erlang Programs Using Percept2 (by Huiqing Li, Simon Thompson) Session: Extending Erlang Software Agents Mobility using Process Migration Mechanism in Distributed Erlang (by Micha? Piotrowski, Wojciech Turek) Actor Scheduling for Multicore Hierarchical Memory Platforms (by Emilio Francesquini, Alfredo Goldman, Jean-Fran?ois Mehaut) Extending Erlang by Utilising RefactorErl (by D?niel Horp?csi) Session: Testing Turning Web Services Descriptions into QuickCheck Models for Automatic Testing (by Mac?as L?pez, Henrique Ferreiro, Miguel A. Francisco, Laura M. Castro) Testing blocking operations with QuickCheck's component library (by Ulf Norell, Hans Svensson, Thomas Arts) Poster Session Scalable Persistent Storage for Erlang: Theory and Practice (by Amir Ghaffari, Natalia Chechina, Phil Trinder, Jon Meredith) Towards an Abstraction for Remote Evaluation Services in Erlang (by Adrian Francalanza, Tyron Zerafa) Property-based testing of RESTful Web Services (by Pablo Lamela Seijas, Simon Thompson, Huiqing Li) --------------------- Program Committee --------------------- Steve Vinoski, Basho Technologies, USA (Workshop Chair) Laura M. Castro, University of A Coru?a, Spain (Program Chair) Lars-Ake Fredlund, Universidad Polit?cnica de Madrid, Spain Kevin Hammond, University of St. Andrews, UK Torben Hoffman, Erlang Solutions Limited, UK Zolt?n Horv?th, E?tv?s Lor?nd University, Hungary Kenneth Lundin, Ericsson AB, Sweden Micka?l R?mond, ProcessOne, France Kenji Rikitake, Basho Japan KK, Japan Simon Thompson, University of Kent, UK -- Laura M. Castro Department of Computer Science Universidade da Coru?a (Spain) http://www.madsgroup.org/staff/laura From co7eb@REDACTED Thu Aug 8 19:34:34 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Thu, 8 Aug 2013 13:34:34 -0400 Subject: [erlang-questions] =?iso-8859-1?q?EDOC_and_remote_type=27s_specif?= =?iso-8859-1?q?ication?= Message-ID: <000001ce945d$914412b0$b3cc3810$@frcuba.co.cu> Hi all, I think this has to do a little with what Nikolaos Bezirgiannis was saying about edoc generation and types, I have this little ?thing? when I was implementing a log module for a project I am developing. For example I use a remote type specification for my own function specification like this: -type log_opt() :: show | write | log. %% Functions specification. %% @doc Logs events information for the system. %% %% When Opts is: %% %%
    %%
  • show logs to the system console.
    • %%
  • write logs to the solid state storage.
    • %%
  • log logs to both system console and solid state storage.
    • %%
  • error logs errors to both system console and solid state storage %% in a special error format.
    • %%
-spec log_event(Opts, FormattedStr, Args) -> ok | error when Opts :: log_opt(), FormattedStr :: io:format(), Args :: term(). The one in red (io:format()) is an exported type from io module, once I generate the documentation for the module, using edoc:file/1 in this case this function is deprecated but I think it works for the propose, this generate a html documentation including links for the local type log_opt() and the remote type io:format(), the link for my own local type is well, the problem is with the imported type (remote one), when I click the io:format() link it of course try to find it in the working directory of my project, how (if it?s worth) I can refer to the truly type definition in the io module documentation? -------------- next part -------------- An HTML attachment was scrubbed... URL: From yoursurrogategod@REDACTED Fri Aug 9 03:38:32 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Thu, 8 Aug 2013 21:38:32 -0400 Subject: [erlang-questions] How to access CouchDB? Message-ID: Hello, I have a small database that's running right now in CouchDB, it's called sample_database. What I'd like to do is to read, create, delete, update documents and run some of the views inside from an Erlang app. I looked online and found this example: http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ >From that, I proceeded to make the following little script: -module(test). -import(queue,[in/1,out/1,new/0]). -export([start/0]). -include("../couch_db.hrl"). -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). start() -> io:format("Hello world!~n~n"), couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). However, when I run this, I get the following error: 18> test:start(). Hello world! ** exception error: undefined function couch_db:open/2 It's obvious I'm not including a library of some sort. I'm curious how I can connect to my little database in the simplest way possible just to start with. Has anyone done this before? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Fri Aug 9 07:36:37 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 9 Aug 2013 07:36:37 +0200 Subject: [erlang-questions] How to access CouchDB? In-Reply-To: References: Message-ID: <1376026597.4898.4.camel@sekic1152.release> Greetings, Where is couch_db.beam located on your system? The Erlang equivalent of including libraries is to do: "erl -pz /path/to/beam_directory" when starting Erlang. bengt On Thu, 2013-08-08 at 21:38 -0400, Yves S. Garret wrote: > Hello, > > > I have a small database that's running right now in CouchDB, it's > called sample_database. What I'd > like to do is to read, create, delete, update documents and run some > of the views inside from an > Erlang app. I looked online and found this example: > http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ > > > > From that, I proceeded to make the following little script: > -module(test). > > > -import(queue,[in/1,out/1,new/0]). > -export([start/0]). > -include("../couch_db.hrl"). > > > -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = > [<<"_admin">>]}}). > > > start() -> > io:format("Hello world!~n~n"), > > > couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). > > > However, when I run this, I get the following error: > 18> test:start(). > Hello world! > > > ** exception error: undefined function couch_db:open/2 > > > It's obvious I'm not including a library of some sort. I'm curious > how I can connect to my little > database in the simplest way possible just to start with. Has anyone > done this before? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From hc@REDACTED Fri Aug 9 09:41:12 2013 From: hc@REDACTED (H.C. v. Stockhausen) Date: Fri, 9 Aug 2013 09:41:12 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? Message-ID: Hello, I need a DB backend for my application but I'd like to be able to swap it out for different DBs if I choose so later on. I would like to code against an interface and tell the application what specific backend to use through config rather than code changes. Is there a pattern for doing that? Hot code upgrades and multi-node are not a real concern at this time - mostly, since I have no experience yet with either - however if doing it right means taking that into account too I'd also like to learn more about that. I am thinking of defining a custom behaviour ("my_crud" perhaps), then to implement it for various DBs and to also let a config driven adapter implement it that I then use to throughout the code to talk to the DB layer. For example, using Mnesia and AWS DynamoDB: - my_crud.erl (behaviour) - my_db_mnesia.erl (implements behaviour) - my_db_dynamo.erl (implements behaviour) - my_db.erl (configurable adapter that also implements behaviour) - my.config my_db:insert(Key, Value). Is that a reasonable approach that makes proper use of Erlang and behaviours or is this just not how one should do it? Thank you for any help & best regards, Hans From qwertymaniac@REDACTED Fri Aug 9 10:28:08 2013 From: qwertymaniac@REDACTED (Harsh J) Date: Fri, 9 Aug 2013 13:58:08 +0530 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: Perhaps you are looking for an ORM. Something like https://github.com/evanmiller/boss_db could probably some your need? On Aug 9, 2013 1:51 PM, "H.C. v. Stockhausen" wrote: > Hello, > > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > > I would like to code against an interface and tell the application > what specific backend to use through config rather than code changes. > > Is there a pattern for doing that? Hot code upgrades and multi-node > are not a real concern at this time - mostly, since I have no > experience yet with either - however if doing it right means taking > that into account too I'd also like to learn more about that. > > I am thinking of defining a custom behaviour ("my_crud" perhaps), then > to implement it for various DBs and to also let a config driven > adapter implement it that I then use to throughout the code to talk to > the DB layer. > > For example, using Mnesia and AWS DynamoDB: > > - my_crud.erl (behaviour) > - my_db_mnesia.erl (implements behaviour) > - my_db_dynamo.erl (implements behaviour) > - my_db.erl (configurable adapter that also implements behaviour) > - my.config > > my_db:insert(Key, Value). > > Is that a reasonable approach that makes proper use of Erlang and > behaviours or is this just not how one should do it? > > Thank you for any help & best regards, > Hans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookchrisd@REDACTED Fri Aug 9 10:30:19 2013 From: cookchrisd@REDACTED (Chris Cook) Date: Fri, 9 Aug 2013 09:30:19 +0100 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: Morning Hans, I abstracted the interface out to a separate layer when I had to do this and set the chosen db as state in startup, so you'd have something like: my_interface:get_content/2 -> (State#state.db):get_content/2 -> my_db_mnesia:get_content/2 -> my_db_dynamo:get_content/2 -> my_db:get_content/2 hope this kind of help and along the right lines. Regards Chris On 9 August 2013 08:41, H.C. v. Stockhausen wrote: > Hello, > > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > > I would like to code against an interface and tell the application > what specific backend to use through config rather than code changes. > > Is there a pattern for doing that? Hot code upgrades and multi-node > are not a real concern at this time - mostly, since I have no > experience yet with either - however if doing it right means taking > that into account too I'd also like to learn more about that. > > I am thinking of defining a custom behaviour ("my_crud" perhaps), then > to implement it for various DBs and to also let a config driven > adapter implement it that I then use to throughout the code to talk to > the DB layer. > > For example, using Mnesia and AWS DynamoDB: > > - my_crud.erl (behaviour) > - my_db_mnesia.erl (implements behaviour) > - my_db_dynamo.erl (implements behaviour) > - my_db.erl (configurable adapter that also implements behaviour) > - my.config > > my_db:insert(Key, Value). > > Is that a reasonable approach that makes proper use of Erlang and > behaviours or is this just not how one should do it? > > Thank you for any help & best regards, > Hans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ddosia@REDACTED Fri Aug 9 13:56:48 2013 From: ddosia@REDACTED (Daniil Churikov) Date: Fri, 9 Aug 2013 04:56:48 -0700 (PDT) Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> Does anybody know could *dialyzer* break through such dynamic module construction, like *Chris Cook* wrote? Is it able to know that * State#state.db* module do not export *get_content/2* function, or it's contract is wrong? -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookchrisd@REDACTED Fri Aug 9 14:40:37 2013 From: cookchrisd@REDACTED (Chris Cook) Date: Fri, 9 Aug 2013 13:40:37 +0100 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> Message-ID: Afternoon Daniil, I'm not sure what your getting at, the module that the atom that State#state.db refers to must export the calling function. Take a look at this link [0] (Only the atomic names bit the second bit has been removed from erlang) for more information about the basics of what is going on. The State is a variable and evaluated like one that's why it is in (). maybe an easier way of coding it would be. -record(state, {db=mneisa_module}); M = State#state.db M:get_content(Tbl, Id) -module(mneisa_module) -export([get_content/2]). -spec(...) get_content(Tbl, Id) -> .... [0] http://userprimary.net/posts/2009/03/04/two-erlang-finds-atomic-names-and-parameterized-modules/ Regards Chris On 9 August 2013 12:56, Daniil Churikov wrote: > Does anybody know could *dialyzer* break through such dynamic module > construction, like *Chris Cook* wrote? Is it able to know that * > State#state.db* module do not export *get_content/2* function, or it's > contract is wrong? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ddosia@REDACTED Fri Aug 9 14:49:59 2013 From: ddosia@REDACTED (Daniil Churikov) Date: Fri, 9 Aug 2013 05:49:59 -0700 (PDT) Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> Message-ID: Hello Chris. I am speaking about static analysis tool named "Dialyzer". This tool is intended to be used as "error-finder". And what I am interested in is whether this tool will be able to understand what module you substitute in * State#state.db* and make warning that this module is not properly written (i.e. miss some functions or this functions will not be able to handle what you will feed them). -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookchrisd@REDACTED Fri Aug 9 14:57:22 2013 From: cookchrisd@REDACTED (Chris Cook) Date: Fri, 9 Aug 2013 13:57:22 +0100 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> Message-ID: I cant remember if it throw an error or not, it probably did and probably glossed over it, if I get chance later this afternoon or if someone gets chance try it and see, only need two module one function in each and call it with a dynamic function, could use pass an atom is in as a variable in to the calling function with V:F(), doesnt need to be in state. On 9 August 2013 13:49, Daniil Churikov wrote: > Hello Chris. > > I am speaking about static analysis tool named "Dialyzer". This tool is > intended to be used as "error-finder". And what I am interested in is > whether this tool will be able to understand what module you substitute in > *State#state.db* and make warning that this module is not properly > written (i.e. miss some functions or this functions will not be able to > handle what you will feed them). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Aug 9 14:58:56 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 09 Aug 2013 14:58:56 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> Message-ID: <5204E790.8080003@ninenines.eu> Whether Dialyzer can do that or not is a bit irrelevant, as this kind of error is quickly apparent when your tests immediately fail. On 08/09/2013 02:49 PM, Daniil Churikov wrote: > Hello Chris. > > I am speaking about static analysis tool named "Dialyzer". This tool is > intended to be used as "error-finder". And what I am interested in is > whether this tool will be able to understand what module you substitute > in *State#state.db* and make warning that this module is not properly > written (i.e. miss some functions or this functions will not be able to > handle what you will feed them). > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From erlang@REDACTED Fri Aug 9 15:04:49 2013 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 9 Aug 2013 15:04:49 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: On Fri, Aug 9, 2013 at 9:41 AM, H.C. v. Stockhausen wrote: > Hello, > > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > > I would like to code against an interface and tell the application > what specific backend to use through config rather than code changes. > > Is there a pattern for doing that? Hot code upgrades and multi-node > are not a real concern at this time - mostly, since I have no > experience yet with either - however if doing it right means taking > that into account too I'd also like to learn more about that. > > I am thinking of defining a custom behaviour ("my_crud" perhaps), then > to implement it for various DBs and to also let a config driven > adapter implement it that I then use to throughout the code to talk to > the DB layer. > > For example, using Mnesia and AWS DynamoDB: > > - my_crud.erl (behaviour) > - my_db_mnesia.erl (implements behaviour) > - my_db_dynamo.erl (implements behaviour) > - my_db.erl (configurable adapter that also implements behaviour) > - my.config > > my_db:insert(Key, Value). > > Is that a reasonable approach that makes proper use of Erlang and > behaviours or is this just not how one should do it? > Yes - this is fine But ... Firstly, this might be overkill. One of the real reasons for the import declaration was to be able to flip the back-end implementation while providing a common interface library: So we'd write: -module(xxx). -import(my_mnesia_backend, [insert/2, lookup/1, ...]) foo(X) -> insert(X, ...) and so on. To change the backend you just change the name of the module in the import declaration That was the idea - but in practice very few libraries are written with compatible back-ends The second problem with doing this is that it turns out to be extremely difficult to make a common front-end - this is essentially the least common denominator of the all the services offered by the individual databases. There are often very good reasons why databases are different - they offer different services. I recently considered making a common front-end for dets and basho's bitcask - this seems reasonable both are disk based key-value crash tolerant stores, so this seemed like a good idea, I'd be able to use one store and then flip to the other if circumstances changed. The problem is that dets and bitcask are rather different - they *have* to be, otherwise why would the basho guys write bitcask. ets and dets play nicely together (ets:from_dets :-) - ets keys and values are terms. bitcask keys and values are binaries. Which will the common interface have? - if you choose (say) terms you'll bias the interface in favour of dets. So now we have a problem - two interfaces can be *functionally* equivalent (ie offer the same interface in terms of types) but the non-functional properties of the interface might be completely different. For example database1 might be optimised for fast reads/slow writes. Database2 might be optimised for fast writes. It seems to me that the non-functional parts of a system are the bits that are difficult to get right - making things (fault- tolerant, scalable, upgradable, ... etc.) is far more difficult than getting types and interfaces right. There are two approaches to making a system with a database: 1) Choose the database first - make your mind up 2) Implement everything or 1) Make an abstraction layer 2) Write application and interface layers 3) configure and decide on database later The hope is that by choosing the second approach, you save work since if you chose the wrong database you can just change the backend later. The problem here is that the non-functional behaviour of the backend you choose causes feature creep in the front end. If the backend has fast reads and slow writes, the front-end software will endup having a design that reflects this. The question you now have to ask is the generalisation and extra interface libraries worth the extra effort? I often start by trying to generalise dissimilar things and make them similar - but get to a point where the generalisation and interfacing becomes the main problem and not solving the original problem. That was a bit long and rambling ... So the answer is "Yes you can do this, and it *might* be a good idea" to see if it *is* a good idea, you'll have to implement things three times (with two data bases) (ie solution with database A alone and no abstraction layer) (database B with no abstraction layer) (A and B with a common abstraction layer) Often the abstraction layer becomes at least as complex as the thing being abstracted- at which point you wish you'd chosen a simpler career like being a brain surgeon or astronaut. Cheers /Joe > Thank you for any help & best regards, > Hans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Fri Aug 9 15:58:54 2013 From: colanderman@REDACTED (Chris King) Date: Fri, 9 Aug 2013 09:58:54 -0400 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: <5204E790.8080003@ninenines.eu> References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> <5204E790.8080003@ninenines.eu> Message-ID: On Fri, Aug 9, 2013 at 8:58 AM, Lo?c Hoguin wrote: > Whether Dialyzer can do that or not is a bit irrelevant, as this kind of > error is quickly apparent when your tests immediately fail. Eh, that argument can apply to most things a static analyzer checks. The point of a static analyzer is to catch these kinds of errors without having to write formal tests. Hans: You can get most of what you want by using the (undocumented) callback syntax to specify the interface (in addition to the pattern you described above). Its syntax is identical to that of "spec" function specifications; use it in the behavior module to declare the types of the behavior's functions. See http://learnyousomeerlang.com/dialyzer#typed-behaviours for an example. Specifying callback types will check that modules claiming to implement that behavior will be type-checked by Dialyzer. However Dialyzer won't be able to check modules that *don't* specify which behavior they implement, and it won't be able to check that the front-end module is using the back-ends according to your behavior spec (there is (AFAIK) currently no way to specify this). (Dialyzer *can* check, I think, that the front-end module is using the *currently configured* back-end in a safe manner (not necessarily according to the behavior spec though). Though I wouldn't rely on this, as it may be a long enough code path between back-end declaration and use to result in a false negative.) From essen@REDACTED Fri Aug 9 16:07:37 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 09 Aug 2013 16:07:37 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> <5204E790.8080003@ninenines.eu> Message-ID: <5204F7A9.5020103@ninenines.eu> On 08/09/2013 03:58 PM, Chris King wrote: > On Fri, Aug 9, 2013 at 8:58 AM, Lo?c Hoguin wrote: >> Whether Dialyzer can do that or not is a bit irrelevant, as this kind of >> error is quickly apparent when your tests immediately fail. > > Eh, that argument can apply to most things a static analyzer checks. > The point of a static analyzer is to catch these kinds of errors > without having to write formal tests. No. A static analyzer cannot be used in place of tests. They're complementary. It's even more true in a dynamic language like Erlang where the static analyzer power is limited, *but* applies to all languages regardless. As an example that comes immediately to mind, heard from discussions, in this case about Haskell, the static analyzer can tell you the program is correct, but when you test it it blows up because it ran out of memory. If your program passes static analyzing, that's great! But you still have no idea if the program will actually work, because you didn't test it. The reverse example is of course the static analyzer telling you about things you didn't think of testing. So use both Dialyzer *and* tests if you want solid programs. Not one or the other. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From colanderman@REDACTED Fri Aug 9 16:55:42 2013 From: colanderman@REDACTED (Chris King) Date: Fri, 9 Aug 2013 10:55:42 -0400 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: <5204F7A9.5020103@ninenines.eu> References: <490eda7e-e2d0-486d-b806-1848a946e2e1@googlegroups.com> <5204E790.8080003@ninenines.eu> <5204F7A9.5020103@ninenines.eu> Message-ID: On Fri, Aug 9, 2013 at 10:07 AM, Lo?c Hoguin wrote: > On 08/09/2013 03:58 PM, Chris King wrote: >> >> Eh, that argument can apply to most things a static analyzer checks. >> The point of a static analyzer is to catch these kinds of errors >> without having to write formal tests. > > No. Yes. There is no point to a static analyzer if, as you assert, the class of errors it checks is subsumed by that of formal tests. Ask yourself: why bother even statically checking syntax? After all, unit tests will catch syntax errors. From g@REDACTED Fri Aug 9 17:34:14 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 9 Aug 2013 10:34:14 -0500 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: Hi Hans, On Fri, Aug 9, 2013 at 2:41 AM, H.C. v. Stockhausen wrote: > Hello, > > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > > I would like to code against an interface and tell the application > what specific backend to use through config rather than code changes. Joe's response to this is quite comprehensive and I'll just echo it here. I would simply defer the decision to build this abstraction until you need it. If you add up the time and effort to build a reusable abstraction that can *actually* survive swapping, you'll find it massively outweighs the time and effort to just change your code straight away. BUT - you're not even at the point you need to change the code. You're still at the first implementation. If there's < 100% chance that you'll need to swap, you're better off waiting until that event occurs to deal with it. You may avoid it altogether AND - if you do hit that point - you'll have more information and will make better decisions. Again, to reiterate Joe's point, there's a intuitive appeal in software to build generic abstractions with swappable implementations (see Java ecosystem, where this is literally at epidemic levels) - but in practice it's *hard, *expensive* and rife with trade offs. I recommend going for the direct solution, every time without exception. If you end up with a set of interchangeable components it will be because you needed that, rather than because you thought you needed it. Apologies for the religious overtones :) Garrett From norton@REDACTED Fri Aug 9 18:21:39 2013 From: norton@REDACTED (=?utf-8?B?44OO44O844OI44OzIOOCuOODp+ODvOOCu+ODlSDjgqbjgqfjgqQg?= =?utf-8?B?44Oz?=) Date: Sat, 10 Aug 2013 01:21:39 +0900 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: <6030DF90-A72D-4817-9421-71707187D2C8@lovely.email.ne.jp> Hans - Hi. I wrote an application that provides an ETS-like interface with a callback interface for a backend implementation. This application (https://github.com/norton/gen-ets) won't fit your needs but it may provide you with an example. As others have mentioned, I'd try to first implement one DB API and implementation that fits your needs to gain experience. You might even find out later that you don't really need a generic layer. Cheers, Joe N On 2013/08/09, at 16:41, H.C. v. Stockhausen wrote: > Hello, > > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > > I would like to code against an interface and tell the application > what specific backend to use through config rather than code changes. > > Is there a pattern for doing that? Hot code upgrades and multi-node > are not a real concern at this time - mostly, since I have no > experience yet with either - however if doing it right means taking > that into account too I'd also like to learn more about that. > > I am thinking of defining a custom behaviour ("my_crud" perhaps), then > to implement it for various DBs and to also let a config driven > adapter implement it that I then use to throughout the code to talk to > the DB layer. > > For example, using Mnesia and AWS DynamoDB: > > - my_crud.erl (behaviour) > - my_db_mnesia.erl (implements behaviour) > - my_db_dynamo.erl (implements behaviour) > - my_db.erl (configurable adapter that also implements behaviour) > - my.config > > my_db:insert(Key, Value). > > Is that a reasonable approach that makes proper use of Erlang and > behaviours or is this just not how one should do it? > > Thank you for any help & best regards, > Hans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Fri Aug 9 18:26:59 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 9 Aug 2013 18:26:59 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: On Fri, Aug 9, 2013 at 9:41 AM, H.C. v. Stockhausen wrote: > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > The key point about swapping out is that it is a *static* property of your program here. So I would just -define(db, foo_db_mnesia). [...] ?db:transaction(fun() -> ...) and so on. There is little reason to make this harder by making the choice of the database dynamic at runtime unless you really need it. Tools, like the dialyzer, will make a best-effort at catching problems, should they occur. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yoursurrogategod@REDACTED Fri Aug 9 18:47:08 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Fri, 9 Aug 2013 12:47:08 -0400 Subject: [erlang-questions] Fwd: How to access CouchDB? In-Reply-To: References: Message-ID: Got it! Thanks! % erl -pz /very/long/path ---------- Forwarded message ---------- From: Yves S. Garret Date: Thu, Aug 8, 2013 at 9:38 PM Subject: How to access CouchDB? To: "erlang-questions@REDACTED" Hello, I have a small database that's running right now in CouchDB, it's called sample_database. What I'd like to do is to read, create, delete, update documents and run some of the views inside from an Erlang app. I looked online and found this example: http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ >From that, I proceeded to make the following little script: -module(test). -import(queue,[in/1,out/1,new/0]). -export([start/0]). -include("../couch_db.hrl"). -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). start() -> io:format("Hello world!~n~n"), couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). However, when I run this, I get the following error: 18> test:start(). Hello world! ** exception error: undefined function couch_db:open/2 It's obvious I'm not including a library of some sort. I'm curious how I can connect to my little database in the simplest way possible just to start with. Has anyone done this before? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Fri Aug 9 18:57:25 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 9 Aug 2013 18:57:25 +0200 Subject: [erlang-questions] How to access CouchDB? In-Reply-To: References: Message-ID: not sure if it can help but you can use http://github.com/benoitc/couchcwhich abstract the internal couchdb api . - benoitc On Friday, August 9, 2013, Yves S. Garret wrote: > Hello, > > I have a small database that's running right now in CouchDB, it's called > sample_database. What I'd > like to do is to read, create, delete, update documents and run some of > the views inside from an > Erlang app. I looked online and found this example: > > http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ > > From that, I proceeded to make the following little script: > -module(test). > > -import(queue,[in/1,out/1,new/0]). > -export([start/0]). > -include("../couch_db.hrl"). > > -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). > > start() -> > io:format("Hello world!~n~n"), > > couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). > > However, when I run this, I get the following error: > 18> test:start(). > Hello world! > > ** exception error: undefined function couch_db:open/2 > > It's obvious I'm not including a library of some sort. I'm curious how I > can connect to my little > database in the simplest way possible just to start with. Has anyone done > this before? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsvancara@REDACTED Fri Aug 9 19:23:22 2013 From: bsvancara@REDACTED (Bohuslav Svancara) Date: Fri, 9 Aug 2013 19:23:22 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: Hello! I tried ContextErlang as an exercise: http://www.guidosalvaneschi.com/wp/software/contexterlang/ The "context programming" concept can be applied very well here (I think). Here is an example what I am doing. (1) The "base" module "mt4_db_interface" implements MySql operations using emysql. It also includes a definition for "context switch". (See mt4_db_interface_context/0 ): -module(mt4_db_interface). -behaviour(context_agent). % ContextErlang -context_call([connect/5,get_c2_id_for_mt4_id/2,update_c2_id_for_mt4_id/3,delete/2]). -context_cast([]). %% gen_server callbacks -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). %% External exports -export([connect/5,get_c2_id_for_mt4_id/2,delete/2,update_c2_id_for_mt4_id/3,start_link/0]). % ContextErlang -include("../../ContextErlang/include/context_agent_api.hrl"). -define(PoolId,mt4). % a definition for emysql:add_pool(...) -record(mt4_db_interface_state, {dbHandle, dummy_database_for_tests=dict:new()}). % in the records.hrl in the real program start_link() -> context_agent:start_link({local, ?MODULE}, ?MODULE, [], []). get_c2_id_for_mt4_id(SystemId,MT4OrderId) -> context_agent:call(?MODULE, {get_c2_id_for_mt4_id, SystemId,MT4OrderId}, 60000). update_c2_id_for_mt4_id(SystemId,MT4OrderId,C2SignalId) -> context_agent:call(?MODULE, {update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, 60000). delete(SystemId,MT4OrderId) -> context_agent:call(?MODULE, {delete, SystemId,MT4OrderId}, 60000). connect(Host, Port, User, Password, DatabaseName) -> context_agent:call(?MODULE, {connect, Host, Port, User, Password, DatabaseName}, 60000). init([]) -> io:format("Starting ~p~n",[?MODULE]), {context,mt4_db_interface_context(),#mt4_db_interface_state{}}. handle_call({connect, Host, Port, User, Password, DatabaseName}, _From, State) -> Reply = do_connect(Host, Port, User, Password, DatabaseName), {reply, Reply, State}; handle_call({get_c2_id_for_mt4_id, SystemId,MT4OrderId}, _From, State) -> Reply = do_get_c2_id_for_mt4_id(?PoolId,SystemId,MT4OrderId), {reply, Reply, State}; handle_call({update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, _From, State) -> Reply = do_replace_c2_id_for_mt4_id(?PoolId,SystemId,MT4OrderId,C2SignalId), {reply, Reply, State}; handle_call({delete, SystemId,MT4OrderId}, _From, State) -> Reply = do_delete(?PoolId,SystemId,MT4OrderId), {reply, Reply, State}; handle_call(_Request, _From, State) -> Reply = ok, {reply, Reply, State}. handle_cast(_Msg, State) -> {noreply, State}. handle_info(_Info, State) -> {noreply, State}. terminate(_Reason, _State) -> ok. code_change(_OldVsn, State, _Extra) -> {ok, State}. %% ============================================================================= %% MySql implementations: emysql:execute(DbId,Select) do_connect(Host, Port, User, Password, DatabaseName)-> % Connect to MySql here emysql:add_pool(?PoolId,...)... ok. do_get_c2_id_for_mt4_id(DbId,SystemId,MT4OrderId) -> emysql:execute(...), .... do_replace_c2_id_for_mt4_id(DbId,SystemId,MT4OrderId,C2SignalId) -> emysql:execute(...), .... do_delete(DbId,SystemId,MT4OrderId) -> emysql:execute(...), .... %% ==================================================================== %% @doc ContextErlang. %% Create a context structure for this module. %% Defined variation 'use_dummy_db_for_mt4' is not active in the production mode. Activated just for unit tests. %% @end %% ==================================================================== mt4_db_interface_context() -> Spec = [ {what_db_to_use_slot, use_dummy_db_for_mt4} ], context_ADT:create(Spec). (2) A variation "use_dummy_db_for_mt4" which implements a fake database and is used for unit tests: %% =========================================================================== %% Author: Bohuslav Svancara %% Created: 2013-07-20 %% Description: Experimental ContextErlang variation module for mt4_db_interface. %% %% This variation is used for unit tests where MySql is very hard to use. %% =========================================================================== -module(use_dummy_db_for_mt4). -export([handle_call/3,on_activation/1,on_deactivation/1,test/0]). -context_call([connect/5,get_c2_id_for_mt4_id/2,update_c2_id_for_mt4_id/3,delete/2]). -context_cast([]). -record(mt4_db_interface_state, {dbHandle, dummy_database_for_tests=dict:new()}). % in records.hrl in a real program handle_call({connect, Host, Port, User, Password, DatabaseName}, _From, State) -> {reply, ok, State#mt4_db_interface_state{ dummy_database_for_tests = dict:new()}}; handle_call({get_c2_id_for_mt4_id, SystemId,MT4OrderId}, _From, State) -> Reply = case dict:find({SystemId,MT4OrderId}, State#mt4_db_interface_state.dummy_database_for_tests) of end, {reply, Reply, State}; handle_call({update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, _From, State) -> NewDict = dict:update({SystemId,MT4OrderId}, fun(_) -> C2SignalId end, C2SignalId, State#mt4_db_interface_state.dummy_database_for_tests), {reply, ok, State#mt4_db_interface_state{dummy_database_for_tests = NewDict}}; handle_call({delete, SystemId,MT4OrderId}, _From, State) -> NewDict = dict:erase({SystemId,MT4OrderId}, State#mt4_db_interface_state.dummy_database_for_tests), {reply, ok, State#mt4_db_interface_state{dummy_database_for_tests = NewDict}}. on_activation(State) -> {ok,State}. on_deactivation(State) -> {ok,State}. (3) I a production code the mt4_db_interface module is used: {ok, Host, Port, User, Password, DatabaseName} = get_db_config(), mt4_db_interface:start_link(), mt4_db_interface:connect(Host, Port, User, Password, DatabaseName),... mt4_db_interface:get_c2_id_for_mt4_id(...) mt4_db_interface:update_c2_id(...) mt4_db_interface:delete(...) (4) In unit tests mt4_db_interface is switched (dynamic switch - the most interesting thing in context programming) to "use_dummy_db_for_mt4" implementation: mt4_db_interface:start_link(), mt4_db_interface:in_cur_context_activate(mt4_db_interface, use_dummy_db_for_mt4, what_db_to_use_slot), mt4_db_interface:connect("DummyHost", 0, "DummyUser", "DummyPassword", "DummyDatabaseName"), mt4_db_interface:get_c2_id_for_mt4_id(...) mt4_db_interface:update_c2_id(...) mt4_db_interface:delete(...) Is there any other person trying context programming in Erlang? Sincerely, Bohuslav Svancara 2013/8/9 H.C. v. Stockhausen > > Hello, > > I need a DB backend for my application but I'd like to be able to swap > it out for different DBs if I choose so later on. > > I would like to code against an interface and tell the application > what specific backend to use through config rather than code changes. > > Is there a pattern for doing that? Hot code upgrades and multi-node > are not a real concern at this time - mostly, since I have no > experience yet with either - however if doing it right means taking > that into account too I'd also like to learn more about that. > > I am thinking of defining a custom behaviour ("my_crud" perhaps), then > to implement it for various DBs and to also let a config driven > adapter implement it that I then use to throughout the code to talk to > the DB layer. > > For example, using Mnesia and AWS DynamoDB: > > - my_crud.erl (behaviour) > - my_db_mnesia.erl (implements behaviour) > - my_db_dynamo.erl (implements behaviour) > - my_db.erl (configurable adapter that also implements behaviour) > - my.config > > my_db:insert(Key, Value). > > Is that a reasonable approach that makes proper use of Erlang and > behaviours or is this just not how one should do it? > > Thank you for any help & best regards, > Hans > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sperber@REDACTED Fri Aug 9 17:52:47 2013 From: sperber@REDACTED (Michael Sperber) Date: Fri, 09 Aug 2013 17:52:47 +0200 Subject: [erlang-questions] Commercial Users of Functional Programming (Sep 22-24, Boston): Call for Participation Message-ID: Functional programming has been at the forefront of a new generation of programming technologies: Companies employing functional programming use it to enable more effective, robust, and flexible software development. The annual CUFP workshop is designed to serve the growing community of commercial users of functional programming: Practitioners meet and collaborate; language designers and users can share ideas about the future of their languages; experts share their expertise on practical functional programming. CUFP 2013 begins with a day of talks about industrial applications of functional programming, followed by two days of tutorials by top-notch language experts including advanced tutorials on special topics. More information about CUFP 2013 is available on the CUFP web site at http://cufp.org/conference/schedule Registration is available at: https://regmaster3.com/2013conf/ICFP13/register.php Note that early-registration discounts end August 22. TALKS, SEPTEMBER 22 Keynote: "Small Talk" Dave Thomas, TBA. Analyzing PHP statically (Julien Verlaguet, Facebook) Introducing Erlang to OpenX (Anthony Molinaro, OpenX) Redesigning the Computer for Security (Tom Hawkins, BAE Systems) End to end Reactive Programming (Jafar Husain, Netflix) Medical Device Automation using Message-Passing Concurrency in Scheme (Vishesh Panchal & BobBurger, Beckman Coulter Inc.) Enabling Microservice Architectures with Scala (Kevin Scaldeferri, Gilt Groupe) Functional Infrastructures (Antoni Batchelli, PalletOps) Realtime MapReduce at Twitter (Sam Ritchie, Twitter Inc) Functional Probabilistic Programming (Avi Pfeffer, Charles River Analytics) Building a commercial development platform Haskell, an experience report. (Gregg Lebovitz, FP Complete) Common Pitfalls of Functional Programming and How to Avoid Them: A Mobile Gaming Platform Case Study (Yasuaki Takebe, GREE, Inc) Building scalable, high-availability distributed systems in Haskell (Jeff Epstein, Parallel Scientific) Functional Reporting (Edward Kmett, S&P Capital IQ) Enterprise Appointment Scheduling with Haskell (Ryan Trinkle, skedge.me) Programming Map/Reduce in Mathematica (Paul-Jean Letourneau, Wolfram) TUTORIALS, SEPTEMBER 23 T1: Haskell Day 1 (Andres L?h) T2 - OCaml tutorial (Yaron Minsky & Anil Madhavapeddy) T3 - Erlang 101 - Your introduction to Concurrency and Multi-core (Francesco Cesarini & Simon Thompson) T4 - (Systematic generation of optimal code with MetaOCaml) Oleg Kiselyov T5 - (Erlang Web frameworks) Steve Vinoski TUTORIALS, SEPTEMBER 24 T6 - Haskell Day 2 (Simon Marlow) T7 - Clojure tutorial (Luke Vander Hart) T8 - The Seductions of Scala (Dean Wampler) T9 - Bending Clojure to your will: Macros and Domain Specific Languages (Leonardo Borges) T10 - Scalding - The Scala Tool for Data Analytics in Hadoop Systems (Dean Wampler) From yoursurrogategod@REDACTED Fri Aug 9 21:22:46 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Fri, 9 Aug 2013 15:22:46 -0400 Subject: [erlang-questions] How to access CouchDB? In-Reply-To: References: Message-ID: On Fri, Aug 9, 2013 at 12:57 PM, Benoit Chesneau wrote: > not sure if it can help but you can use http://github.com/benoitc/couchcwhich abstract the internal couchdb api . > > - benoitc > > > On Friday, August 9, 2013, Yves S. Garret wrote: > >> Hello, >> >> I have a small database that's running right now in CouchDB, it's called >> sample_database. What I'd >> like to do is to read, create, delete, update documents and run some of >> the views inside from an >> Erlang app. I looked online and found this example: >> >> http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ >> >> From that, I proceeded to make the following little script: >> -module(test). >> >> -import(queue,[in/1,out/1,new/0]). >> -export([start/0]). >> -include("../couch_db.hrl"). >> >> -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). >> >> start() -> >> io:format("Hello world!~n~n"), >> >> couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). >> >> However, when I run this, I get the following error: >> 18> test:start(). >> Hello world! >> >> ** exception error: undefined function couch_db:open/2 >> >> It's obvious I'm not including a library of some sort. I'm curious how I >> can connect to my little >> database in the simplest way possible just to start with. Has anyone >> done this before? >> > Thanks. I believe I've resolved most of my problems already, but this is a welcome addition to my little bag of tricks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Aug 9 21:22:57 2013 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 09 Aug 2013 12:22:57 -0700 Subject: [erlang-questions] [ANN] CloudI 1.2.5 Released! Message-ID: <52054191.6090900@gmail.com> Download 1.2.5 from http://sourceforge.net/projects/cloudi/files/latest/download What is CloudI? A Cloud at the lowest level! CloudI (http://cloudi.org/) is a "universal integrator" using an Erlang core, with its actor model, to pursue efficiency, fault-tolerance and scalability. The CloudI API provides a minimal interface to communicate among actors that are called services, so programming language agnostic, database agnostic, and messaging bus agnostic integration can occur. CloudI currently integrates with the programming languages Erlang, C/C++, Java, Python, and Ruby, the databases PostgreSQL, elasticsearch, MySQL, couchdb, memcached, and tokyotyrant, the messaging bus ZeroMQ and the internal CloudI service bus. HTTP is also supported for easy integration with cowboy and elli. If anyone is willing to get involved, don't hesitate to contact me or start looking at the code. Thanks to Mahesh Paolini-Subramanya for the elasticsearch integration, integration help from Juan Jose Comellas and Lukas Larsson for testing/validation, et. al. (tell me if your name isn't mentioned in the AUTHORS file). The details for this release are below: * elasticsearch integration was added in cloudi_service_db_elasticsearch. * External service configuration strings provided for the executable file path, the executable command line arguments, and the executable environment variables, now can expand environment variables set in the shell executing the Erlang VM (e.g., "$USER" or "${USER}" syntax). * Internal services now have a few more service configuration options: * hibernate - always makes the service processes hibernate to conserve memory by doing extra garbage collection processing * reload - will automatically reload the service module or all modules within a service application when the beam file is updated * Both external and internal services can set the scope service configuration option to give service name lookups and subscriptions a separate scope, minimizing contention when using an immediate destination refresh method. * Tests with common test, eunit, and PropEr are now used to validate the build. An initial common test was added to test cloudi_service_db_pgsql. * The cloudi Erlang module received fixes so that it is a better match to the subset of the CloudI API (the cloudi_service Erlang module) it implements. * The services_search function was added to the CloudI Service API which returns the services that have subscribed to a service name. * cloudi_service:service_name_parse/2 was added to make it simpler to parse service name patterns. * The cloudi_core Erlang application is now the only dependency required when including CloudI in an Erlang release. See the hello_world2 example for details. * rebar integration for using the CloudI repository as a rebar dependency was added. See the hello_world3 example for details. * Added a lot of configuration validation to prevent any problems with invalid configuration data for services, ACLs, etc. * Small fixes and additions were added to the websockets integration within cloudi_service_http_cowboy. use_client_ip_prefix is a new option within cloudi_service_http_cowboy. * cloudi_service_http_elli was added to provide elli integration. More load testing is required to determine how it compares to cloudi_service_http_cowboy. * ZeroMQ detection on OSX was fixed. Please mention any problems, issues, or ideas! Thanks, Michael From yoursurrogategod@REDACTED Fri Aug 9 21:36:21 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Fri, 9 Aug 2013 15:36:21 -0400 Subject: [erlang-questions] How to access CouchDB? In-Reply-To: References: Message-ID: On Fri, Aug 9, 2013 at 12:57 PM, Benoit Chesneau wrote: > not sure if it can help but you can use http://github.com/benoitc/couchcwhich abstract the internal couchdb api . > > - benoitc > > > On Friday, August 9, 2013, Yves S. Garret wrote: > >> Hello, >> >> I have a small database that's running right now in CouchDB, it's called >> sample_database. What I'd >> like to do is to read, create, delete, update documents and run some of >> the views inside from an >> Erlang app. I looked online and found this example: >> >> http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ >> >> From that, I proceeded to make the following little script: >> -module(test). >> >> -import(queue,[in/1,out/1,new/0]). >> -export([start/0]). >> -include("../couch_db.hrl"). >> >> -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). >> >> start() -> >> io:format("Hello world!~n~n"), >> >> couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). >> >> However, when I run this, I get the following error: >> 18> test:start(). >> Hello world! >> >> ** exception error: undefined function couch_db:open/2 >> >> It's obvious I'm not including a library of some sort. I'm curious how I >> can connect to my little >> database in the simplest way possible just to start with. Has anyone >> done this before? >> > And I've spoken too soon :) . This is the code that I currently have: -module(test). -import(queue,[in/1,out/1,new/0]). -export([start/0]). % /opt/local/lib/couchdb/erlang/lib/couch-1.3.1/ebin -include("/opt/local/lib/couchdb/erlang/lib/couch-1.3.1/include/couch_db.hrl"). -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"some_user">>]}}). start() -> io:format("Hello world!~n~n"), couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). It compiles just fine, but when I run it, this is the error that I get: > c(test). {ok,test} > test:start(). Hello world! ** exception error: bad argument in function ets:lookup/2 called as ets:lookup(couch_config,{"replicator","db"}) in call from couch_config:get/3 (couch_config.erl, line 62) in call from couch_server:maybe_add_sys_db_callbacks/2 (couch_server.erl, line 89) in call from couch_server:open/2 (couch_server.erl, line 64) in call from couch_db:open/2 (couch_db.erl, line 79) I'll be digging into why this is happening. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Aug 9 22:12:54 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 10 Aug 2013 00:12:54 +0400 Subject: [erlang-questions] Possibly bug in cth_log_redirect? Message-ID: When tests are running in parallel, LogFun in cth_log_redirect is changed to ct_log: https://github.com/erlang/otp/blob/3021fca734f71f8bae966ab67f1400d37f8927bc/lib/common_test/src/cth_log_redirect.erl#L49 Problem is that it must be not ct_log, but tc_log: https://github.com/erlang/otp/blob/3021fca734f71f8bae966ab67f1400d37f8927bc/lib/common_test/src/ct_logs.erl#L44 Should I make patch? From yoursurrogategod@REDACTED Fri Aug 9 22:47:44 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Fri, 9 Aug 2013 16:47:44 -0400 Subject: [erlang-questions] How to access CouchDB? In-Reply-To: References: Message-ID: On Fri, Aug 9, 2013 at 3:36 PM, Yves S. Garret wrote: > On Fri, Aug 9, 2013 at 12:57 PM, Benoit Chesneau wrote: > >> not sure if it can help but you can use http://github.com/benoitc/couchcwhich abstract the internal couchdb api . >> >> - benoitc >> >> >> On Friday, August 9, 2013, Yves S. Garret wrote: >> >>> Hello, >>> >>> I have a small database that's running right now in CouchDB, it's called >>> sample_database. What I'd >>> like to do is to read, create, delete, update documents and run some of >>> the views inside from an >>> Erlang app. I looked online and found this example: >>> >>> http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ >>> >>> From that, I proceeded to make the following little script: >>> -module(test). >>> >>> -import(queue,[in/1,out/1,new/0]). >>> -export([start/0]). >>> -include("../couch_db.hrl"). >>> >>> -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). >>> >>> start() -> >>> io:format("Hello world!~n~n"), >>> >>> couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). >>> >>> However, when I run this, I get the following error: >>> 18> test:start(). >>> Hello world! >>> >>> ** exception error: undefined function couch_db:open/2 >>> >>> It's obvious I'm not including a library of some sort. I'm curious how >>> I can connect to my little >>> database in the simplest way possible just to start with. Has anyone >>> done this before? >>> >> > And I've spoken too soon :) . > > This is the code that I currently have: > > -module(test). > > -import(queue,[in/1,out/1,new/0]). > -export([start/0]). > % /opt/local/lib/couchdb/erlang/lib/couch-1.3.1/ebin > > -include("/opt/local/lib/couchdb/erlang/lib/couch-1.3.1/include/couch_db.hrl"). > > -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"some_user">>]}}). > > start() -> > io:format("Hello world!~n~n"), > > couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). > > It compiles just fine, but when I run it, this is the error that I get: > > c(test). > {ok,test} > > test:start(). > Hello world! > > ** exception error: bad argument > in function ets:lookup/2 > called as ets:lookup(couch_config,{"replicator","db"}) > in call from couch_config:get/3 (couch_config.erl, line 62) > in call from couch_server:maybe_add_sys_db_callbacks/2 > (couch_server.erl, line 89) > in call from couch_server:open/2 (couch_server.erl, line 64) > in call from couch_db:open/2 (couch_db.erl, line 79) > > I'll be digging into why this is happening. > As a work around, I'm thinking of using os:cmd("curl -i -X GET ..."), but that seems like it could be done better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Fri Aug 9 22:51:08 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 9 Aug 2013 22:51:08 +0200 Subject: [erlang-questions] How to access CouchDB? In-Reply-To: References: Message-ID: or just use couchbeam ... On Fri, Aug 9, 2013 at 10:47 PM, Yves S. Garret wrote: > On Fri, Aug 9, 2013 at 3:36 PM, Yves S. Garret > wrote: > >> On Fri, Aug 9, 2013 at 12:57 PM, Benoit Chesneau wrote: >> >>> not sure if it can help but you can use http://github.com/benoitc/couchcwhich abstract the internal couchdb api . >>> >>> - benoitc >>> >>> >>> On Friday, August 9, 2013, Yves S. Garret wrote: >>> >>>> Hello, >>>> >>>> I have a small database that's running right now in CouchDB, it's >>>> called sample_database. What I'd >>>> like to do is to read, create, delete, update documents and run some of >>>> the views inside from an >>>> Erlang app. I looked online and found this example: >>>> >>>> http://www.softwarepassion.com/importing-data-to-couchdb-java-ruby-and-erlang-way/ >>>> >>>> From that, I proceeded to make the following little script: >>>> -module(test). >>>> >>>> -import(queue,[in/1,out/1,new/0]). >>>> -export([start/0]). >>>> -include("../couch_db.hrl"). >>>> >>>> -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"_admin">>]}}). >>>> >>>> start() -> >>>> io:format("Hello world!~n~n"), >>>> >>>> couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). >>>> >>>> However, when I run this, I get the following error: >>>> 18> test:start(). >>>> Hello world! >>>> >>>> ** exception error: undefined function couch_db:open/2 >>>> >>>> It's obvious I'm not including a library of some sort. I'm curious how >>>> I can connect to my little >>>> database in the simplest way possible just to start with. Has anyone >>>> done this before? >>>> >>> >> And I've spoken too soon :) . >> >> This is the code that I currently have: >> >> -module(test). >> >> -import(queue,[in/1,out/1,new/0]). >> -export([start/0]). >> % /opt/local/lib/couchdb/erlang/lib/couch-1.3.1/ebin >> >> -include("/opt/local/lib/couchdb/erlang/lib/couch-1.3.1/include/couch_db.hrl"). >> >> -define(ADMIN_USER_CTX, {user_ctx, #user_ctx{roles = [<<"some_user">>]}}). >> >> start() -> >> io:format("Hello world!~n~n"), >> >> couch_db:open(<<"sample_database">>, [?ADMIN_USER_CTX]). >> >> It compiles just fine, but when I run it, this is the error that I get: >> > c(test). >> {ok,test} >> > test:start(). >> Hello world! >> >> ** exception error: bad argument >> in function ets:lookup/2 >> called as ets:lookup(couch_config,{"replicator","db"}) >> in call from couch_config:get/3 (couch_config.erl, line 62) >> in call from couch_server:maybe_add_sys_db_callbacks/2 >> (couch_server.erl, line 89) >> in call from couch_server:open/2 (couch_server.erl, line 64) >> in call from couch_db:open/2 (couch_db.erl, line 79) >> >> I'll be digging into why this is happening. >> > > As a work around, I'm thinking of using os:cmd("curl -i -X GET ..."), but > that seems like it could > be done better. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Sat Aug 10 00:02:38 2013 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 9 Aug 2013 18:02:38 -0400 Subject: [erlang-questions] Fwd: ErlExec stdin and other questions In-Reply-To: References: Message-ID: I added two new options to erlexec (docs: http://saleyn.github.io/erlexec/, code: https://github.com/saleyn/erlexec): * Ability to communicate with spawned OS processes via STDIN * Ability to specify custom kill timeout (in seconds) if an OS command doesn't respond to exec:stop(OsPid) via {kill_timeout, Seconds} option See examples in the documentation section. Serge On Thu, Aug 8, 2013 at 3:16 AM, OvermindDL1 wrote: > I actually already left an issue on Github with such an example script > yesterday, and system information. :-) > On Aug 7, 2013 9:15 PM, "Serge Aleynikov" wrote: > >> I >> am currently on vacation but will try to look into this some time later >> next week when I return. >> >> The stdout/stderr redirection to Erlang is not yet "production" quality - >> rather an experimental feature introduced last week. I am also working on >> adding stdin support, so that you could send messages to an OS process's >> stdin. This should also be available in a week after I return. >> >> As far as I know erlexec is used in several production systems, and >> except for the stdout/stderr redirection to Erlang is stable. >> >> Upon exiting the VM, the erl-exec port process started by exec >> application receives an end-of-file on the stdin pipe, and sets off the >> alarm, which guarantees that it'll be killed after a timeout if it hasn't >> finished the cleanup (termination) of managed processes. If you don't >> observe that behavior, please provide more information regarding OS you are >> running, and ideally some simple test cases. >> >> >> On Wed, Aug 7, 2013 at 3:36 AM, OvermindDL1 wrote: >> >>> Created a complete test-case example of my issue with erlexec, and I ran >>> this same script on 3 different computers running variations of debian >>> versions. Feel free to toss in application:start(sasl), at the top of the >>> erlang script area if you wish, and yes the (minecraft) server software >>> uses stderr instead of stdout, no I do not know why, and yes as you can see >>> it does not return anything to the erlang system until it is killed on all >>> the systems I tested. Using Erlang's ports you get each line as a message >>> as that is how the server flushes it and it is instant. >>> >>> >>> Yes it is ugly, but it is pared down from what it grew from and no need >>> to re-do the receive's better in such a test. >>> test_exec_mc.sh >>> """ >>> #!/bin/sh >>> >>> [ -d erlexec ] || git clone https://github.com/saleyn/erlexec.git >>> >>> cd erlexec >>> >>> rebar compile >>> >>> [ -f 'minecraft_server.1.6.2.jar' ] || wget ' >>> https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar >>> ' >>> >>> erl -pa $PWD/ebin -eval ' >>> application:start(exec), >>> {_, P, _} = exec:run_link("java -Xms1G -Xmx1G -jar >>> minecraft_server.1.6.2.jar nogui", [{stdout, self()}, {stderr, self()}]), >>> timer:sleep(1000), >>> receive A0 -> io:format("Message ~p: ~p~n", [now(), A0]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A1 -> io:format("Message ~p: ~p~n", [now(), A1]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A2 -> io:format("Message ~p: ~p~n", [now(), A2]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A3 -> io:format("Message ~p: ~p~n", [now(), A3]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A4 -> io:format("Message ~p: ~p~n", [now(), A4]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> timer:sleep(1000), >>> receive A5 -> io:format("Message ~p: ~p~n", [now(), A5]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A6 -> io:format("Message ~p: ~p~n", [now(), A6]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A7 -> io:format("Message ~p: ~p~n", [now(), A7]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A8 -> io:format("Message ~p: ~p~n", [now(), A8]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A9 -> io:format("Message ~p: ~p~n", [now(), A9]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> timer:sleep(10000), >>> receive A10 -> io:format("Message ~p: ~p~n", [now(), A10]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A11 -> io:format("Message ~p: ~p~n", [now(), A11]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A12 -> io:format("Message ~p: ~p~n", [now(), A12]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A13 -> io:format("Message ~p: ~p~n", [now(), A13]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A14 -> io:format("Message ~p: ~p~n", [now(), A14]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> exec:kill(P, 9), >>> receive A15 -> io:format("Message ~p: ~p~n", [now(), A15]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A16 -> io:format("Message ~p: ~p~n", [now(), A16]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A17 -> io:format("Message ~p: ~p~n", [now(), A17]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A18 -> io:format("Message ~p: ~p~n", [now(), A18]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A19 -> io:format("Message ~p: ~p~n", [now(), A19]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> timer:sleep(1000), >>> receive A20 -> io:format("Message ~p: ~p~n", [now(), A20]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A21 -> io:format("Message ~p: ~p~n", [now(), A21]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A22 -> io:format("Message ~p: ~p~n", [now(), A22]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A23 -> io:format("Message ~p: ~p~n", [now(), A23]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> receive A24 -> io:format("Message ~p: ~p~n", [now(), A24]) after 1000 -> >>> io:format("timeout ~p~n", [now()]) end, >>> q().' >>> """ >>> >>> >>> >>> >>> >>> On Tue, Aug 6, 2013 at 10:49 PM, OvermindDL1 wrote: >>> >>>> Excepting this, is there any other better way to launch a program as >>>> another user? I might just use ssh certs and ssh to localhost if not >>>> unless any other ideas? >>>> >>>> >>>> On Tue, Aug 6, 2013 at 4:22 AM, OvermindDL1 wrote: >>>> >>>>> I have been attempting to use ErlExec as I noticed that it had a few >>>>> new features added six days ago to forward the stdout/stderr to an Erlang >>>>> PID, however I am having some issue. The process that erlexec runs >>>>> occasionally does not die when the Erlang VM dies, thus keeping it and its >>>>> children running. >>>>> >>>>> Also I need a way to send commands back to the children, but I am >>>>> either not seeing a command for that or have not figured out the format to >>>>> send a message, which is difficult because of the main issue I have, when I >>>>> have it launch a fairly large server software (that is not a daemon for >>>>> whatever reason, hence why I am wrapping it), the Erlang shell seems to >>>>> lose a great deal of input that I type in, only catching from every 1 in 4 >>>>> I type to 1 in 400 characters I type, and it does not seem to matter based >>>>> on the speed I hit keys or holding a key or anything of the sort, just >>>>> purely on keypress down, and it remains like that until I kill the child >>>>> process that I started through erlexec. >>>>> >>>>> Using the normal Erlang Ports works fine and has no such issues (other >>>>> than the child not coming down with the VM either and no direct kill >>>>> command when the child hangs, which it occasionally does, but os:cmd("kill >>>>> ...") works around that well enough. >>>>> >>>>> Is ErlExec stable enough for high use? Or am I doing something wrong? >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hc@REDACTED Sat Aug 10 18:53:10 2013 From: hc@REDACTED (H.C. v. Stockhausen) Date: Sat, 10 Aug 2013 18:53:10 +0200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: Hi, thank you for all your replies. I appreciate your help. The DB example, even if applicable to my use case, was perhaps a little too specific. I don't necessarily need to abstract at a CRUD level - a higher business level would work too. What I wanted to learn is how to swap out implementations (at whatever level really), so thank you for your samples, explanations and advise. I am happy though to have also learned that my initial ideaI wasn't too bad. If I wanted to allow third parties to substitute their implementations for mine (possibly in a closed source product even?) it could be an option. Best regards, Hans On 9 August 2013 19:23, Bohuslav Svancara wrote: > Hello! > > I tried ContextErlang as an exercise: > http://www.guidosalvaneschi.com/wp/software/contexterlang/ > > The "context programming" concept can be applied very well here (I think). > > Here is an example what I am doing. > > (1) > The "base" module "mt4_db_interface" implements MySql operations using emysql. > It also includes a definition for "context switch". (See > mt4_db_interface_context/0 ): > > -module(mt4_db_interface). > > -behaviour(context_agent). > > % ContextErlang > -context_call([connect/5,get_c2_id_for_mt4_id/2,update_c2_id_for_mt4_id/3,delete/2]). > -context_cast([]). > > %% gen_server callbacks > -export([init/1, handle_call/3, handle_cast/2, handle_info/2, > terminate/2, code_change/3]). > > %% External exports > -export([connect/5,get_c2_id_for_mt4_id/2,delete/2,update_c2_id_for_mt4_id/3,start_link/0]). > > % ContextErlang > -include("../../ContextErlang/include/context_agent_api.hrl"). > > -define(PoolId,mt4). % a definition for emysql:add_pool(...) > > -record(mt4_db_interface_state, {dbHandle, > dummy_database_for_tests=dict:new()}). % in the records.hrl in the > real program > > start_link() -> context_agent:start_link({local, ?MODULE}, ?MODULE, [], []). > > get_c2_id_for_mt4_id(SystemId,MT4OrderId) -> > context_agent:call(?MODULE, {get_c2_id_for_mt4_id, > SystemId,MT4OrderId}, 60000). > > update_c2_id_for_mt4_id(SystemId,MT4OrderId,C2SignalId) -> > context_agent:call(?MODULE, {update_c2_id_for_mt4_id, > SystemId,MT4OrderId,C2SignalId}, 60000). > > delete(SystemId,MT4OrderId) -> context_agent:call(?MODULE, {delete, > SystemId,MT4OrderId}, 60000). > > connect(Host, Port, User, Password, DatabaseName) -> > context_agent:call(?MODULE, {connect, Host, Port, User, Password, > DatabaseName}, 60000). > > init([]) -> > io:format("Starting ~p~n",[?MODULE]), > {context,mt4_db_interface_context(),#mt4_db_interface_state{}}. > > handle_call({connect, Host, Port, User, Password, DatabaseName}, > _From, State) -> > Reply = do_connect(Host, Port, User, Password, DatabaseName), > {reply, Reply, State}; > > handle_call({get_c2_id_for_mt4_id, SystemId,MT4OrderId}, _From, State) -> > Reply = do_get_c2_id_for_mt4_id(?PoolId,SystemId,MT4OrderId), > {reply, Reply, State}; > > handle_call({update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, > _From, State) -> > Reply = do_replace_c2_id_for_mt4_id(?PoolId,SystemId,MT4OrderId,C2SignalId), > {reply, Reply, State}; > > handle_call({delete, SystemId,MT4OrderId}, _From, State) -> > Reply = do_delete(?PoolId,SystemId,MT4OrderId), > {reply, Reply, State}; > > handle_call(_Request, _From, State) -> > Reply = ok, > {reply, Reply, State}. > > handle_cast(_Msg, State) -> > {noreply, State}. > > handle_info(_Info, State) -> > {noreply, State}. > > terminate(_Reason, _State) -> > ok. > > code_change(_OldVsn, State, _Extra) -> > {ok, State}. > > %% ============================================================================= > %% MySql implementations: emysql:execute(DbId,Select) > do_connect(Host, Port, User, Password, DatabaseName)-> > % Connect to MySql here > emysql:add_pool(?PoolId,...)... > ok. > > do_get_c2_id_for_mt4_id(DbId,SystemId,MT4OrderId) -> > emysql:execute(...), .... > do_replace_c2_id_for_mt4_id(DbId,SystemId,MT4OrderId,C2SignalId) -> > emysql:execute(...), .... > do_delete(DbId,SystemId,MT4OrderId) -> emysql:execute(...), .... > > %% ==================================================================== > %% @doc ContextErlang. > %% Create a context structure for this module. > %% Defined variation 'use_dummy_db_for_mt4' is not active in the > production mode. Activated just for unit tests. > %% @end > %% ==================================================================== > mt4_db_interface_context() -> > Spec = [ {what_db_to_use_slot, use_dummy_db_for_mt4} ], > context_ADT:create(Spec). > > > (2) > A variation "use_dummy_db_for_mt4" which implements a fake database > and is used for unit tests: > > %% =========================================================================== > %% Author: Bohuslav Svancara > %% Created: 2013-07-20 > %% Description: Experimental ContextErlang variation module for > mt4_db_interface. > %% > %% This variation is used for unit tests where MySql is very hard to use. > %% =========================================================================== > > -module(use_dummy_db_for_mt4). > -export([handle_call/3,on_activation/1,on_deactivation/1,test/0]). > > -context_call([connect/5,get_c2_id_for_mt4_id/2,update_c2_id_for_mt4_id/3,delete/2]). > -context_cast([]). > > -record(mt4_db_interface_state, {dbHandle, > dummy_database_for_tests=dict:new()}). % in records.hrl in a real > program > > handle_call({connect, Host, Port, User, Password, DatabaseName}, > _From, State) -> > {reply, ok, State#mt4_db_interface_state{ dummy_database_for_tests > = dict:new()}}; > > handle_call({get_c2_id_for_mt4_id, SystemId,MT4OrderId}, _From, State) -> > Reply = case dict:find({SystemId,MT4OrderId}, > State#mt4_db_interface_state.dummy_database_for_tests) of > > end, > {reply, Reply, State}; > > handle_call({update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, > _From, State) -> > NewDict = dict:update({SystemId,MT4OrderId}, fun(_) -> C2SignalId > end, C2SignalId, > State#mt4_db_interface_state.dummy_database_for_tests), > {reply, ok, State#mt4_db_interface_state{dummy_database_for_tests > = NewDict}}; > > handle_call({delete, SystemId,MT4OrderId}, _From, State) -> > NewDict = dict:erase({SystemId,MT4OrderId}, > State#mt4_db_interface_state.dummy_database_for_tests), > {reply, ok, State#mt4_db_interface_state{dummy_database_for_tests > = NewDict}}. > > on_activation(State) -> {ok,State}. > > on_deactivation(State) -> {ok,State}. > > > (3) > I a production code the mt4_db_interface module is used: > > {ok, Host, Port, User, Password, DatabaseName} = get_db_config(), > mt4_db_interface:start_link(), > mt4_db_interface:connect(Host, Port, User, Password, DatabaseName),... > > mt4_db_interface:get_c2_id_for_mt4_id(...) > mt4_db_interface:update_c2_id(...) > mt4_db_interface:delete(...) > > (4) > In unit tests mt4_db_interface is switched (dynamic switch - the most > interesting thing in context programming) to "use_dummy_db_for_mt4" > implementation: > > mt4_db_interface:start_link(), > mt4_db_interface:in_cur_context_activate(mt4_db_interface, > use_dummy_db_for_mt4, what_db_to_use_slot), > mt4_db_interface:connect("DummyHost", 0, "DummyUser", > "DummyPassword", "DummyDatabaseName"), > > mt4_db_interface:get_c2_id_for_mt4_id(...) > mt4_db_interface:update_c2_id(...) > mt4_db_interface:delete(...) > > > Is there any other person trying context programming in Erlang? > > Sincerely, > Bohuslav Svancara > > > 2013/8/9 H.C. v. Stockhausen >> >> Hello, >> >> I need a DB backend for my application but I'd like to be able to swap >> it out for different DBs if I choose so later on. >> >> I would like to code against an interface and tell the application >> what specific backend to use through config rather than code changes. >> >> Is there a pattern for doing that? Hot code upgrades and multi-node >> are not a real concern at this time - mostly, since I have no >> experience yet with either - however if doing it right means taking >> that into account too I'd also like to learn more about that. >> >> I am thinking of defining a custom behaviour ("my_crud" perhaps), then >> to implement it for various DBs and to also let a config driven >> adapter implement it that I then use to throughout the code to talk to >> the DB layer. >> >> For example, using Mnesia and AWS DynamoDB: >> >> - my_crud.erl (behaviour) >> - my_db_mnesia.erl (implements behaviour) >> - my_db_dynamo.erl (implements behaviour) >> - my_db.erl (configurable adapter that also implements behaviour) >> - my.config >> >> my_db:insert(Key, Value). >> >> Is that a reasonable approach that makes proper use of Erlang and >> behaviours or is this just not how one should do it? >> >> Thank you for any help & best regards, >> Hans >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From steven.charles.davis@REDACTED Sat Aug 10 21:07:01 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Sat, 10 Aug 2013 12:07:01 -0700 (PDT) Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: <668456f5-a665-4144-81ca-92c66c2bf9b3@googlegroups.com> Hi, Are you trying to do something like this (just an example)? HTH, /s ===== % -*- mode:erlang -*- {application, my_db, [ {description, "my_db"}, {vsn, "0.1"}, {mod, {my_db_app, []}}, {env, [ {my_db, my_simple_db} ]}, {modules, [ my_db, my_db_app, my_db_sup ]}, {applications, [kernel, stdlib]} ]}. ===== -module(my_db). -export([start/0, stop/0]). -export([create/1, read/1, update/1, delete/1]). start() -> application:start(my_db). stop() -> application:stop(my_db). create({K, V}) -> gen_server:call(?MODULE, {create, K, V}). read(K) -> gen_server:call(?MODULE, {read, K}). update({K, V}) -> gen_server:call(?MODULE, {update, K, V}). delete(K) -> gen_server:call(?MODULE, {delete, K}). ==== -module(my_db_app). -define(TTY(X), io:format("~p~n", [X])). -behaviour(application). -export([start/2, stop/1]). start(_, _) -> {ok, DbModule} = application:get_env(my_db), ?TTY(DbModule), my_db_sup:start_link([DbModule]). stop(State) -> State. ==== -module(my_db_sup). -export([start_link/1]). -behaviour(supervisor). -export([init/1]). %% start_link(Opts) -> supervisor:start_link({local, ?MODULE}, ?MODULE, Opts). %% init([DbModule]) -> Server = {my_db, {DbModule, start_link, [[]]}, permanent, 2000, worker, []}, {ok, {{one_for_all, 0, 1}, [Server]}}. ==== -module(my_simple_db). -export([start_link/1]). -behaviour(gen_server). -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). start_link(Opts) -> gen_server:start_link({local, my_db}, ?MODULE, Opts, []). init(_) -> {ok, dict:new()}. handle_call({create, K, V}, _, Db) -> case dict:is_key(K, Db) of false -> {reply, {ok, K}, dict:store(K, V, Db)}; true -> {reply, {error, {exists, K}}, Db} end; handle_call({read, K}, _, Db) -> case dict:find(K, Db) of {ok, V} -> {reply, {ok, {K, V}}, Db}; error -> {reply, {error, not_found}, Db} end; handle_call({update, K, V}, _, Db) -> case dict:is_key(K, Db) of true -> {reply, {ok, K}, dict:store(K, V, Db)}; false -> {reply, {error, not_found}, Db} end; handle_call({delete, K}, _, Db) -> case dict:is_key(K, Db) of true -> {reply, {ok, K}, dict:erase(K, Db)}; false -> {reply, {error, not_found, K}, Db} end; handle_call(_, _, Db) -> {reply, error, Db}. handle_cast(_, State) -> {noreply, State}. handle_info(_, State) -> {noreply, State}. terminate(_, _State) -> ok. code_change(_, State, _) -> {ok, State}. ==== On Saturday, August 10, 2013 11:53:10 AM UTC-5, H.C. v. Stockhausen wrote: > > Hi, > > thank you for all your replies. I appreciate your help. > > The DB example, even if applicable to my use case, was perhaps a > little too specific. I don't necessarily need to abstract at a CRUD > level - a higher business level would work too. What I wanted to learn > is how to swap out implementations (at whatever level really), so > thank you for your samples, explanations and advise. > > I am happy though to have also learned that my initial ideaI wasn't > too bad. If I wanted to allow third parties to substitute their > implementations for mine (possibly in a closed source product even?) > it could be an option. > > Best regards, > Hans > > On 9 August 2013 19:23, Bohuslav Svancara > > wrote: > > Hello! > > > > I tried ContextErlang as an exercise: > > http://www.guidosalvaneschi.com/wp/software/contexterlang/ > > > > The "context programming" concept can be applied very well here (I > think). > > > > Here is an example what I am doing. > > > > (1) > > The "base" module "mt4_db_interface" implements MySql operations using > emysql. > > It also includes a definition for "context switch". (See > > mt4_db_interface_context/0 ): > > > > -module(mt4_db_interface). > > > > -behaviour(context_agent). > > > > % ContextErlang > > > -context_call([connect/5,get_c2_id_for_mt4_id/2,update_c2_id_for_mt4_id/3,delete/2]). > > > -context_cast([]). > > > > %% gen_server callbacks > > -export([init/1, handle_call/3, handle_cast/2, handle_info/2, > > terminate/2, code_change/3]). > > > > %% External exports > > > -export([connect/5,get_c2_id_for_mt4_id/2,delete/2,update_c2_id_for_mt4_id/3,start_link/0]). > > > > > % ContextErlang > > -include("../../ContextErlang/include/context_agent_api.hrl"). > > > > -define(PoolId,mt4). % a definition for emysql:add_pool(...) > > > > -record(mt4_db_interface_state, {dbHandle, > > dummy_database_for_tests=dict:new()}). % in the records.hrl in the > > real program > > > > start_link() -> context_agent:start_link({local, ?MODULE}, ?MODULE, [], > []). > > > > get_c2_id_for_mt4_id(SystemId,MT4OrderId) -> > > context_agent:call(?MODULE, {get_c2_id_for_mt4_id, > > SystemId,MT4OrderId}, 60000). > > > > update_c2_id_for_mt4_id(SystemId,MT4OrderId,C2SignalId) -> > > context_agent:call(?MODULE, {update_c2_id_for_mt4_id, > > SystemId,MT4OrderId,C2SignalId}, 60000). > > > > delete(SystemId,MT4OrderId) -> context_agent:call(?MODULE, {delete, > > SystemId,MT4OrderId}, 60000). > > > > connect(Host, Port, User, Password, DatabaseName) -> > > context_agent:call(?MODULE, {connect, Host, Port, User, Password, > > DatabaseName}, 60000). > > > > init([]) -> > > io:format("Starting ~p~n",[?MODULE]), > > {context,mt4_db_interface_context(),#mt4_db_interface_state{}}. > > > > handle_call({connect, Host, Port, User, Password, DatabaseName}, > > _From, State) -> > > Reply = do_connect(Host, Port, User, Password, DatabaseName), > > {reply, Reply, State}; > > > > handle_call({get_c2_id_for_mt4_id, SystemId,MT4OrderId}, _From, State) > -> > > Reply = do_get_c2_id_for_mt4_id(?PoolId,SystemId,MT4OrderId), > > {reply, Reply, State}; > > > > handle_call({update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, > > _From, State) -> > > Reply = > do_replace_c2_id_for_mt4_id(?PoolId,SystemId,MT4OrderId,C2SignalId), > > {reply, Reply, State}; > > > > handle_call({delete, SystemId,MT4OrderId}, _From, State) -> > > Reply = do_delete(?PoolId,SystemId,MT4OrderId), > > {reply, Reply, State}; > > > > handle_call(_Request, _From, State) -> > > Reply = ok, > > {reply, Reply, State}. > > > > handle_cast(_Msg, State) -> > > {noreply, State}. > > > > handle_info(_Info, State) -> > > {noreply, State}. > > > > terminate(_Reason, _State) -> > > ok. > > > > code_change(_OldVsn, State, _Extra) -> > > {ok, State}. > > > > %% > ============================================================================= > > > %% MySql implementations: emysql:execute(DbId,Select) > > do_connect(Host, Port, User, Password, DatabaseName)-> > > % Connect to MySql here > > emysql:add_pool(?PoolId,...)... > > ok. > > > > do_get_c2_id_for_mt4_id(DbId,SystemId,MT4OrderId) -> > > emysql:execute(...), .... > > do_replace_c2_id_for_mt4_id(DbId,SystemId,MT4OrderId,C2SignalId) -> > > emysql:execute(...), .... > > do_delete(DbId,SystemId,MT4OrderId) -> emysql:execute(...), .... > > > > %% ==================================================================== > > %% @doc ContextErlang. > > %% Create a context structure for this module. > > %% Defined variation 'use_dummy_db_for_mt4' is not active in the > > production mode. Activated just for unit tests. > > %% @end > > %% ==================================================================== > > mt4_db_interface_context() -> > > Spec = [ {what_db_to_use_slot, use_dummy_db_for_mt4} ], > > context_ADT:create(Spec). > > > > > > (2) > > A variation "use_dummy_db_for_mt4" which implements a fake database > > and is used for unit tests: > > > > %% > =========================================================================== > > %% Author: Bohuslav Svancara > > %% Created: 2013-07-20 > > %% Description: Experimental ContextErlang variation module for > > mt4_db_interface. > > %% > > %% This variation is used for unit tests where MySql is very hard to > use. > > %% > =========================================================================== > > > > -module(use_dummy_db_for_mt4). > > -export([handle_call/3,on_activation/1,on_deactivation/1,test/0]). > > > > > -context_call([connect/5,get_c2_id_for_mt4_id/2,update_c2_id_for_mt4_id/3,delete/2]). > > > -context_cast([]). > > > > -record(mt4_db_interface_state, {dbHandle, > > dummy_database_for_tests=dict:new()}). % in records.hrl in a real > > program > > > > handle_call({connect, Host, Port, User, Password, DatabaseName}, > > _From, State) -> > > {reply, ok, State#mt4_db_interface_state{ dummy_database_for_tests > > = dict:new()}}; > > > > handle_call({get_c2_id_for_mt4_id, SystemId,MT4OrderId}, _From, State) > -> > > Reply = case dict:find({SystemId,MT4OrderId}, > > State#mt4_db_interface_state.dummy_database_for_tests) of > > > > end, > > {reply, Reply, State}; > > > > handle_call({update_c2_id_for_mt4_id, SystemId,MT4OrderId,C2SignalId}, > > _From, State) -> > > NewDict = dict:update({SystemId,MT4OrderId}, fun(_) -> C2SignalId > > end, C2SignalId, > > State#mt4_db_interface_state.dummy_database_for_tests), > > {reply, ok, State#mt4_db_interface_state{dummy_database_for_tests > > = NewDict}}; > > > > handle_call({delete, SystemId,MT4OrderId}, _From, State) -> > > NewDict = dict:erase({SystemId,MT4OrderId}, > > State#mt4_db_interface_state.dummy_database_for_tests), > > {reply, ok, State#mt4_db_interface_state{dummy_database_for_tests > > = NewDict}}. > > > > on_activation(State) -> {ok,State}. > > > > on_deactivation(State) -> {ok,State}. > > > > > > (3) > > I a production code the mt4_db_interface module is used: > > > > {ok, Host, Port, User, Password, DatabaseName} = get_db_config(), > > mt4_db_interface:start_link(), > > mt4_db_interface:connect(Host, Port, User, Password, DatabaseName),... > > > > mt4_db_interface:get_c2_id_for_mt4_id(...) > > mt4_db_interface:update_c2_id(...) > > mt4_db_interface:delete(...) > > > > (4) > > In unit tests mt4_db_interface is switched (dynamic switch - the most > > interesting thing in context programming) to "use_dummy_db_for_mt4" > > implementation: > > > > mt4_db_interface:start_link(), > > mt4_db_interface:in_cur_context_activate(mt4_db_interface, > > use_dummy_db_for_mt4, what_db_to_use_slot), > > mt4_db_interface:connect("DummyHost", 0, "DummyUser", > > "DummyPassword", "DummyDatabaseName"), > > > > mt4_db_interface:get_c2_id_for_mt4_id(...) > > mt4_db_interface:update_c2_id(...) > > mt4_db_interface:delete(...) > > > > > > Is there any other person trying context programming in Erlang? > > > > Sincerely, > > Bohuslav Svancara > > > > > > 2013/8/9 H.C. v. Stockhausen > > >> > >> Hello, > >> > >> I need a DB backend for my application but I'd like to be able to swap > >> it out for different DBs if I choose so later on. > >> > >> I would like to code against an interface and tell the application > >> what specific backend to use through config rather than code changes. > >> > >> Is there a pattern for doing that? Hot code upgrades and multi-node > >> are not a real concern at this time - mostly, since I have no > >> experience yet with either - however if doing it right means taking > >> that into account too I'd also like to learn more about that. > >> > >> I am thinking of defining a custom behaviour ("my_crud" perhaps), then > >> to implement it for various DBs and to also let a config driven > >> adapter implement it that I then use to throughout the code to talk to > >> the DB layer. > >> > >> For example, using Mnesia and AWS DynamoDB: > >> > >> - my_crud.erl (behaviour) > >> - my_db_mnesia.erl (implements behaviour) > >> - my_db_dynamo.erl (implements behaviour) > >> - my_db.erl (configurable adapter that also implements behaviour) > >> - my.config > >> > >> my_db:insert(Key, Value). > >> > >> Is that a reasonable approach that makes proper use of Erlang and > >> behaviours or is this just not how one should do it? > >> > >> Thank you for any help & best regards, > >> Hans > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-q...@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > > erlang-questions mailing list > > erlang-q...@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Sun Aug 11 07:21:46 2013 From: serge@REDACTED (Serge Aleynikov) Date: Sun, 11 Aug 2013 01:21:46 -0400 Subject: [erlang-questions] ErlExec stdin and other questions In-Reply-To: References: Message-ID: <52071F6A.90308@aleynikov.org> I fixed the STDOUT buffering issue with erlexec. Now all STDOUT and STDERR output should arrive as soon as the child OS process generates it: 1> exec:start([]). {ok, <0.35.0>} 2> exec:run("for i in 1 2 3; do sleep 1; echo \"Iter$i\"; done", [{stdout, fun(S,OsPid,D) -> io:format("Got ~w from ~w: ~p\n", [S,OsPid,D]) end}]). {ok,<0.37.0>,2452} Got stdout from 2452: <<"Iter1\n">> Got stdout from 2452: <<"Iter2\n">> Got stdout from 2452: <<"Iter3\n">> See https://github.com/saleyn/erlexec. On Wed, Aug 7, 2013 at 3:36 AM, OvermindDL1 > wrote: Created a complete test-case example of my issue with erlexec, and I ran this same script on 3 different computers running variations of debian versions. Feel free to toss in application:start(sasl), at the top of the erlang script area if you wish, and yes the (minecraft) server software uses stderr instead of stdout, no I do not know why, and yes as you can see it does not return anything to the erlang system until it is killed on all the systems I tested. Using Erlang's ports you get each line as a message as that is how the server flushes it and it is instant. Yes it is ugly, but it is pared down from what it grew from and no need to re-do the receive's better in such a test. test_exec_mc.sh """ #!/bin/sh [ -d erlexec ] || git clone https://github.com/saleyn/erlexec.git cd erlexec rebar compile [ -f 'minecraft_server.1.6.2.jar' ] || wget 'https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar' erl -pa $PWD/ebin -eval ' application:start(exec), {_, P, _} = exec:run_link("java -Xms1G -Xmx1G -jar minecraft_server.1.6.2.jar nogui", [{stdout, self()}, {stderr, self()}]), timer:sleep(1000), receive A0 -> io:format("Message ~p: ~p~n", [now(), A0]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A1 -> io:format("Message ~p: ~p~n", [now(), A1]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A2 -> io:format("Message ~p: ~p~n", [now(), A2]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A3 -> io:format("Message ~p: ~p~n", [now(), A3]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A4 -> io:format("Message ~p: ~p~n", [now(), A4]) after 1000 -> io:format("timeout ~p~n", [now()]) end, timer:sleep(1000), receive A5 -> io:format("Message ~p: ~p~n", [now(), A5]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A6 -> io:format("Message ~p: ~p~n", [now(), A6]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A7 -> io:format("Message ~p: ~p~n", [now(), A7]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A8 -> io:format("Message ~p: ~p~n", [now(), A8]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A9 -> io:format("Message ~p: ~p~n", [now(), A9]) after 1000 -> io:format("timeout ~p~n", [now()]) end, timer:sleep(10000), receive A10 -> io:format("Message ~p: ~p~n", [now(), A10]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A11 -> io:format("Message ~p: ~p~n", [now(), A11]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A12 -> io:format("Message ~p: ~p~n", [now(), A12]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A13 -> io:format("Message ~p: ~p~n", [now(), A13]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A14 -> io:format("Message ~p: ~p~n", [now(), A14]) after 1000 -> io:format("timeout ~p~n", [now()]) end, exec:kill(P, 9), receive A15 -> io:format("Message ~p: ~p~n", [now(), A15]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A16 -> io:format("Message ~p: ~p~n", [now(), A16]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A17 -> io:format("Message ~p: ~p~n", [now(), A17]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A18 -> io:format("Message ~p: ~p~n", [now(), A18]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A19 -> io:format("Message ~p: ~p~n", [now(), A19]) after 1000 -> io:format("timeout ~p~n", [now()]) end, timer:sleep(1000), receive A20 -> io:format("Message ~p: ~p~n", [now(), A20]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A21 -> io:format("Message ~p: ~p~n", [now(), A21]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A22 -> io:format("Message ~p: ~p~n", [now(), A22]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A23 -> io:format("Message ~p: ~p~n", [now(), A23]) after 1000 -> io:format("timeout ~p~n", [now()]) end, receive A24 -> io:format("Message ~p: ~p~n", [now(), A24]) after 1000 -> io:format("timeout ~p~n", [now()]) end, q().' """ On Tue, Aug 6, 2013 at 10:49 PM, OvermindDL1 > wrote: Excepting this, is there any other better way to launch a program as another user? I might just use ssh certs and ssh to localhost if not unless any other ideas? On Tue, Aug 6, 2013 at 4:22 AM, OvermindDL1 > wrote: I have been attempting to use ErlExec as I noticed that it had a few new features added six days ago to forward the stdout/stderr to an Erlang PID, however I am having some issue. The process that erlexec runs occasionally does not die when the Erlang VM dies, thus keeping it and its children running. Also I need a way to send commands back to the children, but I am either not seeing a command for that or have not figured out the format to send a message, which is difficult because of the main issue I have, when I have it launch a fairly large server software (that is not a daemon for whatever reason, hence why I am wrapping it), the Erlang shell seems to lose a great deal of input that I type in, only catching from every 1 in 4 I type to 1 in 400 characters I type, and it does not seem to matter based on the speed I hit keys or holding a key or anything of the sort, just purely on keypress down, and it remains like that until I kill the child process that I started through erlexec. Using the normal Erlang Ports works fine and has no such issues (other than the child not coming down with the VM either and no direct kill command when the child hangs, which it occasionally does, but os:cmd("kill ...") works around that well enough. Is ErlExec stable enough for high use? Or am I doing something wrong? _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Aug 11 15:19:37 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Sun, 11 Aug 2013 15:19:37 +0200 Subject: [erlang-questions] Possibly bug in cth_log_redirect? In-Reply-To: References: Message-ID: <52078F69.8050307@ninenines.eu> On 08/09/2013 10:12 PM, Max Lapshin wrote: > When tests are running in parallel, LogFun in cth_log_redirect is > changed to ct_log: > > https://github.com/erlang/otp/blob/3021fca734f71f8bae966ab67f1400d37f8927bc/lib/common_test/src/cth_log_redirect.erl#L49 > > Problem is that it must be not ct_log, but tc_log: > > https://github.com/erlang/otp/blob/3021fca734f71f8bae966ab67f1400d37f8927bc/lib/common_test/src/ct_logs.erl#L44 I am also hit by this issue. Full error message: =ERROR REPORT==== 11-Aug-2013::15:16:09 === ** gen_event handler cth_log_redirect crashed. ** Was installed in error_logger ** Last event was: {error,<0.390.0>, {emulator,"~s~n", ["Error in process <0.620.0> on node 'ct@REDACTED' with exit value: {{<<18 bytes>>,{stacktrace,[{http_errors,handle,2,[{file,\"test/http_SUITE_data/http_errors.erl\"},{line,37}]},{cowboy_handler,handler_handle,4,[{file,\"src/cowboy_handler.erl\"},{line,115}]},{cowboy_protocol,execute,4,[{file,\"src/cowbo... \n"]}} ** When handler state == ct_log ** Reason == {'function not exported', [{ct_logs,ct_log, [error_logger,50,"System", ["\n",61,"ERROR REPORT",61,61,61,61,32,"11",45,"Aug", 45,"2013",58,58,"17",58,"16",58,"09",32,61,61,61,"\n", "Error in process <0.620.0> on node 'ct@REDACTED' with exit value: {{<<18 bytes>>,{stacktrace,[{http_errors,handle,2,[{file,\"test/http_SUITE_data/http_errors.erl\"},{line,37}]},{cowboy_handler,handler_handle,4,[{file,\"src/cowboy_handler.erl\"},{line,115}]},{cowboy_protocol,execute,4,[{file,\"src/cowbo... \n", "\n"], []], []}, {cth_log_redirect,handle_event,2, [{file,"cth_log_redirect.erl"},{line,91}]}, {gen_event,server_update,4, [{file,"gen_event.erl"},{line,522}]}, {gen_event,server_notify,4, [{file,"gen_event.erl"},{line,504}]}, {gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,266}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]} -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From serge@REDACTED Sun Aug 11 15:31:32 2013 From: serge@REDACTED (Serge Aleynikov) Date: Sun, 11 Aug 2013 09:31:32 -0400 Subject: [erlang-questions] Erlang shell lag when using open_port/2 Message-ID: <52079234.9080106@aleynikov.org> While looking at the issue described here [1] I drilled the issue down to the following problem with erlang:open_port/2 (note: get minecraft_server.1.6.2.jar here [2]): When executing: f(P), P = erlang:open_port({spawn, "java -Xms1G -Xmx1G -jar minecraft_server.1.6.2.jar nogui"}, [binary, exit_status, nouse_stdio]). The Erlang shell has a very noticeable lag between key presses, and seems to "swallow" some typed keys. On the other hand, when running the same command without 'nouse_stdio' option, the Erlang shell behaves as expected. Is this a bug? If not, how can I take advantage of nouse_stdio and not have that impact on the Erlang shell? Serge [1] https://github.com/saleyn/erlexec/issues/17 [2] wget 'https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar' From co7eb@REDACTED Sun Aug 11 17:14:16 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Sun, 11 Aug 2013 11:14:16 -0400 Subject: [erlang-questions] Unlike behavior in -spec and type importing Message-ID: <000001ce96a5$76d65010$6482f030$@frcuba.co.cu> Hi all, I was dealing with edoc and -spec attribute and I found a little not liked behavior regards importing types from another modules. For examples I was setting the specification of start/2 application start point function like this: %%%--------------------------------------------------------------------- %%% @doc %%% Start point of the application. %%% @end %%%--------------------------------------------------------------------- -spec start(StartType, StartArgs) -> {'ok', Pid} | {'ok', Pid, State} when StartType :: application:start_type(), %% @see application module for start_type() type specification. StartArgs :: term(), Pid :: pid(), State :: term(). start(StartType, StartArgs)-> . . . I check the source of application module to see if the start_type() type was exported and it is not. And my module compiles well, and even I changed to StartType :: unknown_module: unknown_type() type and it still compiles, It seems that there are no type checking outside of the module. All this could seems a little nothing important thing but, I like to program in Erlang a lot, even without using any IDE, I like notepad++ with Erlang colors definition. And I?m proud of my code and the language even more since I learned that I can do thing like -spec, -type and documentation. The same still happens with functions, I can do non_declared_module:non_declared_fun() and it will compile. Only in runtime it will fail. Cheers, Ivan. NOTE: I apologies for any bad intention you could think I have with this, I stand for Erlang and its Community. -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Sun Aug 11 17:27:19 2013 From: colanderman@REDACTED (Chris King) Date: Sun, 11 Aug 2013 11:27:19 -0400 Subject: [erlang-questions] Unlike behavior in -spec and type importing In-Reply-To: <000001ce96a5$76d65010$6482f030$@frcuba.co.cu> References: <000001ce96a5$76d65010$6482f030$@frcuba.co.cu> Message-ID: Hi Ivan, On Sun, 11 Aug 2013 11:14:16 -0400, Ivan Carmenates Garc?a wrote: > > >> I check the source of application module to see if the start_type() >> type was exported and it >is not. > > And my module compiles well, and even I changed to StartType :: > unknown_module: unknown_type>() type and it still compiles, > > It seems that there are no type checking outside of the module. Are you using Dialyzer to check types? The Erlang compiler itself does not perform typechecking. You can run Dialyzer (included with OTP) on your source code like this: $ dialyzer --src or, if you compile your code with debugging enabled, you can run it like this: $ dialyzer In addition to inferring types, Dialyzer will check that code adheres to -spec declarations, and it will warn about unexported types. Note that you have to run Dialyzer on *all* your modules, as it only sees the modules you tell it about, and those whose specifications have been explicitly stored in its "PLT" database (by default erts, kernel, and stdlib). -------------- next part -------------- An HTML attachment was scrubbed... URL: From overminddl1@REDACTED Sun Aug 11 19:53:52 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Sun, 11 Aug 2013 11:53:52 -0600 Subject: [erlang-questions] Erlang shell lag when using open_port/2 In-Reply-To: <52079234.9080106@aleynikov.org> References: <52079234.9080106@aleynikov.org> Message-ID: For note, the 'swallowed' keys pressed in the Erlang shell seem to get passed to stdin of the launched child process, and not all input is swallowed, it seems quite random whether it gets passed to the Erlang shell or to the launched process. On Aug 11, 2013 7:34 AM, "Serge Aleynikov" wrote: > While looking at the issue described here [1] I drilled the issue down > to the following problem with erlang:open_port/2 (note: get > minecraft_server.1.6.2.jar here [2]): > > When executing: > > f(P), P = erlang:open_port({spawn, "java -Xms1G -Xmx1G -jar > minecraft_server.1.6.2.jar nogui"}, [binary, exit_status, nouse_stdio]). > > The Erlang shell has a very noticeable lag between key presses, and > seems to "swallow" some typed keys. > > On the other hand, when running the same command without 'nouse_stdio' > option, the Erlang shell behaves as expected. > > Is this a bug? If not, how can I take advantage of nouse_stdio and not > have that impact on the Erlang shell? > > Serge > > [1] https://github.com/saleyn/erlexec/issues/17 > [2] wget > ' > https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar > ' > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sun Aug 11 20:04:35 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sun, 11 Aug 2013 20:04:35 +0200 Subject: [erlang-questions] Erlang shell lag when using open_port/2 In-Reply-To: References: <52079234.9080106@aleynikov.org> Message-ID: Sounds like you have two processes connected to the same stdin source, which both read all the time. Have you tried adding ": > For note, the 'swallowed' keys pressed in the Erlang shell seem to get > passed to stdin of the launched child process, and not all input is > swallowed, it seems quite random whether it gets passed to the Erlang shell > or to the launched process. > On Aug 11, 2013 7:34 AM, "Serge Aleynikov" wrote: > >> While looking at the issue described here [1] I drilled the issue down >> to the following problem with erlang:open_port/2 (note: get >> minecraft_server.1.6.2.jar here [2]): >> >> When executing: >> >> f(P), P = erlang:open_port({spawn, "java -Xms1G -Xmx1G -jar >> minecraft_server.1.6.2.jar nogui"}, [binary, exit_status, nouse_stdio]). >> >> The Erlang shell has a very noticeable lag between key presses, and >> seems to "swallow" some typed keys. >> >> On the other hand, when running the same command without 'nouse_stdio' >> option, the Erlang shell behaves as expected. >> >> Is this a bug? If not, how can I take advantage of nouse_stdio and not >> have that impact on the Erlang shell? >> >> Serge >> >> [1] https://github.com/saleyn/erlexec/issues/17 >> [2] wget >> ' >> https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar >> ' >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From co7eb@REDACTED Sun Aug 11 23:02:31 2013 From: co7eb@REDACTED (=?UTF-8?Q?Ivan_Carmenates_Garc=C3=ADa?=) Date: Sun, 11 Aug 2013 17:02:31 -0400 Subject: [erlang-questions] Unlike behavior in -spec and type importing In-Reply-To: References: <000001ce96a5$76d65010$6482f030$@frcuba.co.cu> Message-ID: <001501ce96d6$1e58d8f0$5b0a8ad0$@frcuba.co.cu> Hi Chris, I didn?t knew about the function of the dialyzer tool application, I read the documentation and I finally got it to work, It was a little hard because I?m using windows but I finally came with a solution: When I tried to start the dialyzer I got an error of it couldn?t find the PLT file etc, etc. I read about an Environment Variable called DIALYZER_PLT and I gave it the value c:\Program Files\erl5.10.1\usr\PLT\countlink.plt So I built the PLT file using the common applications I use in my project dialyzer --build_plt --output_plt "c:\Program Files\erl5.10.1\usr\PLT\countlink.plt" --apps stdlib kernel Them I use another script to update the PLT file with my beams and at the same time get a result. @echo off mode 100, 40 dialyzer --add_to_plt -r "d:\WORK\countlink-1.0\src" --output_plt "c:\Program Files\erl5.10.1\usr\PLT\countlink.plt" echo. echo. echo. echo ************************************************************* echo * Dialyzer Results * echo ************************************************************* dialyzer -r "d:\WORK\countlink-1.0\src" echo. pause I wonder about what for is the PLT file but I think, following my guessing that it content the necessary information of types and functions exporting and many other things. Because I did test it excluding for example, one my beams with debug information which had an exported type, I use that type in another module included in the PLT and I got an error of undefined type, then I include it again and resulted in no errors. That?s very good the Dialyzer application, but I still think that the compiler should be aware of that minor details, maybe not something like unreachable code detection or unnecessary testing like Dialyser do, still it could also do that, Microsoft CShard Compiler does it, ups sorry about that ;). Well I hope in the future the community fix or adds that little things to the Erlang compiler that is so important (I think one of the most important modules). Cheers, Ivan. De: Chris King [mailto:colanderman@REDACTED] Enviado el: domingo, 11 de agosto de 2013 11:27 Para: erlang-questions@REDACTED; Ivan Carmenates Garc?a Asunto: Re: [erlang-questions] Unlike behavior in -spec and type importing Hi Ivan, On Sun, 11 Aug 2013 11:14:16 -0400, Ivan Carmenates Garc?a < co7eb@REDACTED> wrote: I check the source of application module to see if the start_type() type was exported and it is not. And my module compiles well, and even I changed to StartType :: unknown_module: unknown_type() type and it still compiles, It seems that there are no type checking outside of the module. Are you using Dialyzer to check types? The Erlang compiler itself does not perform typechecking. You can run Dialyzer (included with OTP) on your source code like this: $ dialyzer --src or, if you compile your code with debugging enabled, you can run it like this: $ dialyzer In addition to inferring types, Dialyzer will check that code adheres to -spec declarations, and it will warn about unexported types. Note that you have to run Dialyzer on *all* your modules, as it only sees the modules you tell it about, and those whose specifications have been explicitly stored in its "PLT" database (by default erts, kernel, and stdlib). -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Aug 12 06:57:11 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Mon, 12 Aug 2013 16:57:11 +1200 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: References: Message-ID: <6A61745A-762D-4C6E-9AA9-C37C02582780@cs.otago.ac.nz> On 10/08/2013, at 3:34 AM, Garrett Smith wrote: > Joe's response to this is quite comprehensive and I'll just echo it here. I was thrilled to see Joe's response and to see it seconded. The "generic interface" idea makes quite a lot of sense when the things you want to interface to are essentially the same. The major case of that is SQL data bases, where there is an international standard (or are international standards) and the various systems out there (Firebird, MariaDB, Postgresql) are tolerably close, although it's astonishing how different they can be (and not just in nonstandard features). But that's a matter of papering over inessential details of fundamentally similar things. And then you end up with an interface (ODBC or JDBC) that is harder to learn and drive than the thing it interfaces to. A similar example would be the POSIX "ndbm" interface, which could be backed by original nbdm, by sdbm, by gdbm, or by BDB. You end up imposing the limitations of the weakest/most dangerous back end on the most powerful/safest. One of the XP slogans is YAGNI: "You Ain't Gonna Need it". From bengt.kleberg@REDACTED Mon Aug 12 08:43:21 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Mon, 12 Aug 2013 08:43:21 +0200 Subject: [erlang-questions] ot: Bret Victor The Future of Programming - YouTube Message-ID: <1376289801.4796.3.camel@sekic1152.release> Greetings, Good 30 minute talk that pretends to be 40 years old. After 25 minutes he fake prophecies that maybe actors could be picked up by a Swedish telecom company... bengt From sedrik@REDACTED Mon Aug 12 08:48:52 2013 From: sedrik@REDACTED (Fredrik Andersson) Date: Mon, 12 Aug 2013 08:48:52 +0200 Subject: [erlang-questions] ot: Bret Victor The Future of Programming - YouTube In-Reply-To: <1376289801.4796.3.camel@sekic1152.release> References: <1376289801.4796.3.camel@sekic1152.release> Message-ID: I guess you meant to include the youtube link as well. Here is what a search gave me, seems correct. https://www.youtube.com/watch?v=8pTEmbeENF4 On Mon, Aug 12, 2013 at 8:43 AM, Bengt Kleberg wrote: > Greetings, > > Good 30 minute talk that pretends to be 40 years old. After 25 minutes > he fake prophecies that maybe actors could be picked up by a Swedish > telecom company... > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peppe@REDACTED Mon Aug 12 10:20:35 2013 From: peppe@REDACTED (Peter Andersson) Date: Mon, 12 Aug 2013 10:20:35 +0200 Subject: [erlang-questions] Possibly bug in cth_log_redirect? In-Reply-To: <52078F69.8050307@ninenines.eu> References: <52078F69.8050307@ninenines.eu> Message-ID: <52089AD3.4080600@erlang.org> Thanks for reporting this, guys! We'll fix it asap. /Peter Ericsson AB, Erlang/OTP Lo?c Hoguin wrote: > On 08/09/2013 10:12 PM, Max Lapshin wrote: > > When tests are running in parallel, LogFun in cth_log_redirect is > > changed to ct_log: > > > > https://github.com/erlang/otp/blob/3021fca734f71f8bae966ab67f1400d37f8927bc/lib/common_test/src/cth_log_redirect.erl#L49 > > > > Problem is that it must be not ct_log, but tc_log: > > > > https://github.com/erlang/otp/blob/3021fca734f71f8bae966ab67f1400d37f8927bc/lib/common_test/src/ct_logs.erl#L44 > > I am also hit by this issue. > > Full error message: > > =ERROR REPORT==== 11-Aug-2013::15:16:09 === > ** gen_event handler cth_log_redirect crashed. > ** Was installed in error_logger > ** Last event was: {error,<0.390.0>, > {emulator,"~s~n", > ["Error in process <0.620.0> on > node 'ct@REDACTED' with exit value: {{<<18 > bytes>>,{stacktrace,[{http_errors,handle,2,[{file,\"test/http_SUITE_data/http_errors.erl\"},{line,37}]},{cowboy_handler,handler_handle,4,[{file,\"src/cowboy_handler.erl\"},{line,115}]},{cowboy_protocol,execute,4,[{file,\"src/cowbo... > \n"]}} > ** When handler state == ct_log > ** Reason == {'function not exported', > [{ct_logs,ct_log, > [error_logger,50,"System", > ["\n",61,"ERROR > REPORT",61,61,61,61,32,"11",45,"Aug", > > 45,"2013",58,58,"17",58,"16",58,"09",32,61,61,61,"\n", > "Error in process <0.620.0> on node > 'ct@REDACTED' with exit value: {{<<18 > bytes>>,{stacktrace,[{http_errors,handle,2,[{file,\"test/http_SUITE_data/http_errors.erl\"},{line,37}]},{cowboy_handler,handler_handle,4,[{file,\"src/cowboy_handler.erl\"},{line,115}]},{cowboy_protocol,execute,4,[{file,\"src/cowbo... > \n", > "\n"], > []], > []}, > {cth_log_redirect,handle_event,2, > [{file,"cth_log_redirect.erl"},{line,91}]}, > {gen_event,server_update,4, > [{file,"gen_event.erl"},{line,522}]}, > {gen_event,server_notify,4, > [{file,"gen_event.erl"},{line,504}]}, > > {gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,266}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,239}]}]} > > > From eric@REDACTED Tue Aug 13 17:46:07 2013 From: eric@REDACTED (Eric Moritz) Date: Tue, 13 Aug 2013 11:46:07 -0400 Subject: [erlang-questions] How to code against an interface rather than an implementation? In-Reply-To: <6A61745A-762D-4C6E-9AA9-C37C02582780@cs.otago.ac.nz> References: <6A61745A-762D-4C6E-9AA9-C37C02582780@cs.otago.ac.nz> Message-ID: I have found that making a generic interface for my projects tends to be a premature optimization and unless the project needs swappable databases, a proxy module tends to be overkill. The approach I take is to write a module that has an API that hides the details of the implementation. That way in the future, if the need arises, I can completely rewrite the implementation without having to refactor the clients of the API. I can add connection pooling, or completely replace the database with something else. Eric Moritz. -------------- next part -------------- An HTML attachment was scrubbed... URL: From reaperman123456789@REDACTED Wed Aug 14 13:20:44 2013 From: reaperman123456789@REDACTED (reaperman123456789@REDACTED) Date: Wed, 14 Aug 2013 04:20:44 -0700 (PDT) Subject: [erlang-questions] Maintaining state between application failover Message-ID: <6acd9d77-fcd6-4161-a398-bbf597c3150d@googlegroups.com> Assuming a distributed application, how could state between application starts due to failover be maintained? For illustration purposes, consider the following problem: We want a kind of server that delivers unique numbers. Starting at 0, on each request this number is delivered and incremented. For implementation, we use a gen_server process that keeps the current number in it's state. We put that process under a one-for-one supervisor, which serves as the top supervisor of the application. Now, even in a non-distributed setup, the gen_server could not maintain the state between restarts managed by it's supervisor. We could store the current number in the environment of the application itself (which doesn't feel right, but for illustration purposes let's keep it in mind), where it would survive restarts of the gen_server process. In a distributed setup, even the state stored in the application environment would not survive in case of an application failover. When the node on which the application is running dies and is restarted on another node, it starts at 0 again. Performance considerations aside, the current number could be constantly kept and updated permanently in a state file which could then be read at startup. But since the nodes would usually be running on different machines, on failover the application would be restartet on another machine than the one where the file resided, and since the reason for the death of the erlang node is presumable the death of the hardware node, would not be accessible from the application started in failover mode. For keeping the state file accessible everywhere, we would need to put it on a NFS mount or something, but the NFS server would become a critical component in our setup, not to mention the overkill of running an extra machine for the single purpose of sharing a single file which would not exceed a few bytes in size. Using a database of whatever flavor is essentially the same. So, how could state be efficiently maintained in the erlang way of doing things? To clarify, I am *not* asking for a solution to the problem of generating unique numbers, there a probably a thousand ways to do this in a better way, UUID and whatnot. I am asking for ways to maintain state between restarts of a distributed application in a failover scenario. The example problem above is a special case of the general problem I am asking about, made up purely for the purpose of having a simple illustration. A solution of the general problem would automatically solve the special case, anyway ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From co7eb@REDACTED Wed Aug 14 14:53:24 2013 From: co7eb@REDACTED (=?UTF-8?Q?Ivan_Carmenates_Garc=C3=ADa?=) Date: Wed, 14 Aug 2013 08:53:24 -0400 Subject: [erlang-questions] Maintaining state between application failover Message-ID: <000601ce98ed$48422a90$d8c67fb0$@frcuba.co.cu> Hi, I think you should have the state of the server updated in all nodes, so when a node goes down it will be restarted with the current state. If entire application in all nodes goes down I think there is no sense in having the last state you can start over from the initial state (0 in this case) since the application will start entirely over. But if it does have any sense for other purposes, then when the fail over process start you should store the last state in a distribute storage manager like dets or mnesia and then you can restart the entire application having the lasted state stored in one of the distributed nodes (you don?t need to know which node because the distributed storage manager will give you the most updated information). Best Regards, Ivan. De: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] En nombre de reaperman123456789@REDACTED Enviado el: mi?rcoles, 14 de agosto de 2013 7:21 Para: erlang-programming@REDACTED Asunto: [erlang-questions] Maintaining state between application failover Assuming a distributed application, how could state between application starts due to failover be maintained? For illustration purposes, consider the following problem: We want a kind of server that delivers unique numbers. Starting at 0, on each request this number is delivered and incremented. For implementation, we use a gen_server process that keeps the current number in it's state. We put that process under a one-for-one supervisor, which serves as the top supervisor of the application. Now, even in a non-distributed setup, the gen_server could not maintain the state between restarts managed by it's supervisor. We could store the current number in the environment of the application itself (which doesn't feel right, but for illustration purposes let's keep it in mind), where it would survive restarts of the gen_server process. In a distributed setup, even the state stored in the application environment would not survive in case of an application failover. When the node on which the application is running dies and is restarted on another node, it starts at 0 again. Performance considerations aside, the current number could be constantly kept and updated permanently in a state file which could then be read at startup. But since the nodes would usually be running on different machines, on failover the application would be restartet on another machine than the one where the file resided, and since the reason for the death of the erlang node is presumable the death of the hardware node, would not be accessible from the application started in failover mode. For keeping the state file accessible everywhere, we would need to put it on a NFS mount or something, but the NFS server would become a critical component in our setup, not to mention the overkill of running an extra machine for the single purpose of sharing a single file which would not exceed a few bytes in size. Using a database of whatever flavor is essentially the same. So, how could state be efficiently maintained in the erlang way of doing things? To clarify, I am not asking for a solution to the problem of generating unique numbers, there a probably a thousand ways to do this in a better way, UUID and whatnot. I am asking for ways to maintain state between restarts of a distributed application in a failover scenario. The example problem above is a special case of the general problem I am asking about, made up purely for the purpose of having a simple illustration. A solution of the general problem would automatically solve the special case, anyway ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Wed Aug 14 15:27:32 2013 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Wed, 14 Aug 2013 15:27:32 +0200 Subject: [erlang-questions] Generating exprs on the fly Message-ID: Hi, Tools dedicated to generate Erlang code are the erl_{scan,parse,syntax} and compile modules. However, in order to build and load code at runtime one is forced to put one's code into a module and then an exported function. Is it not possible to build and load small pieces of code, such as fun expressions or a list of atoms, without the constraint of being inside a thus generated module/exported function? I imagine it must go against some Erlang rules (wouldn't it be considered global state?), but it somehow feels cleaner than to have to generate a module and export a function just to access runtime-variable code. (What I am actually doing is scanning & parsing a "query string" then building a list comprehension out of it, then executing the LC. Thus the need for a runtime-generated fun-expr). Pierre -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Wed Aug 14 15:39:07 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Wed, 14 Aug 2013 15:39:07 +0200 Subject: [erlang-questions] Generating exprs on the fly In-Reply-To: References: Message-ID: Hello, The answer is no, but you can always use erl_eval. Regards, -- Anthony Ramine Le 14 ao?t 2013 ? 15:27, Pierre Fenoll a ?crit : > Is it not possible to build and load small pieces of code, such as fun expressions or a list of atoms, without the constraint of being inside a thus generated module/exported function? From max.lapshin@REDACTED Wed Aug 14 15:41:32 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 14 Aug 2013 17:41:32 +0400 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? Message-ID: Right now I fight with problem of streaming 10 Gbit/s of video and more from single server. I have met different problems (mostly about locking into single process like with file:read_file), but right now I have one question, it is about memory allocation. I've started strace against one of threads running scheduler and saw that it makes lot of mmap/munmap. For example during second 1376487042 it has called mmap 72 times with sizes 1052672, 2101248. Such lines are very interesting for me: 1376487042.167246 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f49bf000 1376487042.174690 munmap(0x7fe90c6ff000, 1052672) = 0 erlang asked system for one megabyte of memory and after 7 milliseconds returned it back. To make it clear: erlyvideo is reading 700-800 megabytes per second from several hard drives, repack them a bit and sends to network via TCP sockets. It is possible to tell erlang that it can use 20GB or 30 GB or ram and don't return it back at all? I suppose that such often allocation/deallocation leads to high cpu usage of process kswapd0 that (perhaps) tries to do something with such big amount of freed pages. I have looked at erts_alloc.html page. Well, it is scary a bit, just like when I've looked into the airplane cabin. Should I change any options from this page or I shouldn't touch memory allocator at all? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Aug 14 16:26:11 2013 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 14 Aug 2013 16:26:11 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: Hello Max, You have forgot to mention which version of Erlang/OTP you are running. Things have changed quite a bit in the last releases of Erlang on this front. You can get a lot of information about what is actually allocated by looking at erlang:system_info(allocator) and erlang:system_info({allocator,Alloc}). And this information can guide you in you decision of what flags you might want to set. In your case it sounds like maybe you end up allocating a lot of sbcs (single block carriers) instead of getting mbcs (multi block carriers). You can check this by looking at the mbcs and sbcs section in the info you get from erlang:system_info({allocator,$type_alloc}). If you notice you have a lot of sbcs you might want to raise the threashold for when a sbc is used, +MB**sbct 1024. You might want to look at if the calls section in any allocator has almost as many to $type_alloc as it does to mseg_alloc. eg: 1> erlang:system_info({allocator,binary_alloc}). [{instance,0, [{versions,"0.9","3.0"}, -- snip -- {calls,[{binary_alloc,0,77}, {binary_free,0,76}, {binary_realloc,0,0}, {mseg_alloc,0,3}, {mseg_dealloc,0,2}, {mseg_realloc,0,0}, {sys_alloc,0,0}, {sys_free,0,0}, {sys_realloc,0,0}]}]}] We can in the above see that there were 77 calls to binary_alloc and we did 3 mseg_alloc. One mseg_alloc roughly corresponds to a mmap (there is also a mseg cache, so the actual number of mmaps could be less). By playing with sbct and maybe mmbcs (this sets a static mbc that is allocated and kept from though out the lifetime of the beam) you could get better statistics. Note that IIRC mmbcs only works for mbcs so make sure that it is mbcs objects which you are allocating before twiddling with it. Yet another idea could be to play with the mseg cache parameters, MMamcbf, MMrmcbf and MMmcs. By making the cache keep more carriers you could up the cache hit rate and thus reduce mmaps. Lukas On Wed, Aug 14, 2013 at 3:41 PM, Max Lapshin wrote: > Right now I fight with problem of streaming 10 Gbit/s of video and more > from single server. > > I have met different problems (mostly about locking into single process > like with file:read_file), > but right now I have one question, it is about memory allocation. > > I've started strace against one of threads running scheduler and saw that > it makes lot of mmap/munmap. > > For example during second 1376487042 it has called mmap 72 times with > sizes 1052672, 2101248. > > Such lines are very interesting for me: > > 1376487042.167246 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f49bf000 > 1376487042.174690 munmap(0x7fe90c6ff000, 1052672) = 0 > > > erlang asked system for one megabyte of memory and after 7 milliseconds > returned it back. > > To make it clear: erlyvideo is reading 700-800 megabytes per second from > several hard drives, repack them a bit and sends to network via TCP sockets. > > > > It is possible to tell erlang that it can use 20GB or 30 GB or ram and > don't return it back at all? > > I suppose that such often allocation/deallocation leads to high cpu usage > of process kswapd0 that (perhaps) tries to do something with such big > amount of freed pages. > > I have looked at erts_alloc.html page. Well, it is scary a bit, just like > when I've looked into the airplane cabin. Should I change any options from > this page or I shouldn't touch memory allocator at all? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Wed Aug 14 16:35:57 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 14 Aug 2013 16:35:57 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: <520B95CD.5000305@erix.ericsson.se> Max Lapshin wrote: > Right now I fight with problem of streaming 10 Gbit/s of video and more > from single server. > > I have met different problems (mostly about locking into single process > like with file:read_file), > but right now I have one question, it is about memory allocation. > > I've started strace against one of threads running scheduler and saw that > it makes lot of mmap/munmap. > > For example during second 1376487042 it has called mmap 72 times with > sizes 1052672, 2101248. > > Such lines are very interesting for me: > > 1376487042.167246 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f49bf000 > 1376487042.174690 munmap(0x7fe90c6ff000, 1052672) = 0 > > > erlang asked system for one megabyte of memory and after 7 milliseconds > returned it back. > > To make it clear: erlyvideo is reading 700-800 megabytes per second from > several hard drives, repack them a bit and sends to network via TCP sockets. > > > > It is possible to tell erlang that it can use 20GB or 30 GB or ram and > don't return it back at all? > > I suppose that such often allocation/deallocation leads to high cpu usage > of process kswapd0 that (perhaps) tries to do something with such big > amount of freed pages. > > I have looked at erts_alloc.html page. Well, it is scary a bit, just like > when I've looked into the airplane cabin. Should I change any options from > this page or I shouldn't touch memory allocator at all? > > The erts_alloc docs can be a bit intimidating. The "mseg" allocator is the backend that makes the calls to mmap/munmap. It contains a cache of memory segments to reduce the number of mmap/munmap calls. Have you tried to increase the size of the mseg cache with: +MMmcs Max cached segments. The maximum number of memory segments stored in the memory segment cache. Valid range is 0-30. Default value is 10. /Sverker, Erlang/OTP From max.lapshin@REDACTED Wed Aug 14 16:41:26 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 14 Aug 2013 18:41:26 +0400 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: I'm running erlang R16B01 on Debian linux (kernel 3.2.0) Here is my binary allocator info: https://gist.github.com/maxlapshin/6231659 here is mseg_allocator: https://gist.github.com/maxlapshin/6231710 I've restarted erlang with +MMmcs 30: erts-5.10.2/bin/beam.smp -K true -A 160 -MMmcs 30 -stbt s -- ... Looks like number of mmap/munmap has lowered. Will try to understand this allocator information. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon@REDACTED Wed Aug 14 16:52:08 2013 From: jon@REDACTED (Jon Schneider) Date: Wed, 14 Aug 2013 15:52:08 +0100 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? Message-ID: You should also be aware that by default Linux is rather optimistic (over committing). It has various knobs under /proc . Jon > I'm running erlang R16B01 on Debian linux (kernel 3.2.0) > > > Here is my binary allocator info: > https://gist.github.com/maxlapshin/6231659 > > here is mseg_allocator: https://gist.github.com/maxlapshin/6231710 > > I've restarted erlang with +MMmcs 30: > > erts-5.10.2/bin/beam.smp -K true -A 160 -MMmcs 30 -stbt s -- ... > > Looks like number of mmap/munmap has lowered. > > Will try to understand this allocator information. From lukas@REDACTED Wed Aug 14 16:55:54 2013 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 14 Aug 2013 16:55:54 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: Looking quickly at the binary_alloc info you have a lot of the data allocated by sys_alloc. sys_alloc uses malloc directly, so there is no cache there. Maybe try setting +MBmmsbc 2048, I got 2048 by adding mseg_alloc_carriers to sys_alloc_carriers for sbcs and then adding a small buffer. Lukas On Wed, Aug 14, 2013 at 4:41 PM, Max Lapshin wrote: > I'm running erlang R16B01 on Debian linux (kernel 3.2.0) > > > Here is my binary allocator info: > https://gist.github.com/maxlapshin/6231659 > > here is mseg_allocator: https://gist.github.com/maxlapshin/6231710 > > I've restarted erlang with +MMmcs 30: > > erts-5.10.2/bin/beam.smp -K true -A 160 -MMmcs 30 -stbt s -- ... > > Looks like number of mmap/munmap has lowered. > > Will try to understand this allocator information. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Aug 14 16:57:07 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 14 Aug 2013 18:57:07 +0400 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: Oh, thank you, will try it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre@REDACTED Wed Aug 14 18:21:47 2013 From: andre@REDACTED (Andre Nathan) Date: Wed, 14 Aug 2013 13:21:47 -0300 Subject: [erlang-questions] mnesia:transform_table on a cluster Message-ID: <20130814162147.GF18385@andre.mz.digirati.com.br> Hello I have a mnesia cluster of four nodes, all using disc_copies, and I need to add fields to one of those tables. My idea is to first run mnesia:transform_table to change the schema, and then upgrade the code (this upgrade will be an installation of a new version of a package by the OS package manager, shutting off the service and then turning it on again. I don't have a need for a hot upgrade in this situation). Are schema changes replicated to other nodes in the cluster? That is, is it enough to run mnesia:transform_table on one of the nodes, or do I have to run it all nodes of the cluster? Thanks, Andre -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From anotherworldofworld@REDACTED Wed Aug 14 20:54:03 2013 From: anotherworldofworld@REDACTED (Alex toyer) Date: Thu, 15 Aug 2013 00:54:03 +0600 Subject: [erlang-questions] ANN: Ybot-0.3.5 chat bot released Message-ID: Hello, Today i released Ybot-0.3.5. Ybot it char bot written with erlang. Ybot supports now following transports: * IRC * XMPP * Campfire. * HTTP. * HipChat. * Flowdock. * Skype. * Talkerapp. And 2 write - only transports: * SMTP * Twitter ChangeLog: * Documentation for all transports. * Twitter now compatible with twitter api v.1.1 * Exception at the start of XMPP connection was fixed. * Web admin fully (frontend/backend) remade, all js libraries were removed besides Angular.js Ybot at github - https://github.com/0xAX/Ybot Plugins for Ybot - https://github.com/0xAX/ybot-contrib -- best regards, twitter: @0xAX github: 0xAX -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthonym@REDACTED Thu Aug 15 02:12:06 2013 From: anthonym@REDACTED (ANTHONY MOLINARO) Date: Wed, 14 Aug 2013 17:12:06 -0700 Subject: [erlang-questions] Reference counting in NIFs Message-ID: <59C8FD30-9DB6-4813-BDD0-B0033F24AAD5@alumni.caltech.edu> Hi, So I've been using re2 for some time now. I want to compile large regexes and use them from multiple processes. This however leads to a bottleneck because I need to have a gen_server which holds onto the NIF resource returned from the call to re2:compile/2 and pass it to calls to re2:match/3. The re2 engine runs in a few microseconds, but the gen_server message queue can take milliseconds and often under high concurrency it's message queue grows, and it takes even longer. The idea I had was to add an option to re2:compile/2 called {named_pattern, atom()} which when given an atom() keeps a copy of the re2 object on the C++ side (in a map<> at the moment), and then allows you to pass the atom() as the regex argument to re2:match/3. This works fine except in the case where you want to recompile the regex for the name (this happens every so often). Sometimes the recompile works and other times it seg faults. I'm pretty sure the segfaults occur during the switching, since a call like old_handle = named_patterns[copts.name]; named_patterns[copts.name] = new_handle; enif_release_resource (old_handle); occurring in one thread can conflict with another thread calling handle = named_patterns[copts.name]; ? // use re2 from within handle since the handle could be released (and I assume freed). So, I've been trying various things, but ideally, I'd just use the resource reference counting and the GC to make sure I don't leak memory. However, it's a little clear when the references are incremented and decremented from the example in the documentation. The documentation seems to suggest that calling enif_make_resource will add to the reference count, and that enif_release_resource will decrement from the reference count. Also it appears as though enif_keep_resource will also increment, but I'm not sure if enif_alloc_resource also increments (the documentation doesn't mention it). I tried a scheme were I call enif_keep_resource at the beginning of the re2:match/3 call, and enif_release_resource at the end to attempt to keep around the resource, but I still see some segfaults. So I guess my questions are. 1. which enif_* functions change the reference count, and how? 2. is there a better/safer way? I'd like to get this working as basho_bench shows me the ops/sec increasing from 12000 / second to 17000 / second and the mean time decreasing from 0.08 ms to 0.02 ms (the 99.9th increases however, so there must be more outlier's with my scheme for some reason). Sorry for the long email, -Anthony From leechau@REDACTED Thu Aug 15 02:14:19 2013 From: leechau@REDACTED (=?GBK?B?wO6zrA==?=) Date: Thu, 15 Aug 2013 08:14:19 +0800 (CST) Subject: [erlang-questions] How to build telnet client in Erlang? Message-ID: <777acbb.1619.1407f52ac58.Coremail.leechau@126.com> Hi, guys, I am new for Erlang and I wonder if there are any libraries providing APIs like: Handle = telnet_module:connect(Ip, Port, Username, Password). {ok, Msg} = telnet_module:send(Handle, Command). {ok, close} = telnet_module:close(Handle). Someone recommends ct_telnet, but this module seems mainly for test purpose (common test?) and associated closely with eunit. And I can't find any documents about how to configure the target server(including ip, port, username, password, etc.). Some examples show they are specified in a configuration file, but I am not quite sure. There is also a getty.erl (http://ftp.sunet.se/pub/lang/erlang/examples/small_examples/getty.erl), but I find it is actually a tcp server and has nothing to do with connecting to a telnet server. Maybe I can use "telnet" command of shell as the client and use open_port/port_command/port_close to communicate with remote peer? Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhuqling@REDACTED Thu Aug 15 07:45:39 2013 From: zhuqling@REDACTED (Jason Chuh) Date: Thu, 15 Aug 2013 13:45:39 +0800 Subject: [erlang-questions] How to build telnet client in Erlang? In-Reply-To: <777acbb.1619.1407f52ac58.Coremail.leechau@126.com> References: <777acbb.1619.1407f52ac58.Coremail.leechau@126.com> Message-ID: How about ct_telnet? http://www.erlang.org/doc/man/ct_telnet.html 2013/8/15 ?? > Hi, guys, > > I am new for Erlang and I wonder if there are any libraries providing APIs > like: > > Handle = telnet_module:connect(Ip, Port, Username, Password). > {ok, Msg} = telnet_module:send(Handle, Command). > {ok, close} = telnet_module:close(Handle). > > Someone recommends ct_telnet, but this module seems mainly for test > purpose (common test?) and associated closely with eunit. And I can't find > any documents about how to configure the target server(including ip, port, > username, password, etc.). Some examples show they are specified in a > configuration file, but I am not quite sure. > > < /div> > There is also a getty.erl ( > http://ftp.sunet.se/pub/lang/erlang/examples/small_examples/getty.erl), > but I find it is actually a tcp server and has nothing to do with > connecting to a telnet server. > > Maybe I can use "telnet" command of shell as the client and use > open_port/port_command/port_close to communicate with remote peer? > > Thanks a lot! > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Thu Aug 15 10:05:01 2013 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 15 Aug 2013 10:05:01 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: Hello, Did increasing the max mseg_alloc sbc limit help? After thinking a bit more on this issue, I think that my original hypothesis was correct. You are allocating way more sbc than you want to. Moving more of you allocations into mbcs should drastically decrease the number of mmaps needed and also instead of placing one binary within one mmap area, it will place many binaries within a larger mmaped area. Do you have any idea of the average sizes of the binaries that you are allocating? If you do I would try to set the sbct to be high enough that maybe 80% of all binaries fit below it. At the same time you should probably also adjust the smbcs and lmbcs to a higher value, maybe something like smbcs = 0.5*sbct lmbcs = 10*sbct. If you know that you will be using a lot of binaries you could also set smbcs to be higher to start with, or decrease the growth stages (mbcgs) so that you faster get to the lmbcs. You will have to play around with sbct, smbcs, lmbcs and mbcgs and measure performance and memory usage to see what is the best fit for you. When doing this you can remove the mmsbc config again as you will (hopefully) be using very few sbcs. Lukas On Wed, Aug 14, 2013 at 4:57 PM, Max Lapshin wrote: > Oh, thank you, will try it. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Aug 15 10:09:42 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 15 Aug 2013 12:09:42 +0400 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: I will give you more information in the evening, when traffic grows from 5 gbit/s per server to a least 7,5 gbit/s -------------- next part -------------- An HTML attachment was scrubbed... URL: From jozsef.berces@REDACTED Thu Aug 15 11:35:03 2013 From: jozsef.berces@REDACTED (=?iso-8859-1?Q?J=F3zsef_B=E9rces?=) Date: Thu, 15 Aug 2013 09:35:03 +0000 Subject: [erlang-questions] processes for old code Message-ID: <7460EBDDCF52084A849D0F271CE059B81A2CDE@ESESSMB101.ericsson.se> Is there a simpler and more efficient way than [X_Pid || X_Pid <- processes(), check_process_code(X_Pid, Mod)] to get the processes using the old code? Thanks, Jozsef -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Aug 15 11:53:03 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 15 Aug 2013 11:53:03 +0200 Subject: [erlang-questions] Reference counting in NIFs In-Reply-To: <59C8FD30-9DB6-4813-BDD0-B0033F24AAD5@alumni.caltech.edu> References: <59C8FD30-9DB6-4813-BDD0-B0033F24AAD5@alumni.caltech.edu> Message-ID: <520CA4FF.3020509@erix.ericsson.se> ANTHONY MOLINARO wrote: > Hi, > > So I've been using re2 for some time now. I want to compile large regexes and use them from multiple processes. This however leads to a bottleneck because I need to have a gen_server which holds onto the NIF resource returned from the call to re2:compile/2 and pass it to calls to re2:match/3. The re2 engine runs in a few microseconds, but the gen_server message queue can take milliseconds and often under high concurrency it's message queue grows, and it takes even longer. > > The idea I had was to add an option to re2:compile/2 called {named_pattern, atom()} which when given an atom() keeps a copy of the re2 object on the C++ side (in a map<> at the moment), and then allows you to pass the atom() as the regex argument to re2:match/3. This works fine except in the case where you want to recompile the regex for the name (this happens every so often). > > Sometimes the recompile works and other times it seg faults. I'm pretty sure the segfaults occur during the switching, since a call like > > old_handle = named_patterns[copts.name]; > named_patterns[copts.name] = new_handle; > enif_release_resource (old_handle); > > occurring in one thread can conflict with another thread calling > > handle = named_patterns[copts.name]; > ? > // use re2 from within handle > > since the handle could be released (and I assume freed). > > So, I've been trying various things, but ideally, I'd just use the resource reference counting and the GC to make sure I don't leak memory. However, it's a little clear when the references are incremented and decremented from the example in the documentation. > > The documentation seems to suggest that calling enif_make_resource will add to the reference count, and that enif_release_resource will decrement from the reference count. Also it appears as though enif_keep_resource will also increment, but I'm not sure if enif_alloc_resource also increments (the documentation doesn't mention it). > > I tried a scheme were I call enif_keep_resource at the beginning of the re2:match/3 call, and enif_release_resource at the end to attempt to keep around the resource, but I still see some segfaults. > > So I guess my questions are. > > 1. which enif_* functions change the reference count, and how? > alloc_resource: ref=1 release_resource: if (--ref == 0) free make_resource: ++ref keep_resource: ++ref keep_resource and release_resource can also be called by the VM when resource terms are copied or destroyed. > 2. is there a better/safer way? > Why don't use a public ETS table as a name lookup for your compiled regexps. If you have to access shared data in NIFs from several threads then I suggest using the enif_mutex_* interface to do make it thread safe. /Sverker, Erlang/OTP From maxim@REDACTED Thu Aug 15 11:47:20 2013 From: maxim@REDACTED (Maxim Sokhatsky) Date: Thu, 15 Aug 2013 12:47:20 +0300 Subject: [erlang-questions] [ANN] N2O: Erlang Web Framework on WebSockets In-Reply-To: References: , Message-ID: Greetings to All! We are happy to announce that experimental fork of N2O (Nitrogen 2x Optimized) is quite stable to be announced. Let us introduce the reason of refreshing Nitrogen. During our Erlang activities of creating a couple of complex Erlang Web projects we discovered that Nitrogen is extremely useful and productive not only among Erlang frameworks, but even comparable to known modern Web frameworks: https://gist.github.com/5HT/5522302However, we have discovered the Nitrogen limitations on speed, process registration gaps and technology lock with XHR. So we decided to fork and broke the compatibility with Nitrogen saving the general DSL approach and event system. So if your want to try Nitrogen you will have all the benefits of it with N2O too. Let us state new core features: -- WebSockets from the ground up with fallback to XHR using Bullet-- REST modules handlers using code from FarWest-- Control Events for custom JavaScript controls-- New Editor Elements like Medium.js and TinyMCE-- GProc "comet" registrations and Pub/Sub-- Proper ID instead of class attributes-- Erlang on Xen compatible with instructions of how to install on Xen-- DTL templates instead of Nitrogen slow templates-- Compatible enough to port Nitrogen sites-- Bootstrap and LESS styles boilerplate Some legacy code is completely removed and simplified for speed: -- simple_bridge removed-- nprocreg removed-- rendering simplified-- framework code reduced to 3000 LOC. Since one of our goals was to create the fastest web framework in all 7 levels (routing, query, session, templates, dsl, html rendering and javascript rendering) we tuned it up, reaching 15K Req/s (on the same machine raw cowboy yields 30K and raw nginx 60K). We don't send any JS on a first page load, and all JS comes on WebSockets afterwards, so the page is possible to render in one RTT -- in that way we reduced latency. Big news for MVC fans. From now on you can mix DSL and MVC separation with linking DSL controls by an ID that will produce only JS code and will not produce HTML. HTML could be served statically and variables could be picked up from cookies in JS code (this is experimental at the moment). But that is not all. We?ve also created a KVS data model framework where you can store all the data with commonly known KV stores: Riak and Mnesia. The data model includes Users, Group, Subscriptions, Likes, Products, Accounts, Transactions, Payments, Purchases, Feeds, ACL, Entries, Comments. You can store those entities in Mnesia and Riak in an abstract manner. More KV databases are to come along with SQL backends. Here are come sample sites built with N2O: http://igratch.synrc.com/http://skyline.synrc.com/ Sources: https://github.com/5HT/n2oDocumentation: http://synrc.com/framework/web/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim@REDACTED Thu Aug 15 12:35:53 2013 From: maxim@REDACTED (Maxim Sokhatsky) Date: Thu, 15 Aug 2013 13:35:53 +0300 Subject: [erlang-questions] [ANN] N2O Announcement In-Reply-To: References: Message-ID: Greetings to All! We are happy to announce that experimental fork of N2O is quite stable to be announced. Let us introduce the reason of refreshing Nitrogen. During our Erlang activities of creating a couple of complex Erlang Web projects we discovered that Nitrogen is extremely useful and productive not only among Erlang frameworks, but even comparable to known modern Web frameworks:?https://gist.github.com/5HT/5522302 However, we have discovered the Nitrogen limitations on speed, process registration gaps and technology lock with XHR. So we decided to fork and broke the compatibility with Nitrogen saving the general DSL approach and event system. So if your want to try Nitrogen you will have all the benefits of it with N2O too. Let us state new core features: * WebSockets from the ground up with fallback to XHR using Bullet * REST modules handlers using code from FarWest * Control Events for custom JavaScript controls * New Editor Elements like Medium.js and TinyMCE * GProc "comet" registrations and Pub/Sub * Proper ID instead of class attributes * Erlang on Xen compatible with instructions of how to install on Xen * DTL templates instead of Nitrogen slow templates * Compatible enough to port Nitrogen sites * Bootstrap and LESS styles boilerplate Some legacy code is completely removed and simplified for speed: * simple_bridge removed *?nprocreg removed * rendering simplified *?framework code reduced to 3000 LOC. Since one of our goals was to create the fastest web framework in all 7 levels (routing, query, session, templates, dsl, html rendering and javascript rendering) we tuned it up, reaching 15K Req/s (on the same machine raw cowboy yields 30K and raw nginx 60K). We don't send any JS on a first page load, and all JS comes on WebSockets afterwards, so the page is possible to render in one RTT -- in that way we reduced latency. Big news for MVC fans. From now on you can mix DSL and MVC separation with linking DSL controls by an ID that will produce only JS code and will not produce HTML. HTML could be served statically and variables could be picked up from cookies in JS code (this is experimental at the moment). But that is not all. We?ve also created a KVS data model framework where you can store all the data with commonly known KV stores: Riak and Mnesia. The data model includes Users, Group, Subscriptions, Likes, Products, Accounts, Transactions, Payments, Purchases, Feeds, ACL, Entries, Comments. You can store those entities in Mnesia and Riak in an abstract manner. More KV databases are to come along with SQL backends. Here are come sample sites built with N2O: http://igratch.synrc.com/ http://skyline.synrc.com/ Sources:?https://github.com/5HT/n2o Documentation:?http://synrc.com/framework/web/ From pierrefenoll@REDACTED Thu Aug 15 13:36:46 2013 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Thu, 15 Aug 2013 13:36:46 +0200 Subject: [erlang-questions] Generating exprs on the fly In-Reply-To: References: Message-ID: Thanks! That also helped, regarding erl_eval: http://erlang.2086793.n4.nabble.com/erl-eval-function-object-td2115199.html Cheers On 14 August 2013 15:39, Anthony Ramine wrote: > Hello, > > The answer is no, but you can always use erl_eval. > > Regards, > > -- > Anthony Ramine > > Le 14 ao?t 2013 ? 15:27, Pierre Fenoll a ?crit : > > > Is it not possible to build and load small pieces of code, such as fun > expressions or a list of atoms, without the constraint of being inside a > thus generated module/exported function? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Aug 15 16:19:42 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Thu, 15 Aug 2013 16:19:42 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool Message-ID: <520CE37E.4090909@ninenines.eu> Hello friendly people, I would like to make an official announcement of erlang.mk now that all the features I wanted are in. erlang.mk is a rebar replacement. It was initially created for allowing a faster development process than rebar and for better compatibility with Linux build tools. It should work on Linux and OSX with GNU Make installed. Projects using erlang.mk are still compatible with rebar. Dependencies fetched by rebar are stored in the same deps/ directory, and projects using erlang.mk can still be used as rebar dependencies, with or without a rebar.config file. erlang.mk also features a simple package index. Try `make pkg-list` to list all packages currently available. All the packages listed are compatible with erlang.mk with no tweaking required. Makefiles written with erlang.mk are *VERY* simple, here are two examples: * https://github.com/extend/farwest/blob/master/Makefile * https://github.com/extend/cowboy/blob/master/Makefile I wrote about erlang.mk and relx recently on the Nine Nines blog. erlang.mk is the perfect companion to relx. * http://ninenines.eu/articles/erlang.mk-and-relx Here are examples of projects that are using and compatible with erlang.mk: * https://github.com/jlouis/etorrent * https://github.com/extend/cowboy * https://github.com/extend/farwest You can find erlang.mk at the following URL: * https://github.com/extend/erlang.mk Contributions to the package index are of course welcome! The only requirement is that the package is to be compatible with erlang.mk itself. Just send a PR to the erlang.mk project updating the packages.v1.txt! Enjoy! -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From desired.mta@REDACTED Thu Aug 15 17:07:42 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Thu, 15 Aug 2013 17:07:42 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: <520CE37E.4090909@ninenines.eu> References: <520CE37E.4090909@ninenines.eu> Message-ID: On Thu, Aug 15, 2013 at 4:19 PM, Lo?c Hoguin wrote: > Hello friendly people, > > I would like to make an official announcement of erlang.mk now that all the > features I wanted are in. Please include an ability to cleanly override any target so make does not emit warnings about overwritten target. There are a few ways to do it, but I find this the easiest: prefix every target in erlang.mk with a variable: $(.erlang-mk-all)all: That way, if $(.erlang-mk-all) is defined (from my application Makefile before including erlang.mk), your 'all' target will be the one that does not cause me warnings. Motiejus Jak?tys From anthonym@REDACTED Thu Aug 15 17:59:26 2013 From: anthonym@REDACTED (ANTHONY MOLINARO) Date: Thu, 15 Aug 2013 08:59:26 -0700 Subject: [erlang-questions] Reference counting in NIFs In-Reply-To: <520CA4FF.3020509@erix.ericsson.se> References: <59C8FD30-9DB6-4813-BDD0-B0033F24AAD5@alumni.caltech.edu> <520CA4FF.3020509@erix.ericsson.se> Message-ID: On Aug 15, 2013, at 2:53 AM, Sverker Eriksson wrote: >> >> 1. which enif_* functions change the reference count, and how? >> > alloc_resource: ref=1 > release_resource: if (--ref == 0) free > make_resource: ++ref > keep_resource: ++ref > > keep_resource and release_resource can also be called by the VM when resource terms are copied or destroyed. Thanks for this information >> 2. is there a better/safer way? >> > Why don't use a public ETS table as a name lookup for your compiled regexps. > > If you have to access shared data in NIFs from several threads then I suggest using the enif_mutex_* interface to do make it thread safe. Well, because I had tried this and it failed to work, but I realized that I actually made a mistake in the way I tried and it does work. So I'll benchmark this and see if it works. Thanks for the answers, -Anthony From essen@REDACTED Thu Aug 15 18:32:04 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 15 Aug 2013 18:32:04 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: References: <520CE37E.4090909@ninenines.eu> Message-ID: <520D0284.6070408@ninenines.eu> On 08/15/2013 05:07 PM, Motiejus Jak?tys wrote: > On Thu, Aug 15, 2013 at 4:19 PM, Lo?c Hoguin wrote: >> Hello friendly people, >> >> I would like to make an official announcement of erlang.mk now that all the >> features I wanted are in. > > Please include an ability to cleanly override any target so make does > not emit warnings about overwritten target. There are a few ways to do > it, but I find this the easiest: prefix every target in erlang.mk with > a variable: > > $(.erlang-mk-all)all: > > > That way, if $(.erlang-mk-all) is defined (from my application > Makefile before including erlang.mk), your 'all' target will be the > one that does not cause me warnings. Yes I was thinking about this, but so far nobody needed it. If you do, though, patches welcome! -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From co7eb@REDACTED Thu Aug 15 21:45:50 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Thu, 15 Aug 2013 15:45:50 -0400 Subject: [erlang-questions] Why I cannot start sasl from my app file application specification Message-ID: <000001ce99f0$1193e930$34bbbb90$@frcuba.co.cu> Hi all, I have this application specification for an application. {application, x, [{description, "x description"}, {vsn, "1.0"}, {modules, [x, x_sup, x_log]}, {registered, [x, x_sup, x_log]}, {applications, [kernel, stdlib, sasl]}, {mod, {x, []}}, {start_phases, []}]}. Then when I try to start the system I get this error. application:start(x). {error,{not_started,sasl} I also have this config file which I load with werl -config x.config [{sasl, [ {sasl_error_logger, {file, "debug2.log"}}, {errlog_type, error}, %% define the parameters of the rotating log %% the log file directory {error_logger_mf_dir, "d:/WORK/x-1.0/error_logs/"}, %% # bytes per logfile {error_logger_mf_maxbytes, 10485760}, % 10 MB %% maximum number of logfiles {error_logger_mf_maxfiles, 10}]}, {kernel, [ {error_logger, {file, "debug.log" }}]} ]. If I start the sasl whether by hand in the console like this application:start(sasl) or by the command line werl -boot start_sasl all work fine. But not from the normal application specification which should be started from there according to the documentation. Best regards, Ivan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Thu Aug 15 22:23:18 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 15 Aug 2013 16:23:18 -0400 Subject: [erlang-questions] Why I cannot start sasl from my app file application specification In-Reply-To: <000001ce99f0$1193e930$34bbbb90$@frcuba.co.cu> References: <000001ce99f0$1193e930$34bbbb90$@frcuba.co.cu> Message-ID: <20130815202317.GA83287@ferdair.local> That's because unless you do -boot sasl, start SASL manually (with application:start/2) or have it started as part of a release, the SASL application is not actually started, and all the config file does is define variables to be loaded once it's started. If everything went fine with one of the patches I sent the OTP team, R16B02 should see the addition of application:ensure_all_started/1-2 functions that will boot both the application and all of its dependencies, recursively. This will make your use case simpler. Regards, Fred. On 08/15, Ivan Carmenates Garc?a wrote: > Hi all, > > > > I have this application specification for an application. > > > > {application, x, > > [{description, "x description"}, > > {vsn, "1.0"}, > > {modules, [x, x_sup, x_log]}, > > {registered, [x, x_sup, x_log]}, > > {applications, [kernel, stdlib, sasl]}, > > {mod, {x, []}}, > > {start_phases, []}]}. > > > > Then when I try to start the system I get this error. > > > > application:start(x). > > {error,{not_started,sasl} > > > > I also have this config file which I load with werl -config x.config > > > > > > [{sasl, [ > > {sasl_error_logger, {file, "debug2.log"}}, > > {errlog_type, error}, > > %% define the parameters of the rotating log > > %% the log file directory > > {error_logger_mf_dir, "d:/WORK/x-1.0/error_logs/"}, > > %% # bytes per logfile > > {error_logger_mf_maxbytes, 10485760}, % 10 MB > > %% maximum number of logfiles > > {error_logger_mf_maxfiles, 10}]}, > > {kernel, [ > > {error_logger, {file, "debug.log" }}]} > > ]. > > > > If I start the sasl whether by hand in the console like this > application:start(sasl) or by the command line werl -boot start_sasl all > work fine. > > But not from the normal application specification which should be started > from there according to the documentation. > > > > Best regards, > > Ivan. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From co7eb@REDACTED Fri Aug 16 04:27:14 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Thu, 15 Aug 2013 22:27:14 -0400 Subject: [erlang-questions] Why I cannot start sasl from my app file application specification Message-ID: <000501ce9a28$23d90e80$6b8b2b80$@frcuba.co.cu> Hi Fred, Thanks, the best solution I came with for now is to start it manually from start/2 callback function, because I have the Erlang R16B and I don't have that new function implemented. I think that's nice to have another function like ensure_all_started/1-2 in the application module, that makes the entire application to be started as one or fail as one if the applications specified in the app file couldn?t be all started property. But I still think even when the normal application:start/1-2 function don?t required all the child applications to be started, still that, I think it should do start the sasl application, because I think there is no apparent reason why it should not to. Another last thing, and I am sorry to bother you, there is a way to turn off the intromission of the error_logger in the sasl reports?, because I have repeated the same error when happens, one in the sasl like form: CRASH REPORT <0.35.0> 2013-08-15 21:43:07 ============================================================================ === Crashing process initial_call {application_master,init, ['Argument__1','Argument__2','Argument__3', 'Argument__4']} pid <0.35.0> registered_name [] error_info {exit, {bad_return, {{countlink,start,[normal,[{debug_level,all_solid}]]}, {'EXIT', {undef, ... ... and another with the error_logger form: INFO REPORT <0.7.0> 2013-08-15 21:43:07 ============================================================================ === application countlink exited {bad_return, {{countlink,start,[normal,[{debug_level,all_solid}]]}, {'EXIT', {undef, Both reports of the same error but different error_loggers in the same sasl report. I also tried to silent the error_logger with: {kernel, [{error_logger, silent}]} but still... Regards, Ivan. -----Mensaje original----- De: Fred Hebert [mailto:mononcqc@REDACTED] Enviado el: jueves, 15 de agosto de 2013 16:23 Para: Ivan Carmenates Garc?a CC: erlang-questions@REDACTED Asunto: Re: [erlang-questions] Why I cannot start sasl from my app file application specification That's because unless you do -boot sasl, start SASL manually (with application:start/2) or have it started as part of a release, the SASL application is not actually started, and all the config file does is define variables to be loaded once it's started. If everything went fine with one of the patches I sent the OTP team, R16B02 should see the addition of application:ensure_all_started/1-2 functions that will boot both the application and all of its dependencies, recursively. This will make your use case simpler. Regards, Fred. On 08/15, Ivan Carmenates Garc?a wrote: > Hi all, > > > > I have this application specification for an application. > > > > {application, x, > > [{description, "x description"}, > > {vsn, "1.0"}, > > {modules, [x, x_sup, x_log]}, > > {registered, [x, x_sup, x_log]}, > > {applications, [kernel, stdlib, sasl]}, > > {mod, {x, []}}, > > {start_phases, []}]}. > > > > Then when I try to start the system I get this error. > > > > application:start(x). > > {error,{not_started,sasl} > > > > I also have this config file which I load with werl -config x.config > > > > > > [{sasl, [ > > {sasl_error_logger, {file, "debug2.log"}}, > > {errlog_type, error}, > > %% define the parameters of the rotating log > > %% the log file directory > > {error_logger_mf_dir, "d:/WORK/x-1.0/error_logs/"}, > > %% # bytes per logfile > > {error_logger_mf_maxbytes, 10485760}, % 10 MB > > %% maximum number of logfiles > > {error_logger_mf_maxfiles, 10}]}, > > {kernel, [ > > {error_logger, {file, "debug.log" }}]} > > ]. > > > > If I start the sasl whether by hand in the console like this > application:start(sasl) or by the command line werl -boot start_sasl > all work fine. > > But not from the normal application specification which should be > started from there according to the documentation. > > > > Best regards, > > Ivan. > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Fri Aug 16 10:01:01 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 16 Aug 2013 10:01:01 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: <520CE37E.4090909@ninenines.eu> References: <520CE37E.4090909@ninenines.eu> Message-ID: On Thu, Aug 15, 2013 at 4:19 PM, Lo?c Hoguin wrote: > I would like to make an official announcement of erlang.mk now that all > the features I wanted are in. > I have been using erlang.mk for a while now, and recently I converted etorrent to use it as a test of the viability in larger projects. Typical gotchas: * Projects has no Makefile. erlang.mk needs one. So add one! * No `modules` section in the .app file. erlang.mk needs one to replace it. Not adding this makes relx behave badly. * If you use relx, it is more strict in what it accepts. * Relx can't yet overlay sys.config :/ Apart from that, erlang.mk is a bliss to work with. In one project I am working with: Core i5 2.4Ghz approx 2010 Macbook Pro, encrypted disk (this hurts performance like mad): Cold build: Rebar: 40 secs elrang.mk: 42 secs Build where each file is compiled in advance: Rebar: 20 secs erlang.mk: 0.4 secs For my development cycle, this is important enough to spend time rewriting projects to use erlang.mk. Also note that rebar.config and erlang.mk can co-exist, so you don't need to abandon rebar for erlang.mk, which is important. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Fri Aug 16 10:39:27 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Fri, 16 Aug 2013 10:39:27 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: <520CE37E.4090909@ninenines.eu> References: <520CE37E.4090909@ninenines.eu> Message-ID: The big problem with erlang.mk is requiring to have gmake and more importantly wget installed imo. Which makes it quite annoying to distribute on systems that have none of them. It would be interrestin to have the support for curl for example. Also what are the makefile extensions that you really need to require gmake? - benoit On Thu, Aug 15, 2013 at 4:19 PM, Lo?c Hoguin wrote: > Hello friendly people, > > I would like to make an official announcement of erlang.mk now that all > the features I wanted are in. > > erlang.mk is a rebar replacement. It was initially created for allowing a > faster development process than rebar and for better compatibility with > Linux build tools. It should work on Linux and OSX with GNU Make installed. > > Projects using erlang.mk are still compatible with rebar. Dependencies > fetched by rebar are stored in the same deps/ directory, and projects using > erlang.mk can still be used as rebar dependencies, with or without a > rebar.config file. > > erlang.mk also features a simple package index. Try `make pkg-list` to > list all packages currently available. All the packages listed are > compatible with erlang.mk with no tweaking required. > > Makefiles written with erlang.mk are *VERY* simple, here are two examples: > > * https://github.com/extend/**farwest/blob/master/Makefile > * https://github.com/extend/**cowboy/blob/master/Makefile > > I wrote about erlang.mk and relx recently on the Nine Nines blog. > erlang.mk is the perfect companion to relx. > > * http://ninenines.eu/articles/**erlang.mk-and-relx > > Here are examples of projects that are using and compatible with erlang.mk > : > > * https://github.com/jlouis/**etorrent > * https://github.com/extend/**cowboy > * https://github.com/extend/**farwest > > You can find erlang.mk at the following URL: > > * https://github.com/extend/**erlang.mk > > Contributions to the package index are of course welcome! The only > requirement is that the package is to be compatible with erlang.mkitself. Just send a PR to the > erlang.mk project updating the packages.v1.txt! > > Enjoy! > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ttom.kelly@REDACTED Fri Aug 16 11:27:23 2013 From: ttom.kelly@REDACTED (tom kelly) Date: Fri, 16 Aug 2013 10:27:23 +0100 Subject: [erlang-questions] Memory not freed after VM closed In-Reply-To: <039996476abc4a759861ea88dba39a05@DBXPR03MB128.eurprd03.prod.outlook.com> References: <039996476abc4a759861ea88dba39a05@DBXPR03MB128.eurprd03.prod.outlook.com> Message-ID: Hi list, We're confused here by a memory usage problem that we're seeing. We're writing a lot of data chunks to disk and are seeing higher than expected memory usage, that lingers even after the Erlang VM has stopped. I've trimmed it down to a simple minimal testcase, just open an erlang shell, open a dets file and write some data chunks to the dets file. In my test I write 45000 4KB chunks for 180MB of data (I get the data from crypto:rand_bytes/1 but the source doesn't affect the behaviour) and afterwards erlang:memory/0 shows that we're using ~12MB of RAM that agrees with what both "top" & "ps aux" accounts for in the beam process. What's weird is that top also reports (in the total memory used figure) 300MB of RAM has been used up while filling the dets table but does not account for it under any process (the amount of extra RAM consumed always seems to be between 1.5 & 2 times the amount of data written to disk). I can delete all the data from the dets file and close it with no effect on the inconsistent top output. I can then shutdown the Erlang VM, with no effect on the inconsistent top output. While monitoring top I can see that the RAM is only freed after I delete the dets/bitcask file/s from disk, the total RAM accounted for by "ps aux" does not change with this step. Or, if I re-run the test it gets freed up when the dets table is re-opened. I'm seeing this behavior here with R16B: Erlang R16B (erts-5.10.1) [source] [64-bit] [async-threads:10] [hipe] [kernel-poll:false] And my OS: uname -a Linux host 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux But I also see it on R14B02: Erlang R14B02 (erts-5.8.3) [source] [64-bit] [rq:1] [async-threads:0] [hipe] [kernel-poll:false] Linux host 2.6.32-5-amd64 #1 SMP Mon Jan 16 16:22:28 UTC 2012 x86_64 GNU/Linux It seems like way too big an issue not to have been noticed before now, can anyone else reproduce it? We're easily confused here at the best of times ;-) so if anyone can shed any light on what's holding onto this RAM it'd be much appreciated! Ideally we'd like to get this 300MB freed up straight away after writing everything to disk. //TTom. From pan@REDACTED Fri Aug 16 11:35:58 2013 From: pan@REDACTED (Patrik Nyblom) Date: Fri, 16 Aug 2013 11:35:58 +0200 Subject: [erlang-questions] Memory not freed after VM closed In-Reply-To: References: <039996476abc4a759861ea88dba39a05@DBXPR03MB128.eurprd03.prod.outlook.com> Message-ID: <520DF27E.80808@erlang.org> Hi! My guess is that this has nothing to do with Erlang, it's just disk cache. You'll get the memory back when you need it. See for example: http://www.linuxatemyram.com/ for more details. Cheers, Patrik On 08/16/2013 11:27 AM, tom kelly wrote: > Hi list, > > We're confused here by a memory usage problem that we're seeing. We're > writing a lot of data chunks to disk and are seeing higher than > expected memory usage, that lingers even after the Erlang VM has > stopped. > > I've trimmed it down to a simple minimal testcase, just open an erlang > shell, open a dets file and write some data chunks to the dets file. > In my test I write 45000 4KB chunks for 180MB of data (I get the data > from crypto:rand_bytes/1 but the source doesn't affect the behaviour) > and afterwards erlang:memory/0 shows that we're using ~12MB of RAM > that agrees with what both "top" & "ps aux" accounts for in the beam > process. > > What's weird is that top also reports (in the total memory used > figure) 300MB of RAM has been used up while filling the dets table but > does not account for it under any process (the amount of extra RAM > consumed always seems to be between 1.5 & 2 times the amount of data > written to disk). > > I can delete all the data from the dets file and close it with no > effect on the inconsistent top output. I can then shutdown the Erlang > VM, with no effect on the inconsistent top output. > > While monitoring top I can see that the RAM is only freed after I > delete the dets/bitcask file/s from disk, the total RAM accounted for > by "ps aux" does not change with this step. Or, if I re-run the test > it gets freed up when the dets table is re-opened. > > I'm seeing this behavior here with R16B: > Erlang R16B (erts-5.10.1) [source] [64-bit] [async-threads:10] [hipe] > [kernel-poll:false] > > And my OS: > uname -a > Linux host 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux > > But I also see it on R14B02: > Erlang R14B02 (erts-5.8.3) [source] [64-bit] [rq:1] [async-threads:0] > [hipe] [kernel-poll:false] > Linux host 2.6.32-5-amd64 #1 SMP Mon Jan 16 16:22:28 UTC 2012 x86_64 GNU/Linux > > It seems like way too big an issue not to have been noticed before > now, can anyone else reproduce it? > > We're easily confused here at the best of times ;-) so if anyone can > shed any light on what's holding onto this RAM it'd be much > appreciated! Ideally we'd like to get this 300MB freed up straight > away after writing everything to disk. > > //TTom. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From pan@REDACTED Fri Aug 16 12:07:43 2013 From: pan@REDACTED (Patrik Nyblom) Date: Fri, 16 Aug 2013 12:07:43 +0200 Subject: [erlang-questions] New version of PCRE in master - please, please test! Message-ID: <520DF9EF.8080504@erlang.org> Hello all regexp hackers! I've bumped the PCRE version in our master branch, so that we now use the latest and greatest. This includes support for Unicode character properties, among other things. However, this also means potential incompatibilities that's hard to foresee, there might be subtle differences in the regular expression evaluation, after all they have fixed bugs that we never knew was there... It would be nice if heavy (or inspired) regular expression users would try this out - i.e. check out and build from master (aka R17A) and see if their systems still work. It would also bee nice if any trouble was reported within the next two weeks, as I will quit OTP after that and I'm quite well acquainted with the code right now, so we would gain time If I knew about problems asap... The branch I merged also contains fixes/extensions for other issues with re discussed previously on the lists, like: OTP-11205: Correctly handle dupnames in re OTP-10285: re does not handle PCRE_ERROR_LOOP_LIMIT correctly And of course the issues of re not handling Unicode character properties (that comes with the new PCRE version, but there is also a new option to re:run()) The documentation is of course also updated. So please try it out and report any strange behavior on erlang-bugs (if you know they are bugs) or on erlang-questions (if you're unsure). There's no time like the present...:) Cheers, Patrik From desired.mta@REDACTED Fri Aug 16 12:43:13 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Fri, 16 Aug 2013 12:43:13 +0200 Subject: [erlang-questions] Memory not freed after VM closed In-Reply-To: References: <039996476abc4a759861ea88dba39a05@DBXPR03MB128.eurprd03.prod.outlook.com> Message-ID: On Fri, Aug 16, 2013 at 11:27 AM, tom kelly wrote: > Hi list, > > We're confused here by a memory usage problem that we're seeing. We're > writing a lot of data chunks to disk and are seeing higher than > expected memory usage, that lingers even after the Erlang VM has > stopped. > > We're easily confused here at the best of times ;-) so if anyone can > shed any light on what's holding onto this RAM it'd be much > appreciated! Ideally we'd like to get this 300MB freed up straight > away after writing everything to disk. > Hi, Tom, like Patrick said, probably it is disk cache by linux kernel. Run this to verify: $ echo 1 | sudo tee /proc/sys/vm/drop_caches This command instructs kernel to release disk cache. Inspect your memory after running this command and confirm. -- Motiejus Jak?tys From ttom.kelly@REDACTED Fri Aug 16 13:20:07 2013 From: ttom.kelly@REDACTED (tom kelly) Date: Fri, 16 Aug 2013 12:20:07 +0100 Subject: [erlang-questions] Memory not freed after VM closed In-Reply-To: References: <039996476abc4a759861ea88dba39a05@DBXPR03MB128.eurprd03.prod.outlook.com> Message-ID: Hi Patrik, Motiejus, Thanks for the expertise, that does indeed explain all our observations. Writing that file does free up the RAM, and I love the tone of linuxatemyram.com ;-) Our application has to work in some tight HW specs so we have a resource checker that monitors our RAM level among other things. When RAM is low we take certain actions and send connections into a by-pass mode which is sub-optimal but better than crashing from memory starvation. This was being triggered too often on our deployment which brought it to our attention. Based on my new knowledge I can now program this better and use the correct figure for RAM usage to base our decisions on. Hopefully a customer will monitor top & I'll have an occasion to send them that link! Thanks for the answers. //TTom. On 8/16/13, Motiejus Jak?tys wrote: > On Fri, Aug 16, 2013 at 11:27 AM, tom kelly wrote: >> Hi list, >> >> We're confused here by a memory usage problem that we're seeing. We're >> writing a lot of data chunks to disk and are seeing higher than >> expected memory usage, that lingers even after the Erlang VM has >> stopped. >> >> We're easily confused here at the best of times ;-) so if anyone can >> shed any light on what's holding onto this RAM it'd be much >> appreciated! Ideally we'd like to get this 300MB freed up straight >> away after writing everything to disk. >> > > Hi, Tom, > > like Patrick said, probably it is disk cache by linux kernel. Run this > to verify: > > $ echo 1 | sudo tee /proc/sys/vm/drop_caches > > This command instructs kernel to release disk cache. Inspect your > memory after running this command and confirm. > > -- > Motiejus Jak?tys > From dronnikov@REDACTED Fri Aug 16 13:25:20 2013 From: dronnikov@REDACTED (Vladimir Dronnikov) Date: Fri, 16 Aug 2013 15:25:20 +0400 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: References: <520CE37E.4090909@ninenines.eu> Message-ID: I believe curl -L $(PKG_FILE_URL) >$(PKG_FILE) is kinda drop-in replacement for wget -O $(PKG_FILE) $(PKG_FILE_URL) used in erlang.mk. Should be tested On Fri, Aug 16, 2013 at 12:39 PM, Benoit Chesneau wrote: > The big problem with erlang.mk is requiring to have gmake and more > importantly wget installed imo. > > Which makes it quite annoying to distribute on systems that have none of > them. It would be interrestin to have the support for curl for example. > Also what are the makefile extensions that you really need to require gmake? > > - benoit > > > On Thu, Aug 15, 2013 at 4:19 PM, Lo?c Hoguin wrote: > >> Hello friendly people, >> >> I would like to make an official announcement of erlang.mk now that all >> the features I wanted are in. >> >> erlang.mk is a rebar replacement. It was initially created for allowing >> a faster development process than rebar and for better compatibility with >> Linux build tools. It should work on Linux and OSX with GNU Make installed. >> >> Projects using erlang.mk are still compatible with rebar. Dependencies >> fetched by rebar are stored in the same deps/ directory, and projects using >> erlang.mk can still be used as rebar dependencies, with or without a >> rebar.config file. >> >> erlang.mk also features a simple package index. Try `make pkg-list` to >> list all packages currently available. All the packages listed are >> compatible with erlang.mk with no tweaking required. >> >> Makefiles written with erlang.mk are *VERY* simple, here are two >> examples: >> >> * https://github.com/extend/**farwest/blob/master/Makefile >> * https://github.com/extend/**cowboy/blob/master/Makefile >> >> I wrote about erlang.mk and relx recently on the Nine Nines blog. >> erlang.mk is the perfect companion to relx. >> >> * http://ninenines.eu/articles/**erlang.mk-and-relx >> >> Here are examples of projects that are using and compatible with >> erlang.mk: >> >> * https://github.com/jlouis/**etorrent >> * https://github.com/extend/**cowboy >> * https://github.com/extend/**farwest >> >> You can find erlang.mk at the following URL: >> >> * https://github.com/extend/**erlang.mk >> >> Contributions to the package index are of course welcome! The only >> requirement is that the package is to be compatible with erlang.mkitself. Just send a PR to the >> erlang.mk project updating the packages.v1.txt! >> >> Enjoy! >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Fri Aug 16 14:27:48 2013 From: steve@REDACTED (Steve Strong) Date: Fri, 16 Aug 2013 14:27:48 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: <520CE37E.4090909@ninenines.eu> References: <520CE37E.4090909@ninenines.eu> Message-ID: <05C075301D714F86AD3DF71B5DE6A0C0@srstrong.com> Looks good - I like simple! Quick question, does it support multiple applications, for example a project laid out as: /proj /deps /stuff /apps /app1 /app2 Most of our stuff is in that form, with shared dependencies between the various apps. Rebar is quite happy with that format, but I can't see how to persuade erlang.mk to handle that. Cheers, Steve -- Steve Strong Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Thursday, 15 August 2013 at 16:19, Lo?c Hoguin wrote: > Hello friendly people, > > I would like to make an official announcement of erlang.mk now that all > the features I wanted are in. > > erlang.mk is a rebar replacement. It was initially created for allowing > a faster development process than rebar and for better compatibility > with Linux build tools. It should work on Linux and OSX with GNU Make > installed. > > Projects using erlang.mk are still compatible with rebar. Dependencies > fetched by rebar are stored in the same deps/ directory, and projects > using erlang.mk can still be used as rebar dependencies, with or without > a rebar.config file. > > erlang.mk also features a simple package index. Try `make pkg-list` to > list all packages currently available. All the packages listed are > compatible with erlang.mk with no tweaking required. > > Makefiles written with erlang.mk are *VERY* simple, here are two examples: > > * https://github.com/extend/farwest/blob/master/Makefile > * https://github.com/extend/cowboy/blob/master/Makefile > > I wrote about erlang.mk and relx recently on the Nine Nines blog. > erlang.mk is the perfect companion to relx. > > * http://ninenines.eu/articles/erlang.mk-and-relx > > Here are examples of projects that are using and compatible with erlang.mk: > > * https://github.com/jlouis/etorrent > * https://github.com/extend/cowboy > * https://github.com/extend/farwest > > You can find erlang.mk at the following URL: > > * https://github.com/extend/erlang.mk > > Contributions to the package index are of course welcome! The only > requirement is that the package is to be compatible with erlang.mk > itself. Just send a PR to the erlang.mk project updating the > packages.v1.txt! > > Enjoy! > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Aug 16 16:34:56 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 16 Aug 2013 16:34:56 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: References: <520CE37E.4090909@ninenines.eu> Message-ID: <520E3890.5020000@ninenines.eu> On 08/16/2013 10:39 AM, Benoit Chesneau wrote: > The big problem with erlang.mk is requiring to have > gmake and more importantly wget installed imo. wget is only used for fetching the package index file. I'm sure if it doesn't work somewhere it'll be patched eventually. > Which makes it quite annoying to distribute on systems that have none of > them. It would be interrestin to have the support for curl for example. > Also what are the makefile extensions that you really need to require gmake? No idea. Patches are welcome for compatibility with different OS/build tools (as long as it's not "rewrite the whole file" of course, then you're better off just using gmake). > - benoit > > > On Thu, Aug 15, 2013 at 4:19 PM, Lo?c Hoguin > wrote: > > Hello friendly people, > > I would like to make an official announcement of erlang.mk > now that all the features I wanted are in. > > erlang.mk is a rebar replacement. It was > initially created for allowing a faster development process than > rebar and for better compatibility with Linux build tools. It should > work on Linux and OSX with GNU Make installed. > > Projects using erlang.mk are still compatible > with rebar. Dependencies fetched by rebar are stored in the same > deps/ directory, and projects using erlang.mk can > still be used as rebar dependencies, with or without a rebar.config > file. > > erlang.mk also features a simple package index. > Try `make pkg-list` to list all packages currently available. All > the packages listed are compatible with erlang.mk > with no tweaking required. > > Makefiles written with erlang.mk are *VERY* > simple, here are two examples: > > * https://github.com/extend/__farwest/blob/master/Makefile > > * https://github.com/extend/__cowboy/blob/master/Makefile > > > I wrote about erlang.mk and relx recently on the > Nine Nines blog. erlang.mk is the perfect > companion to relx. > > * http://ninenines.eu/articles/__erlang.mk-and-relx > > > Here are examples of projects that are using and compatible with > erlang.mk : > > * https://github.com/jlouis/__etorrent > > * https://github.com/extend/__cowboy > > * https://github.com/extend/__farwest > > > You can find erlang.mk at the following URL: > > * https://github.com/extend/__erlang.mk > > > Contributions to the package index are of course welcome! The only > requirement is that the package is to be compatible with erlang.mk > itself. Just send a PR to the erlang.mk > project updating the packages.v1.txt! > > Enjoy! > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From essen@REDACTED Fri Aug 16 16:42:39 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 16 Aug 2013 16:42:39 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: <05C075301D714F86AD3DF71B5DE6A0C0@srstrong.com> References: <520CE37E.4090909@ninenines.eu> <05C075301D714F86AD3DF71B5DE6A0C0@srstrong.com> Message-ID: <520E3A5F.7000800@ninenines.eu> Well I'm sure if you create a base Makefile (without erlang.mk) that exports DEPS_DIR and then call $(MAKE) on all folders in /apps (which would themselves contain Makefiles that use erlang.mk), it would work just fine. You can still keep only one erlang.mk in your repos and use include ../../erlang.mk instead for example. But know that this folder structure is a rebar thing and not standard (just like /deps you'll say, but that one is insanely useful regardless of the project structure otherwise). On 08/16/2013 02:27 PM, Steve Strong wrote: > Looks good - I like simple! Quick question, does it support multiple > applications, for example a project laid out as: > > /proj > /deps > /stuff > > /apps > /app1 > /app2 > > Most of our stuff is in that form, with shared dependencies between the > various apps. Rebar is quite happy with that format, but I can't see > how to persuade erlang.mk to handle that. > > Cheers, > > Steve > > -- > Steve Strong > Sent with Sparrow > > On Thursday, 15 August 2013 at 16:19, Lo?c Hoguin wrote: > >> Hello friendly people, >> >> I would like to make an official announcement of erlang.mk now that all >> the features I wanted are in. >> >> erlang.mk is a rebar replacement. It was initially created for allowing >> a faster development process than rebar and for better compatibility >> with Linux build tools. It should work on Linux and OSX with GNU Make >> installed. >> >> Projects using erlang.mk are still compatible with rebar. Dependencies >> fetched by rebar are stored in the same deps/ directory, and projects >> using erlang.mk can still be used as rebar dependencies, with or without >> a rebar.config file. >> >> erlang.mk also features a simple package index. Try `make pkg-list` to >> list all packages currently available. All the packages listed are >> compatible with erlang.mk with no tweaking required. >> >> Makefiles written with erlang.mk are *VERY* simple, here are two examples: >> >> * https://github.com/extend/farwest/blob/master/Makefile >> * https://github.com/extend/cowboy/blob/master/Makefile >> >> I wrote about erlang.mk and relx recently on the Nine Nines blog. >> erlang.mk is the perfect companion to relx. >> >> * http://ninenines.eu/articles/erlang.mk-and-relx >> >> Here are examples of projects that are using and compatible with >> erlang.mk: >> >> * https://github.com/jlouis/etorrent >> * https://github.com/extend/cowboy >> * https://github.com/extend/farwest >> >> You can find erlang.mk at the following URL: >> >> * https://github.com/extend/erlang.mk >> >> Contributions to the package index are of course welcome! The only >> requirement is that the package is to be compatible with erlang.mk >> itself. Just send a PR to the erlang.mk project updating the >> packages.v1.txt! >> >> Enjoy! >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From steve@REDACTED Fri Aug 16 16:44:07 2013 From: steve@REDACTED (Steve Strong) Date: Fri, 16 Aug 2013 16:44:07 +0200 Subject: [erlang-questions] [ANN] erlang.mk build tool In-Reply-To: <520E3A5F.7000800@ninenines.eu> References: <520CE37E.4090909@ninenines.eu> <05C075301D714F86AD3DF71B5DE6A0C0@srstrong.com> <520E3A5F.7000800@ninenines.eu> Message-ID: <4BEA9D47600B462DAAA7D339E9AE47B5@srstrong.com> Was guessing that was the answer - I'll give it a go... -- Steve Strong Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Friday, 16 August 2013 at 16:42, Lo?c Hoguin wrote: > Well I'm sure if you create a base Makefile (without erlang.mk) that > exports DEPS_DIR and then call $(MAKE) on all folders in /apps (which > would themselves contain Makefiles that use erlang.mk), it would work > just fine. You can still keep only one erlang.mk in your repos and use > include ../../erlang.mk instead for example. > > But know that this folder structure is a rebar thing and not standard > (just like /deps you'll say, but that one is insanely useful regardless > of the project structure otherwise). > > On 08/16/2013 02:27 PM, Steve Strong wrote: > > Looks good - I like simple! Quick question, does it support multiple > > applications, for example a project laid out as: > > > > /proj > > /deps > > /stuff > > > > /apps > > /app1 > > /app2 > > > > Most of our stuff is in that form, with shared dependencies between the > > various apps. Rebar is quite happy with that format, but I can't see > > how to persuade erlang.mk to handle that. > > > > Cheers, > > > > Steve > > > > -- > > Steve Strong > > Sent with Sparrow > > > > On Thursday, 15 August 2013 at 16:19, Lo?c Hoguin wrote: > > > > > Hello friendly people, > > > > > > I would like to make an official announcement of erlang.mk now that all > > > the features I wanted are in. > > > > > > erlang.mk is a rebar replacement. It was initially created for allowing > > > a faster development process than rebar and for better compatibility > > > with Linux build tools. It should work on Linux and OSX with GNU Make > > > installed. > > > > > > Projects using erlang.mk are still compatible with rebar. Dependencies > > > fetched by rebar are stored in the same deps/ directory, and projects > > > using erlang.mk can still be used as rebar dependencies, with or without > > > a rebar.config file. > > > > > > erlang.mk also features a simple package index. Try `make pkg-list` to > > > list all packages currently available. All the packages listed are > > > compatible with erlang.mk with no tweaking required. > > > > > > Makefiles written with erlang.mk are *VERY* simple, here are two examples: > > > > > > * https://github.com/extend/farwest/blob/master/Makefile > > > * https://github.com/extend/cowboy/blob/master/Makefile > > > > > > I wrote about erlang.mk and relx recently on the Nine Nines blog. > > > erlang.mk is the perfect companion to relx. > > > > > > * http://ninenines.eu/articles/erlang.mk-and-relx > > > > > > Here are examples of projects that are using and compatible with > > > erlang.mk: > > > > > > * https://github.com/jlouis/etorrent > > > * https://github.com/extend/cowboy > > > * https://github.com/extend/farwest > > > > > > You can find erlang.mk at the following URL: > > > > > > * https://github.com/extend/erlang.mk > > > > > > Contributions to the package index are of course welcome! The only > > > requirement is that the package is to be compatible with erlang.mk > > > itself. Just send a PR to the erlang.mk project updating the > > > packages.v1.txt! > > > > > > Enjoy! > > > > > > -- > > > Lo?c Hoguin > > > Erlang Cowboy > > > Nine Nines > > > http://ninenines.eu > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From publicityifl@REDACTED Fri Aug 16 20:17:27 2013 From: publicityifl@REDACTED (publicityifl@REDACTED) Date: Fri, 16 Aug 2013 11:17:27 -0700 (PDT) Subject: [erlang-questions] Call for participation IFL 2013 Message-ID: <520e6cb7.4c560f0a.60bb.5d8d@mx.google.com> CALL FOR PARTICIPATION 25th SYMPOSIUM ON IMPLEMENTATION AND APPLICATION OF FUNCTIONAL LANGUAGES - IFL 2013 RADBOUD UNIVERSITY NIJMEGEN, THE NETHERLANDS ACM In-Cooperation / ACM SIGPLAN AUGUST 28 - 30 2013 "Landgoed Holthurnsche Hof" http://ifl2013.cs.ru.nl [program available - late registration still open] We are proud to announce that the 25th edition of the IFL series returns to its roots at the Radboud University Nijmegen in the Netherlands. The symposium is held from 28th to 30th of August 2013. Scope ----- The goal of the IFL symposia is to bring together researchers actively engaged in the implementation and application of functional and function-based programming languages. IFL 2013 will be a venue for researchers to present and discuss new ideas and concepts, work in progress, and publication-ripe results related to the implementation and application of functional languages and function-based programming. Following the IFL tradition, IFL 2013 will use a post-symposium review process to produce the formal proceedings which will be published in the ACM Digital Library. All participants of IFL 2013 are invited to submit either a draft paper or an extended abstract describing work to be presented at the symposium. At no time may work submitted to IFL be simultaneously submitted to other venues; submissions must adhere to ACM SIGPLAN's republication policy: http://www.sigplan.org/Resources/Policies/Republication The submissions will be screened by the program committee chair to make sure they are within the scope of IFL, and will appear in the draft proceedings distributed at the symposium. Submissions appearing in the draft proceedings are not peer-reviewed publications. Hence, publications that appear only in the draft proceedings do not count as publication for the ACM SIGPLAN republication policy. After the symposium, authors will be given the opportunity to incorporate the feedback from discussions at the symposium and will be invited to submit a revised full article for the formal review process. From the revised submissions, the program committee will select papers for the formal proceedings considering their correctness, novelty, originality, relevance, significance, and clarity. Invited Speaker --------------- Lennart Augustsson, currently employed by the Standard Chartered Bank, well-known for his work on Haskell, parallel Haskell, Cayenne, and Bluespec, is the invited speaker of IFL 2013. The title and abstract of his talk is: "Implementation and Application of Functional Languages - A personal perspective" It is now over 30 years ago since I implemented my first functional language, and over 15 years ago since I wrote my first commercial application. In this talk I will look back to those bygone days and remind you of things that you might have forgotten or never known. The talk will be absolutely free of anything new. Peter Landin Prize ------------------ The Peter Landin Prize is awarded to the best paper presented at the symposium every year. The honoured article is selected by the program committee based on the submissions received for the formal review process. The prize carries a cash award equivalent to 150 Euros. Programme committee ------------------- ? Thomas Arts, Quviq, Gothenburg, Sweden ? Andrew Butterfield, Trinity College, Dublin, Ireland ? Edwin Brady, University of St. Andrews, UK ? Clemens Grelck, University of Amsterdam, Netherlands ? Adam Granicz, IntelliFactory, Budapest, Hungary ? Jeremy Gibbons, University of Oxford, UK ? Fritz Henglein, University of Copenhagen, Denmark ? Stephan Herhut, Intel Labs, Santa Clara, US ? Ralf Hinze (co-chair), University of Oxford, UK ? Zolt?n Horv?th, E?tv?s Lor?nd University, Budapest, Hungary ? Zhenjiang Hu, University of Tokyo, Japan ? Mauro Jaskelioff, Universidad Nacional de Rosario, Argentina ? Johan Jeuring, University of Utrecht, Netherlands ? Rita Loogen, University of Marburg, Germany ? Marco T. Moraz?n, Seton Hall University, New Jersey, US ? Dominic Orchard, University of Cambridge, UK ? Rinus Plasmeijer (chair), Radboud University Nijmegen, Netherlands ? Tim Sheard, Portland State University, US ? Sam Tobin-Hochstadt, Northeastern University / Indiana University, US ? Peter Thiemann, University of Freiburg, Germany ? Simon Thompson, University of Kent, UK Venue ----- The 25th IFL is organized by the Radboud University Nijmegen, Model Based Software Development Department at the Nijmegen Institute for Computing and Information Sciences. The event is held in the Landgoed ?Holthurnsche Hof?, a rural estate in the woodlands surrounding Nijmegen. It can be reached quickly and easily by public transport. Program ------- Wednesday August 28 ------------------- 8:15 registration 8:50 opening 9:00 Marcos Viera First Class Syntax, Semantics, and Their Composition Doaitse Swierstra 9:25 Olivier Danvy Circularity and Lambda Abstraction Peter Thiemann Ian Zerny 9:50 break 10:20 Loic Denuziere Piglets to the rescue Ernesto Rodriguez Adam Granicz 10:45 Simon Fowler Correct and Secure Web Programming using Dependent Types Edwin Brady and Embedded Domain-Specific Languages 11:10 break 11:40 Jennifer Hackett The Under-Performing Unfold: A new approach to optimising Graham Hutton corecursive programs Mauro Jaskelioff 12:05 Laurence Edward Day Compilation ? la Carte Graham Hutton 12:30 Artjoms Sinkarovs Functionally Redundant Declarations for Improved Sven-Bodo Scholz Performance Portability 12:55 lunch 14:00 Xavier Clerc OCaml-Java: Typing Java Accesses from OCaml Programs 14:25 Jonathan Protzenko The implementation of the Mezzo type-checker 14:50 break 15:20 Ralf L?mmel The 101haskell chrestomathy Thomas Schmorleiz Andrei Varanovich 15:45 Chide Groenouwe Instant playful access to serious programming for non-programmers with a visual functional programming language 16:10 break 16:40 Arjan Boeijink Supercompiling Haskell to Hardware Philip H?lzenspies Christiaan Baaij Jan Kuper 17:05 Bas van Gijzel Towards a framework for the implementation and verification Henrik Nilson of translations between argumentation models 17:30 end of talks 18:00 dinner 20:00 25th IFL reunion 23:00 Thursday August 29 ------------------ 9:00 Bas Lijnse Supporting Semi-Structured Work using Higher Order Tasks Jan Martin Jansen 9:25 Vikt?ria Zs?k A Prototype of CPS Systems 9:50 break 10:20 Majed Al Saeed A Critical Analysis of Parallel Functional Profilers Patrick Maier Phil Trinder Lilia Georgieva 10:45 Vladimir Janjic Using Erlang Skeletons to Parallelise Realistic Christopher Brown Medium-Scale Parallel Programs Kevin Hammond 11:10 break 11:40 Lennart Augustsson Invited talk: Implementation and Application of Functional Languages A personal perspective 12:40 lunch 13:45 - 23:00 Social event: excursion, symposium dinner, Peter Landin award Friday August 30 ---------------- 9:00 Malak Aljabri The Design and Implementation of GUMSMP: Hans-Wolfgang Loidl a Multilevel Parallel Haskell Implementation Phil Trinder 9:25 Henrique Ferreiro Kindergarten Cop: Laura Castro Dynamic Nursery Resizing for Increasing Parallelism in GHC Vladimir Janjic David Castro Kevin Hammond 9:50 break 10:20 Fangyong Tang User-Defined Shape Constraints in SAC Clemens Grelck 10:45 Leaf Petersen Measuring the Haskell Gap Todd Anderson Hai Liu Neal Glew 11:10 break 11:40 Marco T. Moraz?n Immediate Dominators in Linear Time 12:05 Merijn Verstraaten On Predicting the Impact of Resource Redistribution Sven-Bodo Scholz in Streaming Applications 12:30 IFL 2014 12:40 lunch 14:00 Melinda T?th Reduction of regression tests for Erlang based on impact analysis Istv?n Boz? Zolt?n Horv?th 14:25 Mac?as L?pez A DSL for Web Services Automatic Test Data Generation Henrique Ferreiro Laura M. Castro Thomas Arts 14:50 break 15:20 Kanae Tsushima A Weighted Type Error Slicer Kenichi Asai 15:45 Ben Thorner A Type Inference Debugger for ML in Education Kathryn Gray 16:10 break 16:40 Nicolas Wu Structured Sharing for Dynamic Programming 17:05 Clemens Grelck Towards Persistent and Parallel Asynchronous Adaptive Specialisation Heinz Wiesinger for Data-Parallel Array Processing in SAC 17:30 closing From seriy.pr@REDACTED Fri Aug 16 20:45:52 2013 From: seriy.pr@REDACTED (=?KOI8-R?B?88XSx8XKIPDSz8jP0s/X?=) Date: Fri, 16 Aug 2013 22:45:52 +0400 Subject: [erlang-questions] xmerl/xpath alternatives? Message-ID: A little outdated, but anyway, you can use HTML/XML parser from Mochiweb https://github.com/mochi/mochiweb/blob/master/src/mochiweb_html.erl and XPath implementation for it's tree https://github.com/retnuh/mochiweb_xpath They are much faster, than xmerl and don't produce any atoms (uses binaries). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Sat Aug 17 03:30:11 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 16 Aug 2013 21:30:11 -0400 Subject: [erlang-questions] Why I cannot start sasl from my app file application specification In-Reply-To: <000501ce9a28$23d90e80$6b8b2b80$@frcuba.co.cu> References: <000501ce9a28$23d90e80$6b8b2b80$@frcuba.co.cu> Message-ID: <20130817013010.GG14660@ferdmbp.local> Answers inline On 08/15, Ivan Carmenates Garc?a wrote: > Hi Fred, > > Thanks, the best solution I came with for now is to start it manually from > start/2 callback function, because I have the Erlang R16B and I don't have > that new function implemented. I think that's nice to have another function > like ensure_all_started/1-2 in the application module, that makes the entire > application to be started as one or fail as one if the applications > specified in the app file couldn't be all started property. > Right. Ideally the 'proper' way to do things would be to use OTP releases which would figure out that dependency graph you have, and boots all applications at start time. A lot of projects, however, never get the ball rolling on releases and instead use ad-hoc start routines, which ensure_all_started/1-2 would at least normalize. > But I still think even when the normal application:start/1-2 function don't > required all the child applications to be started, still that, I think it > should do start the sasl application, because I think there is no apparent > reason why it should not to. > If you do not *need* an application like SASL for your app to work, you might as well not put it as a dependency of your application. That solves the problem directly. OTP code will alert through SASL whether it was started before it or not. > Another last thing, and I am sorry to bother you, there is a way to turn off > the intromission of the error_logger in the sasl reports?, because I have > repeated the same error when happens, one in the sasl like form: > > CRASH REPORT <0.35.0> 2013-08-15 > > > and another with the error_logger form: > > INFO REPORT <0.7.0> 2013-08-15 > > > Both reports of the same error but different error_loggers in the same sasl > report. > > I also tried to silent the error_logger with: {kernel, [{error_logger, > silent}]} but still... Some reports will be duplicated because they come from proc_lib processes and then the supervisor (CRASH REPORT and SUPERVISOR REPORT, respectively). I have no idea on how to get rid of only one or the other of these. You can however get rid of PROGRESS REPORTs by changing the 'errlog_type' SASL variable from 'all' to 'error'. INFO REPORTs, for their part, are messages generated through error_logger. These can be disabled by removing the error_logger handler, either through calling `error_logger:tty(false)' or by setting 'error_logger' as a kernel variable to 'silent'. You can test whether it worked by calling 'error_logger:info_report("abc")' and seeing if you get a report. Now you tried the latest one, but this has to be done as VM argument, either through sys.config, a config file passed with the -config flag, or plainly adding '-kernel error_logger silent' to the 'erl' command line. Doing it after the VM has booted is not an option, but I think this is the one you attempted. Regards, Fred. From wbin00@REDACTED Sat Aug 17 10:00:04 2013 From: wbin00@REDACTED (Bin Wang) Date: Sat, 17 Aug 2013 16:00:04 +0800 Subject: [erlang-questions] How to broadcaset with ranch? Message-ID: Hi, I'm new to ranch. In my application, I need to send some message to all connections. So I'd like to know can I get all connections from ranch, so I could use Transport:send to send them, or I must manage all the created connections by myself? Or is there any other better way? Thanks. Bin Wang From essen@REDACTED Sat Aug 17 10:10:41 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Sat, 17 Aug 2013 10:10:41 +0200 Subject: [erlang-questions] [99s-extend] How to broadcaset with ranch? In-Reply-To: References: Message-ID: <520F3001.5010504@ninenines.eu> On 08/17/2013 10:00 AM, Bin Wang wrote: > Hi, > > I'm new to ranch. In my application, I need to send some message to > all connections. So I'd like to know can I get all connections from > ranch, so I could use Transport:send to send them, or I must manage > all the created connections by myself? Or is there any other better > way? The best way to do that is on your end, using gproc properties. When the connection is accepted, register the process with the property and use the property to send messages to all processes. You don't need to unregister when the connection ends, gproc does that automatically. The hackish way to do that would be to call supervisor:which_children on the ranch_conns_sup supervisor of your listener, but that will slow down the accepting of new connections, so don't do this if you need high accept rates. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From janos.n.hary@REDACTED Sat Aug 17 13:51:23 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Sat, 17 Aug 2013 13:51:23 +0200 Subject: [erlang-questions] Persistent tree datastructure Message-ID: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> Hi, Is there any Erlang library or method to handle trees persistently with ACID properties? I'm looking for something to handle filesystem like data with 1-100million records and varying nesting levels. Thanks in advance, Janos -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sat Aug 17 13:59:50 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sat, 17 Aug 2013 13:59:50 +0200 Subject: [erlang-questions] Persistent tree datastructure In-Reply-To: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> References: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> Message-ID: <89AA9B46-0108-42AF-95E2-0016D93249CF@gmail.com> Why not an sql database? Sergej On Aug 17, 2013, at 1:51 PM, Janos Hary wrote: > Hi, > > Is there any Erlang library or method to handle trees persistently with ACID properties? > I?m looking for something to handle filesystem like data with 1-100million records and varying nesting levels. > > Thanks in advance, > Janos > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomas.kukosa@REDACTED Sat Aug 17 15:12:07 2013 From: tomas.kukosa@REDACTED (Kukosa, Tomas) Date: Sat, 17 Aug 2013 13:12:07 +0000 Subject: [erlang-questions] Persistent tree datastructure In-Reply-To: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> References: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> Message-ID: <771808E86A57134A85559759DD60290110F2164F@MCHP01MSX.global-ad.net> Hi, what about GT.M and egtm. http://www.fisglobal.com/products-technologyplatforms-gtm https://github.com/ztmr/egtm Regards, Tomas ________________________________ From: erlang-questions-bounces@REDACTED [erlang-questions-bounces@REDACTED] on behalf of Janos Hary [janos.n.hary@REDACTED] Sent: Saturday, August 17, 2013 1:51 PM To: erlang-questions@REDACTED Subject: [erlang-questions] Persistent tree datastructure Hi, Is there any Erlang library or method to handle trees persistently with ACID properties? I?m looking for something to handle filesystem like data with 1-100million records and varying nesting levels. Thanks in advance, Janos -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Aug 17 15:20:54 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 17 Aug 2013 15:20:54 +0200 Subject: [erlang-questions] barrel 2.1 released Message-ID: Hi all, I'm happy to announce the new release of barrel. barrel 2.1 is a service release and contains the following changes: new: add socket option to start_listener fix: Fix a bug in barrel_tcp option filter doc: fix typos The full changelog is available here: https://github.com/benoitc/barrel/compare/2.0...2.1 For those who want to use mochiweb with barrel you can use the barrel_mochiweb adapter available here: https://github.com/benoitc/barrel_mochiweb An example of its usage with webmachine is also available: https://github.com/benoitc/barrel_mochiweb/tree/master/examples/webmachine_demo Any feedback is appreciated, Enjoy! - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From seriy.pr@REDACTED Sat Aug 17 15:02:06 2013 From: seriy.pr@REDACTED (Sergey Prokhorov) Date: Sat, 17 Aug 2013 17:02:06 +0400 Subject: [erlang-questions] xmerl/xpath alternatives? In-Reply-To: <5C89CE9B-A94F-4A07-87DA-6C4DA49A0301@mtod.org> References: <5C89CE9B-A94F-4A07-87DA-6C4DA49A0301@mtod.org> Message-ID: <520F744E.8060900@gmail.com> (2'nd try...) A little outdated, but anyway, you can use HTML/XML parser from Mochiweb https://github.com/mochi/mochiweb/blob/master/src/mochiweb_html.erl and XPath implementation for it's tree https://github.com/retnuh/mochiweb_xpath Need to note, that https://github.com/maxlapshin/parsexml is compatible with mochiweb_html by tree format, so XPath implementation can be used with both parsers. They are much faster, than xmerl and don't produce any atoms (uses binaries). From bchesneau@REDACTED Sat Aug 17 18:24:28 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 17 Aug 2013 18:24:28 +0200 Subject: [erlang-questions] barrel 2.1 released In-Reply-To: References: Message-ID: ok I was bored a little so I forked cowboy to make it work with barrel: https://github.com/refuge/cowboy/tree/barrel Only some tests are OK for now, but the hello world and websockets examples are OK. WIP... - benoit On Sat, Aug 17, 2013 at 3:20 PM, Benoit Chesneau wrote: > Hi all, > > I'm happy to announce the new release of barrel. barrel 2.1 is a service > release and contains the following changes: > > new: add socket option to start_listener > fix: Fix a bug in barrel_tcp option filter > doc: fix typos > > The full changelog is available here: > > https://github.com/benoitc/barrel/compare/2.0...2.1 > > For those who want to use mochiweb with barrel you can use the > barrel_mochiweb adapter available here: > > https://github.com/benoitc/barrel_mochiweb > > An example of its usage with webmachine is also available: > > > https://github.com/benoitc/barrel_mochiweb/tree/master/examples/webmachine_demo > > Any feedback is appreciated, > > Enjoy! > > - benoit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Aug 17 23:09:36 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 17 Aug 2013 23:09:36 +0200 Subject: [erlang-questions] barrel 2.1 released In-Reply-To: References: Message-ID: For those who didn't click yet I forgot to mention that barrel is a generic TCP acceptor pool :) - benoit On Sat, Aug 17, 2013 at 3:20 PM, Benoit Chesneau wrote: > Hi all, > > I'm happy to announce the new release of barrel. barrel 2.1 is a service > release and contains the following changes: > > new: add socket option to start_listener > fix: Fix a bug in barrel_tcp option filter > doc: fix typos > > The full changelog is available here: > > https://github.com/benoitc/barrel/compare/2.0...2.1 > > For those who want to use mochiweb with barrel you can use the > barrel_mochiweb adapter available here: > > https://github.com/benoitc/barrel_mochiweb > > An example of its usage with webmachine is also available: > > > https://github.com/benoitc/barrel_mochiweb/tree/master/examples/webmachine_demo > > Any feedback is appreciated, > > Enjoy! > > - benoit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chan.sisowath@REDACTED Mon Aug 19 09:25:11 2013 From: chan.sisowath@REDACTED (chan sisowath) Date: Mon, 19 Aug 2013 15:25:11 +0800 Subject: [erlang-questions] Persistent tree datastructure In-Reply-To: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> References: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> Message-ID: i face this problem once, and i use mnesia with a custom backend to store the tree structure. the custom backend is a simple interface to the filesystem. 2013/8/17 Janos Hary > Hi,**** > > ** ** > > Is there any Erlang library or method to handle trees persistently with > ACID properties? **** > > I?m looking for something to handle filesystem like data with 1-100million > records and varying nesting levels.**** > > ** ** > > Thanks in advance,**** > > Janos**** > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Mon Aug 19 12:44:06 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Mon, 19 Aug 2013 12:44:06 +0200 Subject: [erlang-questions] Persistent tree datastructure In-Reply-To: References: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> Message-ID: <001c01ce9cc9$08bbfba0$1a33f2e0$@gmail.com> This is interesting. Would it be a custom mnesia_access callback module? Or what does the 'custom backend' exactly mean? Can you point me to some documentation or example? Thanks, Janos From: chan sisowath [mailto:chan.sisowath@REDACTED] Sent: Monday, August 19, 2013 9:25 AM To: Janos Hary Cc: erlang-questions Subject: Re: [erlang-questions] Persistent tree datastructure i face this problem once, and i use mnesia with a custom backend to store the tree structure. the custom backend is a simple interface to the filesystem. 2013/8/17 Janos Hary Hi, Is there any Erlang library or method to handle trees persistently with ACID properties? I'm looking for something to handle filesystem like data with 1-100million records and varying nesting levels. Thanks in advance, Janos _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From chan.sisowath@REDACTED Mon Aug 19 14:52:32 2013 From: chan.sisowath@REDACTED (chan sisowath) Date: Mon, 19 Aug 2013 20:52:32 +0800 Subject: [erlang-questions] Persistent tree datastructure In-Reply-To: <001c01ce9cc9$08bbfba0$1a33f2e0$@gmail.com> References: <001b01ce9b40$19a8ace0$4cfa06a0$@gmail.com> <001c01ce9cc9$08bbfba0$1a33f2e0$@gmail.com> Message-ID: my work was based on the work of Dukes Of Erl, Dukes of Erl: Tokyocabinet and *Mnesia* google it, here from china it s hard to provide the link, since his blog is on blogspot tree structure in SQL (MySql) is a bad idea to go at large scale, HDD + VFS is one of the oldest database i know. :). performance are not bad you should test it for your need. the PoC is quite easy to do, all the work was done by Dukes Of Erl. cheers. Chan 2013/8/19 Janos Hary > This is interesting. Would it be a custom mnesia_access callback module? > Or what does the ?custom backend? exactly mean?**** > > Can you point me to some documentation or example?**** > > ** ** > > Thanks,**** > > Janos**** > > ** ** > > *From:* chan sisowath [mailto:chan.sisowath@REDACTED] > *Sent:* Monday, August 19, 2013 9:25 AM > *To:* Janos Hary > *Cc:* erlang-questions > *Subject:* Re: [erlang-questions] Persistent tree datastructure**** > > ** ** > > i face this problem once,**** > > and i use mnesia with a custom backend to store the tree structure.**** > > the custom backend is a simple interface to the filesystem.**** > > ** ** > > 2013/8/17 Janos Hary **** > > Hi,**** > > **** > > Is there any Erlang library or method to handle trees persistently with > ACID properties? **** > > I?m looking for something to handle filesystem like data with 1-100million > records and varying nesting levels.**** > > **** > > Thanks in advance,**** > > Janos**** > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yash_ganthe@REDACTED Mon Aug 19 16:22:27 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Mon, 19 Aug 2013 14:22:27 +0000 Subject: [erlang-questions] Node gets terminated when worker completes Message-ID: Hi, In the supervisor I have: init(_Args) -> WorkerSpec= {?WORKERMOD, {?WORKERMOD,start_link,[]}, temporary, 5000, worker, [?WORKERMOD]}, {ok,{{simple_one_for_one,1,1}, [WorkerSpec]}}. The Worker is a gen_server which terminates gracefully after completing its work. Once this happens, the entire node just shuts down without any message. There is no dump file. Is there anything in the spec above that causes this behavior? I am using Erlang R16B (erts-5.10.1) [source] [64-bit] [async-threads:10] [kernel-poll:false] Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Mon Aug 19 21:36:38 2013 From: g@REDACTED (Garrett Smith) Date: Mon, 19 Aug 2013 14:36:38 -0500 Subject: [erlang-questions] SQLite bindings memory leak Message-ID: Before I dig into an apparent memory leak I'm seeing in the Erlang sqlite3 bindings, I'm wondering if anyone has also. I'm using this repo: git://github.com/Feuerlabs/erlang-sqlite3.git Garrett From sergej.jurecko@REDACTED Mon Aug 19 21:42:53 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Mon, 19 Aug 2013 21:42:53 +0200 Subject: [erlang-questions] SQLite bindings memory leak In-Reply-To: References: Message-ID: <746FABF9-488F-4FA7-9D12-E6156274A6A5@gmail.com> I reported a memory leak here: https://github.com/alexeyr/erlang-sqlite3/issues Sergej On Aug 19, 2013, at 9:36 PM, Garrett Smith wrote: > Before I dig into an apparent memory leak I'm seeing in the Erlang > sqlite3 bindings, I'm wondering if anyone has also. > > I'm using this repo: > > git://github.com/Feuerlabs/erlang-sqlite3.git > > Garrett > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Mon Aug 19 23:04:55 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Mon, 19 Aug 2013 23:04:55 +0200 Subject: [erlang-questions] Node gets terminated when worker completes In-Reply-To: References: Message-ID: What are the last life signs from the worker (or the system)? Have you got sasl started? Den 19/08/2013 17.27 skrev "Yash Ganthe" : > Hi,**** > > ** ** > > In the supervisor I have:**** > > ** ** > > init(_Args) ->**** > > WorkerSpec= {?WORKERMOD,**** > > {?WORKERMOD,start_link,[]},**** > > temporary,**** > > 5000,**** > > worker,**** > > [?WORKERMOD]},**** > > {ok,{{simple_one_for_one,1,1}, [WorkerSpec]}}.**** > > **** > > ** ** > > The Worker is a gen_server which terminates gracefully after completing > its work. Once this happens, the entire node just shuts down without any > message. There is no dump file.**** > > ** ** > > Is there anything in the spec above that causes this behavior?**** > > ** ** > > I am using Erlang R16B (erts-5.10.1) [source] [64-bit] [async-threads:10] > [kernel-poll:false]**** > > ** ** > > Thanks,**** > > Yash**** > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Ltd. It is intended > only for the use of the individual or entity to which it is addressed. If > you are not the intended recipient, you are not authorized to read, retain, > copy, print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Persistent Systems Ltd. does not accept any liability for > virus infected mails. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john_re@REDACTED Tue Aug 20 07:02:20 2013 From: john_re@REDACTED (giovanni_re) Date: Mon, 19 Aug 2013 22:02:20 -0700 Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) Message-ID: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> https://news.ycombinator.com/item?id=6234736 Rewriting a large production system in Go (matt-welsh.blogspot.com) 222 points by qdie 1 day ago | 173 comments http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html MW: PhD U. Cali Forni Berkeley ========== https://news.ycombinator.com/item?id=6234736 derefr 1 day ago | link You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :) reply [GR: Would Googl use an Erlang on its' search engine show?] zaphar 1 day ago | link (full disclosure: I work at google and also like erlang) Erlang has fantastic facilities for robustness and concurrency. What it does not have is type safety and it's terrible at handling text in a performant fashion. So if you don't care about either of those things and only care about robustness and concurrency then Erlang is great. There were internal discussions about Erlang here but the upshot was. We had already basically duplicated Erlangs supervision model in our infrastructure, only we did it for all languages and Erlang didn't offer any benefits in performance for us. It's only benefit would have been the concurrency model. That's much less benefit than Go gives. Go gives you Erlangs concurency model, a similar philosophy of robustness, type safety, all batteries included, and performance. Equating the two languages works in 1 or 2 dimensions but not on all the dimensions google cares about. reply derefr 1 day ago | link Interesting, thanks for that; it's pretty much what I guessed (especially the bit about the supervision tree and hot-code-upgrade advantages being mooted by your infrastructure.) On a tangent, though: > What it does not have is type safety I've tried to work this out before (I'm designing a new language for Erlang's VM), but as far as I can tell, type safety is in practice incompatible with VM-supported hot code upgrade. If you have two services, A and B, and you need to upgrade them both, but you can't "stop the world" to do an atomic upgrade of both A and B together (because you're running a distributed soft real-time system, after all), then you need to switch out A, and then switch out B. So, at some point, on some nodes, A will be running a version with an ABI incompatible with B. In a strongly-typed system, the VM wouldn't allow A's new code to load, since it refers to functions in B with type signatures that don't exist. On the other hand, in a system with pattern-matching and a "let it crash" philosophy, you just let A's new code start up and repeatedly try-and-fail to communicate with B for a while, until B's code gets upgraded as well--and now the types are compatible again. It's an interesting problem. reply laureny 1 day ago | link > type safety is in practice incompatible with VM-supported hot code upgrade. That's not true. First, it's very easy to hot reload changes that have been made to the code that are backward compatible. The JVM spec describes in very specific details what that means (adding or removing a method is not backward compatible, modifying a body is, etc...). This is how Hotswap works, the JVM has been using it for years. As for changes that are backward incompatible, you can still manage them with application level techniques, such as rolling out servers or simply allow two different versions of the class to exist at the same time (JRebel does that, as do other a few other products in the JVM ecosystem). Erlang doesn't really have any advantages over statically typed systems in the hot reload area, and its lack of static typing is a deal breaker for pretty much any serious production deployment. reply rdtsc 20 hours ago | link > lack of static typing is a deal breaker for pretty much any serious production deployment. Are you talking about Google only where they made it a mandate or in general? There are serious production deployments on Python, Ruby, Erlang and Javascript. I will trade expressiveness and less lines of code with a strong but dynamically typed language + tests over more a static typed language with more lines of code all being equal. Or put it another way, if strong typing is the main thing that protects against lack of faults and crashes in production, there is a serious issue that needs to be addressed (just my 2 cents). reply derefr 23 hours ago | link > As for changes that are backward incompatible, you can still manage them with application level techniques, such as rolling out servers or simply allow two different versions of the class to exist at the same time (JRebel does that, as do other a few other products in the JVM ecosystem). Neither of these allow for the whole reason Erlang has hot code upgrade in the first place: allowing to upgrade the code on one side of a TCP connection without dropping the connection to the other side. Tell me how to do that with a static type system :) reply pmahoney 10 hours ago | link Tomcat (and other app servers) has support for doing hot reloads of Java web apps while not reloading the HTTP layer (and not dropping TCP connections). http://www.javacodegeeks.com/2011/06/zero-downtime-deploymen... I have implemented a similar system for JRuby apps running inside a Servlet container. There are many caveats. I don't actually recommend it because for a while you're using nearly twice the memory (and JRuby is particularly memory hungry). Also there are many ways to leak the old class definitions such that they are not GC'd (e.g. thread locals). But it's certainly possible. I suspect that Erlang, Java, and all languages are in the same boat: some parts can be upgraded live in the VM while other parts require a full restart (maybe coordinating with multiple nodes and a load balancer to achieve zero-downtime). reply lenkite 10 hours ago | link Out of curiosity, where/why would such an exotic feature be needed in today's internet architectures where you always front a group of servers with a load balancer ? reply butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If you're running a service where long-lived connections are the norm, "simply fronting a bunch of servers with a load balancer" can require a pretty smart load balancer. E.g. IMAP connections often last hours or even days, and are required to maintain a degree of statefulness. reply DanWaterworth 1 day ago | link Go gives you Erlangs concurency model There are a number of significant differences between Erlang's and Go's concurrency models: Asynchronous vs synchronous communication, per-thread vs per-process heaps, send to process vs send to channel. reply dbaupp 1 day ago | link Go doesn't have strong type safety either; I remember a recent story about a Go stdlib function "accidentally" calling an interface it shouldn't. reply f2f 22 hours ago | link it wasn't accidental -- it was written on purpose by a programmer (a conversion from Writer to WriteCloser). it was immediately acknowledged as an error and eventually may be caught by the standard code examining tool "vet". reply pcwalton 22 hours ago | link What would the static analysis that "vet" is performing enforce to stop this? No interface-to-interface downcasts? reply ..... [It goes on, & on, & on & on, when Go & Erlang party together.] --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/group/BerkTIPGlobal. All Freedom SW, HW & Culture. From snar@REDACTED Tue Aug 20 09:08:48 2013 From: snar@REDACTED (Alexandre Snarskii) Date: Tue, 20 Aug 2013 11:08:48 +0400 Subject: [erlang-questions] xmerl and "encoded" utf-8 attributes ? Message-ID: <20130820070848.GA6513@snar.spb.ru> Hi! one-liner example: taking attribute value from XML: 26> Xl= <<"">>, {El, _} = xmerl_scan:string(binary_to_list(Xl)), [A]=El#xmlElement.attributes, Avb=list_to_binary(A#xmlAttribute.value). <<49,48,48,32,129,209,130,209,32,97,110,100,32,109,111,114,101>> and then conversion to unicode fails after first four bytes: 27> unicode:characters_to_list(Avb). {error,"100 ", <<129,209,130,209,32,97,110,100,32,109,111,114,101>>} What I think is wrong here: first four bytes (49,48,48,32) are fine ("100 " in binary), last bytes (32,97,110,100,32,109,111,114,101) are fine too (" and more"), but middle four looks like "inverted utf8" with first and second byte swapped. Indeed, per utf-8 specification, encoding for 16#441 should be [ 2#110 + , 2#10 + ], or 30> [2#11000000 bor (16#441 bsr 6), 2#10000000 bor (16#441 band 2#0111111)]. [209,129] and for 16#442: 31> [2#11000000 bor (16#442 bsr 6), 2#10000000 bor (16#442 band 2#0111111)]. [209,130] but in provided binary these bytes are reordered as 129,209 and 130,209. After manual byte reordering everything is fine: 29> Avc = <<49,48,48,32,209,129,209,130,32,97,110,100,32,109,111,114,101>>. <<49,48,48,32,209,129,209,130,32,97,110,100,32,109,111,114,101>> 30> unicode:characters_to_list(Avc). [49,48,48,32,1089,1090,32,97,110,100,32,109,111,114,101] Bug ? Feature ? My misunderstanding on how it supposed to work ? -- In theory, there is no difference between theory and practice. But, in practice, there is. From essen@REDACTED Tue Aug 20 09:26:49 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 20 Aug 2013 09:26:49 +0200 Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) In-Reply-To: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> References: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> Message-ID: <52131A39.6000803@ninenines.eu> Why do you paste us HN discussions? There's no better waste of time than to read programmers comparing "sizes". On 08/20/2013 07:02 AM, giovanni_re wrote: > https://news.ycombinator.com/item?id=6234736 > Rewriting a large production system in Go (matt-welsh.blogspot.com) > 222 points by qdie 1 day ago | 173 comments > http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html > > MW: PhD U. Cali Forni Berkeley > > > > ========== https://news.ycombinator.com/item?id=6234736 > derefr 1 day ago | link You know, every time I see some Googler shocked > at the effectiveness and various advantages of coding in Go, I wonder > why Google never adopted Erlang. They could have been getting all these > same advantages (and then some) a decade ago :) > > reply > > [GR: Would Googl use an Erlang on its' search engine show?] > > > > > zaphar 1 day ago | link (full disclosure: I work at google and also like > erlang) Erlang has fantastic facilities for robustness and concurrency. > What it does not have is type safety and it's terrible at handling text > in a performant fashion. So if you don't care about either of those > things and only care about robustness and concurrency then Erlang is > great. There were internal discussions about Erlang here but the upshot > was. We had already basically duplicated Erlangs supervision model in > our infrastructure, only we did it for all languages and Erlang didn't > offer any benefits in performance for us. It's only benefit would have > been the concurrency model. That's much less benefit than Go gives. Go > gives you Erlangs concurency model, a similar philosophy of robustness, > type safety, all batteries included, and performance. Equating the two > languages works in 1 or 2 dimensions but not on all the dimensions > google cares about. > > reply > > derefr 1 day ago | link Interesting, thanks for that; it's pretty much > what I guessed (especially the bit about the supervision tree and > hot-code-upgrade advantages being mooted by your infrastructure.) On a > tangent, though: > What it does not have is type safety I've tried to > work this out before (I'm designing a new language for Erlang's VM), but > as far as I can tell, type safety is in practice incompatible with > VM-supported hot code upgrade. If you have two services, A and B, and > you need to upgrade them both, but you can't "stop the world" to do an > atomic upgrade of both A and B together (because you're running a > distributed soft real-time system, after all), then you need to switch > out A, and then switch out B. So, at some point, on some nodes, A will > be running a version with an ABI incompatible with B. In a > strongly-typed system, the VM wouldn't allow A's new code to load, since > it refers to functions in B with type signatures that don't exist. On > the other hand, in a system with pattern-matching and a "let it crash" > philosophy, you just let A's new code start up and repeatedly > try-and-fail to communicate with B for a while, until B's code gets > upgraded as well--and now the types are compatible again. It's an > interesting problem. > > reply > > laureny 1 day ago | link > type safety is in practice incompatible with > VM-supported hot code upgrade. That's not true. First, it's very easy to > hot reload changes that have been made to the code that are backward > compatible. The JVM spec describes in very specific details what that > means (adding or removing a method is not backward compatible, modifying > a body is, etc...). This is how Hotswap works, the JVM has been using it > for years. As for changes that are backward incompatible, you can still > manage them with application level techniques, such as rolling out > servers or simply allow two different versions of the class to exist at > the same time (JRebel does that, as do other a few other products in the > JVM ecosystem). Erlang doesn't really have any advantages over > statically typed systems in the hot reload area, and its lack of static > typing is a deal breaker for pretty much any serious production > deployment. > > reply > > rdtsc 20 hours ago | link > lack of static typing is a deal breaker for > pretty much any serious production deployment. Are you talking about > Google only where they made it a mandate or in general? There are > serious production deployments on Python, Ruby, Erlang and Javascript. I > will trade expressiveness and less lines of code with a strong but > dynamically typed language + tests over more a static typed language > with more lines of code all being equal. Or put it another way, if > strong typing is the main thing that protects against lack of faults and > crashes in production, there is a serious issue that needs to be > addressed (just my 2 cents). > > reply > > derefr 23 hours ago | link > As for changes that are backward > incompatible, you can still manage them with application level > techniques, such as rolling out servers or simply allow two different > versions of the class to exist at the same time (JRebel does that, as do > other a few other products in the JVM ecosystem). Neither of these allow > for the whole reason Erlang has hot code upgrade in the first place: > allowing to upgrade the code on one side of a TCP connection without > dropping the connection to the other side. Tell me how to do that with a > static type system :) > > reply > > pmahoney 10 hours ago | link Tomcat (and other app servers) has support > for doing hot reloads of Java web apps while not reloading the HTTP > layer (and not dropping TCP connections). > http://www.javacodegeeks.com/2011/06/zero-downtime-deploymen... I have > implemented a similar system for JRuby apps running inside a Servlet > container. There are many caveats. I don't actually recommend it because > for a while you're using nearly twice the memory (and JRuby is > particularly memory hungry). Also there are many ways to leak the old > class definitions such that they are not GC'd (e.g. thread locals). But > it's certainly possible. I suspect that Erlang, Java, and all languages > are in the same boat: some parts can be upgraded live in the VM while > other parts require a full restart (maybe coordinating with multiple > nodes and a load balancer to achieve zero-downtime). > > reply > > lenkite 10 hours ago | link Out of curiosity, where/why would such an > exotic feature be needed in today's internet architectures where you > always front a group of servers with a load balancer ? > > reply > > butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If > you're running a service where long-lived connections are the norm, > "simply fronting a bunch of servers with a load balancer" can require a > pretty smart load balancer. E.g. IMAP connections often last hours or > even days, and are required to maintain a degree of statefulness. > > reply > > DanWaterworth 1 day ago | link Go gives you Erlangs concurency model > There are a number of significant differences between Erlang's and Go's > concurrency models: Asynchronous vs synchronous communication, > per-thread vs per-process heaps, send to process vs send to channel. > > reply > > dbaupp 1 day ago | link Go doesn't have strong type safety either; I > remember a recent story about a Go stdlib function "accidentally" > calling an interface it shouldn't. > > reply > > f2f 22 hours ago | link it wasn't accidental -- it was written on > purpose by a programmer (a conversion from Writer to WriteCloser). it > was immediately acknowledged as an error and eventually may be caught by > the standard code examining tool "vet". > > reply > > pcwalton 22 hours ago | link What would the static analysis that "vet" > is performing enforce to stop this? No interface-to-interface downcasts? > > reply > > ..... > [It goes on, & on, & on & on, > when Go & Erlang party together.] > > --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/group/BerkTIPGlobal. All Freedom SW, HW & Culture. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From peerst@REDACTED Tue Aug 20 12:43:12 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Tue, 20 Aug 2013 12:43:12 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? References: Message-ID: Hi, I'm also interested in reducing the OS allocation and free's but for a different use case: In small embedded systems with no virtual memory (running on RTEMS e.g.) there are several problems with malloc/free: 1. its very slow. 2. the system heap fragments. There is no VM to help. My favorite behaviour would be for Erlang to either grab all memory available or at least a fixed amount on startup and hold on to it forever. Hopefully without fragmenting it too much (I know in case of binary heaps that this isn't possible which is a bit disturbing to me). Can this be achieved by the current ERTS with some parameter settings? Which direction should I look if I want to add behaviour like this? BTW: next best thing is to malloc/free allways same size tiles this would avoid outer fragmentation and could be sped up relatively easily. Cheers, -- Peer On 2013-08-15 08:05:01 +0000, Lukas Larsson said: > Hello, > > Did increasing the max mseg_alloc sbc limit help? > > After thinking a bit more on this issue, I think that my original > hypothesis was correct. You are allocating way more sbc than you want > to. Moving more of you allocations into mbcs should drastically > decrease the number of mmaps needed and also instead of placing one > binary within one mmap area, it will place many binaries within a > larger mmaped area. > > Do you have any idea of the average sizes of the binaries that you are > allocating? If you do I would try to set the sbct to be high enough > that maybe 80% of all binaries fit below it. At the same time you > should probably also adjust the smbcs and lmbcs to a higher value, > maybe something like smbcs = 0.5*sbct lmbcs = 10*sbct. If you know that > you will be using a lot of binaries you could also set smbcs to be > higher to start with, or decrease the growth stages (mbcgs) so that you > faster get to the lmbcs. You will have to play around with sbct, smbcs, > lmbcs and mbcgs and measure performance and memory usage to see what is > the best fit for you. > > When doing this you can remove the mmsbc config again as you will > (hopefully) be using very few sbcs. > > > Lukas > > > > On Wed, Aug 14, 2013 at 4:57 PM, Max Lapshin wrote: > Oh, thank you, will try it. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Tue Aug 20 15:11:29 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 20 Aug 2013 15:11:29 +0200 Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) In-Reply-To: <52131A39.6000803@ninenines.eu> References: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> <52131A39.6000803@ninenines.eu> Message-ID: I remember reading that discussion on Hacker News as for why Google considered but did not pick Erlang. The reason I did not answer was because I don't believe it would be fruitful to do so, but on this mailing list I can't resist. One of the things I've been looking into as of late is the price paid for error handling. When you anticipate error and handle it, there is an overhead to this handling. It is easy to get extreme Queries per second (QPS) rates if you ignore faults and have no way to mitigate them. I believe that the stance of many systems are to claim that the system is void of any internal error and then run without the internal error protection of the system. Outside, you run something like monit in order to keep the system up and restarted should it die. In turn, such systems can be made so they run much faster than the typical Erlang system easily: * There is no need for doing hot code upgrade on the fly. Hence, you can native-compile everything into one static binary, inline like mad and so on. This speeds up a lot of the computational stuff. * You don't need to set a monitor for each call. Since there are no errors (by your claim) the code never goes wrong and hence why set it? * You have no distribution. So everything happens locally in your own memory space. This usually gives complete knowledge and simplifies a lot of possible error paths to "non-existent". To make Go work in practice, you need to adopt this stance: you deploy small statically compiled binaries which are "obviously correct" by being small. And if something is amiss, you do not go for a partial failure or a gradual degradation of the system: off with its head! It is a trade-off. You get more speed, but are ignoring certain failure scenarios. But you are also suddenly going to rely 100% on the code being correct. In your code and in all libraries you use. On Tue, Aug 20, 2013 at 9:26 AM, Lo?c Hoguin wrote: > Why do you paste us HN discussions? There's no better waste of time than > to read programmers comparing "sizes". > > > On 08/20/2013 07:02 AM, giovanni_re wrote: > >> https://news.ycombinator.com/**item?id=6234736 >> Rewriting a large production system in Go (matt-welsh.blogspot.com) >> 222 points by qdie 1 day ago | 173 comments >> http://matt-welsh.blogspot.**com/2013/08/rewriting-large-** >> production-system-in-go.html >> >> MW: PhD U. Cali Forni Berkeley >> >> >> >> ========== https://news.ycombinator.com/**item?id=6234736 >> derefr 1 day ago | link You know, every time I see some Googler shocked >> at the effectiveness and various advantages of coding in Go, I wonder >> why Google never adopted Erlang. They could have been getting all these >> same advantages (and then some) a decade ago :) >> >> reply >> >> [GR: Would Googl use an Erlang on its' search engine show?] >> >> >> >> >> zaphar 1 day ago | link (full disclosure: I work at google and also like >> erlang) Erlang has fantastic facilities for robustness and concurrency. >> What it does not have is type safety and it's terrible at handling text >> in a performant fashion. So if you don't care about either of those >> things and only care about robustness and concurrency then Erlang is >> great. There were internal discussions about Erlang here but the upshot >> was. We had already basically duplicated Erlangs supervision model in >> our infrastructure, only we did it for all languages and Erlang didn't >> offer any benefits in performance for us. It's only benefit would have >> been the concurrency model. That's much less benefit than Go gives. Go >> gives you Erlangs concurency model, a similar philosophy of robustness, >> type safety, all batteries included, and performance. Equating the two >> languages works in 1 or 2 dimensions but not on all the dimensions >> google cares about. >> >> reply >> >> derefr 1 day ago | link Interesting, thanks for that; it's pretty much >> what I guessed (especially the bit about the supervision tree and >> hot-code-upgrade advantages being mooted by your infrastructure.) On a >> tangent, though: > What it does not have is type safety I've tried to >> work this out before (I'm designing a new language for Erlang's VM), but >> as far as I can tell, type safety is in practice incompatible with >> VM-supported hot code upgrade. If you have two services, A and B, and >> you need to upgrade them both, but you can't "stop the world" to do an >> atomic upgrade of both A and B together (because you're running a >> distributed soft real-time system, after all), then you need to switch >> out A, and then switch out B. So, at some point, on some nodes, A will >> be running a version with an ABI incompatible with B. In a >> strongly-typed system, the VM wouldn't allow A's new code to load, since >> it refers to functions in B with type signatures that don't exist. On >> the other hand, in a system with pattern-matching and a "let it crash" >> philosophy, you just let A's new code start up and repeatedly >> try-and-fail to communicate with B for a while, until B's code gets >> upgraded as well--and now the types are compatible again. It's an >> interesting problem. >> >> reply >> >> laureny 1 day ago | link > type safety is in practice incompatible with >> VM-supported hot code upgrade. That's not true. First, it's very easy to >> hot reload changes that have been made to the code that are backward >> compatible. The JVM spec describes in very specific details what that >> means (adding or removing a method is not backward compatible, modifying >> a body is, etc...). This is how Hotswap works, the JVM has been using it >> for years. As for changes that are backward incompatible, you can still >> manage them with application level techniques, such as rolling out >> servers or simply allow two different versions of the class to exist at >> the same time (JRebel does that, as do other a few other products in the >> JVM ecosystem). Erlang doesn't really have any advantages over >> statically typed systems in the hot reload area, and its lack of static >> typing is a deal breaker for pretty much any serious production >> deployment. >> >> reply >> >> rdtsc 20 hours ago | link > lack of static typing is a deal breaker for >> pretty much any serious production deployment. Are you talking about >> Google only where they made it a mandate or in general? There are >> serious production deployments on Python, Ruby, Erlang and Javascript. I >> will trade expressiveness and less lines of code with a strong but >> dynamically typed language + tests over more a static typed language >> with more lines of code all being equal. Or put it another way, if >> strong typing is the main thing that protects against lack of faults and >> crashes in production, there is a serious issue that needs to be >> addressed (just my 2 cents). >> >> reply >> >> derefr 23 hours ago | link > As for changes that are backward >> incompatible, you can still manage them with application level >> techniques, such as rolling out servers or simply allow two different >> versions of the class to exist at the same time (JRebel does that, as do >> other a few other products in the JVM ecosystem). Neither of these allow >> for the whole reason Erlang has hot code upgrade in the first place: >> allowing to upgrade the code on one side of a TCP connection without >> dropping the connection to the other side. Tell me how to do that with a >> static type system :) >> >> reply >> >> pmahoney 10 hours ago | link Tomcat (and other app servers) has support >> for doing hot reloads of Java web apps while not reloading the HTTP >> layer (and not dropping TCP connections). >> http://www.javacodegeeks.com/**2011/06/zero-downtime-**deploymen... >> I have >> implemented a similar system for JRuby apps running inside a Servlet >> container. There are many caveats. I don't actually recommend it because >> for a while you're using nearly twice the memory (and JRuby is >> particularly memory hungry). Also there are many ways to leak the old >> class definitions such that they are not GC'd (e.g. thread locals). But >> it's certainly possible. I suspect that Erlang, Java, and all languages >> are in the same boat: some parts can be upgraded live in the VM while >> other parts require a full restart (maybe coordinating with multiple >> nodes and a load balancer to achieve zero-downtime). >> >> reply >> >> lenkite 10 hours ago | link Out of curiosity, where/why would such an >> exotic feature be needed in today's internet architectures where you >> always front a group of servers with a load balancer ? >> >> reply >> >> butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If >> you're running a service where long-lived connections are the norm, >> "simply fronting a bunch of servers with a load balancer" can require a >> pretty smart load balancer. E.g. IMAP connections often last hours or >> even days, and are required to maintain a degree of statefulness. >> >> reply >> >> DanWaterworth 1 day ago | link Go gives you Erlangs concurency model >> There are a number of significant differences between Erlang's and Go's >> concurrency models: Asynchronous vs synchronous communication, >> per-thread vs per-process heaps, send to process vs send to channel. >> >> reply >> >> dbaupp 1 day ago | link Go doesn't have strong type safety either; I >> remember a recent story about a Go stdlib function "accidentally" >> calling an interface it shouldn't. >> >> reply >> >> f2f 22 hours ago | link it wasn't accidental -- it was written on >> purpose by a programmer (a conversion from Writer to WriteCloser). it >> was immediately acknowledged as an error and eventually may be caught by >> the standard code examining tool "vet". >> >> reply >> >> pcwalton 22 hours ago | link What would the static analysis that "vet" >> is performing enforce to stop this? No interface-to-interface downcasts? >> >> reply >> >> ..... >> [It goes on, & on, & on & on, >> when Go & Erlang party together.] >> >> --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/* >> *group/BerkTIPGlobal . All >> Freedom SW, HW & Culture. >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> >> > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Tue Aug 20 17:04:41 2013 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 20 Aug 2013 17:04:41 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: Hello Peer, On Tue, Aug 20, 2013 at 12:43 PM, Peer Stritzinger wrote: > My favorite behaviour would be for Erlang to either grab all memory > available or at least a fixed amount on startup and hold on to it forever. > Hopefully without fragmenting it too much (I know in case of binary heaps > that this isn't possible which is a bit disturbing to me). > > Can this be achieved by the current ERTS with some parameter settings? > You could request that the main mbc (+Mmmbcs) should be preallocated to a certain size, and this would kind of get the effect you are looking for. However this is controlled on a per allocator level, so you have to figure out which type of memory it is that you are using. It also will only work for any mbcs allocated, all sbcs will still be allocated using mmap/malloc. If you want to force only mbcs you could up the sbct to very big, just remember to up lmbcs as well so that can fit all of your sbcs into the mbc. I do not recommend doing this, but it is possible. > Which direction should I look if I want to add behaviour like this? > > BTW: next best thing is to malloc/free allways same size tiles this would > avoid outer fragmentation and could be sped up relatively easily. > The emulator already tries to do this to the best of its abilities. mbcs mmap requests grow depending on how you setup lmbcs, smbcs and mbcgs. sys_alloc (i.e. malloc) calls are always multiples of +Muycs. You could set smbcs to the same as lmbcs if you want to, this makes it so that all mbcs have that same size. Lukas* * -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Wed Aug 21 04:27:22 2013 From: garry@REDACTED (Garry Hodgson) Date: Tue, 20 Aug 2013 22:27:22 -0400 Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) In-Reply-To: <52131A39.6000803@ninenines.eu> References: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> <52131A39.6000803@ninenines.eu> Message-ID: <5214258A.5010408@research.att.com> except perhaps reading people complaining about it. or people responding to such complaints. On 08/20/2013 03:26 AM, Lo?c Hoguin wrote: > Why do you paste us HN discussions? There's no better waste of time > than to read programmers comparing "sizes". > > On 08/20/2013 07:02 AM, giovanni_re wrote: >> https://news.ycombinator.com/item?id=6234736 >> Rewriting a large production system in Go (matt-welsh.blogspot.com) >> 222 points by qdie 1 day ago | 173 comments >> http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html >> >> >> MW: PhD U. Cali Forni Berkeley >> >> >> >> ========== https://news.ycombinator.com/item?id=6234736 >> derefr 1 day ago | link You know, every time I see some Googler shocked >> at the effectiveness and various advantages of coding in Go, I wonder >> why Google never adopted Erlang. They could have been getting all these >> same advantages (and then some) a decade ago :) >> >> reply >> >> [GR: Would Googl use an Erlang on its' search engine show?] >> >> >> >> >> zaphar 1 day ago | link (full disclosure: I work at google and also like >> erlang) Erlang has fantastic facilities for robustness and concurrency. >> What it does not have is type safety and it's terrible at handling text >> in a performant fashion. So if you don't care about either of those >> things and only care about robustness and concurrency then Erlang is >> great. There were internal discussions about Erlang here but the upshot >> was. We had already basically duplicated Erlangs supervision model in >> our infrastructure, only we did it for all languages and Erlang didn't >> offer any benefits in performance for us. It's only benefit would have >> been the concurrency model. That's much less benefit than Go gives. Go >> gives you Erlangs concurency model, a similar philosophy of robustness, >> type safety, all batteries included, and performance. Equating the two >> languages works in 1 or 2 dimensions but not on all the dimensions >> google cares about. >> >> reply >> >> derefr 1 day ago | link Interesting, thanks for that; it's pretty much >> what I guessed (especially the bit about the supervision tree and >> hot-code-upgrade advantages being mooted by your infrastructure.) On a >> tangent, though: > What it does not have is type safety I've tried to >> work this out before (I'm designing a new language for Erlang's VM), but >> as far as I can tell, type safety is in practice incompatible with >> VM-supported hot code upgrade. If you have two services, A and B, and >> you need to upgrade them both, but you can't "stop the world" to do an >> atomic upgrade of both A and B together (because you're running a >> distributed soft real-time system, after all), then you need to switch >> out A, and then switch out B. So, at some point, on some nodes, A will >> be running a version with an ABI incompatible with B. In a >> strongly-typed system, the VM wouldn't allow A's new code to load, since >> it refers to functions in B with type signatures that don't exist. On >> the other hand, in a system with pattern-matching and a "let it crash" >> philosophy, you just let A's new code start up and repeatedly >> try-and-fail to communicate with B for a while, until B's code gets >> upgraded as well--and now the types are compatible again. It's an >> interesting problem. >> >> reply >> >> laureny 1 day ago | link > type safety is in practice incompatible with >> VM-supported hot code upgrade. That's not true. First, it's very easy to >> hot reload changes that have been made to the code that are backward >> compatible. The JVM spec describes in very specific details what that >> means (adding or removing a method is not backward compatible, modifying >> a body is, etc...). This is how Hotswap works, the JVM has been using it >> for years. As for changes that are backward incompatible, you can still >> manage them with application level techniques, such as rolling out >> servers or simply allow two different versions of the class to exist at >> the same time (JRebel does that, as do other a few other products in the >> JVM ecosystem). Erlang doesn't really have any advantages over >> statically typed systems in the hot reload area, and its lack of static >> typing is a deal breaker for pretty much any serious production >> deployment. >> >> reply >> >> rdtsc 20 hours ago | link > lack of static typing is a deal breaker for >> pretty much any serious production deployment. Are you talking about >> Google only where they made it a mandate or in general? There are >> serious production deployments on Python, Ruby, Erlang and Javascript. I >> will trade expressiveness and less lines of code with a strong but >> dynamically typed language + tests over more a static typed language >> with more lines of code all being equal. Or put it another way, if >> strong typing is the main thing that protects against lack of faults and >> crashes in production, there is a serious issue that needs to be >> addressed (just my 2 cents). >> >> reply >> >> derefr 23 hours ago | link > As for changes that are backward >> incompatible, you can still manage them with application level >> techniques, such as rolling out servers or simply allow two different >> versions of the class to exist at the same time (JRebel does that, as do >> other a few other products in the JVM ecosystem). Neither of these allow >> for the whole reason Erlang has hot code upgrade in the first place: >> allowing to upgrade the code on one side of a TCP connection without >> dropping the connection to the other side. Tell me how to do that with a >> static type system :) >> >> reply >> >> pmahoney 10 hours ago | link Tomcat (and other app servers) has support >> for doing hot reloads of Java web apps while not reloading the HTTP >> layer (and not dropping TCP connections). >> http://www.javacodegeeks.com/2011/06/zero-downtime-deploymen... I have >> implemented a similar system for JRuby apps running inside a Servlet >> container. There are many caveats. I don't actually recommend it because >> for a while you're using nearly twice the memory (and JRuby is >> particularly memory hungry). Also there are many ways to leak the old >> class definitions such that they are not GC'd (e.g. thread locals). But >> it's certainly possible. I suspect that Erlang, Java, and all languages >> are in the same boat: some parts can be upgraded live in the VM while >> other parts require a full restart (maybe coordinating with multiple >> nodes and a load balancer to achieve zero-downtime). >> >> reply >> >> lenkite 10 hours ago | link Out of curiosity, where/why would such an >> exotic feature be needed in today's internet architectures where you >> always front a group of servers with a load balancer ? >> >> reply >> >> butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If >> you're running a service where long-lived connections are the norm, >> "simply fronting a bunch of servers with a load balancer" can require a >> pretty smart load balancer. E.g. IMAP connections often last hours or >> even days, and are required to maintain a degree of statefulness. >> >> reply >> >> DanWaterworth 1 day ago | link Go gives you Erlangs concurency model >> There are a number of significant differences between Erlang's and Go's >> concurrency models: Asynchronous vs synchronous communication, >> per-thread vs per-process heaps, send to process vs send to channel. >> >> reply >> >> dbaupp 1 day ago | link Go doesn't have strong type safety either; I >> remember a recent story about a Go stdlib function "accidentally" >> calling an interface it shouldn't. >> >> reply >> >> f2f 22 hours ago | link it wasn't accidental -- it was written on >> purpose by a programmer (a conversion from Writer to WriteCloser). it >> was immediately acknowledged as an error and eventually may be caught by >> the standard code examining tool "vet". >> >> reply >> >> pcwalton 22 hours ago | link What would the static analysis that "vet" >> is performing enforce to stop this? No interface-to-interface downcasts? >> >> reply >> >> ..... >> [It goes on, & on, & on & on, >> when Go & Erlang party together.] >> >> --- Join the BerkeleyTIP-Global mail list - >> http://groups.google.com/group/BerkTIPGlobal. All Freedom SW, HW & >> Culture. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -- Garry Hodgson AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From seancribbs@REDACTED Wed Aug 21 05:18:19 2013 From: seancribbs@REDACTED (Sean Cribbs) Date: Tue, 20 Aug 2013 22:18:19 -0500 Subject: [erlang-questions] [99s-extend] How to broadcaset with ranch? In-Reply-To: <520F3001.5010504@ninenines.eu> References: <520F3001.5010504@ninenines.eu> Message-ID: This is exactly the sort of thing gen_event is for. I would make each server process register a handler at startup using gen_event:add_sup_handler() and then have the handle_event callback simply relay the event to the server processes. Yes, gproc can do this, but why incur its extra features and overhead? On Sat, Aug 17, 2013 at 3:10 AM, Lo?c Hoguin wrote: > On 08/17/2013 10:00 AM, Bin Wang wrote: > >> Hi, >> >> I'm new to ranch. In my application, I need to send some message to >> all connections. So I'd like to know can I get all connections from >> ranch, so I could use Transport:send to send them, or I must manage >> all the created connections by myself? Or is there any other better >> way? >> > > The best way to do that is on your end, using gproc properties. When the > connection is accepted, register the process with the property and use the > property to send messages to all processes. You don't need to unregister > when the connection ends, gproc does that automatically. > > The hackish way to do that would be to call supervisor:which_children on > the ranch_conns_sup supervisor of your listener, but that will slow down > the accepting of new connections, so don't do this if you need high accept > rates. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew@REDACTED Wed Aug 21 05:38:19 2013 From: andrew@REDACTED (Andrew Thompson) Date: Tue, 20 Aug 2013 23:38:19 -0400 Subject: [erlang-questions] [99s-extend] How to broadcaset with ranch? In-Reply-To: References: <520F3001.5010504@ninenines.eu> Message-ID: <20130821033819.GT19389@hijacked.us> On Tue, Aug 20, 2013 at 10:18:19PM -0500, Sean Cribbs wrote: > This is exactly the sort of thing gen_event is for. I would make each > server process register a handler at startup using > gen_event:add_sup_handler() and then have the handle_event callback simply > relay the event to the server processes. Yes, gproc can do this, but why > incur its extra features and overhead? pg2 also works fairly well. Andrew From hans.bolinder@REDACTED Wed Aug 21 08:55:36 2013 From: hans.bolinder@REDACTED (Hans Bolinder) Date: Wed, 21 Aug 2013 06:55:36 +0000 Subject: [erlang-questions] internal error in lint_module In-Reply-To: References: Message-ID: <56466BD70414EA48969B4064696CF28C081133@ESESSMB207.ericsson.se> Hi, [Mikhail Sobolev on Monday, 31 December 2012:] > A minimal example I managed to come out with is: > > --- cut here --- > -module(p2). > > -export([ > manifest/2 > ]). > > manifest(Module, Name) -> > fun Module:Nine/1. > --- cut here --- > > While compiling with erlc, the following error message is produced: > > --- cut here --- > p2.erl:none: internal error in lint_module; > crash reason: {{case_clause,{location,8}}, > [{erl_lint,loc,1,[{file,"erl_lint.erl"},{line,598}]}, > {erl_lint,add_error,3,[{file,"erl_lint.erl"},{line,588}]}, > {erl_lint,expr_var,4,[{file,"erl_lint.erl"},{line,3155}]}, > {erl_lint,'-expr_list/3-fun-0-',3, > [{file,"erl_lint.erl"},{line,2289}]}, > {lists,foldl,3,[{file,"lists.erl"},{line,1197}]}, > {erl_lint,expr,3,[{file,"erl_lint.erl"},{line,2151}]}, > {erl_lint,exprs,3,[{file,"erl_lint.erl"},{line,2047}]}, > {erl_lint,clause,3,[{file,"erl_lint.erl"},{line,1399}]}]} > --- cut here --- > > If the typo -- Nine -- is fixed, everything compiles just fine. Thanks for the bug report and sorry for the long delay. A fix should appear on the 'maint' branch soon. Best regards, Hans Bolinder, Erlang/OTP team, Ericsson From max.lapshin@REDACTED Wed Aug 21 09:53:05 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 21 Aug 2013 11:53:05 +0400 Subject: [erlang-questions] [99s-extend] How to broadcaset with ranch? In-Reply-To: <20130821033819.GT19389@hijacked.us> References: <520F3001.5010504@ninenines.eu> <20130821033819.GT19389@hijacked.us> Message-ID: It is important to understand that if you use gen_event to register event handlers, than all event traffic will flow through one single process. If you have 100 messages per second and 10 000 subscribers, than one single process (one single core) will send 1 mln of messages per second. If you register subscribers in public named ets table, than each producer will send only 10 000 messages, which can be distributed among cores. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tonyp.nospam@REDACTED Wed Aug 21 13:49:08 2013 From: tonyp.nospam@REDACTED (Tony Pedley) Date: Wed, 21 Aug 2013 12:49:08 +0100 Subject: [erlang-questions] Anyone working on a Erlang OPC-UA Implementation Message-ID: Is anyone working on an implementation on OPC-UA in erlang? Especially a client I would look it at myself but have little time and my Erlang is rusty(well very rusty) so It would take far to long to even get started. However it feels like a natural fit allowing a connection between the enterprise and embedded industrial world while at the same time playing on Erlang's strengths of reliability and concurrency(a process per subscribed variable anyone?) If anyone has a client I would love to test it against our servers. If there is a server even better.... Tony -------------- next part -------------- An HTML attachment was scrubbed... URL: From bayinamine@REDACTED Wed Aug 21 04:11:18 2013 From: bayinamine@REDACTED (Michael Scofield) Date: Wed, 21 Aug 2013 10:11:18 +0800 Subject: [erlang-questions] Does anyone ever success compiled erlang binding zeromq in Windows? Message-ID: I'm trying to use erlzmq2 . The OS is Windows 7 64 bit, and I have already installed MinGW. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john_re@REDACTED Wed Aug 21 14:58:35 2013 From: john_re@REDACTED (giovanni_re) Date: Wed, 21 Aug 2013 05:58:35 -0700 Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) In-Reply-To: <52131A39.6000803@ninenines.eu> References: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> <52131A39.6000803@ninenines.eu> Message-ID: <1377089915.29836.12415249.593CE4EA@webmail.messagingengine.com> On Tue, Aug 20, 2013, at 12:26 AM, Lo?c Hoguin wrote: > Why do you paste us HN discussions? There's no better waste of time than > to read programmers comparing "sizes". She grew up in an Indiana town, Had a good-lookin' mama who never was around. But she grew up tall and she grew up right, with them Indiana boys on them Indiana nights. http://www.sweetlifeericka.com/2011/09/she-grew-up-in-indiana-town.html > > On 08/20/2013 07:02 AM, giovanni_re wrote: > > https://news.ycombinator.com/item?id=6234736 > > Rewriting a large production system in Go (matt-welsh.blogspot.com) > > 222 points by qdie 1 day ago | 173 comments > > http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html > > > > MW: PhD U. Cali Forni Berkeley > > > > > > > > ========== https://news.ycombinator.com/item?id=6234736 > > derefr 1 day ago | link You know, every time I see some Googler shocked > > at the effectiveness and various advantages of coding in Go, I wonder > > why Google never adopted Erlang. They could have been getting all these > > same advantages (and then some) a decade ago :) ... --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/group/BerkTIPGlobal. All Freedom SW, HW & Culture. From essen@REDACTED Wed Aug 21 15:39:21 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 21 Aug 2013 15:39:21 +0200 Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) In-Reply-To: <1377089915.29836.12415249.593CE4EA@webmail.messagingengine.com> References: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> <52131A39.6000803@ninenines.eu> <1377089915.29836.12415249.593CE4EA@webmail.messagingengine.com> Message-ID: <5214C309.1020805@ninenines.eu> On 08/21/2013 02:58 PM, giovanni_re wrote: > On Tue, Aug 20, 2013, at 12:26 AM, Lo?c Hoguin wrote: >> Why do you paste us HN discussions? There's no better waste of time than >> to read programmers comparing "sizes". > > > She grew up in an Indiana town, > Had a good-lookin' mama who never was around. > But she grew up tall and she grew up right, > with them Indiana boys on them Indiana nights. > > > http://www.sweetlifeericka.com/2011/09/she-grew-up-in-indiana-town.html Yes, watching pictures of people smiling is still a better use of time than watching people "argue" endlessly (quotes because nobody ever gets convinced). At least smiles improve your spirits. Both are spam, though. Spam is bad. Please don't spam. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From robert.virding@REDACTED Wed Aug 21 16:26:53 2013 From: robert.virding@REDACTED (Robert Virding) Date: Wed, 21 Aug 2013 15:26:53 +0100 (BST) Subject: [erlang-questions] (Googl never Erlanged) Rewriting a large production system in Go (matt-welsh.blogspot.com) In-Reply-To: References: <1376974940.23770.11836497.65D56B49@webmail.messagingengine.com> <52131A39.6000803@ninenines.eu> Message-ID: <642616242.10758326.1377095213108.JavaMail.zimbra@erlang-solutions.com> You should have added this answer, it would have led to a very interesting discussion! Robert ----- Original Message ----- > From: "Jesper Louis Andersen" > I remember reading that discussion on Hacker News as for why Google > considered but did not pick Erlang. The reason I did not answer was because > I don't believe it would be fruitful to do so, but on this mailing list I > can't resist. > One of the things I've been looking into as of late is the price paid for > error handling. When you anticipate error and handle it, there is an > overhead to this handling. It is easy to get extreme Queries per second > (QPS) rates if you ignore faults and have no way to mitigate them. > I believe that the stance of many systems are to claim that the system is > void of any internal error and then run without the internal error > protection of the system. Outside, you run something like monit in order to > keep the system up and restarted should it die. In turn, such systems can be > made so they run much faster than the typical Erlang system easily: > * There is no need for doing hot code upgrade on the fly. Hence, you can > native-compile everything into one static binary, inline like mad and so on. > This speeds up a lot of the computational stuff. > * You don't need to set a monitor for each call. Since there are no errors > (by your claim) the code never goes wrong and hence why set it? > * You have no distribution. So everything happens locally in your own memory > space. This usually gives complete knowledge and simplifies a lot of > possible error paths to "non-existent". > To make Go work in practice, you need to adopt this stance: you deploy small > statically compiled binaries which are "obviously correct" by being small. > And if something is amiss, you do not go for a partial failure or a gradual > degradation of the system: off with its head! > It is a trade-off. You get more speed, but are ignoring certain failure > scenarios. But you are also suddenly going to rely 100% on the code being > correct. In your code and in all libraries you use. > On Tue, Aug 20, 2013 at 9:26 AM, Lo?c Hoguin < essen@REDACTED > wrote: > > Why do you paste us HN discussions? There's no better waste of time than to > > read programmers comparing "sizes". > > > On 08/20/2013 07:02 AM, giovanni_re wrote: > > > > https://news.ycombinator.com/ item?id=6234736 > > > > > > Rewriting a large production system in Go ( matt-welsh.blogspot.com ) > > > > > > 222 points by qdie 1 day ago | 173 comments > > > > > > http://matt-welsh.blogspot. com/2013/08/rewriting-large- > > > production-system-in-go.html > > > > > > MW: PhD U. Cali Forni Berkeley > > > > > > ========== https://news.ycombinator.com/ item?id=6234736 > > > > > > derefr 1 day ago | link You know, every time I see some Googler shocked > > > > > > at the effectiveness and various advantages of coding in Go, I wonder > > > > > > why Google never adopted Erlang. They could have been getting all these > > > > > > same advantages (and then some) a decade ago :) > > > > > > reply > > > > > > [GR: Would Googl use an Erlang on its' search engine show?] > > > > > > zaphar 1 day ago | link (full disclosure: I work at google and also like > > > > > > erlang) Erlang has fantastic facilities for robustness and concurrency. > > > > > > What it does not have is type safety and it's terrible at handling text > > > > > > in a performant fashion. So if you don't care about either of those > > > > > > things and only care about robustness and concurrency then Erlang is > > > > > > great. There were internal discussions about Erlang here but the upshot > > > > > > was. We had already basically duplicated Erlangs supervision model in > > > > > > our infrastructure, only we did it for all languages and Erlang didn't > > > > > > offer any benefits in performance for us. It's only benefit would have > > > > > > been the concurrency model. That's much less benefit than Go gives. Go > > > > > > gives you Erlangs concurency model, a similar philosophy of robustness, > > > > > > type safety, all batteries included, and performance. Equating the two > > > > > > languages works in 1 or 2 dimensions but not on all the dimensions > > > > > > google cares about. > > > > > > reply > > > > > > derefr 1 day ago | link Interesting, thanks for that; it's pretty much > > > > > > what I guessed (especially the bit about the supervision tree and > > > > > > hot-code-upgrade advantages being mooted by your infrastructure.) On a > > > > > > tangent, though: > What it does not have is type safety I've tried to > > > > > > work this out before (I'm designing a new language for Erlang's VM), but > > > > > > as far as I can tell, type safety is in practice incompatible with > > > > > > VM-supported hot code upgrade. If you have two services, A and B, and > > > > > > you need to upgrade them both, but you can't "stop the world" to do an > > > > > > atomic upgrade of both A and B together (because you're running a > > > > > > distributed soft real-time system, after all), then you need to switch > > > > > > out A, and then switch out B. So, at some point, on some nodes, A will > > > > > > be running a version with an ABI incompatible with B. In a > > > > > > strongly-typed system, the VM wouldn't allow A's new code to load, since > > > > > > it refers to functions in B with type signatures that don't exist. On > > > > > > the other hand, in a system with pattern-matching and a "let it crash" > > > > > > philosophy, you just let A's new code start up and repeatedly > > > > > > try-and-fail to communicate with B for a while, until B's code gets > > > > > > upgraded as well--and now the types are compatible again. It's an > > > > > > interesting problem. > > > > > > reply > > > > > > laureny 1 day ago | link > type safety is in practice incompatible with > > > > > > VM-supported hot code upgrade. That's not true. First, it's very easy to > > > > > > hot reload changes that have been made to the code that are backward > > > > > > compatible. The JVM spec describes in very specific details what that > > > > > > means (adding or removing a method is not backward compatible, modifying > > > > > > a body is, etc...). This is how Hotswap works, the JVM has been using it > > > > > > for years. As for changes that are backward incompatible, you can still > > > > > > manage them with application level techniques, such as rolling out > > > > > > servers or simply allow two different versions of the class to exist at > > > > > > the same time (JRebel does that, as do other a few other products in the > > > > > > JVM ecosystem). Erlang doesn't really have any advantages over > > > > > > statically typed systems in the hot reload area, and its lack of static > > > > > > typing is a deal breaker for pretty much any serious production > > > > > > deployment. > > > > > > reply > > > > > > rdtsc 20 hours ago | link > lack of static typing is a deal breaker for > > > > > > pretty much any serious production deployment. Are you talking about > > > > > > Google only where they made it a mandate or in general? There are > > > > > > serious production deployments on Python, Ruby, Erlang and Javascript. I > > > > > > will trade expressiveness and less lines of code with a strong but > > > > > > dynamically typed language + tests over more a static typed language > > > > > > with more lines of code all being equal. Or put it another way, if > > > > > > strong typing is the main thing that protects against lack of faults and > > > > > > crashes in production, there is a serious issue that needs to be > > > > > > addressed (just my 2 cents). > > > > > > reply > > > > > > derefr 23 hours ago | link > As for changes that are backward > > > > > > incompatible, you can still manage them with application level > > > > > > techniques, such as rolling out servers or simply allow two different > > > > > > versions of the class to exist at the same time (JRebel does that, as do > > > > > > other a few other products in the JVM ecosystem). Neither of these allow > > > > > > for the whole reason Erlang has hot code upgrade in the first place: > > > > > > allowing to upgrade the code on one side of a TCP connection without > > > > > > dropping the connection to the other side. Tell me how to do that with a > > > > > > static type system :) > > > > > > reply > > > > > > pmahoney 10 hours ago | link Tomcat (and other app servers) has support > > > > > > for doing hot reloads of Java web apps while not reloading the HTTP > > > > > > layer (and not dropping TCP connections). > > > > > > http://www.javacodegeeks.com/ 2011/06/zero-downtime- deploymen. .. I have > > > > > > implemented a similar system for JRuby apps running inside a Servlet > > > > > > container. There are many caveats. I don't actually recommend it because > > > > > > for a while you're using nearly twice the memory (and JRuby is > > > > > > particularly memory hungry). Also there are many ways to leak the old > > > > > > class definitions such that they are not GC'd (e.g. thread locals). But > > > > > > it's certainly possible. I suspect that Erlang, Java, and all languages > > > > > > are in the same boat: some parts can be upgraded live in the VM while > > > > > > other parts require a full restart (maybe coordinating with multiple > > > > > > nodes and a load balancer to achieve zero-downtime). > > > > > > reply > > > > > > lenkite 10 hours ago | link Out of curiosity, where/why would such an > > > > > > exotic feature be needed in today's internet architectures where you > > > > > > always front a group of servers with a load balancer ? > > > > > > reply > > > > > > butterflyhug 5 hours ago | link Not all Internet protocols are HTTP. If > > > > > > you're running a service where long-lived connections are the norm, > > > > > > "simply fronting a bunch of servers with a load balancer" can require a > > > > > > pretty smart load balancer. E.g. IMAP connections often last hours or > > > > > > even days, and are required to maintain a degree of statefulness. > > > > > > reply > > > > > > DanWaterworth 1 day ago | link Go gives you Erlangs concurency model > > > > > > There are a number of significant differences between Erlang's and Go's > > > > > > concurrency models: Asynchronous vs synchronous communication, > > > > > > per-thread vs per-process heaps, send to process vs send to channel. > > > > > > reply > > > > > > dbaupp 1 day ago | link Go doesn't have strong type safety either; I > > > > > > remember a recent story about a Go stdlib function "accidentally" > > > > > > calling an interface it shouldn't. > > > > > > reply > > > > > > f2f 22 hours ago | link it wasn't accidental -- it was written on > > > > > > purpose by a programmer (a conversion from Writer to WriteCloser). it > > > > > > was immediately acknowledged as an error and eventually may be caught by > > > > > > the standard code examining tool "vet". > > > > > > reply > > > > > > pcwalton 22 hours ago | link What would the static analysis that "vet" > > > > > > is performing enforce to stop this? No interface-to-interface downcasts? > > > > > > reply > > > > > > ..... > > > > > > [It goes on, & on, & on & on, > > > > > > when Go & Erlang party together.] > > > > > > --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/ > > > group/BerkTIPGlobal . All Freedom SW, HW & Culture. > > > > > > ______________________________ _________________ > > > > > > erlang-questions mailing list > > > > > > erlang-questions@REDACTED > > > > > > http://erlang.org/mailman/ listinfo/erlang-questions > > > > > -- > > > Lo?c Hoguin > > > Erlang Cowboy > > > Nine Nines > > > http://ninenines.eu > > > ______________________________ _________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/ listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From john_re@REDACTED Wed Aug 21 16:42:17 2013 From: john_re@REDACTED (giovanni_re) Date: Wed, 21 Aug 2013 07:42:17 -0700 Subject: [erlang-questions] =?utf-8?q?Lo=C3=AFc_Poet?= References: <1377095796.21987.12460801.20069E96@webmail.messagingengine.com> Message-ID: <1377096137.23188.12462417.5E8721DF@webmail.messagingengine.com> > On Tue, Aug 20, 2013, at 12:26 AM, Lo?c Hoguin wrote: > > Why do you paste us HN discussions? > There's no better waste of time than > to read programmers comparing "sizes". > > > Yes, watching pictures of people smiling is still a better use of time > than watching people "argue" endlessly (quotes because nobody ever gets convinced). > At least smiles improve your spirits. > Both are spam, though. Spam is bad. Please don't spam. Well, I don't know, but I've been told You never slow down, you never grow old Buy me a drink, sing me a song Take me as I come 'cause I can't stay long --- Join the BerkeleyTIP-Global mail list - http://groups.google.com/group/BerkTIPGlobal. All Freedom SW, HW & Culture. From co7eb@REDACTED Wed Aug 21 18:15:04 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Wed, 21 Aug 2013 12:15:04 -0400 Subject: [erlang-questions] Still looking for an Erlang Job (please readme) Message-ID: <000601ce9e89$9e2b2960$da817c20$@frcuba.co.cu> Hi all, I?m still looking for a job in the community, because I like Erlang and I need to work. I have been trying signing in the Erlang jobs websites but I have no luck since I?m from Cuba, but I think that shouldn?t be a problem, but still I have no luck in finding Erlang jobs, not even one of it, I just want one to be my first. I just need the opportunity to learn more Erlang and to demonstrate my intelligence, I think I?m not so bad comparing my code with your written in the Erlang OTP Libraries. Down is one example of the code I?m writing right now. Sorry if bother you, I don?t look for any trouble just a little bit of help and if you think I don?t deserve then I will have no question back. %% %% %CopyrightBegin% %% %% Copyright X Software Corporation 2014. All Rights Reserved. %% %% The contents of this file are subject to the Erlang Public License, %% Version 1.1, (the "License"); you may not use this file except in %% compliance with the License. You should have received a copy of the %% Erlang Public License along with this software. If not, it can be %% retrieved online at http://www.erlang.org/. %% %% Software distributed under the License is distributed on an "AS IS" %% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See %% the License for the specific language governing rights and limitations %% under the License. %% %% %CopyrightEnd% %% %%%------------------------------------------------------------------------- --- %%% @author Ivan Carmenates Garc?a %%% @copyright X Software Corporation 2014 %%% @version 1.0 %%% @doc %%% CountLink start point module. %%% %%% This module contains the basic functions to install, start, stop, configure %%% and update the CountLink System. %%% @end %%%------------------------------------------------------------------------- --- %%% Module attributes definition. -module(countlink). -behavior(application). -vsn(1.0). -author("Ivan Carmenates Garc?a"). -description("CountLink start point module"). %%% Functions importing (module, [function/arity[, ...]]). -import(error_logger, [info_msg/2]). %%% Functions exporting ([function/arity[, ...]]). -export([install_db/0, start_system/0, stop_system/0, start/2, stop/1]). %%%------------------------------------------------------------------------- --- %%% @doc %%% Installs the CountLink System database. %%% @end %%%------------------------------------------------------------------------- --- -spec install_db() -> ok | {error, Reason} when Reason :: term(). install_db() -> info_msg(" *** Application: \"~p\" is installing the database...~n", [?MODULE]), timer:sleep(1000), % some nice waiting. %% Installs the CountLink System database. countlink_db:install(). %%%------------------------------------------------------------------------- --- %%% @doc %%% Starts the CountLink System. %%% @end %%%------------------------------------------------------------------------- --- -spec start_system() -> ok | {error, Reason} when Reason :: term(). start_system() -> application:start(?MODULE). %%%------------------------------------------------------------------------- --- %%% @doc %%% Stops the CountLink System. %%% @end %%%------------------------------------------------------------------------- --- -spec stop_system() -> ok | {error, Reason} when Reason :: term(). stop_system() -> application:stop(?MODULE). %%%------------------------------------------------------------------------- --- %%% @doc %%% Start point of the application. %%% %%% This function is automatically called when the application is started. %%% @end %%%------------------------------------------------------------------------- --- -spec start(Type, StartArgs) -> Return when Type :: term(), % @see application module for start_type() type. StartArgs :: term(), Return :: {ok, pid()} | {ok, pid(), State} | {error, Reason}, State :: term, Reason :: term(). start(Type, StartArgs) -> %% Starts the sasl application. %% TODO 1: put the sasl application start in the application specification and %% use application:ensure_all_started/1 instead of application:start/1, %% this require of the new Erlang version R16B04. application:start(sasl), %% Retrieves the node name. Node = erlang:atom_to_list(node()), %% Logs the application starting information. info_msg(" *** Application: \"~p\" is starting...~n" " Type: ~p StartArgs: ~p~n" " NodeName: ~p~n", [?MODULE, Type, StartArgs, Node]), %% Tries to start the main supervisor. case countlink_sup:start_link(StartArgs) of {ok, Pid} -> {ok, Pid, ready}; Error -> Error end. %%%------------------------------------------------------------------------- --- %%% @doc %%% This function is automatically called when the application is stopped. %%% @end %%%------------------------------------------------------------------------- --- -spec stop(State) -> void when State :: term(). stop(State) -> %% Logs the application stopping information. info_msg(" *** Application \"~p\" is stopped.~n" " Current State: ~p~n", [?MODULE, State]), %% TODO 3: Do the necessary clean up and application stopping like sasl. void. Best Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Wed Aug 21 19:04:25 2013 From: dmercer@REDACTED (David Mercer) Date: Wed, 21 Aug 2013 12:04:25 -0500 Subject: [erlang-questions] [99s-extend] How to broadcaset with ranch? In-Reply-To: <20130821033819.GT19389@hijacked.us> References: <520F3001.5010504@ninenines.eu> <20130821033819.GT19389@hijacked.us> Message-ID: <007401ce9e90$7dd47d40$797d77c0$@gmail.com> On Tuesday, August 20, 2013, Andrew Thompson wrote: > On Tue, Aug 20, 2013 at 10:18:19PM -0500, Sean Cribbs wrote: > > This is exactly the sort of thing gen_event is for. I would make each > > server process register a handler at startup using > > gen_event:add_sup_handler() and then have the handle_event callback > > simply relay the event to the server processes. Yes, gproc can do > > this, but why incur its extra features and overhead? > > pg2 also works fairly well. Can someone summarize the advantages/disadvantages of each: gproc, gen_event, and pg2? Cheers, DBM From mjtruog@REDACTED Wed Aug 21 19:29:28 2013 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 21 Aug 2013 10:29:28 -0700 Subject: [erlang-questions] [99s-extend] How to broadcaset with ranch? In-Reply-To: <007401ce9e90$7dd47d40$797d77c0$@gmail.com> References: <520F3001.5010504@ninenines.eu> <20130821033819.GT19389@hijacked.us> <007401ce9e90$7dd47d40$797d77c0$@gmail.com> Message-ID: <5214F8F8.1020103@gmail.com> On 08/21/2013 10:04 AM, David Mercer wrote: > On Tuesday, August 20, 2013, Andrew Thompson wrote: > >> On Tue, Aug 20, 2013 at 10:18:19PM -0500, Sean Cribbs wrote: >>> This is exactly the sort of thing gen_event is for. I would make each >>> server process register a handler at startup using >>> gen_event:add_sup_handler() and then have the handle_event callback >>> simply relay the event to the server processes. Yes, gproc can do >>> this, but why incur its extra features and overhead? >> pg2 also works fairly well. > Can someone summarize the advantages/disadvantages of each: gproc, gen_event, and pg2? > > Cheers, > > DBM > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > gproc uses gen_leader for a CA-type (from CAP theorem) process lookup that has trouble with partition tolerance due to usage of gen_leader gen_event is just an OTP behaviour, so it would be using the normal process registration to create a single process instance pg2 is a AP-type (from CAP theorem) process lookup you can use gproc with gen_event, if you choose, using the "via" name registration to rely on an external module pg2 doesn't support the via name registration, however, a similar process group installation (similar to pg2) is in cpg (https://github.com/okeuday/cpg/) which does support via name registration, so you could use cpg with gen_event So, your choice depends on whether you choose to use distributed erlang, and if you do choose to use distributed erlang, whether you care about partition tolerance or whether you care about consistency. From yash_ganthe@REDACTED Thu Aug 22 15:10:31 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 22 Aug 2013 13:10:31 +0000 Subject: [erlang-questions] Monitoring distributed nodes Message-ID: Hi, We have a Master erlang node that has an application with a supervisor and multiple, dynamically added worker processes. For each worker process, there is another erlang node dynamically started. We would like to monitor all nodes on one screen and detect failures so that corrective action can be taken. Is there an utility that can let us do this? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yoursurrogategod@REDACTED Thu Aug 22 16:35:21 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Thu, 22 Aug 2013 10:35:21 -0400 Subject: [erlang-questions] Erlang's garbage collector Message-ID: Hi, I'm reading the O'Reilly book on Erlang and I found this paragraph in it: The garbage collector is also generational, meaning that it has several generations of the heap (in Erlang?s case, two). A garbage collection can be shallow or deep. A shallow garbage collection looks only at data in the youngest generation; all data that survives three shallow garbage collections will be moved to the old generation. A deep garbage collection will occur only when a shallow collection fails to recycle enough memory or after a (VM version dependent) number of shallow collections. Am I understanding this correctly? You have two heaps and if a piece of memory keeps getting referenced, it gets moved into the "old" heap and the "new" heap's memory gets freed up a little, yes? -------------- next part -------------- An HTML attachment was scrubbed... URL: From co7eb@REDACTED Thu Aug 22 18:29:45 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Thu, 22 Aug 2013 12:29:45 -0400 Subject: [erlang-questions] de Ivan (amigo de Alexander) In-Reply-To: References: <001401ce9f4c$2762a470$7627ed50$@frcuba.co.cu> Message-ID: <001b01ce9f54$d4c25d60$7e471820$@frcuba.co.cu> Kb man, todo good, el Erlang s?per, la pincha precisamente si me la dan va a ser de Erlang con Java y C++ en Linux. Ahora toy tirando algunos codiguitos en Erlang en un proyecto casi personal ah?, pero me hac?a falta tambi?n un trabajo para ir tirando. Gary siempre est? peleando, cuando vino hace poco a darle la vuelta a la casa, estaba con un ojo malo y protestando jeje. Est? bien ya estoy m?s o menos informado. El tipo me respondi? ahorita y me dijo que por transferencia bancaria puede servir, as? que esa misma va a ser, el banco me queda a dos cuadras. Bueno saludos, me alegra saber que estas bien (aunque sea hablas) jeje, y ya sabes este es mi correo, cualquier cosa de Erlang o de lo que sea que te haga falta me dices por aqu?. Si te interesan trabajos de erlang, hay unos websites donde ponen, tambi?n tienes que registrarte en la comunidad para estar al d?a . http://erlang.org/mailman/listinfo/erlang-questions aqu? revisa a ver si por ah? puedes entrar a la lista de distribuci?n de la comunidad. El correo es erlang-questions@REDACTED para enviar pero eso es despu?s de que te subscribas. Y estos son los sitios de jobs https://erlangcentral.org/jobs/ http://totally-erlang.com/ Y est?n bastante buenos los payments, 150 000 y cositas as?. Saludos. Best Regards, Ivan. -----Mensaje original----- De: Karel ?lvarez [mailto:karelium@REDACTED] Enviado el: jueves, 22 de agosto de 2013 11:53 Para: Ivan Carmenates Garc?a Asunto: Re: de Ivan (amigo de Alexander) Hola Iv?n, s?, te recuerdo... con Gary, peleando y berreao jaja bueno, visa y mastercard son americanas ambas, as? que all? no hay nada de eso. Cuando la gente de otros pa?ses las usan en cuba, les cobran una cuota extra. Yo usaba all? una AIS. Se solicita online (http://www.aisremesascuba.com/) o en una oficina de fincimex. Averigua si en guaimaro hay, sino, en camaguey en la calle gral gomez muy cerca de la tienda el encanto, la oficina queda justo en frente de un fotoservice. Hay otros tipos de cards, unas reciben de mas paises pero cobran mas, etc etc. Pregunta en esa oficina. Tal vez la mejor via es que te env?en directo a una sucursal bancaria. La que te quede mas cerca de tu casa, pregunta como hace alguien de x pa?s para enviarte. No es necesario tener cuenta bancaria, solo tu nombre y los datos de la sucursal. Si es de usa, mx y otros que no se, por western union. Eso llega casi al otro dia y cobran bastante poco. Te pongo todas esas variantes porque la de la tarjeta es la mas comoda para uno pero la mas cara de todas. Saludos, como va el erlang? jaja 2013/8/22 Ivan Carmenates Garc?a : > Hola Karel, soy un amigo de alex, que trabajamos junto pal sistema de > Gary, como esta todo por all?, quisiera saber c?mo se llamaban las > tarjetas de pago que se sacan en Camag?ey, tipo visa y MasterCard, y > de que pa?ses son factibles que me puedan enviar por ah?. > > > > Bueno saludos. > > Ivan. From robert.virding@REDACTED Thu Aug 22 18:41:22 2013 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 22 Aug 2013 17:41:22 +0100 (BST) Subject: [erlang-questions] Core erlang wildcards and sequences In-Reply-To: References: Message-ID: <1307148587.11663103.1377189682223.JavaMail.zimbra@erlang-solutions.com> While the Erlang compiler stacks let this is not really necessary as Core does supports nested calls. It is really not defined in which order the arguments will be evaluated, which is one reason why Erlang does the explicit nested lets. In one sense there it makes no difference as the resultant code will be the same, however the optimisation passes of the compiler more or less assume the flattened nested form. There is one thing missing in Core and that is non-recursive local function definitions. Using letrec defines a recursive function so if you want to reuse local function names as local to a block you can't, you have to invent a new name each time. Almost trivial but still annoying. Robert ----- Original Message ----- > From: "Ludovic Demblans" > To: erlang-questions@REDACTED > Sent: Saturday, 3 August, 2013 3:50:36 PM > Subject: Re: [erlang-questions] Core erlang wildcards and sequences > > Ok so i'll continue to stack let expressions, it seems simple. > > Thanks for the replies, > > Ludovic > > > > > Le Thu, 01 Aug 2013 15:57:30 +0200, Anthony Ramine a > ?crit: > > > Hello, > > > > Replied inline. > > > > > -- > Utilisant le logiciel de courrier r?volutionnaire d'Opera : > http://www.opera.com/mail/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From icfp.publicity@REDACTED Thu Aug 22 19:54:23 2013 From: icfp.publicity@REDACTED (David Van Horn) Date: Thu, 22 Aug 2013 13:54:23 -0400 Subject: [erlang-questions] ICFP 2013 Call for Participation Message-ID: [ Today is the last day of early registration. ] ============================== ================================== CALL FOR PARTICIPATION 18th ACM SIGPLAN International Conference on Functional Programming ICFP 2013 Boston, MA, USA, 25-27 September 2013 Affiliated events 22-24 September, 28 September 2013 http://www.icfpconference.org/icfp2013 ================================================================ ICFP is a forum for researchers and practitioners to hear about the latest developments in the art and science of functional programming. The conference cover the entire spectrum of work, from theory to practice. The keynote speakers of ICFP 2013 are Ulf Norell on dependently typed programming and Simon Peyton Jones on computer science as a school subject. The 40 contributed papers on all aspects of functional programming were selected by the program committee from 133 submissions. Affiliated to ICFP 2013 are: ACM SIGPLAN Commercial Users of Functional Programming Workshop ACM SIGPLAN Workshop on Dependently Typed Programming ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design ACM SIGPLAN Workshop on Functional High-Performance Computing ACM SIGPLAN Erlang Workshop ACM SIGPLAN Functional Programming in Domain-Specific Languages Workshop ACM SIGPLAN Workshop on Generic Programming ACM SIGPLAN Haskell Symposium ACM SIGPLAN Haskell Implementors Workshop ACM SIGPLAN Workshop on Higher-Order Programming with Effects ACM SIGPLAN International Workshop on Logical Frameworks and Meta-Languages: Theory and Practice ACM SIGPLAN Workshop on ML ACM SIGPLAN OCaml Workshop VENUE The conference will take place at Hilton Boston Logan Airport Hotel. REGISTRATION Registration is available at https://regmaster3.com/2013conf/ICFP13/register.php *Note that early registration ends 22 August 2013.* SUPPORT FROM SIGPLAN PAC The SIGPLAN Professional Activities Committee can potentially provide supplemental financial support for attending ICFP in the form of student grants, child-care support, companion travel grants, and long-distance internation travel support. To apply, follow the instructions found here: http://www.sigplan.org/PAC.htm ICFP 2013 PROGRAMME Wednesday, 25 September 9-10 Keynote 1 Interactive Programming with Dependent Types Ulf Norell (Chalmers University of Technology) 10-1020 Break 1020-1100 Session 1: Verification with Grammars and Automata Verified Decision Procedures for MSO on Words Based on Derivatives of Regular Expressions [Functional Pearl] Dmitriy Traytel and Tobias Nipkow C-SHORe Christopher Broadbent, Arnaud Carayol, Matthew Hague and Olivier Serre 1100-1130 Break 1130-1230 Session 2: Data Parallelism Automatic SIMD Vectorization for Haskell Leaf Petersen, Dominic Orchard and Neal Glew Exploiting Vector Instructions with Generalized Stream Fusion Geoffrey Mainland, Roman Leshchinskiy and Simon Peyton Jones Optimising Purely Functional GPU Programs Trevor L. McDonell, Manuel Chakravarty, Gabriele Keller and Ben Lippmeier 1230-14 Lunch 14-15 Session 3: Dependent Types Type-Theory In Color Jean-Philippe Bernardy and Guilhem Moulin Typed Syntactic Meta-programming Dominique Devriese and Frank Piessens Mtac: A Monad for Typed Tactic Programming in Coq Beta Ziliani, Derek Dreyer, Neelakantan Krishnaswami, Aleksandar Nanevski and Viktor Vafeiadis 15-1530 Break 1530-1630 Session 4: Fun in the Afternoon Fun with Semirings [Functional Pearl] Stephen Dolan Efficient Divide-and-Conquer Parsing of Practical Context-Free Languages Jean-Philippe Bernardy and Koen Claessen Functional Geometry and the "Trait? de Lutherie" [Functional Pearl] Harry Mairson 1630-17 Break 17-1740 Session 5: Handling Effects Programming and Reasoning with Algebraic Effects and Dependent Types Edwin Brady Handlers in Action Ohad Kammar, Sam Lindley and Nicolas Oury 1740-18 Program Chair's Report Thursday, 26 September 9-10 Keynote 2 Computer Science as a School Subject Simon Peyton Jones (Microsoft Research, Cambridge, UK) 10-1020 Break 1020-11 Session 6: Concurrency Correctness of an STM Haskell Implementation Manfred Schmidt-Schauss and David Sabel Programming with Permissions in Mezzo Francois Pottier and Jonathan Protzenko 11-1130 Break 1130-1230 Session 7: (Co-)Recursion Wellfounded Recursion with Copatterns Andreas Abel and Brigitte Pientka Productive Coprogramming with Guarded Recursion Robert Atkey and Conor McBride Unifying Structured Recursion Schemes Ralf Hinze, Nicolas Wu and Jeremy Gibbons 1230-14 Lunch 14-15 Session 8: Functional Reactive Programming (and More) Higher-Order Functional Reactive Programming without Spacetime Leaks Neelakantan Krishnaswami Functional Reactive Programming with Liveness Guarantees Alan Jeffrey A Short Cut to Parallelization Theorems Akimasa Morihata 15-1530 Break 1530-1630 Session 9: Lambda-Calculus Using Circular Programs for Higher-Order Syntax [Functional Pearl] Emil Axelsson and Koen Claessen Weak Optimality, and the Meaning of Sharing Thibaut Balabonski System FC with Explicit Kind Equality Stephanie Weirich, Justin Hsu and Richard A. Eisenberg 1630-17 Break 17-1740 Programming Contest Co-Chairs' Report and Awards 1740-18 ICFP 2003 Most Influential Paper Award Friday, 27 September 9-10 Session 10: Monads The Constrained-Monad Problem Neil Sculthorpe, Jan Bracker, George Giorgidze and Andy Gill Simple and Compositional Reification of Monadic Embedded Languages [Functional Pearl] Josef Svenningsson and Bo Joel Svensson Structural Recursion for Querying Ordered Graphs Soichiro Hidaka, Kazuyuki Asada, Zhenjiang Hu, Hiroyuki Kato and Keisuke Nakano 10-1020 Break 1020-11 Session 11: Modular Meta-Theory Modular Monadic Meta-Theory Benjamin Delaware, Steven Keuchel, Tom Schrijvers and Bruno Oliveira Modular and Automated Type-Soundness Verification for Language Extensions Florian Lorenzen and Sebastian Erdweg 11-1130 Break 1130-1230 Session 12: Experience Reports (Chair: John Launchbury) A Nanopass Framework for Commercial Compiler Development [Experience Report] Andrew Keep and R Kent Dybvig Applying Random Testing to a Base Type Environment [Experience Report] Vincent St-Amour and Neil Toronto Functional Programming of mHealth Applications [Experience Report] Christian Petersen, Matthias Gorges, Dustin Dunsmuir, Mark Ansermino and Guy Dumont 1230-14 Lunch 14-15 Session 13: Program Logics Hoare-Style Reasoning with (Algebraic) Continuations Germ?n Andr?s Delbianco and Aleksandar Nanevski Unifying Refinement and Hoare-Style Reasoning in a Logic for Higher-Order Concurrency Aaron Turon, Derek Dreyer and Lars Birkedal The Bedrock Structured Programming System Adam Chlipala 15-1530 Break 1530-1630 Session 14: Language Design A Practical Theory of Language-Integrated Query James Cheney, Sam Lindley and Philip Wadler Calculating Threesomes, with Blame Ronald Garcia Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism Joshua Dunfield and Neelakantan R. Krishnaswami 1630-17 Break 17-1740 Session 15: Analysis and Optimization Optimizing Abstract Abstract Machines J. Ian Johnson, Nicholas Labich, Matthew Might and David Van Horn Testing Noninterference, Quickly Catalin Hritcu, John Hughes, Benjamin C. Pierce, Antal Spector-Zabusky, Dimitrios Vytiniotis, Arthur Azevedo de Amorim and Leonidas Lampropoulos 1740-18 ICFP 2014 Advert & Closing GENERAL CHAIR Greg Morrisett, Harvard University PROGRAM CHAIR Tarmo Uustalu, Institute of Cybernetics, Tallinn PROGRAM COMMITTEE Thorsten Altenkirch, University of Nottingham Olaf Chitil, University of Kent Silvia Ghilezan, University of Novi Sad Michael Hanus, Christian-Albrechts-Universit?t zu Kiel Fritz Henglein, University of Copenhagen Mauro Jaskelioff, Universidad Nacional de Rosario Alan Jeffrey, Alcatel-Lucent Bell Labs Shin-ya Katsumata, Kyoto University Shriram Krishnamurthi Brown University John Launchbury, Galois Ryan Newton, Indiana University Sungwoo Park, Pohang University of Science and Technology Sam Staton, University of Cambridge Nikhil Swamy, Microsoft Research, Redmond, WA Dimitrios Vytiniotis, Microsoft Research, Cambridge From jesper.louis.andersen@REDACTED Thu Aug 22 23:04:36 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 22 Aug 2013 23:04:36 +0200 Subject: [erlang-questions] Erlang's garbage collector In-Reply-To: References: Message-ID: On Thu, Aug 22, 2013 at 4:35 PM, Yves S. Garret wrote: > Am I understanding this correctly? You have two heaps and if a piece of > memory keeps getting > referenced, it gets moved into the "old" heap and the "new" heap's memory > gets freed up a little, > yes? > Well the heap is divided in two regions, old and young. New data is allocated on the young heap. When it fills, the young heap is garbage collected. Data which is dead is removed and data which is still live survives on the young heap. When it has survived enough times (3) then it is promoted to the old heap. The reason for this scheme is that you keep the young heap small and collect it often. Collection is faster in a small heap. Also, since most objects tend to be used only once at most, they are usually dead when the young heap collects. Objects which survive are usually long lived, so they get promoted into the old heap. It is then collected more rarely. The view here is simplified, but it might do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Fri Aug 23 14:13:23 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 23 Aug 2013 14:13:23 +0200 Subject: [erlang-questions] compile: making asm and core official Message-ID: Motivated by a discussion at https://github.com/rebar/rebar/issues/105 and Bjorn-Egil's suggestion, I'd like to ask for opinions on officially supporting 'core' and 'asm' as compile:file/2 options. (1) How likely are you to accept patches which would: * Implement support for compile:file(File, [core]) same as compile:file(File, [asm]). * Officially document 'core' and 'asm' as external names for 'from_asm' and 'from_core'? * Change the existing documentation for 'asm' to not discourage use of the option as much. * Officially document that "erlc foo.core" and "erlc foo.S" have been wired to from_core and from_asm for ages? (2) Document compile_core/3 and compile_asm/3 Alternatively, one could call compile:compile_asm/3 and compile:compile_core/3, but they're internal functions meant to be used only from erl_compile (used by erlc). This would actually be the most backwards compatible solution if we don't want to require a patched compile.beam. So, what about alternatively or additionally documenting compile_core/3 and compile_asm/3? From raimo+erlang-questions@REDACTED Fri Aug 23 12:07:48 2013 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Fri, 23 Aug 2013 12:07:48 +0200 Subject: [erlang-questions] de Ivan (amigo de Alexander) In-Reply-To: <001b01ce9f54$d4c25d60$7e471820$@frcuba.co.cu> References: <001401ce9f4c$2762a470$7627ed50$@frcuba.co.cu> <001b01ce9f54$d4c25d60$7e471820$@frcuba.co.cu> Message-ID: <20130823100748.GA12125@erix.ericsson.se> Plees use english on this list. / Raimo Niskanen, Mailing list admin at erlang dot org On Thu, Aug 22, 2013 at 12:29:45PM -0400, Ivan Carmenates Garc?a wrote: > Kb man, todo good, el Erlang s?per, la pincha precisamente si me la dan va a > ser de Erlang con Java y C++ en Linux. Ahora toy tirando algunos codiguitos > en Erlang en un proyecto casi personal ah?, pero me hac?a falta tambi?n un > trabajo para ir tirando. Gary siempre est? peleando, cuando vino hace poco a > darle la vuelta a la casa, estaba con un ojo malo y protestando jeje. > Est? bien ya estoy m?s o menos informado. El tipo me respondi? ahorita y me > dijo que por transferencia bancaria puede servir, as? que esa misma va a > ser, el banco me queda a dos cuadras. > > Bueno saludos, me alegra saber que estas bien (aunque sea hablas) jeje, y ya > sabes este es mi correo, cualquier cosa de Erlang o de lo que sea que te > haga falta me dices por aqu?. > Si te interesan trabajos de erlang, hay unos websites donde ponen, tambi?n > tienes que registrarte en la comunidad para estar al d?a . > > http://erlang.org/mailman/listinfo/erlang-questions aqu? revisa a ver si por > ah? puedes entrar a la lista de distribuci?n de la comunidad. > El correo es erlang-questions@REDACTED para enviar pero eso es despu?s de > que te subscribas. > > Y estos son los sitios de jobs > > https://erlangcentral.org/jobs/ > http://totally-erlang.com/ > > Y est?n bastante buenos los payments, 150 000 y cositas as?. > Saludos. > > > Best Regards, > Ivan. > > > > -----Mensaje original----- > De: Karel ?lvarez [mailto:karelium@REDACTED] > Enviado el: jueves, 22 de agosto de 2013 11:53 > Para: Ivan Carmenates Garc?a > Asunto: Re: de Ivan (amigo de Alexander) > > Hola Iv?n, s?, te recuerdo... con Gary, peleando y berreao jaja > > bueno, visa y mastercard son americanas ambas, as? que all? no hay nada de > eso. Cuando la gente de otros pa?ses las usan en cuba, les cobran una cuota > extra. > > Yo usaba all? una AIS. Se solicita online > (http://www.aisremesascuba.com/) o en una oficina de fincimex. > Averigua si en guaimaro hay, sino, en camaguey en la calle gral gomez muy > cerca de la tienda el encanto, la oficina queda justo en frente de un > fotoservice. Hay otros tipos de cards, unas reciben de mas paises pero > cobran mas, etc etc. Pregunta en esa oficina. > > Tal vez la mejor via es que te env?en directo a una sucursal bancaria. > La que te quede mas cerca de tu casa, pregunta como hace alguien de x pa?s > para enviarte. No es necesario tener cuenta bancaria, solo tu nombre y los > datos de la sucursal. > > Si es de usa, mx y otros que no se, por western union. Eso llega casi al > otro dia y cobran bastante poco. > > Te pongo todas esas variantes porque la de la tarjeta es la mas comoda para > uno pero la mas cara de todas. > > Saludos, como va el erlang? jaja > > > > 2013/8/22 Ivan Carmenates Garc?a : > > Hola Karel, soy un amigo de alex, que trabajamos junto pal sistema de > > Gary, como esta todo por all?, quisiera saber c?mo se llamaban las > > tarjetas de pago que se sacan en Camag?ey, tipo visa y MasterCard, y > > de que pa?ses son factibles que me puedan enviar por ah?. > > > > > > > > Bueno saludos. > > > > Ivan. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From sasa.juric@REDACTED Fri Aug 23 22:05:21 2013 From: sasa.juric@REDACTED (Sasa Juric) Date: Fri, 23 Aug 2013 22:05:21 +0200 Subject: [erlang-questions] Erlang's garbage collector In-Reply-To: References: Message-ID: <0D78D842-FC0E-433B-8C72-B9F685F82829@gmail.com> Is there a document (other than the source code) where one can find out about the GC internals. Not low level details, but more a general characteristics. In particular, I'm interested how Erlang GC profits from memory isolation. I've seen claims that it works on a process level. Does it mean it runs separately for each scheduler (allowing other schedulers to work)? Since processes are isolated, is a process memory immediately reclaimed after the process terminates? On Aug 22, 2013, at 11:04 PM, Jesper Louis Andersen wrote: > > On Thu, Aug 22, 2013 at 4:35 PM, Yves S. Garret wrote: > Am I understanding this correctly? You have two heaps and if a piece of memory keeps getting > referenced, it gets moved into the "old" heap and the "new" heap's memory gets freed up a little, > yes? > > Well the heap is divided in two regions, old and young. New data is allocated on the young heap. When it fills, the young heap is garbage collected. Data which is dead is removed and data which is still live survives on the young heap. When it has survived enough times (3) then it is promoted to the old heap. > > The reason for this scheme is that you keep the young heap small and collect it often. Collection is faster in a small heap. Also, since most objects tend to be used only once at most, they are usually dead when the young heap collects. Objects which survive are usually long lived, so they get promoted into the old heap. It is then collected more rarely. > > The view here is simplified, but it might do. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Fri Aug 23 23:51:34 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Fri, 23 Aug 2013 23:51:34 +0200 Subject: [erlang-questions] mnesia:write slow down Message-ID: <004001cea04a$f02629a0$d0727ce0$@gmail.com> All, My application uses mnesia for data storage. I saw a very quick performance degradation as data started to build up. I wrote some simple test cases and measured the write performance. I inserted 1.3million records and compared the times needed to insert the first and last 10000. The best I could reach was 22% time increase, but in more lifelike situations it reached 73%. I used only one process to avoid locking problems. Finally I found the possible reason: my record looks like #img{dbid, ...}, where dbid is a continuous counter. If I change dbid to a random number the insert time for the first and last records are nearly the same (actually it varies in a range but there is no systematic slow down). This problem seems to affect fragmented tables even worst. Is it a known problem? What shall I do if the continuous dbid is an important part of my design? Shall I convert it to binary and prepend some salt? I'm happy to provide my test code or time measurements if it helps finding the cause, but please note, I'm not concerned about the overall performance, I just can't release the app if it is slowing down so fast, as the requirement is to handle many more records. Thanks for your time in advance. Janos -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sat Aug 24 00:07:33 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sat, 24 Aug 2013 00:07:33 +0200 Subject: [erlang-questions] Erlang's garbage collector In-Reply-To: <0D78D842-FC0E-433B-8C72-B9F685F82829@gmail.com> References: <0D78D842-FC0E-433B-8C72-B9F685F82829@gmail.com> Message-ID: Den 23/08/2013 23.23 skrev "Sasa Juric" : > > Is there a document (other than the source code) where one can find out about the GC internals. Not low level details, but more a general characteristics. > In particular, I'm interested how Erlang GC profits from memory isolation. I've seen claims that it works on a process level. Does it mean it runs separately for each scheduler (allowing other schedulers to work)? Yes. It runs as part of a process's timeslice - when the process runs out of free heap space. Other schedulers won't be doing anything with the heap of the process anyway, and can carry on without even being notified of the GC taking place. Isolation is indeed beneficial here. > Since processes are isolated, is a process memory immediately reclaimed after the process terminates? Yes. Heap and stack memory, the message queue, and other resources such as ports and tables are released when the process disappears. (It's not atomic, of course - a process can afaik be "terminated" for a short while before it's also "cleaned up after", but that's typically not of importance.) > On Aug 22, 2013, at 11:04 PM, Jesper Louis Andersen wrote: > >> >> On Thu, Aug 22, 2013 at 4:35 PM, Yves S. Garret < yoursurrogategod@REDACTED> wrote: >>> >>> Am I understanding this correctly? You have two heaps and if a piece of memory keeps getting >>> referenced, it gets moved into the "old" heap and the "new" heap's memory gets freed up a little, >>> yes? >> >> >> Well the heap is divided in two regions, old and young. New data is allocated on the young heap. When it fills, the young heap is garbage collected. Data which is dead is removed and data which is still live survives on the young heap. When it has survived enough times (3) then it is promoted to the old heap. >> >> The reason for this scheme is that you keep the young heap small and collect it often. Collection is faster in a small heap. Also, since most objects tend to be used only once at most, they are usually dead when the young heap collects. Objects which survive are usually long lived, so they get promoted into the old heap. It is then collected more rarely. >> >> The view here is simplified, but it might do. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Aug 24 16:42:29 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 24 Aug 2013 18:42:29 +0400 Subject: [erlang-questions] lager and dynamic log cutting via compiled module Message-ID: log4erl logging library has an interesting feature: on loading config it generates module that cuts all log requests that are too verbose on caller side without any ets lookups or calls. lager instead makes ets call to find out current loglevel. I can't say that it is a bad approach, but when my traffic is growing upper 8 Gbit/s from single node, I have to calculate each microsecond. I suppose that thousands of requests per second to single row in ets table for finding out whether to log or not to log may be more blocking than executing precompiled erlang code. So, question: why approach with compiled on-fly module is not used? Just because ets is more standard? Or "shut up and make pull request"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Aug 24 17:05:36 2013 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 24 Aug 2013 17:05:36 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: Message-ID: The Feuerlabs fork of lager does basically this. https://github.com/Feuerlabs/lager/compare/basho:master...master I don't think it ever ended up as a pull request, though. BR, Ulf On 24 Aug 2013, at 16:42, Max Lapshin wrote: > log4erl logging library has an interesting feature: on loading config it generates module that cuts all log requests that are too verbose on caller side without any ets lookups or calls. > > lager instead makes ets call to find out current loglevel. I can't say that it is a bad approach, but when my traffic is growing upper 8 Gbit/s from single node, I have to calculate each microsecond. > > I suppose that thousands of requests per second to single row in ets table for finding out whether to log or not to log may be more blocking than executing precompiled erlang code. > > So, question: why approach with compiled on-fly module is not used? Just because ets is more standard? Or "shut up and make pull request"? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Sat Aug 24 17:17:16 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 24 Aug 2013 19:17:16 +0400 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: Message-ID: It is not so clear for me: do you generate a proxy module on config loading or you make a parse_transform of code that uses lager:debug(..) ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Aug 24 17:45:09 2013 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 24 Aug 2013 17:45:09 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: Message-ID: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> The loglevel check is done using mochiglobal: %% @doc Set the loglevel for a particular backend. set_loglevel(Handler, Level) when is_atom(Level) -> Reply = gen_event:call(lager_event, Handler, {set_loglevel, Level}, infinity), %% recalculate min log level MinLog = minimum_loglevel(get_loglevels()), {_, Traces} = lager_mochiglobal:get(loglevel), lager_mochiglobal:put(loglevel, {MinLog, Traces}), Reply. Basho switched to an ets table in this pull request: https://github.com/basho/lager/pull/88 "Replace mochiglobal with a public ETS table This also removes the dependency on the syntax_tools and compiler applications, which interfered with lager's startup/shutdown order. I've benchmarked this with logbench, and performance doesn't seem to differ significantly from the mochiglobal version, and has much less baggage associated with it." BR, Ulf W On 24 Aug 2013, at 17:17, Max Lapshin wrote: > It is not so clear for me: do you generate a proxy module on config loading or you make a parse_transform of code that uses lager:debug(..) ? Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ludovic@REDACTED Sat Aug 24 14:30:59 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Sat, 24 Aug 2013 14:30:59 +0200 Subject: [erlang-questions] Core erlang wildcards and sequences In-Reply-To: <1307148587.11663103.1377189682223.JavaMail.zimbra@erlang-solutions.com> References: <1307148587.11663103.1377189682223.JavaMail.zimbra@erlang-solutions.com> Message-ID: Hi, So it's better to stick with nested let-exprs. It's quite hard to find docs about core erlang, i have some docs but not examples to hack with. Would you mind to post an example of erlang code which would use letrecs when compiled to core erlang ? If i read well, 'local recursive functions' are sorts of self-aware funs that can call themselves ? And theese funs are NOT able to NOT call themselves so local NON-recursive functions are missing in core erlang ... I'm not sure i really get what "local function" means : funs or functions local from a module perspective. If it's the latter then i don't understand why the calling code should treat recursive functions specifically, because the calling code just call a function and don't have to know what this function does, does it ? Thank you Ludovic ( any option to have Reply to header set to erlang-questions@REDACTED automatically on this mailing list ? ) Le Thu, 22 Aug 2013 18:41:22 +0200, Robert Virding a ?crit: > While the Erlang compiler stacks let this is not really necessary as > Core does supports nested calls. It is really not defined in which order > the arguments will be evaluated, which is one reason why Erlang does the > explicit nested lets. In one sense there it makes no difference as the > resultant code will be the same, however the optimisation passes of the > compiler more or less assume the flattened nested form. > > There is one thing missing in Core and that is non-recursive local > function definitions. Using letrec defines a recursive function so if > you want to reuse local function names as local to a block you can't, > you have to invent a new name each time. Almost trivial but still > annoying. > > Robert > > ----- Original Message ----- >> From: "Ludovic Demblans" >> To: erlang-questions@REDACTED >> Sent: Saturday, 3 August, 2013 3:50:36 PM >> Subject: Re: [erlang-questions] Core erlang wildcards and sequences >> >> Ok so i'll continue to stack let expressions, it seems simple. >> >> Thanks for the replies, >> >> Ludovic >> >> >> >> >> Le Thu, 01 Aug 2013 15:57:30 +0200, Anthony Ramine a >> ?crit: >> >> > Hello, >> > >> > Replied inline. >> > >> >> >> -- >> Utilisant le logiciel de courrier r?volutionnaire d'Opera : >> http://www.opera.com/mail/ >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> -- Utilisant le logiciel de courrier r?volutionnaire d'Opera : http://www.opera.com/mail/ From i.aadi07@REDACTED Sat Aug 24 17:14:35 2013 From: i.aadi07@REDACTED (aadi i) Date: Sat, 24 Aug 2013 08:14:35 -0700 (PDT) Subject: [erlang-questions] slow response of post method of https request using httpc module Message-ID: <51b24890-b15a-4c8e-b19e-65bfe6bc8066@googlegroups.com> Our Application (which uses existing Erlang OTP R15B01 modules) sends https request to external authentication server..and it gets reply and seems work fine under normal cases But under heavy loads some requests are failing since they are consuming more time to do ssl handshake and i have observed the following things during ssl handshake client is taking(our application ) nearly 80 sec to send the certificate after server hello is done with server certificate since our server expects to complete the request- response in 30 sec otherwise it drops the connection hence results to connection failures and effects the performance of application severely and finally i would like to know - Is our application failing to invoke the client certificate quickly .? i mean does httpc module do the file/IO related operations to invoke the certificates which results to slow response under heavy loads..? -Does Erlang has any limitations in ssl handshake procedure ..? Thanks in Advance :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From i.aadi07@REDACTED Sat Aug 24 17:17:56 2013 From: i.aadi07@REDACTED (aadi i) Date: Sat, 24 Aug 2013 20:47:56 +0530 Subject: [erlang-questions] Slow response of post method of https request using httpc erlang module Message-ID: Our Application (which uses existing Erlang OTP R15B01 modules) sends https request to external authentication server..and it gets reply and seems work fine under normal cases But under heavy loads some requests are failing since they are consuming more time to do ssl handshake and i have observed the following things during ssl handshake client is taking(our application ) nearly 80 sec to send the certificate after server hello is done with server certificate since our server expects to complete the request- response in 30 sec otherwise it drops the connection hence results to connection failures and effects the performance of application severely and finally i would like to know - Is our application failing to invoke the client certificate quickly .? i mean does httpc module do the file/IO related operations to invoke the certificates which results to slow response under heavy loads..? -Does Erlang has any limitations in ssl handshake procedure ..? Thanks in Advance :-) adinarayana murthy -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Sat Aug 24 18:51:57 2013 From: bob@REDACTED (Bob Ippolito) Date: Sat, 24 Aug 2013 09:51:57 -0700 Subject: [erlang-questions] Slow response of post method of https request using httpc erlang module In-Reply-To: References: Message-ID: Have you configured httpc to allow more parallel connections to your external server? On Saturday, August 24, 2013, aadi i wrote: > Our Application (which uses existing Erlang OTP R15B01 modules) sends > https request to external authentication server..and it gets reply and > seems work fine under normal cases > > But under heavy loads some requests are failing since they are > consuming more time to do ssl handshake > > and i have observed the following things during ssl handshake > > client is taking(our application ) nearly 80 sec to send the certificate > after server hello is done with server certificate > > since our server expects to complete the request- response in 30 sec > otherwise it drops the connection hence results to connection failures and > effects the performance of application severely > > and finally i would like to know > - Is our application failing to invoke the client certificate quickly .? > i mean does httpc module do the file/IO related operations to invoke the > certificates which results to slow response under heavy loads..? > > -Does Erlang has any limitations in ssl handshake procedure ..? > > > Thanks in Advance :-) > > > adinarayana murthy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Aug 24 22:10:51 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 24 Aug 2013 22:10:51 +0200 Subject: [erlang-questions] [ANN] hackney 0.4.3 has been released Message-ID: Just released a new version of hackney, an HTTP client library in Erlang. Little changes in this one except that the parse transform has been removed. Now the REST syntax `hackney:Method(URL, ...)` (where `Method` can be get, post,put, delete....) thanks to nox is now available at the compilation which also means that you can use it in the shell. Ex. hackney:get("https://friendpaste.com") will start to fetch the Friendpaste page. More info on the release page: https://github.com/benoitc/hackney/releases/tag/0.4.3 Enjoy :) - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Sun Aug 25 00:56:45 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Sat, 24 Aug 2013 15:56:45 -0700 (PDT) Subject: [erlang-questions] [ANN] hackney 0.4.3 has been released In-Reply-To: References: Message-ID: My personal path: 1) Parameterized Modules -> Cool! -> oh no 2) Parse Transform -> Cool! -> oh no 3) Module parameterization -> Cool! -> oh no 4) Higher order function + record -> ....? I'm between 3 & 4 right now. I'm sure it would be "great" to get to "5) higher order function" and eliminate record but I don't see that path yet. /s On Saturday, August 24, 2013 3:10:51 PM UTC-5, Benoit Chesneau wrote: > > Just released a new version of hackney, an HTTP client library in Erlang. > > Little changes in this one except that the parse transform has been > removed. Now the REST syntax `hackney:Method(URL, ...)` (where `Method` can > be get, post,put, delete....) thanks to nox is now available at the > compilation which also means that you can use it in the shell. > > Ex. > > hackney:get("https://friendpaste.com") > > will start to fetch the Friendpaste page. > > More info on the release page: > > https://github.com/benoitc/hackney/releases/tag/0.4.3 > > Enjoy :) > > - benoit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sun Aug 25 17:38:09 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sun, 25 Aug 2013 17:38:09 +0200 Subject: [erlang-questions] [ANN] hackney 0.4.3 has been released In-Reply-To: References: Message-ID: On Sun, Aug 25, 2013 at 12:56 AM, Steve Davis < steven.charles.davis@REDACTED> wrote: > My personal path: > 1) Parameterized Modules -> Cool! -> oh no > 2) Parse Transform -> Cool! -> oh no > 3) Module parameterization -> Cool! -> oh no > 4) Higher order function + record -> ....? > > I'm between 3 & 4 right now. > > I'm sure it would be "great" to get to "5) higher order function" and > eliminate record but I don't see that path yet. > > Not sure what you mean neither what you want. If you are talking about the Client, you can consider it as an opaque value. You generally don't care about its content. When they will be available this record will be probably replaced by a map [1] - benoit [1] http://www.erlang.org/eeps/eep-0043.html > /s > > > On Saturday, August 24, 2013 3:10:51 PM UTC-5, Benoit Chesneau wrote: >> >> Just released a new version of hackney, an HTTP client library in Erlang. >> >> Little changes in this one except that the parse transform has been >> removed. Now the REST syntax `hackney:Method(URL, ...)` (where `Method` can >> be get, post,put, delete....) thanks to nox is now available at the >> compilation which also means that you can use it in the shell. >> >> Ex. >> >> hackney:get("https://**friendpaste.com ") >> >> will start to fetch the Friendpaste page. >> >> More info on the release page: >> >> https://github.com/benoitc/**hackney/releases/tag/0.4.3 >> >> Enjoy :) >> >> - benoit >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean@REDACTED Mon Aug 26 08:00:02 2013 From: sean@REDACTED (Functional Jobs) Date: Mon, 26 Aug 2013 02:00:02 -0400 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <521aeef833890@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Erlang Software Engineer at Smarkets http://functionaljobs.com/jobs/8642-erlang-software-engineer-at-smarkets Cheers, Sean Murphy FunctionalJobs.com From peerst@REDACTED Mon Aug 26 13:49:51 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Mon, 26 Aug 2013 13:49:51 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? References: Message-ID: On 2013-08-20 15:04:41 +0000, Lukas Larsson said: > Hello Peer, > > On Tue, Aug 20, 2013 at 12:43 PM, Peer Stritzinger wrote: > My favorite behaviour would be for Erlang to either grab all memory > available or at least a fixed amount on startup and hold on to it > forever. ?Hopefully without fragmenting it too much (I know in case of > binary heaps that this isn't possible which is a bit disturbing to me). > > Can this be achieved by the current ERTS with some parameter settings? > > You could request that the main mbc (+Mmmbcs) should be preallocated > to a certain size, and this would kind of get the effect you are > looking for. However this is controlled on a per allocator level, so > you have to figure out which type of memory it is that you are using. > It also will only work for any mbcs allocated, all sbcs will still be > allocated using mmap/malloc. If you want to force only mbcs you could > up the sbct to very big, just remember to up lmbcs as well so that can > fit all of your sbcs into the mbc. > > I do not recommend doing this, but it is possible. I see, probably time is better to just optimize the system malloc for erlangs use cases. > Which direction should I look if I want to add behaviour like this? > > BTW: ?next best thing is to malloc/free allways same size tiles this > would avoid outer fragmentation and could be sped up relatively easily. > > The emulator already tries to do this to the best of its abilities. > mbcs mmap requests grow depending on how you setup lmbcs, smbcs and > mbcgs. sys_alloc (i.e. malloc) calls are always multiples of +Muycs. Having allocs multiples of a fixed number helps against small scale fragmentation, on the larger scale if there is no memory mapping there is still possibility of fragmentation with this scheme. > You could set smbcs to the same as lmbcs if you want to, this makes it > so that all mbcs have that same size. I'l try this out, any downsides of doing this? This would remove all fragmentation issues and malloc/free could be sped up by just having a special free list for these fixed size tiles. Cheers -- Peer From wallentin.dahlberg@REDACTED Mon Aug 26 15:52:44 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Mon, 26 Aug 2013 15:52:44 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? In-Reply-To: References: Message-ID: As of R16 MBCs are allocated in fixed sizes (power of 2) and each size has its own freelist. The region sizes for new MBCs grows according to allocating pressure. SBCs are different. The freelist for SBCs is a circular list (but conceptually a stack). Free SBCs are pushed to the freelist (you will usually use the same size very soon, hence you want it first). When allocing a SBC the segment allocator will search the freelist for a segment with suitable size and if none is found it will then map a new segment. The freelist is emptied at the end of the list and these slots are unmapped periodically as well as when cache clearing occurs, i.e. nearing end of memory. The memory allocator is a fairly complex piece of machinery. It balances speed, scalability and fragmentation and does this on multiple levels. You want to get most of your allocations in the MBCs but for large heaps and large binaries this is of course impossible. There is no real downside here but memory tuning can be highly application dependent and there is no real "one size fits all". My suggestion is to tune your system and measure. I do agree that there should be an option to pre-map/allocate the entire space of allocatable memory for the runtime system. It would be better for the segment allocator to handle the entire region itself rather the to rely on the operating system. We will probably add an option for it sometime but I don't know when, maybe for R17 but probably later. // Bj?rn-Egil, still on vacation. 2013/8/26 Peer Stritzinger > On 2013-08-20 15:04:41 +0000, Lukas Larsson said: > > Hello Peer, >> >> On Tue, Aug 20, 2013 at 12:43 PM, Peer Stritzinger >> wrote: >> My favorite behaviour would be for Erlang to either grab all memory >> available or at least a fixed amount on startup and hold on to it forever. >> Hopefully without fragmenting it too much (I know in case of binary heaps >> that this isn't possible which is a bit disturbing to me). >> >> Can this be achieved by the current ERTS with some parameter settings? >> >> You could request that the main mbc (+Mmmbcs) should be preallocated >> to a certain size, and this would kind of get the effect you are looking >> for. However this is controlled on a per allocator level, so you have to >> figure out which type of memory it is that you are using. It also will only >> work for any mbcs allocated, all sbcs will still be allocated using >> mmap/malloc. If you want to force only mbcs you could up the sbct to very >> big, just remember to up lmbcs as well so that can fit all of your sbcs >> into the mbc. >> >> I do not recommend doing this, but it is possible. >> > > I see, probably time is better to just optimize the system malloc for > erlangs use cases. > > > Which direction should I look if I want to add behaviour like this? >> >> BTW: next best thing is to malloc/free allways same size tiles this >> would avoid outer fragmentation and could be sped up relatively easily. >> >> The emulator already tries to do this to the best of its abilities. mbcs >> mmap requests grow depending on how you setup lmbcs, smbcs and mbcgs. >> sys_alloc (i.e. malloc) calls are always multiples of +Muycs. >> > > Having allocs multiples of a fixed number helps against small scale > fragmentation, on the larger scale if there is no memory mapping there is > still possibility of fragmentation with this scheme. > > > You could set smbcs to the same as lmbcs if you want to, this makes it so >> that all mbcs have that same size. >> > > I'l try this out, any downsides of doing this? > > This would remove all fragmentation issues and malloc/free could be sped > up by just having a special free list for these fixed size tiles. > > Cheers > -- Peer > > > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apauley@REDACTED Mon Aug 26 18:31:22 2013 From: apauley@REDACTED (Andreas Pauley) Date: Mon, 26 Aug 2013 18:31:22 +0200 Subject: [erlang-questions] SSL certificate unknown error In-Reply-To: References: Message-ID: Hi Ingela, I'm having the same issue when trying to access a web service over HTTPS. In my case the issue is not present on R13B04, but it is present on R14B03, R15B03-1 and R16B01. I did some debug tracing on R16B01 as suggested above. The ssl app in my case fails validation on an empty field 'X520StateOrProvinceName' in the remote certificate. Even though it looks like the remote certificate may not have valid data in this field, we don't really have control over what certificates external parties use. Google Chrome on Mac OS X claims that this certificate is valid, but in the field "State/Province" it gives "< parser error >" as the value. The first few and last few lines of my debug tracing: 12> ssltest:test(). OTP Version: "R16B01" (<0.104.0>) call ssl_certificate:trusted_cert_and_path([<<48,130,4,175,48,130,3,151,160,3,2,1,2,2,3,1,71,197,48,13,6,9,42,134,72,134, ... (<0.104.0>) call 'OTP-PUB-KEY':decode('X520StateOrProvinceName',<<19,0>>) (<0.104.0>) returned from 'OTP-PUB-KEY':decode/2 -> {error, {asn1, {length,{1,128},[]}}} (<0.104.0>) exception_from {public_key,pkix_decode_cert,2} {error,{badmatch,{error,{asn1,{length,{1,128},[]}}}}} (<0.104.0>) exception_from {ssl_certificate,trusted_cert_and_path,3} {error,{badmatch,{error,{asn1,{length,{1,128},[]}}}}} =ERROR REPORT==== 26-Aug-2013::18:01:40 === SSL: certify: tls_handshake.erl:259:Fatal error: certificate unknown ** exception error: no match of right hand side value {error,{tls_alert,"certificate unknown"}} in function ssltest:test/1 (/var/tmp/ssl/ssltest.erl, line 17) The code I used to test with: -module(ssltest). -export([test/0]). test() -> test("196.38.92.87"). test(Server) -> [application:start(App) || App <- [crypto, asn1, public_key, ssl]], OTPVersion = erlang:system_info(otp_release), io:format("OTP Version: ~p~n", [OTPVersion]), dbg:tracer(), dbg:p(all, [call]), dbg:tpl(ssl_certificate, trusted_cert_and_path, x), dbg:tpl(public_key, pkix_decode_cert, x), dbg:tpl('OTP-PUB-KEY', decode, x), {ok, _SslSocket} = ssl:connect(Server,443,[binary, {active,false}], 2000), % this is line 17 io:format("Successful ssl:connect to ~s~n", [Server]), ok. Please let me know if I can help with more tracing or other info. Kind regards, Andreas Pauley -- http://pauley.org.za/ http://twitter.com/apauley http://www.meetup.com/lambda-luminaries/ http://za.linkedin.com/in/apauley On Tue, Mar 12, 2013 at 10:48 AM, Ingela Andin wrote: > Hi! > > This error typically happens when there is a problem in the ASN-1 > decoding of a certificate. > You could try tracing on public_key:pkix_path_validation. If we can > get the input values > to that function we could create a way to reproduce the error and make > it much easier for us > to fix the problem. > > dbg:tracer(). > dbg:p(all, [call]). > dbg:tpl(public_key, pkix_path_validation, x). > > Regards Ingela Erlang/OTP- team Ericsson AB > > > 2013/3/12, Andrei Soroker : >> On Mon, Mar 11, 2013 at 8:14 PM, Tristan Sloughter >> wrote: >>> I ran into the same problem on R16A. I sent this to erlang-bugs >>> >>> http://erlang.org/pipermail/erlang-bugs/2013-February/003369.html >>> >>> I sadly didn't take care of continued investigation.... But if it still >>> exists in R16B I guess I'll have to now, unless you figure it out :) >> >> I reverted to R15B03-1 for the time being. I'll poke around some more >> later in the week. >> >>> >>> >>> On Mon, Mar 11, 2013 at 10:09 PM, Andrei Soroker >>> wrote: >>>> >>>> Hi again, >>>> >>>> I'm using lhttpc to make some calls over https. Everything worked fine >>>> with R15B03-1, but with R16B I'm seeing this: >>>> >>>> SSL: certify: ssl_handshake.erl:263:Fatal error: certificate unknown >>>> >>>> Thanks, >>>> Andrei >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From andrew@REDACTED Mon Aug 26 19:11:36 2013 From: andrew@REDACTED (Andrew Thompson) Date: Mon, 26 Aug 2013 13:11:36 -0400 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> Message-ID: <20130826171136.GA10745@hijacked.us> I'm willing to reconsider my change to ETS, if any tangible benefit can be seen. We might be able to switch to goldrush for all filtering, not just tracing. Goldrush does the 'recompile a module every time you change the config' trick, but nicely wrapped in a library. The goldrush dep re-introduced the syntax_tools/compiler dependencies anyway, so I've resigned myself to having to have them around. Andrew From ludovic@REDACTED Mon Aug 26 20:27:26 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Mon, 26 Aug 2013 20:27:26 +0200 Subject: [erlang-questions] Core erlang wildcards and sequences In-Reply-To: <989B115C-C7A4-4FB3-A216-BB43BA3ACB61@gmail.com> References: <1307148587.11663103.1377189682223.JavaMail.zimbra@erlang-solutions.com> <989B115C-C7A4-4FB3-A216-BB43BA3ACB61@gmail.com> Message-ID: Ok comprehensions uses letrec ... the generated core is quite hard to understand to me but I have some new toys :) Thank you ! Le Sat, 24 Aug 2013 19:47:37 +0200, Anthony Ramine a ?crit: > Hello, > > Replied inline. > > Regards, > -- Utilisant le logiciel de courrier r?volutionnaire d'Opera : http://www.opera.com/mail/ From pablo.polvorin@REDACTED Tue Aug 27 01:30:47 2013 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Mon, 26 Aug 2013 20:30:47 -0300 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: <20130826171136.GA10745@hijacked.us> References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: Some time ago I've seen performance degradation too, only on tight loops that were doing very little work but had some lager:debug() around. My workaround at that time was to get the current loglevel before entering the loop, and then call lager:debug selectively only if needed. It did have an impact for me in that case On 26 August 2013 14:11, Andrew Thompson wrote: > I'm willing to reconsider my change to ETS, if any tangible benefit can > be seen. We might be able to switch to goldrush for all filtering, not > just tracing. Goldrush does the 'recompile a module every time you > change the config' trick, but nicely wrapped in a library. > > The goldrush dep re-introduced the syntax_tools/compiler dependencies > anyway, so I've resigned myself to having to have them around. > > Andrew > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Pablo Polvorin ProcessOne From paul.joseph.davis@REDACTED Tue Aug 27 04:56:55 2013 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Mon, 26 Aug 2013 21:56:55 -0500 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: <20130826171136.GA10745@hijacked.us> References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: Andrew, Can you explain the reticence about having compiler and syntax_tools as dependencies? At first I was mostly assuming you wanted lager to handle compiler logging or some such so it was a question of startup ordering. Given goldrush I'm wondering if it were a more a general concern about just removing dependencies for file sizes and so on. Paul On Mon, Aug 26, 2013 at 12:11 PM, Andrew Thompson wrote: > I'm willing to reconsider my change to ETS, if any tangible benefit can > be seen. We might be able to switch to goldrush for all filtering, not > just tracing. Goldrush does the 'recompile a module every time you > change the config' trick, but nicely wrapped in a library. > > The goldrush dep re-introduced the syntax_tools/compiler dependencies > anyway, so I've resigned myself to having to have them around. > > Andrew > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew@REDACTED Tue Aug 27 05:14:21 2013 From: andrew@REDACTED (Andrew Thompson) Date: Mon, 26 Aug 2013 23:14:21 -0400 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: <20130827031421.GB10745@hijacked.us> On Mon, Aug 26, 2013 at 09:56:55PM -0500, Paul Davis wrote: > Andrew, > > Can you explain the reticence about having compiler and syntax_tools as > dependencies? At first I was mostly assuming you wanted lager to handle > compiler logging or some such so it was a question of startup ordering. > Given goldrush I'm wondering if it were a more a general concern about just > removing dependencies for file sizes and so on. It was mainly the complexity incurred for application startup by depending on those applications, even though they're only library applications, the application controller insists they be 'started'. That complexity seems to be mostly handled by lager now, so the original reasons may not have any grounding anymore. Andrew From peter@REDACTED Tue Aug 27 09:30:47 2013 From: peter@REDACTED (Peter Membrey) Date: Tue, 27 Aug 2013 09:30:47 +0200 (CEST) Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> Message-ID: <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> Hi guys, I've got a fairly basic C Node set up where I have the main thread running in a loop with ei_xreceive_msg() and a number of "callback" threads that execute functions and write data using ei_send() to the shared socket (connecting to the Erlang node). Originally I had a lot of data corruption (the Erlang node crashing due to corrupt data) because of incorrect locking on socket writes. I added mutexes to the ei_send() calls and this problem seemed to go away. However I've had a couple of occasions where the system has been quiet and then suddenly become busy where corrupt data has still been sent to the Erlang node. I'm positive all the places where I do ei_send() are protected, but that got me wondering about ei_xreceive_msg(). >From what I can find, ei_xreceive_msg() automatically handles heartbeats for you and I guess that means it will send some sort of reply on that socket. If the heart beat is being sent at the same time as some other process tries to write to the socket, is it possible that the two could get interleaved or something? I would honestly have thought it unlikely but I'm running out of ideas. Assuming it's possible, how could I add a mutex in this case? The call itself blocks, so I can't wrap the whole call in the mutex else nothing else will be able to send data, and there's no way to pass a mutex into the call itself. So as far as I can tell, there's no way to protect these writes and prevent them from getting mixed up with other writes on that socket. Does anyone have any ideas? I'm quite willing to accept I could be doing something pretty stupid, but I'm really out of ideas as to what that might be... Thanks in advance! Kind Regards, Peter Membrey From lukas@REDACTED Tue Aug 27 09:51:41 2013 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 27 Aug 2013 09:51:41 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: <20130826171136.GA10745@hijacked.us> References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: On Mon, Aug 26, 2013 at 7:11 PM, Andrew Thompson wrote: > I'm willing to reconsider my change to ETS, if any tangible benefit can > be seen. > One good reason _not_ to switch from ETS tables is that before R16B hot code loading would "stop the world" while doing hot code loading. This can be potentially disastrous depending on your current load and system design. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Aug 27 09:52:28 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 27 Aug 2013 11:52:28 +0400 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: But reconfiguration is done very rarely. -------------- next part -------------- An HTML attachment was scrubbed... URL: From liyusheng2@REDACTED Tue Aug 27 11:10:25 2013 From: liyusheng2@REDACTED (LiYusheng) Date: Tue, 27 Aug 2013 17:10:25 +0800 Subject: [erlang-questions] [ANN] bean2cloud, erlang based HTML5 webapp dev platform Message-ID: Dear Erlangers, We are pleased to announce the release of bean2cloud server, a platform for rapid development of HTML5 webapp/games. It is based on Erlang/Yaws and is open source. The source code can be found at https://bitbucket.org/qiyun/bean2cloud/ It provides the following features: 1. Social development of HTML5 projects. It has distributed version control, data storage backend, online code edition, project deployment, task management, messenger for collaboration. 2. One click to setup the whole enviroment in a machine. no need to do any configuration. 3. One command to pack the whole development enviroment. The whole enviroment can be packed into a single installation package. Can easily deploy the pack in other machines/cloud 4. One command to pack HTML5 projects into APK or iOS installation package. Can be directly installed in all android/iOS phones/pads. Can be directly pushed to all web stores. 5. Write backend logic in your favirate language(s). Either javascript,Erlang,Php all supported. Future languages include Python,Ruby,Java. 6. Use your favorite data store at ease. sqlite,postgresql have builtin support. Future data store includes mysql,riak,mongodb 7. Completely open source, completely free. You can go to http://qyapp.cn/qiyun/main for online tryout. some noteable webapp/games created by this platform are projmgmt (http://qyapp.cn/qiyun/qyprojmgmt_sln/qyprojmgmt) form builder (http://qyapp.cn/qiyun/qyFormBuilder_sln/qyFormBuilder) qiyun data platform support (http://qyapp.cn/qiyun/qydps_sln/qydps) team.r.us (http://qyapp.cn/teamrus) wolf (http://qyapp.cn/ganyouyin/wolf) Twister (http://qyapp.cn/ganyouyin/Twister) To install the environment in your local machine, please do the following: 1. download http://qyapp.cn/b2c/100/bean2cloud_1.00_i386.deb. 2. either double click the downloaded deb, or run dpkg -i bean2cloud_1.00_i386.deb to install it. 3. once installed, open a console(Ctrl+Alt+T), and run "qystart". 4. a browser webapp will show up and you should have bean2cloud installed and running! -------------- next part -------------- An HTML attachment was scrubbed... URL: From liyusheng2@REDACTED Tue Aug 27 11:32:58 2013 From: liyusheng2@REDACTED (LiYusheng) Date: Tue, 27 Aug 2013 17:32:58 +0800 Subject: [erlang-questions] [ANN] bean2cloud, erlang based HTML5 webapp dev platform In-Reply-To: References: Message-ID: download link should be http://qyapp.cn/b2c/100/bean2cloud_1.00_i386.deb hotmail added the last period into the url. td;lr one click to install open source html5 webapp/game dev platform in your local machine which is based on Erlang/Yaws. The source code can be found at https://bitbucket.org/qiyun/bean2cloud/ From: liyusheng2@REDACTED To: erlang-questions@REDACTED Subject: [ANN] bean2cloud, erlang based HTML5 webapp dev platform Date: Tue, 27 Aug 2013 17:10:25 +0800 Dear Erlangers, We are pleased to announce the release of bean2cloud server, a platform for rapid development of HTML5 webapp/games. It is based on Erlang/Yaws and is open source. The source code can be found at https://bitbucket.org/qiyun/bean2cloud/ It provides the following features: 1. Social development of HTML5 projects. It has distributed version control, data storage backend, online code edition, project deployment, task management, messenger for collaboration. 2. One click to setup the whole enviroment in a machine. no need to do any configuration. 3. One command to pack the whole development enviroment. The whole enviroment can be packed into a single installation package. Can easily deploy the pack in other machines/cloud 4. One command to pack HTML5 projects into APK or iOS installation package. Can be directly installed in all android/iOS phones/pads. Can be directly pushed to all web stores. 5. Write backend logic in your favirate language(s). Either javascript,Erlang,Php all supported. Future languages include Python,Ruby,Java. 6. Use your favorite data store at ease. sqlite,postgresql have builtin support. Future data store includes mysql,riak,mongodb 7. Completely open source, completely free. You can go to http://qyapp.cn/qiyun/main for online tryout. some noteable webapp/games created by this platform are projmgmt (http://qyapp.cn/qiyun/qyprojmgmt_sln/qyprojmgmt) form builder (http://qyapp.cn/qiyun/qyFormBuilder_sln/qyFormBuilder) qiyun data platform support (http://qyapp.cn/qiyun/qydps_sln/qydps) team.r.us (http://qyapp.cn/teamrus) wolf (http://qyapp.cn/ganyouyin/wolf) Twister (http://qyapp.cn/ganyouyin/Twister) To install the environment in your local machine, please do the following: 1. download http://qyapp.cn/b2c/100/bean2cloud_1.00_i386.deb. 2. either double click the downloaded deb, or run dpkg -i bean2cloud_1.00_i386.deb to install it. 3. once installed, open a console(Ctrl+Alt+T), and run "qystart". 4. a browser webapp will show up and you should have bean2cloud installed and running! -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Tue Aug 27 12:05:34 2013 From: tony@REDACTED (Tony Rogvall) Date: Tue, 27 Aug 2013 12:05:34 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: But disaster is really boring :-) /Tony On 27 aug 2013, at 09:52, Max Lapshin wrote: > But reconfiguration is done very rarely. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Tue Aug 27 14:08:15 2013 From: ulf@REDACTED (Ulf Wiger) Date: Tue, 27 Aug 2013 14:08:15 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> Message-ID: <67BFCFB7-197F-45ED-B4ED-9EC0769F1246@feuerlabs.com> 27 aug 2013 kl. 12:05 skrev Tony Rogvall : > But disaster is really boring :-) On the contrary, disasters are everything *but* boring. ;-) BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://feuerlabs.com From rtrlists@REDACTED Tue Aug 27 14:35:57 2013 From: rtrlists@REDACTED (Robert Raschke) Date: Tue, 27 Aug 2013 13:35:57 +0100 Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: References: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> <407260818.1367.1377593402004.JavaMail.zimbra@membrey.hk> Message-ID: Just noticed that my phone email decided not to reply-all. On Tue, Aug 27, 2013 at 1:32 PM, Robert Raschke wrote: > Hi Peter, > > that sounds OK, I guess. It's been ages since I spent any time in the deep > down details of how the receive/send works, so It's difficult for me to say > for certain that a mutex around each explicit send is enough. Your theory > of the receive doing a send internally without the protection of a mutex > sounds reasonable. > > I don't think you can handle the TICK messages yourself, but you might dig > in to see if that's possible. > > Another way that comes to mind is to wrap both receive and send in the > same mutex and have receive be "non-blocking" via the timeout version of > the function. > > HTH, > Robby > > > > On Tue, Aug 27, 2013 at 9:50 AM, Peter Membrey wrote: > >> Hi Robert, >> >> I have a thread pool that receives data from another C library and >> generates messages for sending to Erlang. Each message is atomic and self >> contained, so it's not a data stream as such, just a bunch of messages. >> These can be received and handled in any order at the Erlang side so that >> should be okay I'd have thought. The mutex protects ei_send() with the >> intention of ensuring that only one thread can send a message at a time. Of >> course this assumes that when ei_send() returns, the message has actually >> been sent... >> >> I get data corruption on the Erlang node where it seems as though >> messages have become mixed up or inter-mingled. All the ei_send() that I do >> are protected by mutex, and the the main thread of the application simply >> blocks on ei_xreceive() - so in theory it could reply to a heart beat at >> any time which maybe gets interspersed with sends from the call back >> threads because it is not protected by the mutex. >> >> I can't think of any other way to do this, so I'd greatly appreciate any >> guidance you can give me... >> >> Kind regards, >> >> Peter Membrey >> >> ------------------------------ >> *From: *"Robert Raschke" >> *To: *"Peter Membrey" >> *Sent: *Tuesday, 27 August, 2013 4:39:01 PM >> *Subject: *Re:[erlang-questions] Is ei_xreceive_msg() thread safe? >> >> >> You need to call ei_xreceive_msg() regularly, it only handles the pings >> while it is running. Is it possible that you are losing the connection to >> Erlang because of that (see the discussion about net_ticktime in the kernel >> app documentation) ? >> >> But I also note you say you are starting threads to handle work. Is each >> one of them simply sending data to Erlang? Wouldn't that mean you're >> sending randomly interspersed pieces of data? I think you may have to >> investigate handling the comms back to Erlang in more detail. This may >> depend a lot on your use case. But simply mutexing the sends doesn't sound >> right to me. >> >> Robby >> On Aug 27, 2013 8:26 AM, "Peter Membrey" wrote: >> >>> Hi guys, >>> >>> I've got a fairly basic C Node set up where I have the main thread >>> running in a loop with ei_xreceive_msg() and a number of "callback" threads >>> that execute functions and write data using ei_send() to the shared socket >>> (connecting to the Erlang node). >>> >>> Originally I had a lot of data corruption (the Erlang node crashing due >>> to corrupt data) because of incorrect locking on socket writes. I added >>> mutexes to the ei_send() calls and this problem seemed to go away. >>> >>> However I've had a couple of occasions where the system has been quiet >>> and then suddenly become busy where corrupt data has still been sent to the >>> Erlang node. I'm positive all the places where I do ei_send() are >>> protected, but that got me wondering about ei_xreceive_msg(). >>> >>> From what I can find, ei_xreceive_msg() automatically handles heartbeats >>> for you and I guess that means it will send some sort of reply on that >>> socket. If the heart beat is being sent at the same time as some other >>> process tries to write to the socket, is it possible that the two could get >>> interleaved or something? I would honestly have thought it unlikely but I'm >>> running out of ideas. >>> >>> Assuming it's possible, how could I add a mutex in this case? The call >>> itself blocks, so I can't wrap the whole call in the mutex else nothing >>> else will be able to send data, and there's no way to pass a mutex into the >>> call itself. So as far as I can tell, there's no way to protect these >>> writes and prevent them from getting mixed up with other writes on that >>> socket. >>> >>> Does anyone have any ideas? I'm quite willing to accept I could be doing >>> something pretty stupid, but I'm really out of ideas as to what that might >>> be... >>> >>> Thanks in advance! >>> >>> Kind Regards, >>> >>> Peter Membrey >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peerst@REDACTED Tue Aug 27 14:43:19 2013 From: peerst@REDACTED (Peer Stritzinger) Date: Tue, 27 Aug 2013 14:43:19 +0200 Subject: [erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it? References: Message-ID: On 2013-08-26 13:52:44 +0000, Bj?rn-Egil Dahlberg said: > As of R16 MBCs are allocated in fixed sizes (power of 2) and each size > has its own freelist. The region sizes for new MBCs grows according to > allocating pressure. SBCs are different. The freelist for SBCs is a > circular list (but conceptually a stack). Free SBCs are pushed to the > freelist (you will usually use the same size very soon, hence you want > it first). When allocing a SBC the segment allocator will search the > freelist for a segment with suitable size and if none is found it will > then map a new segment. The freelist is emptied at the end of the list > and these slots are unmapped periodically as well as when cache > clearing occurs, i.e. nearing end of memory.? Ok, so no need to add extra fixed size freelists as a malloc/free wrapper if this is already done internally. > The memory allocator is a fairly complex piece of machinery. It > balances speed, scalability and fragmentation and does this on multiple > levels. You want to get most of your allocations in the MBCs but for > large heaps and large binaries this is of course impossible.? > > There is no real downside here but memory tuning can be highly > application dependent and there is no real "one size fits all". My > suggestion is to tune your system and measure. Yeah, will add logging + metrics to the OS malloc/free and tune according to these. > I do agree that there should be an option to pre-map/allocate the > entire space of allocatable memory for the runtime system. It would be > better for the segment allocator to handle the entire region itself > rather the to rely on the operating system. We will probably add an > option for it sometime but I don't know when, maybe for R17 but > probably later. This would be a *very* helpful feature for embedded systems. All the non Erlang stuff is fairly static and if ERTS could grab the rest afterwards and manage it itself this would give the best results and even without tuning. Probably Erlangs memory can be handled even better when it knows right from the start: this is your memory and thats it. Cheers -- Peer > > // Bj?rn-Egil, still on vacation. > > > 2013/8/26 Peer Stritzinger > On 2013-08-20 15:04:41 +0000, Lukas Larsson said: > > Hello Peer, > > On Tue, Aug 20, 2013 at 12:43 PM, Peer Stritzinger wrote: > My favorite behaviour would be for Erlang to either grab all memory > available or at least a fixed amount on startup and hold on to it > forever. ?Hopefully without fragmenting it too much (I know in case of > binary heaps that this isn't possible which is a bit disturbing to me). > > Can this be achieved by the current ERTS with some parameter settings? > > You could request that the main mbc (+Mmmbcs) should be preallocated > to a certain size, and this would kind of get the effect you are > looking for. However this is controlled on a per allocator level, so > you have to figure out which type of memory it is that you are using. > It also will only work for any mbcs allocated, all sbcs will still be > allocated using mmap/malloc. If you want to force only mbcs you could > up the sbct to very big, just remember to up lmbcs as well so that can > fit all of your sbcs into the mbc. > > I do not recommend doing this, but it is possible. > > I see, probably time is better to just optimize the system malloc for > erlangs use cases. > > > Which direction should I look if I want to add behaviour like this? > > BTW: ?next best thing is to malloc/free allways same size tiles this > would avoid outer fragmentation and could be sped up relatively easily. > > The emulator already tries to do this to the best of its abilities. > mbcs mmap requests grow depending on how you setup lmbcs, smbcs and > mbcgs. sys_alloc (i.e. malloc) calls are always multiples of +Muycs. > > Having allocs multiples of a fixed number helps against small scale > fragmentation, on the larger scale if there is no memory mapping there > is still possibility of fragmentation with this scheme. > > > You could set smbcs to the same as lmbcs if you want to, this makes it > so that all mbcs have that same size. > > I'l try this out, any downsides of doing this? > > This would remove all fragmentation issues and malloc/free could be > sped up by just having a special free list for these fixed size tiles. > > Cheers > -- Peer > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From janos.n.hary@REDACTED Tue Aug 27 15:05:58 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Tue, 27 Aug 2013 15:05:58 +0200 Subject: [erlang-questions] mnesia silent failure and table corruption Message-ID: <004801cea326$2d06fb00$8714f100$@gmail.com> All, I started to write records into a table from a simple test function. When the table size reaches 2GB write transactions still report success, but they are silently ignored. The table became unusable, read operations return error, and mnesia cannot recover the table. How shall I avoid such situation? I use R16B01 x64 SMP on Windows 7. The table creation code: mnesia:create_table(tab2, [ {record_name, trec}, {attributes, record_info(fields, trec)}, {disc_only_copies, [node()]} ]) My write function: write_transaction(Tab, Trec) -> F = fun () -> mnesia:write(Tab, Trec, write) end, case mnesia:transaction(F) of {atomic, ok} -> ok; _ -> error end. After reaching 2GB: (test@REDACTED)25> db_test:read_transaction(tab1, 1). {atomic,{error,{{bad_object,read_buckets}, "d:/db_test/tab1.DAT"}}} (test@REDACTED)27> application:start(mnesia). dets: file "d:/db_test/tab2.DAT" not properly closed, repairing ... =ERROR REPORT==== 27-Aug-2013::11:24:18 === Mnesia(test@REDACTED): ** ERROR ** (core dumped to file: "c:/work/dirXDS5/MnesiaCore.test@REDACTED") ** FATAL ** {error,{"Cannot open dets table",tab2, [{file,"d:/db_test/tab2.DAT"}, {keypos,2}, {repair,true}, {type,set}], {no_more_space_on_file,"d:/db_test/tab2.DAT.TMP"}}} =ERROR REPORT==== 27-Aug-2013::11:24:28 === Mnesia(test@REDACTED): ** ERROR ** mnesia_event got unexpected event: {'EXIT', <0.4288.0>, killed} =INFO REPORT==== 27-Aug-2013::11:24:28 === application: mnesia exited: {killed,{mnesia_sup,start,[normal,[]]}} type: temporary {error,{killed,{mnesia_sup,start,[normal,[]]}}} Thanks for your help. Janos From hm@REDACTED Tue Aug 27 16:01:37 2013 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Tue, 27 Aug 2013 16:01:37 +0200 Subject: [erlang-questions] mnesia silent failure and table corruption Message-ID: On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary wrote: > All, > > I started to write records into a table from a simple test function. When > the table size reaches 2GB write transactions still report success, but they > are silently ignored. The table became unusable, read operations return > error, and mnesia cannot recover the table. How shall I avoid such > situation? Try using mnesia:sync_transaction() instead of mnesia:transaction(). Then Mnesia will use syncronized calls to disk_log (and syncronized commits when several nodes are involved). /H?kan From serge@REDACTED Tue Aug 27 16:59:27 2013 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 27 Aug 2013 10:59:27 -0400 Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> References: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> Message-ID: e i_send/receive family of functions are not thread safe, and receive functions do handle heartbeats internally. The way the ei_{connect,send,receive} functions are written it's not possible to do any non-blocking I/O with them or use them in multi-threaded code in the manner you described without modifying the functions. You can take a look at the alternative C++ library that doesn't have such limitations: https://github.com/saleyn/eixx , and offers almost all functionality that ei has. Regards, Serge On Tue, Aug 27, 2013 at 3:30 AM, Peter Membrey wrote: > Hi guys, > > I've got a fairly basic C Node set up where I have the main thread running > in a loop with ei_xreceive_msg() and a number of "callback" threads that > execute functions and write data using ei_send() to the shared socket > (connecting to the Erlang node). > > Originally I had a lot of data corruption (the Erlang node crashing due to > corrupt data) because of incorrect locking on socket writes. I added > mutexes to the ei_send() calls and this problem seemed to go away. > > However I've had a couple of occasions where the system has been quiet and > then suddenly become busy where corrupt data has still been sent to the > Erlang node. I'm positive all the places where I do ei_send() are > protected, but that got me wondering about ei_xreceive_msg(). > > From what I can find, ei_xreceive_msg() automatically handles heartbeats > for you and I guess that means it will send some sort of reply on that > socket. If the heart beat is being sent at the same time as some other > process tries to write to the socket, is it possible that the two could get > interleaved or something? I would honestly have thought it unlikely but I'm > running out of ideas. > > Assuming it's possible, how could I add a mutex in this case? The call > itself blocks, so I can't wrap the whole call in the mutex else nothing > else will be able to send data, and there's no way to pass a mutex into the > call itself. So as far as I can tell, there's no way to protect these > writes and prevent them from getting mixed up with other writes on that > socket. > > Does anyone have any ideas? I'm quite willing to accept I could be doing > something pretty stupid, but I'm really out of ideas as to what that might > be... > > Thanks in advance! > > Kind Regards, > > Peter Membrey > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Tue Aug 27 17:06:14 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Tue, 27 Aug 2013 17:06:14 +0200 Subject: [erlang-questions] mnesia silent failure and table corruption In-Reply-To: References: Message-ID: <004f01cea336$fa6b2390$ef416ab0$@gmail.com> Thanks, but no luck. It behaves the same with sync_transaction. Janos -----Original Message----- From: hawk.mattsson@REDACTED [mailto:hawk.mattsson@REDACTED] On Behalf Of H?kan Mattsson Sent: Tuesday, August 27, 2013 4:02 PM To: Janos Hary Cc: erlang-questions Subject: Re: [erlang-questions] mnesia silent failure and table corruption On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary wrote: > All, > > I started to write records into a table from a simple test function. > When the table size reaches 2GB write transactions still report > success, but they are silently ignored. The table became unusable, > read operations return error, and mnesia cannot recover the table. How > shall I avoid such situation? Try using mnesia:sync_transaction() instead of mnesia:transaction(). Then Mnesia will use syncronized calls to disk_log (and syncronized commits when several nodes are involved). /H?kan From peter@REDACTED Tue Aug 27 17:27:59 2013 From: peter@REDACTED (Peter Membrey) Date: Tue, 27 Aug 2013 17:27:59 +0200 (CEST) Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: References: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> Message-ID: <1396147141.1692.1377617279511.JavaMail.zimbra@membrey.hk> Hi Serge, Thank you very much for clarifying this! At least I know where everything stands now. Does the scenario I'm talking about make sense? Could a heartbeat in the current design end up interleaved with other data? I suspect it would be pretty rare, but we pump 10 million or more messages through this app per day, so even if the chance was a million to one, we'd end up seeing it ten times a day on average... I'm thinking for now that I'll give Robert's suggestion a try. If I call ei_xreceive() with a timeout and wrap a mutex around it, would that then protect the socket? I'm assuming (probably a bad thing) that it doesn't reply in a different thread or something... I guess it would be a bit wasteful to effectively poll the socket every millisecond or so (I can't really block the other threads from writing longer than that) but it seems to be a potential way to resolve this issue in the short term while I come up with a better fix. Does that sound like a reasonable plan? Thanks again! Kind Regards, Peter Membrey ----- Original Message ----- From: "Serge Aleynikov" To: "Peter Membrey" Cc: "Erlang Questions" Sent: Tuesday, August 27, 2013 10:59:27 PM Subject: Re: [erlang-questions] Is ei_xreceive_msg() thread safe? e i_send/receive family of functions are not thread safe, and receive functions do handle heartbeats internally. The way the ei_{connect,send,receive} functions are written it's not possible to do any non-blocking I/O with them or use them in multi-threaded code in the manner you described without modifying the functions. You can take a look at the alternative C++ library that doesn't have such limitations: https://github.com/saleyn/eixx , and offers almost all functionality that ei has. Regards, Serge On Tue, Aug 27, 2013 at 3:30 AM, Peter Membrey < peter@REDACTED > wrote: Hi guys, I've got a fairly basic C Node set up where I have the main thread running in a loop with ei_xreceive_msg() and a number of "callback" threads that execute functions and write data using ei_send() to the shared socket (connecting to the Erlang node). Originally I had a lot of data corruption (the Erlang node crashing due to corrupt data) because of incorrect locking on socket writes. I added mutexes to the ei_send() calls and this problem seemed to go away. However I've had a couple of occasions where the system has been quiet and then suddenly become busy where corrupt data has still been sent to the Erlang node. I'm positive all the places where I do ei_send() are protected, but that got me wondering about ei_xreceive_msg(). >From what I can find, ei_xreceive_msg() automatically handles heartbeats for you and I guess that means it will send some sort of reply on that socket. If the heart beat is being sent at the same time as some other process tries to write to the socket, is it possible that the two could get interleaved or something? I would honestly have thought it unlikely but I'm running out of ideas. Assuming it's possible, how could I add a mutex in this case? The call itself blocks, so I can't wrap the whole call in the mutex else nothing else will be able to send data, and there's no way to pass a mutex into the call itself. So as far as I can tell, there's no way to protect these writes and prevent them from getting mixed up with other writes on that socket. Does anyone have any ideas? I'm quite willing to accept I could be doing something pretty stupid, but I'm really out of ideas as to what that might be... Thanks in advance! Kind Regards, Peter Membrey _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.polvorin@REDACTED Tue Aug 27 18:26:53 2013 From: pablo.polvorin@REDACTED (Pablo Polvorin) Date: Tue, 27 Aug 2013 13:26:53 -0300 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Hello, might be interesting to the emysql users, just published a first post on some optimizations we have been doing to it http://blog.process-one.net/optimizing-erlang-applications-emysql/ It is focused on queries returning large result sets, that was our original problem. There are more things done but not yet had time to complete a writting about it. Note this isn't on oficial emysql repo, Jesper did try to include some of the changes, but at that time he found some problems (thanks!), solved now on our repo HEAD. On 5 July 2013 14:03, Garrett Smith wrote: > On Fri, Jul 5, 2013 at 10:43 AM, Jesper Louis Andersen > wrote: >> On Fri, Jul 5, 2013 at 4:27 PM, Garrett Smith wrote: >>> >>> >>> At first glance, it looks like I'd be losing the connect timeout >>> enhancements from here: >>> >>> https://github.com/gar1t/Emysql >>> >> >> Yes, we are currently missing those changes in the code base. I only took >> branches which were updated fairly recently on github, and yours were >> overlooked for some reason. >> >> I like most of the changes you have in there and most of them definitely >> should be part of the full release. I also agree that your functional change >> to the gen_tcp:recv timeout is something we want to fix. The only slight >> worry is that it may be a too big behavioral change for some people. The >> other changes are changes I want as well, so i'll probably pull them unless >> you want to prepare a PR yourself. > > I'll have to look again, but the intent was that default behavior is unchanged. > >> I guess the initial reason for having the TCP timeout was to catch slow >> queries and get rid of them quickly, rather than waiting around. This makes >> a lot of sense in an environment where you have a large amount of small >> queries and are not allowed to wait around. But for larger queries and >> procedure calls, this is definitely not what one wants. > > A default timeout other than infinity strikes me as a very bad idea. > There are many reasons, but to your point in particular, a client > close during a transaction will trigger a rollback, which can have > profoundly negative effects on the system performance -- in many cases > much worse then just waiting for the query to finish. > > Not that this should influence the library feature set, but I tend to > deal with problematic long-running queries as an out-of-band problem. > E.g. a separate monitor that looks for problem connections and kills > them (could be a sys admin in the simplest case). > >> The whole decoder loop looks weird as well. It should be possible to make it >> smarter than what is it currently, but this is a future change. >> >> Just drop me a note if you want to push it yourself or I should get to work. >> But it won't be this weekend, so I'll only be able to look at this Monday at >> the earliest, perhaps late Sunday. > > I can tackle this -- but it likely won't be within days :) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Pablo Polvorin ProcessOne From serge@REDACTED Tue Aug 27 18:43:02 2013 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 27 Aug 2013 12:43:02 -0400 Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: <1396147141.1692.1377617279511.JavaMail.zimbra@membrey.hk> References: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> <1396147141.1692.1377617279511.JavaMail.zimbra@membrey.hk> Message-ID: I think a cleaner/safer solution would be not to have multi-threaded code calling ei functions. You can create a FIFO queue (pipe, or user-level queue with eventfd() file descriptor) that the other threads would write to in a thread-safe manner. This queue would have an associated file descriptor that could be added to the select set together with the file descriptor returned by ei_xconnect_tmo(). This way you can poll for input from the socket connected to an Erlang node and be awaken when some other thread writes something to the output queue - all in the same ei-handling thread and have all socket-related I/O serialized safely. On Tue, Aug 27, 2013 at 11:27 AM, Peter Membrey wrote: > Hi Serge, > > Thank you very much for clarifying this! At least I know where everything > stands now. > > Does the scenario I'm talking about make sense? Could a heartbeat in the > current design end up interleaved with other data? I suspect it would be > pretty rare, but we pump 10 million or more messages through this app per > day, so even if the chance was a million to one, we'd end up seeing it ten > times a day on average... > > I'm thinking for now that I'll give Robert's suggestion a try. If I call > ei_xreceive() with a timeout and wrap a mutex around it, would that then > protect the socket? I'm assuming (probably a bad thing) that it doesn't > reply in a different thread or something... > > I guess it would be a bit wasteful to effectively poll the socket every > millisecond or so (I can't really block the other threads from writing > longer than that) but it seems to be a potential way to resolve this issue > in the short term while I come up with a better fix. > > Does that sound like a reasonable plan? > > Thanks again! > > Kind Regards, > > Peter Membrey > > ------------------------------ > *From: *"Serge Aleynikov" > *To: *"Peter Membrey" > *Cc: *"Erlang Questions" > *Sent: *Tuesday, August 27, 2013 10:59:27 PM > *Subject: *Re: [erlang-questions] Is ei_xreceive_msg() thread safe? > > e > i_send/receive family of functions are not thread safe, and receive > functions do handle heartbeats internally. The way the > ei_{connect,send,receive} functions are written it's not possible to do any > non-blocking I/O with them or use them in multi-threaded code in the manner > you described without modifying the functions. > > You can take a look at the alternative C++ library that doesn't have such > limitations: > https://github.com/saleyn/eixx > , and offers almost all functionality that ei has. > > Regards, > > Serge > > > On Tue, Aug 27, 2013 at 3:30 AM, Peter Membrey wrote: > >> Hi guys, >> >> I've got a fairly basic C Node set up where I have the main thread >> running in a loop with ei_xreceive_msg() and a number of "callback" threads >> that execute functions and write data using ei_send() to the shared socket >> (connecting to the Erlang node). >> >> Originally I had a lot of data corruption (the Erlang node crashing due >> to corrupt data) because of incorrect locking on socket writes. I added >> mutexes to the ei_send() calls and this problem seemed to go away. >> >> However I've had a couple of occasions where the system has been quiet >> and then suddenly become busy where corrupt data has still been sent to the >> Erlang node. I'm positive all the places where I do ei_send() are >> protected, but that got me wondering about ei_xreceive_msg(). >> >> From what I can find, ei_xreceive_msg() automatically handles heartbeats >> for you and I guess that means it will send some sort of reply on that >> socket. If the heart beat is being sent at the same time as some other >> process tries to write to the socket, is it possible that the two could get >> interleaved or something? I would honestly have thought it unlikely but I'm >> running out of ideas. >> >> Assuming it's possible, how could I add a mutex in this case? The call >> itself blocks, so I can't wrap the whole call in the mutex else nothing >> else will be able to send data, and there's no way to pass a mutex into the >> call itself. So as far as I can tell, there's no way to protect these >> writes and prevent them from getting mixed up with other writes on that >> socket. >> >> Does anyone have any ideas? I'm quite willing to accept I could be doing >> something pretty stupid, but I'm really out of ideas as to what that might >> be... >> >> Thanks in advance! >> >> Kind Regards, >> >> Peter Membrey >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Aug 27 19:06:02 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 27 Aug 2013 19:06:02 +0200 Subject: [erlang-questions] Async ssl_accept, ssl:close, gen_tcp:send and ssl:send In-Reply-To: References: Message-ID: <521CDC7A.7070908@ninenines.eu> Hey Ingela, On 07/04/2013 10:50 PM, Ingela Andin wrote: > 2013/6/29 Lo?c Hoguin > > > Hello, > > I'd like to start a discussion on making additions to the gen_tcp > and ssl API to allow it to handle more asynchronous operations. The > reasons for allowing them async are as follow: > > * async ssl_accept: This call can take a long time because > there's a few round trips between the two endpoints to perform the > handshake. Therefore there's a lot of waiting going on, not just > busy operations. It could be more interesting to not block a process > for this because the process could for example initialize the state > for the connection while it waits for the handshake to be completed, > reducing the latency of applications. > > > Is it not enough that ssl:transport_accept and ssl:ssl_accept can/should > be called by different processes? Problem comes when trying to guarantee that ssl_accept will be ran before any reading is done. If we can have ssl_accept ran asynchronously the message order guarantees this with no extra work and without blocking any process. If the async ssl_accept fails, I'm more than fine finding out about it when trying to read right after. > * async ssl:close: This call can also take a long time for > similar reasons. I think the same can apply to gen_tcp? Anyway if > you have a long running process that holds a connection at some > point, and wants to close it, it might interrupt its normal > workflow. A temporary solution would be to have a process dedicated > to closing these sockets and pass it ownership, but if that can be > avoided... Same for this really. We don't really care to know that the socket is closed properly most of the time. We just want to close it and free resources and move on. > * async send: Send also blocks, I suppose because it waits for > the TCP ack before returning. Sometimes you don't have the luxury to > wait, so it would be good to have an alternative documented > solution. An undocumented solution can be found here: > http://www.erlang-factory.com/__conference/SFBay2013/speakers/__GeoffCant > This one is unrelated and not problematic per se but would also be pretty cool to have. > I will not go in to the other points now, as I am supposed to be on > vacation, so I will go back to that! I hope you had a great time! -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From overminddl1@REDACTED Tue Aug 27 21:16:49 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Tue, 27 Aug 2013 13:16:49 -0600 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: I love these kind of articles on just how bug fixes and optimizations are done. Look forward to the next. On Aug 27, 2013 10:27 AM, "Pablo Polvorin" wrote: > Hello, might be interesting to the emysql users, just published a > first post on some optimizations we have been doing to it > http://blog.process-one.net/optimizing-erlang-applications-emysql/ > > It is focused on queries returning large result sets, that was our > original problem. There are more things done but not yet had time > to complete a writting about it. Note this isn't on oficial emysql > repo, Jesper did try to include some of the changes, but at that time > he found some problems (thanks!), solved now on our repo HEAD. > > > > On 5 July 2013 14:03, Garrett Smith wrote: > > On Fri, Jul 5, 2013 at 10:43 AM, Jesper Louis Andersen > > wrote: > >> On Fri, Jul 5, 2013 at 4:27 PM, Garrett Smith wrote: > >>> > >>> > >>> At first glance, it looks like I'd be losing the connect timeout > >>> enhancements from here: > >>> > >>> https://github.com/gar1t/Emysql > >>> > >> > >> Yes, we are currently missing those changes in the code base. I only > took > >> branches which were updated fairly recently on github, and yours were > >> overlooked for some reason. > >> > >> I like most of the changes you have in there and most of them definitely > >> should be part of the full release. I also agree that your functional > change > >> to the gen_tcp:recv timeout is something we want to fix. The only slight > >> worry is that it may be a too big behavioral change for some people. The > >> other changes are changes I want as well, so i'll probably pull them > unless > >> you want to prepare a PR yourself. > > > > I'll have to look again, but the intent was that default behavior is > unchanged. > > > >> I guess the initial reason for having the TCP timeout was to catch slow > >> queries and get rid of them quickly, rather than waiting around. This > makes > >> a lot of sense in an environment where you have a large amount of small > >> queries and are not allowed to wait around. But for larger queries and > >> procedure calls, this is definitely not what one wants. > > > > A default timeout other than infinity strikes me as a very bad idea. > > There are many reasons, but to your point in particular, a client > > close during a transaction will trigger a rollback, which can have > > profoundly negative effects on the system performance -- in many cases > > much worse then just waiting for the query to finish. > > > > Not that this should influence the library feature set, but I tend to > > deal with problematic long-running queries as an out-of-band problem. > > E.g. a separate monitor that looks for problem connections and kills > > them (could be a sys admin in the simplest case). > > > >> The whole decoder loop looks weird as well. It should be possible to > make it > >> smarter than what is it currently, but this is a future change. > >> > >> Just drop me a note if you want to push it yourself or I should get to > work. > >> But it won't be this weekend, so I'll only be able to look at this > Monday at > >> the earliest, perhaps late Sunday. > > > > I can tackle this -- but it likely won't be within days :) > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Pablo Polvorin > ProcessOne > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.l.france@REDACTED Tue Aug 27 21:18:40 2013 From: aaron.l.france@REDACTED (Aaron France) Date: Tue, 27 Aug 2013 21:18:40 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Thoroughly enjoyed. Looking forward to the next. On Tue, Aug 27, 2013 at 9:16 PM, OvermindDL1 wrote: > I love these kind of articles on just how bug fixes and optimizations are > done. Look forward to the next. > On Aug 27, 2013 10:27 AM, "Pablo Polvorin" > wrote: > >> Hello, might be interesting to the emysql users, just published a >> first post on some optimizations we have been doing to it >> http://blog.process-one.net/optimizing-erlang-applications-emysql/ >> >> It is focused on queries returning large result sets, that was our >> original problem. There are more things done but not yet had time >> to complete a writting about it. Note this isn't on oficial emysql >> repo, Jesper did try to include some of the changes, but at that time >> he found some problems (thanks!), solved now on our repo HEAD. >> >> >> >> On 5 July 2013 14:03, Garrett Smith wrote: >> > On Fri, Jul 5, 2013 at 10:43 AM, Jesper Louis Andersen >> > wrote: >> >> On Fri, Jul 5, 2013 at 4:27 PM, Garrett Smith wrote: >> >>> >> >>> >> >>> At first glance, it looks like I'd be losing the connect timeout >> >>> enhancements from here: >> >>> >> >>> https://github.com/gar1t/Emysql >> >>> >> >> >> >> Yes, we are currently missing those changes in the code base. I only >> took >> >> branches which were updated fairly recently on github, and yours were >> >> overlooked for some reason. >> >> >> >> I like most of the changes you have in there and most of them >> definitely >> >> should be part of the full release. I also agree that your functional >> change >> >> to the gen_tcp:recv timeout is something we want to fix. The only >> slight >> >> worry is that it may be a too big behavioral change for some people. >> The >> >> other changes are changes I want as well, so i'll probably pull them >> unless >> >> you want to prepare a PR yourself. >> > >> > I'll have to look again, but the intent was that default behavior is >> unchanged. >> > >> >> I guess the initial reason for having the TCP timeout was to catch slow >> >> queries and get rid of them quickly, rather than waiting around. This >> makes >> >> a lot of sense in an environment where you have a large amount of small >> >> queries and are not allowed to wait around. But for larger queries and >> >> procedure calls, this is definitely not what one wants. >> > >> > A default timeout other than infinity strikes me as a very bad idea. >> > There are many reasons, but to your point in particular, a client >> > close during a transaction will trigger a rollback, which can have >> > profoundly negative effects on the system performance -- in many cases >> > much worse then just waiting for the query to finish. >> > >> > Not that this should influence the library feature set, but I tend to >> > deal with problematic long-running queries as an out-of-band problem. >> > E.g. a separate monitor that looks for problem connections and kills >> > them (could be a sys admin in the simplest case). >> > >> >> The whole decoder loop looks weird as well. It should be possible to >> make it >> >> smarter than what is it currently, but this is a future change. >> >> >> >> Just drop me a note if you want to push it yourself or I should get to >> work. >> >> But it won't be this weekend, so I'll only be able to look at this >> Monday at >> >> the earliest, perhaps late Sunday. >> > >> > I can tackle this -- but it likely won't be within days :) >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -- >> Pablo Polvorin >> ProcessOne >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Wed Aug 28 12:04:04 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Wed, 28 Aug 2013 12:04:04 +0200 Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: <1396147141.1692.1377617279511.JavaMail.zimbra@membrey.hk> References: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> <1396147141.1692.1377617279511.JavaMail.zimbra@membrey.hk> Message-ID: <521DCB14.6030805@erix.ericsson.se> On 08/27/2013 05:27 PM, Peter Membrey wrote: > Hi Serge, > > Thank you very much for clarifying this! At least I know where everything stands now. > > Does the scenario I'm talking about make sense? Could a heartbeat in the current design end up interleaved with other data? I suspect it would be pretty rare, but we pump 10 million or more messages through this app per day, so even if the chance was a million to one, we'd end up seeing it ten times a day on average... Yes, it make sense. The ei_send/receive API is not thread safe for the same file descriptor. > I'm thinking for now that I'll give Robert's suggestion a try. If I call ei_xreceive() with a timeout and wrap a mutex around it, would that then protect the socket? I'm assuming (probably a bad thing) that it doesn't reply in a different thread or something... > > I guess it would be a bit wasteful to effectively poll the socket every millisecond or so (I can't really block the other threads from writing longer than that) but it seems to be a potential way to resolve this issue in the short term while I come up with a better fix. > > Does that sound like a reasonable plan? You could use select() to wait for a message to arrive on the socket. Then lock your mutex and call ei_xreceive(). It might not be such a good idea to poll by repeated calls to ei_xreceive() with a short timeout. You can get situations when the timeout triggers after receiving half a message. That will make all following calls to ei_xreceive() get out of synch and probably fail. I guess the probability of this scenario depends on the message sizes and the speed and reliability of your connection. Calling ei_xreceive() without a timeout and my suggestion with select() on the other hand could make a hickup on the net to cause all your sending threads to freeze waiting for the mutex. /Sverker > > Thanks again! > > Kind Regards, > > Peter Membrey > > ----- Original Message ----- > > From: "Serge Aleynikov" > To: "Peter Membrey" > Cc: "Erlang Questions" > Sent: Tuesday, August 27, 2013 10:59:27 PM > Subject: Re: [erlang-questions] Is ei_xreceive_msg() thread safe? > > e > i_send/receive family of functions are not thread safe, and receive functions do handle heartbeats internally. The way the ei_{connect,send,receive} functions are written it's not possible to do any non-blocking I/O with them or use them in multi-threaded code in the manner you described without modifying the functions. > > You can take a look at the alternative C++ library that doesn't have such limitations: > https://github.com/saleyn/eixx > , and offers almost all functionality that ei has. > > Regards, > > Serge > > > On Tue, Aug 27, 2013 at 3:30 AM, Peter Membrey < peter@REDACTED > wrote: > > > Hi guys, > > I've got a fairly basic C Node set up where I have the main thread running in a loop with ei_xreceive_msg() and a number of "callback" threads that execute functions and write data using ei_send() to the shared socket (connecting to the Erlang node). > > Originally I had a lot of data corruption (the Erlang node crashing due to corrupt data) because of incorrect locking on socket writes. I added mutexes to the ei_send() calls and this problem seemed to go away. > > However I've had a couple of occasions where the system has been quiet and then suddenly become busy where corrupt data has still been sent to the Erlang node. I'm positive all the places where I do ei_send() are protected, but that got me wondering about ei_xreceive_msg(). > > >From what I can find, ei_xreceive_msg() automatically handles heartbeats for you and I guess that means it will send some sort of reply on that socket. If the heart beat is being sent at the same time as some other process tries to write to the socket, is it possible that the two could get interleaved or something? I would honestly have thought it unlikely but I'm running out of ideas. > > Assuming it's possible, how could I add a mutex in this case? The call itself blocks, so I can't wrap the whole call in the mutex else nothing else will be able to send data, and there's no way to pass a mutex into the call itself. So as far as I can tell, there's no way to protect these writes and prevent them from getting mixed up with other writes on that socket. > > Does anyone have any ideas? I'm quite willing to accept I could be doing something pretty stupid, but I'm really out of ideas as to what that might be... > > Thanks in advance! > > Kind Regards, > > Peter Membrey > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavchernin@REDACTED Wed Aug 28 13:54:24 2013 From: nadavchernin@REDACTED (Nadav Chernin) Date: Wed, 28 Aug 2013 14:54:24 +0300 Subject: [erlang-questions] Question about lists:filter Message-ID: Hi, all I'm learning Erlang with Project Euler. This is my solution for problem 4. *-module(euler).* *-export([problem4/0]).* * * *cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)].* *isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)).* *problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())).* * * Before it, my code was: *-module(euler).* *-export([problem4/0]).* * * *cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)].* *isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)).* *problem4()->lists:max(lists:filter(isPal,cand())).* * * Difference from my final solution - using isPal instead fun(A)->isPal(A) end But exception occured during execution: exception error: no function clause matching lists:filter(isPal, [10000,10100,10200,10300,10400,10500,10600, 10700,10800,10900,11000,11100,11200,11300, 11400,11500,11600,11700,11800,11900,12000, 12100,12200,12300,12400,12500,12600|...]) Please, explain me why i can't use function as is, without using lambda function. I come to Erlang from Haskell, and there i can do this, Thanks, Nadav -------------- next part -------------- An HTML attachment was scrubbed... URL: From jose.valim@REDACTED Wed Aug 28 13:58:32 2013 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Wed, 28 Aug 2013 08:58:32 -0300 Subject: [erlang-questions] Question about lists:filter In-Reply-To: References: Message-ID: In your problem4() function: *problem4()->lists:max(lists:filter(isPal,cand())).* * * isPal is an atom. It is simply a value. If you want to pass the function isPal declared above as a function, you need to write: *problem4()->lists:max(lists:filter(fun isPal/1,cand())).* * * Functions in Erlang are identified by name *and* arity. That's why you need to explicitly add the arity and use the fun syntax to pass functions around. *Jos? Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer On Wed, Aug 28, 2013 at 8:54 AM, Nadav Chernin wrote: > Hi, all > > I'm learning Erlang with Project Euler. > > This is my solution for problem 4. > > > *-module(euler).* > *-export([problem4/0]).* > * > * > *cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)].* > *isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)).* > *problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())).* > * > * > Before it, my code was: > > *-module(euler).* > *-export([problem4/0]).* > * > * > *cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)].* > *isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)).* > *problem4()->lists:max(lists:filter(isPal,cand())).* > * > * > Difference from my final solution - using isPal instead fun(A)->isPal(A) > end > > But exception occured during execution: > > exception error: no function clause matching > lists:filter(isPal, > > [10000,10100,10200,10300,10400,10500,10600, > > 10700,10800,10900,11000,11100,11200,11300, > > 11400,11500,11600,11700,11800,11900,12000, > 12100,12200,12300,12400,12500,12600|...]) > > Please, explain me why i can't use function as is, without using lambda > function. > I come to Erlang from Haskell, and there i can do this, > > Thanks, Nadav > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Wed Aug 28 13:59:06 2013 From: steve@REDACTED (Steve Strong) Date: Wed, 28 Aug 2013 13:59:06 +0200 Subject: [erlang-questions] Question about lists:filter In-Reply-To: References: Message-ID: In your second sample, isPal is just an atom, hence the error. Instead, use "fun isPal/1" to indicate that you're passing a function. Cheers, Steve -- Steve Strong Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Wednesday, 28 August 2013 at 13:54, Nadav Chernin wrote: > Hi, all > > I'm learning Erlang with Project Euler. > > This is my solution for problem 4. > > > -module(euler). > -export([problem4/0]). > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())). > > Before it, my code was: > > -module(euler). > -export([problem4/0]). > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > problem4()->lists:max(lists:filter(isPal,cand())). > > > Difference from my final solution - using isPal instead fun(A)->isPal(A) end > > But exception occured during execution: > > exception error: no function clause matching > lists:filter(isPal, > [10000,10100,10200,10300,10400,10500,10600, > 10700,10800,10900,11000,11100,11200,11300, > 11400,11500,11600,11700,11800,11900,12000, > 12100,12200,12300,12400,12500,12600|...]) > > > Please, explain me why i can't use function as is, without using lambda function. > I come to Erlang from Haskell, and there i can do this, > > Thanks, Nadav > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Wed Aug 28 14:01:42 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Wed, 28 Aug 2013 14:01:42 +0200 Subject: [erlang-questions] Question about lists:filter In-Reply-To: References: Message-ID: <1377691302.4778.14.camel@sekic1152.release> Greetings, isPal is an atom. If you want to make it into a fun you have to write: fun isPal/1 bengt On Wed, 2013-08-28 at 14:54 +0300, Nadav Chernin wrote: > Hi, all > > > I'm learning Erlang with Project Euler. > > > This is my solution for problem 4. > > > > > -module(euler). > -export([problem4/0]). > > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())). > > > Before it, my code was: > > > -module(euler). > -export([problem4/0]). > > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > problem4()->lists:max(lists:filter(isPal,cand())). > > > Difference from my final solution - using isPal instead > fun(A)->isPal(A) end > > > But exception occured during execution: > > > exception error: no function clause matching > lists:filter(isPal, > > [10000,10100,10200,10300,10400,10500,10600, > > 10700,10800,10900,11000,11100,11200,11300, > > 11400,11500,11600,11700,11800,11900,12000, > > 12100,12200,12300,12400,12500,12600|...]) > > > Please, explain me why i can't use function as is, without using > lambda function. > I come to Erlang from Haskell, and there i can do this, > > > Thanks, Nadav > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From peter@REDACTED Wed Aug 28 14:17:49 2013 From: peter@REDACTED (Peter Membrey) Date: Wed, 28 Aug 2013 14:17:49 +0200 (CEST) Subject: [erlang-questions] Is ei_xreceive_msg() thread safe? In-Reply-To: <521DCB14.6030805@erix.ericsson.se> References: <1738474072.1073.1377588140542.JavaMail.zimbra@membrey.hk> <1124670680.1085.1377588647393.JavaMail.zimbra@membrey.hk> <1396147141.1692.1377617279511.JavaMail.zimbra@membrey.hk> <521DCB14.6030805@erix.ericsson.se> Message-ID: <1111093852.2336.1377692269179.JavaMail.zimbra@membrey.hk> Hi Sverker, Coincidentally with some help of a colleague and friend, I have implemented your suggested solution. It seems like a good half way house and means I don't need to put a sleep in there and poll the socket directly. I am a bit worried about the receive though. The timeout is necessary (although hopefully rarely required) but if it means I could only get a partial message, that would obviously be very bad. On the other hand, ending up blocked inside the mutex would also be very bad. I did come up with one potential solution for this problem - I could modify ei_xreceive() to accept a mutex as an argument. Then if a heart beat request is received, the mutex can be applied either side of the socket operation. This would mean that a mutex lock would only be taken when it was actually needed and would also mean that I wouldn't have to use the _tmo version (which could corrupt data) as even if the request blocks, it won't be in a critical section and so won't block the senders. The change would add a new function, but wouldn't alter the behaviour of existing functions. It seems a simple and elegant solution, but I'm not sure there'd be much support in accepting such a change upstream :) Cheers, Pete ----- Original Message ----- From: "Sverker Eriksson" To: "Peter Membrey" Cc: "Serge Aleynikov" , "Erlang Questions" Sent: Wednesday, August 28, 2013 6:04:04 PM Subject: Re: [erlang-questions] Is ei_xreceive_msg() thread safe? On 08/27/2013 05:27 PM, Peter Membrey wrote: Hi Serge, Thank you very much for clarifying this! At least I know where everything stands now. Does the scenario I'm talking about make sense? Could a heartbeat in the current design end up interleaved with other data? I suspect it would be pretty rare, but we pump 10 million or more messages through this app per day, so even if the chance was a million to one, we'd end up seeing it ten times a day on average... Yes, it make sense. The ei_send/receive API is not thread safe for the same file descriptor.
I'm thinking for now that I'll give Robert's suggestion a try. If I call ei_xreceive() with a timeout and wrap a mutex around it, would that then protect the socket? I'm assuming (probably a bad thing) that it doesn't reply in a different thread or something... I guess it would be a bit wasteful to effectively poll the socket every millisecond or so (I can't really block the other threads from writing longer than that) but it seems to be a potential way to resolve this issue in the short term while I come up with a better fix. Does that sound like a reasonable plan?
You could use select() to wait for a message to arrive on the socket. Then lock your mutex and call ei_xreceive(). It might not be such a good idea to poll by repeated calls to ei_xreceive() with a short timeout. You can get situations when the timeout triggers after receiving half a message. That will make all following calls to ei_xreceive() get out of synch and probably fail. I guess the probability of this scenario depends on the message sizes and the speed and reliability of your connection. Calling ei_xreceive() without a timeout and my suggestion with select() on the other hand could make a hickup on the net to cause all your sending threads to freeze waiting for the mutex. /Sverker
Thanks again! Kind Regards, Peter Membrey ----- Original Message ----- From: "Serge Aleynikov" To: "Peter Membrey" Cc: "Erlang Questions" Sent: Tuesday, August 27, 2013 10:59:27 PM Subject: Re: [erlang-questions] Is ei_xreceive_msg() thread safe? e i_send/receive family of functions are not thread safe, and receive functions do handle heartbeats internally. The way the ei_{connect,send,receive} functions are written it's not possible to do any non-blocking I/O with them or use them in multi-threaded code in the manner you described without modifying the functions. You can take a look at the alternative C++ library that doesn't have such limitations: https://github.com/saleyn/eixx , and offers almost all functionality that ei has. Regards, Serge On Tue, Aug 27, 2013 at 3:30 AM, Peter Membrey < peter@REDACTED > wrote: Hi guys, I've got a fairly basic C Node set up where I have the main thread running in a loop with ei_xreceive_msg() and a number of "callback" threads that execute functions and write data using ei_send() to the shared socket (connecting to the Erlang node). Originally I had a lot of data corruption (the Erlang node crashing due to corrupt data) because of incorrect locking on socket writes. I added mutexes to the ei_send() calls and this problem seemed to go away. However I've had a couple of occasions where the system has been quiet and then suddenly become busy where corrupt data has still been sent to the Erlang node. I'm positive all the places where I do ei_send() are protected, but that got me wondering about ei_xreceive_msg(). >From what I can find, ei_xreceive_msg() automatically handles heartbeats for you and I guess that means it will send some sort of reply on that socket. If the heart beat is being sent at the same time as some other process tries to write to the socket, is it possible that the two could get interleaved or something? I would honestly have thought it unlikely but I'm running out of ideas. Assuming it's possible, how could I add a mutex in this case? The call itself blocks, so I can't wrap the whole call in the mutex else nothing else will be able to send data, and there's no way to pass a mutex into the call itself. So as far as I can tell, there's no way to protect these writes and prevent them from getting mixed up with other writes on that socket. Does anyone have any ideas? I'm quite willing to accept I could be doing something pretty stupid, but I'm really out of ideas as to what that might be... Thanks in advance! Kind Regards, Peter Membrey _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions
-------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavchernin@REDACTED Wed Aug 28 14:48:10 2013 From: nadavchernin@REDACTED (Nadav Chernin) Date: Wed, 28 Aug 2013 15:48:10 +0300 Subject: [erlang-questions] Question about lists:filter In-Reply-To: <1377691302.4778.14.camel@sekic1152.release> References: <1377691302.4778.14.camel@sekic1152.release> Message-ID: Hi, thanks for your quick answer. Nadav On Wed, Aug 28, 2013 at 3:01 PM, Bengt Kleberg wrote: > Greetings, > > isPal is an atom. If you want to make it into a fun you have to write: > fun isPal/1 > > > bengt > > On Wed, 2013-08-28 at 14:54 +0300, Nadav Chernin wrote: > > Hi, all > > > > > > I'm learning Erlang with Project Euler. > > > > > > This is my solution for problem 4. > > > > > > > > > > -module(euler). > > -export([problem4/0]). > > > > > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > > problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())). > > > > > > Before it, my code was: > > > > > > -module(euler). > > -export([problem4/0]). > > > > > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > > problem4()->lists:max(lists:filter(isPal,cand())). > > > > > > Difference from my final solution - using isPal instead > > fun(A)->isPal(A) end > > > > > > But exception occured during execution: > > > > > > exception error: no function clause matching > > lists:filter(isPal, > > > > [10000,10100,10200,10300,10400,10500,10600, > > > > 10700,10800,10900,11000,11100,11200,11300, > > > > 11400,11500,11600,11700,11800,11900,12000, > > > > 12100,12200,12300,12400,12500,12600|...]) > > > > > > Please, explain me why i can't use function as is, without using > > lambda function. > > I come to Erlang from Haskell, and there i can do this, > > > > > > Thanks, Nadav > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Aug 29 00:22:09 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Thu, 29 Aug 2013 10:22:09 +1200 Subject: [erlang-questions] Question about lists:filter In-Reply-To: References: Message-ID: <58C3B481-8D0B-4D4B-9E2B-BA5CF5D5C6D2@cs.otago.ac.nz> On 28/08/2013, at 11:54 PM, Nadav Chernin wrote: > Hi, all > > I'm learning Erlang with Project Euler. > > This is my solution for problem 4. > > > -module(euler). > -export([problem4/0]). > > cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)]. > isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())). Other people have pointed out that "isPal" should have been "fun isPal/1". Let me be the one to remind you that your keyboard has a space key. candidates() -> Range = lists:seq(100, 999), [X*Y || X <- Range, Y <- Range]. is_palindrome(X) -> integer_to_list(X) == lists:reverse(integer_to_list(X)). problem_4() -> lists:max([A || A <- candidates(), is_palindrome(A)]). is much easier to read. Now it is easy to see that candidates() contains a _lot_ of duplication. We cannot distinguish X * Y from Y * X, so why generate both? candidates() -> Range = lists:seq(100, 999), [X * Y || X <- Range, Y <- Range, Y >= X]. % ^^^^^^ Then there is the question about why you were bothering to pass is_palindrome/1 as an argument at all instead of using a list comprehension: Another thing that might speed this up is noting that you are generating numbers in the range 10,000 to 999,801, so problem_4() -> lists:max([A || A <- candidates(), case integer_to_list(A) of [X,Y,_,Y,X] -> true ; [X,Y,Z,Z,Y,X] -> true ; _ -> false end]). makes half as many lists. From ludovic@REDACTED Thu Aug 29 00:29:39 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Thu, 29 Aug 2013 00:29:39 +0200 Subject: [erlang-questions] Question about lists:filter In-Reply-To: References: Message-ID: lists:filter/2 want a fun for it first argument. isPal is not a fun but an atom. You must use lists:filter(fun isPal/1, [...]). Remember, in Erlang, as several functions can share the same name (with different arities), a complete function name goes with arity. so : fun functionName/Arity Cheers Le Wed, 28 Aug 2013 13:54:24 +0200, Nadav Chernin a ?crit: > Hi, all > > I'm learning Erlang with Project Euler. > > This is my solution for problem 4. > > > *-module(euler).* > *-export([problem4/0]).* > * > * > *cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)].* > *isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)).* > *problem4()->lists:max(lists:filter(fun(A)->isPal(A) end,cand())).* > * > * > Before it, my code was: > > *-module(euler).* > *-export([problem4/0]).* > * > * > *cand()->[X*Y||X<-lists:seq(100,999),Y<-lists:seq(100,999)].* > *isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)).* > *problem4()->lists:max(lists:filter(isPal,cand())).* > * > * > Difference from my final solution - using isPal instead fun(A)->isPal(A) > end > > But exception occured during execution: > > exception error: no function clause matching > lists:filter(isPal, > [10000,10100,10200,10300,10400,10500,10600, > 10700,10800,10900,11000,11100,11200,11300, > 11400,11500,11600,11700,11800,11900,12000, > 12100,12200,12300,12400,12500,12600|...]) > > Please, explain me why i can't use function as is, without using lambda > function. > I come to Erlang from Haskell, and there i can do this, > > Thanks, Nadav -- Utilisant le logiciel de courrier r?volutionnaire d'Opera : http://www.opera.com/mail/ From ok@REDACTED Thu Aug 29 07:05:26 2013 From: ok@REDACTED (ok@REDACTED) Date: Thu, 29 Aug 2013 17:05:26 +1200 Subject: [erlang-questions] Question about lists:filter In-Reply-To: <58C3B481-8D0B-4D4B-9E2B-BA5CF5D5C6D2@cs.otago.ac.nz> References: <58C3B481-8D0B-4D4B-9E2B-BA5CF5D5C6D2@cs.otago.ac.nz> Message-ID: <73d7d2832292203a258ad02310151d8a.squirrel@chasm.otago.ac.nz> I proposed (a) eliminating obvious duplication from cand() and (b) checking palindromicity by pattern matching rather than making two copies of a list and reversing one of them. I _believed_ the result should be faster. reductions: go down by a factor of 1.88; run time: goes down by a factor of 3.95. >> isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). One thing that will surprise anyone coming to Erlang from Haskell is that the two calls to integer_to_list(X) are _not_ merged into a single call by the compiler, so for an n-digit number, you get 3n list cells here. Since Erlang functions in general can have side effects, it is in general unsafe to merge calls to the same function with the same arguments. In the Erlang version I tried, the compiler treated integer_to_list/1 like any other function. From fxmywc@REDACTED Thu Aug 29 09:03:29 2013 From: fxmywc@REDACTED (fxmy wang) Date: Thu, 29 Aug 2013 15:03:29 +0800 Subject: [erlang-questions] How to get a worker's pid() after restarted by supervisor? Message-ID: Hi dear fellow erlangers, New to Erlang, I'm building a small comment backend for practice. By my design there are many worker processes that handle comments posted from different URLs, and these workers are all under simple_one_for_one supervision. I also decide to start a dispatcher process to map certain URL comment to corresponding worker by worker_sup:start_child(URL) and then store returned Pid. So here comes the problem, I expect that worker may be restarted by supervisor. Is there a way to get the worker's Pid after restarted by supervisor, so that I can keep track of which URL comment should go for which worker process? Hope express myself clearly cause not a native speaker :) B.R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Thu Aug 29 11:10:31 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Thu, 29 Aug 2013 12:10:31 +0300 Subject: [erlang-questions] How to get a worker's pid() after restarted by supervisor? In-Reply-To: References: Message-ID: Hello, You have to maintain mapping of uri to pid in external table (e.g ETS) However, there is number of open source project that addresses this issue. For example gproc by Ulf. However, I had exactly the same task / problem like your and made a library to manage those processes. You can find it here: http://github.com/fogfish/pts Best Regards, Dmitry >-|-|-*> On 29.8.2013, at 10.03, fxmy wang wrote: > > Hi dear fellow erlangers, > > New to Erlang, I'm building a small comment backend for practice. By my design there are many worker processes that handle comments posted from different URLs, and these workers are all under simple_one_for_one supervision. > > I also decide to start a dispatcher process to map certain URL comment to corresponding worker by worker_sup:start_child(URL) and then store returned Pid. > > So here comes the problem, I expect that worker may be restarted by supervisor. Is there a way to get the worker's Pid after restarted by supervisor, so that I can keep track of which URL comment should go for which worker process? > > Hope express myself clearly cause not a native speaker :) > > B.R. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ulf@REDACTED Thu Aug 29 12:35:30 2013 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 29 Aug 2013 12:35:30 +0200 Subject: [erlang-questions] mnesia and storage_properties Message-ID: <58788D1A-345D-4ACA-B25D-75CEDE3D3780@feuerlabs.com> Playing around with mnesia storage properties, I decided to check what happens if e.g. duplicate {ets,Ps} entries are given for the same table. First, I note that mnesia allows it. Second, I note that ets doesn't allow me to check concurrency "tweaks" via ets:info/2 - bummer. Still... Eshell V5.10.2 (abort with ^G) 1> mnesia:start(). ok 2> dbg:tracer(), dbg:tp(ets,new,[]), dbg:p(all,[c]). {ok,[{matched,nonode@REDACTED,40}]} 3> mnesia:create_table(t,[{storage_properties,[{ets,[{write_concurrency,true},{read_concurrency,true}]}]}]). (<0.44.0>) call ets:new(mnesia_trans_store,[bag,public]) (<0.40.0>) call ets:new(t,[{keypos,2}, public,named_table,set, {write_concurrency,true}, {read_concurrency,true}]) {atomic,ok} 4> mnesia:delete_table(t). (<0.44.0>) call ets:new(mnesia_trans_store,[bag,public]) {atomic,ok} 5> mnesia:create_table(t,[{storage_properties,[{ets,[{write_concurrency,true}]},{ets,[{read_concurrency,true}]}]}]). (<0.44.0>) call ets:new(mnesia_trans_store,[bag,public]) (<0.40.0>) call ets:new(t,[{keypos,2},public,named_table,set,{read_concurrency,true}]) {atomic,ok} So, in the second case, when the two ets properties are given as separate {ets, Ps} entries, mnesia accepts the definition, but ignores the *first* entry. 6> rp(mnesia:table_info(t,cstruct)). {cstruct,t,set, [nonode@REDACTED], [],[],0,read_write,false,[],[],false,t, [key,val], [],[], [{ets,[{read_concurrency,true}]}, {ets,[{write_concurrency,true}]}], {{1377,772088,582589},nonode@REDACTED}, {{2,0},[]}} The problem (unless it is that mnesia_schema doesn't merge the properties, or disallow multiple entries in the first place) seems to be in mnesia_loader: create_table(Tab, TabSize, Storage, Cs) -> StorageProps = val({Tab, storage_properties}), if ...; (Storage == ram_copies) or (Storage == disc_copies) -> EtsOpts = proplists:get_value(ets, StorageProps, []), Args = [{keypos, 2}, public, named_table, Cs#cstruct.type | EtsOpts], end. Note that mnesia_schema seems to have reversed the list before storing it in the cstruct. In fact, what it does is call lists:sort/1 on the list. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From yash_ganthe@REDACTED Thu Aug 29 13:31:18 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 29 Aug 2013 11:31:18 +0000 Subject: [erlang-questions] Monitoring distributed nodes Message-ID: I found github.com/beamspirit/bigwig. But it has no support for monitoring multiple nodes the way appmon does. Regards, Yash From: Yash Ganthe Sent: Thursday, August 22, 2013 6:40 PM To: erlang-questions@REDACTED Subject: Monitoring distributed nodes Hi, We have a Master erlang node that has an application with a supervisor and multiple, dynamically added worker processes. For each worker process, there is another erlang node dynamically started. We would like to monitor all nodes on one screen and detect failures so that corrective action can be taken. Is there an utility that can let us do this? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From js@REDACTED Thu Aug 29 14:11:03 2013 From: js@REDACTED (Schneider) Date: Thu, 29 Aug 2013 14:11:03 +0200 Subject: [erlang-questions] Mini Server does not reciv Message-ID: <521F3A57.5070703@globe.de> Hi List, I tried playing around with a baby example of a server, just printing the data it receives. But I don't get any output: the Code: > -module(mini_web_srv). > > -export([start/0,send/0]). > > start() -> > { ok, Listen } = gen_tcp:listen(8080, > [binary, > {packet, 4}, > {reuseaddr, true}, > {active, true} > ] > ), > spawn(fun() -> par_connect(Listen) end). > > par_connect(Listen) -> > {ok, Socket } = gen_tcp:accept(Listen), > spawn(fun() -> par_connect(Listen) end), > loop(Socket). > > loop(Socket) -> > receive > {tcp, Socket, Bin} -> > io:format("received ~p~n",[Bin]), > loop(Socket); > {tcp_closed, _} -> > io:format("closed connection"); > {tcp_error, Socket, Reason} -> > io:format("Error on socket ~p reason: ~p~n", [Socket, Reason]) > end. > > send() -> > send_data("foo"). > > send_data(Data) -> > {ok, Send} = gen_tcp:connect("localhost", 8080,[binary, {packet, > 0} ]), > gen_tcp:send(Send, Data). and the commands used in the EShell: > Eshell V5.8.5 (abort with ^G) > 1> c("/home/j_schn14/src/erlang/mini_web_srv/mini_web_srv", [{outdir, > "/home/j_schn14/src/erlang/mini_web_srv/"}]). > c("/home/j_schn14/src/erlang/mini_web_srv/mini_web_srv", [{outdir, > "/home/j_schn14/src/erlang/mini_web_srv/"}]). > {ok,mini_web_srv} > 2> mini_web_srv:start(). > <0.39.0> > 3> > 3> > 3> mini_web_srv:send(). > Accepted Connection on #Port<0.2075> > ok > 4> > 4> {ok, SendSocket} = gen_tcp:connect("localhost", 8080, [binary, > {packet, 0} ]). > Accepted Connection on #Port<0.2077> > {ok,#Port<0.2076>} > 5> > 5> gen_tcp:send(SendSocket,<<"foo">>). > ok > 6> > 6> gen_tcp:close(SendSocket). > ok > closed connection7> > 7> I would expect after line 3 (to be exact, after the "Accepted Connection on #Port<0.2075>") and after line 5 to get "received foo" e.g. "received <>" printed on the screen. what am I doing wrong? bg, Johannes -- GLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nsterGLOBE Development GmbH K?nigsberger Strasse 260 48157 M?nster 0251/5205 390 From yash_ganthe@REDACTED Thu Aug 29 14:45:36 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 29 Aug 2013 12:45:36 +0000 Subject: [erlang-questions] odbc module invoking stored procedure with OUT params In-Reply-To: <6FB0C92A7B7B0840A0AA3530CEDA268565D7D47C@HJ-MBX2.persistent.co.in> References: <6FB0C92A7B7B0840A0AA3530CEDA268565D7D47C@HJ-MBX2.persistent.co.in> Message-ID: Hi, We have a DB2 stored proc that takes OUT params and returns a CURSOR so that our Erlang module can get a List of records using the odbc module. We invoke the stored proc using the code: Params = [{{sql_varchar, 40}, out, [[]]},{{sql_varchar, 40}, out, [[]]}]. {selected, Results} = odbc:param_query(Ref,"call SP_NAME(?,?)",Params). However, when we execute the stored proc, the OUT params get set with a value that indicates that the proc execution is successful, but no records are fetched from the cursor into the list. The output of param_query is : {executed,2,[{"00000","NO ERROR"}]} Can you suggest what needs to be done to read the OUT params as well as the records from the stored proc results? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From i.aadi07@REDACTED Thu Aug 29 15:32:46 2013 From: i.aadi07@REDACTED (aadi i) Date: Thu, 29 Aug 2013 06:32:46 -0700 (PDT) Subject: [erlang-questions] Slow response of post method of https request using httpc erlang module In-Reply-To: References: Message-ID: <23021060-7364-4448-b225-dcb0e3fd6bd5@googlegroups.com> Hi Bob, We send thousands of httpc requests(post) in parallel to server. Which internally does ssl communication for every request. What do you mean by " configuring httpc to allow more parallel connections"? How can we do it. Thanks, Aadi On Saturday, 24 August 2013 22:21:57 UTC+5:30, Bob Ippolito wrote: > > Have you configured httpc to allow more parallel connections to your > external server? > > On Saturday, August 24, 2013, aadi i wrote: > >> Our Application (which uses existing Erlang OTP R15B01 modules) sends >> https request to external authentication server..and it gets reply and >> seems work fine under normal cases >> >> But under heavy loads some requests are failing since they are >> consuming more time to do ssl handshake >> >> and i have observed the following things during ssl handshake >> >> client is taking(our application ) nearly 80 sec to send the certificate >> after server hello is done with server certificate >> >> since our server expects to complete the request- response in 30 sec >> otherwise it drops the connection hence results to connection failures and >> effects the performance of application severely >> >> and finally i would like to know >> - Is our application failing to invoke the client certificate quickly >> .? i mean does httpc module do the file/IO related operations to invoke the >> certificates which results to slow response under heavy loads..? >> >> -Does Erlang has any limitations in ssl handshake procedure ..? >> >> >> Thanks in Advance :-) >> >> >> adinarayana murthy >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Thu Aug 29 15:52:35 2013 From: dmercer@REDACTED (David Mercer) Date: Thu, 29 Aug 2013 08:52:35 -0500 Subject: [erlang-questions] Question about lists:filter In-Reply-To: <73d7d2832292203a258ad02310151d8a.squirrel@chasm.otago.ac.nz> References: <58C3B481-8D0B-4D4B-9E2B-BA5CF5D5C6D2@cs.otago.ac.nz> <73d7d2832292203a258ad02310151d8a.squirrel@chasm.otago.ac.nz> Message-ID: <020901cea4bf$049cb250$0dd616f0$@gmail.com> On Thursday, August 29, 2013, Richard A. O'Keefe wrote: > >> isPal(X)->integer_to_list(X)==lists:reverse(integer_to_list(X)). > > One thing that will surprise anyone coming to Erlang from Haskell is > that the two calls to integer_to_list(X) are _not_ merged into a single > call by the compiler, so for an n-digit number, you get 3n list cells > here. Curiously enough, though, timingwise the following 2 seem to do roughly the same: is_palindrome1(X) -> integer_to_list(X) == lists:reverse(integer_to_list(X)). is_palindrome2(X) -> Digits = integer_to_list(X), Digits == lists:reverse(Digits). Cheers, DBM From vinoski@REDACTED Thu Aug 29 16:47:55 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Thu, 29 Aug 2013 10:47:55 -0400 Subject: [erlang-questions] Mini Server does not reciv In-Reply-To: <521F3A57.5070703@globe.de> References: <521F3A57.5070703@globe.de> Message-ID: On Thu, Aug 29, 2013 at 8:11 AM, Schneider wrote: > Hi List, > > I tried playing around with a baby example of a server, just printing the > data it receives. But I don't get any output: > > the Code: > > -module(mini_web_srv). >> >> -export([start/0,send/0]). >> >> start() -> >> { ok, Listen } = gen_tcp:listen(8080, >> [binary, >> {packet, 4}, >> > > send_data(Data) -> >> {ok, Send} = gen_tcp:connect("localhost", 8080,[binary, {packet, 0} >> ]), >> gen_tcp:send(Send, Data). >> > It fails because you open the server with {packet,4} and the client with {packet,0}. The client and server must agree on the packet value for them to be able to communicate successfully. To make it work you can either remove the packet option entirely from both client and server, or set the client to {packet,4} to match the server. You might want to look at https://gist.github.com/vinoski/4996859 to see an example of a very small yet operational web server. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Thu Aug 29 17:43:27 2013 From: bob@REDACTED (Bob Ippolito) Date: Thu, 29 Aug 2013 08:43:27 -0700 Subject: [erlang-questions] Slow response of post method of https request using httpc erlang module In-Reply-To: <23021060-7364-4448-b225-dcb0e3fd6bd5@googlegroups.com> References: <23021060-7364-4448-b225-dcb0e3fd6bd5@googlegroups.com> Message-ID: http://erlang.org/doc/man/httpc.html see set_options and max_sessions. On Thursday, August 29, 2013, aadi i wrote: > Hi Bob, > > We send thousands of httpc requests(post) in parallel to server. Which > internally does ssl communication for every request. > > What do you mean by " configuring httpc to allow more parallel > connections"? How can we do it. > > Thanks, > Aadi > > On Saturday, 24 August 2013 22:21:57 UTC+5:30, Bob Ippolito wrote: >> >> Have you configured httpc to allow more parallel connections to your >> external server? >> >> On Saturday, August 24, 2013, aadi i wrote: >> >>> Our Application (which uses existing Erlang OTP R15B01 modules) sends >>> https request to external authentication server..and it gets reply and >>> seems work fine under normal cases >>> >>> But under heavy loads some requests are failing since they are >>> consuming more time to do ssl handshake >>> >>> and i have observed the following things during ssl handshake >>> >>> client is taking(our application ) nearly 80 sec to send the certificate >>> after server hello is done with server certificate >>> >>> since our server expects to complete the request- response in 30 sec >>> otherwise it drops the connection hence results to connection failures and >>> effects the performance of application severely >>> >>> and finally i would like to know >>> - Is our application failing to invoke the client certificate quickly >>> .? i mean does httpc module do the file/IO related operations to invoke the >>> certificates which results to slow response under heavy loads..? >>> >>> -Does Erlang has any limitations in ssl handshake procedure ..? >>> >>> >>> Thanks in Advance :-) >>> >>> >>> adinarayana murthy >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Aug 30 03:01:47 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 30 Aug 2013 13:01:47 +1200 Subject: [erlang-questions] Question about lists:filter In-Reply-To: <020901cea4bf$049cb250$0dd616f0$@gmail.com> References: <58C3B481-8D0B-4D4B-9E2B-BA5CF5D5C6D2@cs.otago.ac.nz> <73d7d2832292203a258ad02310151d8a.squirrel@chasm.otago.ac.nz> <020901cea4bf$049cb250$0dd616f0$@gmail.com> Message-ID: <8F218518-8E50-4296-BCE7-B721B59572C2@cs.otago.ac.nz> On 30/08/2013, at 1:52 AM, David Mercer wrote: > > Curiously enough, though, timingwise the following 2 seem to do roughly the same: > > is_palindrome1(X) -> > integer_to_list(X) == lists:reverse(integer_to_list(X)). > > is_palindrome2(X) -> > Digits = integer_to_list(X), > Digits == lists:reverse(Digits). > Presumably integer_to_list/1 does the bulk of its work in C. There are two kinds of cost involved in code like this: the time it takes to build the garbage and the space the garbage takes. The two versions both create 3N list cells which almost instantly become garbage. The most space-efficient (and the most time-efficient) method, at least for unboxed integers, is to do the palindrome test with arithmetic, not using lists at all. original code: 9.88 seconds (integer_to_list twice, reverse once) revised list: 3.06 seconds (integer_to_list once, pattern match) arithmetic: 2.14 seconds (no lists at all) From garry@REDACTED Fri Aug 30 04:16:33 2013 From: garry@REDACTED (Garry Hodgson) Date: Thu, 29 Aug 2013 22:16:33 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: Message-ID: <52200081.3070302@research.att.com> i would be love to get together with some other erlang folks, if this is still on the table. On 04/10/2013 02:59 PM, Joe Armstrong wrote: > Hello, > > Somebody, and I forget who asked me, and I forget when, to drop them a > line > if I was every in New York. > > I'll be in New York from 10 - 14 sept, if anybody is interested in a > meetup. > > /Joe -- Garry Hodgson AT&T Chief Security Office (CSO) "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." From heinz@REDACTED Fri Aug 30 04:39:20 2013 From: heinz@REDACTED (Heinz Nikolaus Gies) Date: Thu, 29 Aug 2013 22:39:20 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <52200081.3070302@research.att.com> References: <52200081.3070302@research.att.com> Message-ID: <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> I'll do my best to make it too :) Cheers, Heinz On Aug 29, 2013, at 22:16, Garry Hodgson wrote: > i would be love to get together with some other > erlang folks, if this is still on the table. > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >> Hello, >> >> Somebody, and I forget who asked me, and I forget when, to drop them a line >> if I was every in New York. >> >> I'll be in New York from 10 - 14 sept, if anybody is interested in a meetup. >> >> /Joe > > -- > Garry Hodgson > AT&T Chief Security Office (CSO) > > "This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited." > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From roberto@REDACTED Fri Aug 30 05:06:34 2013 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 29 Aug 2013 23:06:34 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> Message-ID: I moved to NYC. If this happens I'm in too. r. On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies wrote: > I'll do my best to make it too :) > > Cheers, > Heinz > On Aug 29, 2013, at 22:16, Garry Hodgson wrote: > > > i would be love to get together with some other > > erlang folks, if this is still on the table. > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > >> Hello, > >> > >> Somebody, and I forget who asked me, and I forget when, to drop them a > line > >> if I was every in New York. > >> > >> I'll be in New York from 10 - 14 sept, if anybody is interested in a > meetup. > >> > >> /Joe > > > > -- > > Garry Hodgson > > AT&T Chief Security Office (CSO) > > > > "This e-mail and any files transmitted with it are AT&T property, are > confidential, and are intended solely for the use of the individual or > entity to whom this e-mail is addressed. If you are not one of the named > recipient(s) or otherwise have reason to believe that you have received > this message in error, please notify the sender and delete this message > immediately from your computer. Any other use, retention, dissemination, > forwarding, printing, or copying of this e-mail is strictly prohibited." > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Fri Aug 30 05:17:52 2013 From: bob@REDACTED (Bob Ippolito) Date: Thu, 29 Aug 2013 20:17:52 -0700 Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> Message-ID: I'm in NYC from Sep 5-13 so I could probably also attend On Thursday, August 29, 2013, Roberto Ostinelli wrote: > I moved to NYC. If this happens I'm in too. > > r. > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies > > wrote: > >> I'll do my best to make it too :) >> >> Cheers, >> Heinz >> On Aug 29, 2013, at 22:16, Garry Hodgson > >> wrote: >> >> > i would be love to get together with some other >> > erlang folks, if this is still on the table. >> > >> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >> >> Hello, >> >> >> >> Somebody, and I forget who asked me, and I forget when, to drop them a >> line >> >> if I was every in New York. >> >> >> >> I'll be in New York from 10 - 14 sept, if anybody is interested in a >> meetup. >> >> >> >> /Joe >> > >> > -- >> > Garry Hodgson >> > AT&T Chief Security Office (CSO) >> > >> > "This e-mail and any files transmitted with it are AT&T property, are >> confidential, and are intended solely for the use of the individual or >> entity to whom this e-mail is addressed. If you are not one of the named >> recipient(s) or otherwise have reason to believe that you have received >> this message in error, please notify the sender and delete this message >> immediately from your computer. Any other use, retention, dissemination, >> forwarding, printing, or copying of this e-mail is strictly prohibited." >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED > 'erlang-questions@REDACTED');> >> > http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED > 'erlang-questions@REDACTED');> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fxmywc@REDACTED Fri Aug 30 03:43:09 2013 From: fxmywc@REDACTED (fxmy wang) Date: Fri, 30 Aug 2013 09:43:09 +0800 Subject: [erlang-questions] How to get a worker's pid() after restarted by supervisor? In-Reply-To: References: Message-ID: Thanks Dmitry for the advise :) I worked out a way like bellow to automatically get informed new worker process after restarted by supervisor. If you can't get supervisor to voluntarily send restarted worker Pid, you just let the worker himself to inform you :| Though it seems kind of "ugly" ================================== =====dispatcher.erl ================================== -behaviour(gen_server). %% Locally register( dispatcher, self() ). ... handle_call( {startWorker, Args}, _From, State) -> worker_sup:start_child( Arg), receive {workerStarted, Args, Pid} -> %% store {Args, Pid} into ets table & monitor this Pid. {reply, {ok, Pid}, State} after ?TIMEOUT -> {reply, {cantStartWorker, Args}, State} end. %%This should happen only when a worker is restarted by supervisor handle_info( {workerStared, Args, Pid}, State) -> %% Update ets table & monitor Pid {noreply, State}; %% This should happen when a worker exits normally handle_info( {'DOWN', _Ref, process, Pid, normal}, State) -> %% Update ets table {noreply, State}. ======================================== =====worker.erl ======================================== -behaviour(gen_server). init(Args) -> %% Do things necessary dispatcher ! {workerStarted, Args, self() }, {ok, State}. So far seems good ,cheers XD B.R. 2013/8/29 Dmitry Kolesnikov > Hello, > > You have to maintain mapping of uri to pid in external table (e.g ETS) > However, there is number of open source project that addresses this issue. > For example gproc by Ulf. > > However, I had exactly the same task / problem like your and made a > library to manage those processes. You can find it here: > http://github.com/fogfish/pts > > > Best Regards, > Dmitry >-|-|-*> > > > On 29.8.2013, at 10.03, fxmy wang wrote: > > > > > Hi dear fellow erlangers, > > > > New to Erlang, I'm building a small comment backend for practice. By my > design there are many worker processes that handle comments posted from > different URLs, and these workers are all under simple_one_for_one > supervision. > > > > I also decide to start a dispatcher process to map certain URL comment > to corresponding worker by worker_sup:start_child(URL) and then store > returned Pid. > > > > So here comes the problem, I expect that worker may be restarted by > supervisor. Is there a way to get the worker's Pid after restarted by > supervisor, so that I can keep track of which URL comment should go for > which worker process? > > > > Hope express myself clearly cause not a native speaker :) > > > > B.R. > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From snaky@REDACTED Fri Aug 30 08:50:36 2013 From: snaky@REDACTED (Yuri Lukyanov) Date: Fri, 30 Aug 2013 10:50:36 +0400 Subject: [erlang-questions] [erlang-patches] cover & export_all In-Reply-To: References: <51DD171D.2090802@erlang.org> Message-ID: Hi Peti, Did you have a chance to push your fix to the main repo? On Wed, Jul 10, 2013 at 12:34 PM, Peti G?m?ri wrote: > Hi Fredrik, > > sorry, what about this one (maybe it was a wrong ?) > > git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all > > On Wed, Jul 10, 2013 at 9:11 AM, Fredrik wrote: >> >> On 07/10/2013 03:19 AM, Peti G?m?ri wrote: >> >> Hi OTP team, >> >> here is a patch that addresses the problem Yuri described. >> (I haven't added any tests or docs though) >> >> is it true that now you also accept pull requests? >> >> >> >> git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all >> >> >> https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all >> >> https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all.patch >> >> >> br >> Peter >> >> On Wed, Jul 10, 2013 at 12:34 AM, Peti G?m?ri >> wrote: >>> >>> Hi Yuri, >>> >>> You are right, while cover compiling from source works (you can use this >>> as a workaround): >>> > cover:compile(cover_test, UserOptions = [export_all, debug_info]). >>> {ok,cover_test} >>> > cover_test:test(). >>> ok >>> >>> (because UserOptions from the arguments is taken when cover recompiles >>> the instrumented forms) >>> when cover compiling from beam UserOptions = [] is taken. >>> This could be fixed in cover by taking the compile options from the beam >>> file as you assumed how it works. >>> >>> Actually the documentation still says that "Only options defining include >>> file directories and macros are passed to compile:file/2, everything else is >>> ignored." apparently a patch from Tobias Schlager added export_all to the >>> allowed options of cover:compile for the exact same use case as you had. But >>> this is missing from cover:compile_beam. >>> >>> May be I try to come up with a patch >>> >>> br >>> Peter >>> >>> >>> >>> On Tue, Jul 9, 2013 at 11:56 AM, Yuri Lukyanov wrote: >>>> >>>> It seems that there is no way to cover-compile modules with export_all. >>>> Here is a simple example: >>>> >>>> cover_test.erl: >>>> >>>> -module(cover_test). >>>> test() -> ok. >>>> >>>> >>>> $ erl >>>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] >>>> [async-threads:0] [hipe] [kernel-poll:false] >>>> >>>> Eshell V5.9.1 (abort with ^G) >>>> 1> c(cover_test, [debug_info,export_all]). >>>> {ok,cover_test} >>>> 2> cover_test:test(). >>>> ok >>>> 3> cover:compile_beam(cover_test). >>>> {ok,cover_test} >>>> 4> cover_test:test(). >>>> ** exception error: undefined function cover_test:test/0 >>>> 5> >>>> >>>> >>>> Could someone explain why it is like this? Is it done on purpose? >>>> Maybe it's a bug? >>>> The reason I want modules to be cover-compiled with +export_all is >>>> that it is sometimes convinient to have unit tests outside of a >>>> module. Before unit tests are run the modules get compiled with >>>> +export_all for tests to be able to access private functions. But the >>>> situation is that it is not possibe in this case to enable coverage >>>> analysys. >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> >> >> >> _______________________________________________ >> erlang-patches mailing list >> erlang-patches@REDACTED >> http://erlang.org/mailman/listinfo/erlang-patches >> >> Hello Peti, >> I am getting, fatal: Couldn't find remote ref pg?-cover-export-all when I >> am trying to fetch. >> Yes pull requests are accepted. >> >> -- >> >> BR Fredrik Gustafsson >> Erlang OTP Team > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From dmercer@REDACTED Fri Aug 30 15:17:38 2013 From: dmercer@REDACTED (David Mercer) Date: Fri, 30 Aug 2013 08:17:38 -0500 Subject: [erlang-questions] Question about lists:filter In-Reply-To: <8F218518-8E50-4296-BCE7-B721B59572C2@cs.otago.ac.nz> References: <58C3B481-8D0B-4D4B-9E2B-BA5CF5D5C6D2@cs.otago.ac.nz> <73d7d2832292203a258ad02310151d8a.squirrel@chasm.otago.ac.nz> <020901cea4bf$049cb250$0dd616f0$@gmail.com> <8F218518-8E50-4296-BCE7-B721B59572C2@cs.otago.ac.nz> Message-ID: <028801cea583$4d31a100$e794e300$@gmail.com> On Thursday, August 29, 2013, Richard A. O'Keefe wrote: > On 30/08/2013, at 1:52 AM, David Mercer wrote: > > > > > Curiously enough, though, timingwise the following 2 seem to do > roughly the same: > > > > is_palindrome1(X) -> > > integer_to_list(X) == lists:reverse(integer_to_list(X)). > > > > is_palindrome2(X) -> > > Digits = integer_to_list(X), > > Digits == lists:reverse(Digits). > > > > Presumably integer_to_list/1 does the bulk of its work in C. > There are two kinds of cost involved in code like this: the time it > takes to build the garbage and the space the garbage takes. The two > versions both create 3N list cells which almost instantly become > garbage. Why does the second one create 3N cells. Isn't it only 2: Digits and the result of lists:reverse(Digits)? DBM From mahesh@REDACTED Fri Aug 30 19:43:58 2013 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Fri, 30 Aug 2013 13:43:58 -0400 Subject: [erlang-questions] [ANN] erlang_cassandra - (yet another) thrift-based Cassandra client Message-ID: As the subject-line sez., this is yet another thrift-based erlang client for Cassandra, and is available here --> https://github.com/dieswaytoofast/erlang_cassandra There really isn't that much documentation - the whole thing is just a shim on the (currently) complete list of thrift commands as listed here So why this? Why not lpgauth's excellent cassanderl? No real reason - this uses a more recent version of thrift (whatever), and uses poolboy instead of discount. The semantics between the two are different, and our use case is more poolboy-ish than dispcount-ish. In any case, if you've got any questions, comments or issues, shoot me an email (or github/issue it)? Cheers Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Google+ | Blog | Twitter | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mallen@REDACTED Fri Aug 30 22:22:59 2013 From: mallen@REDACTED (Mark Allen) Date: Fri, 30 Aug 2013 15:22:59 -0500 Subject: [erlang-questions] Erlang APIs and JSON Message-ID: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> I am wondering if anyone has opinions on how to deserialize JSON HTTP request bodies into Erlang terms. Tuple wrapped proplists seems to be how it comes back from mochijson2:decode() and jiffy:decode(). Erlson returns orddicts hidden in a record-like syntax via a parse-transform. Does anyone further deserialize proplist(s) into records? Why or why not? What are the best (or even suggested) practices for dealing with JSON data? This might be a FAQ. If so, kindly point me toward the Fine Reading Material. Thanks. Mark Allen Sr Software Developer Alert Logic E: mallen@REDACTED From g@REDACTED Fri Aug 30 22:41:36 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 30 Aug 2013 15:41:36 -0500 Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: Hi Mark, On Fri, Aug 30, 2013 at 3:22 PM, Mark Allen wrote: > I am wondering if anyone has opinions on how to deserialize JSON HTTP request bodies into Erlang terms. Tuple wrapped proplists seems to be how it comes back from mochijson2:decode() and jiffy:decode(). Erlson returns orddicts hidden in a record-like syntax via a parse-transform. It's initially odd, but {Proplist} has emerged as a standard-ish representation of a JSON associative array in Erlang. It's better IMO than dealing with the "what is this list - a string or an array?" question. > Does anyone further deserialize proplist(s) into records? Why or why not? All the time. If your data prefers to be represented as records, do that -- and make the interface to JSON do the work once and be done with it. This is similar to pulling data out of a database -- every driver will have its own (weird) representation. I like to get it into the application-native format as quickly as possible, apply validation as early as possible. Alternatively, if your JSON is acting more as a lookup table -- strip off the one-tuple and use proplists:get_value - it's *very* efficient with small lists (based on rough measurements, < 200 items). > What are the best (or even suggested) practices for dealing with JSON data? I like the "make it work" and then "make it work better only as needed" best practice :) JSON and Erlang terms are similar enough that naively one might expect there to be zero conversion cost (both in time and logical complexity). I think if you give up on seamlessness and treat JSON as an arbitrarily complex format -- and each driver as arbitrarily inconvenient -- you just wrap the messiness and get on with it. > This might be a FAQ. If so, kindly point me toward the Fine Reading Material. Just my two cents. If you end up publishing this and generating Google ad revenue, I'd appreciate a beer. Garrett From essen@REDACTED Fri Aug 30 23:57:04 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 30 Aug 2013 23:57:04 +0200 Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: <52211530.5030103@ninenines.eu> Hi Mark, long time no see! On 08/30/2013 10:41 PM, Garrett Smith wrote: > Hi Mark, > > On Fri, Aug 30, 2013 at 3:22 PM, Mark Allen wrote: >> I am wondering if anyone has opinions on how to deserialize JSON HTTP request bodies into Erlang terms. Tuple wrapped proplists seems to be how it comes back from mochijson2:decode() and jiffy:decode(). Erlson returns orddicts hidden in a record-like syntax via a parse-transform. > > It's initially odd, but {Proplist} has emerged as a standard-ish > representation of a JSON associative array in Erlang. It's better IMO > than dealing with the "what is this list - a string or an array?" > question. The question doesn't come up in jsx because it doesn't use strings. So you got Proplist instead of {Proplist} and <<"string">> instead of "string". The JSON is still checked to be valid utf-8 even if it gives you binaries at the end so no functionality is lost. I don't think one is more used than the other though. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From bob@REDACTED Sat Aug 31 00:06:57 2013 From: bob@REDACTED (Bob Ippolito) Date: Fri, 30 Aug 2013 15:06:57 -0700 Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: <52211530.5030103@ninenines.eu> References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> <52211530.5030103@ninenines.eu> Message-ID: On Fri, Aug 30, 2013 at 2:57 PM, Lo?c Hoguin wrote: > Hi Mark, long time no see! > > > On 08/30/2013 10:41 PM, Garrett Smith wrote: > >> Hi Mark, >> >> On Fri, Aug 30, 2013 at 3:22 PM, Mark Allen >> wrote: >> >>> I am wondering if anyone has opinions on how to deserialize JSON HTTP >>> request bodies into Erlang terms. Tuple wrapped proplists seems to be how >>> it comes back from mochijson2:decode() and jiffy:decode(). Erlson returns >>> orddicts hidden in a record-like syntax via a parse-transform. >>> >> >> It's initially odd, but {Proplist} has emerged as a standard-ish >> representation of a JSON associative array in Erlang. It's better IMO >> than dealing with the "what is this list - a string or an array?" >> question. >> > > The question doesn't come up in jsx because it doesn't use strings. So you > got Proplist instead of {Proplist} and <<"string">> instead of "string". > The JSON is still checked to be valid utf-8 even if it gives you binaries > at the end so no functionality is lost. > > I don't think one is more used than the other though. How are 0-length arrays distinguished from 0-length proplists? This is why mochijson2 wraps proplists as {struct,P}. It also uses binaries for strings, and validates the UTF8. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.chamard@REDACTED Fri Aug 30 23:58:24 2013 From: david.chamard@REDACTED (David Chamard) Date: Fri, 30 Aug 2013 17:58:24 -0400 Subject: [erlang-questions] Erlang inter node communication issue Message-ID: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org> Hi Everyone, I am unable to get 2 erlang nodes to talk to each other through net_adm:ping. I am running on centos 6.3 and I am on amazon ec2. Port 4369 is open and I can telnet to it from both servers The cookies are the same when I do a get_cookie. When I try to do a net_adm:ping it hangs for a while and return a pang. I can resolve the hostname properly. Anyone could guide me on what the issue could be? Thank you, David From alisdairsullivan@REDACTED Sat Aug 31 00:30:22 2013 From: alisdairsullivan@REDACTED (alisdair sullivan) Date: Fri, 30 Aug 2013 15:30:22 -0700 (PDT) Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: Message-ID: <1377901822.55993.YahooMailAndroidMobile@web160306.mail.bf1.yahoo.com> the empty object is deserialized to '[{}]'. the proplists module considers this a valid proplist with zero keys -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sat Aug 31 00:44:22 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sat, 31 Aug 2013 00:44:22 +0200 Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: When I ran into the "how to treat JSON input?" question some time ago, this was my answer: https://github.com/eriksoe/valijate (I've been tempted to mention it several times recently when JSON was discussed on this list. This time I give in.) It's a tool for validating the JSON, *and* turn it into a form suitable for further use in Erlang. The form of the output follows the form of the typespec given to the validator: % A JSON value: > S = "{\"title\": \"Hogwarts, a History\", \"authors\":[\"Bathilda Bagshot\"], "++ " \"price\": 3.99, \"available\": true}". % A type/format specification: > T = {object, [{<<"authors">>, {array,string}}, {<<"title">>, string}, {<<"available">>, boolean}, {<<"price">>, number}]}. % Parse, validate and convert: > J = mochijson2:decode(S). > {ok,{Authors,Title,Available,Price}} = valijate:validate(J,T). => {ok,{[<<"Bathilda Bagshot">>], <<"Hogwarts, a History">>,true,3.99}} Valijate works with the data structures used by mochijson2 and ejson. I've tried to make it flexible (with an expressive typespec language), and with decent error reporting: > S2 = "{\"title\": \"Hogwarts, a History\", \"authors\":[\"Bathilda Bagshot\", 12], "++ " \"price\": 3.99, \"available\": true}". > E = valijate:validate(mochijson2:decode(S2),T). => {validation_error,[<<"authors">>,1], {wrong_type,12,number,string}} > valijate:error_to_english(E). => "At path .authors[1] : Value 12 has type number, but string was expected" When it turned out that I wanted to do similar validation of configuration values from Erlang's .config files, I even added support for validation of Erlang terms as well. If valijate turns out to be useful (or just nearly so) to anyone else, please tell me - then I could polish it up and do a proper [ANN] of it... /Erik 2013/8/30 Garrett Smith > Hi Mark, > > On Fri, Aug 30, 2013 at 3:22 PM, Mark Allen wrote: > > I am wondering if anyone has opinions on how to deserialize JSON HTTP > request bodies into Erlang terms. Tuple wrapped proplists seems to be how > it comes back from mochijson2:decode() and jiffy:decode(). Erlson returns > orddicts hidden in a record-like syntax via a parse-transform. > > It's initially odd, but {Proplist} has emerged as a standard-ish > representation of a JSON associative array in Erlang. It's better IMO > than dealing with the "what is this list - a string or an array?" > question. > > > Does anyone further deserialize proplist(s) into records? Why or why > not? > > All the time. If your data prefers to be represented as records, do > that -- and make the interface to JSON do the work once and be done > with it. This is similar to pulling data out of a database -- every > driver will have its own (weird) representation. I like to get it into > the application-native format as quickly as possible, apply validation > as early as possible. > > Alternatively, if your JSON is acting more as a lookup table -- strip > off the one-tuple and use proplists:get_value - it's *very* efficient > with small lists (based on rough measurements, < 200 items). > > > What are the best (or even suggested) practices for dealing with JSON > data? > > I like the "make it work" and then "make it work better only as > needed" best practice :) > > JSON and Erlang terms are similar enough that naively one might expect > there to be zero conversion cost (both in time and logical > complexity). I think if you give up on seamlessness and treat JSON as > an arbitrarily complex format -- and each driver as arbitrarily > inconvenient -- you just wrap the messiness and get on with it. > > > This might be a FAQ. If so, kindly point me toward the Fine Reading > Material. > > Just my two cents. If you end up publishing this and generating Google > ad revenue, I'd appreciate a beer. > > Garrett > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sat Aug 31 01:38:27 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sat, 31 Aug 2013 01:38:27 +0200 Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: 2013/8/30 Garrett Smith > Alternatively, if your JSON is acting more as a lookup table -- strip > off the one-tuple and use proplists:get_value - it's *very* efficient > with small lists (based on rough measurements, < 200 items). > I feel obliged to point out that for {Key,Value} lists, it usually makes better sense to use lists:keyfind/3 instead of functions from the proplists module. Performance-wise, because it's faster by a factor of 3x for small lists and 10-15x for lists of length 30+ (keyfind is implemented in C, and so is hard to beat). And arguably semantic-wise as well, because the proplists module don't work quite the way many believe (it's not just [{_,_}] it handles). Whether that micro-benchmark difference has any impact on your system as a whole depends of course on how heavily it's used -- it did on one I worked on once --, but my view is that, all other things being equal, there's no reason to adopt proplists as a habit if it offers no other advantages over more lightweight functions. (Assuming that 'dict' is the obvious alternative when dealing with lookup tables: My quick meaurements indicate that proplists:get_value() becomes slower than dict:fetch() at 6 elements; for lists:keyfind() vs. dict that doesn't happen until ~140 elements. At which point proplists is 16x slower than both.) Just my 2? - no nitpicking at all, it's just that you brought up performance... (in the same sentence as proplists) /Erik [like pushing a button] -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sat Aug 31 01:59:02 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sat, 31 Aug 2013 01:59:02 +0200 Subject: [erlang-questions] Erlang inter node communication issue In-Reply-To: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org> References: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org> Message-ID: For one thing, you should be aware that while EPMD is part of the the process, and therefore that it's true that its port should be accessible, the inter-node connection does not go through EPMD. For you, this probably means that you need to open other ports. Erlang can be told to use a port in a specific range - the "how" is described, for instance, here: https://sites.google.com/site/gettingalongwitherlang/home/distributed-programming/port-range-for-erlang-distribution-protocol the short version being: add the parameters -kernel inet_dist_listen_min ${FROM} inet_dist_listen_max ${TO} If I'm guessing correctly, you'll need to pick a port range, add those parameters to "erl", and open for the ports in that range. (To find out if I'm guessing correcly: run "epmd -names" on host A; this will tell you the port number of your Erlang node. Then, on host B, check if you can connect to that port, using e.g. "telnet A ".) Hoping this helps /Erik 2013/8/30 David Chamard > Hi Everyone, > > I am unable to get 2 erlang nodes to talk to each other through > net_adm:ping. > I am running on centos 6.3 and I am on amazon ec2. > > Port 4369 is open and I can telnet to it from both servers > > The cookies are the same when I do a get_cookie. > When I try to do a net_adm:ping it hangs for a while and return a pang. > I can resolve the hostname properly. > > Anyone could guide me on what the issue could be? > > Thank you, > > David > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Sat Aug 31 02:27:54 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 30 Aug 2013 17:27:54 -0700 (PDT) Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: +10 On Friday, August 30, 2013 6:38:27 PM UTC-5, Erik S?e S?rensen wrote: > > I feel obliged to point out that for {Key,Value} lists, it usually makes > better sense to use lists:keyfind/3 instead of functions from the proplists > module. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roe.adrian@REDACTED Sat Aug 31 08:00:33 2013 From: roe.adrian@REDACTED (Adrian Roe) Date: Sat, 31 Aug 2013 07:00:33 +0100 Subject: [erlang-questions] Erlang inter node communication issue In-Reply-To: References: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org> Message-ID: <14918805-B04B-4CD6-869F-74C05C80D6AC@gmail.com> In addition to Erik's comments, one thing we have found on ec2 instances is that you have to be particularly careful to ensure erlang nodes come up with the nodes' public IP addresses (for example we've never had any luck with erl -sname myname..., only explicitly using erl -name myname@REDACTED) It's worth a little hassle at the start - erlang and ec2 were made for each other :) Adrian Sent from my iPhone On 31 Aug 2013, at 00:59, Erik S?e S?rensen wrote: > For one thing, you should be aware that while EPMD is part of the the process, and therefore that it's true that its port should be accessible, the inter-node connection does not go through EPMD. > For you, this probably means that you need to open other ports. > Erlang can be told to use a port in a specific range - the "how" is described, for instance, here: > https://sites.google.com/site/gettingalongwitherlang/home/distributed-programming/port-range-for-erlang-distribution-protocol > the short version being: add the parameters > -kernel inet_dist_listen_min ${FROM} inet_dist_listen_max ${TO} > If I'm guessing correctly, you'll need to pick a port range, add those parameters to "erl", and open for the ports in that range. > > (To find out if I'm guessing correcly: run "epmd -names" on host A; this will tell you the port number of your Erlang node. Then, on host B, check if you can connect to that port, using e.g. "telnet A ".) > > Hoping this helps > /Erik > > > 2013/8/30 David Chamard >> Hi Everyone, >> >> I am unable to get 2 erlang nodes to talk to each other through net_adm:ping. >> I am running on centos 6.3 and I am on amazon ec2. >> >> Port 4369 is open and I can telnet to it from both servers >> >> The cookies are the same when I do a get_cookie. >> When I try to do a net_adm:ping it hangs for a while and return a pang. >> I can resolve the hostname properly. >> >> Anyone could guide me on what the issue could be? >> >> Thank you, >> >> David >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Sat Aug 31 08:59:00 2013 From: ivan@REDACTED (Ivan uemlianin) Date: Sat, 31 Aug 2013 07:59:00 +0100 Subject: [erlang-questions] Erlang inter node communication issue In-Reply-To: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org> References: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org> Message-ID: <5D0F6BB2-E151-460B-A630-6B24C6B3EEA0@llaisdy.com> Dear David Each nodes will need another port (or range of ports) available, and a full name. I had the same trouble: http://llaisdy.wordpress.com/2011/05/17/connecting-erlang-nodes/ Best wishes Ivan -- festina lente On 30 Aug 2013, at 22:58, David Chamard wrote: > Hi Everyone, > > I am unable to get 2 erlang nodes to talk to each other through net_adm:ping. > I am running on centos 6.3 and I am on amazon ec2. > > Port 4369 is open and I can telnet to it from both servers > > The cookies are the same when I do a get_cookie. > When I try to do a net_adm:ping it hangs for a while and return a pang. > I can resolve the hostname properly. > > Anyone could guide me on what the issue could be? > > Thank you, > > David > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ddosia@REDACTED Sat Aug 31 14:17:10 2013 From: ddosia@REDACTED (Daniil Churikov) Date: Sat, 31 Aug 2013 05:17:10 -0700 (PDT) Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: <86860bc5-a371-4949-9053-0e3fd559cceb@googlegroups.com> I wrote tiny wrapper around exprecs parse_trans [1] that converts json into records on the fly. It helps a lot together with jesse [2] (json validation tool). json deserealization could be done with help of any library like mochijson2 or jsx. Also there is interesting thing called jerg [3]. This tool generates hrl files with erlang records from json schemas. Hope this helps. [1] https://github.com/uwiger/parse_trans [2] https://github.com/klarna/jesse [3] https://github.com/ddossot/jerg -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Sat Aug 31 14:44:51 2013 From: g@REDACTED (Garrett Smith) Date: Sat, 31 Aug 2013 07:44:51 -0500 Subject: [erlang-questions] Erlang APIs and JSON In-Reply-To: References: <6845EF9E-06D2-486E-BAE7-821D4656BE63@alertlogic.com> Message-ID: On Fri, Aug 30, 2013 at 6:38 PM, Erik S?e S?rensen wrote: > 2013/8/30 Garrett Smith >> >> Alternatively, if your JSON is acting more as a lookup table -- strip >> off the one-tuple and use proplists:get_value - it's *very* efficient >> with small lists (based on rough measurements, < 200 items). > > I feel obliged to point out that for {Key,Value} lists, it usually makes > better sense to use lists:keyfind/3 instead of functions from the proplists > module. Performance-wise, because it's faster by a factor of 3x for small > lists and 10-15x for lists of length 30+ (keyfind is implemented in C, and > so is hard to beat). And arguably semantic-wise as well, because the > proplists module don't work quite the way many believe (it's not just > [{_,_}] it handles). You're absolutely right! I misspoke here -- the benchmarks I was referred to use the *lists* module. This silly benchmark shows at least that member tests for small lists are roughly equivalent to members tests for small sets and dicts. Note this is more a framework for getting *some* information on storage/retrieval options and any absolute truth (as has been pointed out many times in previous threads - benchmark results can change wildly and unexpectedly under different circumstances). https://github.com/gar1t/erlang-bench/blob/master/name-lookup.escript > Whether that micro-benchmark difference has any impact on your system as a > whole depends of course on how heavily it's used -- it did on one I worked > on once --, but my view is that, all other things being equal, there's no > reason to adopt proplists as a habit if it offers no other advantages over > more lightweight functions. > > (Assuming that 'dict' is the obvious alternative when dealing with lookup > tables: My quick meaurements indicate that proplists:get_value() becomes > slower than dict:fetch() at 6 elements; for lists:keyfind() vs. dict that > doesn't happen until ~140 elements. At which point proplists is 16x slower > than both.) > > Just my 2? - no nitpicking at all, it's just that you brought up > performance... > (in the same sentence as proplists) Definitely grateful for the correction and elaboration! This is an *Erlang* list after all :) Garrett From anotherworldofworld@REDACTED Sat Aug 31 21:39:15 2013 From: anotherworldofworld@REDACTED (Alex toyer) Date: Sun, 1 Sep 2013 01:39:15 +0600 Subject: [erlang-questions] Collection of erlang bookmarks Message-ID: Hello list, some times ago i started to collect links from the Web about Erlang and today i publish it - https://github.com/0xAX/erlang-bookmarks I think it will be useful for somebody. Feel free to contribute. -- best regards, twitter: @0xAX github: 0xAX -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.chamard@REDACTED Sat Aug 31 16:49:09 2013 From: david.chamard@REDACTED (David Chamard) Date: Sat, 31 Aug 2013 10:49:09 -0400 Subject: [erlang-questions] Erlang inter node communication issue In-Reply-To: <5D0F6BB2-E151-460B-A630-6B24C6B3EEA0@llaisdy.com> References: <28782DC2867DA947B19FA7519EE8FCDEAF89376635@MDC-MAIL-CMS01.ubisoft.org>, <5D0F6BB2-E151-460B-A630-6B24C6B3EEA0@llaisdy.com> Message-ID: <28782DC2867DA947B19FA7519EE8FCDEAF89376639@MDC-MAIL-CMS01.ubisoft.org> Thank you everyone , I really appreciate the time you took to answer my question. The issue was the additional port needed to be open. The cluster is up and running. Cheers, David ________________________________________ From: Ivan uemlianin [ivan@REDACTED] Sent: 31 August 2013 02:59 To: David Chamard Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Erlang inter node communication issue Dear David Each nodes will need another port (or range of ports) available, and a full name. I had the same trouble: http://llaisdy.wordpress.com/2011/05/17/connecting-erlang-nodes/ Best wishes Ivan -- festina lente On 30 Aug 2013, at 22:58, David Chamard > wrote: Hi Everyone, I am unable to get 2 erlang nodes to talk to each other through net_adm:ping. I am running on centos 6.3 and I am on amazon ec2. Port 4369 is open and I can telnet to it from both servers The cookies are the same when I do a get_cookie. When I try to do a net_adm:ping it hangs for a while and return a pang. I can resolve the hostname properly. Anyone could guide me on what the issue could be? Thank you, David _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions