From silasdb@REDACTED Wed Sep 1 05:16:49 2010 From: silasdb@REDACTED (Silas Silva) Date: Wed, 1 Sep 2010 00:16:49 -0300 Subject: Modules and function "encapsulation" Message-ID: <20100901031645.GA2088@hope.tifa.renegado> Hello all. I'm a newbie in Erlang and functional programming. Just got amazed to see how fast I could make a MVC application in Erlang, using its great pattern matching capabilities (not present in mainstream imperative languages) and other features. A doubt regarding modules, though, is in my mind: In my MVC application, the Model (for now, an interface to a Mnesia database) has the start/0 (exported) function. The Controller (which imports Model's start/0) also defines its own start/0 function. When trying to compile the Controller, I got the following error: defining imported function start/0 I decided to make some research on my favourite search engine and found this thread on an Erlang forum: http://www.trapexit.org/forum/viewtopic.php?t=11131&view=previous&sid=8d05d8be807d18af632ad00ea13e7027 There, people talks about overloading. I can have start/0 in one module and start/1, start/2 etc. in another module, which "extends" the start function, but I can't have two start/0 functions. That is okay, but I thought that start/0 for a module would be totally different from start/0 from another module and both would not conflict. So, why there is the module prefixing syntax (controller:start()), -import and -export? How do you deal with that? You just make functions name unique among all modules? This approach looks like in a big C program (e. g.: OS monolithic kernel), where symbols cannot conflict because, even though they are compiled in different .o files, the linker complains when putting everything together. Any recommendation? Thanks! -- Silas Silva From geoffrey.biggs@REDACTED Wed Sep 1 05:57:24 2010 From: geoffrey.biggs@REDACTED (Geoffrey Biggs) Date: Wed, 01 Sep 2010 12:57:24 +0900 Subject: [erlang-questions] Modules and function "encapsulation" In-Reply-To: <20100901031645.GA2088@hope.tifa.renegado> References: <20100901031645.GA2088@hope.tifa.renegado> Message-ID: <4C7DCF24.9070407@aist.go.jp> On 01/09/10 12:16, Silas Silva wrote: > Hello all. > > I'm a newbie in Erlang and functional programming. Just got amazed to > see how fast I could make a MVC application in Erlang, using its great > pattern matching capabilities (not present in mainstream imperative > languages) and other features. > > A doubt regarding modules, though, is in my mind: > > In my MVC application, the Model (for now, an interface to a Mnesia > database) has the start/0 (exported) function. The Controller (which > imports Model's start/0) also defines its own start/0 function. When > trying to compile the Controller, I got the following error: > > defining imported function start/0 > > I decided to make some research on my favourite search engine and found > this thread on an Erlang forum: > http://www.trapexit.org/forum/viewtopic.php?t=11131&view=previous&sid=8d05d8be807d18af632ad00ea13e7027 > > There, people talks about overloading. I can have start/0 in one module > and start/1, start/2 etc. in another module, which "extends" the start > function, but I can't have two start/0 functions. That is okay, but I > thought that start/0 for a module would be totally different from > start/0 from another module and both would not conflict. So, why there > is the module prefixing syntax (controller:start()), -import and > -export? > > How do you deal with that? You just make functions name unique among > all modules? This approach looks like in a big C program (e. g.: OS > monolithic kernel), where symbols cannot conflict because, even though > they are compiled in different .o files, the linker complains when > putting everything together. > > Any recommendation? > > Thanks! > Generally, you shouldn't use the -import statement. Call functions from other models explicitly using the model:function syntax (such as controller:start()). Using -import makes the code harder to follow and, as you discovered, leads to name clashes. Geoff From silasdb@REDACTED Wed Sep 1 06:16:01 2010 From: silasdb@REDACTED (Silas Silva) Date: Wed, 1 Sep 2010 01:16:01 -0300 Subject: [erlang-questions] Modules and function "encapsulation" In-Reply-To: <4C7DCF24.9070407@aist.go.jp> References: <20100901031645.GA2088@hope.tifa.renegado> <4C7DCF24.9070407@aist.go.jp> Message-ID: <20100901041559.GB2088@hope.tifa.renegado> On Wed, Sep 01, 2010 at 12:57:24PM +0900, Geoffrey Biggs wrote: > Generally, you shouldn't use the -import statement. Call functions from > other models explicitly using the model:function syntax (such as > controller:start()). Using -import makes the code harder to follow and, > as you discovered, leads to name clashes. > > Geoff Thank you! I didn't realize I could use other modules functions without -import'ing them... That is because I was getting name clashes. Thanks. -- Silas Silva From ulf.wiger@REDACTED Wed Sep 1 08:33:41 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Wed, 01 Sep 2010 08:33:41 +0200 Subject: [erlang-questions] Modules and function "encapsulation" In-Reply-To: <20100901041559.GB2088@hope.tifa.renegado> References: <20100901031645.GA2088@hope.tifa.renegado> <4C7DCF24.9070407@aist.go.jp> <20100901041559.GB2088@hope.tifa.renegado> Message-ID: <4C7DF3C5.5080302@erlang-solutions.com> On 01/09/2010 06:16, Silas Silva wrote: > On Wed, Sep 01, 2010 at 12:57:24PM +0900, Geoffrey Biggs wrote: >> Generally, you shouldn't use the -import statement. Call functions from >> other models explicitly using the model:function syntax (such as >> controller:start()). Using -import makes the code harder to follow and, >> as you discovered, leads to name clashes. >> >> Geoff > > Thank you! I didn't realize I could use other modules functions without > -import'ing them... That is because I was getting name clashes. > Thanks. As implied in that thread, what -import does is that it lets you call a remote function just as if it were a local function or a BIF. E.g. using -import(lists, [reverse/1]). I can write reverse(L) rather than lists:reverse(L), but it is purely a syntactical convenience, and as you discovered, knows nothing of overriding. The EUC talk mentioned, by Richard Carlsson, has been implemented as an experimental feature. http://www.trapexit.org/Extend_Module Note the experimental status. This is not to say it doesn't work - I daresay it does - but that there is no commitment to keep the feature as-is. BR, Ulf W -- Ulf Wiger CTO, Erlang Solutions Ltd, formerly Erlang Training & Consulting Ltd http://www.erlang-solutions.com From ulf.wiger@REDACTED Wed Sep 1 10:16:00 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Wed, 01 Sep 2010 10:16:00 +0200 Subject: [erlang-questions] Why Distributed Is So Hard In-Reply-To: <60497EC4-0B3D-40DB-BA27-B2955A114ADF@souja.net> References: <60497EC4-0B3D-40DB-BA27-B2955A114ADF@souja.net> Message-ID: <4C7E0BC0.3020107@erlang-solutions.com> On 30/08/2010 12:01, Jayson Vantuyl wrote: > Hey everybody, > > While it's not Erlang-specific, I know a lot of people here care a > lot about distributed programming. I had a really good question > from someone the other day and it spawned a blog post [...] > > http://is.gd/eLhKF Nice blog post. For the record, I ran Erlang on MS-DOS back in 1992-93. I believe it was the first (sorta) working Erlang environment I had access to - that, and a partly-broken port to MacOS 6. I later shelled out $1000 and got a HP-UX version on a QIC tape in 1993. I'll admit that it was a huge improvement... I could now run Distributed Erlang, and enthusiastically demoed a distributed ping-pong program to some of my colleagues, who looked like they wondered what I had been smoking. http://dilbert.com/strips/comic/1992-09-08/ So MS-DOS was actually capable of some advanced concurrency even without diving into TSR magic or utilizing 80x86 virtualization techniques. :) Why it didn't catch on like wildfire already then is a mystery I have yet to unravel... BR, Ulf W -- Ulf Wiger CTO, Erlang Solutions Ltd, formerly Erlang Training & Consulting Ltd http://www.erlang-solutions.com From daniel.goertzen@REDACTED Wed Sep 1 18:17:45 2010 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Wed, 1 Sep 2010 11:17:45 -0500 Subject: ssh client Message-ID: I want to write a program that will chat with another remote program via SSH. Erlang SSH is a bit complex, and I was unable to find an example of what I wanted, so I thought I would confirm my approach here before barking up the wrong tree. Here is what I was going to do: 1. Open a connection with ssh:connect() 2. Open a channel with ssh_connection:session_channel() 3. Call ssh_channel:start_link() to create my handler process. 4. In my callback module, init() will call ssh_connection:exec() to invoke the remote program. 5. Incoming data gets sent to my handle_ssh_message() callback. I send data with ssh_connection:send() Do I have it right? It would be nice if there was a "port" abstraction for an ssh channel, so I could treat locally and remotely invoked programs identically... I guess it wouldn't be hard to write my own wrapper for that. Thanks, Dan. From sperka@REDACTED Wed Sep 1 18:38:25 2010 From: sperka@REDACTED (=?UTF-8?B?U3ZhdG9wbHVrIMWgcGVya2E=?=) Date: Wed, 01 Sep 2010 18:38:25 +0200 Subject: Accessing function specifications Message-ID: <4C7E8181.5050105@gmail.com> Hi, would anybody know if it is possible to access type specification of a function in the same module at runtime ? And secondly is it possible to declare different specification for each function clause (yeah, pretty dirty but the end justifies the means ;) ? To be more specific, I'll outline my problem: I have a function with several clauses, each of which returns a list of tuples of the same size but type of value on particular positions differ for each clause. Some return 'condensed' tuples with a list on some position (it virtually represents a set of tuples one would obtain by making a cartesian product of atomic values with a list, which i do not want to do because results could be huge and i have functions for processing this kind of tuples). I would like to use specifications for functions as input for building infrastracture for processing their results before actually evaluating them because it is part of a larger machinery which needs proper settings (and probably optimisation). Thank you very much in advance! Svatopluk ?perka From fritchie@REDACTED Wed Sep 1 18:50:26 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 01 Sep 2010 11:50:26 -0500 Subject: [erlang-questions] ACM Erlang Workshop: accepted papers & schedule In-Reply-To: Message of "Mon, 30 Aug 2010 09:49:17 +0300." Message-ID: <31577.1283359826@snookles.snookles.com> Guy Wiener wrote: gw> Is there a timetable for this program, or at least an approximated gw> end time? Guy, both the Erlang and Haskell workshops will be ending at 6pm on Thursday. I don't recall exactly the start time, but memory suggests that it's 9am or perhaps 8:45am. (The start time hasn't been cast in stone, but it's at least in concrete that's still drying.) -Scott From sg2342@REDACTED Wed Sep 1 21:31:15 2010 From: sg2342@REDACTED (Stefan Grundmann) Date: Wed, 1 Sep 2010 19:31:15 +0000 Subject: [erlang-questions] ssh application leaks processes(?) In-Reply-To: References: <201008251923.43899.sg2342@googlemail.com> Message-ID: <201009011931.15502.sg2342@googlemail.com> unfortunately this leads to a) not so helpful error messages (when ssh_subsystem_sup reaches its restart limit and kills itself) and b) a (very) small memory leak in ssh_system_sup (the child never gets deleted) i think the way it should be solved is to add support for stopping a ssh_subsystem_sup to ssh_system_sup and call this from ssh_connection_manager:terminate/2. On Wednesday 25 August 2010 21:48:49 Attila Rajmund Nohl wrote: > I've patched our local installation about a year ago to use > 'permanent'. It works fine. > > 2010/8/25, Stefan Grundmann : > > Hi, > > while using the ssh-2.0 application i encountered the following problem: > > > > The ssh_system_sup supervisor (created via ssh:daemon/1,2,3) has as > > children > > the ssh_acceptor_sup supervisor and for every connected client an > > ssh_subsystem_sup supervisor. When a client connection is closed one > > would expect that all processes in the supervision tree of this > > connection die - which is not the case: the ssh_subsystem_sup process > > and one child (ssh_channel_sup) stay alive. > > > > The reason for this behavior is that the child specs of the > > ssh_channel_sup and ssh_connection_sup in > > lib/ssh-2.0/src/ssh_subsystem_sup,erl (line 75 and line 88) define a > > 'transient' Restart and while the subtree supervised by > > ssh_connection_sup as well as the ssh_connection_sup process itself > > terminates > > on connection close, the ssh_channel_sup process does not. Which leaves 2 > > processes alive that won't be of any further use. > > > > If the Restart is changed to 'permanent' the problem goes away. > > > > Since lib/ssh-2.0/src/ssh_subsystem_sup,erl contains (line 76 and line > > 89) commented out code that sets Restart to permanent; i would like to > > know what the reason for a transient Restart might be. > > > > best regards > > Stefan Grundmann > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From kostis@REDACTED Wed Sep 1 20:09:36 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 01 Sep 2010 21:09:36 +0300 Subject: [erlang-questions] Accessing function specifications In-Reply-To: <4C7E8181.5050105@gmail.com> References: <4C7E8181.5050105@gmail.com> Message-ID: <4C7E96E0.4030000@cs.ntua.gr> Svatopluk ?perka wrote: > Hi, > > would anybody know if it is possible to access type specification of a > function in the same module at runtime ? Yes, it is possible. (*) > And secondly is it possible to > declare different specification for each function clause (yeah, pretty > dirty but the end justifies the means ;) ? Yes, this is also possible ;-) (**) Cheers, Kostis (*) Though the only tool I know of is not released, so effectively the answer to your question is: "You have to do it yourself somehow". (**) Look for overloaded specs here: http://www.erlang.org/doc/reference_manual/typespec.html#id2272692 From igorrs@REDACTED Thu Sep 2 07:24:32 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Thu, 2 Sep 2010 02:24:32 -0300 Subject: Mnesia vs. timeouts Message-ID: Hello. I've recently had a problem with a pool of servers running Mnesia. The clients reach the pool through a load balancer and the tables (most of which are disc_only_copies) are fragmented and replicated throughout the servers. This means that Mnesia, on each server handling a request, will usually need to contact other servers. The problem happened when the number of requests from the clients was suddenly multiplied by 100 during some seconds. The kernel (system) CPU time immediately reached the top on every server and the clients began to timeout and later retry the failed requests (what worsened the problem, of course). >From the logs in the servers, I noticed some dirty reads taking more than 10 minutes to finish (way after the clients had given up on those operations). My question is: how can I set a timeout for every Mnesia activity (which may be distributed) and make sure that, after that time, no operation related to that activity will be left hanging on any node? By just killing the process that called mnesia:activity, am I guaranteed to get that result? Thanks. Igor. -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From silasdb@REDACTED Thu Sep 2 08:23:52 2010 From: silasdb@REDACTED (Silas Silva) Date: Thu, 2 Sep 2010 03:23:52 -0300 Subject: Converting list of tagged tuples to a record Message-ID: <20100902062350.GA2658@hope.tifa.renegado> Hello all, In my newbie application, I receive a bunch of data from HTTP's POST (I'm using Mochiweb), in the following format (supposing a person's data): [{first_name, value}, {last_name, value}, ...] And I have the definition of the record for the person: -record(person, {first_name, last_name, ...}). alistair in the #erlang channel in freenode, gave me a nice solution to put this list in the record: lists:foldl(fun add_to_record/2, #person{}, List). add_to_record({first_name, value}, rec) -> rec#person{first_name = value}; add_to_record({last_name, value}, rec) -> rec#person{last_name = value}; ... That works. My questions are: 1. Are there other [good practice] ways to transform information coming from a UI (which, must of the times, come as a list) in well specified records? 2. Is it common to use records of records? Is it possible? For example: -record(name, {last_name, first_name}). -record(person, name, ...) Thanks! -- Silas Silva From bengt.kleberg@REDACTED Thu Sep 2 08:48:52 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 02 Sep 2010 08:48:52 +0200 Subject: [erlang-questions] Converting list of tagged tuples to a record In-Reply-To: <20100902062350.GA2658@hope.tifa.renegado> References: <20100902062350.GA2658@hope.tifa.renegado> Message-ID: <1283410132.4852.17.camel@seasc1137> Greetings, After 10 years of programming Erlang I still use the method described below (1) when moving data from a user interface. The same goes for using records in records. It is rather awkward to update the innermost records, so it helps to use functions. bengt On Thu, 2010-09-02 at 08:23 +0200, Silas Silva wrote: > Hello all, > > In my newbie application, I receive a bunch of data from HTTP's POST > (I'm using Mochiweb), in the following format (supposing a person's > data): > > [{first_name, value}, {last_name, value}, ...] > > And I have the definition of the record for the person: > > -record(person, {first_name, last_name, ...}). > > alistair in the #erlang channel in freenode, gave me a nice solution to > put this list in the record: > > lists:foldl(fun add_to_record/2, #person{}, List). > > add_to_record({first_name, value}, rec) -> > rec#person{first_name = value}; > > add_to_record({last_name, value}, rec) -> > rec#person{last_name = value}; > > ... > > That works. My questions are: > > 1. Are there other [good practice] ways to transform information coming > from a UI (which, must of the times, come as a list) in well > specified records? > > 2. Is it common to use records of records? Is it possible? For example: > > -record(name, {last_name, first_name}). > -record(person, name, ...) > > Thanks! > From dangud@REDACTED Thu Sep 2 10:43:48 2010 From: dangud@REDACTED (Dan Gudmundsson) Date: Thu, 2 Sep 2010 10:43:48 +0200 Subject: [erlang-questions] Mnesia vs. timeouts In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 7:24 AM, Igor Ribeiro Sucupira wrote: > My question is: how can I set a timeout for every Mnesia activity > (which may be distributed) and make sure that, after that time, no > operation related to that activity will be left hanging on any node? > By just killing the process that called mnesia:activity, am I > guaranteed to get that result? > Killing the process shall cleanup everything (eventually :-)), if not it is a bug. Most of the activity/transaction is driven from the client process in mnesia so it should work well. But if you have long search times in dets on a remote node, a queued request will not be removed from the inbox of the dets process just because you killed the requester. But you can't know if the data was commit'ed or not. /Dan From sperka@REDACTED Thu Sep 2 10:45:19 2010 From: sperka@REDACTED (=?UTF-8?B?U3ZhdG9wbHVrIMWgcGVya2E=?=) Date: Thu, 02 Sep 2010 10:45:19 +0200 Subject: [erlang-questions] Accessing function specifications In-Reply-To: <4C7E96E0.4030000@cs.ntua.gr> References: <4C7E8181.5050105@gmail.com> <4C7E96E0.4030000@cs.ntua.gr> Message-ID: <4C7F641F.50107@gmail.com> Thanks for the reply. I believe dialyzer could serve as inspiration for how to extract specs. Anyways I've figured other, more 'manual' way of how to do what I need. On 9/1/10 20:09 , Kostis Sagonas wrote: > Svatopluk ?perka wrote: >> Hi, >> >> would anybody know if it is possible to access type specification of >> a function in the same module at runtime ? > > Yes, it is possible. (*) > >> And secondly is it possible to declare different specification for >> each function clause (yeah, pretty dirty but the end justifies the >> means ;) ? > > Yes, this is also possible ;-) (**) > > Cheers, > Kostis > > (*) Though the only tool I know of is not released, so effectively > the answer to your question is: "You have to do it yourself somehow". > > (**) Look for overloaded specs here: > http://www.erlang.org/doc/reference_manual/typespec.html#id2272692 From hm@REDACTED Thu Sep 2 10:49:02 2010 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Thu, 2 Sep 2010 10:49:02 +0200 Subject: [erlang-questions] Mnesia vs. timeouts In-Reply-To: References: Message-ID: cupira wrote: > Hello. > > I've recently had a problem with a pool of servers running Mnesia. > The clients reach the pool through a load balancer and the tables > (most of which are disc_only_copies) are fragmented and replicated > throughout the servers. This means that Mnesia, on each server > handling a request, will usually need to contact other servers. This is a rather unfortunate setup that does not scale. In order to make this to scale, you would need to try to achieve that each request only should access local replicas. Mnesia has a concept of foreign keys that can be used to co-allocate fragments from different tables. > The problem happened when the number of requests from the clients was > suddenly multiplied by 100 during some seconds. The kernel (system) > CPU time immediately reached the top on every server and the clients > began to timeout and later retry the failed requests (what worsened > the problem, of course). > From the logs in the servers, I noticed some dirty reads taking more > than 10 minutes to finish (way after the clients had given up on those > operations). > > My question is: how can I set a timeout for every Mnesia activity > (which may be distributed) and make sure that, after that time, no > operation related to that activity will be left hanging on any node? > By just killing the process that called mnesia:activity, am I > guaranteed to get that result? Mnesia has no such timeout. I would not recommend killing the Mnesia related processes, even if Mnesia is designed to cope with that. The core problem seems to be that your application does not scale. Killing processes in panic does not solve the real problem. /H?kan From dmitrii@REDACTED Thu Sep 2 16:18:53 2010 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Thu, 2 Sep 2010 17:18:53 +0300 Subject: erlang-neo4j Message-ID: <94DD34A3-9ABA-40FB-A568-C284F5C2C070@gmail.com> I'd like to introduce cali, another library to interface erlang and neo4j, http://github.com/dmitriid/cali. The other one is nerlo, http://github.com/nerlo/nerlo Cali is an early version which is more of an experiment rather than anything else, but it works. It's built on top of Gremlin, and if you don't know what Gremlin is, do read presentations linked in the README :) Anyway, it's quite simple, no fancy stuff other than ad-hoc Gremlin queries. There are several examples in the README to get you started. If anyone's willing to help, do jump in :) From igorrs@REDACTED Thu Sep 2 17:08:57 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Thu, 2 Sep 2010 12:08:57 -0300 Subject: [erlang-questions] Mnesia vs. timeouts In-Reply-To: References: Message-ID: 2010/9/2 H?kan Mattsson : > cupira wrote: >> Hello. >> >> I've recently had a problem with a pool of servers running Mnesia. >> The clients reach the pool through a load balancer and the tables >> (most of which are disc_only_copies) are fragmented and replicated >> throughout the servers. This means that Mnesia, on each server >> handling a request, will usually need to contact other servers. > > This is a rather unfortunate setup that does not scale. In order to make > this to scale, you would need to try to achieve that each request only > should access local replicas. By replicas, do you mean Mnesia clones? Since I'm using replication also for data safety, I can't see a "local" scenario that would make sense. Even if you have two nodes in the same server, each one writing a clone to a different external storage, that would still be non-ideal. Besides, this scenario wouldn't be free of the problems of cloning to two different machines, since one of the external storages might still be overloaded some day. That said... replication was not really the problem in the situation I described, since the huge majority of the operations were dirty reads (not writes). If what you meant was actually fragments (not replicas), you would be saying that I shouldn't run distributed activities through Mnesia (?). Actually, I can avoid distributed activities for my simple dirty reads: just check with Mnesia which nodes hold the data and then contact one of them via RPC. I'm planning that, indeed. But if you tell people they can't run distributed activities, Mnesia starts to become less useful. The third possibility is: you meant the clients should also reach Mnesia by the exact server that holds all the necessary data. In this scenario, Mnesia would be almost worthless, so I don't think you mean that. I'll post again today, to discuss some solutions. Thanks. Igor. > Mnesia has a concept of foreign keys > that can be used to co-allocate fragments from different tables. > >> The problem happened when the number of requests from the clients was >> suddenly multiplied by 100 during some seconds. The kernel (system) >> CPU time immediately reached the top on every server and the clients >> began to timeout and later retry the failed requests (what worsened >> the problem, of course). >> From the logs in the servers, I noticed some dirty reads taking more >> than 10 minutes to finish (way after the clients had given up on those >> operations). >> >> My question is: how can I set a timeout for every Mnesia activity >> (which may be distributed) and make sure that, after that time, no >> operation related to that activity will be left hanging on any node? >> By just killing the process that called mnesia:activity, am I >> guaranteed to get that result? > > Mnesia has no such timeout. I would not recommend killing the Mnesia > related processes, even if Mnesia is designed to cope with that. The core > problem seems to be that your application does not scale. Killing processes > in panic does not solve the real problem. > > /H?kan -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From igorrs@REDACTED Thu Sep 2 17:08:57 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Thu, 2 Sep 2010 12:08:57 -0300 Subject: [erlang-questions] Mnesia vs. timeouts In-Reply-To: References: Message-ID: 2010/9/2 H?kan Mattsson : > cupira wrote: >> Hello. >> >> I've recently had a problem with a pool of servers running Mnesia. >> The clients reach the pool through a load balancer and the tables >> (most of which are disc_only_copies) are fragmented and replicated >> throughout the servers. This means that Mnesia, on each server >> handling a request, will usually need to contact other servers. > > This is a rather unfortunate setup that does not scale. In order to make > this to scale, you would need to try to achieve that each request only > should access local replicas. By replicas, do you mean Mnesia clones? Since I'm using replication also for data safety, I can't see a "local" scenario that would make sense. Even if you have two nodes in the same server, each one writing a clone to a different external storage, that would still be non-ideal. Besides, this scenario wouldn't be free of the problems of cloning to two different machines, since one of the external storages might still be overloaded some day. That said... replication was not really the problem in the situation I described, since the huge majority of the operations were dirty reads (not writes). If what you meant was actually fragments (not replicas), you would be saying that I shouldn't run distributed activities through Mnesia (?). Actually, I can avoid distributed activities for my simple dirty reads: just check with Mnesia which nodes hold the data and then contact one of them via RPC. I'm planning that, indeed. But if you tell people they can't run distributed activities, Mnesia starts to become less useful. The third possibility is: you meant the clients should also reach Mnesia by the exact server that holds all the necessary data. In this scenario, Mnesia would be almost worthless, so I don't think you mean that. I'll post again today, to discuss some solutions. Thanks. Igor. > Mnesia has a concept of foreign keys > that can be used to co-allocate fragments from different tables. > >> The problem happened when the number of requests from the clients was >> suddenly multiplied by 100 during some seconds. The kernel (system) >> CPU time immediately reached the top on every server and the clients >> began to timeout and later retry the failed requests (what worsened >> the problem, of course). >> From the logs in the servers, I noticed some dirty reads taking more >> than 10 minutes to finish (way after the clients had given up on those >> operations). >> >> My question is: how can I set a timeout for every Mnesia activity >> (which may be distributed) and make sure that, after that time, no >> operation related to that activity will be left hanging on any node? >> By just killing the process that called mnesia:activity, am I >> guaranteed to get that result? > > Mnesia has no such timeout. I would not recommend killing the Mnesia > related processes, even if Mnesia is designed to cope with that. The core > problem seems to be that your application does not scale. Killing processes > in panic does not solve the real problem. > > /H?kan -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From rapsey@REDACTED Thu Sep 2 17:32:16 2010 From: rapsey@REDACTED (Rapsey) Date: Thu, 2 Sep 2010 17:32:16 +0200 Subject: [erlang-questions] Converting list of tagged tuples to a record In-Reply-To: <1283410132.4852.17.camel@seasc1137> References: <20100902062350.GA2658@hope.tifa.renegado> <1283410132.4852.17.camel@seasc1137> Message-ID: I use these two functions (rec2prop, prop2rec): rec2prop(Rec, RecordFields) -> loop_rec(RecordFields, 1, Rec, []). loop_rec([H|T], N, Rec, L) -> loop_rec(T, N+1, Rec, [{H, element(N+1, Rec)}|L]); loop_rec([], _, _, L) -> L. prop2rec(Prop, RecName, DefRec, RecordFields) -> loop_fields(erlang:make_tuple(tuple_size(DefRec), RecName), RecordFields, DefRec, Prop, 2). loop_fields(Tuple, [Field|T], DefRec, Props, N) -> case lists:keysearch(Field, 1, Props) of {value, {_, Val}} -> loop_fields(setelement(N, Tuple, Val), T, DefRec, Props, N+1); false -> loop_fields(setelement(N, Tuple, element(N, DefRec)), T, DefRec, Props, N+1) end; loop_fields(Tuple, [], _, _, _) -> Tuple. And then for every record a define (#cdat{} is the name of the record): -define(R2P(Record), rec2prop(Record, record_info(fields, cdat))). -define(P2R(Prop), prop2rec(Prop, cdat, #cdat{}, record_info(fields, cdat))). It's not exactly optimal, but at least I don't have to write a function for every record (just two short defines). Sergej On Thu, Sep 2, 2010 at 8:48 AM, Bengt Kleberg wrote: > Greetings, > > After 10 years of programming Erlang I still use the method described > below (1) when moving data from a user interface. > The same goes for using records in records. It is rather awkward to > update the innermost records, so it helps to use functions. > > > bengt > > On Thu, 2010-09-02 at 08:23 +0200, Silas Silva wrote: > > Hello all, > > > > In my newbie application, I receive a bunch of data from HTTP's POST > > (I'm using Mochiweb), in the following format (supposing a person's > > data): > > > > [{first_name, value}, {last_name, value}, ...] > > > > And I have the definition of the record for the person: > > > > -record(person, {first_name, last_name, ...}). > > > > alistair in the #erlang channel in freenode, gave me a nice solution to > > put this list in the record: > > > > lists:foldl(fun add_to_record/2, #person{}, List). > > > > add_to_record({first_name, value}, rec) -> > > rec#person{first_name = value}; > > > > add_to_record({last_name, value}, rec) -> > > rec#person{last_name = value}; > > > > ... > > > > That works. My questions are: > > > > 1. Are there other [good practice] ways to transform information coming > > from a UI (which, must of the times, come as a list) in well > > specified records? > > > > 2. Is it common to use records of records? Is it possible? For example: > > > > -record(name, {last_name, first_name}). > > -record(person, name, ...) > > > > Thanks! > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From armando@REDACTED Thu Sep 2 18:09:19 2010 From: armando@REDACTED (Armando Di Cianno) Date: Thu, 2 Sep 2010 12:09:19 -0400 Subject: Getting INFO_REPORT level messages into sasl error logger Message-ID: Clearly, I'm doing something daft again. For the life of me, I cannot figure out how to get INFO_REPORT messages into the sasl error logger. I have an app that starts sasl, and passes in `-sasl sasl_error_logger '{file, \"${LOG_PATH}/erlang.log\"}'"`. Error level messages get into this log. I'm using the error_logger:{info,warning,error}_msg family of commands to add thorough logging to my app. However, I simply cannot figure out from the docs if there is a different log level settings I need to set, or if there's a different log (e.g. sasl_info_logger or something like that). I've seen options for "setting" the output level of warning messages, but nothing for info level reports. If I start my app non-detached, then I see the INFO_REPORT messages just fine in the output to the console. Thanks for any help in this matter. Cheers, __armando From boris.muehmer@REDACTED Thu Sep 2 19:17:18 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Thu, 2 Sep 2010 19:17:18 +0200 Subject: Book "Concurrent Programming in ERLANG" (2nd Ed.): Source for example programmes? Message-ID: I just got my copy of "Concurrent Programming in ERLANG" (2nd ED; actually "on demand" version by Pearson Education), and I really love the "2nd Part" of it! Is there anywhere an archive with the example programmes available? Thanks in advance, - boris PS: Please don't tell me if the samples are hidden somewhere in the Erlang source archive... ;-) PPS: Of course, please tell me if they are hidden in the Erlang source archive, just don't make me feel too stupid. From tuncer.ayaz@REDACTED Thu Sep 2 20:17:28 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 2 Sep 2010 20:17:28 +0200 Subject: xml_from_edocs.escript HiPE segfault in R14B dev branch Message-ID: FYI, When building erlang with HiPE enabled and native-libs in the current dev tree you might encounter a segfault with xml_from_edocs.escript. This will usually happen if you call 'make docs'. Rickard and OTP team has git bisected the issue to the following first bad commit which exposes a possible bug in HiPE: a8b8ec5e858da86531933b545f752f436e411b58 Load native code for modules loaded before the code server This commit is only bad because it exposed the bug by causing more native compiled code to be executed than before. The HiPE team is informed and will hopefully fix it before R14B. If you happen to experience the issue just make sure to build the docs with a non-hipe beam. From mevans@REDACTED Thu Sep 2 20:56:39 2010 From: mevans@REDACTED (Evans, Matthew) Date: Thu, 2 Sep 2010 14:56:39 -0400 Subject: [erlang-questions] Converting list of tagged tuples to a record In-Reply-To: <20100902062350.GA2658@hope.tifa.renegado> References: <20100902062350.GA2658@hope.tifa.renegado> Message-ID: Using a preprocessor is another, perhaps more generic way of doing it. %% The record -record(person, {first_name, last_name, ...}). %% Create a preprocessor -define(PERSON_RECORD,record_info(fields, person)). %% Function that will convert a list of tuples to the record create_record_from_list([],_Data,Result) -> erlang:list_to_tuple(Result); create_record_from_list([Field|Rest],Data,Result) -> case lists:keysearch(Field,1,Data) of {value,{_,Value}} -> create_record_from_list(Rest,Data,lists:append(Result,[Value])); _ -> create_record_from_list(Rest,Data,lists:append(Result,[undefined])) end. Then to convert the list of tuples "DataList" to a record call: Record = create_record_from_list(?PERSON_RECORD,DataList,[person]), Or something like that :) Matt -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Silas Silva Sent: Thursday, September 02, 2010 2:24 AM To: erlang-questions@REDACTED Subject: [erlang-questions] Converting list of tagged tuples to a record Hello all, In my newbie application, I receive a bunch of data from HTTP's POST (I'm using Mochiweb), in the following format (supposing a person's data): [{first_name, value}, {last_name, value}, ...] And I have the definition of the record for the person: -record(person, {first_name, last_name, ...}). alistair in the #erlang channel in freenode, gave me a nice solution to put this list in the record: lists:foldl(fun add_to_record/2, #person{}, List). add_to_record({first_name, value}, rec) -> rec#person{first_name = value}; add_to_record({last_name, value}, rec) -> rec#person{last_name = value}; ... That works. My questions are: 1. Are there other [good practice] ways to transform information coming from a UI (which, must of the times, come as a list) in well specified records? 2. Is it common to use records of records? Is it possible? For example: -record(name, {last_name, first_name}). -record(person, name, ...) Thanks! -- Silas Silva ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From igorrs@REDACTED Thu Sep 2 21:30:26 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Thu, 2 Sep 2010 16:30:26 -0300 Subject: [erlang-questions] Mnesia vs. timeouts In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 5:43 AM, Dan Gudmundsson wrote: > On Thu, Sep 2, 2010 at 7:24 AM, Igor Ribeiro Sucupira wrote: >> My question is: how can I set a timeout for every Mnesia activity >> (which may be distributed) and make sure that, after that time, no >> operation related to that activity will be left hanging on any node? >> By just killing the process that called mnesia:activity, am I >> guaranteed to get that result? >> > > Killing the process shall cleanup everything (eventually :-)), if not > it is a bug. > > Most of the activity/transaction is driven from the client process in mnesia so > it should work well. Good to know that. Might be useful. > But if you have long search times in dets on a remote node, a queued > request will not be > removed from the inbox of the dets process just because you killed the > requester. Well... sooner or later, grown-ups have to deal with the situation of slow/overloaded servers. :-) Basically, what has to be done is: 1) Avoid sending more load to a slow server. 2) Don't let a slow server create slowness in the others. I understand this is difficult to achieve in the general case, so I'm not expecting Mnesia to give it to me for free. What I'm planning to do is: 1) For the dirty reads that constitute the majority of the operations in my system, don't let Mnesia perform remote calls. Instead, make an RPC to the node that holds the desired fragment and set a timeout. The database operation will never be interrupted, but at least nobody but the slow server will be waiting for it. 2) Wrap that RPC on a circuit breaker, keeping track of timeouts and temporarily avoiding calls to slow servers. Of course, there are other operations going on (including transactions writing data), but they only account for about 10 per second for the whole pool, which is nothing compared to the amount of those dirty reads. Can you see anything else that might help? Thanks. Igor. > But you can't know if the data was commit'ed or not. > > > /Dan -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From steve@REDACTED Thu Sep 2 23:09:29 2010 From: steve@REDACTED (Steve Strong) Date: Thu, 2 Sep 2010 23:09:29 +0200 Subject: Dynamically add node to Distributed Application Message-ID: Hi, I'm working on the design of an Erlang based system which will be hosted on EC2. Right now, a distributed application with a set of nodes looks like the right way forward, but one of my requirements is to be able to dynamically spin up new amazon instances and add them to the nodes within the distributed app. However, I can't see a way of adding nodes to a running distributed app - I am, however, a self-confessed Erlang noob, so any pointers would be much appreciated! Cheers, Steve From mevans@REDACTED Thu Sep 2 23:33:53 2010 From: mevans@REDACTED (Evans, Matthew) Date: Thu, 2 Sep 2010 17:33:53 -0400 Subject: Mnesia/dets question - does this make sense? Message-ID: Hi, I have a fragmented disc_only mnesia table. During tests I have noticed that it appears that the access (insert) times to the fragment improve over time, and then slow down after a period of inactivity. Do the tables/fragments get opened and then closed based on a timer that would affect initial inert times? For example, the table has 10 fragments. I see insert times in the 4000us range for the first 10 to 15 inserts, and then insert times averaging 400us subsequently. If there is no activity on the table for a few minutes access times are again in the 4000us range for the first 10 to 15 inserts. As I said before it's almost like there is an overhead to get things started/warmed up (e.g. opening the fragments/files), once all the fragments are opened the speed is fine. Can anyone confirm this? Thanks Matt From rvirding@REDACTED Fri Sep 3 00:00:34 2010 From: rvirding@REDACTED (Robert Virding) Date: Fri, 3 Sep 2010 00:00:34 +0200 Subject: [erlang-questions] Book "Concurrent Programming in ERLANG" (2nd Ed.): Source for example programmes? In-Reply-To: References: Message-ID: As far as I can remember we never collected together the all code from the examples and put them in some form of archive. I don't know if anyone else has done this. Robert On 2 September 2010 19:17, Boris M?hmer wrote: > I just got my copy of "Concurrent Programming in ERLANG" > (2nd ED; actually "on demand" version by Pearson Education), > and I really love the "2nd Part" of it! > > > Is there anywhere an archive with the example programmes available? > > > Thanks in advance, > ?- boris > > > PS: > ? ?Please don't tell me if the samples are hidden somewhere > ? ?in the Erlang source archive... ;-) > > PPS: > ? ?Of course, please tell me if they are hidden in the Erlang > ? ?source archive, just don't make me feel too stupid. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From zeno490@REDACTED Fri Sep 3 00:58:16 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Thu, 2 Sep 2010 18:58:16 -0400 Subject: [erlang-questions] Mnesia/dets question - does this make sense? In-Reply-To: References: Message-ID: Disc_only tables probably write to disk right away and probably don't buffer so writes can be atomic. I'm thinking your problem is more likely at the OS/HDD driver level. It is possible your OS/HDD is caching the data in memory for short amounts of time in order to speed up subsequent access to said data, once flushed you pay the full price of accessing that section of the disk again (worse if the disk had stopped spinning). For example, Windows 7 is actually quite good at this, using all available memory that isn't being used by applications to buffer IO (or something like that). Just a wild guess though, you'll have to profile to understand what is happening (perhaps at the kernel level to see how much time is spend actually writing the data by the kernel VS time spent in erlang code). 2cents Nicholas On Thu, Sep 2, 2010 at 5:33 PM, Evans, Matthew wrote: > Hi, > > I have a fragmented disc_only mnesia table. During tests I have noticed > that it appears that the access (insert) times to the fragment improve over > time, and then slow down after a period of inactivity. > > Do the tables/fragments get opened and then closed based on a timer that > would affect initial inert times? > > For example, the table has 10 fragments. I see insert times in the 4000us > range for the first 10 to 15 inserts, and then insert times averaging 400us > subsequently. If there is no activity on the table for a few minutes access > times are again in the 4000us range for the first 10 to 15 inserts. As I > said before it's almost like there is an overhead to get things > started/warmed up (e.g. opening the fragments/files), once all the fragments > are opened the speed is fine. > > Can anyone confirm this? > > Thanks > > Matt > From kaiduanx@REDACTED Fri Sep 3 03:10:58 2010 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Thu, 2 Sep 2010 21:10:58 -0400 Subject: [erlang-questions] Dynamically add node to Distributed Application In-Reply-To: References: Message-ID: You need to consider the two things, 1. Distribution from erlang that means you need to set up an erlang cluster so that each node can ping each other in the cluster. Use net_adm:ping(Node) to join a node into the cluster. You will get notification when a node leaves the cluster from erlang distribution if you monitor the nodes in cluster via erlang:monitor(Node,flag). 2. Distribution on your application, this is your application's logic to handle the distribution. For example, how to dispatch the load to the nodes in erlang cluster, how to handle the scenario where a node joins/leaves the cluster from your application point of view. Kaiduan On Thu, Sep 2, 2010 at 5:09 PM, Steve Strong wrote: > Hi, > > I'm working on the design of an Erlang based system which will be hosted on > EC2. ?Right now, a distributed application with a set of nodes looks like > the right way forward, but one of my requirements is to be able to > dynamically spin up new amazon instances and add them to the nodes within > the distributed app. > > However, I can't see a way of adding nodes to a running distributed app - I > am, however, a self-confessed Erlang noob, so any pointers would be much > appreciated! > > Cheers, > > Steve > From g9414002.pccu.edu.tw@REDACTED Fri Sep 3 03:21:56 2010 From: g9414002.pccu.edu.tw@REDACTED (=?UTF-8?B?6buD6ICA6LOiIChZYXUtSHNpZW4gSHVhbmcp?=) Date: Fri, 3 Sep 2010 09:21:56 +0800 Subject: [erlang-questions] Dynamically add node to Distributed Application In-Reply-To: References: Message-ID: Hi, Steve! I'm a EC2 noob. But I recently wrote somethings in HP high performance computer. I have a monitoring module that takes all working nodes in the cluster. Part of code is loop(Pid, NStates) -> receive {Pid, get_nodes} -> Pid ! [ N || N <- NStates, is_atom(N) ], loop(Pid, NStates); {Pid, refresh} -> manage(Pid, NStates); {Pid, stop} -> [ slave:stop(N) || N <- NStates, is_atom(N) ], Pid ! ok, ok end. First it starts a process that end with keeping the loop. The loop knows the PID of the master node and a list of node states, NStates, that contains either NodeNames or {idle, NodeNames}. Then when I want to add a node, it may be a new request such as {Pid, add_nodes, ListOfIdleNodes} -> loop(Pid, NStates ++ ListOfIdleNodes); ListOfIdleNodes contains a set of {idle, NodeName}. After adding nodes by sending this message, the master node shell send refresh command and new nodes may work. Best regards, Y.H.H. On Fri, Sep 3, 2010 at 5:09 AM, Steve Strong wrote: > Hi, > > I'm working on the design of an Erlang based system which will be hosted on > EC2. Right now, a distributed application with a set of nodes looks like > the right way forward, but one of my requirements is to be able to > dynamically spin up new amazon instances and add them to the nodes within > the distributed app. > > However, I can't see a way of adding nodes to a running distributed app - I > am, however, a self-confessed Erlang noob, so any pointers would be much > appreciated! > > Cheers, > > Steve > From bengt.kleberg@REDACTED Fri Sep 3 08:20:06 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 03 Sep 2010 08:20:06 +0200 Subject: [erlang-questions] Converting list of tagged tuples to a record In-Reply-To: References: <20100902062350.GA2658@hope.tifa.renegado> Message-ID: <1283494806.4861.13.camel@seasc1137> Greetings, I would like to know if there is a simple way to handle backwards compatible changes? My interfaces tend to evolve so that there is no longer a one to one mapping from the properties to the record. Ex: the boolean property is_receive has been replaced with {direction , "send" | "receive"} and the internal record only has member 'direction', but old programs that have is_receive is still in use. bengt On Thu, 2010-09-02 at 20:56 +0200, Evans, Matthew wrote: > Using a preprocessor is another, perhaps more generic way of doing it. > > > > > > %% The record > > -record(person, {first_name, last_name, ...}). > > > > > > %% Create a preprocessor > > -define(PERSON_RECORD,record_info(fields, person)). > > > > > > %% Function that will convert a list of tuples to the record > > create_record_from_list([],_Data,Result) -> > > erlang:list_to_tuple(Result); > > create_record_from_list([Field|Rest],Data,Result) -> > > case lists:keysearch(Field,1,Data) of > > {value,{_,Value}} -> > > create_record_from_list(Rest,Data,lists:append(Result,[Value])); > > _ -> > > create_record_from_list(Rest,Data,lists:append(Result,[undefined])) > > end. > > > > > > > > Then to convert the list of tuples "DataList" to a record call: > > > > Record = create_record_from_list(?PERSON_RECORD,DataList,[person]), > > > > > > Or something like that :) > > > > Matt > > > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Silas Silva > Sent: Thursday, September 02, 2010 2:24 AM > To: erlang-questions@REDACTED > Subject: [erlang-questions] Converting list of tagged tuples to a record > > > > Hello all, > > > > In my newbie application, I receive a bunch of data from HTTP's POST > > (I'm using Mochiweb), in the following format (supposing a person's > > data): > > > > [{first_name, value}, {last_name, value}, ...] > > > > And I have the definition of the record for the person: > > > > -record(person, {first_name, last_name, ...}). > > > > alistair in the #erlang channel in freenode, gave me a nice solution to > > put this list in the record: > > > > lists:foldl(fun add_to_record/2, #person{}, List). > > > > add_to_record({first_name, value}, rec) -> > > rec#person{first_name = value}; > > > > add_to_record({last_name, value}, rec) -> > > rec#person{last_name = value}; > > > > ... > > > > That works. My questions are: > > > > 1. Are there other [good practice] ways to transform information coming > > from a UI (which, must of the times, come as a list) in well > > specified records? > > > > 2. Is it common to use records of records? Is it possible? For example: > > > > -record(name, {last_name, first_name}). > > -record(person, name, ...) > > > > Thanks! > > > > -- > > Silas Silva > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From tuncer.ayaz@REDACTED Fri Sep 3 13:40:22 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 3 Sep 2010 13:40:22 +0200 Subject: Erlang User Conference 2010 Hackathon Message-ID: Erlang User Conference 2010 Hackathon Same as last year we're organizing a hackathon the day after EUC. The event is again kindly hosted by Klarna. Space is limited to around 15 persons. If you're interested please reply to Jordi and me off-list. If you intend to work on something specific don't forget to include it in your reply. Where: Klarna AB Norra stationsgatan 61 113 43 Stockholm When: Nov. 17th 2010 From mevans@REDACTED Fri Sep 3 16:25:13 2010 From: mevans@REDACTED (Evans, Matthew) Date: Fri, 3 Sep 2010 10:25:13 -0400 Subject: [erlang-questions] Mnesia/dets question - does this make sense? In-Reply-To: References: Message-ID: Possibly. In this case we are running on Linux, but the file system is EXT3 running on flash/SSD, so there shouldn't be any spinning disks that need to get moving. There are of course buffers down in the kernel space, but I imagine that they are global and not per file, so they shouldn't cause this behavior. It's not a huge issue, but could be if/when we need to grow the number of fragments. Matt ________________________________ From: Nicholas Frechette [mailto:zeno490@REDACTED] Sent: Thursday, September 02, 2010 6:58 PM To: Evans, Matthew Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Mnesia/dets question - does this make sense? Disc_only tables probably write to disk right away and probably don't buffer so writes can be atomic. I'm thinking your problem is more likely at the OS/HDD driver level. It is possible your OS/HDD is caching the data in memory for short amounts of time in order to speed up subsequent access to said data, once flushed you pay the full price of accessing that section of the disk again (worse if the disk had stopped spinning). For example, Windows 7 is actually quite good at this, using all available memory that isn't being used by applications to buffer IO (or something like that). Just a wild guess though, you'll have to profile to understand what is happening (perhaps at the kernel level to see how much time is spend actually writing the data by the kernel VS time spent in erlang code). 2cents Nicholas On Thu, Sep 2, 2010 at 5:33 PM, Evans, Matthew > wrote: Hi, I have a fragmented disc_only mnesia table. During tests I have noticed that it appears that the access (insert) times to the fragment improve over time, and then slow down after a period of inactivity. Do the tables/fragments get opened and then closed based on a timer that would affect initial inert times? For example, the table has 10 fragments. I see insert times in the 4000us range for the first 10 to 15 inserts, and then insert times averaging 400us subsequently. If there is no activity on the table for a few minutes access times are again in the 4000us range for the first 10 to 15 inserts. As I said before it's almost like there is an overhead to get things started/warmed up (e.g. opening the fragments/files), once all the fragments are opened the speed is fine. Can anyone confirm this? Thanks Matt From zabrane3@REDACTED Fri Sep 3 17:17:25 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Fri, 3 Sep 2010 17:17:25 +0200 Subject: Suffix array implementation in Erlang? Message-ID: Hi list, I'm looking after a fast implementation of "Suffix array" in Erlang: http://en.wikipedia.org/wiki/Suffix_array Any pointer/code will be very welcome! -- Regards Zabrane From anthonym@REDACTED Fri Sep 3 18:54:14 2010 From: anthonym@REDACTED (Anthony Molinaro) Date: Fri, 3 Sep 2010 09:54:14 -0700 Subject: [erlang-questions] Dynamically add node to Distributed Application In-Reply-To: References: Message-ID: <20100903165414.GA6126@alumni.caltech.edu> Perhaps this will work for you http://code.google.com/p/nodefinder/ it has an ec2nodefinder package which will use ec2-describe-instances, we used this in conjunction with http://code.google.com/p/erlrc/ at my previous company to dynamically add ec2 instances and have them join our cluster. We also used http://code.google.com/p/schemafinder/ to allow distributed mnesia instances to shuttle around data to newly created nodes. -Anthony On Thu, Sep 02, 2010 at 11:09:29PM +0200, Steve Strong wrote: > Hi, > > I'm working on the design of an Erlang based system which will be hosted on > EC2. Right now, a distributed application with a set of nodes looks like > the right way forward, but one of my requirements is to be able to > dynamically spin up new amazon instances and add them to the nodes within > the distributed app. > > However, I can't see a way of adding nodes to a running distributed app - I > am, however, a self-confessed Erlang noob, so any pointers would be much > appreciated! > > Cheers, > > Steve -- ------------------------------------------------------------------------ Anthony Molinaro From mevans@REDACTED Fri Sep 3 19:08:54 2010 From: mevans@REDACTED (Evans, Matthew) Date: Fri, 3 Sep 2010 13:08:54 -0400 Subject: [erlang-questions] Suffix array implementation in Erlang? In-Reply-To: References: Message-ID: Sounds like a good use for a NIF implementation. -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of zabrane Mikael Sent: Friday, September 03, 2010 11:17 AM To: Erlang Questions Mailinglist Subject: [erlang-questions] Suffix array implementation in Erlang? Hi list, I'm looking after a fast implementation of "Suffix array" in Erlang: http://en.wikipedia.org/wiki/Suffix_array Any pointer/code will be very welcome! -- Regards Zabrane ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From zabrane3@REDACTED Fri Sep 3 19:13:19 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Fri, 3 Sep 2010 19:13:19 +0200 Subject: [erlang-questions] Suffix array implementation in Erlang? In-Reply-To: References: Message-ID: Hi Metthew, Yep, this is what I'm planning is no pure Erlang implementation is available. -- Regards Zabrane 2010/9/3 Evans, Matthew : > Sounds like a good use for a NIF implementation. > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of zabrane Mikael > Sent: Friday, September 03, 2010 11:17 AM > To: Erlang Questions Mailinglist > Subject: [erlang-questions] Suffix array implementation in Erlang? > > Hi list, > > I'm looking after a fast implementation of "Suffix array" in Erlang: > http://en.wikipedia.org/wiki/Suffix_array > > Any pointer/code will be very welcome! > > -- > Regards > Zabrane > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From zabrane3@REDACTED Fri Sep 3 19:13:19 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Fri, 3 Sep 2010 19:13:19 +0200 Subject: [erlang-questions] Suffix array implementation in Erlang? In-Reply-To: References: Message-ID: Hi Metthew, Yep, this is what I'm planning is no pure Erlang implementation is available. -- Regards Zabrane 2010/9/3 Evans, Matthew : > Sounds like a good use for a NIF implementation. > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of zabrane Mikael > Sent: Friday, September 03, 2010 11:17 AM > To: Erlang Questions Mailinglist > Subject: [erlang-questions] Suffix array implementation in Erlang? > > Hi list, > > I'm looking after a fast implementation of "Suffix array" in Erlang: > http://en.wikipedia.org/wiki/Suffix_array > > Any pointer/code will be very welcome! > > -- > Regards > Zabrane > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From fritchie@REDACTED Fri Sep 3 19:22:36 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 03 Sep 2010 12:22:36 -0500 Subject: ACM Erlang Workshop is almost here: 30 September 2010 In-Reply-To: Message of "Mon\, 23 Aug 2010 23\:12\:01 CDT." <1589.1282623121@snookles.snookles.com> Message-ID: <81378.1283534556@snookles.snookles.com> Good afternoon, everyone. The 2010 ACM Erlang Workshop is now less than a month away. There are plenty of seats still available, so please make your registration & travel plans now to Baltimore, Maryland, USA. :-) The URL http://www.erlang.org/workshop/2010/ has the full program & schedule. We're happy to have David Smith from Basho Technologies as our invited speaker(*). The title of his talk is "Concurrent Life, the Universe, and Everything Rebar" We will also be adding 5-minute lightning talks at the end of the day, as many as we can fit in before they kick us out of the room at 6pm. I'll include a synopsis of the schedule below. The papers this year are excellent. Several of them are going to spark a lot of discussion & debate: at lunch, at session breaks, and over dinner and beer/whiskey/beverage-of-your-choice afterward. Also, I suggest that you look at the CUFP program (Commercial Users of Functional Programming) for lots of interesting tutorials and presentations. I look forward to seeing you in Baltimore! -Scott Workshop Chair, 2010 ACM Erlang Workshop ------------------------- Keynote Address * Invited Talk: David Smith, Basho Technologies: "Concurrent Life, the Universe, and Everything Rebar" Testing and Programming * From Test Cases to FSMs: Augmented Test Driven Development and Property Inference. Thomas Arts and Simon Thompson * Coordinating and Visualizing Independent Behaviors in Erlang. Guy Wiener, Gera Weiss, and Assaf Marron Theory and Practice * A Unified Semantics for Future Erlang. Hans Svensson, Lars-?ke Fredlund, and Clara Benac Earle * Chain Replication In Theory and in Practice. Scott Lystig Fritchie Language Issues and Extensions * Analysis of Preprocessor Constructs in Erlang. R?bert Kitlei, Istv?n Boz?, Tam?s Kozsik, M?t? Tejfel, and Melinda T?th * Generic Load Regulation Framework for Erlang. Ulf Wiger Implementations, Tools, and Demos * Implementing a Multiagent Negotiation Protocol in Erlang. Alvaro Fernandez, Clara Benac Earle, and Lars-?ke Fredlund * QuickChecking Refactoring Tools. D?niel Drienyovszky, D?niel Horp?csi, and Simon Thompson * Using Erlang to Implement an Autonomous Build and Distribution System for Software Projects. Tino Breddin Latest news from the Erlang/OTP team at Ericsson Lightning Talks: 5-minute presentations (strict time limit) until 6pm ------------------------- (*) In the interest of transparency, David's name was suggested by other Program Committee members long before I started work at Basho. After the PC agreed on David (whose nickname is "Dizzy"), and he accepted, I've had multiple colleagues from Basho tell me that they can't wait to hear Dizzy talk ... they don't care about the topic, they're all certain(**) that it's going to be really good. (**) No pressure, Dizzy, honest. :-) From paul-trapexit@REDACTED Fri Sep 3 19:52:12 2010 From: paul-trapexit@REDACTED (Paul Mineiro) Date: Fri, 3 Sep 2010 10:52:12 -0700 (PDT) Subject: [erlang-questions] Dynamically add node to Distributed Application In-Reply-To: <20100903165414.GA6126@alumni.caltech.edu> References: <20100903165414.GA6126@alumni.caltech.edu> Message-ID: If you use this, make sure your ec2 instances are in the same redundancy zone. It seemed to make a difference (in ping times and partition event likelihood). If the goal is a highly dynamic cluster, I would encourage you to try and minimize the distribution coupling, and shoot for an architecture of Erlang nodes "flying in loose formation". Mnesia in particular is not really designed for the case of nodes being introduced and removed frequently (i.e., schemafinder was a bit dangerous for that). Having Erlang maintain a fully connected grid at the distribution layer seemed safe enough for smallish clusters (i.e., nodefinder was useful and caused no headaches at the scale of < 40 nodes), and rpc:call/5 with a timeout was a useful primitive, but trying to abstract away local from remote was something we quickly abandoned. We also never used the distributed aspects of the app controller. We did use pg2 alot, but when starting up an app we had to run some checks to wait until it was consistent, since the time it took to settle down was variable. 2c, -- p On Fri, 3 Sep 2010, Anthony Molinaro wrote: > Perhaps this will work for you > > http://code.google.com/p/nodefinder/ > > it has an ec2nodefinder package which will use ec2-describe-instances, > we used this in conjunction with > > http://code.google.com/p/erlrc/ > > at my previous company to dynamically add ec2 instances and have them > join our cluster. > > We also used > > http://code.google.com/p/schemafinder/ > > to allow distributed mnesia instances to shuttle around data to newly > created nodes. > > -Anthony > > On Thu, Sep 02, 2010 at 11:09:29PM +0200, Steve Strong wrote: > > Hi, > > > > I'm working on the design of an Erlang based system which will be hosted on > > EC2. Right now, a distributed application with a set of nodes looks like > > the right way forward, but one of my requirements is to be able to > > dynamically spin up new amazon instances and add them to the nodes within > > the distributed app. > > > > However, I can't see a way of adding nodes to a running distributed app - I > > am, however, a self-confessed Erlang noob, so any pointers would be much > > appreciated! > > > > Cheers, > > > > Steve > > -- > ------------------------------------------------------------------------ > Anthony Molinaro > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From mevans@REDACTED Fri Sep 3 22:28:55 2010 From: mevans@REDACTED (Evans, Matthew) Date: Fri, 3 Sep 2010 16:28:55 -0400 Subject: Ordered set and DETS Message-ID: Hi, Does anyone know if there are any roadmap plans to implement ordered_set table types in DETS? Thanks From senthilkumar.peelikkampatti@REDACTED Sat Sep 4 00:07:06 2010 From: senthilkumar.peelikkampatti@REDACTED (Senthilkumar Peelikkampatti) Date: Fri, 3 Sep 2010 17:07:06 -0500 Subject: high performance cache Message-ID: Hi, I am looking for LRU caching framework in Erlang with thresholds to control the amount of data written to it. I explored memcached Erlang derivatives and ETS based solutions but not sure about which way to go. Please share your experience using caching framework. I am not looking for distributed cache. -- Regards, Senthilkumar Peelikkampatti, From rvirding@REDACTED Sat Sep 4 01:09:42 2010 From: rvirding@REDACTED (Robert Virding) Date: Sat, 4 Sep 2010 01:09:42 +0200 Subject: [erlang-questions] Suffix array implementation in Erlang? In-Reply-To: References: Message-ID: Why do a NIF implementation? Or don't you plan to use erlang data structures? Robert On 3 September 2010 19:13, zabrane Mikael wrote: > Hi Metthew, > > Yep, this is what I'm planning is no pure Erlang implementation is available. > > -- > Regards > Zabrane > > > 2010/9/3 Evans, Matthew : >> Sounds like a good use for a NIF implementation. >> >> -----Original Message----- >> From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of zabrane Mikael >> Sent: Friday, September 03, 2010 11:17 AM >> To: Erlang Questions Mailinglist >> Subject: [erlang-questions] Suffix array implementation in Erlang? >> >> Hi list, >> >> I'm looking after a fast implementation of "Suffix array" in Erlang: >> http://en.wikipedia.org/wiki/Suffix_array >> >> Any pointer/code will be very welcome! >> >> -- >> Regards >> Zabrane >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From fritchie@REDACTED Sat Sep 4 01:32:03 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 03 Sep 2010 18:32:03 -0500 Subject: [erlang-questions] ACM Erlang Workshop is almost here: 30 September 2010 In-Reply-To: Message of "Fri, 03 Sep 2010 12:22:36 CDT." <81378.1283534556@snookles.snookles.com> Message-ID: <98984.1283556723@snookles.snookles.com> I have an update from David Smith, the ACM Erlang Workshop's invited speaker this year. He has refined the title of this talk: Rebar, Bitcask and how chemotherapy made me a better developer. (take 2) See the original at: http://twitter.com/dizzyd/status/22927890584 -Scott Workshop Chair, 2010 ACM Erlang Workshop http://www.erlang.org/workshop/2010/ From zabrane3@REDACTED Sat Sep 4 13:29:50 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Sat, 4 Sep 2010 13:29:50 +0200 Subject: [erlang-questions] Suffix array implementation in Erlang? In-Reply-To: References: Message-ID: Hi Robert, 2010/9/4 Robert Virding : > Why do a NIF implementation? Performance ! There's a lot of rock solid C implementations out there. > Or don't you plan to use erlang data structures? Yep, but before reinventing the wheel, I'd like to know if a pure Erlang implementation exists. -- Regards Zabrane From tony@REDACTED Sat Sep 4 13:10:26 2010 From: tony@REDACTED (Tony Rogvall) Date: Sat, 4 Sep 2010 13:10:26 +0200 Subject: [erlang-questions] Ordered set and DETS In-Reply-To: References: Message-ID: <3392D0EC-D3E4-4B13-AE18-583A632FF94E@rogvall.se> +1 Please /Tony On 3 sep 2010, at 22.28, Evans, Matthew wrote: > Hi, > > Does anyone know if there are any roadmap plans to implement ordered_set table types in DETS? > > Thanks From jesper.louis.andersen@REDACTED Sat Sep 4 14:24:51 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 4 Sep 2010 14:24:51 +0200 Subject: [erlang-questions] high performance cache In-Reply-To: References: Message-ID: On Sat, Sep 4, 2010 at 12:07 AM, Senthilkumar Peelikkampatti wrote: > I am looking for LRU caching framework in Erlang with thresholds to control > the amount of data written to it. I explored memcached Erlang derivatives > and ETS based solutions but not sure about which way to go. Please share > your experience using caching framework. I am not looking for distributed > cache. Etorrent has a very very simple LRU janitorialization on file descriptors. The principle is this: Every FD is governed by a process. These processes enters themself into a janitor ETS table. Upon reaching a high watermark, we do a full scan of the ETS table, find the oldest beasts not accessed recently and then kill them. It is viable because the number of open descriptors tend to be fairly small, so we are not paying that much for a full table scan. Code: http://github.com/jlouis/etorrent/blob/master/src/etorrent_fs_janitor.erl If the table ends up being very big, I guess one should clean it up based on a timer and a heuristic on how much stale data is in the table. Also, beware of storing too much data in ETS as it is copied into the memory space of the process in question. Same caveats apply as when you try to send a 32mb binary tree from one process to the other (in the default GC/allocator). My guess is you can take the code above and use it for inspiration to build an LRU ETS cache. It shouldn't take more than a couple of days max to get it into shape. -- J. From gleber.p@REDACTED Sat Sep 4 16:13:28 2010 From: gleber.p@REDACTED (Gleb Peregud) Date: Sat, 4 Sep 2010 16:13:28 +0200 Subject: [erlang-questions] high performance cache In-Reply-To: References: Message-ID: Cherly[1] might be interesting for you 1: http://github.com/cliffmoon/cherly On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti wrote: > Hi, > > I am looking for LRU caching framework in Erlang with thresholds to control > the amount of data written to it. I explored memcached Erlang derivatives > and ETS based solutions but not sure about which way to go. Please share > your experience using caching framework. I am not looking for distributed > cache. > > -- > Regards, > Senthilkumar Peelikkampatti, > From rvirding@REDACTED Sat Sep 4 16:33:33 2010 From: rvirding@REDACTED (Robert Virding) Date: Sat, 4 Sep 2010 16:33:33 +0200 Subject: [erlang-questions] Suffix array implementation in Erlang? In-Reply-To: References: Message-ID: On 4 September 2010 13:29, zabrane Mikael wrote: > Hi Robert, > > 2010/9/4 Robert Virding : >> Why do a NIF implementation? > > Performance ! There's a lot of rock solid C implementations out there. > >> Or don't you plan to use erlang data structures? > > Yep, but before reinventing the wheel, I'd like to know if a pure > Erlang implementation exists. Where I was trying (unsuccessfully) to go was that if you were going to use normal erlang data structures (lists/tuples) to build the tree and store general erlang terms in it then I would be surprised if you can use destructive algorithms for the array as they work on the normal Erlang heaps. In which case you might not be able to directly use these C implementations. In which case it is probably easier to work on the tree directly in Erlang instead of in C in a NIF, seeing you have to modify the algorithms anyway. I can't remember what it says about destructive operations on heap data in the NIF docs. Robert From igorrs@REDACTED Sat Sep 4 18:31:26 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Sat, 4 Sep 2010 13:31:26 -0300 Subject: [erlang-questions] Mnesia/dets question - does this make sense? In-Reply-To: References: Message-ID: Just another guess that's probably wrong: do you have write-back caching enabled for your storage? If so, does disabling it make writing times more uniform? Igor. On Fri, Sep 3, 2010 at 11:25 AM, Evans, Matthew wrote: > Possibly. In this case we are running on Linux, but the file system is EXT3 running on flash/SSD, so there shouldn't be any spinning disks that need to get moving. > > There are of course buffers down in the kernel space, but I imagine that they are global and not per file, so they shouldn't cause this behavior. > > It's not a huge issue, but could be if/when we need to grow the number of fragments. > > Matt > > ________________________________ > From: Nicholas Frechette [mailto:zeno490@REDACTED] > Sent: Thursday, September 02, 2010 6:58 PM > To: Evans, Matthew > Cc: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Mnesia/dets question - does this make sense? > > Disc_only tables probably write to disk right away and probably don't buffer so writes can be atomic. > I'm thinking your problem is more likely at the OS/HDD driver level. It is possible your OS/HDD is caching the data in memory for short amounts of time in order to speed up subsequent access to said data, once flushed you pay the full price of accessing that section of the disk again (worse if the disk had stopped spinning). For example, Windows 7 is actually quite good at this, using all available memory that isn't being used by applications to buffer IO (or something like that). > > Just a wild guess though, you'll have to profile to understand what is happening (perhaps at the kernel level to see how much time is spend actually writing the data by the kernel VS time spent in erlang code). > 2cents > Nicholas > On Thu, Sep 2, 2010 at 5:33 PM, Evans, Matthew > wrote: > Hi, > > I have a fragmented disc_only mnesia table. During tests I have noticed that it appears that the access (insert) times to the fragment improve over time, and then slow down after a period of inactivity. > > Do the tables/fragments get opened and then closed based on a timer that would affect initial inert times? > > For example, the table has 10 fragments. I see insert times in the 4000us range for the first 10 to 15 inserts, and then insert times averaging 400us subsequently. If there is no activity on the table for a few minutes access times are again in the 4000us range for the first 10 to 15 inserts. As I said before it's almost like there is an overhead to get things started/warmed up (e.g. opening the fragments/files), once all the fragments are opened the speed is fine. > > Can anyone confirm this? > > Thanks > > Matt > > -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From ulf.wiger@REDACTED Sat Sep 4 19:16:03 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Sat, 04 Sep 2010 19:16:03 +0200 Subject: [erlang-questions] Ordered set and DETS In-Reply-To: References: Message-ID: <4C827ED3.7080004@erlang-solutions.com> On 09/03/2010 10:28 PM, Evans, Matthew wrote: > Hi, > > Does anyone know if there are any roadmap plans to implement ordered_set table types in DETS? > > Thanks It may be possible to build on the combination of Tokyo Tyrant and my sortable serialization library, which I mentioned in this post a while ago: http://www.erlang.org/cgi-bin/ezmlm-cgi/4/47285 That was part of the justification for adding prefix matching and matchspec-style functionality in the sext library in the first place. BR, Ulf W From zeno490@REDACTED Sat Sep 4 23:00:27 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Sat, 4 Sep 2010 17:00:27 -0400 Subject: [erlang-questions] high performance cache In-Reply-To: References: Message-ID: I believe the most efficient caching algorithm is the landlord algo. Google it, i'm sure you'll find it. Great algo for caching slow fetching resources (files, sounds, webpages, etc.). Not sure if the is an implementation in erlang already or not but it should be fairly trivial to implement yourself. On Sat, Sep 4, 2010 at 10:13 AM, Gleb Peregud wrote: > Cherly[1] might be interesting for you > > 1: http://github.com/cliffmoon/cherly > > On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti > wrote: > > Hi, > > > > I am looking for LRU caching framework in Erlang with thresholds to > control > > the amount of data written to it. I explored memcached Erlang derivatives > > and ETS based solutions but not sure about which way to go. Please share > > your experience using caching framework. I am not looking for distributed > > cache. > > > > -- > > Regards, > > Senthilkumar Peelikkampatti, > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From anthonym@REDACTED Sun Sep 5 00:50:32 2010 From: anthonym@REDACTED (Anthony Molinaro) Date: Sat, 4 Sep 2010 15:50:32 -0700 Subject: [erlang-questions] Ordered set and DETS In-Reply-To: References: Message-ID: <20100904225032.GB16109@alumni.caltech.edu> On Fri, Sep 03, 2010 at 04:28:55PM -0400, Evans, Matthew wrote: > Does anyone know if there are any roadmap plans to implement ordered_set table types in DETS? It might be a tad out of date now, but a couple years ago a company I worked for had a consultant write mnesiax http://code.google.com/p/mnesiaex/ Which made the mnesia storage backend a behaviour IIRC. Then we used tcerl http://code.google.com/p/tcerl/ to keep ordered_set disk backed tables in mnesia. It's probably bit-rotted a bit but maybe worth dusting off? -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro From hirotnkg@REDACTED Sun Sep 5 08:02:43 2010 From: hirotnkg@REDACTED (Yoshi) Date: Sat, 4 Sep 2010 23:02:43 -0700 (PDT) Subject: Global names of hidden node Message-ID: Hello, Is there any way to make global registered names of hidden node visible after I connect to the hidden node from other hidden node? In my distributed Erlang network, I don't want nodes connect transitively and using -hidden flag, but this leads to the situation I cannot use globally registered names on other nodes even after I connect to those nodes using net_kernel:connect_node/1. For example, when I start a node, with -hidden flag, I cannot see the global registered names on 'node_foo@REDACTED' even after I connect to it below: hiro@REDACTED:~$ erl -pa ebin -hidden -setcookie harmonia_cookie -sname 'bar_node@REDACTED' Erlang R13B04 (erts-5.7.5) [source] [rq:1] [async-threads:0] [kernel-poll:false] Eshell V5.7.5 (abort with ^G) (bar_node@REDACTED)1> global:registered_names(). [] (bar_node@REDACTED)2> net_kernel:connect_node(node_foo@REDACTED). true (bar_node@REDACTED)3> global:registered_names(). [] It seems I can use rpc:call/4 and global:register_name/2 in order to do what I want: (bar_node@REDACTED)4> rpc:call(node_foo@REDACTED, global, registered_names, []). [hm_router_foo,hm_config_if_foo,hm_stabilizer_foo, hm_name_server,hm_ds_foo,foo,hm_table_foo] (bar_node@REDACTED)5> Pid = rpc:call(node_foo@REDACTED, global, whereis_name, [hm_router_foo]). <6351.53.0> (bar_node@REDACTED)6> global:register_name(hm_router_foo, Pid). yes (bar_node@REDACTED)7> global:registered_names(). [hm_router_foo] (bar_node@REDACTED)8> gen_server:call({global, hm_router_foo}, {find_successor, 38, nil}). {hm_router_foo,90} (bar_node@REDACTED)9> But is this a proper way and is there a better way? Or I should not consider to do this in the first place? Thank you From zeno490@REDACTED Sun Sep 5 18:51:48 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Sun, 5 Sep 2010 12:51:48 -0400 Subject: [erlang-questions] high performance cache In-Reply-To: References: Message-ID: For those interested, the landlord algorithm was published by Neal E. Young at (a quick google search later): http://www.cs.ucr.edu/~neal/publications The algorithm paper itself is under On-line file caching (click on it to see a brief description and to download/view the full pdf). Some of his other papers are also of interest. Per the webpage above: *? Copyrights are reserved by the publishers. Download for personal and limited academic use only. * On Sat, Sep 4, 2010 at 5:00 PM, Nicholas Frechette wrote: > I believe the most efficient caching algorithm is the landlord algo. Google > it, i'm sure you'll find it. Great algo for caching slow fetching resources > (files, sounds, webpages, etc.). Not sure if the is an implementation in > erlang already or not but it should be fairly trivial to implement yourself. > > > On Sat, Sep 4, 2010 at 10:13 AM, Gleb Peregud wrote: > >> Cherly[1] might be interesting for you >> >> 1: http://github.com/cliffmoon/cherly >> >> On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti >> wrote: >> > Hi, >> > >> > I am looking for LRU caching framework in Erlang with thresholds to >> control >> > the amount of data written to it. I explored memcached Erlang >> derivatives >> > and ETS based solutions but not sure about which way to go. Please share >> > your experience using caching framework. I am not looking for >> distributed >> > cache. >> > >> > -- >> > Regards, >> > Senthilkumar Peelikkampatti, >> > >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > From ttom.kelly@REDACTED Sun Sep 5 20:45:57 2010 From: ttom.kelly@REDACTED (tom kelly) Date: Sun, 5 Sep 2010 19:45:57 +0100 Subject: Random behaviour Message-ID: Hello List, I just found a bahaviour I didn't expect in OTPs random module, it's not necessarily a bug but I thought I'd share it here all the same. I wrote a simple random data generator using random:uniform() to help me track a bug that was triggered by certain input data that I couldn't catch. The minimal example is attached in demo.erl. If you run the test function it will probably pass a few times but will eventually hit my "bug" and fail. Whenever the bug is triggered the test function gets a badmatch and the current process dies. The behaviour I wasn't expecting now occurs, everytime I repeat the test function I get the exact same sequence of numbers from random:uniform. I can call random:uniform from the shell to move it on one step in the sequence and probably pass another test, but when it fails again my process dies and again I repeatedly get a repeated sequence of numbers. The sequence from this second fail is different from the first fail so it hasn't just gone back to the start. I don't expect to be saved from my own bugs but just thought this was a strange behaviour from random. If anyone is using random in their application they should be aware of this. //Tom. 8> demo:test_my_buggy_code(). ok - [15,21,70,16,56,22,46,43,1,57] ok 9> demo:test_my_buggy_code(). ok - [41,31,6,58,99,34,19,21,4,89] ok 10> demo:test_my_buggy_code(). ok - [83,33,26,81,2,3,5,6,99,57] ok 11> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, 72,41,80] in function demo:test_my_buggy_code/0 in call from random:uniform/0 12> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, 72,41,80] in function demo:test_my_buggy_code/0 in call from random:uniform/0 13> random:uniform(1). 1 14> demo:test_my_buggy_code(). ok - [67,88,90,80,59,30,72,41,80,8] ok 15> demo:test_my_buggy_code(). ok - [60,68,65,8,27,61,1,94,82,65] ok 16> demo:test_my_buggy_code(). ok - [47,2,62,94,25,97,94,87,27,32] ok 17> demo:test_my_buggy_code(). ok - [59,21,10,14,17,24,93,78,34,28] ok 18> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, 60,42,2] in function demo:test_my_buggy_code/0 in call from random:uniform/0 19> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, 60,42,2] in function demo:test_my_buggy_code/0 in call from random:uniform/0 20> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, 60,42,2] in function demo:test_my_buggy_code/0 in call from random:uniform/0 21> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: demo.erl Type: application/octet-stream Size: 320 bytes Desc: not available URL: From zabrane3@REDACTED Sun Sep 5 20:48:36 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Sun, 5 Sep 2010 20:48:36 +0200 Subject: [erlang-questions] high performance cache In-Reply-To: References: Message-ID: Thanks Nicolas 2010/9/5 Nicholas Frechette : > For those interested, the landlord algorithm was published by Neal E. Young > at (a quick google search later): > http://www.cs.ucr.edu/~neal/publications > > The algorithm paper itself is under On-line file caching (click on it to see > a brief description and to download/view the full pdf). > Some of his other papers are also of interest. > > Per the webpage above: > *? Copyrights are reserved by the publishers. > Download for personal and limited academic use only. * > > On Sat, Sep 4, 2010 at 5:00 PM, Nicholas Frechette wrote: > >> I believe the most efficient caching algorithm is the landlord algo. Google >> it, i'm sure you'll find it. Great algo for caching slow fetching resources >> (files, sounds, webpages, etc.). Not sure if the is an implementation in >> erlang already or not but it should be fairly trivial to implement yourself. >> >> >> On Sat, Sep 4, 2010 at 10:13 AM, Gleb Peregud wrote: >> >>> Cherly[1] might be interesting for you >>> >>> 1: http://github.com/cliffmoon/cherly >>> >>> On Sat, Sep 4, 2010 at 00:07, Senthilkumar Peelikkampatti >>> wrote: >>> > Hi, >>> > >>> > I am looking for LRU caching framework in Erlang with thresholds to >>> control >>> > the amount of data written to it. I explored memcached Erlang >>> derivatives >>> > and ETS based solutions but not sure about which way to go. Please share >>> > your experience using caching framework. I am not looking for >>> distributed >>> > cache. >>> > >>> > -- >>> > Regards, >>> > Senthilkumar Peelikkampatti, >>> > >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >>> >> > -- Regards Zabrane From vladdu55@REDACTED Sun Sep 5 21:13:29 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 5 Sep 2010 21:13:29 +0200 Subject: [erlang-questions] Random behaviour In-Reply-To: References: Message-ID: On Sun, Sep 5, 2010 at 20:45, tom kelly wrote: > I just found a bahaviour I didn't expect in OTPs random module, it's not > necessarily a bug but I thought I'd share it here all the same. Hi Tom, The problem is in your code... test_my_buggy_code() -> Data = lists:map(fun(_) -> random:uniform(100) end, lists:seq(1,10)), Data = my_buggy_code(Data), io:format("ok - ~p~n",[Data]). my_buggy_code(Data) -> N = random:uniform(10), if N > 3 -> Data; true -> [1|Data] end. If N is larger than 3, then my_buggy_code returns the same list and the second line in test_my_buggy_code matches. Otherwise, Data can never match [1|Data]. You probably want Data = lists:map(fun(_) -> random:uniform(100) end, lists:seq(1,10)), ProcessedData = my_buggy_code(Data), io:format("ok - ~p~n",[ProcessedData]). best regards, Vlad From ok@REDACTED Mon Sep 6 00:02:51 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 6 Sep 2010 10:02:51 +1200 Subject: [erlang-questions] Modules and function "encapsulation" In-Reply-To: <4C7DF3C5.5080302@erlang-solutions.com> References: <20100901031645.GA2088@hope.tifa.renegado> <4C7DCF24.9070407@aist.go.jp> <20100901041559.GB2088@hope.tifa.renegado> <4C7DF3C5.5080302@erlang-solutions.com> Message-ID: On Sep 1, 2010, at 6:33 PM, Ulf Wiger wrote: > The EUC talk mentioned, by Richard Carlsson, has been implemented as > an experimental feature. http://www.trapexit.org/Extend_Module > > Note the experimental status. This is not to say it doesn't work - > I daresay it does - but that there is no commitment to keep the > feature as-is. I read the slides, but I'm afraid they are rather vague. Is there a more precise specification anywhere? At first blush, this does nothing that - (1) explicitly importing the base module's exports into the extension module, and - (2) allowing the extension module to re-export imported functions would not do. The only problem with doing it that way is that the Erlang compiler won't LET you: % cat -n foo.erl 1 -module(foo). 2 -import(bar, [ugh/1]). 3 -export([ugh/1, zoo/2]). 4 5 zoo(X, Y) -> ugh(X+Y). 6 % erlc foo.erl ./foo.erl:3: function ugh/1 undefined Relaxing this apparently pointless restriction would be a far simpler change to the language than adding -extends; it would provide a very very simple way for a module to extend MANY modules; and it would be much better than -extends because it would be EXPLICIT. As it stands, -extends(bar) says "I don't know and I don't care what bar exports. I trust bar TOTALLY. Whatever bar exports, today, tomorrow, or until the Sun goes out, I export that too." That really does not seem to be in the spirit of Erlang to me. So I must have misunderstood the proposal. From rvirding@REDACTED Mon Sep 6 02:29:12 2010 From: rvirding@REDACTED (Robert Virding) Date: Mon, 6 Sep 2010 02:29:12 +0200 Subject: [erlang-questions] Modules and function "encapsulation" In-Reply-To: References: <20100901031645.GA2088@hope.tifa.renegado> <4C7DCF24.9070407@aist.go.jp> <20100901041559.GB2088@hope.tifa.renegado> <4C7DF3C5.5080302@erlang-solutions.com> Message-ID: The way I have understood it is that it does exactly what you say in your last paragraph. I don't like it, but I could live with it, though I probably would not use it. It has the feeling of viewing modules as OO classes and wishing to provide a way of inheriting them. :-) Robert On 6 September 2010 00:02, Richard O'Keefe wrote: > > On Sep 1, 2010, at 6:33 PM, Ulf Wiger wrote: >> The EUC talk mentioned, by Richard Carlsson, has been implemented as >> an experimental feature. http://www.trapexit.org/Extend_Module >> >> Note the experimental status. This is not to say it doesn't work - >> I daresay it does - but that there is no commitment to keep the >> feature as-is. > > I read the slides, but I'm afraid they are rather vague. > Is there a more precise specification anywhere? > > At first blush, this does nothing that > ?- (1) explicitly importing the base module's exports into > ? ? ? the extension module, and > ?- (2) allowing the extension module to re-export imported > ? ? ? functions > would not do. ?The only problem with doing it that way is > that the Erlang compiler won't LET you: > % cat -n foo.erl > ? ? 1 ?-module(foo). > ? ? 2 ?-import(bar, [ugh/1]). > ? ? 3 ?-export([ugh/1, zoo/2]). > ? ? 4 > ? ? 5 ?zoo(X, Y) -> ugh(X+Y). > ? ? 6 > % erlc foo.erl > ./foo.erl:3: function ugh/1 undefined > > Relaxing this apparently pointless restriction would be a far > simpler change to the language than adding -extends; it would > provide a very very simple way for a module to extend MANY > modules; and it would be much better than -extends because it > would be EXPLICIT. ?As it stands, -extends(bar) says > "I don't know and I don't care what bar exports. > ?I trust bar TOTALLY. > ?Whatever bar exports, today, tomorrow, or until the Sun > ?goes out, I export that too." > That really does not seem to be in the spirit of Erlang to me. > > So I must have misunderstood the proposal. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From senthilkumar.peelikkampatti@REDACTED Mon Sep 6 05:33:21 2010 From: senthilkumar.peelikkampatti@REDACTED (Senthilkumar Peelikkampatti) Date: Sun, 5 Sep 2010 22:33:21 -0500 Subject: gen_server as Pid based callback Message-ID: I have a 3rd party utility module which takes Pid as one of the input to later callback. I want to explore how gen_server registered/supervised process can be used for this purpose. How to use registered/supervised gen_server in place of Pid. -- Regards, Senthilkumar Peelikkampatti, From john.hughes@REDACTED Mon Sep 6 07:26:25 2010 From: john.hughes@REDACTED (John Hughes) Date: Mon, 6 Sep 2010 07:26:25 +0200 Subject: [erlang-questions] Random behaviour In-Reply-To: References: Message-ID: <6FB7DAFE1ABF46789DA9D4A262F70D2D@HALL> Hi Tom, Every Erlang process has its own random number seed, and it's initialised to a CONSTANT value when the process starts! Hence given foo() -> io:format("~p\n",[[random:uniform(10) || _ <- lists:seq(1,10)]]). then you get different values from multiple calls in the SAME process: 2> foo:foo(). [1,5,8,10,6,4,6,10,7,5] ok 3> foo:foo(). [6,2,3,7,2,6,3,5,5,1] ok 4> foo:foo(). [6,5,5,4,1,6,10,4,2,3] ok but the same value every time in a NEW process: 5> spawn(foo,foo,[]). [1,5,8,10,6,4,6,10,7,5] <0.45.0> 6> spawn(foo,foo,[]). [1,5,8,10,6,4,6,10,7,5] <0.47.0> 7> spawn(foo,foo,[]). [1,5,8,10,6,4,6,10,7,5] <0.49.0> I guess it makes sense--you don't want to pay the price of initialising a random number seed every time you start a process--but it is a bit of a gotcha. If you want different random numbers in different processes (which you usually do), then you need to initialise the seed differently when you start the process. The usual trick is to use now(), since a seed just happens to be three integers: baz() -> {A,B,C} = now(), random:seed(A,B,C), foo(). and now 10> spawn(foo,baz,[]). [10,7,9,4,7,1,4,8,3,6] <0.61.0> 11> spawn(foo,baz,[]). [10,5,1,6,5,5,9,3,6,2] <0.63.0> 12> spawn(foo,baz,[]). [8,7,9,7,9,6,1,9,4,3] <0.65.0> One thing to be aware of is that calls to now() close together in time give you seeds the initially generate similar numbers: 13> spawn(foo,baz,[]),spawn(foo,baz,[]). [10,1,8,1,8,1,8,10,1,9] [10,1,7,1,6,5,7,8,8,2] <0.68.0> 14> spawn(foo,baz,[]),spawn(foo,baz,[]). [5,9,8,2,2,3,3,9,5,8] [5,9,7,2,9,6,2,7,2,1] <0.71.0> 15> spawn(foo,baz,[]),spawn(foo,baz,[]). [10,5,2,3,4,4,3,1,6,3] [10,5,1,4,2,8,3,9,3,7] <0.74.0> Note the first two numbers generated are always the same in both processes, even though they start with different seeds. If you're going to start many processes and you want to be sure they have independent random numbers then you have to be a bit more careful how you initialise the seed--but if there are going to be seconds or more between each process start then you don't have to worry. John ----- Original Message ----- From: tom kelly To: erlang-questions@REDACTED Sent: Sunday, September 05, 2010 8:45 PM Subject: [erlang-questions] Random behaviour Hello List, I just found a bahaviour I didn't expect in OTPs random module, it's not necessarily a bug but I thought I'd share it here all the same. I wrote a simple random data generator using random:uniform() to help me track a bug that was triggered by certain input data that I couldn't catch. The minimal example is attached in demo.erl. If you run the test function it will probably pass a few times but will eventually hit my "bug" and fail. Whenever the bug is triggered the test function gets a badmatch and the current process dies. The behaviour I wasn't expecting now occurs, everytime I repeat the test function I get the exact same sequence of numbers from random:uniform. I can call random:uniform from the shell to move it on one step in the sequence and probably pass another test, but when it fails again my process dies and again I repeatedly get a repeated sequence of numbers. The sequence from this second fail is different from the first fail so it hasn't just gone back to the start. I don't expect to be saved from my own bugs but just thought this was a strange behaviour from random. If anyone is using random in their application they should be aware of this. //Tom. 8> demo:test_my_buggy_code(). ok - [15,21,70,16,56,22,46,43,1,57] ok 9> demo:test_my_buggy_code(). ok - [41,31,6,58,99,34,19,21,4,89] ok 10> demo:test_my_buggy_code(). ok - [83,33,26,81,2,3,5,6,99,57] ok 11> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, 72,41,80] in function demo:test_my_buggy_code/0 in call from random:uniform/0 12> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, 72,41,80] in function demo:test_my_buggy_code/0 in call from random:uniform/0 13> random:uniform(1). 1 14> demo:test_my_buggy_code(). ok - [67,88,90,80,59,30,72,41,80,8] ok 15> demo:test_my_buggy_code(). ok - [60,68,65,8,27,61,1,94,82,65] ok 16> demo:test_my_buggy_code(). ok - [47,2,62,94,25,97,94,87,27,32] ok 17> demo:test_my_buggy_code(). ok - [59,21,10,14,17,24,93,78,34,28] ok 18> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, 60,42,2] in function demo:test_my_buggy_code/0 in call from random:uniform/0 19> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, 60,42,2] in function demo:test_my_buggy_code/0 in call from random:uniform/0 20> demo:test_my_buggy_code(). ** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, 60,42,2] in function demo:test_my_buggy_code/0 in call from random:uniform/0 21> ------------------------------------------------------------------------------ ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From bob@REDACTED Mon Sep 6 07:39:50 2010 From: bob@REDACTED (Bob Ippolito) Date: Mon, 6 Sep 2010 13:39:50 +0800 Subject: [erlang-questions] Random behaviour In-Reply-To: <6FB7DAFE1ABF46789DA9D4A262F70D2D@HALL> References: <6FB7DAFE1ABF46789DA9D4A262F70D2D@HALL> Message-ID: You might also want to look at the crypto module for generating random numbers instead of the random module. The random module uses an ancient algorithm and has issues with seeding (as you have noticed). On Mon, Sep 6, 2010 at 1:26 PM, John Hughes wrote: > Hi Tom, > > Every Erlang process has its own random number seed, and it's initialised to a CONSTANT value when the process starts! Hence given > > foo() -> > ? ?io:format("~p\n",[[random:uniform(10) || _ <- lists:seq(1,10)]]). > > then you get different values from multiple calls in the SAME process: > > 2> foo:foo(). > [1,5,8,10,6,4,6,10,7,5] > ok > 3> foo:foo(). > [6,2,3,7,2,6,3,5,5,1] > ok > 4> foo:foo(). > [6,5,5,4,1,6,10,4,2,3] > ok > > but the same value every time in a NEW process: > > 5> spawn(foo,foo,[]). > [1,5,8,10,6,4,6,10,7,5] > <0.45.0> > 6> spawn(foo,foo,[]). > [1,5,8,10,6,4,6,10,7,5] > <0.47.0> > 7> spawn(foo,foo,[]). > [1,5,8,10,6,4,6,10,7,5] > <0.49.0> > > I guess it makes sense--you don't want to pay the price of initialising a random number seed every time you start a process--but it is a bit of a gotcha. > > If you want different random numbers in different processes (which you usually do), then you need to initialise the seed differently when you start the process. The usual trick is to use now(), since a seed just happens to be three integers: > > baz() -> > ? ?{A,B,C} = now(), > ? ?random:seed(A,B,C), > ? ?foo(). > > and now > > 10> spawn(foo,baz,[]). > [10,7,9,4,7,1,4,8,3,6] > <0.61.0> > 11> spawn(foo,baz,[]). > [10,5,1,6,5,5,9,3,6,2] > <0.63.0> > 12> spawn(foo,baz,[]). > [8,7,9,7,9,6,1,9,4,3] > <0.65.0> > > One thing to be aware of is that calls to now() close together in time give you seeds the initially generate similar numbers: > > 13> spawn(foo,baz,[]),spawn(foo,baz,[]). > [10,1,8,1,8,1,8,10,1,9] > [10,1,7,1,6,5,7,8,8,2] > <0.68.0> > 14> spawn(foo,baz,[]),spawn(foo,baz,[]). > [5,9,8,2,2,3,3,9,5,8] > [5,9,7,2,9,6,2,7,2,1] > <0.71.0> > 15> spawn(foo,baz,[]),spawn(foo,baz,[]). > [10,5,2,3,4,4,3,1,6,3] > [10,5,1,4,2,8,3,9,3,7] > <0.74.0> > > Note the first two numbers generated are always the same in both processes, even though they start with different seeds. If you're going to start many processes and you want to be sure they have independent random numbers then you have to be a bit more careful how you initialise the seed--but if there are going to be seconds or more between each process start then you don't have to worry. > > John > ?----- Original Message ----- > ?From: tom kelly > ?To: erlang-questions@REDACTED > ?Sent: Sunday, September 05, 2010 8:45 PM > ?Subject: [erlang-questions] Random behaviour > > > ?Hello List, > > > ?I just found a bahaviour I didn't expect in OTPs random module, it's not necessarily a bug but I thought I'd share it here all the same. > > > ?I wrote a simple random data generator using random:uniform() to help me track a bug that was triggered by certain input data that I couldn't catch. The minimal example is attached in demo.erl. > > > ?If you run the test function it will probably pass a few times but will eventually hit my "bug" and fail. Whenever the bug is triggered the test function gets a badmatch and the current process dies. The behaviour I wasn't expecting now occurs, everytime I repeat the test function I get the exact same sequence of numbers from random:uniform. > > > ?I can call random:uniform from the shell to move it on one step in the sequence and probably pass another test, but when it fails again my process dies and again I repeatedly get a repeated sequence of numbers. The sequence from this second fail is different from the first fail so it hasn't just gone back to the start. > > > ?I don't expect to be saved from my own bugs but just thought this was a strange behaviour from random. If anyone is using random in their application they should be aware of this. > > > ?//Tom. > > > > > ?8> demo:test_my_buggy_code(). > ?ok - [15,21,70,16,56,22,46,43,1,57] > ?ok > ?9> demo:test_my_buggy_code(). > ?ok - [41,31,6,58,99,34,19,21,4,89] > ?ok > ?10> demo:test_my_buggy_code(). > ?ok - [83,33,26,81,2,3,5,6,99,57] > ?ok > ?11> demo:test_my_buggy_code(). > ?** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 72,41,80] > ? ? ? in function ?demo:test_my_buggy_code/0 > ? ? ? in call from random:uniform/0 > ?12> demo:test_my_buggy_code(). > ?** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 72,41,80] > ? ? ? in function ?demo:test_my_buggy_code/0 > ? ? ? in call from random:uniform/0 > ?13> random:uniform(1). > ?1 > ?14> demo:test_my_buggy_code(). > ?ok - [67,88,90,80,59,30,72,41,80,8] > ?ok > ?15> demo:test_my_buggy_code(). > ?ok - [60,68,65,8,27,61,1,94,82,65] > ?ok > ?16> demo:test_my_buggy_code(). > ?ok - [47,2,62,94,25,97,94,87,27,32] > ?ok > ?17> demo:test_my_buggy_code(). > ?ok - [59,21,10,14,17,24,93,78,34,28] > ?ok > ?18> demo:test_my_buggy_code(). > ?** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 60,42,2] > ? ? ? in function ?demo:test_my_buggy_code/0 > ? ? ? in call from random:uniform/0 > ?19> demo:test_my_buggy_code(). > ?** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 60,42,2] > ? ? ? in function ?demo:test_my_buggy_code/0 > ? ? ? in call from random:uniform/0 > ?20> demo:test_my_buggy_code(). > ?** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 60,42,2] > ? ? ? in function ?demo:test_my_buggy_code/0 > ? ? ? in call from random:uniform/0 > ?21> > > > > > ------------------------------------------------------------------------------ > > > > ?________________________________________________________________ > ?erlang-questions (at) erlang.org mailing list. > ?See http://www.erlang.org/faq.html > ?To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From ulf.wiger@REDACTED Mon Sep 6 08:38:46 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 06 Sep 2010 08:38:46 +0200 Subject: [erlang-questions] Modules and function "encapsulation" In-Reply-To: References: <20100901031645.GA2088@hope.tifa.renegado> <4C7DCF24.9070407@aist.go.jp> <20100901041559.GB2088@hope.tifa.renegado> <4C7DF3C5.5080302@erlang-solutions.com> Message-ID: <4C848C76.10902@erlang-solutions.com> On 09/06/2010 12:02 AM, Richard O'Keefe wrote: > > As it stands, -extends(bar) says > "I don't know and I don't care what bar exports. > I trust bar TOTALLY. > Whatever bar exports, today, tomorrow, or until the Sun > goes out, I export that too." > That really does not seem to be in the spirit of Erlang to me. OTOH, it is pretty much exactly what one wants when e.g. providing a mnesia callback module - it is required to support all operations in the callback API, but in many cases, you really only care about modifying a few of the functions. When I last made a custom mnesia callback, it was to update vector clocks for the unsplit behaviour. The mnesia callback required looked something like this: -module(my_mnesia_cb). ... -include(?table_defs.hrl?). -export_records([....]). write(Tid, Ts, Tab, Rec, LockKind) -> Rec1 = try Old = ?#get-?(Rec, [vclock]), ?#set-?( Rec, [{vclock, unsplit_vclock:increment(node(), Old)}]) catch error:badarg -> Rec end, mnesia:write(Tid,Ts,Tab,Rec1,LockKind). Of course, in this case, the "..." could use the -extends() option, or it could explicitly import the needed callback functions. We can easily enough find out which ones they are: $ grep 'Mod:' $otp/lib/mnesia/src/mnesia.erl Mod:lock(Tid, Ts, LockItem, LockKind); Mod:write(Tid, Ts, Tab, Val, LockKind); Mod:delete(Tid, Ts, Tab, Key, LockKind); Mod:delete_object(Tid, Ts, Tab, Val, LockKind); Mod:read(Tid, Ts, Tab, Key, LockKind); Mod:first(Tid, Ts, Tab); Mod:last(Tid, Ts, Tab); Mod:next(Tid,Ts,Tab,Key); Mod:prev(Tid,Ts,Tab,Key); Mod:foldl(Tid, Ts, Fun, Acc, Tab, LockKind); Mod:foldr(Tid, Ts, Fun, Acc, Tab, LockKind); Mod:match_object(Tid, Ts, Tab, Pat, LockKind); Mod:select(Tid, Ts, Tab, Pat, LockKind); Mod:select(Tid, Ts, Tab, Pat, NObjects, LockKind); Mod:select_cont(Tid,Ts,Cont); Mod:all_keys(Tid, Ts, Tab, read); Mod:index_match_object(Tid, Ts, Tab, Pat, Attr, LockKind); Mod:index_read(Tid, Ts, Tab, Key, Attr, read); Mod:table_info(Tid, Ts, Tab, Item); Mod:clear_table(Tid, Ts, Tab, '_'); ...but in this case, I really only care about the write function, now and forever, unless mnesia would add another function that I'd have to wrap. Whether this is such a representative case that it warrants a language extension, I really can't say, and personally, I've yet to use -extends() myself. Of course, without -extends(), or at least your suggested change for -import(), writing such a mnesia callback is tedious and error-prone, since the majority of the line count goes into re-mapping functions that we really do not care about. BR, Ulf W From ttom.kelly@REDACTED Mon Sep 6 10:08:49 2010 From: ttom.kelly@REDACTED (tom kelly) Date: Mon, 6 Sep 2010 09:08:49 +0100 Subject: [erlang-questions] Random behaviour In-Reply-To: <6FB7DAFE1ABF46789DA9D4A262F70D2D@HALL> References: <6FB7DAFE1ABF46789DA9D4A262F70D2D@HALL> Message-ID: Hi John, Thanks for the comprehensive explanation! //Tom. On Mon, Sep 6, 2010 at 6:26 AM, John Hughes wrote: > Hi Tom, > > Every Erlang process has its own random number seed, and it's initialised > to a CONSTANT value when the process starts! Hence given > > foo() -> > io:format("~p\n",[[random:uniform(10) || _ <- lists:seq(1,10)]]). > then you get different values from multiple calls in the SAME process: > > 2> foo:foo(). > [1,5,8,10,6,4,6,10,7,5] > ok > 3> foo:foo(). > [6,2,3,7,2,6,3,5,5,1] > ok > 4> foo:foo(). > [6,5,5,4,1,6,10,4,2,3] > ok > > but the same value every time in a NEW process: > > 5> spawn(foo,foo,[]). > [1,5,8,10,6,4,6,10,7,5] > <0.45.0> > 6> spawn(foo,foo,[]). > [1,5,8,10,6,4,6,10,7,5] > <0.47.0> > 7> spawn(foo,foo,[]). > [1,5,8,10,6,4,6,10,7,5] > <0.49.0> > > I guess it makes sense--you don't want to pay the price of initialising a > random number seed every time you start a process--but it is a bit of a > gotcha. > > If you want different random numbers in different processes (which you > usually do), then you need to initialise the seed differently when you start > the process. The usual trick is to use now(), since a seed just happens to > be three integers: > > baz() -> > {A,B,C} = now(), > random:seed(A,B,C), > foo(). > and now > > 10> spawn(foo,baz,[]). > [10,7,9,4,7,1,4,8,3,6] > <0.61.0> > 11> spawn(foo,baz,[]). > [10,5,1,6,5,5,9,3,6,2] > <0.63.0> > 12> spawn(foo,baz,[]). > [8,7,9,7,9,6,1,9,4,3] > <0.65.0> > One thing to be aware of is that calls to now() close together in time give > you seeds the initially generate similar numbers: > > 13> spawn(foo,baz,[]),spawn(foo,baz,[]). > [10,1,8,1,8,1,8,10,1,9] > [10,1,7,1,6,5,7,8,8,2] > <0.68.0> > 14> spawn(foo,baz,[]),spawn(foo,baz,[]). > [5,9,8,2,2,3,3,9,5,8] > [5,9,7,2,9,6,2,7,2,1] > <0.71.0> > 15> spawn(foo,baz,[]),spawn(foo,baz,[]). > [10,5,2,3,4,4,3,1,6,3] > [10,5,1,4,2,8,3,9,3,7] > <0.74.0> > Note the first two numbers generated are always the same in both processes, > even though they start with different seeds. If you're going to start many > processes and you want to be sure they have independent random numbers then > you have to be a bit more careful how you initialise the seed--but if there > are going to be seconds or more between each process start then you don't > have to worry. > > John > > ----- Original Message ----- > *From:* tom kelly > *To:* erlang-questions@REDACTED > *Sent:* Sunday, September 05, 2010 8:45 PM > *Subject:* [erlang-questions] Random behaviour > > Hello List, > > I just found a bahaviour I didn't expect in OTPs random module, it's not > necessarily a bug but I thought I'd share it here all the same. > > I wrote a simple random data generator using random:uniform() to help me > track a bug that was triggered by certain input data that I couldn't catch. > The minimal example is attached in demo.erl. > > If you run the test function it will probably pass a few times but will > eventually hit my "bug" and fail. Whenever the bug is triggered the test > function gets a badmatch and the current process dies. The behaviour I > wasn't expecting now occurs, everytime I repeat the test function I get the > exact same sequence of numbers from random:uniform. > > I can call random:uniform from the shell to move it on one step in the > sequence and probably pass another test, but when it fails again my process > dies and again I repeatedly get a repeated sequence of numbers. The sequence > from this second fail is different from the first fail so it hasn't just > gone back to the start. > > I don't expect to be saved from my own bugs but just thought this was a > strange behaviour from random. If anyone is using random in their > application they should be aware of this. > > //Tom. > > > 8> demo:test_my_buggy_code(). > ok - [15,21,70,16,56,22,46,43,1,57] > ok > 9> demo:test_my_buggy_code(). > ok - [41,31,6,58,99,34,19,21,4,89] > ok > 10> demo:test_my_buggy_code(). > ok - [83,33,26,81,2,3,5,6,99,57] > ok > 11> demo:test_my_buggy_code(). > ** exception error: no match of right hand side value > [1,39,67,88,90,80,59,30, > 72,41,80] > in function demo:test_my_buggy_code/0 > in call from random:uniform/0 > 12> demo:test_my_buggy_code(). > ** exception error: no match of right hand side value > [1,39,67,88,90,80,59,30, > 72,41,80] > in function demo:test_my_buggy_code/0 > in call from random:uniform/0 > 13> random:uniform(1). > 1 > 14> demo:test_my_buggy_code(). > ok - [67,88,90,80,59,30,72,41,80,8] > ok > 15> demo:test_my_buggy_code(). > ok - [60,68,65,8,27,61,1,94,82,65] > ok > 16> demo:test_my_buggy_code(). > ok - [47,2,62,94,25,97,94,87,27,32] > ok > 17> demo:test_my_buggy_code(). > ok - [59,21,10,14,17,24,93,78,34,28] > ok > 18> demo:test_my_buggy_code(). > ** exception error: no match of right hand side value > [1,3,59,96,93,32,99,34, > 60,42,2] > in function demo:test_my_buggy_code/0 > in call from random:uniform/0 > 19> demo:test_my_buggy_code(). > ** exception error: no match of right hand side value > [1,3,59,96,93,32,99,34, > 60,42,2] > in function demo:test_my_buggy_code/0 > in call from random:uniform/0 > 20> demo:test_my_buggy_code(). > ** exception error: no match of right hand side value > [1,3,59,96,93,32,99,34, > 60,42,2] > in function demo:test_my_buggy_code/0 > in call from random:uniform/0 > 21> > > ------------------------------ > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From dmitrii@REDACTED Mon Sep 6 11:37:56 2010 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Mon, 6 Sep 2010 12:37:56 +0300 Subject: Dialyzer woes Message-ID: I'm trying to run dialyzer under both MacOS X and Ubuntu. Every time I run it I get the following error: dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() Is this curable? From kenneth.lundin@REDACTED Mon Sep 6 12:14:56 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Mon, 6 Sep 2010 12:14:56 +0200 Subject: [erlang-questions] Dialyzer woes In-Reply-To: References: Message-ID: What version of Erlang/OTP are you running? I think that this kind of problem is already corrected in R14A. Dialyzer is not supposed to fail with error if a remote type is not found. It should just report it as a warning. /Kenneth , Erlang/OTP Ericsson On Mon, Sep 6, 2010 at 11:37 AM, Dmitrii Dimandt wrote: > I'm trying to run dialyzer under both MacOS X and Ubuntu. Every time I run it I get the following error: > > dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() > > > Is this curable? > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From zvi.avraham@REDACTED Mon Sep 6 12:23:21 2010 From: zvi.avraham@REDACTED (Zvi) Date: Mon, 6 Sep 2010 03:23:21 -0700 (PDT) Subject: Is this code tail recursive? Message-ID: Is this code tail recursive? f() -> receive Pattern -> f() after 1000 timeout end. Thanks. Zvi From max.lapshin@REDACTED Mon Sep 6 12:31:18 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 6 Sep 2010 14:31:18 +0400 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: References: Message-ID: On Mon, Sep 6, 2010 at 2:23 PM, Zvi wrote: > Is this code tail recursive? > > f() -> > ? receive > ? ? ? ?Pattern -> ?f() > ? after 1000 > ? ? ? ?timeout > ? end. Yes, of course. Do not forget about difference between f() and ?MODULE:f() From watson.timothy@REDACTED Mon Sep 6 12:45:15 2010 From: watson.timothy@REDACTED (Tim Watson) Date: Mon, 6 Sep 2010 11:45:15 +0100 Subject: [erlang-questions] gen_server as Pid based callback In-Reply-To: References: Message-ID: > later callback. I want to explore how gen_server registered/supervised > process can be used for this purpose. > How to use registered/supervised gen_server in place of Pid. > -- Starting a process with gen_server:start returns {ok, Pid}, so you can always stash the Pid away for later use. If you want the Pid for a registered process you can call erlang:whereis/1 for a locally registered process and/or global:whereis_name/1 for a globally registered name. Not sure what you're asking about supervised processes. This stuff is very well documented - http://erlang.org/doc. Hope that helps, Tim From kostis@REDACTED Mon Sep 6 12:52:29 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 06 Sep 2010 13:52:29 +0300 Subject: [erlang-questions] Dialyzer woes In-Reply-To: References: Message-ID: <4C84C7ED.2080006@cs.ntua.gr> Dmitrii Dimandt wrote: > I'm trying to run dialyzer under both MacOS X and Ubuntu. Every time I run it I get the following error: > > dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() > > > Is this curable? It's a bit difficult to tell you exactly what to do as you do not specify which Erlang/OTP version you are using but the answer to your question is "Yes". Depending on your OTP version, you have two options: 1. Include the 'syntax_tools' application in your PLT. 2. Upgrade to a later Erlang/OTP version. Kostis From ulf.wiger@REDACTED Mon Sep 6 13:56:24 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 06 Sep 2010 13:56:24 +0200 Subject: strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <4C827ED3.7080004@erlang-solutions.com> References: <4C827ED3.7080004@erlang-solutions.com> Message-ID: <4C84D6E8.2080602@erlang-solutions.com> So when exercising my QuickCheck suite for the sext (sortable serialization) library, I stumbled upon an intermittent test case failure, which was particularly vexing. The failing case was the sorting property, which says that, for any terms, comparing the encoded terms should yield the same result as comparing the original terms. The failing term pairs are {-1.0, 1}, {1.0, 1}, {2.0, 2} etc. The terms are not strictly equal, so they have to be encoded differently (otherwise, the encoding property breaks). That is, decode(encode(2.0)) =:= 2.0 must hold true*. In this sense, the values in each pair are distinctly different. * For some definition of true. In the case of sext, I specify that the IEEE 754 binary64 representation of the float, as used in the bit syntax notation, must be the same. Yet, they have no defined ordering! Eshell V5.7.5 (abort with ^G) 1> 1 < 1.0. false 2> 1.0 < 1. false 3> 1.0 =:= 1. false This means that the result of lists:sort/1 is not defined in this particular case: Eshell V5.7.5 (abort with ^G) 1> lists:sort([1,1.0]). [1,1.0] 2> lists:sort([1.0,1]). [1.0,1] Naturally, lists:usort/1 has a corresponding behaviour: 3> lists:usort([2,2.0]). [2] 4> lists:usort([2.0,2]). [2.0] I think this is a good argument in favour of adding operators "strictly <" and "strictly >", which I suggest could be :< and >:. I made a simple hack where the parser recognized these tokens and expanded it into something similar to: ':<'(A,B) -> A == B andalso (is_integer(A) andalso is_float(B)) orelse A < B. '>:'(A,B) -> A == B andalso (is_float(A) andalso is_integer(B)) orelse A < B. but rather as a form rewriting excercise in erl_parse.yrl: wrap_op({op,Pos,':<',L,R}) -> {op,Pos,'orelse', {op,Pos,'andalso', {op,Pos,'==',L,R}, {op,Pos,'andalso', {call,13,{atom,Pos,is_integer},[L]}, {call,14,{atom,Pos,is_float},[R]}}}, {op,Pos,'<',L,R}}; wrap_op({op,Pos,'>:',L,R}) -> {op,Pos,'orelse', {op,Pos,'andalso', {op,Pos,'==',L,R}, {op,Pos,'andalso', {call,13,{atom,Pos,is_float},[L]}, {call,14,{atom,Pos,is_integer},[R]}}}, {op,Pos,'>',L,R}}; wrap_op(Op) -> Op. This way, it works in guards as well, and will be BW compatible with existing tools, but with the disadvantage that the original source cannot be reconstituted. I could try to formulate this as an EEP, if you all think it's a reasonable (although quite marginal) suggestion. An alternative would be to re-formulate the property of sext such that the encoded terms sort the same way as the original terms in all cases where it is reasonable to rely on the sort order in the first place. This is to some extent already true, at least in regard to encode/decode symmetry, in other cases involving floats. This alternative likely has less impact. :) BR, Ulf W On 09/04/2010 07:16 PM, Ulf Wiger wrote: > On 09/03/2010 10:28 PM, Evans, Matthew wrote: >> Hi, >> >> Does anyone know if there are any roadmap plans to implement ordered_set table types in DETS? >> >> Thanks > > It may be possible to build on the combination of Tokyo Tyrant > and my sortable serialization library, which I mentioned in > this post a while ago: > > http://www.erlang.org/cgi-bin/ezmlm-cgi/4/47285 > > That was part of the justification for adding prefix > matching and matchspec-style functionality in the sext > library in the first place. > > BR, > Ulf W > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From dmitrii@REDACTED Mon Sep 6 14:19:57 2010 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Mon, 6 Sep 2010 15:19:57 +0300 Subject: [erlang-questions] Dialyzer woes In-Reply-To: <4C84C7ED.2080006@cs.ntua.gr> References: <4C84C7ED.2080006@cs.ntua.gr> Message-ID: <2BB99040-59CD-4B14-806F-5057CC5C1617@gmail.com> Indeed, my bad. Had my mind elsewhere :) My Erlang/OTP version is: Erlang R13B04 (erts-5.7.5) So here's an account of my quest for dialyzer :) > dialyzer --build_plt -r . dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() > dialyzer --build_plt -r . /opt/local/lib/erlang/lib/syntax_tools-1.6.5/ dialyzer: Analysis failed with error: Cannot locate module file to resolve the remote type: file:filename() > dialyzer --build_plt -r . /opt/local/lib/erlang/lib/ dialyzer: Could not get abstract code for file: /opt/local/lib/erlang/lib/ssl-3.10.8/examples/certs/ebin/make_certs.beam (please recompile it with +debug_info) I could of course upgrade my Erlang instalation, but I guess that feels a bit wrong ? to upgrade it just because of dialyzer :) I guess, I'll try to get this working as it's now. Is there a way to recompile erlang libs with debug_info? > Dmitrii Dimandt wrote: >> I'm trying to run dialyzer under both MacOS X and Ubuntu. Every time I run it I get the following error: >> dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() >> Is this curable? > > It's a bit difficult to tell you exactly what to do as you do not specify which Erlang/OTP version you are using but the answer to your question is "Yes". Depending on your OTP version, you have two options: > > 1. Include the 'syntax_tools' application in your PLT. > 2. Upgrade to a later Erlang/OTP version. > > Kostis From kostis@REDACTED Mon Sep 6 14:32:33 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 06 Sep 2010 15:32:33 +0300 Subject: [erlang-questions] Dialyzer woes In-Reply-To: <2BB99040-59CD-4B14-806F-5057CC5C1617@gmail.com> References: <4C84C7ED.2080006@cs.ntua.gr> <2BB99040-59CD-4B14-806F-5057CC5C1617@gmail.com> Message-ID: <4C84DF61.2020603@cs.ntua.gr> Dmitrii Dimandt wrote: > Indeed, my bad. Had my mind elsewhere :) > > My Erlang/OTP version is: Erlang R13B04 (erts-5.7.5) > > So here's an account of my quest for dialyzer :) > >> dialyzer --build_plt -r . > dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() > >> dialyzer --build_plt -r . /opt/local/lib/erlang/lib/syntax_tools-1.6.5/ > dialyzer: Analysis failed with error: Cannot locate module file to resolve the remote type: file:filename() > >> dialyzer --build_plt -r . /opt/local/lib/erlang/lib/ > dialyzer: Could not get abstract code for file: /opt/local/lib/erlang/lib/ssl-3.10.8/examples/certs/ebin/make_certs.beam (please recompile it with +debug_info) I think you are making your life unnecessarily complicated. Use the following command which will most probably be sufficient: dialyzer --build_plt --apps erts kernel stdlib syntax_tools > I could of course upgrade my Erlang instalation, but I guess that feels a bit wrong ? to upgrade it just because of dialyzer :) Trust me: there are more reasons to upgrade. kostis From zeno490@REDACTED Mon Sep 6 17:03:39 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Mon, 6 Sep 2010 11:03:39 -0400 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <4C84D6E8.2080602@erlang-solutions.com> References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> Message-ID: Is that not the expected behavior per the erlang doc? http://www.erlang.org/doc/reference_manual/expressions.html Section 7.11 Term Comparisons (copied below the simplicity) Lists are compared element by element. Tuples are ordered by size, two tuples with the same size are compared element by element. If one of the compared terms is an integer and the other a float, the integer is first converted into a float, unless the operator is one of =:= and =/=. If the integer is too big to fit in a float no conversion is done, but the order is determined by inspecting the sign of the numbers. Returns the Boolean value of the expression, true or false. afaik, 1==1.0 -> true and 1=:=1.0 -> false makes sense to me. This means that of course, you can't trust =:= for ordering elements. Conceptually speaking, comparing floating points and integers, one can only infer an ordering if one converts one into the other since ordering is only defined within each's realm. Incidentaly, what would you expect of the following: 1 < {foo, 2} 1 > {foo, 2} 1 == {foo, 2} 1 =:= {foo, 2} IMO a better way to define an ordering in a case like these, when sorting, would be to pass a function that defines the ordering based on input values. I'm not sure extending the language makes sense here. 2cents Nicholas On Mon, Sep 6, 2010 at 7:56 AM, Ulf Wiger wrote: > > So when exercising my QuickCheck suite for the sext > (sortable serialization) library, I stumbled upon an intermittent > test case failure, which was particularly vexing. > > The failing case was the sorting property, which says that, > for any terms, comparing the encoded terms should yield > the same result as comparing the original terms. > > The failing term pairs are {-1.0, 1}, {1.0, 1}, {2.0, 2} etc. > The terms are not strictly equal, so they have to be encoded > differently (otherwise, the encoding property breaks). > That is, decode(encode(2.0)) =:= 2.0 must hold true*. > In this sense, the values in each pair are distinctly different. > > * For some definition of true. In the case of sext, I specify that > the IEEE 754 binary64 representation of the float, as used in > the bit syntax notation, must be the same. > > Yet, they have no defined ordering! > > Eshell V5.7.5 (abort with ^G) > 1> 1 < 1.0. > false > 2> 1.0 < 1. > false > 3> 1.0 =:= 1. > false > > This means that the result of lists:sort/1 is not defined in this > particular case: > > Eshell V5.7.5 (abort with ^G) > 1> lists:sort([1,1.0]). > [1,1.0] > 2> lists:sort([1.0,1]). > [1.0,1] > > Naturally, lists:usort/1 has a corresponding behaviour: > > 3> lists:usort([2,2.0]). > [2] > 4> lists:usort([2.0,2]). > [2.0] > > > I think this is a good argument in favour of adding operators > "strictly <" and "strictly >", which I suggest could be :< and >:. > > I made a simple hack where the parser recognized these tokens and > expanded it into something similar to: > > ':<'(A,B) -> > A == B > andalso (is_integer(A) > andalso is_float(B)) > orelse A < B. > > '>:'(A,B) -> > A == B > andalso (is_float(A) > andalso is_integer(B)) > orelse A < B. > > but rather as a form rewriting excercise in erl_parse.yrl: > > wrap_op({op,Pos,':<',L,R}) -> > {op,Pos,'orelse', > {op,Pos,'andalso', > {op,Pos,'==',L,R}, > {op,Pos,'andalso', > {call,13,{atom,Pos,is_integer},[L]}, > {call,14,{atom,Pos,is_float},[R]}}}, > {op,Pos,'<',L,R}}; > wrap_op({op,Pos,'>:',L,R}) -> > {op,Pos,'orelse', > {op,Pos,'andalso', > {op,Pos,'==',L,R}, > {op,Pos,'andalso', > {call,13,{atom,Pos,is_float},[L]}, > {call,14,{atom,Pos,is_integer},[R]}}}, > {op,Pos,'>',L,R}}; > wrap_op(Op) -> > Op. > > > This way, it works in guards as well, and will be BW > compatible with existing tools, but with the disadvantage > that the original source cannot be reconstituted. > > I could try to formulate this as an EEP, if you all think > it's a reasonable (although quite marginal) suggestion. > > An alternative would be to re-formulate the property of sext > such that the encoded terms sort the same way as the original > terms in all cases where it is reasonable to rely on the sort > order in the first place. This is to some extent already true, > at least in regard to encode/decode symmetry, in other cases > involving floats. This alternative likely has less impact. :) > > BR, > Ulf W > > On 09/04/2010 07:16 PM, Ulf Wiger wrote: > > On 09/03/2010 10:28 PM, Evans, Matthew wrote: > >> Hi, > >> > >> Does anyone know if there are any roadmap plans to implement ordered_set > table types in DETS? > >> > >> Thanks > > > > It may be possible to build on the combination of Tokyo Tyrant > > and my sortable serialization library, which I mentioned in > > this post a while ago: > > > > http://www.erlang.org/cgi-bin/ezmlm-cgi/4/47285 > > > > That was part of the justification for adding prefix > > matching and matchspec-style functionality in the sext > > library in the first place. > > > > BR, > > Ulf W > > > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ulf.wiger@REDACTED Mon Sep 6 17:30:02 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 06 Sep 2010 17:30:02 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> Message-ID: <4C8508FA.6040208@erlang-solutions.com> On 09/06/2010 05:03 PM, Nicholas Frechette wrote: > Is that not the expected behavior per the erlang doc? > http://www.erlang.org/doc/reference_manual/expressions.html Well, yes, but that in itself is no guarantee that it is a good solution. ;-) > Incidentaly, what would you expect of the following: > 1 < {foo, 2} > 1 > {foo, 2} > 1 == {foo, 2} > 1 =:= {foo, 2} There is nothing strange or unambiguous there. Erlang has a global term comparison order so e.g. 1 < {foo,2} is perfectly well defined. The int->float coercion is fine... until it's not. That's why we have =:= and =/=, because in some cases, 1 and 1.0 really are distinctly different. I have managed to get by programming erlang for the better part of 17 years before I was finaly bitten by this, so I'm perfectly willing to go on happily without ever seeing this corrected. However, I really don't see the disadvantage of ensuring that the global comparison order can be guaranteed across the board, without any surprising corner cases, and I think lists:sort/1 et al should really be able to provide such guarantees. BTW, there was another interesting edge case some years ago, that proved very troublesome for mnesia. Pids did not sort the same way on all connected nodes in a distributed erlang network. This broke some fundamental assumptions in mnesia's deadlock detection algorithm. It has long-since been fixed, but having a truly universal sort order really is a very good thing. BR, Ulf W From zabrane3@REDACTED Tue Sep 7 00:03:06 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 7 Sep 2010 00:03:06 +0200 Subject: Mnesia warning: dets: file "/path/foo.DAT" not properly closed, repairing ... Message-ID: Hi guys, hi Ulf, Time to time, I got the following Mnesia message: Mnesia warning: dets: file "/path/foo.DAT" not properly closed, repairing ... But I'm 100% sure that I've correctly closed it before. Any idea? -- Regards Zabrane From zeno490@REDACTED Tue Sep 7 01:04:14 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Mon, 6 Sep 2010 19:04:14 -0400 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <4C8508FA.6040208@erlang-solutions.com> References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> Message-ID: I see what you mean by 'number' < 'tuple' defining an order. Essentially, there is a category missing to (on that same page/section): The arguments may be of different data types. The following order is defined: number < atom < reference < fun < port < pid < tuple < list < bit string 'number' could be split into two: integer and floating point. Something like: integer < floating point < atom < reference < etc. Such that: 1 < 1.0 -> true instead of false? 1 > 1.0 -> false as currently This would allow lists:sort to sort but then, the behavior would be one that isn't intuitive. (ie: 1 < 0.5 -> true) The other way around this is as you suggested to add matching strict operators (:>, :<, :=>, :<=) but then that still won't fix lists:sort (without a code change to it obviously). And when one is doing sorting, one will have to stick to one set of operators or the other, else the order isn't potentially defined. Though this seems to apply to numbers only as the order is explicitly defined for any other pair of terms. Your operators would thus only be required for comparing numbers. Perhaps it would make more sense to have =:= sometime coerce so that == and =:= are equivalent for numbers only? Since there isn't any defined ordering for them without that operator coercing (if you mix <, > and =:=). Incidentaly, matching would then work with numbers: case 1.0 of 1 -> currently_doesnt_match; _ -> currently_always_matches end. (Side note: Does the compiler optimize the above and strip it since it's a constant branch?) Looking elsewhere, I believe in lisp the following applies: (< 1 1.0) -> false (< 1 2.0) -> true (= 1 1.0) -> true (eq 3 3) might be true or false, depending on the implementation. (eq 3 3.0) is false. (eq 3.0 3.0) might be true or false, depending on the implementation. (eql 3 3) is true. (eql 3 3.0) is false. (eql 3.0 3.0) is true. (equal 3 3) is true. (equal 3 3.0) is false. (equal 3.0 3.0) is true. (equalp 3 3) is true. (equalp 3 3.0) is true. (equalp 3.0 3.0) is true. This is the case because in lisp +-=<>*/ etc are functions that coerce. Given a list of terms, a single function would be evaluated on them to sort them, a function you'd explicitly define (or defined in the library). This could lead to weird results if you mix types: (< 1 "foo") -> true (no order defined for these, unlike in erlang) (< "foo" 1) -> true (= 1 "foo") -> false Sadly, my lisp is a bit rusty and I'm not sure I can adequately explain the above behavior. Nonetheless, when sorting or otherwise extracting an ordering between terms, in lisp one will most likely end up using the comparison operators and not the equality predicates (eq, eql, equal, equalp) thus avoiding the issue you ran into (at least when not mixing types, I presume when one is mixing types that can't coerce, one has to provide a function in order to get strict ordering). Maybe someone with more lisp experience than I have can elaborate further here but that is my understanding. Thus imo, when ordering things in erlang, one should use the comparison operators that coerce and use the exact match operators when one is attempting to match. Otherwise, you are setting yourself up for a world of pain. lists:sort is probably a "stable sort" in that if elements are equal, they appear in the sorted list in the order they appeared in the unsorted one which explains the behavior you see: lists:sort([2, 1, 1.0]) -> [1, 1.0, 2] lists:sort([2, 1.0, 1]) -> [1.0, 1, 2] lists:sort([1, 2, 3, 2.0]) -> [1, 2, 2.0, 3] lists:sort([1, 2.0, 3, 2]) -> [1, 2.0, 2, 3] Sorry for the lenghty email. Nicholas On Mon, Sep 6, 2010 at 11:30 AM, Ulf Wiger wrote: > On 09/06/2010 05:03 PM, Nicholas Frechette wrote: > > Is that not the expected behavior per the erlang doc? > > http://www.erlang.org/doc/reference_manual/expressions.html > > Well, yes, but that in itself is no guarantee that it is a > good solution. ;-) > > > > Incidentaly, what would you expect of the following: > > 1 < {foo, 2} > > 1 > {foo, 2} > > 1 == {foo, 2} > > 1 =:= {foo, 2} > > There is nothing strange or unambiguous there. > Erlang has a global term comparison order so e.g. 1 < {foo,2} is > perfectly well defined. > > The int->float coercion is fine... until it's not. That's why > we have =:= and =/=, because in some cases, 1 and 1.0 really > are distinctly different. > > I have managed to get by programming erlang for the better part of > 17 years before I was finaly bitten by this, so I'm perfectly willing > to go on happily without ever seeing this corrected. > > However, I really don't see the disadvantage of ensuring that the > global comparison order can be guaranteed across the board, without > any surprising corner cases, and I think lists:sort/1 et al should > really be able to provide such guarantees. > > BTW, there was another interesting edge case some years ago, that > proved very troublesome for mnesia. Pids did not sort the same > way on all connected nodes in a distributed erlang network. This broke > some fundamental assumptions in mnesia's deadlock detection algorithm. > It has long-since been fixed, but having a truly universal sort order > really is a very good thing. > > BR, > Ulf W > From john.hughes@REDACTED Tue Sep 7 01:37:16 2010 From: john.hughes@REDACTED (John Hughes) Date: Tue, 7 Sep 2010 01:37:16 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <4C84D6E8.2080602@erlang-solutions.com> References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> Message-ID: I think this is a good idea... it's handy to have a real total ordering on values, which this would give. Then sort should be modified to use <: instead of <, which is very unlikely to cause any problems. But note there would need to be TWO versions of usort, one with (approximately) today's behaviour, and one giving uniqueness up to matching. musort, maybe? John ----- Original Message ----- From: "Ulf Wiger" > > So when exercising my QuickCheck suite for the sext > (sortable serialization) library, I stumbled upon an intermittent > test case failure, which was particularly vexing. > > The failing case was the sorting property, which says that, > for any terms, comparing the encoded terms should yield > the same result as comparing the original terms. > > The failing term pairs are {-1.0, 1}, {1.0, 1}, {2.0, 2} etc. > The terms are not strictly equal, so they have to be encoded > differently (otherwise, the encoding property breaks). > That is, decode(encode(2.0)) =:= 2.0 must hold true*. > In this sense, the values in each pair are distinctly different. > > * For some definition of true. In the case of sext, I specify that > the IEEE 754 binary64 representation of the float, as used in > the bit syntax notation, must be the same. > > Yet, they have no defined ordering! > > Eshell V5.7.5 (abort with ^G) > 1> 1 < 1.0. > false > 2> 1.0 < 1. > false > 3> 1.0 =:= 1. > false > > This means that the result of lists:sort/1 is not defined in this > particular case: > > Eshell V5.7.5 (abort with ^G) > 1> lists:sort([1,1.0]). > [1,1.0] > 2> lists:sort([1.0,1]). > [1.0,1] > > Naturally, lists:usort/1 has a corresponding behaviour: > > 3> lists:usort([2,2.0]). > [2] > 4> lists:usort([2.0,2]). > [2.0] > > > I think this is a good argument in favour of adding operators > "strictly <" and "strictly >", which I suggest could be :< and >:. > From rvirding@REDACTED Tue Sep 7 02:20:10 2010 From: rvirding@REDACTED (Robert Virding) Date: Tue, 7 Sep 2010 02:20:10 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <4C84D6E8.2080602@erlang-solutions.com> References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> Message-ID: The original source of this problem is that we got the comparison operators wrong. With 20/20 hindsight I think we should have had >, >=, ... as purely arithmetic comparisons, with or without conversion, and had a separate set of operators @>, @>=, ... as pure term comparison with no conversion. While we can't change the existing operators we can add the pure term comparison operators. I would prefer to have a complete set with consistent names even if this means duplicating =:= and =/=. There is one problem with your definitions of :< and >:, which is that they don't work if the arguments are structures with embedded numbers as you only check at the top-level. Defining a comparison order should be done. Robert On 6 September 2010 13:56, Ulf Wiger wrote: > > So when exercising my QuickCheck suite for the sext > (sortable serialization) library, I stumbled upon an intermittent > test case failure, which was particularly vexing. > > The failing case was the sorting property, which says that, > for any terms, comparing the encoded terms should yield > the same result as comparing the original terms. > > The failing term pairs are {-1.0, 1}, {1.0, 1}, {2.0, 2} etc. > The terms are not strictly equal, so they have to be encoded > differently (otherwise, the encoding property breaks). > That is, decode(encode(2.0)) =:= 2.0 must hold true*. > In this sense, the values in each pair are distinctly different. > > * For some definition of true. In the case of sext, I specify that > the IEEE 754 binary64 representation of the float, as used in > the bit syntax notation, must be the same. > > Yet, they have no defined ordering! > > Eshell V5.7.5 ?(abort with ^G) > 1> 1 < 1.0. > false > 2> 1.0 < 1. > false > 3> 1.0 =:= 1. > false > > This means that the result of lists:sort/1 is not defined in this > particular case: > > Eshell V5.7.5 ?(abort with ^G) > 1> lists:sort([1,1.0]). > [1,1.0] > 2> lists:sort([1.0,1]). > [1.0,1] > > Naturally, lists:usort/1 has a corresponding behaviour: > > 3> lists:usort([2,2.0]). > [2] > 4> lists:usort([2.0,2]). > [2.0] > > > I think this is a good argument in favour of adding operators > "strictly <" and "strictly >", which I suggest could be :< and >:. > > I made a simple hack where the parser recognized these tokens and > expanded it into something similar to: > > ':<'(A,B) -> > ? ?A == B > ? ? ? ?andalso (is_integer(A) > ? ? ? ? ? ? ? ? andalso is_float(B)) > ? ? ? ?orelse A < B. > > '>:'(A,B) -> > ? ?A == B > ? ? ? ?andalso (is_float(A) > ? ? ? ? ? ? ? ? andalso is_integer(B)) > ? ? ? ?orelse A < B. > > but rather as a form rewriting excercise in erl_parse.yrl: > > wrap_op({op,Pos,':<',L,R}) -> > ? ?{op,Pos,'orelse', > ? ? {op,Pos,'andalso', > ? ? ? ? {op,Pos,'==',L,R}, > ? ? ? ? {op,Pos,'andalso', > ? ? ? ? ? ? {call,13,{atom,Pos,is_integer},[L]}, > ? ? ? ? ? ? {call,14,{atom,Pos,is_float},[R]}}}, > ? ? {op,Pos,'<',L,R}}; > wrap_op({op,Pos,'>:',L,R}) -> > ? ?{op,Pos,'orelse', > ? ? {op,Pos,'andalso', > ? ? ? ? {op,Pos,'==',L,R}, > ? ? ? ? {op,Pos,'andalso', > ? ? ? ? ? ? {call,13,{atom,Pos,is_float},[L]}, > ? ? ? ? ? ? {call,14,{atom,Pos,is_integer},[R]}}}, > ? ? {op,Pos,'>',L,R}}; > wrap_op(Op) -> > ? ?Op. > > > This way, it works in guards as well, and will be BW > compatible with existing tools, but with the disadvantage > that the original source cannot be reconstituted. > > I could try to formulate this as an EEP, if you all think > it's a reasonable (although quite marginal) suggestion. > > An alternative would be to re-formulate the property of sext > such that the encoded terms sort the same way as the original > terms in all cases where it is reasonable to rely on the sort > order in the first place. This is to some extent already true, > at least in regard to encode/decode symmetry, in other cases > involving floats. This alternative likely has less impact. :) > > BR, > Ulf W > > On 09/04/2010 07:16 PM, Ulf Wiger wrote: >> On 09/03/2010 10:28 PM, Evans, Matthew wrote: >>> Hi, >>> >>> Does anyone know if there are any roadmap plans to implement ordered_set table types in DETS? >>> >>> Thanks >> >> It may be possible to build on the combination of Tokyo Tyrant >> and my sortable serialization library, which I mentioned in >> this post a while ago: >> >> http://www.erlang.org/cgi-bin/ezmlm-cgi/4/47285 >> >> That was part of the justification for adding prefix >> matching and matchspec-style functionality in the sext >> library in the first place. >> >> BR, >> Ulf W >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From rzezeski@REDACTED Tue Sep 7 05:31:26 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Mon, 6 Sep 2010 23:31:26 -0400 Subject: [erlang-questions] Using ETS for large amounts of data? In-Reply-To: References: <50021.64.81.32.110.1282977701.squirrel@duomark.com> <20100829222700.GF23023@alumni.caltech.edu> Message-ID: On Mon, Aug 30, 2010 at 5:56 AM, Hynek Vychodil wrote: > I have very similar experience. > > So keeping it in one big binary and store only pointers will save you > 300 - 400 MB of data depending of length of item (16-26B). Anyway > using better tuned k/v storage would be better. > > Did you check erlang:memory/0 by any chance? If I run your example in the shell directly vs. it it's own process I get drastically different results, memory wise. It seems when running your example in the shell that ERTS holds onto all the binaries. 36> ets:info(T). [{memory,161554577}, {owner,<0.31.0>}, {heir,none}, {name,foo2}, {size,10000000}, {node,nonode@REDACTED}, {named_table,false}, {type,set}, {keypos,1}, {protection,private}] 37> erlang:memory(). [{total,2108524432}, {processes,1150232}, {processes_used,1135752}, {system,2107374200}, {atom,619969}, {atom_used,593262}, {binary,646419816}, {code,5920745}, {ets,1452847016}] Note that I'm running a 64-bit VM. Here is the same information when running in a separate process spawned in the shell. 21> Pid ! {from,get}. Info: [{memory,141554577}, {owner,<0.70.0>}, {heir,none}, {name,foo}, {size,10000000}, {node,nonode@REDACTED}, {named_table,false}, {type,set}, {keypos,1}, {protection,protected}] {from,get} 22> erlang:memory(). [{total,1302212544}, {processes,1245760}, {processes_used,1232080}, {system,1300966784}, {atom,619969}, {atom_used,593175}, {binary,22272}, {code,5920745}, {ets,1292837352}] Notice that the binary memory varies greatly between the two methods. I recently ran a bunch of tests on binaries to try to understand their behavior better. I'm also using them to handle large (500M+) CSV files. I noticed that if I split the CSV file into a list of lists (i.e. a list of all the column values) using the binary:split function it consumed a _lot_ of memory. I thought that this would be efficient because after reading the docs I was under the impression that the sub binaries would simply reference the bigger binary off-heap but the behavior I noticed seemed to indicate that depending on the size of the binary resulting from split it might reside on the heap. If that's actually what was happening, I'm not sure. I ended up re-writing my functions to transform the CSV a line at a time and build up a new binary in an accumulator. This performed much better and uses much less memory. Anyways, this is slightly tangential to the problem being discussed so maybe I'll just post a new thread. -Ryan From kenji.rikitake@REDACTED Tue Sep 7 06:41:57 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 7 Sep 2010 13:41:57 +0900 Subject: [erlang-questions] IPV6_V6ONLY In-Reply-To: <20100824075631.GA854@erix.ericsson.se> References: <201008231711.o7NHBgvY074811@pluto.hedeland.org> <20100824075631.GA854@erix.ericsson.se> Message-ID: <20100907044157.GA20933@k2r.org> In the message <20100824075631.GA854@REDACTED> dated Tue, Aug 24, 2010 at 09:56:07AM +0200, Raimo Niskanen writes: > It seems the situation has changed since Stevens' Unix Network Programming > - The Sockets Networking API, third edition. There it seems one of the strong > points of a IPv6 socket is that it accepts IPv4 connections _unless_ the > IPV6_V6ONLY socket option is set. > > Where can these network community discussions be followed where > can one find a motivation to why FreeBSD has diverged from Stevens' book > (if that is the case)... See RFC3493 Section 5.3, and http://www.ops.ietf.org/lists/v6ops/v6ops.2003/msg01140.html My opinion is that just because Linux does it is insufficient for making it as the default for the IPV6_V6ONLY behavior. And even on FreeBSD you can make a choice by sysctl param of "net.inet6.ip6.v6only" (see inet6(4)). I'd rather want to leave the choice to the operating system. Kenji Rikitake From kenji.rikitake@REDACTED Tue Sep 7 06:57:56 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 7 Sep 2010 13:57:56 +0900 Subject: [erlang-questions] ssh client In-Reply-To: References: Message-ID: <20100907045756.GB20933@k2r.org> http://github.com/jj1bdx/sshrpc/ Far from a complete one - but might be useful. Kenji Rikitake In the message dated Wed, Sep 01, 2010 at 11:17:21AM -0500, Daniel Goertzen writes: > I want to write a program that will chat with another remote program via > SSH. Erlang SSH is a bit complex, and I was unable to find an example of > what I wanted, so I thought I would confirm my approach here before barking > up the wrong tree. Here is what I was going to do: > > > 1. Open a connection with ssh:connect() > 2. Open a channel with ssh_connection:session_channel() > 3. Call ssh_channel:start_link() to create my handler process. > 4. In my callback module, init() will call ssh_connection:exec() to invoke > the remote program. > 5. Incoming data gets sent to my handle_ssh_message() callback. I send data > with ssh_connection:send() > > Do I have it right? > > It would be nice if there was a "port" abstraction for an ssh channel, so I > could treat locally and remotely invoked programs identically... I guess it > wouldn't be hard to write my own wrapper for that. > > Thanks, > Dan. From jay@REDACTED Tue Sep 7 06:59:00 2010 From: jay@REDACTED (jay@REDACTED) Date: Mon, 6 Sep 2010 21:59:00 -0700 (PDT) Subject: [erlang-questions] Using ETS for large amounts of data? Message-ID: <57029.206.111.154.162.1283835540.squirrel@duomark.com> I wrote: > > If you use and ETS table, you have to use integer index offsets as the > return value or the binary data will be copied into the ETS on insert and > out again on every lookup match. And Ryan quickly proved me wrong: > I think the documentation needs to be updated because I don't witness this > behavior in R14A. For example, I can read a 436M CSV file, store it in ETS > and not see my memory usage go up. Furthermore, I can then use lookup and > match 400MB of the binary and also not see the memory go up. The ETS table is kept in a separate memory space to avoid the garbage collection overhead. Whenever an entry is accessed it is copied into the erlang process space. In the case of binaries, they can live in the binary heap which is shared for a node and only the erlang reference structure copied out. If the binary is small enough (less than the overhead of a reference structure, I think 32 bytes on a 32-bit system) it is cheaper to transfer the binary than the structure reference. That is the case when sending binaries from one process to another, I am not sure with the ETS table since it is a special optimized implementation. To see the memory usage of ETS copies, you would need to do a lot of lookups. Lists, tuples and other structures will be copied on every lookup, but binaries apparently follow the same pattern as message passing of binaries. If you created a process which was used as a replacement for ETS containing a tree, array or tuple, lookups would end up not requiring extra memory within the process but would incur a copy when sent as a message to the other process requesting the lookup (and for binaries, the behavior would be roughly the same as with ETS tables). To accrue the memory savings, the lookup data structure would have to be in the same process and the data returned from a lookup has to be used as is. ------- Anthony described his solution using integer offsets, an ETS table and binary pattern matching to extract a sub-binary from the big binary... > I didn't try to find out if just storing the 26 bytes as > the values in the ets table was more efficient memory wise, but what > I have now seems to work well. This approach is close enough to optimal. Pattern matching binaries used to be slower, but now is quite efficient. 26 byte binaries should get passed to and from ETS / processes essentially as immediate data since only binaries bigger than 32 bytes are stored on the binary heap. If it meets your performance requirements, you are done until you change the requirements. -------- Ryan makes some observations about binary memory usage: > If I run your example in the > shell directly vs. it it's own process I get drastically different > results, memory wise. It seems when running your example in the > shell that ERTS holds onto all the binaries. The shell has a feature to access the results of previous commands (the N previous results, I forget how many). Try the following: 32> X = 5 * 5. 25 33> v(32). 25 The function shell:v/1 returns the result value of the expression for the corresponding prompt number. Therefore the binaries cannot be discarded until they roll off the history stack. Ryan also noted excess memory usage with binary:split: I believe binary pattern matching is now preferred over split. If you do create sub-binaries but do not want all of them, remember that none of the memory occupied by the underlying large binary can be reclaimed as long as the sub-binaries reference it. If you filter the sub-binary list, you should make a new binary copy for each of the retained sub-binaries to allow the old large binary to be recycled. jay From kenji.rikitake@REDACTED Tue Sep 7 07:13:10 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 7 Sep 2010 14:13:10 +0900 Subject: [erlang-questions] Random behaviour In-Reply-To: References: <6FB7DAFE1ABF46789DA9D4A262F70D2D@HALL> Message-ID: <20100907051310.GC20933@k2r.org> And you can try another PRNG module of SIMD-oriented Fast Mersenne Twister at http://github.com/jj1bdx/sfmt/ (I'm the code maintainer so give it a try and let me know how it works) Note: you still need an individual seeding per process as described below. My preference is to seed the generator frequently with crypto:rand_bytes/1. Kenji Rikitake In the message dated Mon, Sep 06, 2010 at 01:39:26PM +0800, Bob Ippolito writes: > Date: Mon, 6 Sep 2010 13:39:50 +0800 > From: Bob Ippolito > Subject: Re: [erlang-questions] Random behaviour > To: John Hughes > Cc: tom kelly , erlang-questions@REDACTED > > You might also want to look at the crypto module for generating random > numbers instead of the random module. The random module uses an > ancient algorithm and has issues with seeding (as you have noticed). > > On Mon, Sep 6, 2010 at 1:26 PM, John Hughes wrote: > > Hi Tom, > > > > Every Erlang process has its own random number seed, and it's initialised to a CONSTANT value when the process starts! Hence given > > > > foo() -> > > ?? ??io:format("~p\n",[[random:uniform(10) || _ <- lists:seq(1,10)]]). > > > > then you get different values from multiple calls in the SAME process: > > > > 2> foo:foo(). > > [1,5,8,10,6,4,6,10,7,5] > > ok > > 3> foo:foo(). > > [6,2,3,7,2,6,3,5,5,1] > > ok > > 4> foo:foo(). > > [6,5,5,4,1,6,10,4,2,3] > > ok > > > > but the same value every time in a NEW process: > > > > 5> spawn(foo,foo,[]). > > [1,5,8,10,6,4,6,10,7,5] > > <0.45.0> > > 6> spawn(foo,foo,[]). > > [1,5,8,10,6,4,6,10,7,5] > > <0.47.0> > > 7> spawn(foo,foo,[]). > > [1,5,8,10,6,4,6,10,7,5] > > <0.49.0> > > > > I guess it makes sense--you don't want to pay the price of initialising a random number seed every time you start a process--but it is a bit of a gotcha. > > > > If you want different random numbers in different processes (which you usually do), then you need to initialise the seed differently when you start the process. The usual trick is to use now(), since a seed just happens to be three integers: > > > > baz() -> > > ?? ??{A,B,C} = now(), > > ?? ??random:seed(A,B,C), > > ?? ??foo(). > > > > and now > > > > 10> spawn(foo,baz,[]). > > [10,7,9,4,7,1,4,8,3,6] > > <0.61.0> > > 11> spawn(foo,baz,[]). > > [10,5,1,6,5,5,9,3,6,2] > > <0.63.0> > > 12> spawn(foo,baz,[]). > > [8,7,9,7,9,6,1,9,4,3] > > <0.65.0> > > > > One thing to be aware of is that calls to now() close together in time give you seeds the initially generate similar numbers: > > > > 13> spawn(foo,baz,[]),spawn(foo,baz,[]). > > [10,1,8,1,8,1,8,10,1,9] > > [10,1,7,1,6,5,7,8,8,2] > > <0.68.0> > > 14> spawn(foo,baz,[]),spawn(foo,baz,[]). > > [5,9,8,2,2,3,3,9,5,8] > > [5,9,7,2,9,6,2,7,2,1] > > <0.71.0> > > 15> spawn(foo,baz,[]),spawn(foo,baz,[]). > > [10,5,2,3,4,4,3,1,6,3] > > [10,5,1,4,2,8,3,9,3,7] > > <0.74.0> > > > > Note the first two numbers generated are always the same in both processes, even though they start with different seeds. If you're going to start many processes and you want to be sure they have independent random numbers then you have to be a bit more careful how you initialise the seed--but if there are going to be seconds or more between each process start then you don't have to worry. > > > > John > > ??----- Original Message ----- > > ??From: tom kelly > > ??To: erlang-questions@REDACTED > > ??Sent: Sunday, September 05, 2010 8:45 PM > > ??Subject: [erlang-questions] Random behaviour > > > > > > ??Hello List, > > > > > > ??I just found a bahaviour I didn't expect in OTPs random module, it's not necessarily a bug but I thought I'd share it here all the same. > > > > > > ??I wrote a simple random data generator using random:uniform() to help me track a bug that was triggered by certain input data that I couldn't catch. The minimal example is attached in demo.erl. > > > > > > ??If you run the test function it will probably pass a few times but will eventually hit my "bug" and fail. Whenever the bug is triggered the test function gets a badmatch and the current process dies. The behaviour I wasn't expecting now occurs, everytime I repeat the test function I get the exact same sequence of numbers from random:uniform. > > > > > > ??I can call random:uniform from the shell to move it on one step in the sequence and probably pass another test, but when it fails again my process dies and again I repeatedly get a repeated sequence of numbers. The sequence from this second fail is different from the first fail so it hasn't just gone back to the start. > > > > > > ??I don't expect to be saved from my own bugs but just thought this was a strange behaviour from random. If anyone is using random in their application they should be aware of this. > > > > > > ??//Tom. > > > > > > > > > > ??8> demo:test_my_buggy_code(). > > ??ok - [15,21,70,16,56,22,46,43,1,57] > > ??ok > > ??9> demo:test_my_buggy_code(). > > ??ok - [41,31,6,58,99,34,19,21,4,89] > > ??ok > > ??10> demo:test_my_buggy_code(). > > ??ok - [83,33,26,81,2,3,5,6,99,57] > > ??ok > > ??11> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 72,41,80] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??12> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 72,41,80] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??13> random:uniform(1). > > ??1 > > ??14> demo:test_my_buggy_code(). > > ??ok - [67,88,90,80,59,30,72,41,80,8] > > ??ok > > ??15> demo:test_my_buggy_code(). > > ??ok - [60,68,65,8,27,61,1,94,82,65] > > ??ok > > ??16> demo:test_my_buggy_code(). > > ??ok - [47,2,62,94,25,97,94,87,27,32] > > ??ok > > ??17> demo:test_my_buggy_code(). > > ??ok - [59,21,10,14,17,24,93,78,34,28] > > ??ok > > ??18> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 60,42,2] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??19> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 60,42,2] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??20> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 60,42,2] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??21> > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > ??________________________________________________________________ > > ??erlang-questions (at) erlang.org mailing list. > > ??See http://www.erlang.org/faq.html > > ??To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From ulf.wiger@REDACTED Tue Sep 7 07:24:10 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Tue, 07 Sep 2010 07:24:10 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> Message-ID: <4C85CC7A.5000508@erlang-solutions.com> On 09/07/2010 02:20 AM, Robert Virding wrote: > > There is one problem with your definitions of :< and >:, which is that > they don't work if the arguments are structures with embedded numbers > as you only check at the top-level. Yes, of course you're right! Let's say it's just a quick-and-dirty to illustrate the concept (which it certainly was, too). A proper implementation would have to be done inside the emulator. BR, Ulf W From ok@REDACTED Tue Sep 7 08:27:35 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 7 Sep 2010 18:27:35 +1200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> Message-ID: On Sep 7, 2010, at 11:04 AM, Nicholas Frechette wrote: > Incidentaly, matching would then work with numbers: > case 1.0 of > 1 -> currently_doesnt_match; > _ -> currently_always_matches > end. Trust me, you *really* don't want to go there. To start with, while -0.0 and +0.0 compare equal under IEEE rules, they also BEHAVE DIFFERENTLY. But both of them are numerically equal to the integer 0. I've seen what happened in VM/PROLOG, where they had a "feature" of allowing fuzzy matching in clause heads. It made trivial cases look kewl, but it made reasoning about programs incredibly hard, because X = Y and Y = Z did not imply X = Z. Heck, even the order in which you wrote the arguments could matter. Floats have the property that there is an number X such that X = X+1. Integers do not. Let's *never* confuse them. (One of the things I just *love* about Ada and Haskell is that they do not allow 'mixed mode arithmetic'.) > > Thus imo, when ordering things in erlang, one should use the comparison > operators that coerce Oh no, no, no. Please NO! That way lies failure of transitivity, which is a DISASTER for sorting. More precisely, there is a way to do comparison-by-coercion that works, and there's one that doesn't, and the one language designers tend to pick is the one that doesn't. Suppose there are two kinds of numbers: Q and F. Q numbers are exact (integers, rationals). F numbers are inexact (floats). If you compare Q1 <= Q2, no problem. If you compare F1 <= F2, no problem. What happens if you compare Q1 <= F2 or F1 <= Q2? The popular technique is what Fortran, Pascal, and C do: convert the Q number to F format, and do F comparison: Q1 <= F2 iff uglify(Q1) <= F2 F1 <= Q2 iff F1 <= uglify(Q2). The problem with that is that it is possible to find numbers X Y Z such that X <= Y Y <= Z *and* X > Z Like I said, disastrous for sorting. The right way to do it is Q1 <= F2 iff Q1 <= make_exact(F2) F1 <= Q2 iff make_exact(F1) <= Q2 This is not particularly easy, and it requires some care. Arguably we want (1 bsl 8000) < 1.0/0.0 so that -infinity < all finite numbers < +infinity From dmitrii@REDACTED Tue Sep 7 08:39:55 2010 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Tue, 7 Sep 2010 09:39:55 +0300 Subject: [erlang-questions] Dialyzer woes In-Reply-To: <4C84DF61.2020603@cs.ntua.gr> References: <4C84C7ED.2080006@cs.ntua.gr> <2BB99040-59CD-4B14-806F-5057CC5C1617@gmail.com> <4C84DF61.2020603@cs.ntua.gr> Message-ID: <0C14E407-0A72-4AF4-8AAC-FA37F01B8748@gmail.com> > Dmitrii Dimandt wrote: >> Indeed, my bad. Had my mind elsewhere :) >> My Erlang/OTP version is: Erlang R13B04 (erts-5.7.5) >> So here's an account of my quest for dialyzer :) >>> dialyzer --build_plt -r . >> dialyzer: Analysis failed with error: Cannot locate module erl_syntax to resolve the remote type: erl_syntax:syntaxTree() >>> dialyzer --build_plt -r . /opt/local/lib/erlang/lib/syntax_tools-1.6.5/ >> dialyzer: Analysis failed with error: Cannot locate module file to resolve the remote type: file:filename() >>> dialyzer --build_plt -r . /opt/local/lib/erlang/lib/ >> dialyzer: Could not get abstract code for file: /opt/local/lib/erlang/lib/ssl-3.10.8/examples/certs/ebin/make_certs.beam (please recompile it with +debug_info) > > I think you are making your life unnecessarily complicated. Use the following command which will most probably be sufficient: > > dialyzer --build_plt --apps erts kernel stdlib syntax_tools > > Thanks! It worked! >> I could of course upgrade my Erlang instalation, but I guess that feels a bit wrong ? to upgrade it just because of dialyzer :) > > Trust me: there are more reasons to upgrade. > > kostis Guess, I'll have to this sooner or later :) From rvirding@REDACTED Tue Sep 7 10:09:01 2010 From: rvirding@REDACTED (Robert Virding) Date: Tue, 7 Sep 2010 10:09:01 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> Message-ID: On 7 September 2010 08:27, Richard O'Keefe wrote: > > ... > > (One of the things I just *love* about Ada and Haskell > is that they do not allow 'mixed mode arithmetic'.) This was also something we got wrong. We tried to be helpful and wanted to make it easy to use. To be fair though we never saw arithmetic as an important part of the language and in the beginning we didn't even have floats. Robert From tomasz.pastuch@REDACTED Tue Sep 7 13:28:07 2010 From: tomasz.pastuch@REDACTED (Tomasz Pastuch) Date: 07 Sep 2010 13:28:07 +0200 Subject: How to design credit control =?UTF-8?q?application=3F?= Message-ID: <20100907112807.8843F763ED4@fwb.poczta.interia.pl> Dear all, I?m quite new in erlang language, but I?m fall in love with it after reading Joe?s book. Now I?m trying to create a prototype of application just to see if erlang is suitable for our purpose. Let?s imagine an telco application for soft real-time credit control. From a stack (i.e. diameter, radius) we receive the information; - open a session for user - reserve money/debit money from "balance" - close session for user The sessions can last for hours, but every 2minutes we have another request. Each credit control request reserves new money and debit the money from the previous (2m ago) reservation. Each user can have many concurrent sessions (>1000). The total number of concurrent session will be max 300K. I?m thinking about designing such system in erlang. We want to have fully redundant system - we cannot loose customer balances. From the beginning I was thinking to model that situation using : - customer session -> spawn many erlang process at each open session, use mnesia replicated table for storing reserved amount for each session - customer balances -> use mnesia replicated table The problems I have encountered: 1. Mnesia transactions cannot spread over two tables, so I can loose the reserve transaction (If the node fails after decreasing the value from balances and begin storing new reservation record). If I store the reservation inside the balance record, then probably I have problems with many (> 1000) concurrent sessions for each balance (the record will be simply big and the operations of updating the balance can be slower). 2. If we use processes for modeling user sessions then how to replicate those processes to the replica ? Maybe the better solution is to insert only a record into mnesia for each session, but how then I can process those sessions parallel (when the new request came)? I?m probably not the first person who is going to create such application ? could you advise me how I should design the system in Erlang ? I appreciate for any help. Best regards --------------------------------------------- Volskwagen, Audi, BMW, bezpo?rednio z Niemiec! http://linkint.pl/f27f7 From erlang@REDACTED Tue Sep 7 14:05:00 2010 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 7 Sep 2010 14:05:00 +0200 Subject: bug or feature in erl -pa Message-ID: $ erl -pa '../ebin' > m(foo) ... works - the autoloader finds foo.beam in ../ebin BUT $ erl -pa ../ebin > cd(".."). > m(foo). fails :-) Should ../ebin be converted to an absolute path in the "erl -pa ..." command? /Joe From attila.r.nohl@REDACTED Tue Sep 7 15:55:15 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Tue, 7 Sep 2010 15:55:15 +0200 Subject: [erlang-questions] fixed 80 columns in io:format In-Reply-To: <4C069EB0.8050908@erlang-solutions.com> References: <4C069EB0.8050908@erlang-solutions.com> Message-ID: 2010/6/2, Ulf Wiger : [...] > However, if you want the shell to make use of a wider tty, > the culprit is this: > > http://github.com/erlang/otp/blob/dev/lib/stdlib/src/shell.erl#L1424 > > pp(V, I, RT) -> > io_lib_pretty:print(V, I, columns(), ?LINEMAX, ?CHAR_MAX, > record_print_fun(RT)). > > The ?CHAR_MAX macro expands to 60, which sets the actual > limit for the column depth. If this were -1 instead, the > pretty printer would make use of the full width of the window. With ?CHAR_MAX (I'm not sure if it shows, but the result from the lists:seq is printed in two lines, the io:format is printed in 3 lines): 1> L=lists:seq(1,60). [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22, 23,24,25,26,27,28,29|...] 2> io:format("~p~n", [L]). [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28, 29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,55,56,57,58,59,60] ok With -1 (the result from the lists:seq is printed in one line, the io:format is still printed in 3 lines): 1> L=lists:seq(1,60). [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29|...] 2> io:format("~p~n", [L]). [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28, 29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,55,56,57,58,59,60] ok My terminal is actually 182 characters wide xterm. Is there a way to tell erlang to use all this width? From rzezeski@REDACTED Tue Sep 7 15:58:23 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Tue, 7 Sep 2010 09:58:23 -0400 Subject: [erlang-questions] Using ETS for large amounts of data? In-Reply-To: <57029.206.111.154.162.1283835540.squirrel@duomark.com> References: <57029.206.111.154.162.1283835540.squirrel@duomark.com> Message-ID: On Tue, Sep 7, 2010 at 12:59 AM, wrote: > > Ryan also noted excess memory usage with binary:split: > > I believe binary pattern matching is now preferred over split. If you do > create sub-binaries but do not want all of them, remember that none of the > memory occupied by the underlying large binary can be reclaimed as long as > the sub-binaries reference it. If you filter the sub-binary list, you > should make a new binary copy for each of the retained sub-binaries to > allow the old large binary to be recycled. > > AFAIK, the binary module was introduced in R14A. In fact, I upgraded to R14A just so I could make use of the binary pattern matching in this module. What I do with binary:split would take much more code with binary matching (I think). Maybe you were thinking of erlang:split_binary/2? Anyways, I digress. I don't want to hijack the original thread. I'll post a new thread later today. -Ryan From olivier.boudeville@REDACTED Tue Sep 7 15:54:50 2010 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Tue, 7 Sep 2010 15:54:50 +0200 Subject: [erlang-questions] Random behaviour In-Reply-To: <20100907051310.GC20933@k2r.org> Message-ID: Hi, Shouldn't the link read "http://github.com/jj1bdx/sfmt-erlang" ? By the way, regarding that seeding: how could it be performed reliably without the crypto module? I believe that crypto needs libssl which may not be available on some platforms. Any idea? On a side note, maybe a neat addition around sfmt would be to provide a function which performs an efficient uniform shuffle of a user-specified list (linked to the "Speedy unsort" discussion, http://groups.google.com/group/erlang-programming/browse_thread/thread/ff5c9d88f78e2456). Unless it would be too far stretched from the purpose of such a library? Thanks in advance for any hint, Olivier Boudeville. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 kenji.rikitake@REDACTED Envoy? par : erlang-questions@REDACTED 07/09/2010 07:15 A ttom.kelly@REDACTED cc erlang-questions@REDACTED Objet Re: [erlang-questions] Random behaviour And you can try another PRNG module of SIMD-oriented Fast Mersenne Twister at http://github.com/jj1bdx/sfmt/ (I'm the code maintainer so give it a try and let me know how it works) Note: you still need an individual seeding per process as described below. My preference is to seed the generator frequently with crypto:rand_bytes/1. Kenji Rikitake In the message dated Mon, Sep 06, 2010 at 01:39:26PM +0800, Bob Ippolito writes: > Date: Mon, 6 Sep 2010 13:39:50 +0800 > From: Bob Ippolito > Subject: Re: [erlang-questions] Random behaviour > To: John Hughes > Cc: tom kelly , erlang-questions@REDACTED > > You might also want to look at the crypto module for generating random > numbers instead of the random module. The random module uses an > ancient algorithm and has issues with seeding (as you have noticed). > > On Mon, Sep 6, 2010 at 1:26 PM, John Hughes wrote: > > Hi Tom, > > > > Every Erlang process has its own random number seed, and it's initialised to a CONSTANT value when the process starts! Hence given > > > > foo() -> > > ?? ??io:format("~p\n",[[random:uniform(10) || _ <- lists:seq(1,10)]]). > > > > then you get different values from multiple calls in the SAME process: > > > > 2> foo:foo(). > > [1,5,8,10,6,4,6,10,7,5] > > ok > > 3> foo:foo(). > > [6,2,3,7,2,6,3,5,5,1] > > ok > > 4> foo:foo(). > > [6,5,5,4,1,6,10,4,2,3] > > ok > > > > but the same value every time in a NEW process: > > > > 5> spawn(foo,foo,[]). > > [1,5,8,10,6,4,6,10,7,5] > > <0.45.0> > > 6> spawn(foo,foo,[]). > > [1,5,8,10,6,4,6,10,7,5] > > <0.47.0> > > 7> spawn(foo,foo,[]). > > [1,5,8,10,6,4,6,10,7,5] > > <0.49.0> > > > > I guess it makes sense--you don't want to pay the price of initialising a random number seed every time you start a process--but it is a bit of a gotcha. > > > > If you want different random numbers in different processes (which you usually do), then you need to initialise the seed differently when you start the process. The usual trick is to use now(), since a seed just happens to be three integers: > > > > baz() -> > > ?? ??{A,B,C} = now(), > > ?? ??random:seed(A,B,C), > > ?? ??foo(). > > > > and now > > > > 10> spawn(foo,baz,[]). > > [10,7,9,4,7,1,4,8,3,6] > > <0.61.0> > > 11> spawn(foo,baz,[]). > > [10,5,1,6,5,5,9,3,6,2] > > <0.63.0> > > 12> spawn(foo,baz,[]). > > [8,7,9,7,9,6,1,9,4,3] > > <0.65.0> > > > > One thing to be aware of is that calls to now() close together in time give you seeds the initially generate similar numbers: > > > > 13> spawn(foo,baz,[]),spawn(foo,baz,[]). > > [10,1,8,1,8,1,8,10,1,9] > > [10,1,7,1,6,5,7,8,8,2] > > <0.68.0> > > 14> spawn(foo,baz,[]),spawn(foo,baz,[]). > > [5,9,8,2,2,3,3,9,5,8] > > [5,9,7,2,9,6,2,7,2,1] > > <0.71.0> > > 15> spawn(foo,baz,[]),spawn(foo,baz,[]). > > [10,5,2,3,4,4,3,1,6,3] > > [10,5,1,4,2,8,3,9,3,7] > > <0.74.0> > > > > Note the first two numbers generated are always the same in both processes, even though they start with different seeds. If you're going to start many processes and you want to be sure they have independent random numbers then you have to be a bit more careful how you initialise the seed--but if there are going to be seconds or more between each process start then you don't have to worry. > > > > John > > ??----- Original Message ----- > > ??From: tom kelly > > ??To: erlang-questions@REDACTED > > ??Sent: Sunday, September 05, 2010 8:45 PM > > ??Subject: [erlang-questions] Random behaviour > > > > > > ??Hello List, > > > > > > ??I just found a bahaviour I didn't expect in OTPs random module, it's not necessarily a bug but I thought I'd share it here all the same. > > > > > > ??I wrote a simple random data generator using random:uniform() to help me track a bug that was triggered by certain input data that I couldn't catch. The minimal example is attached in demo.erl. > > > > > > ??If you run the test function it will probably pass a few times but will eventually hit my "bug" and fail. Whenever the bug is triggered the test function gets a badmatch and the current process dies. The behaviour I wasn't expecting now occurs, everytime I repeat the test function I get the exact same sequence of numbers from random:uniform. > > > > > > ??I can call random:uniform from the shell to move it on one step in the sequence and probably pass another test, but when it fails again my process dies and again I repeatedly get a repeated sequence of numbers. The sequence from this second fail is different from the first fail so it hasn't just gone back to the start. > > > > > > ??I don't expect to be saved from my own bugs but just thought this was a strange behaviour from random. If anyone is using random in their application they should be aware of this. > > > > > > ??//Tom. > > > > > > > > > > ??8> demo:test_my_buggy_code(). > > ??ok - [15,21,70,16,56,22,46,43,1,57] > > ??ok > > ??9> demo:test_my_buggy_code(). > > ??ok - [41,31,6,58,99,34,19,21,4,89] > > ??ok > > ??10> demo:test_my_buggy_code(). > > ??ok - [83,33,26,81,2,3,5,6,99,57] > > ??ok > > ??11> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 72,41,80] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??12> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,39,67,88,90,80,59,30, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 72,41,80] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??13> random:uniform(1). > > ??1 > > ??14> demo:test_my_buggy_code(). > > ??ok - [67,88,90,80,59,30,72,41,80,8] > > ??ok > > ??15> demo:test_my_buggy_code(). > > ??ok - [60,68,65,8,27,61,1,94,82,65] > > ??ok > > ??16> demo:test_my_buggy_code(). > > ??ok - [47,2,62,94,25,97,94,87,27,32] > > ??ok > > ??17> demo:test_my_buggy_code(). > > ??ok - [59,21,10,14,17,24,93,78,34,28] > > ??ok > > ??18> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 60,42,2] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??19> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 60,42,2] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??20> demo:test_my_buggy_code(). > > ??** exception error: no match of right hand side value [1,3,59,96,93,32,99,34, > > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 60,42,2] > > ?? ?? ?? in function ??demo:test_my_buggy_code/0 > > ?? ?? ?? in call from random:uniform/0 > > ??21> > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > ??________________________________________________________________ > > ??erlang-questions (at) erlang.org mailing list. > > ??See http://www.erlang.org/faq.html > > ??To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. From kenji.rikitake@REDACTED Tue Sep 7 16:35:14 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 7 Sep 2010 23:35:14 +0900 Subject: [erlang-questions] Random behaviour In-Reply-To: References: <20100907051310.GC20933@k2r.org> Message-ID: <20100907143514.GA31677@k2r.org> In the message dated Tue, Sep 07, 2010 at 03:54:26PM +0200, Olivier BOUDEVILLE writes: > Shouldn't the link read "http://github.com/jj1bdx/sfmt-erlang" ? You are right. My mistake. > By the way, regarding that seeding: how could it be performed reliably > without the crypto module? I believe that crypto needs libssl which may > not be available on some platforms. Any idea? /dev/random either on Linux or on FreeBSD will work as (at least practically) a reliable seeder, though I suggest crypto:rand_bytes/1 (with OpenSSL). > On a side note, maybe a neat addition around sfmt would be to provide a > function which performs an efficient uniform shuffle of a user-specified > list (linked to the "Speedy unsort" discussion, > http://groups.google.com/group/erlang-programming/browse_thread/thread/ff5c9d88f78e2456). > Unless it would be too far stretched from the purpose of such a library? I remember the discussion, though I don't really recall the shuffling algorithm itself now. SFMT is basically a 32bit integer PRNG so you can use it for whatever you want. sfmt:gen_rand32_max/1 will be helpful (generating the integer sequence between 0 to N-1) Regards, Kenji Rikitake From freza@REDACTED Tue Sep 7 19:12:18 2010 From: freza@REDACTED (Jachym Holecek) Date: Tue, 7 Sep 2010 18:12:18 +0100 Subject: [erlang-questions] fixed 80 columns in io:format In-Reply-To: References: <4C069EB0.8050908@erlang-solutions.com> Message-ID: <20100907171218.GA28797@hanele> Hello, # Attila Rajmund Nohl 2010-09-07: > My terminal is actually 182 characters wide xterm. Is there a way to > tell erlang to use all this width? Didn't follow the thread closely, but maybe 3> io:format("~90p~n", [lists:seq(1,100)]). [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32, 33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61, 62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90, 91,92,93,94,95,96,97,98,99,100] will help (note the use of precision field in format spec). Regards, -- Jachym From daniel.goertzen@REDACTED Tue Sep 7 21:17:04 2010 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Tue, 7 Sep 2010 14:17:04 -0500 Subject: [erlang-questions] ssh client In-Reply-To: <20100907045756.GB20933@k2r.org> References: <20100907045756.GB20933@k2r.org> Message-ID: Thanks Kenji. I did come across that and it was helpful. In the end, ssh_sftp and ssh_sftpd were the readily available examples I was looking for. This might be of interest to the list, so I'll share it... I stress tested Erlang and SSH by establishing 1000 concurrent SSH client sessions. Each server sent back an 8 byte message every 250ms. On my Windows Core2Duo 3GHz, system load was ~25%. To compare, I also wrote the client in Python using Paramiko for SSH (1 OS thread per connection). CPU load and memory usage was consistent with the Erlang solution, but it died at 400 connections due to address space exhaustion (thread stacks were too big...probably fixable, but I didn't try too hard) Also, I wrote a Python client that used plink.exe for SSH. (1 OS process per connection). CPU load was consistent with the Erlang solution, but memory use was insane. It died at ~500 connections due to lack of file descriptors (also probably fixable, but I didn't try). In all cases, the server was a single linux machine running Erlang/SSH. The actual use case for all this is to talk SSH to 1000 individual embedded systems on a large LAN. I am pleased with the way Erlang has behaved. Dan. On Mon, Sep 6, 2010 at 11:57 PM, Kenji Rikitake wrote: > http://github.com/jj1bdx/sshrpc/ > > Far from a complete one - but might be useful. > Kenji Rikitake > > In the message RJhO4NVkvUYsd@REDACTED> > dated Wed, Sep 01, 2010 at 11:17:21AM -0500, > Daniel Goertzen writes: > > I want to write a program that will chat with another remote program via > > SSH. Erlang SSH is a bit complex, and I was unable to find an example of > > what I wanted, so I thought I would confirm my approach here before > barking > > up the wrong tree. Here is what I was going to do: > > > > > > 1. Open a connection with ssh:connect() > > 2. Open a channel with ssh_connection:session_channel() > > 3. Call ssh_channel:start_link() to create my handler process. > > 4. In my callback module, init() will call ssh_connection:exec() to > invoke > > the remote program. > > 5. Incoming data gets sent to my handle_ssh_message() callback. I send > data > > with ssh_connection:send() > > > > Do I have it right? > > > > It would be nice if there was a "port" abstraction for an ssh channel, so > I > > could treat locally and remotely invoked programs identically... I guess > it > > wouldn't be hard to write my own wrapper for that. > > > > Thanks, > > Dan. > -- Daniel Goertzen ----------------- dang@REDACTED (work) daniel.goertzen@REDACTED (home) ----------------- 1 204 272 6149 (home/office) 1 204 470 8360 (mobile) ----------------- From jay@REDACTED Tue Sep 7 21:44:51 2010 From: jay@REDACTED (jay@REDACTED) Date: Tue, 7 Sep 2010 12:44:51 -0700 (PDT) Subject: [erlang-questions] Using ETS for large amounts of data? In-Reply-To: References: <57029.206.111.154.162.1283835540.squirrel@duomark.com> Message-ID: <46990.74.1.186.35.1283888691.squirrel@duomark.com> Yes, I was thinking of erlang:split_binary/2. I haven't used the new binary module yet. > On Tue, Sep 7, 2010 at 12:59 AM, wrote: > >> >> Ryan also noted excess memory usage with binary:split: >> >> I believe binary pattern matching is now preferred over split. If you >> do >> create sub-binaries but do not want all of them, remember that none of >> the >> memory occupied by the underlying large binary can be reclaimed as long >> as >> the sub-binaries reference it. If you filter the sub-binary list, you >> should make a new binary copy for each of the retained sub-binaries to >> allow the old large binary to be recycled. >> >> > AFAIK, the binary module was introduced in R14A. In fact, I upgraded to > R14A just so I could make use of the binary pattern matching in this > module. > What I do with binary:split would take much more code with binary > matching > (I think). Maybe you were thinking of erlang:split_binary/2? > > Anyways, I digress. I don't want to hijack the original thread. I'll > post > a new thread later today. > > -Ryan > From tuncer.ayaz@REDACTED Wed Sep 8 02:38:51 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 8 Sep 2010 02:38:51 +0200 Subject: xml_from_edocs.escript HiPE segfault in R14B dev branch In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 8:17 PM, Tuncer Ayaz wrote: > FYI, > > When building erlang with HiPE enabled and native-libs in the current > dev tree you might encounter a segfault with xml_from_edocs.escript. > This will usually happen if you call 'make docs'. > > Rickard and OTP team has git bisected the issue to the following first > bad commit which exposes a possible bug in HiPE: > a8b8ec5e858da86531933b545f752f436e411b58 > Load native code for modules loaded before the code server > > This commit is only bad because it exposed the bug by causing more > native compiled code to be executed than before. > > The HiPE team is informed and will hopefully fix it before R14B. > > If you happen to experience the issue just make sure to build the docs > with a non-hipe beam. It looks like Paul Guyot's erts_gc_after_bif_call patch corrected the segfault: 8bbcecd938f46127bc40ff7aed91d4e519d0cd5b Fix call to erts_gc_after_bif_call in hipe glue A remaining issue with a native enabled beam is that you can easily reproduce a hang of the vm if you try to compile rebar or use rebar. Interestingly this is again on invocation of an escript. $ hg clone http://bitbucket.org/basho/rebar $ cd rebar $ make clean all [...] Recompile: src/rebar_utils Recompile: src/rebar_xref ^Cmake: *** [all] Interrupt From caox@REDACTED Wed Sep 8 05:21:03 2010 From: caox@REDACTED (caox) Date: Wed, 8 Sep 2010 11:21:03 +0800 Subject: How to release term created by enif_make_copy Message-ID: Hi In my experiment of erlang nif, I use an global env to store term received from erlang proc s via enif_make_copy(). I am wondering how could I free some specific terms created on this env but not all of them? I didn't find any lib funcs to do this job. Both enif_clear_env() and enif_free_env() do whole cleanup, and enif_free is annotated to be used only for memory allocated by enif_alloc. From rzezeski@REDACTED Wed Sep 8 06:52:14 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Wed, 8 Sep 2010 00:52:14 -0400 Subject: large binaries split into many small pieces Message-ID: I've been writing an Erlang app (my first one) for the last 2 and 1/2 months and one of it's duties is to parse/transform _large_ CSV files. Some of them 11+ million lines and 600+ MB in size and they will only get larger as time goes on. Luckily I've got 128GB of RAM to work with, so I decided to do everything in memory via binaries. My initial, naive attempt was to read the file into one large binary and then split that into it's individual columns. The end result would be a list of lists, each sublist being a list of binaries representing the columns. e.g. [[<<"col1">>,<<"col2">>], [<<"val1">>,<<"val2">>]] This chewed up a _lot_ of memory, and my runtimes were nothing to write home about. A few days ago I sat down and tried to read up more on binaries, their idiomatic usage, pitfalls, etc. I ended up re-implementing my CSV functions to build a new binary while parsing the source binary. My memory usage went down drastically (although it spikes b/c of ephemeral garbage, lots of temp binaries) and my runtimes improved greatly! Anyways, while reading the docs on binary handling in the efficiency guide I got the notion that if you match/split a large binary the resulting sub-binary will reference the larger, off-heap one. That is, if you pull a chunk of a large binary it's storage is essentially free. However, after running some tests I feel like a lot of heap memory is allocated. I wrote a more formal test tonight, and honestly, I'm not sure I understand things any better. I created a bunch of test runs on a 27MB CSV file with 597971 lines each with 8 columns. list_of_cols: break each column into it's own binary list_of_lines: break each line into it's own binary list_of_32byte: break into 32 byte chunks list_of_64byte: " " list_of_128byte: ... Each test shows certain memory stats for each stage of the test: start, after send, after run, after GC, and after shutdown. The main thing to focus on is the difference in process allocated memory between "after send data" and "after run" stages. Here are my results and attached is the code. 2> binary_test:run_tests(). *** After read total | processes used | binary 35.57MB | 0.81MB | 26.78MB ==========RUNNING list_of_cols============================ *** Start total | processes used | binary 35.53MB | 0.77MB | 26.78MB *** After send data total | processes used | binary 35.53MB | 0.77MB | 26.78MB *** After run total | processes used | binary 400.45MB | 365.68MB | 26.79MB *** After GC total | processes used | binary 400.45MB | 365.68MB | 26.78MB *** After shutdown total | processes used | binary 35.55MB | 0.78MB | 26.78MB ==========DONE list_of_cols=============================== ==========RUNNING list_of_lines=========================== *** Start total | processes used | binary 35.55MB | 0.78MB | 26.78MB *** After send data total | processes used | binary 35.55MB | 0.78MB | 26.78MB *** After run total | processes used | binary 84.53MB | 49.75MB | 26.79MB *** After GC total | processes used | binary 84.53MB | 49.75MB | 26.78MB *** After shutdown total | processes used | binary 35.55MB | 0.78MB | 26.78MB ==========DONE list_of_lines============================== ==========RUNNING list_of_32byte========================== *** Start total | processes used | binary 35.55MB | 0.78MB | 26.78MB *** After send data total | processes used | binary 35.56MB | 0.79MB | 26.79MB *** After run total | processes used | binary 96.76MB | 61.99MB | 26.78MB *** After GC total | processes used | binary 112.07MB | 77.30MB | 26.78MB *** After shutdown total | processes used | binary 35.55MB | 0.78MB | 26.78MB ==========DONE list_of_32byte============================= ==========RUNNING list_of_64byte========================== *** Start total | processes used | binary 35.55MB | 0.78MB | 26.78MB *** After send data total | processes used | binary 35.56MB | 0.79MB | 26.78MB *** After run total | processes used | binary 66.90MB | 32.13MB | 26.78MB *** After GC total | processes used | binary 66.90MB | 32.13MB | 26.78MB *** After shutdown total | processes used | binary 35.56MB | 0.78MB | 26.78MB ==========DONE list_of_64byte============================= ==========RUNNING list_of_128byte========================= *** Start total | processes used | binary 35.56MB | 0.78MB | 26.78MB *** After send data total | processes used | binary 35.56MB | 0.79MB | 26.78MB *** After run total | processes used | binary 51.61MB | 16.83MB | 26.78MB *** After GC total | processes used | binary 51.61MB | 16.83MB | 26.79MB *** After shutdown total | processes used | binary 35.55MB | 0.77MB | 26.79MB ==========DONE list_of_128byte============================ ==========RUNNING list_of_256byte========================= *** Start total | processes used | binary 35.55MB | 0.78MB | 26.79MB *** After send data total | processes used | binary 35.55MB | 0.78MB | 26.79MB *** After run total | processes used | binary 45.82MB | 11.05MB | 26.79MB *** After GC total | processes used | binary 45.83MB | 11.06MB | 26.79MB *** After shutdown total | processes used | binary 35.56MB | 0.78MB | 26.79MB ==========DONE list_of_256byte============================ ==========RUNNING list_of_512byte========================= *** Start total | processes used | binary 35.54MB | 0.77MB | 26.79MB *** After send data total | processes used | binary 35.55MB | 0.77MB | 26.79MB *** After run total | processes used | binary 41.89MB | 7.12MB | 26.78MB *** After GC total | processes used | binary 41.89MB | 7.13MB | 26.78MB *** After shutdown total | processes used | binary 35.55MB | 0.78MB | 26.78MB ==========DONE list_of_512byte============================ I went ahead and did some _rough_ calculations on how much each sub-binary costs on the process heap by taking the difference in processes_used memory and dividing by the number of sub-binaries created. I got the following numbers: list_of_cols: 80B list_of_lines: 86B list_of_32byte: 73B list_of_64byte: 75B list_of_128byte: 76B list_of_256byte: 99B list_of_512byte: 122B So does this mean a Refc/sub-binary costs roughly 80B of memory? Am I thinking about this too much? Probably. Just to be clear, I'm now happy with the performance of my CSV routines. I'm writing this because I want to understand the underlying binary implementation better and because I spent too much time setting up my tests not to post this :). If you read this far, thank you. -Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: binary_test.erl Type: application/octet-stream Size: 3443 bytes Desc: not available URL: From tuncer.ayaz@REDACTED Wed Sep 8 14:07:27 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 8 Sep 2010 14:07:27 +0200 Subject: xml_from_edocs.escript HiPE segfault in R14B dev branch In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 2:38 AM, Tuncer Ayaz wrote: > It looks like Paul Guyot's erts_gc_after_bif_call patch corrected the segfault: > 8bbcecd938f46127bc40ff7aed91d4e519d0cd5b > Fix call to erts_gc_after_bif_call in hipe glue Forgot to add that this is part of otp.git dev and therefore the fix is already part of the soon to be R14B. From jesper.louis.andersen@REDACTED Wed Sep 8 14:52:28 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 8 Sep 2010 14:52:28 +0200 Subject: [erlang-questions] How to design credit control application? In-Reply-To: <20100907112807.8843F763ED4@fwb.poczta.interia.pl> References: <20100907112807.8843F763ED4@fwb.poczta.interia.pl> Message-ID: On Tue, Sep 7, 2010 at 1:28 PM, Tomasz Pastuch wrote: > I?m thinking about designing such system in erlang. We want to have fully redundant system - we cannot loose customer balances. [..] > I?m probably not the first person who is going to create such application ? could you advise me how I should design the system in Erlang ? I appreciate for any help. Some general comments: The customer balances are part of the "error kernel" in the sense that if a customer has been billed, it is very important to track this in a safe way. In other words, the part of the code which is responsible for this should be made as simple as possible - with all potentially dangerous operations running compartamentalized in separate processes. Also that kernel should probably, as you hint, be written in a redundant way as otherwise you may loose it as soon as a machine dies to the chainsaw-machine-murderer. In other words: Identify the error kernel! Figure out what processes you can't afford to loose and protect them. Once you have an idea of where the error kernel is, you can begin to think about the processes which need less protection in the system. Whether to use mnesia or not is something I will skip commenting on. My knowledge with mnesia is too weak to really understand it. -- J. From dmercer@REDACTED Wed Sep 8 15:31:17 2010 From: dmercer@REDACTED (David Mercer) Date: Wed, 8 Sep 2010 08:31:17 -0500 Subject: [erlang-questions] bug or feature in erl -pa In-Reply-To: References: Message-ID: <002401cb4f5a$1fbc5bc0$5f351340$@com> On September 07, Joe Armstrong wrote: > Should ../ebin be converted to an absolute path in the "erl -pa ..." > command? I would think so. I would generally expect unquoted relative paths at the command line to exhibit the behavior as if expanded by the shell. DBM From dmercer@REDACTED Wed Sep 8 15:44:10 2010 From: dmercer@REDACTED (David Mercer) Date: Wed, 8 Sep 2010 08:44:10 -0500 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> Message-ID: <002b01cb4f5b$eba94b70$c2fbe250$@com> On September 07, Robert Virding wrote: > On 7 September 2010 08:27, Richard O'Keefe wrote: > > > > ... > > > > (One of the things I just *love* about Ada and Haskell > > is that they do not allow 'mixed mode arithmetic'.) > > This was also something we got wrong. We tried to be helpful and > wanted to make it easy to use. To be fair though we never saw > arithmetic as an important part of the language and in the beginning > we didn't even have floats. I wonder if the equality operator should even be applicable to floats. After all, float F1 really represents a number between F1 - epsilon and F2 + epsilon. From freza@REDACTED Wed Sep 8 15:48:38 2010 From: freza@REDACTED (Jachym Holecek) Date: Wed, 8 Sep 2010 14:48:38 +0100 Subject: [erlang-questions] bug or feature in erl -pa In-Reply-To: <002401cb4f5a$1fbc5bc0$5f351340$@com> References: <002401cb4f5a$1fbc5bc0$5f351340$@com> Message-ID: <20100908134838.GA3741@hanele> # David Mercer 2010-09-08: > On September 07, Joe Armstrong wrote: > > > Should ../ebin be converted to an absolute path in the "erl -pa ..." > > command? > > I would think so. I would generally expect unquoted relative paths at the > command line to exhibit the behavior as if expanded by the shell. ... which to leave them relative to CWD. Both "." and ".." are real directories, not some kind of illusion fabricated by shell. :-) (Just amused by the contradiction in your response, not really disagreeing with the idea.) Regards, -- Jachym From Antonio.Musumeci@REDACTED Wed Sep 8 15:35:35 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Wed, 8 Sep 2010 09:35:35 -0400 Subject: System limit bringing down rex and the VM Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> Is this an expected behavior? Shouldn't there be a way to more graciously handle this type of thing? =ERROR REPORT==== 8-Sep-2010::09:28:26 === ** Generic server rex terminating ** Last message in was {call,mnesia_lib,get_node_number,[],<0.55.0>} ** When Server state == {0,nil} ** Reason for termination == ** {system_limit,[{erlang,spawn_opt, [{erlang,apply, [#Fun,[]], [monitor]}]}, {erlang,spawn_monitor,1}, {rpc,handle_call_call,6}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} =CRASH REPORT==== 8-Sep-2010::09:28:26 === crasher: initial call: rpc:init/1 pid: <0.12.0> registered_name: rex exception exit: {system_limit, [{erlang,spawn_opt, [{erlang,apply, [#Fun,[]], [monitor]}]}, {erlang,spawn_monitor,1}, {rpc,handle_call_call,6}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} in function gen_server:terminate/6 ancestors: [kernel_sup,<0.10.0>] messages: [] links: [<0.11.0>] dictionary: [] trap_exit: true status: running heap_size: 377 stack_size: 24 reductions: 143 neighbours: =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === Supervisor: {local,kernel_sup} Context: child_terminated Reason: {system_limit, [{erlang,spawn_opt, [{erlang,apply, [#Fun,[]], [monitor]}]}, {erlang,spawn_monitor,1}, {rpc,handle_call_call,6}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} Offender: [{pid,<0.12.0>}, {name,rex}, {mfargs,{rpc,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === Supervisor: {local,kernel_sup} Context: shutdown Reason: reached_max_restart_intensity Offender: [{pid,<0.12.0>}, {name,rex}, {mfargs,{rpc,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] =ERROR REPORT==== 8-Sep-2010::09:28:26 === Mnesia(igo@REDACTED): ** ERROR ** mnesia_recover got unexpected info: {'EXIT', <0.84.0>, shutdown} From boris.muehmer@REDACTED Wed Sep 8 16:09:35 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Wed, 8 Sep 2010 16:09:35 +0200 Subject: Erlang module for "controlling" (linux/unix) commands? Message-ID: Is there a module for controlling (linux/unix) "shell" commands? I am looking for something more "special" than "os:cmd/1", because I need to get the exit code of the command. Thanks in advance, - boris From zoltan.ban@REDACTED Wed Sep 8 16:16:02 2010 From: zoltan.ban@REDACTED (=?iso-8859-1?Q?Zolt=E1n_B=E1n?=) Date: Wed, 8 Sep 2010 16:16:02 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: References: Message-ID: <0B3B7637B1893048A4F2CFFEFCF350E9015FDAF9@ESESSCMS0356.eemea.ericsson.se> os:cmd("ls /nodir >/dev/null 2>&1; echo $?"). cheers Z -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Boris M?hmer Sent: den 8 september 2010 16:10 To: erlang-questions Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? Is there a module for controlling (linux/unix) "shell" commands? I am looking for something more "special" than "os:cmd/1", because I need to get the exit code of the command. Thanks in advance, - boris ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From tuncer.ayaz@REDACTED Wed Sep 8 16:22:55 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 8 Sep 2010 16:22:55 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 4:09 PM, Boris M?hmer wrote: > Is there a module for controlling (linux/unix) "shell" commands? > > I am looking for something more "special" than "os:cmd/1", > because I need to get the exit code of the command. Take a look at erlang:open_port/2. call erlang:open_port({spawn_executable, Command}, PortSettings) where PortSettings at least contains the exit_status option You might also have to pass the use_stdio and {line, L} options. The exit code will be delivered as a message: receive {Port, {data, ...}} -> ... {Port,{exit_status,Status}} -> ... end From boris.muehmer@REDACTED Wed Sep 8 16:24:11 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Wed, 8 Sep 2010 16:24:11 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: <0B3B7637B1893048A4F2CFFEFCF350E9015FDAF9@ESESSCMS0356.eemea.ericsson.se> References: <0B3B7637B1893048A4F2CFFEFCF350E9015FDAF9@ESESSCMS0356.eemea.ericsson.se> Message-ID: 2010/9/8 Zolt?n B?n : > os:cmd("ls /nodir >/dev/null 2>&1; echo $?"). I already thought about this... but sometimes I need the output as well and I don't want to start parsing the result. I would have prefered something like x:cmd(Command) -> {ReturnCode, Output} - boris From tuncer.ayaz@REDACTED Wed Sep 8 16:25:16 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 8 Sep 2010 16:25:16 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 4:22 PM, Tuncer Ayaz wrote: > On Wed, Sep 8, 2010 at 4:09 PM, Boris M?hmer wrote: >> Is there a module for controlling (linux/unix) "shell" commands? >> >> I am looking for something more "special" than "os:cmd/1", >> because I need to get the exit code of the command. > > Take a look at erlang:open_port/2. > > call erlang:open_port({spawn_executable, Command}, PortSettings) > where PortSettings at least contains the exit_status option .?You might > also have to pass the use_stdio and {line, L} options. To be precise: use_stdio and {line, L} might be useful. That's what I wanted to write. > The exit code will be delivered as a message: > receive > ? ?{Port, {data, ...}} -> ... > ? ?{Port,{exit_status,Status}} -> ... > end From boris.muehmer@REDACTED Wed Sep 8 16:29:31 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Wed, 8 Sep 2010 16:29:31 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: References: Message-ID: 2010/9/8 Tuncer Ayaz : > On Wed, Sep 8, 2010 at 4:22 PM, Tuncer Ayaz wrote: >> On Wed, Sep 8, 2010 at 4:09 PM, Boris M?hmer wrote: >>> [...] >> Take a look at erlang:open_port/2. > [...] >> The exit code will be delivered as a message: >> receive >> ? ?{Port, {data, ...}} -> ... >> ? ?{Port,{exit_status,Status}} -> ... >> end Thanks, I think that is what I was looking for! - boris From mevans@REDACTED Wed Sep 8 17:24:13 2010 From: mevans@REDACTED (Evans, Matthew) Date: Wed, 8 Sep 2010 11:24:13 -0400 Subject: mnesia and QLC in dirty mode Message-ID: Hi, What is the exact syntax for doing a QLC query in dirty mode? (sicm@REDACTED)149> Cursor = mnesia:activity(sync_dirty, fun qlc:cursor/1, [qlc:q([R || R<- mnesia:table(cmfs_index_table)])], read). {qlc_cursor,{<0.3115.0>,<0.3102.0>}} (sicm@REDACTED)150> qlc:next_answers(Cursor,1). ** exception error: undefined function read:select/6 in function mnesia:'-table/2-fun-1-'/4 in call from qlc:reply/4 in call from qlc:next_loop/3 (sicm@REDACTED)151> If I wrap the qlc:next_answers in an mnesia:activity I get: (sicm@REDACTED)2> mnesia:activity(sync_dirty, fun qlc:next_answers/2, [Cursor,1], read). ** exception exit: {undef,[{read,select, [{sync_dirty,<0.625.0>}, non_transaction,cmfs_index_table, [{'$1',[],['$1']}], 100,read]}, {mnesia,'-table/2-fun-1-',4}, {qlc,reply,4}, {qlc,next_loop,3}, {mnesia_tm,non_transaction,5}, {erl_eval,do_apply,5}, {shell,exprs,7}, {shell,eval_exprs,7}]} in function mnesia_tm:non_transaction/5 in call from mnesia_tm:non_transaction/5 Any ideas? Thanks Matt From igorrs@REDACTED Wed Sep 8 17:29:24 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Wed, 8 Sep 2010 12:29:24 -0300 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> Message-ID: I know this is obvious, but the first step should be adjusting your OS limits so that your system, under expected conditions, never fails because of them. I have seen a couple of funny behaviours when exhausting system limits in the QA environment, so I wouldn't count with gracious failures from Erlang/Mnesia/etc. Good luck. Igor. On Wed, Sep 8, 2010 at 10:35 AM, Musumeci, Antonio S wrote: > Is this an expected behavior? Shouldn't there be a way to more graciously handle this type of thing? > > =ERROR REPORT==== 8-Sep-2010::09:28:26 === > ** Generic server rex terminating > ** Last message in was {call,mnesia_lib,get_node_number,[],<0.55.0>} > ** When Server state == {0,nil} > ** Reason for termination == > ** {system_limit,[{erlang,spawn_opt, > ? ? ? ? ? ? ? ? ? ? ? ? ?[{erlang,apply, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [#Fun,[]], > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [monitor]}]}, > ? ? ? ? ? ? ? ? ?{erlang,spawn_monitor,1}, > ? ? ? ? ? ? ? ? ?{rpc,handle_call_call,6}, > ? ? ? ? ? ? ? ? ?{gen_server,handle_msg,5}, > ? ? ? ? ? ? ? ? ?{proc_lib,init_p_do_apply,3}]} > > =CRASH REPORT==== 8-Sep-2010::09:28:26 === > ?crasher: > ? ?initial call: rpc:init/1 > ? ?pid: <0.12.0> > ? ?registered_name: rex > ? ?exception exit: {system_limit, > ? ? ? ? ? ? ? ? ? ? ? ?[{erlang,spawn_opt, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{erlang,apply, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[#Fun,[]], > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[monitor]}]}, > ? ? ? ? ? ? ? ? ? ? ? ? {erlang,spawn_monitor,1}, > ? ? ? ? ? ? ? ? ? ? ? ? {rpc,handle_call_call,6}, > ? ? ? ? ? ? ? ? ? ? ? ? {gen_server,handle_msg,5}, > ? ? ? ? ? ? ? ? ? ? ? ? {proc_lib,init_p_do_apply,3}]} > ? ? ?in function ?gen_server:terminate/6 > ? ?ancestors: [kernel_sup,<0.10.0>] > ? ?messages: [] > ? ?links: [<0.11.0>] > ? ?dictionary: [] > ? ?trap_exit: true > ? ?status: running > ? ?heap_size: 377 > ? ?stack_size: 24 > ? ?reductions: 143 > ?neighbours: > > =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === > ? ? Supervisor: {local,kernel_sup} > ? ? Context: ? ?child_terminated > ? ? Reason: ? ? {system_limit, > ? ? ? ? ? ? ? ? ? ? [{erlang,spawn_opt, > ? ? ? ? ? ? ? ? ? ? ? ? ?[{erlang,apply, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [#Fun,[]], > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [monitor]}]}, > ? ? ? ? ? ? ? ? ? ? ?{erlang,spawn_monitor,1}, > ? ? ? ? ? ? ? ? ? ? ?{rpc,handle_call_call,6}, > ? ? ? ? ? ? ? ? ? ? ?{gen_server,handle_msg,5}, > ? ? ? ? ? ? ? ? ? ? ?{proc_lib,init_p_do_apply,3}]} > ? ? Offender: ? [{pid,<0.12.0>}, > ? ? ? ? ? ? ? ? ?{name,rex}, > ? ? ? ? ? ? ? ? ?{mfargs,{rpc,start_link,[]}}, > ? ? ? ? ? ? ? ? ?{restart_type,permanent}, > ? ? ? ? ? ? ? ? ?{shutdown,2000}, > ? ? ? ? ? ? ? ? ?{child_type,worker}] > > > =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === > ? ? Supervisor: {local,kernel_sup} > ? ? Context: ? ?shutdown > ? ? Reason: ? ? reached_max_restart_intensity > ? ? Offender: ? [{pid,<0.12.0>}, > ? ? ? ? ? ? ? ? ?{name,rex}, > ? ? ? ? ? ? ? ? ?{mfargs,{rpc,start_link,[]}}, > ? ? ? ? ? ? ? ? ?{restart_type,permanent}, > ? ? ? ? ? ? ? ? ?{shutdown,2000}, > ? ? ? ? ? ? ? ? ?{child_type,worker}] > > > =ERROR REPORT==== 8-Sep-2010::09:28:26 === > Mnesia(igo@REDACTED): ** ERROR ** mnesia_recover got unexpected info: {'EXIT', > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? <0.84.0>, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? shutdown} > -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From igorrs@REDACTED Wed Sep 8 17:33:05 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Wed, 8 Sep 2010 12:33:05 -0300 Subject: [erlang-questions] mnesia and QLC in dirty mode In-Reply-To: References: Message-ID: You are passing the atom read as the callback module: http://erlang.org/doc/man/mnesia.html#activity-4 On Wed, Sep 8, 2010 at 12:24 PM, Evans, Matthew wrote: > Hi, > > What is the exact syntax for doing a QLC query in dirty mode? > > > (sicm@REDACTED)149> Cursor = ?mnesia:activity(sync_dirty, fun qlc:cursor/1, [qlc:q([R || R<- mnesia:table(cmfs_index_table)])], read). > {qlc_cursor,{<0.3115.0>,<0.3102.0>}} > > (sicm@REDACTED)150> qlc:next_answers(Cursor,1). > ** exception error: undefined function read:select/6 > ? ? in function ?mnesia:'-table/2-fun-1-'/4 > ? ? in call from qlc:reply/4 > ? ? in call from qlc:next_loop/3 > (sicm@REDACTED)151> > > > If I wrap the qlc:next_answers in an mnesia:activity I get: > > (sicm@REDACTED)2> mnesia:activity(sync_dirty, fun qlc:next_answers/2, [Cursor,1], read). > ** exception exit: {undef,[{read,select, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{sync_dirty,<0.625.0>}, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?non_transaction,cmfs_index_table, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{'$1',[],['$1']}], > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?100,read]}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {mnesia,'-table/2-fun-1-',4}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {qlc,reply,4}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {qlc,next_loop,3}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {mnesia_tm,non_transaction,5}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {erl_eval,do_apply,5}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {shell,exprs,7}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {shell,eval_exprs,7}]} > ? ? in function ?mnesia_tm:non_transaction/5 > ? ? in call from mnesia_tm:non_transaction/5 > > > Any ideas? > > Thanks > > Matt > -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From mevans@REDACTED Wed Sep 8 17:40:34 2010 From: mevans@REDACTED (Evans, Matthew) Date: Wed, 8 Sep 2010 11:40:34 -0400 Subject: [erlang-questions] mnesia and QLC in dirty mode In-Reply-To: References: Message-ID: Yup, That was it...Well spotted :) Thanks Matt -----Original Message----- From: Igor Ribeiro Sucupira [mailto:igorrs@REDACTED] Sent: Wednesday, September 08, 2010 11:33 AM To: Evans, Matthew Cc: Erlang Questions Mailinglist Subject: Re: [erlang-questions] mnesia and QLC in dirty mode You are passing the atom read as the callback module: http://erlang.org/doc/man/mnesia.html#activity-4 On Wed, Sep 8, 2010 at 12:24 PM, Evans, Matthew wrote: > Hi, > > What is the exact syntax for doing a QLC query in dirty mode? > > > (sicm@REDACTED)149> Cursor = ?mnesia:activity(sync_dirty, fun qlc:cursor/1, [qlc:q([R || R<- mnesia:table(cmfs_index_table)])], read). > {qlc_cursor,{<0.3115.0>,<0.3102.0>}} > > (sicm@REDACTED)150> qlc:next_answers(Cursor,1). > ** exception error: undefined function read:select/6 > ? ? in function ?mnesia:'-table/2-fun-1-'/4 > ? ? in call from qlc:reply/4 > ? ? in call from qlc:next_loop/3 > (sicm@REDACTED)151> > > > If I wrap the qlc:next_answers in an mnesia:activity I get: > > (sicm@REDACTED)2> mnesia:activity(sync_dirty, fun qlc:next_answers/2, [Cursor,1], read). > ** exception exit: {undef,[{read,select, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{sync_dirty,<0.625.0>}, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?non_transaction,cmfs_index_table, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{'$1',[],['$1']}], > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?100,read]}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {mnesia,'-table/2-fun-1-',4}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {qlc,reply,4}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {qlc,next_loop,3}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {mnesia_tm,non_transaction,5}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {erl_eval,do_apply,5}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {shell,exprs,7}, > ? ? ? ? ? ? ? ? ? ? ? ? ? {shell,eval_exprs,7}]} > ? ? in function ?mnesia_tm:non_transaction/5 > ? ? in call from mnesia_tm:non_transaction/5 > > > Any ideas? > > Thanks > > Matt > -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From Antonio.Musumeci@REDACTED Wed Sep 8 18:22:31 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Wed, 8 Sep 2010 12:22:31 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> OS limits? This is the BEAM process limit. Rex doesn't gracefully handle errors which in this case is triggered by mnesia_recover. I don't see the point in rex not handling it more gracefully either by returning the error to the caller or restarting. Rex may be a base process but it's not that important IMO to cause the VM to shutdown. I'd rather not have to implement OOM killer like behavior or some other watchdog process just so my VM doesn't crash due to hitting otherwise arbitrary max process limits. -----Original Message----- From: Igor Ribeiro Sucupira [mailto:igorrs@REDACTED] Sent: Wednesday, September 08, 2010 11:29 AM To: Musumeci, Antonio S (Enterprise Infrastructure) Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM I know this is obvious, but the first step should be adjusting your OS limits so that your system, under expected conditions, never fails because of them. I have seen a couple of funny behaviours when exhausting system limits in the QA environment, so I wouldn't count with gracious failures from Erlang/Mnesia/etc. Good luck. Igor. On Wed, Sep 8, 2010 at 10:35 AM, Musumeci, Antonio S wrote: > Is this an expected behavior? Shouldn't there be a way to more graciously handle this type of thing? > > =ERROR REPORT==== 8-Sep-2010::09:28:26 === > ** Generic server rex terminating > ** Last message in was {call,mnesia_lib,get_node_number,[],<0.55.0>} > ** When Server state == {0,nil} > ** Reason for termination == > ** {system_limit,[{erlang,spawn_opt, > [{erlang,apply, > [#Fun,[]], > [monitor]}]}, > {erlang,spawn_monitor,1}, > {rpc,handle_call_call,6}, > {gen_server,handle_msg,5}, > {proc_lib,init_p_do_apply,3}]} > > =CRASH REPORT==== 8-Sep-2010::09:28:26 === > crasher: > initial call: rpc:init/1 > pid: <0.12.0> > registered_name: rex > exception exit: {system_limit, > [{erlang,spawn_opt, > [{erlang,apply, > [#Fun,[]], > [monitor]}]}, > {erlang,spawn_monitor,1}, > {rpc,handle_call_call,6}, > {gen_server,handle_msg,5}, > {proc_lib,init_p_do_apply,3}]} > in function gen_server:terminate/6 > ancestors: [kernel_sup,<0.10.0>] > messages: [] > links: [<0.11.0>] > dictionary: [] > trap_exit: true > status: running > heap_size: 377 > stack_size: 24 > reductions: 143 > neighbours: > > =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === > Supervisor: {local,kernel_sup} > Context: child_terminated > Reason: {system_limit, > [{erlang,spawn_opt, > [{erlang,apply, > [#Fun,[]], > [monitor]}]}, > {erlang,spawn_monitor,1}, > {rpc,handle_call_call,6}, > {gen_server,handle_msg,5}, > {proc_lib,init_p_do_apply,3}]} > Offender: [{pid,<0.12.0>}, > {name,rex}, > {mfargs,{rpc,start_link,[]}}, > {restart_type,permanent}, > {shutdown,2000}, > {child_type,worker}] > > > =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === > Supervisor: {local,kernel_sup} > Context: shutdown > Reason: reached_max_restart_intensity > Offender: [{pid,<0.12.0>}, > {name,rex}, > {mfargs,{rpc,start_link,[]}}, > {restart_type,permanent}, > {shutdown,2000}, > {child_type,worker}] > > > =ERROR REPORT==== 8-Sep-2010::09:28:26 === > Mnesia(igo@REDACTED): ** ERROR ** mnesia_recover got > unexpected info: {'EXIT', > > <0.84.0>, > > shutdown} -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From freza@REDACTED Wed Sep 8 19:07:38 2010 From: freza@REDACTED (Jachym Holecek) Date: Wed, 8 Sep 2010 18:07:38 +0100 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> Message-ID: <20100908170738.GA6576@hanele> # Musumeci, Antonio S 2010-09-08: > OS limits? This is the BEAM process limit. [...] Which you can adjust with '+P ' emulator flag to a sufficiently large value and be done with it, in the spirit of what Igor suggests. You're supposed to provide the runtime system with enough resources (be it OS or emulator settings) to handle the expected load. Dealing with this kind of errors "more gracefully" would be too much pain[1] or simply impossible[2], AFAIU. Regards, -- Jachym [1] Process table limit -- every single call to any variant of spawn() may fail this way. There's plenty of those. There's a similar limit on # of ETS tabs. [2] Out of memory -- if there's no memory, where do you get the memory to deal with OOM error? From Antonio.Musumeci@REDACTED Wed Sep 8 19:46:36 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Wed, 8 Sep 2010 13:46:36 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <20100908170738.GA6576@hanele> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> I'm sorry but that's ridiculous. Limits exist to *limit* not to crash. I see no reasons that rex should cause the VM to fail because it can't spawn a child to handle the rpc request. When a node is down it returns {badrpc,nodedown}. If a process can't be spawned shouldn't it return a similar error? At least be configured to not have the supervisor exit and therefore bring down the entire system? [1] That's not what is being argued. Obviously spawn will fail if there are too many processes. The problem is that rex exits when it happens which causes the entire system to fail. Does any modern OS suddenly reboot when the same type of limit is reached? Do they shut down when virtual memory is exhausted? No, they return errors and allow the developer to handle the issue as well as the situation allows. [2] The Linux kernel for example keeps a certain amount of memory for itself. Besides, just because you can't acquire some resource at T0 does not mean 1) you need the same resource to handle the error and/or 2) that it won't be available at time T1. Increasing the process count isn't a solution. It's a hack that pushes the problem out further. -----Original Message----- From: Jachym Holecek [mailto:freza@REDACTED] Sent: Wednesday, September 08, 2010 1:08 PM To: Musumeci, Antonio S (Enterprise Infrastructure) Cc: Igor Ribeiro Sucupira; erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM # Musumeci, Antonio S 2010-09-08: > OS limits? This is the BEAM process limit. [...] Which you can adjust with '+P ' emulator flag to a sufficiently large value and be done with it, in the spirit of what Igor suggests. You're supposed to provide the runtime system with enough resources (be it OS or emulator settings) to handle the expected load. Dealing with this kind of errors "more gracefully" would be too much pain[1] or simply impossible[2], AFAIU. Regards, -- Jachym [1] Process table limit -- every single call to any variant of spawn() may fail this way. There's plenty of those. There's a similar limit on # of ETS tabs. [2] Out of memory -- if there's no memory, where do you get the memory to deal with OOM error? From boris.muehmer@REDACTED Wed Sep 8 19:48:38 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Wed, 8 Sep 2010 19:48:38 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: References: Message-ID: I'm using the following code (test_command/0 is my entry point for the tests): %%% ================================================================ %%% test_command %%% ================================================================ test_command() -> MysqlCmd = "/usr/local/bin/mysql", Args = [ "--host=mysql-host" ,"--user=MySqlSuperUser" ,"--password=MySecretPassword" ,"-e" ,"SELECT user,password,host FROM mysql.user;" ], Result = execute_command(MysqlCmd, Args), Line = "----------------------------------------------------------------------------", io:format("~s~nResult:~n~p~n~s~n",[Line, Result, Line]), ok. %%% ================================================================ %%% execute_command %%% ================================================================ execute_command(Command, Arguments) when is_list(Command), is_list(Arguments) -> PortSettings = [ {args, Arguments} ,exit_status ,stderr_to_stdout ,{line, 1024} ], Port = erlang:open_port({spawn_executable, Command}, PortSettings), execute_command_loop(Port, []). %%% ================================================================ %%% execute_command_loop %%% ================================================================ execute_command_loop(Port, Data) -> receive {Port,{exit_status, ExitStatus}} -> {ExitStatus, lists:reverse(Data)}; {Port,{data, NewData}} -> execute_command_loop(Port, [NewData | Data]); Other -> io:format("Other:~p~n",[Other]), execute_command_loop(Port, Data) end. I'm not sure if this is "good" code... it works... also when errors occur (e.g. "mysql.users" instead of "mysql.user", or adding a non-existing column). Is it possible to circumvent the "lists:reverse/1" call in "execute_command_loop/2"??? - boris From boris.muehmer@REDACTED Wed Sep 8 19:50:25 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Wed, 8 Sep 2010 19:50:25 +0200 Subject: [erlang-questions] Erlang module for "controlling" (linux/unix) commands? In-Reply-To: References: Message-ID: 2010/9/8 Boris M?hmer > Is it possible to circumvent the "lists:reverse/1" call in "execute_command_loop/2"??? I know how to write it without the "lists:reverse/1", but is it "good" to do it? - boris From mark@REDACTED Wed Sep 8 20:00:16 2010 From: mark@REDACTED (Mark Scandariato) Date: Wed, 8 Sep 2010 14:00:16 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> Message-ID: Last time I looked, the process limit was there to allow the process table to be preallocated at startup. The default limit is 32768. It is not at all ridiculous to determine the needs of your application and increase the process limit to match. On Wed, Sep 8, 2010 at 1:46 PM, Musumeci, Antonio S < Antonio.Musumeci@REDACTED> wrote: > I'm sorry but that's ridiculous. Limits exist to *limit* not to crash. I > see no reasons that rex should cause the VM to fail because it can't spawn a > child to handle the rpc request. When a node is down it returns > {badrpc,nodedown}. If a process can't be spawned shouldn't it return a > similar error? At least be configured to not have the supervisor exit and > therefore bring down the entire system? > > [1] That's not what is being argued. Obviously spawn will fail if there are > too many processes. The problem is that rex exits when it happens which > causes the entire system to fail. Does any modern OS suddenly reboot when > the same type of limit is reached? Do they shut down when virtual memory is > exhausted? No, they return errors and allow the developer to handle the > issue as well as the situation allows. > > [2] The Linux kernel for example keeps a certain amount of memory for > itself. Besides, just because you can't acquire some resource at T0 does not > mean 1) you need the same resource to handle the error and/or 2) that it > won't be available at time T1. > > Increasing the process count isn't a solution. It's a hack that pushes the > problem out further. > > -----Original Message----- > From: Jachym Holecek [mailto:freza@REDACTED] > Sent: Wednesday, September 08, 2010 1:08 PM > To: Musumeci, Antonio S (Enterprise Infrastructure) > Cc: Igor Ribeiro Sucupira; erlang-questions@REDACTED > Subject: Re: [erlang-questions] System limit bringing down rex and the VM > > # Musumeci, Antonio S 2010-09-08: > > OS limits? This is the BEAM process limit. [...] > > Which you can adjust with '+P ' emulator flag to a sufficiently > large value and be done with it, in the spirit of what Igor suggests. > > You're supposed to provide the runtime system with enough resources (be it > OS or emulator settings) to handle the expected load. Dealing with this kind > of errors "more gracefully" would be too much pain[1] or simply > impossible[2], AFAIU. > > Regards, > -- Jachym > > [1] Process table limit -- every single call to any variant of spawn() may > fail > this way. There's plenty of those. There's a similar limit on # of ETS > tabs. > > [2] Out of memory -- if there's no memory, where do you get the memory to > deal > with OOM error? > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From Antonio.Musumeci@REDACTED Wed Sep 8 20:12:36 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Wed, 8 Sep 2010 14:12:36 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com><20100908170738.GA6576@hanele><01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> And what if the needs are dynamic in nature? Or if I *want* the limit and need to take some action when it occurs? There is no technical reason not to handle the error. It's purely a design decision or something that was overlooked. Limits are for limiting. Crashing is not an appropriate action to a definite and controllable problem. If you didn't care about whether or not something crashed there would be no limits at all. This is no different from a whole OS crashing because some process consumed too much memory or file handles. I shouldn't have to build my own spawn wrapper to keep track of the number of processes. The VM already does this. Besides, this problem couldn't be fully addressed that way. This is caused by mnesia reacting to some event causing mnesia_recover to send a rpc. I'm not in control of that. -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Mark Scandariato Sent: Wednesday, September 08, 2010 2:00 PM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM Last time I looked, the process limit was there to allow the process table to be preallocated at startup. The default limit is 32768. It is not at all ridiculous to determine the needs of your application and increase the process limit to match. On Wed, Sep 8, 2010 at 1:46 PM, Musumeci, Antonio S < Antonio.Musumeci@REDACTED> wrote: > I'm sorry but that's ridiculous. Limits exist to *limit* not to crash. > I see no reasons that rex should cause the VM to fail because it can't > spawn a child to handle the rpc request. When a node is down it > returns {badrpc,nodedown}. If a process can't be spawned shouldn't it > return a similar error? At least be configured to not have the > supervisor exit and therefore bring down the entire system? > > [1] That's not what is being argued. Obviously spawn will fail if > there are too many processes. The problem is that rex exits when it > happens which causes the entire system to fail. Does any modern OS > suddenly reboot when the same type of limit is reached? Do they shut > down when virtual memory is exhausted? No, they return errors and > allow the developer to handle the issue as well as the situation allows. > > [2] The Linux kernel for example keeps a certain amount of memory for > itself. Besides, just because you can't acquire some resource at T0 > does not mean 1) you need the same resource to handle the error and/or > 2) that it won't be available at time T1. > > Increasing the process count isn't a solution. It's a hack that pushes > the problem out further. > > -----Original Message----- > From: Jachym Holecek [mailto:freza@REDACTED] > Sent: Wednesday, September 08, 2010 1:08 PM > To: Musumeci, Antonio S (Enterprise Infrastructure) > Cc: Igor Ribeiro Sucupira; erlang-questions@REDACTED > Subject: Re: [erlang-questions] System limit bringing down rex and the > VM > > # Musumeci, Antonio S 2010-09-08: > > OS limits? This is the BEAM process limit. [...] > > Which you can adjust with '+P ' emulator flag to a > sufficiently large value and be done with it, in the spirit of what Igor suggests. > > You're supposed to provide the runtime system with enough resources > (be it OS or emulator settings) to handle the expected load. Dealing > with this kind of errors "more gracefully" would be too much pain[1] > or simply impossible[2], AFAIU. > > Regards, > -- Jachym > > [1] Process table limit -- every single call to any variant of spawn() > may fail > this way. There's plenty of those. There's a similar limit on # of > ETS tabs. > > [2] Out of memory -- if there's no memory, where do you get the memory > to deal > with OOM error? > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From jared.nance@REDACTED Wed Sep 8 20:36:34 2010 From: jared.nance@REDACTED (Jared Nance) Date: Wed, 8 Sep 2010 11:36:34 -0700 Subject: supervisor woes Message-ID: Hello List- Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} and so the supervisor for these workers is started with a child spec that looks like {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. Thanks for whatever help you can offer on this very frustrating issue. Jared N From adam.kocoloski@REDACTED Wed Sep 8 20:59:22 2010 From: adam.kocoloski@REDACTED (Adam Kocoloski) Date: Wed, 8 Sep 2010 11:59:22 -0700 Subject: [erlang-questions] supervisor woes In-Reply-To: References: Message-ID: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: > Hello List- > Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like > > {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} > > and so the supervisor for these workers is started with a child spec that looks like > > {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} > > and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: > > {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} > > it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. > > Thanks for whatever help you can offer on this very frustrating issue. > > Jared N Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? Adam From jared.nance@REDACTED Wed Sep 8 21:16:07 2010 From: jared.nance@REDACTED (Jared Nance) Date: Wed, 8 Sep 2010 12:16:07 -0700 Subject: [erlang-questions] supervisor woes In-Reply-To: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> References: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> Message-ID: <375BE91C-FB72-406F-9439-8E94F85CC5BE@gmail.com> Hi Adam- Here's the definition of ios_faux_card:start_link/1 start_link(CardSlot) when is_atom(CardSlot) -> gen_server:start_link({local,CardSlot}, ?MODULE, [CardSlot,2], []). Is this the problem? JN On Sep 8, 2010, at 11:59 AM, Adam Kocoloski wrote: > On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: > >> Hello List- >> Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like >> >> {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} >> >> and so the supervisor for these workers is started with a child spec that looks like >> >> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >> >> and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: >> >> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >> >> it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. >> >> Thanks for whatever help you can offer on this very frustrating issue. >> >> Jared N > > Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? > > Adam > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From jared.nance@REDACTED Wed Sep 8 22:21:27 2010 From: jared.nance@REDACTED (Jared Nance) Date: Wed, 8 Sep 2010 13:21:27 -0700 Subject: [erlang-questions] supervisor woes In-Reply-To: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> References: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> Message-ID: <72DD29DC-B389-4329-9225-8309F505DB8C@gmail.com> Also, as a point of clarification: If I start the ios_card_super manually with the multiple children, it works without a hitch - all of the ios_faux_cards get registered correctly and behave normally. It's only when I try to start the ios_card_super as a supervised node in a supervision tree that I have the problem. On Sep 8, 2010, at 11:59 AM, Adam Kocoloski wrote: > On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: > >> Hello List- >> Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like >> >> {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} >> >> and so the supervisor for these workers is started with a child spec that looks like >> >> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >> >> and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: >> >> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >> >> it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. >> >> Thanks for whatever help you can offer on this very frustrating issue. >> >> Jared N > > Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? > > Adam > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From mixolyde@REDACTED Wed Sep 8 22:21:22 2010 From: mixolyde@REDACTED (Brian Williams) Date: Wed, 8 Sep 2010 16:21:22 -0400 Subject: [erlang-questions] supervisor woes In-Reply-To: <375BE91C-FB72-406F-9439-8E94F85CC5BE@gmail.com> References: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> <375BE91C-FB72-406F-9439-8E94F85CC5BE@gmail.com> Message-ID: This may help you get more detailed error reports, but I'm not sure it works if you're running outside of an application. application:load(sasl). application:start(sasl). * generate your error * I usually get better error messages when this is running. On Wed, Sep 8, 2010 at 3:16 PM, Jared Nance wrote: > Hi Adam- > Here's the definition of ios_faux_card:start_link/1 > > start_link(CardSlot) when is_atom(CardSlot) -> > ? ? ? ?gen_server:start_link({local,CardSlot}, ?MODULE, [CardSlot,2], []). > > Is this the problem? > > JN > > On Sep 8, 2010, at 11:59 AM, Adam Kocoloski wrote: > >> On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: >> >>> Hello List- >>> Thanks in advance for whatever help you can offer. ?I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. ?The problem I am having is with one of my supervisors which manages 4 worker nodes. ?Each node is a gen_server that uses the same callback module, but with different arguments to start_link. ?The childspec for the worker nodes looks like >>> >>> {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} >>> >>> and so the supervisor for these workers is started with a child spec that looks like >>> >>> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >>> >>> and that works just fine. ?the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. ?if i try starting it with this child spec instead: >>> >>> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >>> >>> it refuses to start. ?what's more, the only response I get out of it is {error, shutdown}. ?I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. >>> >>> Thanks for whatever help you can offer on this very frustrating issue. >>> >>> Jared N >> >> Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? >> >> Adam >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Brian E. Williams mixolyde@REDACTED http://www.techhouse.us/wordpress-mu/brianw "Never attribute to malice that which can be adequately explained by stupidity." - Hanlon's Razor From anthonym@REDACTED Wed Sep 8 22:43:50 2010 From: anthonym@REDACTED (Anthony Molinaro) Date: Wed, 8 Sep 2010 13:43:50 -0700 Subject: [erlang-questions] supervisor woes In-Reply-To: <72DD29DC-B389-4329-9225-8309F505DB8C@gmail.com> References: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> <72DD29DC-B389-4329-9225-8309F505DB8C@gmail.com> Message-ID: <20100908204350.GA34876@alumni.caltech.edu> What does the child spec for the supervisor calling your supervisor look like? Maybe you are calling your supervisor a worker in that spec? -Anthony On Wed, Sep 08, 2010 at 01:21:27PM -0700, Jared Nance wrote: > Also, as a point of clarification: If I start the ios_card_super manually with the multiple children, it works without a hitch - all of the ios_faux_cards get registered correctly and behave normally. It's only when I try to start the ios_card_super as a supervised node in a supervision tree that I have the problem. > > On Sep 8, 2010, at 11:59 AM, Adam Kocoloski wrote: > > > On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: > > > >> Hello List- > >> Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like > >> > >> {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} > >> > >> and so the supervisor for these workers is started with a child spec that looks like > >> > >> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} > >> > >> and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: > >> > >> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} > >> > >> it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. > >> > >> Thanks for whatever help you can offer on this very frustrating issue. > >> > >> Jared N > > > > Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? > > > > Adam > > > > > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- ------------------------------------------------------------------------ Anthony Molinaro From jared.nance@REDACTED Wed Sep 8 22:50:04 2010 From: jared.nance@REDACTED (Jared Nance) Date: Wed, 8 Sep 2010 13:50:04 -0700 Subject: [erlang-questions] supervisor woes In-Reply-To: <20100908204350.GA34876@alumni.caltech.edu> References: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> <72DD29DC-B389-4329-9225-8309F505DB8C@gmail.com> <20100908204350.GA34876@alumni.caltech.edu> Message-ID: <982C53AE-49BC-40A2-9F88-F98CCC5ADD1C@gmail.com> Thanks for the sasl hint, it's definitely nice to have slightly more verbose error messages. The supervisor calling my supervisor is the top level supervisor, started via an application behavior that calls ios_overlord:start_link/1, which in turn invokes supervisor:start_link/3. On Sep 8, 2010, at 1:43 PM, Anthony Molinaro wrote: > > What does the child spec for the supervisor calling your supervisor look like? > Maybe you are calling your supervisor a worker in that spec? > > -Anthony > > On Wed, Sep 08, 2010 at 01:21:27PM -0700, Jared Nance wrote: >> Also, as a point of clarification: If I start the ios_card_super manually with the multiple children, it works without a hitch - all of the ios_faux_cards get registered correctly and behave normally. It's only when I try to start the ios_card_super as a supervised node in a supervision tree that I have the problem. >> >> On Sep 8, 2010, at 11:59 AM, Adam Kocoloski wrote: >> >>> On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: >>> >>>> Hello List- >>>> Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like >>>> >>>> {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} >>>> >>>> and so the supervisor for these workers is started with a child spec that looks like >>>> >>>> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >>>> >>>> and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: >>>> >>>> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >>>> >>>> it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. >>>> >>>> Thanks for whatever help you can offer on this very frustrating issue. >>>> >>>> Jared N >>> >>> Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? >>> >>> Adam >>> >>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > -- > ------------------------------------------------------------------------ > Anthony Molinaro From freza@REDACTED Wed Sep 8 23:03:18 2010 From: freza@REDACTED (Jachym Holecek) Date: Wed, 8 Sep 2010 22:03:18 +0100 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> Message-ID: <20100908210318.GA978@hanele> # Musumeci, Antonio S 2010-09-08: > I'm sorry but that's ridiculous. It's not. > Limits exist to *limit* not to crash. Crash is a kind of limit and very efficient at that, isn't it? > I see no reasons that rex should cause the VM to fail [...] You asked a technical question and received more than one valid answer to it in very short time. Thus your problem is solved and this discussion is over. Whether you accept the underlying reasoning or not on a personal level is irrelevant, unless you can show the thing that disturbs you is an actual engineering problem. Have a nice evening, -- Jachym From chandrashekhar.mullaparthi@REDACTED Wed Sep 8 23:09:54 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 8 Sep 2010 22:09:54 +0100 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> Message-ID: On 8 September 2010 19:12, Musumeci, Antonio S < Antonio.Musumeci@REDACTED> wrote: > And what if the needs are dynamic in nature? Or if I *want* the limit and > need to take some action when it occurs? There is no technical reason not to > handle the error. It's purely a design decision or something that was > overlooked. > Limits are for limiting. Crashing is not an appropriate action to a definite > and controllable problem. If you didn't care about whether or not something > crashed there would be no limits at all. This is no different from a whole > OS crashing because some process consumed too much memory or file handles. > This is the way erlang has been designed. If it hits a system limit, it crashes. Simply because it cannot cope. By not crashing, and returning error codes, you get defensive programming which makes programs harder to read/write/test. As far as BEAM is concerned, the ability to spawn a process is a fundamental requirement. If that cannot be met, all bets are off. I suspect the resource exhaustion is being caused by your own code, and a bit of overload control, coupled with configuration of BEAM as pointed out by others should solve the problem. > I shouldn't have to build my own spawn wrapper to keep track of the number > of processes. The VM already does this. Besides, this problem couldn't be > fully addressed that way. You don't have to. I suspect you need to do some sort of load regulation in your system. > This is caused by mnesia reacting to some event causing mnesia_recover to > send a rpc. I'm not in control of that. > The error message you see about mnesia_recover is the "effect", not the "cause". cheers Chandru From chandrashekhar.mullaparthi@REDACTED Wed Sep 8 23:28:42 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 8 Sep 2010 22:28:42 +0100 Subject: [erlang-questions] How to design credit control application? In-Reply-To: <20100907112807.8843F763ED4@fwb.poczta.interia.pl> References: <20100907112807.8843F763ED4@fwb.poczta.interia.pl> Message-ID: On 7 September 2010 12:28, Tomasz Pastuch wrote: > Dear all, > > I?m quite new in erlang language, but I?m fall in love with it after > reading Joe?s book. > Now I?m trying to create a prototype of application just to see if erlang > is suitable for our purpose. Let?s imagine an telco application for soft > real-time credit control. From a stack (i.e. diameter, radius) we receive > the information; > - open a session for user > - reserve money/debit money from "balance" > - close session for user > > The sessions can last for hours, but every 2minutes we have another > request. Each credit control request reserves new money and debit the money > from the previous (2m ago) reservation. Each user can have many concurrent > sessions (>1000). The total number of concurrent session will be max 300K. > > I?m thinking about designing such system in erlang. We want to have fully > redundant system - we cannot loose customer balances. > We have a very similar system running here, but we handle only the DIAMETER protocol using Erlang. We have a centralised prepay charging system, which handles all funds. This prepay charging system operates in a master/slave configuration. The slave will not handle any monetary operations as long as the master is alive. The master and slave use some sort of clustering mechanism so that they both can't serve traffic at the same time. You are better off following a similar model. Split out the database which handles customer balance, and have your protocol operations in separate nodes. This way, errors in your protocol handling nodes will not affect your system which handles customer balance. > From the beginning I was thinking to model that situation using : > - customer session -> spawn many erlang process at each open session, use > mnesia replicated > table for storing reserved amount for each session > - customer balances -> use mnesia replicated table > > The problems I have encountered: > > 1. Mnesia transactions cannot spread over two tables, so I can loose the > reserve transaction (If the node fails after decreasing the value from > balances and begin storing new reservation record). If I store the > reservation inside the balance record, then probably I have problems with > many (> 1000) concurrent sessions for each balance (the record will be > simply big and the operations of updating the balance can be slower). > Yes you can. You can have a transaction cover any number of tables, and the operation is guaranteed to be atomic. Either all changes are done, or none are. There is no reason to have all reservations in the balance record. Reservations can be in their own table, probably having a reservation id as the primary index, and the customer id as the secondary index. > > 2. If we use processes for modeling user sessions then how to replicate > those processes to the replica ? Maybe the better solution is to insert only > a record into mnesia for each session, but how then I can process those > sessions parallel (when the new request came)? > Sorry, I don't understand what you're saying here. Can you please clarify? cheers Chandru From jared.nance@REDACTED Thu Sep 9 00:17:33 2010 From: jared.nance@REDACTED (Jared Nance) Date: Wed, 8 Sep 2010 15:17:33 -0700 Subject: [erlang-questions] supervisor woes In-Reply-To: <20100908204350.GA34876@alumni.caltech.edu> References: <6DAD465F-6C87-4E6D-979A-6D369465B405@gmail.com> <72DD29DC-B389-4329-9225-8309F505DB8C@gmail.com> <20100908204350.GA34876@alumni.caltech.edu> Message-ID: solved. Basically I hadn't realized that if in the child spec, the argument to child:start_link was a list of length N, child:start_link/N would be called. This resulted in {'EXIT', undef, {....}} coming back at me once I had sasl running, and helped me track it down to this. Thanks for all the help. JN On Sep 8, 2010, at 1:43 PM, Anthony Molinaro wrote: > > What does the child spec for the supervisor calling your supervisor look like? > Maybe you are calling your supervisor a worker in that spec? > > -Anthony > > On Wed, Sep 08, 2010 at 01:21:27PM -0700, Jared Nance wrote: >> Also, as a point of clarification: If I start the ios_card_super manually with the multiple children, it works without a hitch - all of the ios_faux_cards get registered correctly and behave normally. It's only when I try to start the ios_card_super as a supervised node in a supervision tree that I have the problem. >> >> On Sep 8, 2010, at 11:59 AM, Adam Kocoloski wrote: >> >>> On Sep 8, 2010, at 11:36 AM, Jared Nance wrote: >>> >>>> Hello List- >>>> Thanks in advance for whatever help you can offer. I'm putting together a supervision tree which is rather simple - at the moment, there is a supervisor that manages several other supervisors, and those supervisors each manage only worker nodes. The problem I am having is with one of my supervisors which manages 4 worker nodes. Each node is a gen_server that uses the same callback module, but with different arguments to start_link. The childspec for the worker nodes looks like >>>> >>>> {cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]} >>>> >>>> and so the supervisor for these workers is started with a child spec that looks like >>>> >>>> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >>>> >>>> and that works just fine. the problem arises when i try to add a second (or third, or fourth) gen_server to the list of processes that the ios_card_super is supervising. if i try starting it with this child spec instead: >>>> >>>> {{one_for_one, 5, 10}, [{ios_card_super, {ios_card_super, start_link, [{cardA, {ios_faux_card, start_link, [cardA]}, permanent, brutal_kill, worker, [ios_faux_card]},{cardB,{ios_faux_card, start_link, [cardB]}, permanent, brutal_kill, worker, [ios_faux_card]}], permanent, infinity, supervisor, [ios_card_super]} >>>> >>>> it refuses to start. what's more, the only response I get out of it is {error, shutdown}. I have put io:format debug statements in the ios_faux_card module and it's clear that start_link/1is never being called when there are 2+ workers to supervise. >>>> >>>> Thanks for whatever help you can offer on this very frustrating issue. >>>> >>>> Jared N >>> >>> Hi Jared, you wouldn't happen to be trying to register the process in ios_faux_card, would you? >>> >>> Adam >>> >>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > -- > ------------------------------------------------------------------------ > Anthony Molinaro From bile@REDACTED Thu Sep 9 00:34:52 2010 From: bile@REDACTED (bile@REDACTED) Date: Wed, 8 Sep 2010 18:34:52 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> Message-ID: <20100908183452.0c1df9d4@otis> > This is the way erlang has been designed. If it hits a system limit, > it crashes. Simply because it cannot cope. By not crashing, and > returning error codes, you get defensive programming which makes > programs harder to read/write/test. As far as BEAM is concerned, the > ability to spawn a process is a fundamental requirement. If that > cannot be met, all bets are off. I suspect the resource exhaustion is > being caused by your own code, and a bit of overload control, coupled > with configuration of BEAM as pointed out by others should solve the > problem. How many other limits cause the platform to shit the bed? I suspect few. There is a massive difference between the entire platform collapsing and RPC not working / restarting. If spawning of processes is so fundamental why do the core processes fit into the process limit? Linux as mentioned before prevents userland from wrecking things by safeguarding some amount of RAM for itself. Couldn't BEAM do the same? Why doesn't it auto shutdown if the limit is hit? Why no warnings from the system? Why is it triggered by rex? A minor component of the base process tree. Those who defend this behavior are not consistent. The behavior of the core processes are not consistent. Just look at the code. There is no reason to take control away from the developer. Especially when it means the entire platform will collapse from underneath them for something entirely controllable. > > > I shouldn't have to build my own spawn wrapper to keep track of the > > number of processes. The VM already does this. Besides, this > > problem couldn't be fully addressed that way. > > > You don't have to. I suspect you need to do some sort of load > regulation in your system. Load regulation? My system is designed to support arbitrary process creation. I was maxing out the processes as a scale test. If for some reason it can't spawn new processes then I want control over what happens next. Rex and the supervisor's behavior takes that control from me. At best I can poll the process count and warn that the system will soon fail but am powerless to do anything about it. The failure of a non-essential component of the system should not cause the VM to fail just like a bad process in an OS should cause it to halt. Could one of you please explain to me how that analogy is incorrect? > > > This is caused by mnesia reacting to some event causing > > mnesia_recover to send a rpc. I'm not in control of that. > > > > The error message you see about mnesia_recover is the "effect", not > the "cause". The mnesia_recover rpc call is the catalyst for the failure. It's issuing the RPC command. The error is caused by the rpc failure. From igorrs@REDACTED Thu Sep 9 05:13:35 2010 From: igorrs@REDACTED (Igor Ribeiro Sucupira) Date: Thu, 9 Sep 2010 00:13:35 -0300 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> Message-ID: I think you noticed I wasn't trying to justify the crash: I was just being realistic and telling you to not rely on gracious failures. I personally set limits in a way that, if the VM crashes because of them, it's probably because something was already very wrong and my application wouldn't be able to work properly anyway. But I understand this is not acceptable in some scenarios. You might want to limit the resources used by your application (for example, because there are other applications running on the same server) and keep it running with limited functionality if some limit is reached. I remember having written at least one C++ application that keeps doing what it can if it's not possible to spawn a thread. Just telling you it might be an ambitious goal. Igor. On Wed, Sep 8, 2010 at 1:22 PM, Musumeci, Antonio S wrote: > > OS limits? This is the BEAM process limit. Rex doesn't gracefully handle errors which in this case is triggered by mnesia_recover. I don't see the point in rex not handling it more gracefully either by > returning the error to the caller or restarting. Rex may be a base process but it's not that important IMO to cause the VM to shutdown. I'd rather not have to implement OOM killer like behavior or some > other watchdog process just so my VM doesn't crash due to hitting otherwise arbitrary max process limits. > > -----Original Message----- > From: Igor Ribeiro Sucupira [mailto:igorrs@REDACTED] > Sent: Wednesday, September 08, 2010 11:29 AM > To: Musumeci, Antonio S (Enterprise Infrastructure) > Cc: erlang-questions@REDACTED > Subject: Re: [erlang-questions] System limit bringing down rex and the VM > > I know this is obvious, but the first step should be adjusting your OS limits so that your system, under expected conditions, never fails because of them. > I have seen a couple of funny behaviours when exhausting system limits in the QA environment, so I wouldn't count with gracious failures from Erlang/Mnesia/etc. > > Good luck. > Igor. > > On Wed, Sep 8, 2010 at 10:35 AM, Musumeci, Antonio S wrote: > > Is this an expected behavior? Shouldn't there be a way to more graciously handle this type of thing? > > > > =ERROR REPORT==== 8-Sep-2010::09:28:26 === > > ** Generic server rex terminating > > ** Last message in was {call,mnesia_lib,get_node_number,[],<0.55.0>} > > ** When Server state == {0,nil} > > ** Reason for termination == > > ** {system_limit,[{erlang,spawn_opt, > > ? ? ? ? ? ? ? ? ? ? ? ? ?[{erlang,apply, > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [#Fun,[]], > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [monitor]}]}, > > ? ? ? ? ? ? ? ? ?{erlang,spawn_monitor,1}, > > ? ? ? ? ? ? ? ? ?{rpc,handle_call_call,6}, > > ? ? ? ? ? ? ? ? ?{gen_server,handle_msg,5}, > > ? ? ? ? ? ? ? ? ?{proc_lib,init_p_do_apply,3}]} > > > > =CRASH REPORT==== 8-Sep-2010::09:28:26 === > > ?crasher: > > ? ?initial call: rpc:init/1 > > ? ?pid: <0.12.0> > > ? ?registered_name: rex > > ? ?exception exit: {system_limit, > > ? ? ? ? ? ? ? ? ? ? ? ?[{erlang,spawn_opt, > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{erlang,apply, > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[#Fun,[]], > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[monitor]}]}, > > ? ? ? ? ? ? ? ? ? ? ? ? {erlang,spawn_monitor,1}, > > ? ? ? ? ? ? ? ? ? ? ? ? {rpc,handle_call_call,6}, > > ? ? ? ? ? ? ? ? ? ? ? ? {gen_server,handle_msg,5}, > > ? ? ? ? ? ? ? ? ? ? ? ? {proc_lib,init_p_do_apply,3}]} > > ? ? ?in function ?gen_server:terminate/6 > > ? ?ancestors: [kernel_sup,<0.10.0>] > > ? ?messages: [] > > ? ?links: [<0.11.0>] > > ? ?dictionary: [] > > ? ?trap_exit: true > > ? ?status: running > > ? ?heap_size: 377 > > ? ?stack_size: 24 > > ? ?reductions: 143 > > ?neighbours: > > > > =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === > > ? ? Supervisor: {local,kernel_sup} > > ? ? Context: ? ?child_terminated > > ? ? Reason: ? ? {system_limit, > > ? ? ? ? ? ? ? ? ? ? [{erlang,spawn_opt, > > ? ? ? ? ? ? ? ? ? ? ? ? ?[{erlang,apply, > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [#Fun,[]], > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [monitor]}]}, > > ? ? ? ? ? ? ? ? ? ? ?{erlang,spawn_monitor,1}, > > ? ? ? ? ? ? ? ? ? ? ?{rpc,handle_call_call,6}, > > ? ? ? ? ? ? ? ? ? ? ?{gen_server,handle_msg,5}, > > ? ? ? ? ? ? ? ? ? ? ?{proc_lib,init_p_do_apply,3}]} > > ? ? Offender: ? [{pid,<0.12.0>}, > > ? ? ? ? ? ? ? ? ?{name,rex}, > > ? ? ? ? ? ? ? ? ?{mfargs,{rpc,start_link,[]}}, > > ? ? ? ? ? ? ? ? ?{restart_type,permanent}, > > ? ? ? ? ? ? ? ? ?{shutdown,2000}, > > ? ? ? ? ? ? ? ? ?{child_type,worker}] > > > > > > =SUPERVISOR REPORT==== 8-Sep-2010::09:28:26 === > > ? ? Supervisor: {local,kernel_sup} > > ? ? Context: ? ?shutdown > > ? ? Reason: ? ? reached_max_restart_intensity > > ? ? Offender: ? [{pid,<0.12.0>}, > > ? ? ? ? ? ? ? ? ?{name,rex}, > > ? ? ? ? ? ? ? ? ?{mfargs,{rpc,start_link,[]}}, > > ? ? ? ? ? ? ? ? ?{restart_type,permanent}, > > ? ? ? ? ? ? ? ? ?{shutdown,2000}, > > ? ? ? ? ? ? ? ? ?{child_type,worker}] > > > > > > =ERROR REPORT==== 8-Sep-2010::09:28:26 === > > Mnesia(igo@REDACTED): ** ERROR ** mnesia_recover got > > unexpected info: {'EXIT', > > > > <0.84.0>, > > > > shutdown} > > -- > "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. -- "The secret of joy in work is contained in one word - excellence. To know how to do something well is to enjoy it." - Pearl S. Buck. From ok@REDACTED Thu Sep 9 05:17:03 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 9 Sep 2010 15:17:03 +1200 Subject: 14A wants 'fop', what/where? Message-ID: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> I'm installing R14A. documentation : fop is missing. Using fakefop to generate placeholder PDF files. This is the first Erlang installation I've done where this came up. Presumably fop is a processor for XML-FO. Where do I get it? From rzezeski@REDACTED Thu Sep 9 05:34:46 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Wed, 8 Sep 2010 23:34:46 -0400 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: On Wed, Sep 8, 2010 at 11:17 PM, Richard O'Keefe wrote: > I'm installing R14A. > > documentation : > fop is missing. > Using fakefop to generate placeholder PDF files. > > This is the first Erlang installation I've done where this came up. > Presumably fop is a processor for XML-FO. Where do I get it? > > > I think this is only needed if you want to locally generate the documentation as PDF? I could be wrong, but I'm fairly certain it's not needed for a working VM. Of course, maybe you already know that. Straight from INSTALL.md #### Building Documentation #### * `xsltproc` -- XSLT processor. A tool for applying XSLT stylesheets to XML documents. Can be downloaded from . * `fop` -- Apache FOP print formatter (requires Java). Can be downloaded from . If you're on OSX or Linux you should be able to use your favorite package manager to install. -Ryan From ok@REDACTED Thu Sep 9 05:58:32 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 9 Sep 2010 15:58:32 +1200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <002b01cb4f5b$eba94b70$c2fbe250$@com> References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> <002b01cb4f5b$eba94b70$c2fbe250$@com> Message-ID: <098DD2F8-6311-43E3-946E-519C81089A87@cs.otago.ac.nz> On Sep 9, 2010, at 1:44 AM, David Mercer wrote: > I wonder if the equality operator should even be applicable to floats. > After all, float F1 really represents a number between F1 - epsilon and F2 + > epsilon. This is a popular misconception. There is no fuzziness in the number that a floating-point number represents. The fuzziness is in the *operations*. IEEE defines the floating point operations in terms of applying a dynamically selected rounding algorithm to the infinitely precise result of a mathematical operation on unambiguous inputs. In particular, note that floating point add, subtract, multiply, and compare are *exact* for integers expressed as floating point numbers, provided the results are not too big. Before 64-bit integers were readily available, I've used IEEE doubles to get 53-bit integers. In any case, we _can't_ drop term equality from Erlang, because it's an essential part of pattern matching. Arithmetic equality, maybe. From ok@REDACTED Thu Sep 9 06:33:56 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 9 Sep 2010 16:33:56 +1200 Subject: erlang:max/2 and erlang:min/2 Message-ID: I see that max/2 and min/2 are now in Erlang. You would expect them to satisfy [min(X,Y),max(X,Y)] is a permutation of [X,Y] max(X,Y) is max(Y,X) min(X,Y) is min(Y,X) amongst other things. When X and Y are distinct terms that are equal as numbers, max(X,Y) and min(X,Y) both return X. sort3([A,B,C]) -> [U,V] = sort2([A,B]), [W,Z] = sort2([V,C]), [X,Y] = sort2([U,W]), [X,Y,Z]. sort2([A,B]) -> [min(A,B),max(A,B)]. You'd expect this to sort. But it doesn't: > sort3:sort3([0,0.0,-0.0]). [0,0,0] It really is very confusing for it to make a difference whether you write max(X, 0) or max(0, X), but in R14A it does. The *really* big problem here is that max/2 and min/2 are (otherwise) based on *term* comparison, and there *can't* be two identical but distinguishable terms. In order to get a total order on numbers, it seems necessary to rule that -0.0 < 0 < 0.0 and this extends to a general rule that if X and Y are numbers that are numerically equal but of different types, the floating point one is smaller if its sign bit is set, otherwise the floating point one is bigger. The root of the problem is trying to make one ordering serve as both a numeric ordering and a term ordering. From michael.eugene.turner@REDACTED Thu Sep 9 07:42:01 2010 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Thu, 9 Sep 2010 14:42:01 +0900 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: It's unfortunate that Erlang installation still has these dependencies, when you consider that xmerl offers something like XSLT functionality, and (last I checked), there's something called NErlGuten that might substitute entirely for FOP. I suppose it's only a minor annoyance. OTOH, there's this theory that PHP won out over a huge field of server-side scripting languages in the late 90s (some of them probably better) because it was the easiest to install. -michael turner On Thu, Sep 9, 2010 at 12:34 PM, Ryan Zezeski wrote: > On Wed, Sep 8, 2010 at 11:17 PM, Richard O'Keefe > wrote: > > > I'm installing R14A. > > > > documentation : > > fop is missing. > > Using fakefop to generate placeholder PDF files. > > > > This is the first Erlang installation I've done where this came up. > > Presumably fop is a processor for XML-FO. Where do I get it? > > > > > > > I think this is only needed if you want to locally generate the > documentation as PDF? I could be wrong, but I'm fairly certain it's not > needed for a working VM. Of course, maybe you already know that. > > Straight from INSTALL.md > > #### Building Documentation #### > > * `xsltproc` -- XSLT processor. A tool for applying XSLT stylesheets > to XML documents. Can be downloaded from > . > * `fop` -- Apache FOP print formatter (requires Java). Can be downloaded > from . > > If you're on OSX or Linux you should be able to use your favorite package > manager to install. > > -Ryan > From max.lapshin@REDACTED Thu Sep 9 08:05:57 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 9 Sep 2010 10:05:57 +0400 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: On Thu, Sep 9, 2010 at 9:42 AM, Michael Turner wrote: > this theory that PHP won out over a huge field of server-side scripting > languages in the late 90s (some of them probably better) because it was the > easiest to install. > Erlang in installable almost anywhere without any fop. It is optional dependency From ulf.wiger@REDACTED Thu Sep 9 11:14:33 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Thu, 09 Sep 2010 11:14:33 +0200 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <20100908183452.0c1df9d4@otis> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> Message-ID: <4C88A579.6000609@erlang-solutions.com> On 09/09/2010 12:34 AM, bile@REDACTED wrote: > > How many other limits cause the platform to shit the bed? I suspect > few. Actually, hitting the system limits themselves will not cause the VM to crash. OOM is the only exception I can think of right now. Trying to spawn a process when the max number of processes has been reached will simply raise an exception. However, some (most) application code will not include to cope with the situation that you can't spawn a process, so most likely, applications will come crashing down when this happens. Which one goes first is mainly up to chance. Erlang is a concurrency-oriented language. The spawn() function is about as fundamental as new() in OO languages. In most other languages, you treat the spawning of processes as something scary that you don't want to do unless you absolutely have to. Also, the limits are usually pretty low. In Erlang, you can raise the limit to > 200 million processes, if you have enough memory for it. The default limit is fairly low (32,000) mainly for historical reasons, but also because it makes sense to keep memory footprint low by default, and 32K processes is plenty enough for most uses. > Those who defend this behavior are not consistent. The behavior > of the core processes are not consistent. Just look at the code. It would be better if you mention specific instances rather than asking people to "just look at the code". OTOH, we can probably stipulate that the code base is inconsistent in many ways.... In general, application code does not cope with system limits being exhausted. This is in line with Erlang's "let it crash" philosophy as well as the fact that if you want true robustness in the kind of products Erlang was designed for, you have to have a redundant setup anyway. The thing about redundancy is that it works best if the failing side fails quickly and distinctly, rather than trying in vain to correct the problem locally. This is the essence of "fail-fast" programming. Although this is not 100% consistently implemented in Erlang/OTP either, you might want to keep in mind that Erlang has been breaking new ground in this respect, and most of the people who've worked on OTP components over the years were originally steeped in the same programming mindset as everyone else. ;-) >>> I shouldn't have to build my own spawn wrapper to keep track of the >>> number of processes. The VM already does this. Besides, this >>> problem couldn't be fully addressed that way. >> >> You don't have to. I suspect you need to do some sort of load >> regulation in your system. > > Load regulation? My system is designed to support arbitrary process > creation. I was maxing out the processes as a scale test. If for some > reason it can't spawn new processes then I want control over what > happens next. Rex and the supervisor's behavior takes that control > from me. At best I can poll the process count and warn that the system > will soon fail but am powerless to do anything about it. With any programming language or operating environment, you have the responsibility as a developer to understand and respect the fundamental assumptions made when developing the environment. If your requirements don't match well enough, it might be better to find another language/environment that fits your problem better. You seem to be saying that you shouldn't have to worry about the user of your application throwing more work at you than the system is capable of handling? This may be a valid requirement in some domains, but Erlang is fundamentally a language for developing messaging systems, which have to cope with overload situations (including Denial-of-Service attacks) in a structured way. When subjected to a DoS attack, you typically don't just want to accept the challenge and likely die honorably as a result. The normal way to do that is to push back, or shed load so that it doesn't overwhelm the core components in your system. You might compare this with the recurring discussions about active vs passive sockets. Sockets in POSIX are by design passive, as you have to explicitly read data from the buffer, but in Erlang, the default is that the VM empties the buffer and delivers the data to the socket owner asynchronously. I think this was the wrong default, and the recommendation is to use passive, or {active,once} to avoid being swamped by input from the network. This is in line with the idea of not accepting more work than you can cope with. > The failure of > a non-essential component of the system should not cause the VM to fail > just like a bad process in an OS should cause it to halt. Could one > of you please explain to me how that analogy is incorrect? RPC is not a non-essential component, especially not to mnesia. Mnesia assumes that rpc will work, unless something really bad has happened. The recommended behaviour when something really bad happens in Erlang is to die and let the rest of the system take over. BR, Ulf W From bile@REDACTED Thu Sep 9 12:13:08 2010 From: bile@REDACTED (bile@REDACTED) Date: Thu, 9 Sep 2010 06:13:08 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> Message-ID: <20100909061308.585d6089@otis> On Wed, 8 Sep 2010 22:42:36 -0500 Paul Fisher wrote: > > There is no sane thing that the vm could do in the face of such > process usage. Erlang crashes so that you can figure out how to > correct the design flaw, by looking at the dump. > That's simply false. The sane thing to do is to return an error so the developer may attempt to handle the situation. There is no flaw. Hitting the max process limit is not a guaranteed fatal situation and I should not be kept from attempting to respond it it. The fact that this happens by chance due to rex and the supervisor's setup rather than built into the VM totally blows your argument out of the water. Erlang doesn't crash in the normal sense. It's not a true fatal error. It could be triggered by calling rpc:stop(). If it were an actual design decision it should be clearly documented and part of the base VM behavior... not a side effect of rex's design. From chandrashekhar.mullaparthi@REDACTED Thu Sep 9 12:15:38 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Thu, 9 Sep 2010 11:15:38 +0100 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <20100908183452.0c1df9d4@otis> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> Message-ID: On 8 September 2010 23:34, wrote: > > How many other limits cause the platform to shit the bed? I suspect > few. There is a massive difference between the entire platform > collapsing and RPC not working / restarting. If spawning of processes > is so fundamental why do the core processes fit into the process > limit? Linux as mentioned before prevents userland from wrecking things > by safeguarding some amount of RAM for itself. Couldn't BEAM do the > same? Why doesn't it auto shutdown if the limit is hit? Why no warnings > from the system? Why is it triggered by rex? A minor component of the > base process tree. > As Ulf pointed out, beam does throw an exception. It is indeed the rex process which decided to give up. What I meant to say was that almost all applications written in erlang assume they are operating within the system limits. In the case of rex, it decides to die if it can't spawn a process. rex is only active if you are using distributed erlang. That is a design decision, and it is a valid one - atleast for those of us who use it a lot in real world situations. > There is no reason to take control away from the developer. Especially > when it means the entire platform will collapse from underneath them > for something entirely controllable. > > It is a trade off. You have complete control over what happens in the system when you program in C. Doesn't necessarily mean that is the best choice. > > > The error message you see about mnesia_recover is the "effect", not > > the "cause". > > The mnesia_recover rpc call is the catalyst for the failure. It's > issuing the RPC command. The error is caused by the rpc failure. > You are wrong. Infact if you look closely at it, mnesia is trying to make an rpc call when it was trying to dump core, which means it was already dying at that point. You need to dig deeper to find the real cause. Chandru From jesper.louis.andersen@REDACTED Thu Sep 9 13:34:11 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 9 Sep 2010 13:34:11 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: <098DD2F8-6311-43E3-946E-519C81089A87@cs.otago.ac.nz> References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> <002b01cb4f5b$eba94b70$c2fbe250$@com> <098DD2F8-6311-43E3-946E-519C81089A87@cs.otago.ac.nz> Message-ID: On Thu, Sep 9, 2010 at 5:58 AM, Richard O'Keefe wrote: > > In any case, we _can't_ drop term equality from Erlang, because it's an > essential part of pattern matching. ?Arithmetic equality, maybe. > Fun aside relating to this point: Standard ML defines specific types as eqtypes or types which can be compared for equality. The 'real' type of ML is not an eqtype because of the above mentioned trouble: comparing two reals (IEEE FPs) for equality is almost often wrong. But as Richard writes, this has ramifications for term equality. A term is an eqtype iff all its primitive types are eqtypes and all composite types are eqtypes. That is, you can have terms which you cannot compare for equality with the '=' operator. For composite values, this does not rule out partial pattern matches on only some components however. The MosML implementation of SML get this wrong and allows reals to be compared for equality. This also means that programs which compile in MosML does not compile in other SML implementations. It has bitten me more than once. -- J. From Antonio.Musumeci@REDACTED Thu Sep 9 14:15:15 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Thu, 9 Sep 2010 08:15:15 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <4C88A579.6000609@erlang-solutions.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com><20100908170738.GA6576@hanele><01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <4C88A579.6000609@erlang-solutions.com> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DB50@NYWEXMBX2129.msad.ms.com> I understand your points completely... however, there is certainly a difference from having an erlang process die and allowing it's peers to handle the cleanup and having the erlang vm die. The vm has no peer in that way. Yes Mnesia needs RPC... but so do a lot of things and if the pattern is to be followed that you die and allow the peers to respond... that's not what is happening here. Rex dies and brings the world down with it. Mnesia is unable to respond to the issue. The mnesia code shows that it is prepared for {badrpc,_} errors. Rex failed. OK. Let me (or mnesia) decide what to do next. Why is this such a controversial idea? This isn't about mnesia though. It's about rex. Anything could have triggered the crash. I fail to see why rex is the arbiter of the vm's fate when the process limit is reached. People keep implying it was designed this way but the behavior is sporadic and inconsistent with similar issues. If such a behavior was designed it should live in the VM (not that I support this behavior) and not a process that in practice is totally recoverable from when it fails (ie {badrpc,nodedown}). -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Ulf Wiger Sent: Thursday, September 09, 2010 5:15 AM To: erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM On 09/09/2010 12:34 AM, bile@REDACTED wrote: > > How many other limits cause the platform to shit the bed? I suspect > few. Actually, hitting the system limits themselves will not cause the VM to crash. OOM is the only exception I can think of right now. Trying to spawn a process when the max number of processes has been reached will simply raise an exception. However, some (most) application code will not include to cope with the situation that you can't spawn a process, so most likely, applications will come crashing down when this happens. Which one goes first is mainly up to chance. Erlang is a concurrency-oriented language. The spawn() function is about as fundamental as new() in OO languages. In most other languages, you treat the spawning of processes as something scary that you don't want to do unless you absolutely have to. Also, the limits are usually pretty low. In Erlang, you can raise the limit to > 200 million processes, if you have enough memory for it. The default limit is fairly low (32,000) mainly for historical reasons, but also because it makes sense to keep memory footprint low by default, and 32K processes is plenty enough for most uses. > Those who defend this behavior are not consistent. The behavior of the > core processes are not consistent. Just look at the code. It would be better if you mention specific instances rather than asking people to "just look at the code". OTOH, we can probably stipulate that the code base is inconsistent in many ways.... In general, application code does not cope with system limits being exhausted. This is in line with Erlang's "let it crash" philosophy as well as the fact that if you want true robustness in the kind of products Erlang was designed for, you have to have a redundant setup anyway. The thing about redundancy is that it works best if the failing side fails quickly and distinctly, rather than trying in vain to correct the problem locally. This is the essence of "fail-fast" programming. Although this is not 100% consistently implemented in Erlang/OTP either, you might want to keep in mind that Erlang has been breaking new ground in this respect, and most of the people who've worked on OTP components over the years were originally steeped in the same programming mindset as everyone else. ;-) >>> I shouldn't have to build my own spawn wrapper to keep track of the >>> number of processes. The VM already does this. Besides, this problem >>> couldn't be fully addressed that way. >> >> You don't have to. I suspect you need to do some sort of load >> regulation in your system. > > Load regulation? My system is designed to support arbitrary process > creation. I was maxing out the processes as a scale test. If for some > reason it can't spawn new processes then I want control over what > happens next. Rex and the supervisor's behavior takes that control > from me. At best I can poll the process count and warn that the system > will soon fail but am powerless to do anything about it. With any programming language or operating environment, you have the responsibility as a developer to understand and respect the fundamental assumptions made when developing the environment. If your requirements don't match well enough, it might be better to find another language/environment that fits your problem better. You seem to be saying that you shouldn't have to worry about the user of your application throwing more work at you than the system is capable of handling? This may be a valid requirement in some domains, but Erlang is fundamentally a language for developing messaging systems, which have to cope with overload situations (including Denial-of-Service attacks) in a structured way. When subjected to a DoS attack, you typically don't just want to accept the challenge and likely die honorably as a result. The normal way to do that is to push back, or shed load so that it doesn't overwhelm the core components in your system. You might compare this with the recurring discussions about active vs passive sockets. Sockets in POSIX are by design passive, as you have to explicitly read data from the buffer, but in Erlang, the default is that the VM empties the buffer and delivers the data to the socket owner asynchronously. I think this was the wrong default, and the recommendation is to use passive, or {active,once} to avoid being swamped by input from the network. This is in line with the idea of not accepting more work than you can cope with. > The failure of > a non-essential component of the system should not cause the VM to > fail just like a bad process in an OS should cause it to halt. Could > one of you please explain to me how that analogy is incorrect? RPC is not a non-essential component, especially not to mnesia. Mnesia assumes that rpc will work, unless something really bad has happened. The recommended behaviour when something really bad happens in Erlang is to die and let the rest of the system take over. BR, Ulf W ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From Antonio.Musumeci@REDACTED Thu Sep 9 14:37:36 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Thu, 9 Sep 2010 08:37:36 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DB56@NYWEXMBX2129.msad.ms.com> So rex failing and bringing down the VM is a purposeful design decision? Aren't their cleaner ways to do this? Why not a watchdog that pays attention to resources and halts the VM when hit? Why isn't it built into the system itself? This may be a valid behavior for your situations but it's not for mine. I don't like or want my system to just collapse without an ability to do something about it when it's possible to do so. And as I recall if you start erlang in minimal mode and manually start rex it is not part of the primary supervision tree and can therefore be killed. Why doesn't it then force the VM to fail? ________________________________ From: Chandru [mailto:chandrashekhar.mullaparthi@REDACTED] Sent: Thursday, September 09, 2010 6:16 AM To: bile@REDACTED Cc: Musumeci, Antonio S (Enterprise Infrastructure); erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM On 8 September 2010 23:34, > wrote: How many other limits cause the platform to shit the bed? I suspect few. There is a massive difference between the entire platform collapsing and RPC not working / restarting. If spawning of processes is so fundamental why do the core processes fit into the process limit? Linux as mentioned before prevents userland from wrecking things by safeguarding some amount of RAM for itself. Couldn't BEAM do the same? Why doesn't it auto shutdown if the limit is hit? Why no warnings from the system? Why is it triggered by rex? A minor component of the base process tree. As Ulf pointed out, beam does throw an exception. It is indeed the rex process which decided to give up. What I meant to say was that almost all applications written in erlang assume they are operating within the system limits. In the case of rex, it decides to die if it can't spawn a process. rex is only active if you are using distributed erlang. That is a design decision, and it is a valid one - atleast for those of us who use it a lot in real world situations. There is no reason to take control away from the developer. Especially when it means the entire platform will collapse from underneath them for something entirely controllable. It is a trade off. You have complete control over what happens in the system when you program in C. Doesn't necessarily mean that is the best choice. > The error message you see about mnesia_recover is the "effect", not > the "cause". The mnesia_recover rpc call is the catalyst for the failure. It's issuing the RPC command. The error is caused by the rpc failure. You are wrong. Infact if you look closely at it, mnesia is trying to make an rpc call when it was trying to dump core, which means it was already dying at that point. You need to dig deeper to find the real cause. Chandru From chandrashekhar.mullaparthi@REDACTED Thu Sep 9 15:02:46 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Thu, 9 Sep 2010 14:02:46 +0100 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DB56@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <01489E237C2C8F45AF7898E135F48E8019A3B1DB56@NYWEXMBX2129.msad.ms.com> Message-ID: On 9 September 2010 13:37, Musumeci, Antonio S < Antonio.Musumeci@REDACTED> wrote: > And as I recall if you start erlang in minimal mode and manually start rex > it is not part of the primary supervision tree and can therefore be killed. > Why doesn't it then force the VM to fail? > This is a bit like saying that if you disconnect the brakes on a car, and then drive it, it doesn't stop when you press the brakes! rex isn't supposed to be started like that in a "real" system. I think what everyone is trying to tell you is, from what you've written so far, there is nothing in erlang which is making it impossible for you to design a robust system. In other words, you are barking up the wrong tree. Chandru From sedrik@REDACTED Thu Sep 9 15:03:51 2010 From: sedrik@REDACTED (Fredrik Andersson) Date: Thu, 9 Sep 2010 15:03:51 +0200 Subject: Behaviour of queue Message-ID: Hi all I have a question concerning the behaviour of the erlang queue module. Does it store everything in memory or will it fallback to storing things on disk if memory is running short? If it does not I need to implement a layer on top of queue that will fallback to store things on disk if needed. Thanks in advance From Antonio.Musumeci@REDACTED Thu Sep 9 15:10:17 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Thu, 9 Sep 2010 09:10:17 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01935968-E02D-456B-9237-1D2D05B69E52@alertlogic.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <20100909061308.585d6089@otis> <01935968-E02D-456B-9237-1D2D05B69E52@alertlogic.com> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DB6F@NYWEXMBX2129.msad.ms.com> I did read it... It's just not relevant. It doesn't matter if it's my application or not. I have no control over the core system. It's not possible to do anything but mitigate the problem by trying not to spawn processes above some totally arbitrary value. If I only had used 80% of the limit rex could suddenly get tons of incoming calls and spawn the other 20% till it crashed. It's a design flaw to crash a system due to a recoverable error. Rex doesn't crash when a remote node is down so why should it do so when it can't fulfill the rpc request because of other issues? -----Original Message----- From: Paul Fisher [mailto:pfisher@REDACTED] Sent: Thursday, September 09, 2010 8:58 AM To: Cc: Chandru; Musumeci, Antonio S (Enterprise Infrastructure); erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM And you apparently failed to read any of the rest of what I wrote. The root cause of the problem is that your application is using > 99% of the available processes without any consideration for limit. This is a design flaw in your application. On Sep 9, 2010, at 5:13 AM, wrote: > On Wed, 8 Sep 2010 22:42:36 -0500 > Paul Fisher wrote: >> >> There is no sane thing that the vm could do in the face of such >> process usage. Erlang crashes so that you can figure out how to >> correct the design flaw, by looking at the dump. >> > > That's simply false. The sane thing to do is to return an error so the > developer may attempt to handle the situation. There is no flaw. > Hitting the max process limit is not a guaranteed fatal situation and > I should not be kept from attempting to respond it it. > > The fact that this happens by chance due to rex and the supervisor's > setup rather than built into the VM totally blows your argument out of > the water. Erlang doesn't crash in the normal sense. > It's not a true fatal error. It could be triggered by calling > rpc:stop(). If it were an actual design decision it should be clearly > documented and part of the base VM behavior... not a side effect of > rex's design. -- paul From tuncer.ayaz@REDACTED Thu Sep 9 15:30:00 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 9 Sep 2010 15:30:00 +0200 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: On Thu, Sep 9, 2010 at 5:17 AM, Richard O'Keefe wrote: > I'm installing R14A. > > documentation : fop is missing. Using fakefop to generate > placeholder PDF files. > > This is the first Erlang installation I've done where this came up. > Presumably fop is a processor for XML-FO. Where do I get it? FOP has been a requirement for creating PDF files before R14A. I've patched in fakefop support exactly because FOP is not a widespread tool available on standard development machines. The fact that FOP needs a lot of memory that has to be upped (with a JVM option) to not OOM and takes pretty long for processing are other disadvantages you probably don't want to experience. FOP is sadly neither space nor time efficient for processing the Erlang documentation. xsltproc (from libxslt) which you probably have installed is enough to create man pages and html docs. From Antonio.Musumeci@REDACTED Thu Sep 9 15:33:49 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Thu, 9 Sep 2010 09:33:49 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <01489E237C2C8F45AF7898E135F48E8019A3B1DB56@NYWEXMBX2129.msad.ms.com> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DB7F@NYWEXMBX2129.msad.ms.com> I don't see how that analogy is appropriate. If I removed the breaks I expect it not to work. If I remove rpc I expect rpc not to work. If a break pad fails I still expect the other parts of the car to continue working. I can replace the breaking system without replacing the car. If the engine has some problem I'll get a check engine notification... the car won't just shut down. Besides, if starting it manually is really a no-no then it probably shouldn't have a :start/0 and :stop/0 as shortcuts. "nothing in erlang which is making it impossible for you to design a robust system." So long as work around behaviors that force my hand. If the system is intended to be robust, to deal with major issues by crashing and letting others clean up... then why doesn't rpc work the same way? Or perhaps the supervisor? Why is the crashing behavior left up to a side effect rather than explicit? It hardly seems like this was purposefully designed this way. There are other conditions which cause rpc to be unable to fulfill the request. I don't see why process limits are different. When I run out of file descriptors in some of my software they don't crash... they attempt to prioritize the resources I do have and free up things if possible. Running out of FDs is not the end of the world, neither is running out of processes. Hell, in C you can attempt to recover from a sigsegv or worse. In general it may be pointless but in some situations maybe not. It's for me to decide. ________________________________ From: Chandru [mailto:chandrashekhar.mullaparthi@REDACTED] Sent: Thursday, September 09, 2010 9:03 AM To: Musumeci, Antonio S (Enterprise Infrastructure) Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM On 9 September 2010 13:37, Musumeci, Antonio S > wrote: And as I recall if you start erlang in minimal mode and manually start rex it is not part of the primary supervision tree and can therefore be killed. Why doesn't it then force the VM to fail? This is a bit like saying that if you disconnect the brakes on a car, and then drive it, it doesn't stop when you press the brakes! rex isn't supposed to be started like that in a "real" system. I think what everyone is trying to tell you is, from what you've written so far, there is nothing in erlang which is making it impossible for you to design a robust system. In other words, you are barking up the wrong tree. Chandru From boris.muehmer@REDACTED Thu Sep 9 16:38:18 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Thu, 9 Sep 2010 16:38:18 +0200 Subject: Erlang User Groups in Germany / Nuremberg??? Message-ID: English: Are there any "Erlang User Groups" in Germany ... or even in Nuremberg? Deutsch: Gibt es "Erlang?'Benutzer' Gruppen" in Deutschland? Vielleicht auch in N?rnberg? (Irgendwie kommt es mir so vor, als w?re Erlang in Deutschland nicht so popul?r?!?!?!) N?mbercherisch: Gebts a "Erlang User Group" in N?mberch? ??- boris From ulf.wiger@REDACTED Thu Sep 9 17:46:00 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Thu, 09 Sep 2010 17:46:00 +0200 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DB50@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com><20100908170738.GA6576@hanele><01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <4C88A579.6000609@erlang-solutions.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DB50@NYWEXMBX2129.msad.ms.com> Message-ID: <4C890138.6010602@erlang-solutions.com> On 09/09/2010 02:15 PM, Musumeci, Antonio S wrote: > I understand your points completely... however, there is certainly a > difference from having an erlang process die and allowing it's peers > to handle the cleanup and having the erlang vm die. The vm has no > peer in that way. It does have a peer if you are running with redundancy, and Erlang/OTP was primarily designed for systems where redundancy is pretty much a given. > Yes Mnesia needs RPC... but so do a lot of things > and if the pattern is to be followed that you die and allow the peers > to respond... that's not what is happening here. Rex dies and brings > the world down with it. Mnesia is unable to respond to the issue. The > mnesia code shows that it is prepared for {badrpc,_} errors. Yes, but 'badrpc' means "I was not able to communicate with the other node" - not "I wasn't even able to try". Mnesia is quite prepared to handle the case that other nodes disappear. In this case, a system resource has been exhausted, and as Chandru pointed out, the crash came as mnesia was trying to create a core dump, which means your system was going down anyway. This is fairly typical if you exhaust a system limit. Even if you could theoretically write code to handle it, most of the libraries you likely want to use have not been designed that way, so something is going to break. If it were only rex, it is easy enough to write your own RPC library that behaves differently. But mnesia is not prepared to cope with not being able to spawn a process, or create an ETS table, which is another system limit that can bring down mnesia. You call it a completely arbitrary limit, but it is no more arbitrary than the number of open file descriptors you may have. You are not forced to accept the default limit, and just like with the file descriptor limit, you probably couldn't for any system of size. Try setting the process limit to a suitably high number. BR, Ulf W From als@REDACTED Thu Sep 9 18:56:52 2010 From: als@REDACTED (Anthony Shipman) Date: Fri, 10 Sep 2010 02:56:52 +1000 Subject: FYI escript gotcha wrt receive Message-ID: <201009100256.52736.als@iinet.net.au> I had an escript that spawned a process to run a receive loop to fetch messages. The messages were returned by the receive in a scrambled order wrt the sending process. I found the cause and solution buried deep. The erl_eval module page has this at the bottom: Bugs The evaluator is not complete. receive cannot be handled properly and the escript page says By default, the script will be interpreted. You can force it to be compiled by including the following line somewhere in the script file: -mode(compile). Using compile mode solved the receive problem. I think that it would be useful for the escript page to say why one would want to use compile mode instead of interpret. -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From freza@REDACTED Thu Sep 9 19:04:00 2010 From: freza@REDACTED (Jachym Holecek) Date: Thu, 9 Sep 2010 18:04:00 +0100 Subject: [erlang-questions] Behaviour of queue In-Reply-To: References: Message-ID: <20100909170400.GB1428@hanele> # Fredrik Andersson 2010-09-09: > I have a question concerning the behaviour of the erlang queue module. > Does it store everything in memory or will it fallback to storing > things on disk if memory is running short? If it does not I need to > implement a layer on top of queue that will fallback to store things > on disk if needed. It's implemented as a pair of lists, that is all data lives on heap of given process. Other option to implement a queue is to use ETS table, but that's in-memory too so not sure it helps. You could go with DETS table too, but I don't have first-hand experience with it. Regards, -- Jachym From Antonio.Musumeci@REDACTED Thu Sep 9 19:33:55 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Thu, 9 Sep 2010 13:33:55 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <4C890138.6010602@erlang-solutions.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com><20100908170738.GA6576@hanele><01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <4C88A579.6000609@erlang-solutions.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DB50@NYWEXMBX2129.msad.ms.com> <4C890138.6010602@erlang-solutions.com> Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3B1DC55@NYWEXMBX2129.msad.ms.com> {badrpc,_} seems to me to mean that the rpc failed. The second element is for why. Not being able to spawn is a why. It's obvious that other services can't handle the exhaustion of processes. I'm seeing mnesia, rex and timer_server in my dump. If you kill timer_server though it restarts. If I kill mnesia it's just dead. But mnesia is forbidden from dealing with the failure of rex because of it's behavior and I'm unable to deal with the possible failure of mnesia for the same reason. They are separate (but in this case connected) issues and should be addressed separately. Obviously I could set the limit arbitrarily high but then what's the point of the limit? Again, I don't see what the problem is with giving developers more control. If you want the VM to die when the process limit is reached then you can do so easily. Telling people just to raise limits is whitewashing the problem and hides possible reliability improvements. -----Original Message----- From: Ulf Wiger [mailto:ulf.wiger@REDACTED] Sent: Thursday, September 09, 2010 11:46 AM To: Musumeci, Antonio S (Enterprise Infrastructure) Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] System limit bringing down rex and the VM On 09/09/2010 02:15 PM, Musumeci, Antonio S wrote: > I understand your points completely... however, there is certainly a > difference from having an erlang process die and allowing it's peers > to handle the cleanup and having the erlang vm die. The vm has no peer > in that way. It does have a peer if you are running with redundancy, and Erlang/OTP was primarily designed for systems where redundancy is pretty much a given. > Yes Mnesia needs RPC... but so do a lot of things and if the pattern > is to be followed that you die and allow the peers to respond... > that's not what is happening here. Rex dies and brings the world down > with it. Mnesia is unable to respond to the issue. The mnesia code > shows that it is prepared for {badrpc,_} errors. Yes, but 'badrpc' means "I was not able to communicate with the other node" - not "I wasn't even able to try". Mnesia is quite prepared to handle the case that other nodes disappear. In this case, a system resource has been exhausted, and as Chandru pointed out, the crash came as mnesia was trying to create a core dump, which means your system was going down anyway. This is fairly typical if you exhaust a system limit. Even if you could theoretically write code to handle it, most of the libraries you likely want to use have not been designed that way, so something is going to break. If it were only rex, it is easy enough to write your own RPC library that behaves differently. But mnesia is not prepared to cope with not being able to spawn a process, or create an ETS table, which is another system limit that can bring down mnesia. You call it a completely arbitrary limit, but it is no more arbitrary than the number of open file descriptors you may have. You are not forced to accept the default limit, and just like with the file descriptor limit, you probably couldn't for any system of size. Try setting the process limit to a suitably high number. BR, Ulf W From tuncer.ayaz@REDACTED Thu Sep 9 20:06:40 2010 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 9 Sep 2010 20:06:40 +0200 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: On Thu, Sep 9, 2010 at 3:30 PM, Tuncer Ayaz wrote: > On Thu, Sep 9, 2010 at 5:17 AM, Richard O'Keefe wrote: >> I'm installing R14A. >> >> documentation : fop is missing. Using fakefop to generate >> placeholder PDF files. >> >> This is the first Erlang installation I've done where this came up. >> Presumably fop is a processor for XML-FO. Where do I get it? > > FOP has been a requirement for creating PDF files before R14A. I've > patched in fakefop support exactly because FOP is not a widespread > tool available on standard development machines. The fact that FOP > needs a lot of memory that has to be upped (with a JVM option) to not > OOM and takes pretty long for processing are other disadvantages you > probably don't want to experience. FOP is sadly neither space nor time > efficient for processing the Erlang documentation. Also FOP makes use of Apache Batik and therefore requires X11. This makes it harder to run in a graphics-less environment. > xsltproc (from libxslt) which you probably have installed is enough to > create man pages and html docs. From rvirding@REDACTED Thu Sep 9 20:08:44 2010 From: rvirding@REDACTED (Robert Virding) Date: Thu, 9 Sep 2010 20:08:44 +0200 Subject: [erlang-questions] Behaviour of queue In-Reply-To: <20100909170400.GB1428@hanele> References: <20100909170400.GB1428@hanele> Message-ID: On 9 September 2010 19:04, Jachym Holecek wrote: > # Fredrik Andersson 2010-09-09: >> I have a question concerning the behaviour of the erlang queue module. >> Does it store everything in memory or will it fallback to storing >> things on disk if memory is running short? If it does not I need to >> implement a layer on top of queue that will fallback to store things >> on disk if needed. > > It's implemented as a pair of lists, that is all data lives on heap > of given process. Other option to implement a queue is to use ETS > table, but that's in-memory too so not sure it helps. You could go > with DETS table too, but I don't have first-hand experience with it. What Jachym was implying but never really explicitly stated is that the queue module is so simple that it is easier to completely rewrite it with the storage mechanism of your choice rather than add another layer on top of it. In this he is completely right. Robert From boris.muehmer@REDACTED Thu Sep 9 20:13:30 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Thu, 9 Sep 2010 20:13:30 +0200 Subject: Erlang User Groups in Germany / Nuremberg??? In-Reply-To: References: Message-ID: Falls jemand aus dem N?rnberger Raum (also auch F?rth und Erlangen) Interesse an einer Anwendergruppe f?r Erlang hat, kann sie/er mal unter http://groups.google.com/group/erlang-nuernberg reinschauen... - boris From sedrik@REDACTED Thu Sep 9 22:13:43 2010 From: sedrik@REDACTED (Fredrik Andersson) Date: Thu, 9 Sep 2010 22:13:43 +0200 Subject: [erlang-questions] Behaviour of queue In-Reply-To: References: <20100909170400.GB1428@hanele> Message-ID: On Thu, Sep 9, 2010 at 10:13 PM, Fredrik Andersson wrote: > On Thu, Sep 9, 2010 at 8:08 PM, Robert Virding wrote: >> On 9 September 2010 19:04, Jachym Holecek wrote: >>> # Fredrik Andersson 2010-09-09: >>>> I have a question concerning the behaviour of the erlang queue module. >>>> Does it store everything in memory or will it fallback to storing >>>> things on disk if memory is running short? If it does not I need to >>>> implement a layer on top of queue that will fallback to store things >>>> on disk if needed. >>> >>> It's implemented as a pair of lists, that is all data lives on heap >>> of given process. Other option to implement a queue is to use ETS >>> table, but that's in-memory too so not sure it helps. You could go >>> with DETS table too, but I don't have first-hand experience with it. >> >> What Jachym was implying but never really explicitly stated is that >> the queue module is so simple that it is easier to completely rewrite >> it with the storage mechanism of your choice rather than add another >> layer on top of it. In this he is completely right. >> >> Robert > > Ok, Since we already have another layer on top of the queue module in > my current project I will simply extend that to write stuff to disk > when needed. > > Thanks :) Apparently I sent this message to Robert only. Here you go list :) > >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > From ok@REDACTED Fri Sep 10 04:27:48 2010 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 10 Sep 2010 14:27:48 +1200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) In-Reply-To: References: <4C827ED3.7080004@erlang-solutions.com> <4C84D6E8.2080602@erlang-solutions.com> <4C8508FA.6040208@erlang-solutions.com> <002b01cb4f5b$eba94b70$c2fbe250$@com> <098DD2F8-6311-43E3-946E-519C81089A87@cs.otago.ac.nz> Message-ID: On Sep 9, 2010, at 11:34 PM, Jesper Louis Andersen wrote: > Fun aside relating to this point: Standard ML defines specific types > as eqtypes or types which can be compared for equality. The 'real' > type of ML is not an eqtype because of the above mentioned trouble: > comparing two reals (IEEE FPs) for equality is almost often wrong. I see this claim a lot, but I wonder how true it is. In *beginners'* code, yes, sure, but then, almost everything in beginners' code is wrong at some point. Attempts to deal with the problem by banning = don't really help, because you can always define fun eql x (y:float) = not (x < y orelse x > y); after which eql 0.0 ~0.0 is not only defined but true. If it comes to that, the MLton compiler complains every time you use polymorphic equality at all. > The MosML implementation of SML get this wrong and allows reals to be > compared for equality. This also means that programs which compile in > MosML does not compile in other SML implementations. It has bitten me > more than once. ocaml also does this. From zabrane3@REDACTED Fri Sep 10 09:10:53 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Fri, 10 Sep 2010 09:10:53 +0200 Subject: Call a calback function when quitting Erlang shell with CTRL+C + q. Message-ID: Hi List, I'd like to know if it's possible to tell Erlang shell to register a callback somewhere, so when I stop my shell with the keystrocke "CTRL+C" followed by "q" letter, the callback gets called? This will be useful to do some cleaning before (temporaries, closing files ...) quitting. Advices? -- Regards Zabrane From ulf.wiger@REDACTED Fri Sep 10 09:33:27 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Fri, 10 Sep 2010 09:33:27 +0200 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DC55@NYWEXMBX2129.msad.ms.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com><20100908170738.GA6576@hanele><01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com><01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <4C88A579.6000609@erlang-solutions.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DB50@NYWEXMBX2129.msad.ms.com> <4C890138.6010602@erlang-solutions.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DC55@NYWEXMBX2129.msad.ms.com> Message-ID: <4C89DF47.4060909@erlang-solutions.com> On 09/09/2010 07:33 PM, Musumeci, Antonio S wrote: > > I'm seeing mnesia, rex and timer_server in my dump. If you > kill timer_server though it restarts. Actually, I consider this a bug. Let's check to see what the result is of killing timer_server. Eshell V5.7.5 (abort with ^G) 1> F = fun() -> timer:send_after(15000,self(),hello), receive Msg -> io:fwrite("got ~p~n", [Msg]) end end. #Fun 2> f(P), P = spawn(F), time(). {9,25,48} got hello 3> time(). {9,26,6} 4> whereis(timer_server). <0.38.0> 5> f(P), P = spawn(F), time(). {9,26,22} 6> exit(whereis(timer_server),kill). true 7> whereis(timer_server). <0.43.0> 8> time(). {9,27,0} 9> process_info(P). [{current_function,{erl_eval,receive_clauses,6}}, {initial_call,{erlang,apply,2}}, {status,waiting}, {message_queue_len,0}, {messages,[]}, ... So killing timer_server caused it to bounce back, but in the process, it forgot all outstanding requests, so any processes depending on the reliable service of the timer server are now left hanging, with no indication whatsoever that something went wrong. Personally, I think it would be much better if the timer server would in fact stay dead, and bring the whole node down with it - that, or make sure that its dying and restarting is truly transparent. Choosing a middle way of merely pretending to be robust is the worst possible choice. Rather than concluding that the OTP team are incompetent in matters of robustness (as there is overwhelming evidence that they are anything but), I'd like to see this as yet another example of how desperately difficult and dangerous it is to go down the path you're suggesting. It may seem like a respectful thing to do, but you take on a very heavy burden, and may well be much more likely to compound the problem rather than helping it. BR, Ulf W From chandrashekhar.mullaparthi@REDACTED Fri Sep 10 11:13:10 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Fri, 10 Sep 2010 10:13:10 +0100 Subject: [erlang-questions] Behaviour of queue In-Reply-To: References: Message-ID: On 9 September 2010 14:03, Fredrik Andersson wrote: > Hi all > > I have a question concerning the behaviour of the erlang queue module. > Does it store everything in memory or will it fallback to storing > things on disk if memory is running short? If it does not I need to > implement a layer on top of queue that will fallback to store things > on disk if needed. > > You can simulate a queue using an ordered_set mnesia table with the value of now() as the primary key. That might save you writing an extra layer on top. cheers Chandru From erlang@REDACTED Fri Sep 10 11:41:19 2010 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 10 Sep 2010 11:41:19 +0200 Subject: [erlang-questions] Behaviour of queue In-Reply-To: References: Message-ID: Queue as robert said, are really easy. The simplest possible queue is just a pair of lists {In, Out}. As new elements are added they are appended to In So we can write: new_queue() -> {[], []}. add(X, {In,Out}) -> {[X|In}, Out}. What about extracting elements from the queue - these come from Out, so ... remove({In, [X|Out]} -> {X, {In, Out}}; ... But hang on, what happens in Out is empty - well if In is also empty we have nothing to remove so: remove({[], []}) -> empty; Otherwise In is non empty and Out is empty so we revers In and move it to Out and try again remove({In, []}) -> remove({[], reverse(In)}). The nice thing about this is that amortized cost of using the queue is constant. We create a single cons cell when adding to the In list, and a single cons cell when moving an element from the In to Out list - each element is only touched once as it moves from input to output. This is by far the best method for short queue non persistent queues. This pattern - a pair of two lists, one reversed one in normal order is used over and over again in many guises - it's the "natural" representation for a text editor. Files are queues of lines. The current line being edited is a queue of characters, etc. Well worth remembering the pattern ... /Joe On Fri, Sep 10, 2010 at 11:13 AM, Chandru wrote: > On 9 September 2010 14:03, Fredrik Andersson wrote: > >> Hi all >> >> I have a question concerning the behaviour of the erlang queue module. >> Does it store everything in memory or will it fallback to storing >> things on disk if memory is running short? If it does not I need to >> implement a layer on top of queue that will fallback to store things >> on disk if needed. >> >> > You can simulate a queue using an ordered_set mnesia table with the value of > now() as the primary key. That might save you writing an extra layer on top. > > cheers > Chandru > From bile@REDACTED Fri Sep 10 12:27:08 2010 From: bile@REDACTED (bile@REDACTED) Date: Fri, 10 Sep 2010 06:27:08 -0400 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <4C89DF47.4060909@erlang-solutions.com> References: <01489E237C2C8F45AF7898E135F48E8019A3B1D9AF@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA34@NYWEXMBX2129.msad.ms.com> <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <4C88A579.6000609@erlang-solutions.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DB50@NYWEXMBX2129.msad.ms.com> <4C890138.6010602@erlang-solutions.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DC55@NYWEXMBX2129.msad.ms.com> <4C89DF47.4060909@erlang-solutions.com> Message-ID: <20100910062708.0699fc04@otis> If it's not documented it's a bug, otherwise just a bad behavior. If you can do something about it it's a feature. Agreed, a middle way is bad unless clearly documented and preferably there is some way to handle it reasonably. Perhaps ets with the heir option with a custom primary supervisor that would take ownership of the data while it restarted it and then transfer it back? Looking over the timer code it looks that it does a catch spawn so it seems that the spawn shouldn't have been the reason for it's crash. I'll have to look at it more closely to see what happened in my case. On Fri, 10 Sep 2010 09:33:27 +0200 Ulf Wiger wrote: > On 09/09/2010 07:33 PM, Musumeci, Antonio S wrote: > > > > I'm seeing mnesia, rex and timer_server in my dump. If you > > kill timer_server though it restarts. > > Actually, I consider this a bug. > > Let's check to see what the result is of killing timer_server. > > Eshell V5.7.5 (abort with ^G) > 1> F = fun() -> > timer:send_after(15000,self(),hello), > receive > Msg -> > io:fwrite("got ~p~n", [Msg]) > end > end. > #Fun > 2> f(P), P = spawn(F), time(). > {9,25,48} > got hello > 3> time(). > {9,26,6} > 4> whereis(timer_server). > <0.38.0> > 5> f(P), P = spawn(F), time(). > {9,26,22} > 6> exit(whereis(timer_server),kill). > true > 7> whereis(timer_server). > <0.43.0> > 8> time(). > {9,27,0} > 9> process_info(P). > [{current_function,{erl_eval,receive_clauses,6}}, > {initial_call,{erlang,apply,2}}, > {status,waiting}, > {message_queue_len,0}, > {messages,[]}, > ... > > So killing timer_server caused it to bounce back, but in the process, > it forgot all outstanding requests, so any processes depending on the > reliable service of the timer server are now left hanging, with no > indication whatsoever that something went wrong. > > Personally, I think it would be much better if the timer server would > in fact stay dead, and bring the whole node down with it - that, or > make sure that its dying and restarting is truly transparent. Choosing > a middle way of merely pretending to be robust is the worst possible > choice. > > Rather than concluding that the OTP team are incompetent in matters > of robustness (as there is overwhelming evidence that they are > anything but), I'd like to see this as yet another example of how > desperately difficult and dangerous it is to go down the path you're > suggesting. It may seem like a respectful thing to do, but you take > on a very heavy burden, and may well be much more likely to compound > the problem rather than helping it. > > BR, > Ulf W > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From matthew@REDACTED Fri Sep 10 15:07:58 2010 From: matthew@REDACTED (Matthew Sackman) Date: Fri, 10 Sep 2010 14:07:58 +0100 Subject: From whence do monitors come? Message-ID: <20100910130757.GA25976@rabbitmq.com> gen.erl has the following code in do_call: catch erlang:send(Process, {Label, {self(), Mref}, Request}, [noconnect]), receive {Mref, Reply} -> erlang:demonitor(Mref, [flush]), {ok, Reply}; {'DOWN', Mref, _, _, noconnection} -> exit({nodedown, Node}); {'DOWN', Mref, _, _, Reason} -> exit(Reason) If the callee is something like a gen_server, and on handling that msg returns a {stop, Reply, Reason, State} quple, there seems to be a race between the reply and the monitor. Or more simply, say the callee does something like: receive {'$gen_call', From, _Msg} -> gen_server:reply(From, done) end. There seems to be a race - what's stopping the DOWN from the monitor overtaking the reply from the callee? The only thing that I can think of is that the DOWN is considered to come *from* the callee. Is that right? Matthew From michael.eugene.turner@REDACTED Fri Sep 10 15:17:25 2010 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Fri, 10 Sep 2010 22:17:25 +0900 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: On Thu, Sep 9, 2010 at 3:05 PM, Max Lapshin wrote: > On Thu, Sep 9, 2010 at 9:42 AM, Michael Turner > wrote: > > this theory that PHP won out over a huge field of server-side scripting > > languages in the late 90s (some of them probably better) because it was > the > > easiest to install. > > > > Erlang in installable almost anywhere without any fop. It is optional > dependency > Yes, but ISTR when I last did an Erlang build for a machine with no pre-built executable anywhere, I found that it's an enabled option. I had to look pretty hard, then finally ask around, to figure out to disable it. Finally, I gave up on that, and figured out where to get FOP and build it. And I didn't even want the PDFs, I was just trying to get through the build. -michael turner From ulf.wiger@REDACTED Fri Sep 10 15:21:02 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Fri, 10 Sep 2010 15:21:02 +0200 Subject: [erlang-questions] From whence do monitors come? In-Reply-To: <20100910130757.GA25976@rabbitmq.com> References: <20100910130757.GA25976@rabbitmq.com> Message-ID: <4C8A30BE.4090100@erlang-solutions.com> On 09/10/2010 03:07 PM, Matthew Sackman wrote: > gen.erl has the following code in do_call: > > catch erlang:send(Process, {Label, {self(), Mref}, Request}, > [noconnect]), > receive > {Mref, Reply} -> > erlang:demonitor(Mref, [flush]), > {ok, Reply}; > {'DOWN', Mref, _, _, noconnection} -> > exit({nodedown, Node}); > {'DOWN', Mref, _, _, Reason} -> > exit(Reason) > > If the callee is something like a gen_server, and on handling that msg > returns a {stop, Reply, Reason, State} quple, there seems to be a race > between the reply and the monitor. Or more simply, say the callee does > something like: > > receive {'$gen_call', From, _Msg} -> gen_server:reply(From, done) end. > > There seems to be a race - what's stopping the DOWN from the monitor > overtaking the reply from the callee? > > The only thing that I can think of is that the DOWN is considered to > come *from* the callee. Is that right? This is the only conclusion that would be compatible with the ordering guarantee, IMHO. The gen_server explicitly sends the reply before it dies, and it would be almost impossible to write an efficient and safe implementation of do_call if one couldn't be sure that the DOWN message will never be delivered before the last message sent to the same recipient. The ordering guarantee should be clearly documented in the reference manual, together with a clear statement on how EXIT signals and DOWN messages play along with that guarantee. BR, Ulf W From mononcqc@REDACTED Fri Sep 10 18:48:57 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 10 Sep 2010 12:48:57 -0400 Subject: local renaming with a module attribute Message-ID: I would like to discuss the idea of adding a new module attribute allowing to rename a module when doing explicit external calls (local renaming, if I can coin the term). The attribute -rename(long_module_name, name) would allow the programmer to use 'name' instead of 'long_module_name' inside a given module. The reason for this is the regular necessity to manually 'namespace' modules when writing an application. Given the fictive application 'robot', the application structure fits best under the following fictive modules: robot.erl -- interface functions to control your robot brain.erl -- fsm that represents the robot brain sup.erl -- app supervisor mapping.erl -- allows the robot to represent its own map to navigate speech.erl -- allows voice recognition Under the current system, we need to avoid possible name clashes and instead write the modules as: robot.erl robot_brain.erl robot_sup.erl robot_mapping.erl robot_speech.erl Which then gets repeated on every function call that might follow from within the application itself. Most of these namespaced modules are necessary only to prevent conflicts and add no information to a program, only cruft to mentally skip over. The new module attribute (say, '-rename(Old, New).') would allow to change: some_fun(X) -> robot_mapping:funcall(X+1, robot_speech:empty()) end. to the following: -rename(robot_mapping, mapping). -rename(robot_speech, speech). some_fun(X) -> mapping:funcall(X+1, speech:empty()) end. Which allows us to retain all useful information about a module without the manual namespacing or requiring to import the functions. A naive implementation could simply expand all static 'New' external calls to the 'Old' value for everything to be transparent to the outside world. ---- Local renaming overlaps the current packages system ( http://www.erlang.se/publications/packages.html) a bit in its approach to namespace. However, there are a few noticeable difference, two of which are: - packages are based on the directory structure of the source files. Local renaming is restricted to a single module at a time and bears no relationship with the organization of source files on the OS. - packages automatically kick in whenever the module name has periods in it. Local renaming is explicit. - packages can support a hierarchy. Local renaming has no hierarchy at all. I don't think one could necessarily kill the other, although it could render it much less useful as a whole. I would also not want to do the package system's trial, but I do believe errors and abuses in the local renaming system are much simpler to solve and handle than errors and abuses under the packages model. The reason is simple: local renaming is local to a module, packages are global to every module. ---- There are already a few potential problems that are easy to see in advance. The first of these has to do with variables to carry module names around. Let's reuse the robot example. The following is a legal call: X = robot_brain, X:analyze([a,t,f,e,b,s,c,z]). However, if we say: -rename(robot_brain, brain). ... X = brain, X:analyze([a,t,f,e,b,s,c,z]). then the call should fail because 'brain' won't be expanded. This example is trivial, but the issue is the same with module names coming from function calls or messages. I don't know how packages handle this, maybe they do it better. Another issue is to avoid renaming modules that were already renamed. Not because it's a technical problem, but rather because I believe we should strive for simplicity rather than flexibility on that one. That is up for discussion I guess. Another potential weakness of the idea is that it only prevents one kind of namespacing conflicts. In no way does it address ETS table names clashes, registered processes clashes, etc. It might get confusing to use the namespace for one but not the others. So, what's your opinion? You might find the whole thing interesting or totally worthless, but in either case I find this list's opinions worth reading on this kind of issue. From anthonym@REDACTED Fri Sep 10 19:02:02 2010 From: anthonym@REDACTED (Anthony Molinaro) Date: Fri, 10 Sep 2010 10:02:02 -0700 Subject: [erlang-questions] System limit bringing down rex and the VM In-Reply-To: <01489E237C2C8F45AF7898E135F48E8019A3B1DB7F@NYWEXMBX2129.msad.ms.com> References: <20100908170738.GA6576@hanele> <01489E237C2C8F45AF7898E135F48E8019A3B1DA75@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DA8A@NYWEXMBX2129.msad.ms.com> <20100908183452.0c1df9d4@otis> <01489E237C2C8F45AF7898E135F48E8019A3B1DB56@NYWEXMBX2129.msad.ms.com> <01489E237C2C8F45AF7898E135F48E8019A3B1DB7F@NYWEXMBX2129.msad.ms.com> Message-ID: <20100910170202.GA53082@alumni.caltech.edu> On Thu, Sep 09, 2010 at 09:33:49AM -0400, Musumeci, Antonio S wrote: > So long as work around behaviors that force my hand. If the system is intended to be robust, to deal with major issues by crashing and letting others clean up... then why doesn't rpc work the same way? Or perhaps the supervisor? Why is the crashing behavior left up to a side effect rather than explicit? It hardly seems like this was purposefully designed this way. There are other conditions which cause rpc to be unable to fulfill the request. I don't see why process limits are different. When I run out of file descriptors in some of my software they don't crash... they attempt to prioritize the resources I do have and free up things if possible. Running out of FDs is not the end of the world, neither is running out of processes. Hell, in C you can attempt to recover from a sigsegv or worse. In general it may be pointless but in some situations maybe not. It's for me to decide. Are you running heart? http://www.erlang.org/doc/man/heart.html With heart you have an external process which restarts the vm if it crashes. Now of course I've had buggy linked-in drivers crash the vm, and cause it to restart over and over and over again, but it does seem to keep it running. So there is still a way to crash and let others cleanup. It's not always what you want, but when it happens likely means a bug in your system somewhere. I see no problem with you starting a minimal vm and spawning the rex process yourself so you can capture errors and try to deal with them. You've stated this can be done, so why not just do it? Sincerely, -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro From vladdu55@REDACTED Fri Sep 10 19:02:34 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Fri, 10 Sep 2010 19:02:34 +0200 Subject: [erlang-questions] local renaming with a module attribute In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 18:48, Fred Hebert wrote: > I would like to discuss the idea of adding a new module attribute allowing > to rename a module when doing explicit external calls (local renaming, if I > can coin the term). The attribute -rename(long_module_name, name) would > allow the programmer to use 'name' instead of 'long_module_name' inside a > given module. Hi! What I usually use is -define(mapping, robot_mapping). ... ?mapping:func(X) I use lower-cased name on purpose in this case. regards, Vlad From pascalchapier@REDACTED Fri Sep 10 20:05:23 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Fri, 10 Sep 2010 20:05:23 +0200 Subject: [erlang-questions] strictly less-than (Re: [erlang-questions] Ordered set and DETS) Message-ID: I didn't know that float appears lately in Erlang. It should be the reason why we have this weird type of number. Integer is a mathematical group with a pretty good representation in Erlang where there is no size limit. +, - and * are defined in that group, and the Erlang algorithms are consistent with the mathematical operations. Float is nothing but a subset of the rational numbers. In this subset, there is no guarantee that the result of a mathematical addition between 2 float is a float. IEEE gives the rules to round the math result to the closest float. IEEE +, -, *, / are perfectly defined functions, but they have not the properties of their math counterpart: 1> Y = 1.7e308. 1.7e308 2> W = -Y. -1.7e308 3> D = 9.975e291 . 9.975e291 4> Y =:= Y + D. true %% Hummm ! 5> (Y + D*1.1) - Y . 1.99584030953472e292 6> ((Y + D*1.1) - Y) > 2*D . true %% i.e. D*1.1 > 2*D with D > 0.0 ! 7> (D+Y)+W. 0.0 %% really ? 8> D+(Y+W). 9.975e291 %% very good, but different! 9> In these examples, all variable are floats, this shows that one must be aware of the float representation when using them. Mixing floats with integer, complex calculation, FPU internal representation and code implementation can even create a bigger mess. In real life, this kind of problem is not likely to occur, because we generally use float in a small range, far from the maximum value, and we are generally not interested in very very small values - except 0.0 (on my side I change unit if the values I am using are smaller than 1/1000 or bigger than 10000). Nevertheless, the smallest relative difference between two float is around 10e-16 (in 64bits), and this is sometime a problem. I wrote a small 3D simulation program as a hobby. The float were very useful, but I had to write my own operators for <, > and ==, based on a global value NEAR, which define a relative limit where float representation is not relevant, knowing my problem and the algorithms I use. My conclusion is that it is a bad idea to compare floats, a worse idea to check their equality, and an error to mix integers and floats is such operations, using only implicit conversion and standard IEEE operations. The same conclusion applies to pattern matching. I think that it is a good idea to say that the target is to have separate float and integer types, and separate them in Erlang terms comparison. The problem I see, is that it is not possible to detect at compilation a mix in comparison or operation, and this seems be necessary to have an intermediate phase where numbers and numbers operations or comparisons have a deprecate status (i.e. operation without explicit conversion). Is it acceptable to have warnings at run time, and is it feasible to have these warnings controlled by a flag, for backward compatibility? I am not sure that the problem is enough critical to be worth the creation of new syntax operation or the dynamic check . Pascal. From norbu09@REDACTED Sat Sep 11 00:59:18 2010 From: norbu09@REDACTED (lenz) Date: Sat, 11 Sep 2010 10:59:18 +1200 Subject: LinuxConf AU 2011 Parallel programming miniconf Message-ID: hi, for the second year there is a parallel programming miniconf at LCA and call for papers opened. LinuxConf AU is in Brisbane in January next year and we would love to see some more erlang talks. for the north hemispherical of you, this is summer down here, don't miss some beautiful days on the gold coast and some really relaxed geeks. http://multicorelca.wordpress.com/2010/09/10/cfp-for-the-open-source-multicore-and-parallel-computing-miniconf/ cheers lenz -- twitter: @norbu09 current project: iWantMyName.com painless domain registration (finally) From zeno490@REDACTED Sat Sep 11 01:53:02 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Fri, 10 Sep 2010 19:53:02 -0400 Subject: [erlang-questions] erlang:max/2 and erlang:min/2 In-Reply-To: References: Message-ID: This is fairly standard as far as I know in languages without strong typing in function arguments (when that is the case, usually coercion happens which hides this behavior from you). For example in ruby: >> [1, 1.0].min => 1 >> [1.0, 1].min => 1.0 >> And again in python: >>> min([1, 1.0]) 1 >>> min([1.0, 1]) 1.0 >>> For these languages (and erlang), order DOES matter. This isn't unlike a naive implementation of min/max in C++: #define max(A, B) A >= B ? A : B However, even using c++0x: auto SomeResult = max(1, 1.0); // what happens here? what is the type of SomeResult? (float) See note below auto SomeResult = max(1.0, 1); // what happens here? what is the type of SomeResult? (float) See note below *** Note below (really a side note, you can skip this) *** The 'ternary' operator is special (in C/C++). In the case above, both A and B can't be of different types. The type will be coerced to a common base type only if possible, otherwise compilation will fail. See for details: http://msdn.microsoft.com/en-us/library/e4213hs1%28VS.71%29.aspx (applies to other compilers as well) The ternary operator can also be used as an l-value (for fun and profit) :) ((A >= B) ? A : B) = 123; // Don't do this.. otherwise your coworkers will want to kill me for teaching you *** In C++ at least, both types will be converted to a common base type with the ternary operator or you'll traditionally write functions which will coerce the arguments for you in order to use hardware specific instructions to do the selection: max(A,B) becomes 2 instructions for floats/doubles (on powerpc, same for min): C = A - B, C >= 0 ? B : A (powerpc has an instruction to compare a float against zero and select the result (fsel)) For integers, you can go either with a branch or branch-free version. Mixing types will always go to the float/double impl for obvious reasons. As far as I know, in a loosely type language like erlang, you have to know your data... if you sort/compare mixed types (integers/floats/otherwise), you have to handle that case and be aware of the (potential) consequences. I for one disagree with the global/total ordering imposed on mixed types. But then again, I mainly come from strongly typed languages and I am not aware of the original logic behind it, perhaps it makes perfect sense. This thread should perhaps be merged with the 'strictly less then' thread since it voices a similar issue regarding term comparison and ordering. Nicholas On Thu, Sep 9, 2010 at 12:33 AM, Richard O'Keefe wrote: > I see that max/2 and min/2 are now in Erlang. > You would expect them to satisfy > [min(X,Y),max(X,Y)] is a permutation of [X,Y] > max(X,Y) is max(Y,X) > min(X,Y) is min(Y,X) > amongst other things. > > When X and Y are distinct terms that are equal as numbers, > max(X,Y) and min(X,Y) both return X. > > sort3([A,B,C]) -> > [U,V] = sort2([A,B]), > [W,Z] = sort2([V,C]), > [X,Y] = sort2([U,W]), > [X,Y,Z]. > > sort2([A,B]) -> > [min(A,B),max(A,B)]. > > You'd expect this to sort. But it doesn't: > > sort3:sort3([0,0.0,-0.0]). > [0,0,0] > > It really is very confusing for it to make a difference > whether you write max(X, 0) or max(0, X), but in R14A it does. > > The *really* big problem here is that max/2 and min/2 are > (otherwise) based on *term* comparison, and there *can't* be > two identical but distinguishable terms. > > In order to get a total order on numbers, it seems necessary > to rule that > -0.0 < 0 < 0.0 > and this extends to a general rule that > if X and Y are numbers that are numerically equal > but of different types, the floating point one is > smaller if its sign bit is set, otherwise the > floating point one is bigger. > > The root of the problem is trying to make one ordering > serve as both a numeric ordering and a term ordering. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From tony@REDACTED Sat Sep 11 15:10:16 2010 From: tony@REDACTED (Tony Rogvall) Date: Sat, 11 Sep 2010 15:10:16 +0200 Subject: [erlang-questions] erlang:max/2 and erlang:min/2 In-Reply-To: References: Message-ID: <4F7C7CC9-6440-48AF-9C84-E62B7FDB54C8@rogvall.se> > > The ternary operator can also be used as an l-value (for fun and profit) :) > ((A >= B) ? A : B) = 123; // Don't do this.. otherwise your coworkers will > want to kill me for teaching you Ultra cool. This is perfect for obscure programming. I will integrate this in the standard workflow here at the office! Sadly gcc complains a bit about this! tassign.c:12: warning: target of assignment not really an lvalue; this will be a hard error in the future with the option -fno-non-lvalue-assign This will generate error: tassign.c:12: error: lvalue required as left operand of assignment Unfortunately I could not find any option to turn off the irritating warning. Thanks ;-) /Tony From matthew@REDACTED Sat Sep 11 16:00:03 2010 From: matthew@REDACTED (Matthew Sackman) Date: Sat, 11 Sep 2010 15:00:03 +0100 Subject: Erlang, Linux, libaio Message-ID: <20100911140003.GB31522@wellquite.org> Hi, I'm considering trying to do a linked-in driver for file operations using libaio. I'm just wondering whether a) anyone's tried this before; b) are there any plans for the Erlang kernel file driver to support libaio; c) are there any reasons why this would be a bad idea? Matthew From rvirding@REDACTED Sat Sep 11 16:41:49 2010 From: rvirding@REDACTED (Robert Virding) Date: Sat, 11 Sep 2010 16:41:49 +0200 Subject: [erlang-questions] From whence do monitors come? In-Reply-To: <4C8A30BE.4090100@erlang-solutions.com> References: <20100910130757.GA25976@rabbitmq.com> <4C8A30BE.4090100@erlang-solutions.com> Message-ID: It's the [flush] option to demonitor which saves you here. Apart from removing the monitor it automatically removes any monitor message form that process in the process mailbox. And yes the {'DOWN',...} message does work as a normal message. Robert On 10 September 2010 15:21, Ulf Wiger wrote: > On 09/10/2010 03:07 PM, Matthew Sackman wrote: >> gen.erl has the following code in do_call: >> >> ? ? ? ? ? catch erlang:send(Process, {Label, {self(), Mref}, Request}, >> ? ? ? ? ? ? ? ? [noconnect]), >> ? ? ? ? ? receive >> ? ? ? ? ? ? ? {Mref, Reply} -> >> ? ? ? ? ? ? ? ? ? erlang:demonitor(Mref, [flush]), >> ? ? ? ? ? ? ? ? ? {ok, Reply}; >> ? ? ? ? ? ? ? {'DOWN', Mref, _, _, noconnection} -> >> ? ? ? ? ? ? ? ? ? exit({nodedown, Node}); >> ? ? ? ? ? ? ? {'DOWN', Mref, _, _, Reason} -> >> ? ? ? ? ? ? ? ? ? exit(Reason) >> >> If the callee is something like a gen_server, and on handling that msg >> returns a {stop, Reply, Reason, State} quple, there seems to be a race >> between the reply and the monitor. Or more simply, say the callee does >> something like: >> >> receive {'$gen_call', From, _Msg} -> gen_server:reply(From, done) end. >> >> There seems to be a race - what's stopping the DOWN from the monitor >> overtaking the reply from the callee? >> >> The only thing that I can think of is that the DOWN is considered to >> come *from* the callee. Is that right? > > This is the only conclusion that would be compatible with the ordering > guarantee, IMHO. The gen_server explicitly sends the reply before it > dies, and it would be almost impossible to write an efficient and > safe implementation of do_call if one couldn't be sure that the DOWN > message will never be delivered before the last message sent to the > same recipient. > > The ordering guarantee should be clearly documented in the reference > manual, together with a clear statement on how EXIT signals and > DOWN messages play along with that guarantee. > > BR, > Ulf W > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From matthew@REDACTED Sat Sep 11 16:53:18 2010 From: matthew@REDACTED (Matthew Sackman) Date: Sat, 11 Sep 2010 15:53:18 +0100 Subject: [erlang-questions] From whence do monitors come? In-Reply-To: References: <20100910130757.GA25976@rabbitmq.com> <4C8A30BE.4090100@erlang-solutions.com> Message-ID: <20100911145317.GC31522@wellquite.org> On Sat, Sep 11, 2010 at 04:41:49PM +0200, Robert Virding wrote: > It's the [flush] option to demonitor which saves you here. Apart from > removing the monitor it automatically removes any monitor message form > that process in the process mailbox. No, you've misunderstood the problem. The problem is that unless the DOWN message is considered to come *from* the terminating process, there is no guarantee of the order in which the reply and the DOWN message will arrive. The problematic case being the potential for the DOWN to arrive before the reply. > And yes the {'DOWN',...} message > does work as a normal message. Does that also hold for exit signals which are being sent to processes that have trap_exit on? Matthew From boris.okner@REDACTED Sun Sep 12 06:24:58 2010 From: boris.okner@REDACTED (bokner) Date: Sat, 11 Sep 2010 21:24:58 -0700 (PDT) Subject: node connectivity problem Message-ID: <68fff81e-9300-4040-b6da-541e68f1a863@c21g2000vba.googlegroups.com> Hello everyone, Struggling with connecting 2 nodes running on separate boxes. Tried to make sure that there is no usual problems with cookie synchronization, DNS or firewall. First, I run epmd in debug mode as recommended by Erlang docs: epmd -d -d Then on box #1: erl -name xmpp1@`hostname` -kernel inet_dist_listen_min 6000 inet_dist_listen_max 6050 -setcookie testcookie and on box #2: erl -name xmpp2@`hostname` -kernel inet_dist_listen_min 6000 inet_dist_listen_max 6050 -setcookie testcookie For example, on box #2: Erlang (BEAM) emulator version 5.6.4 [source] [64-bit] [smp:4] [async- threads:0] [kernel-poll:false] Eshell V5.6.4 (abort with ^G) (xmpp2@REDACTED)1> net_adm:ping('xmpp1@REDACTED'). pang epmd on server1.net shows following: epmd: Sun Sep 12 01:40:32 2010: opening connection on file descriptor 6 epmd: Sun Sep 12 01:40:32 2010: got 8 bytes ***** 00000000 00 06 7a 78 6d 70 70 31 |..zxmpp1| epmd: Sun Sep 12 01:40:32 2010: ** got PORT2_REQ epmd: Sun Sep 12 01:40:32 2010: got 18 bytes ***** 00000000 77 00 17 70 4d 00 00 05 00 05 00 05 78 6d 70 70 | w..pM.......xmpp| ***** 00000010 31 00 |1.| epmd: Sun Sep 12 01:40:32 2010: ** sent PORT2_RESP (ok) for "xmpp1" epmd: Sun Sep 12 01:40:32 2010: closing connection on file descriptor 6 i.e., appears to receive ping request from second node and respond with ok. Tshark listening on epmd port (TCP 4369) gives following (I replaced real IPs with server names): 1 0.000000 server2.net -> server1.net TCP 43809 > epmd [SYN] Seq=0 Win=5840 Len=0 MSS=1460 SACK_PERM=1 TSV=776213773 TSER=0 WS=5 2 0.000433 server1.net -> server2.net TCP epmd > 43809 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSV=1595930818 TSER=776213773 WS=6 3 0.000483 server2.net -> server1.net TCP 43809 > epmd [ACK] Seq=1 Ack=1 Win=5856 Len=0 TSV=776213773 TSER=1595930818 4 0.000545 server2.net -> server1.net EPMD 43809 > epmd [PSH, ACK] Seq=1 Ack=1 Win=5856 Len=8 TSV=776213773 TSER=1595930818 5 0.001445 server1.net -> server2.net TCP epmd > 43809 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=1595930818 TSER=776213773 6 0.001466 server1.net -> server2.net EPMD epmd > 43809 [PSH, ACK] Seq=1 Ack=9 Win=5824 Len=18 TSV=1595930818 TSER=776213773 7 0.001474 server2.net -> server1.net TCP 43809 > epmd [ACK] Seq=9 Ack=19 Win=5856 Len=0 TSV=776213773 TSER=1595930818 8 0.001481 server1.net -> server2.net TCP epmd > 43809 [FIN, ACK] Seq=19 Ack=9 Win=5824 Len=0 TSV=1595930818 TSER=776213773 9 0.001623 server2.net -> server1.net TCP 43809 > epmd [FIN, ACK] Seq=9 Ack=20 Win=5856 Len=0 TSV=776213773 TSER=1595930818 10 0.001990 server1.net -> server2.net TCP epmd > 43809 [ACK] Seq=20 Ack=10 Win=5824 Len=0 TSV=1595930818 TSER=776213773 So it looks to me that there is no firewall issues, as epmd instances talk to each other. What am I missing? Your advise is very much appreciated! Best regards, Boris From mk@REDACTED Sun Sep 12 12:48:38 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 12 Sep 2010 12:48:38 +0200 Subject: node to node message passing Message-ID: <4C8CB006.6070908@amberbio.com> Hi Erlangers. During some test with node to node communication, I sent a large binary from a process on node A to a process on another node, node B. I also sent some smaller messages from other processes on node A to other processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked the net tick communication, so node A and B disconnected from each other even though the large message was being transferred! After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent sequentially from the sending node A to the receiving node. If that is correct, I think some improvements are needed. The problem to solve is basically that small messages, including the net tick, should get through more or less independently of the presence of large messages. The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up a lot of the hardware bandwidth even on another tcp connection. My suggestion is something like the following. For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the right process. The fragments ensure that no single transfer will occupy the connection for very long. There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say. Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers for backup purposes can have low priority. The send process then switches between the queues in some way, that could be very similar to context switching priorities. More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance. There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in a timeout in a receive loop. I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small messages can overtake large ones. There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require a certain order. I am relatively new to Erlang and I really enjoy it. Kudos to all involved! Cheers, Morten Krogh. From michael.santos@REDACTED Sun Sep 12 13:44:16 2010 From: michael.santos@REDACTED (Michael Santos) Date: Sun, 12 Sep 2010 07:44:16 -0400 Subject: [erlang-questions] Re: node connectivity problem In-Reply-To: <68fff81e-9300-4040-b6da-541e68f1a863@c21g2000vba.googlegroups.com> References: <68fff81e-9300-4040-b6da-541e68f1a863@c21g2000vba.googlegroups.com> Message-ID: <20100912114416.GA20816@ecn.lan> On Sat, Sep 11, 2010 at 09:24:58PM -0700, bokner wrote: > Then on box #1: > erl -name xmpp1@`hostname` -kernel inet_dist_listen_min 6000 > inet_dist_listen_max 6050 -setcookie testcookie > > and on box #2: > erl -name xmpp2@`hostname` -kernel inet_dist_listen_min 6000 > inet_dist_listen_max 6050 -setcookie testcookie > epmd on server1.net shows following: > > epmd: Sun Sep 12 01:40:32 2010: opening connection on file descriptor > 6 > epmd: Sun Sep 12 01:40:32 2010: got 8 bytes > ***** 00000000 00 06 7a 78 6d 70 70 31 > |..zxmpp1| > epmd: Sun Sep 12 01:40:32 2010: ** got PORT2_REQ > epmd: Sun Sep 12 01:40:32 2010: got 18 bytes > ***** 00000000 77 00 17 70 4d 00 00 05 00 05 00 05 78 6d 70 70 | > w..pM.......xmpp| > ***** 00000010 31 00 |1.| > epmd: Sun Sep 12 01:40:32 2010: ** sent PORT2_RESP (ok) for "xmpp1" > epmd: Sun Sep 12 01:40:32 2010: closing connection on file descriptor > 6 > > i.e., appears to receive ping request from second node and respond > with ok. The request to epmd from xmpp2 -> xmppp1: ***** 00000000 00 06 7a 78 6d 70 70 31 |..zxmpp1| 00 06: Message length is 6 bytes 7a: Message type is PORT2_PLEASE_REQ 78 6d 70 70 31: Node name is "xmpp1" The response from epmd on xmpp1 -> xmpp2: ***** 00000000 77 00 17 70 4d 00 00 05 00 05 00 05 78 6d 70 70 | w..pM.......xmpp| ***** 00000010 31 00 |1.| 77: Message type is PORT2_RESP 00: Result is "ok" 17 70: Port number is 6000 4d: Node type is Erlang node 00: Protocol is TCP 05: Higest version of protocol supported 05: Lowest version of protocol supported 00 05: Node name length is 5 bytes 78 6d 70 70 31: Node name is "xmpp1" During the next phase of the protocol, xmpp2 will connect to server1.net on port 6000 (the distribution port of xmpp1) and go through the authentication handshake. Try sniffing port 6000 to see if a TCP connection is established. If everything looks ok, start up 2 distributed nodes on server1.net (e.g., xmpp1@REDACTED, xmpp2@REDACTED) and see if they are able to ping each other. From boris.okner@REDACTED Sun Sep 12 18:04:18 2010 From: boris.okner@REDACTED (Boris Okner) Date: Sun, 12 Sep 2010 12:04:18 -0400 Subject: [erlang-questions] Re: node connectivity problem In-Reply-To: <20100912114416.GA20816@ecn.lan> References: <68fff81e-9300-4040-b6da-541e68f1a863@c21g2000vba.googlegroups.com> <20100912114416.GA20816@ecn.lan> Message-ID: Thanks Michael, I followed your advice and figured it was a firewall problem. Thanks a lot, that was really helpful. Regards, Boris On Sun, Sep 12, 2010 at 7:44 AM, Michael Santos wrote: > On Sat, Sep 11, 2010 at 09:24:58PM -0700, bokner wrote: > > > Then on box #1: > > erl -name xmpp1@`hostname` -kernel inet_dist_listen_min 6000 > > inet_dist_listen_max 6050 -setcookie testcookie > > > > and on box #2: > > erl -name xmpp2@`hostname` -kernel inet_dist_listen_min 6000 > > inet_dist_listen_max 6050 -setcookie testcookie > > > epmd on server1.net shows following: > > > > epmd: Sun Sep 12 01:40:32 2010: opening connection on file descriptor > > 6 > > epmd: Sun Sep 12 01:40:32 2010: got 8 bytes > > ***** 00000000 00 06 7a 78 6d 70 70 31 > > |..zxmpp1| > > epmd: Sun Sep 12 01:40:32 2010: ** got PORT2_REQ > > epmd: Sun Sep 12 01:40:32 2010: got 18 bytes > > ***** 00000000 77 00 17 70 4d 00 00 05 00 05 00 05 78 6d 70 70 | > > w..pM.......xmpp| > > ***** 00000010 31 00 |1.| > > epmd: Sun Sep 12 01:40:32 2010: ** sent PORT2_RESP (ok) for "xmpp1" > > epmd: Sun Sep 12 01:40:32 2010: closing connection on file descriptor > > 6 > > > > i.e., appears to receive ping request from second node and respond > > with ok. > > The request to epmd from xmpp2 -> xmppp1: > > ***** 00000000 00 06 7a 78 6d 70 70 31 |..zxmpp1| > > 00 06: Message length is 6 bytes > 7a: Message type is PORT2_PLEASE_REQ > 78 6d 70 70 31: Node name is "xmpp1" > > The response from epmd on xmpp1 -> xmpp2: > > ***** 00000000 77 00 17 70 4d 00 00 05 00 05 00 05 78 6d 70 70 | > w..pM.......xmpp| > ***** 00000010 31 00 |1.| > > 77: Message type is PORT2_RESP > 00: Result is "ok" > 17 70: Port number is 6000 > 4d: Node type is Erlang node > 00: Protocol is TCP > 05: Higest version of protocol supported > 05: Lowest version of protocol supported > 00 05: Node name length is 5 bytes > 78 6d 70 70 31: Node name is "xmpp1" > > During the next phase of the protocol, xmpp2 will connect to server1.net > on port 6000 (the distribution port of xmpp1) and go through the > authentication handshake. Try sniffing port 6000 to see if a TCP > connection is established. > > If everything looks ok, start up 2 distributed nodes on server1.net > (e.g., xmpp1@REDACTED, xmpp2@REDACTED) and see if they are able > to ping each other. > > > From jh@REDACTED Sun Sep 12 19:51:17 2010 From: jh@REDACTED (Jan Huwald) Date: Sun, 12 Sep 2010 19:51:17 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <4C8CB006.6070908@amberbio.com> References: <4C8CB006.6070908@amberbio.com> Message-ID: <201009121951.20575.jh@sotun.de> Am Sunday 12 September 2010 12:48:38 schrieb Morten Krogh: > Hi Erlangers. > > During some test with node to node communication, I sent a large binary > from a process on node A > to a process on another node, node B. I also sent some smaller messages > from other processes on node A to other > processes on node B. It turned out that the large message blocked the > later messages. Furthermore, it even blocked > the net tick communication, so node A and B disconnected from each other > even though the large message was being transferred! > > After looking a bit around, I have come to the understanding that Erlang > uses one tcp connection between two nodes, and messages are sent > sequentially from the sending node A to the receiving node. > > If that is correct, I think some improvements are needed. IMO the programmer has to take precautions on its own, if he expetcs to handle large messages. This is analogous to large memory requirements of single process. Erlang is not well suited for one of both, natively - which is good, because it keeps things simple. In case of large messages (or large process heaps) tradeoffs have to be made. Your proposed solution is one example for (a) the fact that these tradeoffs can be excluded from the language core (b) involving tradeoffs not everybody is agreeing on. > The problem to solve is basically that small messages, including the net > tick, should get through more or less independently of > the presence of large messages. > > The simplest would be to have several connections, but that doesn't > fully solve the problem. A large message will still take up > a lot of the hardware bandwidth even on another tcp connection. > > My suggestion is something like the following. > > For communication between node A and node B, there is a process (send > process) on each node, that coordinates all messages. The send process > keeps queues of different priorities around, e.g., a high priority, > medium priority and low priority. Messages are split up into fragments of > a maximum size. The receiver(node B) send process assembles the > fragments into the original message and delivers it locally to the > right process. The fragments ensure that no single transfer will occupy > the connection for very long. > There will be a function send_priority where the user can specify a > priority. The usual send will default to medium, say. > Net tick will use high priority, of course. Small messages that are > needed to produce a web application response can have high priority. > File transfers > for backup purposes can have low priority. > The send process then switches between the queues in some way, that > could be very similar to context switching priorities. > > More advanced, the send processes could occasionally probe the > connection with packets to estimate latency and bandwidth. Those figures > could then be used > to calculate fragment sizes. High bandwidth, high latency would require > large fragments. Low bandwidth, low latency small fragments for instance. > There could even be a function send_estimated_transfer_time that sends a > message and has a return value of estimated transfer time, which could > be used in > a timeout in a receive loop. Actively probing seems to me like a recipe for far more failures than it offers benefit. Passively probing might be ok, if the run-time overhead is small. But an actual implementation, which does not get biased by nonstationary activity of the application, seems quite complex to me. > I have actually implemented my own small module for splitting messages > into fragments, and it solves the issues; net tick goes through, and small > messages can overtake large ones. > > There is of course an issue when the sending and receiving process is > the same for several messages. Either the guaranteed message order > should be given up, or the > coordinators should keep track of that as well. Personally, I think > guaranteed message order should be given up. Erlang should model the > real world as > much as possible, and learn from it. In the real world, two letters > going from person A to person B, can definitely arrive in the opposite > order > of the one in which they were sent. And as node to node communication > will be over larger and larger distances, it is totally unnatural to > require > a certain order. > > I am relatively new to Erlang and I really enjoy it. Kudos to all involved! > > Cheers, > > Morten Krogh. Regards, Jan From ok@REDACTED Mon Sep 13 04:43:55 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 13 Sep 2010 14:43:55 +1200 Subject: [erlang-questions] erlang:max/2 and erlang:min/2 In-Reply-To: References: Message-ID: <33A3A8B6-098C-4C1C-8C8A-16A4BF38A6AF@cs.otago.ac.nz> On Sep 11, 2010, at 11:53 AM, Nicholas Frechette wrote: > This is fairly standard as far as I know in languages without strong typing in function arguments (when that is the case, usually coercion happens which hides this behavior from you). > For example in ruby: > >> [1, 1.0].min > => 1 > >> [1.0, 1].min > => 1.0 The traditional retort to this is "And if all your friends jumped off a cliff, would you copy that too?" > > For these languages (and erlang), order DOES matter. And all that means is that they are buggy too. > > This isn't unlike a naive implementation of min/max in C++: > #define max(A, B) A >= B ? A : B Er, that is a couple of steps of beginnerness _below_ "naive". #define max(A, B) ((A) >= (B) ? (A) : (B)) is normally required, and of course the preferred method in C++ is template T max(T a, T b) { return a >= b ? a : b; } for obvious reasons. > > However, even using c++0x: > auto SomeResult = max(1, 1.0); // what happens here? what is the type of SomeResult? (float) See note below > auto SomeResult = max(1.0, 1); // what happens here? what is the type of SomeResult? (float) See note below What happens here is quite irrelevant to dynamically typed languages. The if-then-else operator has to return a result of *some* definite type, so there really are only two choices for it. Either it demands that both choices be the same type, or else it has to convert at least one of them. This has no significance whatever for a language that *doesn't* require all arms of an if to have the same type and *doesn't* have to convert. > > *** Note below (really a side note, you can skip this) *** > The 'ternary' operator is special (in C/C++). In the case above, both A and B can't be of different types. I think "can't" was meant to be "can". > The type will be coerced to a common base type only if possible, otherwise compilation will fail. > See for details: http://msdn.microsoft.com/en-us/library/e4213hs1%28VS.71%29.aspx (applies to other compilers as well) > The ternary operator can also be used as an l-value (for fun and profit) :) NOT IN STANDARD C OR C++ IT CANNOT! > ((A >= B) ? A : B) = 123; // Don't do this.. otherwise your coworkers will want to kill me for teaching you > *** One reason they will be displeased with you is that most compilers reject this. A typical error message: "foo.c", line 2: warning: conditional expression is not a valid lvalue > > In C++ at least, both types will be converted to a common base type with the ternary operator or you'll traditionally write functions which will coerce the arguments for you in order to use hardware specific instructions to do the selection: > max(A,B) becomes 2 instructions for floats/doubles (on powerpc, same for min): C = A - B, C >= 0 ? B : A (powerpc has an instruction to compare a float against zero and select the result (fsel)) On a SPARC or a Pentium or an ARM, it would be [assume A is in register x] [assume B is in register y] [assume result is wanted in register x] compare x with y one compare instruction move y to x if x < y one conditional move It is not clear to me how this is supposed to be relevant to Erlang, which has to deal with mixed integer/float comparisons, or to the question of how you implement max/2 and min/2 so that they satisfy the laws that a programmer *ought* to be able to take for granted. C and C++ can get away with (user-provided) definitions that convert basically because they can't have arrays whose elements are of varying types, so if you do auto x = a[0]; for (int i = 1; i < n; i++) x = max(x, a[i]); there _can't_ be any problems introduced by conversion, because all the a[i] must be the *same* type. But in Erlang, the elements of a list CAN be of different types. Intuitions based on experience with strongly typed languages (Ada, Clean, Haskell) would tell you that mixed-mode arithmetic should simply be banned. Any time it is intentional, an explicit coercion can be provided by the programmer. Intuitions based on experience with weakly typed languages (C, C++, Java, C#) would mislead you into thinking that operations that don't actually obey the mathematical laws they are supposed to are OK because thanks to the all-elements-of-an-array-are-the- same-type setup, uses of < and max DO obey the laws in question. (Sorting a floating-point array is unproblematic in C *IF* (1) there is at most one -infinity (2) there is at most one +infinity (3) there are no NaNs (4) either you don't care about the order of -0 and +0 or they don't both occur. Sorting a mixed integer/float array just can't happen.) From kenneth.lundin@REDACTED Mon Sep 13 09:43:35 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Mon, 13 Sep 2010 09:43:35 +0200 Subject: [erlang-questions] 14A wants 'fop', what/where? In-Reply-To: References: <9EBAB75E-D1FF-4DFE-8AAE-6CB56D4DC2AD@cs.otago.ac.nz> Message-ID: On Fri, Sep 10, 2010 at 3:17 PM, Michael Turner wrote: > On Thu, Sep 9, 2010 at 3:05 PM, Max Lapshin wrote: > >> On Thu, Sep 9, 2010 at 9:42 AM, Michael Turner >> wrote: >> > this theory that PHP won out over a huge field of server-side scripting >> > languages in the late 90s (some of them probably better) because it was >> the >> > easiest to install. >> > >> >> Erlang in installable almost anywhere without any fop. It is optional >> dependency >> > > Yes, but ISTR when I last did an Erlang build for a machine with no > pre-built executable anywhere, I found that it's an enabled option. ?I had > to look pretty hard, then finally ask around, to figure out to disable it. > ?Finally, I gave up on that, and figured out where to get FOP and build it. > ?And I didn't even want the PDFs, I was just trying to get through the > build. When we introduced the new way to build documentation (with dependencies to xsltproc and Apache FOP) in R13B03 the building of the PDFs was mandatory and thus the dependency to Apache FOP. This was unintentional and in R14A this was solved (thanks to Tuncer Ayaz) with a FOP replacement which generates dummy pdf files if Apache FOP cannot be found by configure. And also note that, there is absolutely no need to build the documentation from source at all if you are not developing new documentation or want to contribute and improve the documentation. You can read the documentation on-line on http://erlang.org or if you want the documentation locally you can download the documentation from http://erlang.org/download. The build of the Erlang/OTP SW is not dependent on the build of the documentation. The documentation package is totally platform independent. /Regards Kenneth, Erlang/OTP Ericsson > > -michael turner > From alavrik@REDACTED Mon Sep 13 10:07:35 2010 From: alavrik@REDACTED (Anton Lavrik) Date: Mon, 13 Sep 2010 03:07:35 -0500 Subject: Announce: the Piqi project Message-ID: <20100913080735.GA23321@debian.localdomain> It is my pleasure to announce the Piqi project to the Erlang community. I've just released a new version which features Erlang support. Piqi is a cross-language data serialization system compatible with Google Protocol Buffers with native support for Erlang and OCaml. OCaml and Erlang programmers can use Piqi to conveniently serialize and deserialize data using a portable binary format which can be understood by programs written in Python, Java, C++ and other languages supported by Google Protocol Buffers. In this sense, Piqi provides Protocol Buffers support for OCaml and Erlang. Compared to Protocol Buffers, Piqi relies on a richer data definition language which has a more natural mapping to functional languages. For instance, in addition to records and enums, Piqi supports variants (tagged unions), variant subtyping, lists and type aliases. Some other Piqi features include: - Support for data schema evolution (Protocol Buffers style). - "import" and "include" for reusing type definitions from other modules. - Powerful extensions mechanism for records and variants. - Structured default values for optional fields. Compatibility between Piqi and Protocol Buffers is maintained as follows: - Piqi specifications can be automatically converted to equivalent Protobuf specifications and vice versa (with very few exceptions). - Piqi and Protocol Buffers use the same binary encoding as a serialization format. Piqi comes with several additional tools: - Piq -- a high-level data representation language. It has full support for Piqi data types and it is easier to read, write and edit compared to XML or JSON. - Piqi tools for validating, pretty-printing and converting data between Piq, JSON and portable binary encoding. Erlang support highlights ========================= - Generated Erlang type and record definitions as well as serialization/deserialization code are EEP8-compliant and provide all necessary type information for Dialyzer. - Support for recursive record types. - Support for Unicode strings which can be passed as string() (i.e. list of Unicode codepoints) or utf-8 binary(). - Support for custom Erlang identifiers. For example, if some identifier from existing Proto or Piqi specification conflicts with an Erlang keyword, it is possible to define a custom Erlang-specific identifier without changing the original one and breaking backward compatibility. - Capitalized and "CamelCase" Piqi/Protobuf identifiers are automatically converted to uncapitalized and "camel_case" Erlang ids. - Configurable prefixes for type and record identifiers (since Erlang doesn't have any kind of namespace support for types and records, we need to prefix them to avoid conflicts with names defined in other modules). - Tested with Erlang/OTP R13B04 and R14A. Further references ================== Homepage: http://piqi.org GitHub repository and several Erlang examples: http://github.com/alavrik/piqi http://github.com/alavrik/piqi/tree/master/examples/addressbook/ Details of Piqi to Erlang mapping: http://piqi.org/doc/mappings/erlang/ Your comments, suggestions and contributions are welcome! Don't hesitate to contact me if you have any questions. Anton From geoffrey.biggs@REDACTED Mon Sep 13 10:20:02 2010 From: geoffrey.biggs@REDACTED (Geoffrey Biggs) Date: Mon, 13 Sep 2010 17:20:02 +0900 Subject: Shutting down part of a supervision tree Message-ID: <4C8DDEB2.2030606@aist.go.jp> Evening all, What is the accepted procedure for shutting down a part of a supervision tree? In my case, I have several CORBA objects that come and go. I'm managing them in a supervisor tree. The same supervisor that manages the CORBA objects also manages other Erlang processes. They are all meant to be shut down together; i.e. the supervisor, the workers and the CORBA objects operate as a single unit that comes and goes as necessary. The problem is that when they go, they need to de-register from a naming service. I've tried adding the de-registration call to the terminate/2 function in the CORBA object provider, but I can't seem to find something that guarantees it gets called in a timely fashion, if at all. This causes my tests to fail, because the next test tries to create a new instance of the object and can't register it due to the old registration still existing on the name server (shows the tests are doing their job, though). Methods I've tried to shut down the whole construction are: -Calling "exit(S, normal)" on the supervisor. -Directly casting stop to the CORBA objects, then calling "exit(S, normal)" on the supervisor. -Same as above, but with various sleeps in the hope that this gives the CORBA objects time to do their thing (even though the child_specs give them 30 seconds to shut down). -Using supervisor:terminate_child on the CORBA objects. So far the best combination seems to be casting stop to the CORBA object (which is what I would expect to work), but even so, the supervisor seems to just dump the whole lot before this cast has time to take effect half the time - and it's not even consistent. Any advice on shutting down parts of a supervisor tree is appreciated. Geoff From xramtsov@REDACTED Mon Sep 13 10:23:14 2010 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Mon, 13 Sep 2010 18:23:14 +1000 Subject: [erlang-questions] Announce: the Piqi project In-Reply-To: <20100913080735.GA23321@debian.localdomain> References: <20100913080735.GA23321@debian.localdomain> Message-ID: <4C8DDF72.1050004@gmail.com> 13.09.2010 18:07, Anton Lavrik wrote: > It is my pleasure to announce the Piqi project to the Erlang community. I've > just released a new version which features Erlang support. > > > Piqi is a cross-language data serialization system compatible with Google > Protocol Buffers with native support for Erlang and OCaml. > > OCaml and Erlang programmers can use Piqi to conveniently serialize and > deserialize data using a portable binary format which can be understood by > programs written in Python, Java, C++ and other languages supported by Google > Protocol Buffers. In this sense, Piqi provides Protocol Buffers support for > OCaml and Erlang. > > Compared to Protocol Buffers, Piqi relies on a richer data definition language > which has a more natural mapping to functional languages. For instance, in > addition to records and enums, Piqi supports variants (tagged unions), variant > subtyping, lists and type aliases. > > Some other Piqi features include: > - Support for data schema evolution (Protocol Buffers style). > - "import" and "include" for reusing type definitions from other modules. > - Powerful extensions mechanism for records and variants. > - Structured default values for optional fields. > > Compatibility between Piqi and Protocol Buffers is maintained as follows: > - Piqi specifications can be automatically converted to equivalent Protobuf > specifications and vice versa (with very few exceptions). > - Piqi and Protocol Buffers use the same binary encoding as a serialization > format. > > Piqi comes with several additional tools: > - Piq -- a high-level data representation language. It has full support for Piqi > data types and it is easier to read, write and edit compared to XML or JSON. > - Piqi tools for validating, pretty-printing and converting data between Piq, > JSON and portable binary encoding. > > > Erlang support highlights > ========================= > > - Generated Erlang type and record definitions as well as > serialization/deserialization code are EEP8-compliant and provide all necessary > type information for Dialyzer. > > - Support for recursive record types. > > - Support for Unicode strings which can be passed as string() (i.e. list of > Unicode codepoints) or utf-8 binary(). > > - Support for custom Erlang identifiers. For example, if some identifier from > existing Proto or Piqi specification conflicts with an Erlang keyword, it is > possible to define a custom Erlang-specific identifier without changing the > original one and breaking backward compatibility. > > - Capitalized and "CamelCase" Piqi/Protobuf identifiers are automatically > converted to uncapitalized and "camel_case" Erlang ids. > > - Configurable prefixes for type and record identifiers (since Erlang doesn't > have any kind of namespace support for types and records, we need to prefix > them to avoid conflicts with names defined in other modules). > > - Tested with Erlang/OTP R13B04 and R14A. > > > Further references > ================== > > Homepage: > > http://piqi.org > > GitHub repository and several Erlang examples: > > http://github.com/alavrik/piqi > http://github.com/alavrik/piqi/tree/master/examples/addressbook/ > > Details of Piqi to Erlang mapping: > > http://piqi.org/doc/mappings/erlang/ > > > Your comments, suggestions and contributions are welcome! Don't hesitate to > contact me if you have any questions. > Wow, that sounds impressive. The only problem I see is the license... -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From geoffrey.biggs@REDACTED Mon Sep 13 10:24:33 2010 From: geoffrey.biggs@REDACTED (Geoffrey Biggs) Date: Mon, 13 Sep 2010 17:24:33 +0900 Subject: Re-entrant CORBA services Message-ID: <4C8DDFC1.1010101@aist.go.jp> Morning all, I have a couple of tightly-interacting CORBA objects. It's a little too tight for my tastes, but that's OO for you. The problem is that a call to one service causes that service to block until a result is returned. Several of the calls to object A require making calls to object B. Unfortunately, one of the calls to object B requires its own calls back to object A to finish (whether or not it succeeds). This means that object B blocks waiting for object A to respond, while object A is blocked waiting for object B to respond. How have others solved this sort of interdependency? For the Erlang (not using CORBA interfaces) equivalents of these objects, I just fired off a third process that made the appropriate calls, then sent the result back to the first object as an event. I can't find a way to do this for the CORBA objects. Geoff From ulf.wiger@REDACTED Mon Sep 13 11:09:02 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 11:09:02 +0200 Subject: [erlang-questions] Announce: the Piqi project In-Reply-To: <4C8DDF72.1050004@gmail.com> References: <20100913080735.GA23321@debian.localdomain> <4C8DDF72.1050004@gmail.com> Message-ID: <4C8DEA2E.8070907@erlang-solutions.com> On 09/13/2010 10:23 AM, Evgeniy Khramtsov wrote: > 13.09.2010 18:07, Anton Lavrik wrote: >> It is my pleasure to announce the Piqi project to the Erlang >> community. I've >> just released a new version which features Erlang support. >> > > Wow, that sounds impressive. The only problem I see is the license... The Apache 2.0 license? What's wrong with that one? Or is it the contents of the NOTICE file that worries you? BR, Ulf W From jahakala@REDACTED Mon Sep 13 11:11:34 2010 From: jahakala@REDACTED (Jani Hakala) Date: Mon, 13 Sep 2010 12:11:34 +0300 Subject: [erlang-questions] Re-entrant CORBA services In-Reply-To: <4C8DDFC1.1010101@aist.go.jp> (Geoffrey Biggs's message of "Mon\, 13 Sep 2010 17\:24\:33 +0900") References: <4C8DDFC1.1010101@aist.go.jp> Message-ID: <87hbhum2ih.fsf@pingviini.priv> Geoffrey Biggs writes: > Unfortunately, one of the calls to object B requires its own calls back > to object A to finish (whether or not it succeeds). This means that > object B blocks waiting for object A to respond, while object A is > blocked waiting for object B to respond. > How about dividing this call from object A to object B to two calls to object B? First one would return at that point where B now calls A and the second call would give the object B the results of the call. Any parameters required by object A could be passed as the result from the first call. AMI/AMH [1] might provide another solution if they are available. Or perhaps multiple servants for object A. I prefer to avoid two-way calls with CORBA as much as possible. I once tried those with C++ and ACE/TAO, and it seemed to be too much trouble. I didn't try AMI/AMH or multiple servants though as I was just learning CORBA basics. Jani Hakala [1] asynchronous method invocation, asynchronous method handling. From torben.lehoff@REDACTED Mon Sep 13 13:14:02 2010 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Mon, 13 Sep 2010 13:14:02 +0200 Subject: [erlang-questions] Announce: the Piqi project In-Reply-To: <20100913080735.GA23321@debian.localdomain> References: <20100913080735.GA23321@debian.localdomain> Message-ID: Hi Anton, It looks very interesting - in fact so much that I will ask a few questions (always a good sign, right?). On the higher level: Are there any plans for providing a way to specify your own binary encoding? On the lower level: If I have a PDU where one field is present if another field has a specific value how would that look in Piqi? And what if the dependent field is separated from its condition field by many other fields? Example: (M means mandatory, C means conditional) Field1 M Field2 M . Field15 C - present if Field2 is 4. I am dealing with a protocol that has this kind of behaviour and I am looking into writing my own binary grammar, but I would much rather piggy bag on something else if many reasons on top of laziness ;-) Cheers, Torben On Mon, Sep 13, 2010 at 10:07, Anton Lavrik wrote: > > It is my pleasure to announce the Piqi project to the Erlang community. > I've > just released a new version which features Erlang support. > > > Piqi is a cross-language data serialization system compatible with Google > Protocol Buffers with native support for Erlang and OCaml. > > OCaml and Erlang programmers can use Piqi to conveniently serialize and > deserialize data using a portable binary format which can be understood by > programs written in Python, Java, C++ and other languages supported by > Google > Protocol Buffers. In this sense, Piqi provides Protocol Buffers support for > OCaml and Erlang. > > Compared to Protocol Buffers, Piqi relies on a richer data definition > language > which has a more natural mapping to functional languages. For instance, in > addition to records and enums, Piqi supports variants (tagged unions), > variant > subtyping, lists and type aliases. > > Some other Piqi features include: > - Support for data schema evolution (Protocol Buffers style). > - "import" and "include" for reusing type definitions from other modules. > - Powerful extensions mechanism for records and variants. > - Structured default values for optional fields. > > Compatibility between Piqi and Protocol Buffers is maintained as follows: > - Piqi specifications can be automatically converted to equivalent Protobuf > specifications and vice versa (with very few exceptions). > - Piqi and Protocol Buffers use the same binary encoding as a serialization > format. > > Piqi comes with several additional tools: > - Piq -- a high-level data representation language. It has full support for > Piqi > data types and it is easier to read, write and edit compared to XML or > JSON. > - Piqi tools for validating, pretty-printing and converting data between > Piq, > JSON and portable binary encoding. > > > Erlang support highlights > ========================= > > - Generated Erlang type and record definitions as well as > serialization/deserialization code are EEP8-compliant and provide all > necessary > type information for Dialyzer. > > - Support for recursive record types. > > - Support for Unicode strings which can be passed as string() (i.e. list of > Unicode codepoints) or utf-8 binary(). > > - Support for custom Erlang identifiers. For example, if some identifier > from > existing Proto or Piqi specification conflicts with an Erlang keyword, it > is > possible to define a custom Erlang-specific identifier without changing > the > original one and breaking backward compatibility. > > - Capitalized and "CamelCase" Piqi/Protobuf identifiers are automatically > converted to uncapitalized and "camel_case" Erlang ids. > > - Configurable prefixes for type and record identifiers (since Erlang > doesn't > have any kind of namespace support for types and records, we need to > prefix > them to avoid conflicts with names defined in other modules). > > - Tested with Erlang/OTP R13B04 and R14A. > > > Further references > ================== > > Homepage: > > http://piqi.org > > GitHub repository and several Erlang examples: > > http://github.com/alavrik/piqi > http://github.com/alavrik/piqi/tree/master/examples/addressbook/ > > Details of Piqi to Erlang mapping: > > http://piqi.org/doc/mappings/erlang/ > > > Your comments, suggestions and contributions are welcome! Don't hesitate to > contact me if you have any questions. > > > Anton > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- http://www.linkedin.com/in/torbenhoffmann From max.lapshin@REDACTED Mon Sep 13 14:58:25 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 13 Sep 2010 16:58:25 +0400 Subject: ets vs list Message-ID: erlyvideo has video streams inside, each video stream is an object, that tracks list of subscribed clients. Each client can be in several states. On each incoming frame clients are checked in this list and frame is delivered to each. Also, clients may change their state and it happens more rare, than selecting them from list. Proportions are so: from 1000 clients, about 850 are usually in state "active", 120 in "starting", 30 are "paused" "active" clients receive each frame. I was using ets to store their pids, but today I've created benchmark: http://gist.github.com/577086 and it seems, that my choice was wrong: 15> c(bm_ets). {ok,bm_ets} 16> bm_ets:run(). Initialized, starting [RO] ETS: 3987268, List: 422858 [RW] ETS: 4043659, List: 1213914 ok I can interpret these results so: list is 10 times faster than ets on often full table scans and is 4 times faster, than 50/50:scan and update My questions are: 1) are my conclusions right? 2) what ets is for, if list is fast enough on inserts? ets happened to be unusable in creating timeshift (storing large binaries) because of copy, it is much slower on extracting large amount of small objects. Where should I use it? From ulf.wiger@REDACTED Mon Sep 13 15:11:27 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 15:11:27 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: Message-ID: <4C8E22FF.1070000@erlang-solutions.com> On 09/13/2010 02:58 PM, Max Lapshin wrote: > > My questions are: > 1) are my conclusions right? > 2) what ets is for, if list is fast enough on inserts? ets happened to > be unusable in creating timeshift (storing large binaries) because of > copy, it is much slower on extracting large amount of small objects. > Where should I use it? In your benchmark, you are doing full scans of the data set. Lists are hard to beat then. Ets tables are primarily good when you need random access, in which case lists are pretty bad, with O(N) complexity. On the other hand, for small sets, even lists can be quite competitive for random access. BR, Ulf W From alexey.v.romanov@REDACTED Mon Sep 13 15:16:10 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Mon, 13 Sep 2010 17:16:10 +0400 Subject: Make clean doesn't clean some .o files Message-ID: At least, when cross-compiling. Namely, I got stuck with code compiled for Intel 30386 and not removed by `make clean` in otp_src/erts/emulator/zlib/obj/arm-none-linux-gnueabi/opt/*.o otp_src/erts/emulator/pcre/obj/arm-none-linux-gnueabi/opt/*.o Is this a bug in the build system? Yours, Alexey Romanov From jesper.louis.andersen@REDACTED Mon Sep 13 15:45:47 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 13 Sep 2010 15:45:47 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: Message-ID: On Mon, Sep 13, 2010 at 2:58 PM, Max Lapshin wrote: > and it seems, that my choice was wrong: > In addition to the comments of Ulf: When you run the ets:select/2 call, you are copying all of the ETS table into the memory space of the process and then operate on it. You don't need that for the List case since the memory is already in the heap of the process (Line 47). Line 48 can be merged with the match-spec in Line 46 so you avoid to create yet another list. This will also mean you (at least in theory) copy less data. In Haskell, you can rely on the optimizer to fuse several traversals of a list into a single run over the list. This is not the case for strict/eager languages, so you need to optimize list access to only run a single pass. -- J. From rapsey@REDACTED Mon Sep 13 15:49:39 2010 From: rapsey@REDACTED (Rapsey) Date: Mon, 13 Sep 2010 15:49:39 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: Message-ID: If you need random access and fast traversal, dict and gb_trees are all good and quite fast options. Sergej On Mon, Sep 13, 2010 at 3:45 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Mon, Sep 13, 2010 at 2:58 PM, Max Lapshin > wrote: > > > and it seems, that my choice was wrong: > > > > In addition to the comments of Ulf: When you run the ets:select/2 > call, you are copying all of the ETS table into the memory space of > the process and then operate on it. You don't need that for the List > case since the memory is already in the heap of the process (Line 47). > Line 48 can be merged with the match-spec in Line 46 so you avoid to > create yet another list. This will also mean you (at least in theory) > copy less data. > > In Haskell, you can rely on the optimizer to fuse several traversals > of a list into a single run over the list. This is not the case for > strict/eager languages, so you need to optimize list access to only > run a single pass. > > -- > J. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From nick@REDACTED Mon Sep 13 16:10:20 2010 From: nick@REDACTED (Niclas Eklund) Date: Mon, 13 Sep 2010 16:10:20 +0200 (CEST) Subject: [erlang-questions] Shutting down part of a supervision tree In-Reply-To: <4C8DDEB2.2030606@aist.go.jp> References: <4C8DDEB2.2030606@aist.go.jp> Message-ID: Hello! The Erlang CORBA objects are roughly gen_server-wrappers (using the create option {pseudo, true} is the exception since these objects are not processes). Hence, you should have a look at the behavior of gen_server (http://www.erlang.org/doc/man/gen_server.html#Module:terminate-2) regarding the terminate function. E.g if the process trap exit signals, the IDL file(s) has been compiled using the IC option handle_info or not etc. These links will probably be usefull to read as well: * http://www.erlang.org/doc/man/Module_Interface.html * http://www.erlang.org/doc/apps/orber/ch_stubs.html Best regards, Niclas @ Erlang/OTP On Mon, 13 Sep 2010, Geoffrey Biggs wrote: > Evening all, > > What is the accepted procedure for shutting down a part of a supervision > tree? > > In my case, I have several CORBA objects that come and go. I'm managing > them in a supervisor tree. The same supervisor that manages the CORBA > objects also manages other Erlang processes. They are all meant to be > shut down together; i.e. the supervisor, the workers and the CORBA > objects operate as a single unit that comes and goes as necessary. > > The problem is that when they go, they need to de-register from a naming > service. I've tried adding the de-registration call to the terminate/2 > function in the CORBA object provider, but I can't seem to find > something that guarantees it gets called in a timely fashion, if at all. > This causes my tests to fail, because the next test tries to create a > new instance of the object and can't register it due to the old > registration still existing on the name server (shows the tests are > doing their job, though). Methods I've tried to shut down the whole > construction are: > > -Calling "exit(S, normal)" on the supervisor. > -Directly casting stop to the CORBA objects, then calling "exit(S, > normal)" on the supervisor. > -Same as above, but with various sleeps in the hope that this gives the > CORBA objects time to do their thing (even though the child_specs give > them 30 seconds to shut down). > -Using supervisor:terminate_child on the CORBA objects. > > So far the best combination seems to be casting stop to the CORBA object > (which is what I would expect to work), but even so, the supervisor > seems to just dump the whole lot before this cast has time to take > effect half the time - and it's not even consistent. > > Any advice on shutting down parts of a supervisor tree is appreciated. > > Geoff > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ulf.wiger@REDACTED Mon Sep 13 16:29:21 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 16:29:21 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: Message-ID: <4C8E3541.7050400@erlang-solutions.com> On 09/13/2010 02:58 PM, Max Lapshin wrote: > > 2) what ets is for, if list is fast enough on inserts? The main unique feature of ets is that data residing in ets is not garbage-collected. Roughly speaking, the cost of GC is proportional to the amount of live data, so if you have very large data sets, process heap-based data will have an increasing hidden cost in GC sweep/copy time, even though access times may look excellent in benchmarks. Running benchmarks for longer periods should allow you to include amortized GC cost in the results, but you will also get a bunch of other noise. Tracing on GC events with timestamps may be a more accurate method. BR, Ulf From zeno490@REDACTED Mon Sep 13 16:41:00 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Mon, 13 Sep 2010 10:41:00 -0400 Subject: [erlang-questions] erlang:max/2 and erlang:min/2 In-Reply-To: <33A3A8B6-098C-4C1C-8C8A-16A4BF38A6AF@cs.otago.ac.nz> References: <33A3A8B6-098C-4C1C-8C8A-16A4BF38A6AF@cs.otago.ac.nz> Message-ID: Yes precisely, I was arguing mixed mode arithmetic should be banned or have no defined behavior to force programmers to think and be sure to know what they are doing. I agree with your claim that the behavior is currently unintuitive and prone to error. I simply meant to illustrate erlang isn't alone in that respect and that when one programs in a language, one has to know the potential pitfalls associated with it... I wouldn't be surprised if at this point they were unwilling to correct this situation considering it is of fairly high impact. Although perhaps it could be done with compiler/erl switches to turn on/off the old/new behavior. The bit about the ternary operator is to show that for most of these languages, a similar construct is likely used which causes order to matter. In erlang, it could have been done as follow (wild guess as it is probably a nif but likely of a similar nature given the observed result): max(A, B) when A >= B -> A; max(_, B) -> B. A more correct implementation could have been (not actually compiled to test but you get the idea): %% Integer/integer max(A, B) when is_integer(A) and is_integer(B) and A >= B -> A; max(A, B) when is_integer(A) and is_integer(B) -> B; %% Float/float max(A, B) when is_float(A) and is_float(B) and A >= B -> A; max(A, B) when is_float(A) and is_float(B) -> B; %% Mixed integer/float coerced result as float but could be made to raise an exception/exit process max(A, B) when is_integer(A) and is_float(B) and float(A) >= B -> float(A); %% Not sure if float() can be used in a guard? max(A, B) when is_integer(A) and is_float(B) -> B; max(A, B) when is_float(A) and is_integer(B) and A >= float(B) -> A; max(A, B) when is_float(A) and is_integer(B) -> float(B); %% General case of other types, would it make sense though? max(A, B) when A >= B -> A; max(_, B) -> B. Unfortunately, the above doesn't extend to tuples/lists/other types. *** Side note *** Also, while it is true the ternary operator can't always be used as an l-value, some compilers still allow it as it must have been once part of the standard. For example, wikipedia still mentions it at: http://en.wikipedia.org/wiki/%3F: The link I included (which you may not have read) also says: "The type of the result is the common type, and it is an l-value if both the second and third operands are of the same type and both are l-values." Unfortunately I can't find a copy of the C++ standard laying around but here are some results on the compilers I currently have on hand: In VS2008, the below compiles for win32, for the xbox360 (somewhat expected since they are both Microsoft compilers) and perhaps surprisingly on sony's compiler for the PS3 (PPU) (which is oftentimes more strict than GCC but obviously not for this special case) : void* A; void* B; ((someCondition) ? A : B) = NULL; Alas, I digress. Nicholas On Sun, Sep 12, 2010 at 10:43 PM, Richard O'Keefe wrote: > > On Sep 11, 2010, at 11:53 AM, Nicholas Frechette wrote: > > > This is fairly standard as far as I know in languages without strong > typing in function arguments (when that is the case, usually coercion > happens which hides this behavior from you). > > For example in ruby: > > >> [1, 1.0].min > > => 1 > > >> [1.0, 1].min > > => 1.0 > > The traditional retort to this is "And if all your friends jumped off > a cliff, would you copy that too?" > > > > For these languages (and erlang), order DOES matter. > > And all that means is that they are buggy too. > > > > > This isn't unlike a naive implementation of min/max in C++: > > #define max(A, B) A >= B ? A : B > > Er, that is a couple of steps of beginnerness _below_ "naive". > #define max(A, B) ((A) >= (B) ? (A) : (B)) > is normally required, and of course the preferred method in C++ > is > template > T max(T a, T b) { > return a >= b ? a : b; > } > for obvious reasons. > > > > However, even using c++0x: > > auto SomeResult = max(1, 1.0); // what happens here? what is the type of > SomeResult? (float) See note below > > auto SomeResult = max(1.0, 1); // what happens here? what is the type of > SomeResult? (float) See note below > > What happens here is quite irrelevant to dynamically typed languages. > The if-then-else operator has to return a result of *some* definite type, > so there really are only two choices for it. > Either it demands that both choices be the same type, > or else it has to convert at least one of them. > > This has no significance whatever for a language that *doesn't* require > all arms of an if to have the same type and *doesn't* have to convert. > > > > *** Note below (really a side note, you can skip this) *** > > The 'ternary' operator is special (in C/C++). In the case above, both A > and B can't be of different types. > > I think "can't" was meant to be "can". > > > The type will be coerced to a common base type only if possible, > otherwise compilation will fail. > > See for details: > http://msdn.microsoft.com/en-us/library/e4213hs1%28VS.71%29.aspx (applies > to other compilers as well) > > The ternary operator can also be used as an l-value (for fun and profit) > :) > > NOT IN STANDARD C OR C++ IT CANNOT! > > > ((A >= B) ? A : B) = 123; // Don't do this.. otherwise your coworkers > will want to kill me for teaching you > > *** > > One reason they will be displeased with you is that most compilers reject > this. > A typical error message: > "foo.c", line 2: warning: conditional expression is not a valid > lvalue > > > > > > In C++ at least, both types will be converted to a common base type with > the ternary operator or you'll traditionally write functions which will > coerce the arguments for you in order to use hardware specific instructions > to do the selection: > > max(A,B) becomes 2 instructions for floats/doubles (on powerpc, same for > min): C = A - B, C >= 0 ? B : A (powerpc has an instruction to compare a > float against zero and select the result (fsel)) > > On a SPARC or a Pentium or an ARM, it would be > [assume A is in register x] > [assume B is in register y] > [assume result is wanted in register x] > compare x with y one compare instruction > move y to x if x < y one conditional move > > It is not clear to me how this is supposed to be relevant to Erlang, > which has to deal with mixed integer/float comparisons, or to the > question of how you implement max/2 and min/2 so that they satisfy > the laws that a programmer *ought* to be able to take for granted. > > C and C++ can get away with (user-provided) definitions that convert > basically because they can't have arrays whose elements are of varying > types, so if you do > auto x = a[0]; > for (int i = 1; i < n; i++) x = max(x, a[i]); > there _can't_ be any problems introduced by conversion, because > all the a[i] must be the *same* type. > > But in Erlang, the elements of a list CAN be of different types. > > Intuitions based on experience with strongly typed languages > (Ada, Clean, Haskell) would tell you that mixed-mode arithmetic > should simply be banned. Any time it is intentional, an explicit > coercion can be provided by the programmer. > > Intuitions based on experience with weakly typed languages > (C, C++, Java, C#) would mislead you into thinking that operations > that don't actually obey the mathematical laws they are supposed > to are OK because thanks to the all-elements-of-an-array-are-the- > same-type setup, uses of < and max DO obey the laws in question. > (Sorting a floating-point array is unproblematic in C > *IF* (1) there is at most one -infinity > (2) there is at most one +infinity > (3) there are no NaNs > (4) either you don't care about the order of -0 and +0 > or they don't both occur. > Sorting a mixed integer/float array just can't happen.) > > > > From mk@REDACTED Mon Sep 13 16:41:42 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 16:41:42 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E3541.7050400@erlang-solutions.com> References: <4C8E3541.7050400@erlang-solutions.com> Message-ID: <4C8E3826.30004@amberbio.com> What exactly does it mean that data residing in ets isn't garbage collected? Does it mean that if a {key, Value} pair is in the ets table T, and I write ets:delete(T, Key) then Value will not be garbage collected. Morten. On 9/13/10 4:29 PM, Ulf Wiger wrote: > On 09/13/2010 02:58 PM, Max Lapshin wrote: >> 2) what ets is for, if list is fast enough on inserts? > The main unique feature of ets is that data residing in > ets is not garbage-collected. > > Roughly speaking, the cost of GC is proportional to the > amount of live data, so if you have very large data sets, > process heap-based data will have an increasing hidden > cost in GC sweep/copy time, even though access times may > look excellent in benchmarks. > > Running benchmarks for longer periods should allow you > to include amortized GC cost in the results, but you > will also get a bunch of other noise. Tracing on GC > events with timestamps may be a more accurate method. > > BR, > Ulf > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From max.lapshin@REDACTED Mon Sep 13 16:44:15 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 13 Sep 2010 18:44:15 +0400 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E3541.7050400@erlang-solutions.com> References: <4C8E3541.7050400@erlang-solutions.com> Message-ID: But data from ets will be copied on each incoming frame, so it will also be GC-ed From nick@REDACTED Mon Sep 13 16:46:12 2010 From: nick@REDACTED (Niclas Eklund) Date: Mon, 13 Sep 2010 16:46:12 +0200 (CEST) Subject: [erlang-questions] Re-entrant CORBA services In-Reply-To: <87hbhum2ih.fsf@pingviini.priv> References: <4C8DDFC1.1010101@aist.go.jp> <87hbhum2ih.fsf@pingviini.priv> Message-ID: Hello! The drawback with oneway CORBA operations (i.e. asynchronous calls/gen_server:cast) is that the exact behavior isn't specified by the OMG specifications, which might give different results dependening on which ORBs you're using. Besides using a oneway operation, perhaps you can use corba:reply/2 (see http://www.erlang.org/doc/man/corba.html#reply-2), which is the same thing as gen_server:reply/2. If it's posssible for you to use Orber pseudo objects, you'll be able to avoid deadlock (see for example the CosNaming_NamingContextExt_impl.erl module how these can be implemented). The exact answer depends on if A and B resides on the same ORB (Orber I assume in this case) or inter-ORB communication, with an ORB from another vendor, takes place. Best regards, Niclas @ Erlang/OTP On Mon, 13 Sep 2010, Jani Hakala wrote: > Geoffrey Biggs writes: > >> Unfortunately, one of the calls to object B requires its own calls back >> to object A to finish (whether or not it succeeds). This means that >> object B blocks waiting for object A to respond, while object A is >> blocked waiting for object B to respond. >> > How about dividing this call from object A to object B to two calls > to object B? First one would return at that point where B now calls A > and the second call would give the object B the results of the > call. Any parameters required by object A could be passed as the > result from the first call. > > AMI/AMH [1] might provide another solution if they are available. Or > perhaps multiple servants for object A. > > I prefer to avoid two-way calls with CORBA as much as possible. I > once tried those with C++ and ACE/TAO, and it seemed to be too much > trouble. I didn't try AMI/AMH or multiple servants though as I was > just learning CORBA basics. > > Jani Hakala > > [1] asynchronous method invocation, asynchronous method handling. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ulf.wiger@REDACTED Mon Sep 13 16:58:04 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 16:58:04 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E3826.30004@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> Message-ID: <4C8E3BFC.8060103@erlang-solutions.com> On 09/13/2010 04:41 PM, Morten Krogh wrote: > What exactly does it mean that data residing in ets isn't > garbage collected? > > Does it mean that if a {key, Value} pair is in the ets table T, > and I write > > ets:delete(T, Key) > > then Value will not be garbage collected. > > Morten. If you delete an object from an ets table, it will be freed on the spot - think malloc/free. If you don't explicitly delete it, it will stay there, and will never be touched by a garbage collector. The garbage collector cares about process heaps and binaries, but not about ets. Once an object is copied into ets, it is outside the jurisdiction of the GC; when read from ets, it is copied onto the process heap, and this copy becomes subject to GC. Erlang's per-process garbage collector periodically copies live data to a new instance of the process heap. There are a few variations of the theme, but basically, that's how it works. When some data is no longer referenced, it becomes garbage on the heap, and eventually, the heap will fill up with garbage and live data, triggering the garbage collector. In this sense, GC is really a misnomer, since it collects, and copies, live data, rather than messing about with the garbage. This is also the reason why GC cost can be said to be proportional to the amount of live data, rather than the amount of garbage. Producing lots of garbage, though, will trigger GC more frequently, causing more scans and copying of the live data. BR, Ulf W From rvirding@REDACTED Mon Sep 13 16:58:44 2010 From: rvirding@REDACTED (Robert Virding) Date: Mon, 13 Sep 2010 16:58:44 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E3826.30004@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> Message-ID: All data in an ets table will be garbage collected! It is perfectly safe to insert and delete data as often as you need. What Ulf meant was that an ets table is not garbage collected in the same way as a normal process and that you don't run into same problems with overly long garbage collections times as you can with a processes that have a *LOT* of local data. This is why it is safe to have large ets tables and why you should be careful in having *LARGE* databases as local data to a process. This is a problem using dict, gb_trees and array, and lists. The downside of this is that data has to be copied from an ets table into the process heap before it can be used, which is why we have ets:match and ets:select which allow you to perform more tests on potential data before copying them into the process heap. This is a benefit using dict, gb_trees and array, and lists. So they are all *safe* but perform differently. Robert On 13 September 2010 16:41, Morten Krogh wrote: > ?What exactly does it mean that data residing in ets isn't garbage > collected? > > Does it mean that if a {key, Value} pair is in the ets table T, > and I write > > ets:delete(T, Key) > > then Value will not be garbage collected. > > Morten. > > On 9/13/10 4:29 PM, Ulf Wiger wrote: >> >> On 09/13/2010 02:58 PM, Max Lapshin wrote: >>> >>> 2) what ets is for, if list is fast enough on inserts? >> >> The main unique feature of ets is that data residing in >> ets is not garbage-collected. >> >> Roughly speaking, the cost of GC is proportional to the >> amount of live data, so if you have very large data sets, >> process heap-based data will have an increasing hidden >> cost in GC sweep/copy time, even though access times may >> look excellent in benchmarks. >> >> Running benchmarks for longer periods should allow you >> to include amortized GC cost in the results, but you >> will also get a bunch of other noise. Tracing on GC >> events with timestamps may be a more accurate method. >> >> BR, >> Ulf >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From alan@REDACTED Mon Sep 13 17:08:52 2010 From: alan@REDACTED (Alan Labouseur) Date: Mon, 13 Sep 2010 11:08:52 -0400 Subject: =?windows-1252?Q?What_OS_threads_get_used_in_Erlang=92s_abstract?= =?windows-1252?Q?_machine,_BEAM=3F?= Message-ID: <9B99B9F9-A868-4EA9-860A-B3F17F23970C@labouseur.com> Folks, I?ve begun studying Erlang and find the BEAM runtime environment fascinating. It?s commonly stated that in Erlang, processes belong to the language rather than the OS (meaning the runtime, meaning BEAM in this case). These are the lightweight, ?green processes? that Erlang is getting famous for. It?s further stated (on page 5 of http://www.cs.ucsb.edu/~puneet/reports/erlang.pdf) that BEAM uses one (1) OS thread per CPU core for scheduling and another OS thread for i/o. So I wonder: From what thread do the CPU cycles needed to actually execute Erlang code come from? Further, if I?m running on a dual core machine I would expect -- based on what I?ve read so far -- to see three (3) threads running under the BEAM process: two schedulers (one for each core) and one i/o thread. But I see 10. Sometimes 11. Sometimes it starts at 13 and, like high-quality amplifiers, goes to 11. I?m confused. Any insight will be appreciated. Thanks, - Alan Labouseur From alan@REDACTED Mon Sep 13 17:08:52 2010 From: alan@REDACTED (Alan Labouseur) Date: Mon, 13 Sep 2010 11:08:52 -0400 Subject: =?windows-1252?Q?What_OS_threads_get_used_in_Erlang=92s_abstract?= =?windows-1252?Q?_machine,_BEAM=3F?= Message-ID: <9B99B9F9-A868-4EA9-860A-B3F17F23970C@labouseur.com> Folks, I?ve begun studying Erlang and find the BEAM runtime environment fascinating. It?s commonly stated that in Erlang, processes belong to the language rather than the OS (meaning the runtime, meaning BEAM in this case). These are the lightweight, ?green processes? that Erlang is getting famous for. It?s further stated (on page 5 of http://www.cs.ucsb.edu/~puneet/reports/erlang.pdf) that BEAM uses one (1) OS thread per CPU core for scheduling and another OS thread for i/o. So I wonder: From what thread do the CPU cycles needed to actually execute Erlang code come from? Further, if I?m running on a dual core machine I would expect -- based on what I?ve read so far -- to see three (3) threads running under the BEAM process: two schedulers (one for each core) and one i/o thread. But I see 10. Sometimes 11. Sometimes it starts at 13 and, like high-quality amplifiers, goes to 11. I?m confused. Any insight will be appreciated. Thanks, - Alan Labouseur From mk@REDACTED Mon Sep 13 17:16:17 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 17:16:17 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> Message-ID: <4C8E4041.1080301@amberbio.com> Ulf and Robert, thanks for your answers. It makes sense that delete calls free in the malloc sense. What I don't get now, is what data the data in ets that is not garbage collected, actually is. Does the ets process have data beyond the table itself? Are we talking about intermediate values used in the match function etc? Morten. On 9/13/10 4:58 PM, Robert Virding wrote: > All data in an ets table will be garbage collected! It is perfectly > safe to insert and delete data as often as you need. > > What Ulf meant was that an ets table is not garbage collected in the > same way as a normal process and that you don't run into same problems > with overly long garbage collections times as you can with a processes > that have a *LOT* of local data. This is why it is safe to have large > ets tables and why you should be careful in having *LARGE* databases > as local data to a process. This is a problem using dict, gb_trees and > array, and lists. > > The downside of this is that data has to be copied from an ets table > into the process heap before it can be used, which is why we have > ets:match and ets:select which allow you to perform more tests on > potential data before copying them into the process heap. This is a > benefit using dict, gb_trees and array, and lists. > > So they are all *safe* but perform differently. > > Robert > > > On 13 September 2010 16:41, Morten Krogh wrote: >> What exactly does it mean that data residing in ets isn't garbage >> collected? >> >> Does it mean that if a {key, Value} pair is in the ets table T, >> and I write >> >> ets:delete(T, Key) >> >> then Value will not be garbage collected. >> >> Morten. >> >> On 9/13/10 4:29 PM, Ulf Wiger wrote: >>> On 09/13/2010 02:58 PM, Max Lapshin wrote: >>>> 2) what ets is for, if list is fast enough on inserts? >>> The main unique feature of ets is that data residing in >>> ets is not garbage-collected. >>> >>> Roughly speaking, the cost of GC is proportional to the >>> amount of live data, so if you have very large data sets, >>> process heap-based data will have an increasing hidden >>> cost in GC sweep/copy time, even though access times may >>> look excellent in benchmarks. >>> >>> Running benchmarks for longer periods should allow you >>> to include amortized GC cost in the results, but you >>> will also get a bunch of other noise. Tracing on GC >>> events with timestamps may be a more accurate method. >>> >>> BR, >>> Ulf >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ulf.wiger@REDACTED Mon Sep 13 17:43:38 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 17:43:38 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E4041.1080301@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> Message-ID: <4C8E46AA.9050301@erlang-solutions.com> On 09/13/2010 05:16 PM, Morten Krogh wrote: > Ulf and Robert, > thanks for your answers. It makes sense that delete calls free in the > malloc sense. > > What I don't get now, is what data the data in ets that is not garbage > collected, actually is. Hmm, not sure I understand the question, but data copied into ETS is managed separately from the process heap data. The Erlang VM has a system of "memory carriers" to manage the allocation of different types of data, in order to fight memory fragmentation. ETS data, which tends to be more long lived and of different granularity than process heaps, is stored in separate carrier blocks. Binaries are stored in other types of carriers, etc. But they are just regular Erlang objects. Basically, you have a few different kinds of data inside the VM. To name a few: - The atom table, is never garbage-collected; atoms are never deleted - ETS tables, whole tables are automatically removed when the owner process dies; individual objects remain while the table remains, unless explicitly deleted - Large binaries, reside on a shared heap; they are reference-counted - Process heaps, are managed with a stop-and-copy garbage collector; when a process dies, the whole process heap is freed, informing monitoring and linked processes, as well as decrementing binary reference counters and deleting any ETS tables owned by the process. > Does the ets process have data beyond the table itself? > Are we talking about intermediate values used in the match function etc? Ah, the select function actually executes in a special 'pseudo process' inside the Erlang VM. It has its own little stack machine, called the PAM ("PAM" stands for "Patrik's Abstract Machine", I think, where "Patrik" stands for Patrik Nyblom). erts/emulator/beam/erl_db_util.c So garbage resulting from a select operation accumulates on the pseudo process heap and is subject to the same kind of GC as any regular process (I assume). There's a _lot_ to memory management in the Erlang VM. I don't claim to understand more than a tiny part of it. BR, Ulf W From alavrik@REDACTED Mon Sep 13 17:50:41 2010 From: alavrik@REDACTED (Anton Lavrik) Date: Mon, 13 Sep 2010 10:50:41 -0500 Subject: [erlang-questions] Announce: the Piqi project In-Reply-To: <4C8DDF72.1050004@gmail.com> References: <20100913080735.GA23321@debian.localdomain> <4C8DDF72.1050004@gmail.com> Message-ID: On Mon, Sep 13, 2010 at 3:23 AM, Evgeniy Khramtsov wrote: > 13.09.2010 18:07, Anton Lavrik wrote: >> >> It is my pleasure to announce the Piqi project to the Erlang community. >> I've >> just released a new version which features Erlang support. >> > > Wow, that sounds impressive. The only problem I see is the license... What problem with the license do you mean? Apache v2 is one of the most permissive and clear open-source licenses. The only part of Piqi source distribution that links with Erlang programs is Piqi runtime support library, which is literally a single Erlang module: http://github.com/alavrik/piqi/blob/master/piqirun-erlang/src/piqirun.erl In addition to Apache v2 license header it also contain a copyright notice and a copy of MIT license from protobuffs library (http://github.com/ngerakines/erlang_protobuffs), since I reused some code from it. MIT license is as permissive as Apache v2. Anton From xramtsov@REDACTED Mon Sep 13 18:03:58 2010 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Tue, 14 Sep 2010 02:03:58 +1000 Subject: [erlang-questions] Announce: the Piqi project In-Reply-To: References: <20100913080735.GA23321@debian.localdomain> <4C8DDF72.1050004@gmail.com> Message-ID: <4C8E4B6E.8090804@gmail.com> 14.09.2010 01:50, Anton Lavrik wrote: > On Mon, Sep 13, 2010 at 3:23 AM, Evgeniy Khramtsov wrote: > >> >> Wow, that sounds impressive. The only problem I see is the license... >> > What problem with the license do you mean? > > Apache v2 is one of the most permissive and clear open-source > licenses. The only part of Piqi source distribution that links with > Erlang programs is Piqi runtime support library, which is literally a > single Erlang module: > http://github.com/alavrik/piqi/blob/master/piqirun-erlang/src/piqirun.erl > > In addition to Apache v2 license header it also contain a copyright > notice and a copy of MIT license from protobuffs library > (http://github.com/ngerakines/erlang_protobuffs), since I reused some > code from it. MIT license is as permissive as Apache v2. > Indeed, I see. Sorry, I've misunderstood the license :( Thank you for the explanation and your project :) -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From mk@REDACTED Mon Sep 13 18:10:21 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 18:10:21 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E46AA.9050301@erlang-solutions.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> Message-ID: <4C8E4CED.4040602@amberbio.com> Thanks again. My question might have been hard to understand because I made an assumption that might be wrong, namely that an ets table has a corresponding process. I don't mean the owning process, but an implicit process that is spawned automatically when the table is created, and that all insertions and deletions are going through that process. Is that just my imagination? Is an ets table only a hash table and nothing more? No corresponding process that serializes all access to it? ETS:lookup is just a function call, not a hidden message pass, receive construct or what? Morten. On 9/13/10 5:43 PM, Ulf Wiger wrote: > On 09/13/2010 05:16 PM, Morten Krogh wrote: >> Ulf and Robert, >> thanks for your answers. It makes sense that delete calls free in the >> malloc sense. >> >> What I don't get now, is what data the data in ets that is not garbage >> collected, actually is. > Hmm, not sure I understand the question, but data copied into ETS is > managed separately from the process heap data. The Erlang VM has > a system of "memory carriers" to manage the allocation of different > types of data, in order to fight memory fragmentation. ETS data, which > tends to be more long lived and of different granularity than process > heaps, is stored in separate carrier blocks. Binaries are stored in > other types of carriers, etc. But they are just regular Erlang objects. > > Basically, you have a few different kinds of data inside the VM. > To name a few: > - The atom table, is never garbage-collected; atoms are never deleted > - ETS tables, whole tables are automatically removed when the owner > process dies; individual objects remain while the table remains, > unless explicitly deleted > - Large binaries, reside on a shared heap; they are reference-counted > - Process heaps, are managed with a stop-and-copy garbage collector; > when a process dies, the whole process heap is freed, informing > monitoring and linked processes, as well as decrementing binary > reference counters and deleting any ETS tables owned by the process. > >> Does the ets process have data beyond the table itself? >> Are we talking about intermediate values used in the match function etc? > > Ah, the select function actually executes in a special 'pseudo process' > inside the Erlang VM. It has its own little stack machine, called the > PAM ("PAM" stands for "Patrik's Abstract Machine", I think, where > "Patrik" stands for Patrik Nyblom). > > erts/emulator/beam/erl_db_util.c > > So garbage resulting from a select operation accumulates on the > pseudo process heap and is subject to the same kind of GC as > any regular process (I assume). > > There's a _lot_ to memory management in the Erlang VM. I don't > claim to understand more than a tiny part of it. > > BR, > Ulf W > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From pascalchapier@REDACTED Mon Sep 13 18:15:05 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Mon, 13 Sep 2010 18:15:05 +0200 Subject: [erlang-questions] Shutting down part of a supervision tree Message-ID: Thank you Niclas, I was fighting with a similar problem for several days, and I didn't notice the condition for terminate function execution. With trap_exit = true, everything work as expected with my first (and simple) version... Best regards, Pascal. From alavrik@REDACTED Mon Sep 13 18:17:25 2010 From: alavrik@REDACTED (Anton Lavrik) Date: Mon, 13 Sep 2010 11:17:25 -0500 Subject: [erlang-questions] Announce: the Piqi project In-Reply-To: References: <20100913080735.GA23321@debian.localdomain> Message-ID: Hi Torben, I'm not planning on adding support of other binary encodings. One of the reasons why I started the project was that I didn't want to deal with custom binary or text encodings for application-level protocols and use a single general-purpose system instead. For this reason, I picked one binary encoding and built everything around it. Adding support for another binary encoding will clearly break this model and, also, break compatibility with Google Protocol Buffers, which is one of the key features. So Piqi is definitely not a good match for what you're looking for, since you already have a binary protocol that you need to parse, and Piqi is clearly not a protocol parsing tool. If you're targeting Erlang platform, I can't think of anything simpler than just using plain Erlang and its bit syntax for parsing protocols. This way you can take advantage of Erlang's flexibility and get fairly high-level and expressive code for protocol parsers. If you need lower-level or high-performance parsers, you might want to take a look at BinPAC. I haven't used it myself, but it looks relevant to what you've described: http://www.bro-ids.org/wiki/index.php/BinPAC Best, Anton On Mon, Sep 13, 2010 at 6:14 AM, Torben Hoffmann wrote: > Hi Anton, > > It looks very interesting - in fact so much that I will ask a few questions > (always a good sign, right?). > > On the higher level: Are there any plans for providing a way to specify your > own binary encoding? > > On the lower level: If I have a PDU where one field is present if another > field has a specific value how would that look in Piqi? And what if the > dependent field is separated from its condition field by many other fields? > > Example: (M means mandatory, C means conditional) > > Field1 M > Field2 M > . > Field15 C - present if Field2 is 4. > > I am dealing with a protocol that has this kind of behaviour and I am > looking into writing my own binary grammar, but I would much rather piggy > bag on something else if many reasons on top of laziness ;-) > > Cheers, > Torben > > On Mon, Sep 13, 2010 at 10:07, Anton Lavrik wrote: >> >> It is my pleasure to announce the Piqi project to the Erlang community. >> I've >> just released a new version which features Erlang support. From krab@REDACTED Mon Sep 13 18:10:52 2010 From: krab@REDACTED (Kresten Krab Thorup) Date: Mon, 13 Sep 2010 18:10:52 +0200 Subject: =?Windows-1252?Q?Re:_[erlang-questions]_What_OS_threads_get_used_in_Erlan?= =?Windows-1252?Q?g=92s_abstract_machine,_BEAM=3F?= In-Reply-To: <9B99B9F9-A868-4EA9-860A-B3F17F23970C@labouseur.com> References: <9B99B9F9-A868-4EA9-860A-B3F17F23970C@labouseur.com> Message-ID: <926B736C-252E-4A41-A7A0-4E93C0A698DE@trifork.com> Here is AFAIK, the basic scenario: Erlang code will be run in as many "green threads" as there are processes; the process limit is controlled by the +P flag. The green threads are mapped on to S threads, where S is the number of cores/CPUs. The fact that these threads are also called schedulers can seem somewhat confusing, but from the VMs point of view they are. From the developer's point of view, they are the threads that run your erlang code. The number S can be controlled with the +S option to the erl command line. In addition hereto, there are a number of so-called "Async Threads". That's a thread pool which is used by I/O processes called linked in drivers, to react to select / poll etc. The number of asynch threads is dynamic, but limited by the +A flag. So, the 11 threads you see on a dual-core may be 2 schedulers, and 9 async threads. For instance. Read more about the flags here ... http://www.erlang.org/doc/man/erl.html Kresten On Sep 13, 2010, at 17:08 , Alan Labouseur wrote: Folks, I?ve begun studying Erlang and find the BEAM runtime environment fascinating. It?s commonly stated that in Erlang, processes belong to the language rather than the OS (meaning the runtime, meaning BEAM in this case). These are the lightweight, ?green processes? that Erlang is getting famous for. It?s further stated (on page 5 of http://www.cs.ucsb.edu/~puneet/reports/erlang.pdf) that BEAM uses one (1) OS thread per CPU core for scheduling and another OS thread for i/o. So I wonder: From what thread do the CPU cycles needed to actually execute Erlang code come from? Further, if I?m running on a dual core machine I would expect -- based on what I?ve read so far -- to see three (3) threads running under the BEAM process: two schedulers (one for each core) and one i/o thread. But I see 10. Sometimes 11. Sometimes it starts at 13 and, like high-quality amplifiers, goes to 11. I?m confused. Any insight will be appreciated. Thanks, - Alan Labouseur ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED Kresten Krab Thorup, CTO, Trifork From ulf.wiger@REDACTED Mon Sep 13 18:25:49 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 18:25:49 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E4CED.4040602@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> Message-ID: <4C8E508D.3040507@erlang-solutions.com> On 09/13/2010 06:10 PM, Morten Krogh wrote: > Thanks again. > > My question might have been hard to understand because I made an > assumption that might be wrong, namely that > an ets table has a corresponding process. I don't mean the owning > process, but an implicit process that is spawned automatically > when the table is created, and that all insertions and deletions are > going through that process. Is that just my imagination? > Is an ets table only a hash table and nothing more? No corresponding > process that serializes all access to it? ETS:lookup is just a function > call, not a hidden > message pass, receive construct or what? > > Morten. ETS tables do not have a corresponding process (except for the temporary pseudo process during select()). Serialization is done with mutexes, so essentially, it's just a hash table or B-tree. That said, Erlang purists often hide behind the fact that you _could_ model ETS tables with processes and message passing (thus, process spawning and messaging are still the only means of side-effecting in Erlang ;-) It's not terribly difficult, but hard to do efficiently. The main issue is named tables, since you need a table registry, which implies another set of messages. I did this once (and as it happened, Robert V was doing the same thing at the same time - I was doing it for the Erlang Processor, and he was doing it for his own VM, VEE). As far as I recall, performance of my ETS emulation was ca 30x slower than regular ETS, at least on named tables*. :) This was many years ago, so the numbers may be completely different today. BR, Ulf W * But, as it happened, the Erlang Processor used ca 30x fewer clock cycles than my workstation to execute Erlang code, so ultimately I thought myself no worse off. :) From mk@REDACTED Mon Sep 13 19:11:46 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 19:11:46 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E508D.3040507@erlang-solutions.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> Message-ID: <4C8E5B52.9030506@amberbio.com> Thanks. I just tried in the shell and saw that there were no new processes after creating an ETS table. fine. Surely, you could model ets with a process and message passing. Why should it be slower? The only overhead should be the message passing. So, I guess your benchmark might depend on the size of the key/value pairs. For large values, the performance should be the same, for small values the process might be a bit slower. So, if we compare ets with dict or the process dictionary, then ets should perform really badly on large values, it seems. Suppose we have a giant list, that we store in the tables with some key, i.e, store(key) = [..,..,,,...............................] If we want to traverse that list, dict and pd would start immediately, whereas the list from ets would have to be copied before traversing could start. And after reading the list from ets it would have to be freed at the next garbage collection, where as the dict and pd didn't generate any garbage. Morten. On 9/13/10 6:25 PM, Ulf Wiger wrote: > On 09/13/2010 06:10 PM, Morten Krogh wrote: >> Thanks again. >> >> My question might have been hard to understand because I made an >> assumption that might be wrong, namely that >> an ets table has a corresponding process. I don't mean the owning >> process, but an implicit process that is spawned automatically >> when the table is created, and that all insertions and deletions are >> going through that process. Is that just my imagination? >> Is an ets table only a hash table and nothing more? No corresponding >> process that serializes all access to it? ETS:lookup is just a function >> call, not a hidden >> message pass, receive construct or what? >> >> Morten. > ETS tables do not have a corresponding process (except for the > temporary pseudo process during select()). Serialization is done > with mutexes, so essentially, it's just a hash table or B-tree. > > That said, Erlang purists often hide behind the fact that you > _could_ model ETS tables with processes and message passing > (thus, process spawning and messaging are still the only > means of side-effecting in Erlang ;-) > > It's not terribly difficult, but hard to do efficiently. The > main issue is named tables, since you need a table registry, > which implies another set of messages. I did this once (and > as it happened, Robert V was doing the same thing at the same > time - I was doing it for the Erlang Processor, and he was > doing it for his own VM, VEE). As far as I recall, performance > of my ETS emulation was ca 30x slower than regular ETS, at > least on named tables*. :) This was many years ago, so the > numbers may be completely different today. > > BR, > Ulf W > > * But, as it happened, the Erlang Processor used ca 30x fewer > clock cycles than my workstation to execute Erlang code, so > ultimately I thought myself no worse off. :) > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ulf.wiger@REDACTED Mon Sep 13 20:12:32 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 13 Sep 2010 20:12:32 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E5B52.9030506@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> Message-ID: <4C8E6990.7000805@erlang-solutions.com> On 09/13/2010 07:11 PM, Morten Krogh wrote: > > > Thanks. I just tried in the shell and saw that there were no new > processes after creating an ETS table. fine. > > Surely, you could model ets with a process and message passing. Why > should it be slower? The only overhead should be the message passing. No, there are several other overheads. For instance, you must somehow implement a hash table from immutable dynamically typed data structures, whereas in ETS (in C), you can use mutable arrays. The dict module, for example, uses a tree structure of tuples. Once you've located the leaf tuple where the object is to reside, you have to create a new copy of that tuple, which means you must also update (copy) the parent tuple, etc. Also, when traversing the tree, you cannot avoid the type checks ensuring that it's actually a tuple you're stepping into. In the case of C, you will not be performing runtime type checks on the hash space. OTOH, with SMP Erlang, you have to protect the ETS tables with mutexes, whereas on the process heap, you don't have to worry about contention. > So, I guess your benchmark might depend on the size of the key/value > pairs. For large values, the performance should be the same, for small > values the process might be a bit slower. In the case I described, the task was to model the ETS API faithfully, which meant a separate process per table. This also included copying, as data sent in messages is copied from process to process. Otherwise, if you use dict, gb_trees, etc. or the process dictionary, you can avoid the copying and get quite good performance if the stored data structures are large. > So, if we compare ets with dict or the process dictionary, then ets > should perform really badly on large values, it seems. Yes. OTOH, the process dictionary is difficult to share... BR, Ulf W From mk@REDACTED Mon Sep 13 21:05:19 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 21:05:19 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E6990.7000805@erlang-solutions.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> Message-ID: <4C8E75EF.7000702@amberbio.com> > No, there are several other overheads. For instance, you must somehow > implement a hash table from immutable dynamically typed data structures, > whereas in ETS (in C), you can use mutable arrays. The dict module, > for example, uses a tree structure of tuples. Once you've located > the leaf tuple where the object is to reside, you have to create > a new copy of that tuple, which means you must also update (copy) > the parent tuple, etc. But it is to some extent self imposed that you want a functional data structure. You could have used the process dictionary for this or some foreign interface. Nowadays nif, but that was not available to you then. Why didn't you use the process dictionary? It is made for this situation. Morten. > Also, when traversing the tree, you cannot avoid the type checks > ensuring that it's actually a tuple you're stepping into. In the > case of C, you will not be performing runtime type checks on the > hash space. > > OTOH, with SMP Erlang, you have to protect the ETS tables with > mutexes, whereas on the process heap, you don't have to worry > about contention. > >> So, I guess your benchmark might depend on the size of the key/value >> pairs. For large values, the performance should be the same, for small >> values the process might be a bit slower. > In the case I described, the task was to model the ETS API > faithfully, which meant a separate process per table. This also > included copying, as data sent in messages is copied from > process to process. Otherwise, if you use dict, gb_trees, etc. > or the process dictionary, you can avoid the copying and get > quite good performance if the stored data structures are large. > >> So, if we compare ets with dict or the process dictionary, then ets >> should perform really badly on large values, it seems. > Yes. OTOH, the process dictionary is difficult to share... > > > BR, > Ulf W > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From mk@REDACTED Mon Sep 13 21:40:30 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 21:40:30 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <201009121951.20575.jh@sotun.de> References: <4C8CB006.6070908@amberbio.com> <201009121951.20575.jh@sotun.de> Message-ID: <4C8E7E2E.708@amberbio.com> Hi Jan Thanks for your answer. I guess we really disagree on this:) I was stunned when I saw the nodes disconnecting in the middle of a large message passing, and I think this must be improved at low level in the VM. I see it as similar to context switching. Let me explain why I do that. But first, I will comment on the claim, that you and others make, that Erlang is not suited for large amounts of data. Why not? Erlang is implemented in C. Binaries can be stored as efficiently as in any other language. A binary can be sent to a socket using C functions. Where is the fundamental problem? Is this claim of Erlang being unsuited for large messages not just because people represent data with a space inefficient data structure, e.g., using a list of 4 byte integers instead of a binary. Back to message passing. A cluster of Erlang nodes, need to solve many tasks simultaneously. A task could, in Erlang style, be solved by many cooperating processes, which use message passing to communicate. There are many "tasks" being solved simultaneously, and they can have vastly different time profiles, and priorities. One task could be a fast response to a web application, and it might be implemented as a json/html process that communicates with several data base processes, maybe using a security process as well. They will communicate with mostly small message of size <100 bytes say. At the same time, there could be huge file tansfers or backups of the data base processes going on. Any example suffices. But the point is that the small task should finish rather fast independently of when it is started, and indepedently of what the large task is doing. This is basically a rationale for multi tasking and context switching. And the Erlang vm does this for processes, and everybody agrees it is a good thing. Now muylti tasking only works if all aspects of the task (computation) can go ahead without being blocked. You can context switch as much as you want, but it doesn't help if the fast task gets stuck waiting for a message pass. so it is essential that the large messages can be preempted, and that the bandwidth can be assigned to the fast task. It is probably obvious what I am saying. All parts of the computation must switch between tasks including the message passing between nodes on distinct computers. Otherwise you get a bottleneck. It would be like context switching in the registers and CPU, but the memory bus saying, "sorry CPU, you cannot get the data you are requesting now, because I am still transferring data for the previous process, the one you just preempted". Cheers, Morten. On 9/12/10 7:51 PM, Jan Huwald wrote: > > Am Sunday 12 September 2010 12:48:38 schrieb Morten Krogh: >> Hi Erlangers. >> >> During some test with node to node communication, I sent a large binary >> from a process on node A >> to a process on another node, node B. I also sent some smaller messages >> from other processes on node A to other >> processes on node B. It turned out that the large message blocked the >> later messages. Furthermore, it even blocked >> the net tick communication, so node A and B disconnected from each other >> even though the large message was being transferred! >> >> After looking a bit around, I have come to the understanding that Erlang >> uses one tcp connection between two nodes, and messages are sent >> sequentially from the sending node A to the receiving node. >> >> If that is correct, I think some improvements are needed. > IMO the programmer has to take precautions on its own, if he expetcs to handle > large messages. This is analogous to large memory requirements of single > process. Erlang is not well suited for one of both, natively - which is good, > because it keeps things simple. > > In case of large messages (or large process heaps) tradeoffs have to be made. > Your proposed solution is one example for (a) the fact that these tradeoffs > can be excluded from the language core (b) involving tradeoffs not everybody > is agreeing on. > > >> The problem to solve is basically that small messages, including the net >> tick, should get through more or less independently of >> the presence of large messages. >> >> The simplest would be to have several connections, but that doesn't >> fully solve the problem. A large message will still take up >> a lot of the hardware bandwidth even on another tcp connection. >> >> My suggestion is something like the following. >> >> For communication between node A and node B, there is a process (send >> process) on each node, that coordinates all messages. The send process >> keeps queues of different priorities around, e.g., a high priority, >> medium priority and low priority. Messages are split up into fragments of >> a maximum size. The receiver(node B) send process assembles the >> fragments into the original message and delivers it locally to the >> right process. The fragments ensure that no single transfer will occupy >> the connection for very long. >> There will be a function send_priority where the user can specify a >> priority. The usual send will default to medium, say. >> Net tick will use high priority, of course. Small messages that are >> needed to produce a web application response can have high priority. >> File transfers >> for backup purposes can have low priority. >> The send process then switches between the queues in some way, that >> could be very similar to context switching priorities. >> >> More advanced, the send processes could occasionally probe the >> connection with packets to estimate latency and bandwidth. Those figures >> could then be used >> to calculate fragment sizes. High bandwidth, high latency would require >> large fragments. Low bandwidth, low latency small fragments for instance. >> There could even be a function send_estimated_transfer_time that sends a >> message and has a return value of estimated transfer time, which could >> be used in >> a timeout in a receive loop. > Actively probing seems to me like a recipe for far more failures than it > offers benefit. Passively probing might be ok, if the run-time overhead is > small. But an actual implementation, which does not get biased by > nonstationary activity of the application, seems quite complex to me. > > >> I have actually implemented my own small module for splitting messages >> into fragments, and it solves the issues; net tick goes through, and small >> messages can overtake large ones. >> >> There is of course an issue when the sending and receiving process is >> the same for several messages. Either the guaranteed message order >> should be given up, or the >> coordinators should keep track of that as well. Personally, I think >> guaranteed message order should be given up. Erlang should model the >> real world as >> much as possible, and learn from it. In the real world, two letters >> going from person A to person B, can definitely arrive in the opposite >> order >> of the one in which they were sent. And as node to node communication >> will be over larger and larger distances, it is totally unnatural to >> require >> a certain order. >> >> I am relatively new to Erlang and I really enjoy it. Kudos to all involved! >> >> Cheers, >> >> Morten Krogh. > Regards, > Jan > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From fritchie@REDACTED Mon Sep 13 21:51:26 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Mon, 13 Sep 2010 14:51:26 -0500 Subject: [erlang-questions] node to node message passing In-Reply-To: Message of "Mon, 13 Sep 2010 21:40:30 +0200." <4C8E7E2E.708@amberbio.com> Message-ID: <13219.1284407486@snookles.snookles.com> Morten Krogh wrote: mk> I was stunned when I saw the nodes disconnecting in the middle of a mk> large message passing, and I think this must be improved at low mk> level in the VM. I see it as similar to context switching. Morten, if you had some runnable code that demonstrated the behavior, we would all have a much more precise description of what's happening. :-) And a description of the networking environment, e.g. 10GBit local Ethernet vs. 9.6Kbps/sec to lunar orbit. (Running in virtual machine environments (Xen, VMware, ...) can also play tricky games.) If I recall correctly ... the net_kernel doesn't rely solely on the tick messages between nodes: any message from node A -> B should reset B's tick timeout counter. -Scott From erlang@REDACTED Mon Sep 13 21:55:57 2010 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 13 Sep 2010 21:55:57 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <4C8CB006.6070908@amberbio.com> References: <4C8CB006.6070908@amberbio.com> Message-ID: On Sun, Sep 12, 2010 at 12:48 PM, Morten Krogh wrote: > ?Hi Erlangers. > > During some test with node to node communication, I sent a large binary from > a process on node A > to a process on another node, node B. I also sent some smaller messages from > other processes on node A to other > processes on node B. It turned out that the large message blocked the later > messages. Furthermore, it even blocked > the net tick communication, so node A and B disconnected from each other > even though the large message was being transferred! Just for clarification could you say what you mean by "large", "small" etc. - I have no idea what this means - it might mean 10's of MBytes it might mean GBytes - without knowing I have no idea as to how realistic your expectations are. /Joe > > After looking a bit around, I have come to the understanding that Erlang > uses one tcp connection between two nodes, and messages are sent > sequentially from the sending node A to the receiving node. > > If that is correct, I think some improvements are needed. > > The problem to solve is basically that small messages, including the net > tick, should get through more or less independently of > the presence of large messages. > > The simplest would be to have several connections, but that doesn't fully > solve the problem. A large message will still take up > a lot of the hardware bandwidth even on another tcp connection. > > My suggestion is something like the following. > > For communication between node A and node B, there is a process (send > process) on each node, that coordinates all messages. The send process > keeps queues of different priorities around, e.g., a high priority, medium > priority and low priority. Messages are split up into fragments of > a maximum size. The receiver(node B) send process assembles the fragments > into the original message and delivers it locally to the > right process. The fragments ensure that no single transfer will occupy the > connection for very long. > There will be a function send_priority where the user can specify a > priority. The usual send will default to medium, say. > Net tick will use high priority, of course. Small messages that are needed > to produce a web application response can have high priority. File transfers > for backup purposes can have low priority. > The send process then switches between the queues in some way, that could be > very similar to context switching priorities. > > More advanced, the send processes could occasionally probe the connection > with packets to estimate latency and bandwidth. Those figures could then be > used > to calculate fragment sizes. High bandwidth, high latency would require > large fragments. Low bandwidth, low latency small fragments for instance. > There could even be a function send_estimated_transfer_time that sends a > message and has a return value of estimated transfer time, which could be > used in > a timeout in a receive loop. > > > I have actually implemented my own small module for splitting messages into > fragments, and it solves the issues; net tick goes through, and small > messages can overtake large ones. > > There is of course an issue when the sending and receiving process is the > same for several messages. Either the guaranteed message order should be > given up, or the > coordinators should keep track of that as well. Personally, I think > guaranteed message order should be given up. Erlang should model the real > world as > much as possible, and learn from it. In the real world, two letters going > from person A to person B, can definitely arrive in the opposite order > of the one in which they were sent. And as node to node communication will > be over larger and larger distances, it is totally unnatural to require > a certain order. > > I am relatively new to Erlang and I really enjoy it. Kudos to all involved! > > Cheers, > > Morten Krogh. > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From mk@REDACTED Mon Sep 13 22:41:42 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 13 Sep 2010 22:41:42 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: References: <4C8CB006.6070908@amberbio.com> Message-ID: <705462AF-8D0A-410F-A429-4959114D3394@amberbio.com> Hi Scott and Joe Sure, let us start with the disconnection, even though there is more to it than that. Two mac computers on a local wireless network. Slow connection, using the mac activity I see a transfer speed of around 400kB/s ~ 3Mb/s But hang on, this is a fundamental issue, I believe. It is just more clear at slow connections. Start two nodes, one on each computer erl -name 'node@REDACTED' erl -name 'node@REDACTED' connect them with net_adm:ping. nodes() gives the right result. Create 50MB binary. (node@REDACTED)7> B = binary:copy(<<"b">>, 50000000). Send the binary to the registered shell at the other node. The shell is registered as ten. (node@REDACTED)7> {ten, 'node@REDACTED'} ! B. Watch the network and see transfers of around 300-440 kB/s. On the other node (node@REDACTED)12> now(). {1284,408613,791811} (node@REDACTED)13> =ERROR REPORT==== 13-Sep-2010::22:11:27 === ** Node 'node@REDACTED' not responding ** ** Removing (timedout) connection ** (node@REDACTED)13> now(). {1284,408693,688267} The disconnect happened after around 1 minute, which is the default net_ticktime. Let us check this assumption. (node@REDACTED)14> net_kernel:set_net_ticktime(5). change_initiated (node@REDACTED)15> net_kernel:get_net_ticktime(). {ongoing_change_to,5} (node@REDACTED)16> net_kernel:get_net_ticktime(). {ongoing_change_to,5} (node@REDACTED)17> net_kernel:get_net_ticktime(). 5 The net_ticktime is now 5 seconds. (node@REDACTED)18> now(). {1284,408904,738430} (node@REDACTED)19> =ERROR REPORT==== 13-Sep-2010::22:15:11 === ** Node 'node@REDACTED' not responding ** ** Removing (timedout) connection ** (node@REDACTED)19> now(). {1284,408913,335812} 8 seconds, but I was very slow at typing now(). Try again. (node@REDACTED)20> now(). {1284,408927,35639} (node@REDACTED)21> now(). =ERROR REPORT==== 13-Sep-2010::22:15:32 === ** Node 'node@REDACTED' not responding ** ** Removing (timedout) connection ** =ERROR REPORT==== 13-Sep-2010::22:15:32 === The global_name_server locker process received an unexpected message: {{#Ref<0.0.0.155>,'node@REDACTED'},true} {1284,408932,778710} 5 seconds as expected. Anyway, net_tick is not precise. It could have been 4-6. The file was 50MB, but I could have made it smaller, obviously. But the real point is not just the disconnect. The real point is that messages block others. I am not giving the code here, but you can easily reproduce it, if it is reproducible :) Start two processes on 'node@REDACTED' say two1 and two2. Start two processes on 'node@REDACTED;, say ten1 and ten2. Set net_ticktime to a very high number. Send the 50 MB binary or so from two1 to ten1. Shortly after send a 10 byte message from two2 to ten2. Let ten2 and ten2 write to io:format when they receive a message. Nothing happens for a long time, the network is busy, and then both ten1 and ten2 write their output. But does anyone actually know? Do messages queue up like this test seems to indicate. Cheers, Morten. On Sep 13, 2010, at 9:55 PM, Joe Armstrong wrote: > On Sun, Sep 12, 2010 at 12:48 PM, Morten Krogh wrote: >> Hi Erlangers. >> >> During some test with node to node communication, I sent a large binary from >> a process on node A >> to a process on another node, node B. I also sent some smaller messages from >> other processes on node A to other >> processes on node B. It turned out that the large message blocked the later >> messages. Furthermore, it even blocked >> the net tick communication, so node A and B disconnected from each other >> even though the large message was being transferred! > > Just for clarification could you say what you mean by "large", "small" > etc. - I have no idea what this means - it might mean 10's of MBytes > it might mean GBytes - without knowing I have no idea as to how > realistic your expectations are. > > /Joe > >> >> After looking a bit around, I have come to the understanding that Erlang >> uses one tcp connection between two nodes, and messages are sent >> sequentially from the sending node A to the receiving node. >> >> If that is correct, I think some improvements are needed. >> >> The problem to solve is basically that small messages, including the net >> tick, should get through more or less independently of >> the presence of large messages. >> >> The simplest would be to have several connections, but that doesn't fully >> solve the problem. A large message will still take up >> a lot of the hardware bandwidth even on another tcp connection. >> >> My suggestion is something like the following. >> >> For communication between node A and node B, there is a process (send >> process) on each node, that coordinates all messages. The send process >> keeps queues of different priorities around, e.g., a high priority, medium >> priority and low priority. Messages are split up into fragments of >> a maximum size. The receiver(node B) send process assembles the fragments >> into the original message and delivers it locally to the >> right process. The fragments ensure that no single transfer will occupy the >> connection for very long. >> There will be a function send_priority where the user can specify a >> priority. The usual send will default to medium, say. >> Net tick will use high priority, of course. Small messages that are needed >> to produce a web application response can have high priority. File transfers >> for backup purposes can have low priority. >> The send process then switches between the queues in some way, that could be >> very similar to context switching priorities. >> >> More advanced, the send processes could occasionally probe the connection >> with packets to estimate latency and bandwidth. Those figures could then be >> used >> to calculate fragment sizes. High bandwidth, high latency would require >> large fragments. Low bandwidth, low latency small fragments for instance. >> There could even be a function send_estimated_transfer_time that sends a >> message and has a return value of estimated transfer time, which could be >> used in >> a timeout in a receive loop. >> >> >> I have actually implemented my own small module for splitting messages into >> fragments, and it solves the issues; net tick goes through, and small >> messages can overtake large ones. >> >> There is of course an issue when the sending and receiving process is the >> same for several messages. Either the guaranteed message order should be >> given up, or the >> coordinators should keep track of that as well. Personally, I think >> guaranteed message order should be given up. Erlang should model the real >> world as >> much as possible, and learn from it. In the real world, two letters going >> from person A to person B, can definitely arrive in the opposite order >> of the one in which they were sent. And as node to node communication will >> be over larger and larger distances, it is totally unnatural to require >> a certain order. >> >> I am relatively new to Erlang and I really enjoy it. Kudos to all involved! >> >> Cheers, >> >> Morten Krogh. >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From tushar.erlang@REDACTED Mon Sep 13 22:47:01 2010 From: tushar.erlang@REDACTED (Tushar Deshpande) Date: Mon, 13 Sep 2010 16:47:01 -0400 Subject: About behavior of OTP's supervisor-worker architecture Message-ID: Hi, I've a question about OTP's supervisor-worker architecture. I understand that OTP allows us to write fault-tolerant apps. This is made possible by supervisor-worker architecture. A supervisor manages several workers. If a worker (or a group of workers) fails then supervisor is able to restart it. The worker is restarted and it resumes with the same state that it had before crash. Now, let's consider following situation. A worker process has two possible implementations, P and Q. Worker P runs under normal conditions. Worker Q is supposed to run in case P fails. If worker P crashes then, supervisor is notified about the crash. Typically, the supervisor would restart worker P. But, I would like the supervisor to behave in a different manner. In case the worker P fails, the supervisor should start the worker Q. The worker Q should begin its execution with the same state that P had at the point of crash. Is it possible to write an OTP application that does this? If yes, then do I need to customize the supervisor code. Best Regards, Tushar Deshpande From torben.lehoff@REDACTED Mon Sep 13 23:24:18 2010 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Mon, 13 Sep 2010 23:24:18 +0200 Subject: [erlang-questions] About behavior of OTP's supervisor-worker architecture In-Reply-To: References: Message-ID: Hi Tushar, The OTP supervisor module does not support your desired behaviour, but you can use the supervisor and the monitor facility of Erlang to implement it. A rough design: The supervisor should have dynamic children and in the function where the call to supervisor:start_child/2 is done the pid of the started child should be passed on to a gen_server process on the side that will call erlang:monitor/2. When the child dies the monitoring process will get a 'DOWN' message which ought to contain enough information to start a new process - you just have to include the state data of the process in the Reason for termination. So you have to write some code to make it work, but you can reuse the supervisor and implement the monitoring process using gen_server. If you are really keen on this you can even implement your own OTP module with its own behaviour and all, but I would recommend that you get the thing to work first in order to avoid too many balls in the air in the beginning. Cheers, Torben On Mon, Sep 13, 2010 at 22:47, Tushar Deshpande wrote: > Hi, > > I've a question about OTP's supervisor-worker architecture. > > I understand that OTP allows us to write fault-tolerant apps. > This is made possible by supervisor-worker architecture. A > supervisor manages several workers. If a worker (or a group > of workers) fails then supervisor is able to restart it. The worker > is restarted and it resumes with the same state that it had > before crash. > > Now, let's consider following situation. > > A worker process has two possible implementations, P and Q. > Worker P runs under normal conditions. Worker Q is supposed > to run in case P fails. > > If worker P crashes then, supervisor is notified about the crash. > Typically, the supervisor would restart worker P. > > But, I would like the supervisor to behave in a different manner. > In case the worker P fails, the supervisor should start the worker Q. > The worker Q should begin its execution with the same state that > P had at the point of crash. > > Is it possible to write an OTP application that does this? If yes, > then do I need to customize the supervisor code. > > > Best Regards, > > Tushar Deshpande > -- http://www.linkedin.com/in/torbenhoffmann From tony@REDACTED Mon Sep 13 23:38:39 2010 From: tony@REDACTED (Tony Rogvall) Date: Mon, 13 Sep 2010 23:38:39 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <4C8CB006.6070908@amberbio.com> References: <4C8CB006.6070908@amberbio.com> Message-ID: <85CADA57-5C8B-41F2-8A30-B67F9E55C0C9@rogvall.se> Hi Morten! On 12 sep 2010, at 12.48, Morten Krogh wrote: > Hi Erlangers. > > During some test with node to node communication, I sent a large binary from a process on node A > to a process on another node, node B. I also sent some smaller messages from other processes on node A to other > processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked > the net tick communication, so node A and B disconnected from each other even though the large message was being transferred! > This is one of the things that should have been fixed a long time ago (I think) May be I am also the one that should have done it while I had the chance ;-) > After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent > sequentially from the sending node A to the receiving node. > > If that is correct, I think some improvements are needed. > I can only agree here. > The problem to solve is basically that small messages, including the net tick, should get through more or less independently of > the presence of large messages. > > The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up > a lot of the hardware bandwidth even on another tcp connection. > > My suggestion is something like the following. > > For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process > keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of > a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the > right process. The fragments ensure that no single transfer will occupy the connection for very long. > There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say. > Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers > for backup purposes can have low priority. > The send process then switches between the queues in some way, that could be very similar to context switching priorities. Yes, multiplexing over one TCP channel I think is a reasonable way to go. > > More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used > to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance. > There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in > a timeout in a receive loop. > Why not take one step at a time, Erlang/OTP is and should be a process of improvements. Trying to do too much is just going to delay the implementation finding it's way into the main branch. > > I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small > messages can overtake large ones. > Can't wait to see the implementation, do you have a git somehere ? > There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the > coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as > much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order > of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require > a certain order. Lets try to stick to the existing "semantics" here, then the implementation will have a chance to get realized. > > I am relatively new to Erlang and I really enjoy it. Kudos to all involved! > Well you certainly stumbled into the right place ;-) Regards /Tony From geoffrey.biggs@REDACTED Tue Sep 14 00:03:36 2010 From: geoffrey.biggs@REDACTED (Geoffrey Biggs) Date: Tue, 14 Sep 2010 07:03:36 +0900 Subject: [erlang-questions] Re-entrant CORBA services In-Reply-To: <87hbhum2ih.fsf@pingviini.priv> References: <4C8DDFC1.1010101@aist.go.jp> <87hbhum2ih.fsf@pingviini.priv> Message-ID: <4C8E9FB8.8010407@aist.go.jp> On 13/09/10 18:11, Jani Hakala wrote: > How about dividing this call from object A to object B to two calls > to object B? First one would return at that point where B now calls A > and the second call would give the object B the results of the > call. Any parameters required by object A could be passed as the > result from the first call. Unfortunately, I can't change the IDL, which includes the call from object A to object B. It is defined by a standard and I must comply with that standard. Otherwise I would have used a set of one-way calls, just as I did in the pure Erlang version of the objects. > AMI/AMH [1] might provide another solution if they are available. Or > perhaps multiple servants for object A. > > I prefer to avoid two-way calls with CORBA as much as possible. I > once tried those with C++ and ACE/TAO, and it seemed to be too much > trouble. I didn't try AMI/AMH or multiple servants though as I was > just learning CORBA basics. I agree; I prefer one-way calls for the same reasons, and particularly for the reason that lead to this post. It's a good real-life example of how Erlang has a better RPC system than CORBA. Geoff From geoffrey.biggs@REDACTED Tue Sep 14 00:13:18 2010 From: geoffrey.biggs@REDACTED (Geoffrey Biggs) Date: Tue, 14 Sep 2010 07:13:18 +0900 Subject: [erlang-questions] Re-entrant CORBA services In-Reply-To: References: <4C8DDFC1.1010101@aist.go.jp> <87hbhum2ih.fsf@pingviini.priv> Message-ID: <4C8EA1FE.2050906@aist.go.jp> On 13/09/10 23:46, Niclas Eklund wrote: > The drawback with oneway CORBA operations (i.e. asynchronous > calls/gen_server:cast) is that the exact behavior isn't specified by the > OMG specifications, which might give different results dependening on > which ORBs you're using. > Besides using a oneway operation, perhaps you can use corba:reply/2 (see > http://www.erlang.org/doc/man/corba.html#reply-2), which is the same > thing as gen_server:reply/2. If it's posssible for you to use Orber > pseudo objects, you'll be able to avoid deadlock (see for example the > CosNaming_NamingContextExt_impl.erl module how these can be > implemented). The exact answer depends on if A and B resides on the same > ORB (Orber I assume in this case) or inter-ORB communication, with an > ORB from another vendor, takes place. Thanks for the advice. I'll have a look at pseudo-objects. I'm not sure if I can use them or not until I give them a go. :) Not being able to send info messages may be a problem for initialising the service with values it needs, though. A and B will not necessarily reside on the same ORB. If B resides on an ORB implemented in an object-oriented, everything works happily. If B resides on an ORB such as Orber, it won't; this is a fault of the RPC specification in the IDL, not Orber. But it's something I have to figure out how to work around. I do, overall, find the Orber type to be much nicer to work with, though. Geoff From ulf.wiger@REDACTED Tue Sep 14 00:17:23 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Tue, 14 Sep 2010 00:17:23 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E75EF.7000702@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> <4C8E75EF.7000702@amberbio.com> Message-ID: <4C8EA2F3.4070807@erlang-solutions.com> On 09/13/2010 09:05 PM, Morten Krogh wrote: > >> No, there are several other overheads. For instance, you must somehow >> implement a hash table from immutable dynamically typed data structures, >> whereas in ETS (in C), you can use mutable arrays. The dict module, >> for example, uses a tree structure of tuples. Once you've located >> the leaf tuple where the object is to reside, you have to create >> a new copy of that tuple, which means you must also update (copy) >> the parent tuple, etc. > > But it is to some extent self imposed that you want a functional data > structure. Sure. I'm not trying to criticize Erlang, but merely to point out what makes ETS tables implemented in the VM faster than an Emulation of them in Erlang code. > You could have used the process dictionary for this or some foreign > interface. > Nowadays nif, but that was not available to you then. But in both cases (for Robert and for me), the whole _point_ was to implement them in Erlang, since we wanted to get the functionality in place without having to spend a lot of time with a native implementation. The goal was not speed at all, but productivity. BR, Ulf W From geoffrey.biggs@REDACTED Tue Sep 14 01:16:27 2010 From: geoffrey.biggs@REDACTED (Geoffrey Biggs) Date: Tue, 14 Sep 2010 08:16:27 +0900 Subject: [erlang-questions] Shutting down part of a supervision tree In-Reply-To: References: <4C8DDEB2.2030606@aist.go.jp> Message-ID: <4C8EB0CB.70101@aist.go.jp> On 13/09/10 23:10, Niclas Eklund wrote: > The Erlang CORBA objects are roughly gen_server-wrappers (using the > create option {pseudo, true} is the exception since these objects are > not processes). Hence, you should have a look at the behavior of > gen_server > (http://www.erlang.org/doc/man/gen_server.html#Module:terminate-2) > regarding the terminate function. E.g if the process trap exit signals, > the IDL file(s) has been compiled using the IC option handle_info or not > etc. > > These links will probably be usefull to read as well: > > * http://www.erlang.org/doc/man/Module_Interface.html > * http://www.erlang.org/doc/apps/orber/ch_stubs.html Thanks for the advice. I managed to track done one issue to a subtle typo in my terminate spec (which is somewhat embarassing), and now I have a better idea of how to manage CORBA processes. However, I'm still not sure what the accepted method of telling a supervisor to stop cleanly is, including cleaning up all its children neatly. Is "exit(S, normal)" usually used? Geoff From vinoski@REDACTED Tue Sep 14 02:06:04 2010 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 13 Sep 2010 20:06:04 -0400 Subject: [erlang-questions] Re-entrant CORBA services In-Reply-To: <4C8E9FB8.8010407@aist.go.jp> References: <4C8DDFC1.1010101@aist.go.jp> <87hbhum2ih.fsf@pingviini.priv> <4C8E9FB8.8010407@aist.go.jp> Message-ID: On Mon, Sep 13, 2010 at 6:03 PM, Geoffrey Biggs wrote: > > I agree; I prefer one-way calls for the same reasons, and particularly > for the reason that lead to this post. CORBA oneway calls aren't really equivalent to async send calls. When my colleagues and I wrote the first ORBs back in the early 90s while we also worked on the CORBA spec, we intended oneways to be best-effort calls implemented over UDP. As ORBs evolved most implemented oneways over TCP, but there was nothing in the CORBA spec requiring any particular quality-of-service from a oneway call; indeed, a spec-compliant ORB implementation could just drop all oneways on the floor if it wanted to. Much later we added the AMI stuff and as part of that we added a number of delivery guarantees to oneways that applications could choose to use. Ultimately, unless you know for sure what sorts of delivery guarantees the ORB implementation you're using gives you regarding oneways, or unless it supports AMI, my advice is to avoid oneways. In all the CORBA work I've done I think I used them only once or twice. > It's a good real-life example of > how Erlang has a better RPC system than CORBA. Ah, don't get me started. :-) This really depends. When you consider the original intent of RPC, which was to make remote calls look just like local calls, then CORBA does just fine, and does just as well as Erlang does. But of course that goal ignores a whole bunch of real-world stuff that really can't be ignored, which is why the very concept of RPC is fundamentally flawed given that a remote call can never be "just like" a local call. See: http://qconlondon.com/dl/qcon-london-2009/slides/SteveVinoski_RPCAndItsOffspringConvenientYetFundamentallyFlawed.pdf for many more details. --steve From rvirding@REDACTED Tue Sep 14 02:40:38 2010 From: rvirding@REDACTED (Robert Virding) Date: Tue, 14 Sep 2010 02:40:38 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8E75EF.7000702@amberbio.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> <4C8E75EF.7000702@amberbio.com> Message-ID: On 13 September 2010 21:05, Morten Krogh wrote: > >> No, there are several other overheads. For instance, you must somehow >> implement a hash table from immutable dynamically typed data structures, >> whereas in ETS (in C), you can use mutable arrays. The dict module, >> for example, uses a tree structure of tuples. Once you've located >> the leaf tuple where the object is to reside, you have to create >> a new copy of that tuple, which means you must also update (copy) >> the parent tuple, etc. > > But it is to some extent self imposed that you want a functional data > structure. > You could have used the process dictionary for this or some foreign > interface. > Nowadays nif, but that was not available to you then. > > Why didn't you use the process dictionary? It is made for this situation. The problem comes back to memory management and garbage collection. A process heap is not collected incrementally but in one go. During that time the cpu/core/thread running that process will stop. This is no problem as most process heaps are not large and there is no noticeable pause. Also the collector is optimised to keep these times down. It has to be done in this way as there no other way to detect when terms are free except by scanning the heap.* In an ets table heap however, as you do explicit insert and delete, collection takes part every time you do a delete so there are no long pauses at all. This is irrespective of how big the ets table is, which means is to possible to have LARGE tables without any pause times for collection. But this also means that the data in an ets table cannot exist in a normal process heap but must be separate from it. This is why you get the copying of data between processes and ets tables. This is why you have ets:match and ets:select which allow you to do more checking of data in the ets pseudo-process before copying it over to a process heap, which is a big win when scanning a table. Now using the process dictionary would allow you to use the same algorithms for managing the table itself as with ets tables but it doesn't solve the big problem of where to store the actual data. Either you keep it in the process heap (which is done today) but then incur the all problems with having LARGE process heaps, or you keep it in a separate heap but then incur all the problems with copying. In either case there is not a clear benefit in using the process dictionary. You get the same problem if you were to try to use NIFs. Where do you put the data? The process heap should only contain valid erlang terms and does not support mutable data, which limits how much smarter you can actually make it by not using plain erlang. Though you could probably improve dict by adding some NIFs, definitely something to consider for the future. There is another point as well and that is that using the process dictionary gives the code side effects and makes it non-functional. Ah, you might say, but does not using ets do the same thing? Yes, but ets tables behave as processes (and can be emulated using processes) which confines all side effects to inter-process communication. Which is a Good Thing. If this worries you is another thing but it is nice to be able to follow the flow of state through your code as much as possible. As usual it all boils down to which trade-offs you are willing to make. Ets tables and dict/gb_trees give you two different choices with different properties. Sorry this became a bit longer than I had originally intended. Robert * Yes, there are garbage collection algorithms that incrementally collect data which would allow you to have LARGE process heaps without incurring long pause for collection. These are, however, more complex and costly and it is not clear how much you actually gain by using them. Multi-core environments don't help either in this respect. From tushar.erlang@REDACTED Tue Sep 14 06:11:21 2010 From: tushar.erlang@REDACTED (Tushar Deshpande) Date: Tue, 14 Sep 2010 00:11:21 -0400 Subject: [erlang-questions] About behavior of OTP's supervisor-worker architecture In-Reply-To: References: Message-ID: Hi Torben, I appreciate your help a lot. I think that I could build a prototype system without using OTP. ?As you suggested, I can use pure erlang processes with one process as a monitor and other two processes as workers. Best Regards, Tushar Deshpande From toby@REDACTED Tue Sep 14 06:48:04 2010 From: toby@REDACTED (Toby Thain) Date: Tue, 14 Sep 2010 00:48:04 -0400 Subject: [erlang-questions] erlang:max/2 and erlang:min/2 In-Reply-To: References: <33A3A8B6-098C-4C1C-8C8A-16A4BF38A6AF@cs.otago.ac.nz> Message-ID: <5C039B5C-71BC-4EE3-9E86-D52C46608326@telegraphics.com.au> On 13-Sep-10, at 10:41 AM, Nicholas Frechette wrote: > ... > *** Side note *** > Also, while it is true the ternary operator can't always be used as an > l-value, some compilers still allow it as it must have been once > part of the > standard. While I haven't internalised the relevant standards the way ROK has, there's no reason to assume this was EVER 'standardised' behaviour. Vendors will make arbitrary and/or egregious divergences from standards. That's the problem. --T > ...snipped anecdotes... > Alas, I digress. > Nicholas From torben.lehoff@REDACTED Tue Sep 14 08:06:43 2010 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Tue, 14 Sep 2010 08:06:43 +0200 Subject: [erlang-questions] About behavior of OTP's supervisor-worker architecture In-Reply-To: References: Message-ID: Hi Tushar! Use OTP for everything! Add the missing special functionality with plain Erlang. If common enough make your own behaviour. I have tried to code without OTP and ended up coding gen_server and gen_fsm myself. Cheers, Torben On 14 Sep 2010 06:11, "Tushar Deshpande" wrote: Hi Torben, I appreciate your help a lot. I think that I could build a prototype system without using OTP. As you suggested, I can use pure erlang processes with one process as a monitor and other two processes as workers. Best Regards, Tushar Deshpande From ulf.wiger@REDACTED Tue Sep 14 09:40:45 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Tue, 14 Sep 2010 09:40:45 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> <4C8E75EF.7000702@amberbio.com> Message-ID: <4C8F26FD.8070201@erlang-solutions.com> On 09/14/2010 02:40 AM, Robert Virding wrote: > > > * Yes, there are garbage collection algorithms that incrementally > collect data which would allow you to have LARGE process heaps without > incurring long pause for collection. These are, however, more complex > and costly and it is not clear how much you actually gain by using > them. Multi-core environments don't help either in this respect. I think this is one thing that is seldom overlooked. Erlang is a bit unusual in its use of per-process garbage collection, but this is a model that fits multicore very well. The enablers are the fine-grained concurrency together with share-nothing semantics, and the result is that Erlang can pretty robustly achieve soft real-time characteristics even when the VM is carrying gigabytes of data, and still have a relatively simple garbage collector. BR, Ulf W From johanmon@REDACTED Tue Sep 14 11:02:41 2010 From: johanmon@REDACTED (Johan Montelius) Date: Tue, 14 Sep 2010 11:02:41 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: <4C8F26FD.8070201@erlang-solutions.com> References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> <4C8E75EF.7000702@amberbio.com> <4C8F26FD.8070201@erlang-solutions.com> Message-ID: While we're talking ets tables: Wouldn't it be nice to be able to send table references across nodes? Today when you create a new tabel you get a table reference that is an integer. The table can only be accessed by processes in the same node and god know what will happen if you send the reference over to another node where there is another table registered under the same number. Since tables have a mutable state they should be modeled as processess. When you create a table you get a process identifier. Locally on the node where the table was created all operations are done as usual i.e. direct access to the table without the overhead of sending messages. When you send the process identifier over to another node however, a process identifier is sent that refers to a proxy process on the originating node. Using the functions in the ets module using a regular process identifier would result in messages being sent to that process and replies being received. Just like accessing a local ets table (slower and we have to have some error codes if the process is dead). One could the also open up for a asynchronous access to the table: Ref = ets:lookup_and_send_me_the_reply(Table, Key), do_something(), receive {Ref, Value} -> How's that :-) Johan -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From erlang@REDACTED Tue Sep 14 11:04:28 2010 From: erlang@REDACTED (Joe Armstrong) Date: Tue, 14 Sep 2010 11:04:28 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <85CADA57-5C8B-41F2-8A30-B67F9E55C0C9@rogvall.se> References: <4C8CB006.6070908@amberbio.com> <85CADA57-5C8B-41F2-8A30-B67F9E55C0C9@rogvall.se> Message-ID: Interesting - I think what you want is a message passing layer that "just works" this should survive virtually all failures. What happens if you send a large message between A and B and B is "totally destroyed" during the sending? Think B is a laptop and has just been crushed with a sledge hammer. You buy a new B - reinstall the software and ... should the data transfer proceed? I think you need an introspection layer for things like this - you need to start the transfer, then go back later and ask how it it doing? - you might get the answer "B is broken - waiting to be repaired" - but having said "transfer X from A to B" the system should "try forever" to do this. You might want "transfer X from A to B and tell Y if you can't do this within time T" which is completely different. Exactly what is "built-in" to the standard inter-process message passing semantics is a mute point. I think what you want belongs to a separate library function. Actually I have > 1 macs at home - with bad wireless connectivity between certain rooms - so this is a not uninteresting problem. Some kind of "bit-torrent" algorithm might be better - suppose connectivity between A and C is bad - but between A and B and B and C is good - what then? A very interesting problem ... some kind of gossip algorithm with bit-torrent type fragment sharing seems in order. Since it's your own network, bit-torreent type fairness is not relevant ... You have twiched my programming nerve ... /Joe On Mon, Sep 13, 2010 at 11:38 PM, Tony Rogvall wrote: > Hi Morten! > > On 12 sep 2010, at 12.48, Morten Krogh wrote: > >> Hi Erlangers. >> >> During some test with node to node communication, I sent a large binary from a process on node A >> to a process on another node, node B. I also sent some smaller messages from other processes on node A to other >> processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked >> the net tick communication, so node A and B disconnected from each other even though the large message was being transferred! >> > > This is one of the things that should have been fixed a long time ago (I think) > May be I am also the one that should have done it while I had the chance ;-) > > >> After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent >> sequentially from the sending node A to the receiving node. >> >> If that is correct, I think some improvements are needed. >> > > I can only agree here. > >> The problem to solve is basically that small messages, including the net tick, should get through more or less independently of >> the presence of large messages. >> >> The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up >> a lot of the hardware bandwidth even on another tcp connection. >> >> My suggestion is something like the following. >> >> For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process >> keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of >> a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the >> right process. The fragments ensure that no single transfer will occupy the connection for very long. >> There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say. >> Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers >> for backup purposes can have low priority. >> The send process then switches between the queues in some way, that could be very similar to context switching priorities. > > Yes, multiplexing over one TCP channel I think is a reasonable way to go. > >> >> More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used >> to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance. >> There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in >> a timeout in a receive loop. >> > > Why not take one step at a time, Erlang/OTP is and should be a process of improvements. Trying to do too much > is just going to delay the implementation finding it's way into the main branch. > >> >> I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small >> messages can overtake large ones. >> > Can't wait to see the implementation, do you have a git somehere ? > >> There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the >> coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as >> much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order >> of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require >> a certain order. > > Lets try to stick to the existing "semantics" here, then the implementation will have a chance to get realized. > >> >> I am relatively new to Erlang and I really enjoy it. Kudos to all involved! >> > Well you certainly stumbled into the right place ;-) > > Regards > > /Tony > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From jesper.louis.andersen@REDACTED Tue Sep 14 12:41:40 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 14 Sep 2010 12:41:40 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: References: <4C8CB006.6070908@amberbio.com> <85CADA57-5C8B-41F2-8A30-B67F9E55C0C9@rogvall.se> Message-ID: On Tue, Sep 14, 2010 at 11:04 AM, Joe Armstrong wrote: > > Some kind of "bit-torrent" algorithm might be better - suppose > connectivity between A and > C is bad - but between A and B and B and C is good - what then? Bittorrent does nothing to protect against a killed TCP connection. If the protocol gets a disconnect, it will simply grab another peer from a pool of pending peers and try to connect to it. It may be the same peer, or it may not. If the line is merely slow, the fragmentation of the data ensures a grab eventually where the faster connections pass more data around. > Since it's your own network, bit-torreent type fairness is not relevant ... Most of the bittorrent protocol is there to solve a specific problem: most people are on slow, copper based xDSL type lines. These are asymmetric and congest easily in the upstream path. Hence, a lot of the algorithm is devoted to meticulously manage the limited upstream bandwidth to balance the somewhat unlimited downstream bandwidth and maximize it (until the file has been downloaded in which case we change strategy). The algorithm is not that much about fairness. It doesn't really care if anybody gets a "fair" share of the file. Rather it tries to optimize the downstream bandwidth for the client - but to do that the protocol is designed such that you must send data back. If you ignore the sending, you will eventually get your file. The point is just that you will get it later than everybody else. Another interesting property is that clients "sort themselves" into classes where a class is determined by its upload speed. Thus the fast peers tend to get the file fast and then distribute it to other peers. The brilliance of the protocol lies in swarm intelligence. Each peer knows nothing -- but the information it has gathered itself. Each peer use no historic data older than 10 seconds. Yet, data is transferred at speeds that rival or surpass any other P2P network of critical mass on the internet. > > You have twiched my programming nerve ... > > /Joe > > > > > > > On Mon, Sep 13, 2010 at 11:38 PM, Tony Rogvall wrote: >> Hi Morten! >> >> On 12 sep 2010, at 12.48, Morten Krogh wrote: >> >>> Hi Erlangers. >>> >>> During some test with node to node communication, I sent a large binary from a process on node A >>> to a process on another node, node B. I also sent some smaller messages from other processes on node A to other >>> processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked >>> the net tick communication, so node A and B disconnected from each other even though the large message was being transferred! >>> >> >> This is one of the things that should have been fixed a long time ago (I think) >> May be I am also the one that should have done it while I had the chance ;-) >> >> >>> After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent >>> sequentially from the sending node A to the receiving node. >>> >>> If that is correct, I think some improvements are needed. >>> >> >> I can only agree here. >> >>> The problem to solve is basically that small messages, including the net tick, should get through more or less independently of >>> the presence of large messages. >>> >>> The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up >>> a lot of the hardware bandwidth even on another tcp connection. >>> >>> My suggestion is something like the following. >>> >>> For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process >>> keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of >>> a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the >>> right process. The fragments ensure that no single transfer will occupy the connection for very long. >>> There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say. >>> Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers >>> for backup purposes can have low priority. >>> The send process then switches between the queues in some way, that could be very similar to context switching priorities. >> >> Yes, multiplexing over one TCP channel I think is a reasonable way to go. >> >>> >>> More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used >>> to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance. >>> There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in >>> a timeout in a receive loop. >>> >> >> Why not take one step at a time, Erlang/OTP is and should be a process of improvements. Trying to do too much >> is just going to delay the implementation finding it's way into the main branch. >> >>> >>> I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small >>> messages can overtake large ones. >>> >> Can't wait to see the implementation, do you have a git somehere ? >> >>> There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the >>> coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as >>> much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order >>> of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require >>> a certain order. >> >> Lets try to stick to the existing "semantics" here, then the implementation will have a chance to get realized. >> >>> >>> I am relatively new to Erlang and I really enjoy it. Kudos to all involved! >>> >> Well you certainly stumbled into the right place ;-) >> >> Regards >> >> /Tony >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- J. From jesper.louis.andersen@REDACTED Tue Sep 14 12:45:12 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 14 Sep 2010 12:45:12 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <705462AF-8D0A-410F-A429-4959114D3394@amberbio.com> References: <4C8CB006.6070908@amberbio.com> <705462AF-8D0A-410F-A429-4959114D3394@amberbio.com> Message-ID: On Mon, Sep 13, 2010 at 10:41 PM, Morten Krogh wrote: > > But the real point is not just the disconnect. The real point is that messages block others. > The advantage of a serial stream is probably that you don't need an ID on each message to order them. If messages are allowed to be multiplexed, you need the ordering. The aside: SCTP is the protocol you want for this. -- J. From vladdu55@REDACTED Tue Sep 14 15:55:02 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 14 Sep 2010 15:55:02 +0200 Subject: application meta-information Message-ID: Hi all, As part of my work with erlide, I need to work with projects/applications that don't follow the standard OTP structure. More precisely, the largest user base doesn't :-) The tooling needs to know about the project structure and currently the needed meta-information is stored in a highly Eclipse-dependent way. What I want to do is to change that and have it in a format where it can be used by other tools too. For this to be useful, we should have an agreement about what information is needed and where it is to be stored. Here I will propose a way to do that and look forward suggestions and criticism. ------------- Tools that work on source code need to be able to understand the codebase's structure and dependencies. For example a builder needs to locate the source code and all the needed include files; a launcher needs to find all the required beam files. In the standard libraries there are two files that describe this information: the .app file and the Emakefile. Other build tools use other formats or infer the information at runtime. However, all these make an important assumption: the code must be structured as an OTP application. Also, there are important bits of data that aren't present in these files, like the location of the application's dependencies, because at runtime the applications can be located anyway. What I'd like to suggest is to extend the .app format to include information about - the layout of the source code - the location of dependencies - compiler options This information should be present in a file named "meta" in the root directory of the application. The contents is an #appmeta term like below, but with the options expanded as a property list (i.e. just like in regular .app files): -type options() :: [{atom(), any()}]. -type meta_path() :: string() | {atom(), string()}. -record(layout, { src = ["src"] :: [meta_path()], include = ["include"] :: [meta_path()], ebin = "ebin" :: meta_path(), doc = "doc" :: [meta_path()] }). -type meta_module() :: module() | {module(), options()}. -type meta_app() :: atom() | {atom(), meta_path()} | {atom, #layout{}}. -record(meta, { description = "" :: string(), id = "" :: string(), vsn = "" :: string(), modules = source :: [meta_module()] | 'source', maxT = infinity :: integer() | 'infinity', registered = [] :: [atom()], included_applications = [] :: [meta_app()], applications = [] :: [meta_app()], env = [] :: [{any(), any()}], mod = undefined :: {module(), any()} | 'undefined', start_phases = undefined :: [{atom(), any()}] | 'undefined', otp_version = undefined :: string() | 'undefined', layout = #layout{}, compiler_options = [] :: options() }). -record(appmeta, {name, meta=#meta{}}). The additions and changes are as follows: * otp_version specifies the minimum required version of the runtime in a format like "R13B" * layout describes where source, include, ebin and doc directories are placed, if the setup is not standard * compiler_options specifies the global options to use when compiling * 'source' instead of a module list in modules means "all modules in the source directories" * a module can be specified by name with or without a list of extra compiler options that supersede the global ones * applications and include_applications can be specified by name, by name and a path (these assume the app is a standard OTP apps or a "meta" file can be found), or by name and a precise layout of the source code. This is so that the source code for libraries and such (if available) can be located. * paths can be specified as regular strings, or by a pair {variable, path} where the variable is to be interpreted (tool-specific) to a path that is to be prepended to the one specified. For example, {'PRJ_1', "my/path"} where PRJ1 evaluates to "/home/me/prj_1" gives a result of "home/me/prj_1/my/path". Variables can be retrieved from the environment or from the tool that processes the file (Eclipse, for example). With variables it is possible to use the same settings for multiple users/environments and just set the variables to the right values, or to quickly switch to a different version of a library. * unknown options are ignored, other tools could add specific ones. * if an .app file exists, values should be merged with these (for backwards compatibility). Have to specify what happens if there are mismatches and who (which tool) is responsible to check the consistency. * even the data in the 'info' file used by the OTP applications can be added here Where/how can this be used: - as a makefile, since it contains all details needed - to generate the .app file (so that data isn't duplicated) - to be able to locate the right dependencies to be able to provide code completion, show documentation and other goodies - to automatically locate the binaries for all dependencies when launching an application The point is that it's better to have a single standard place to find all this information than having it spread in different places and tool-dependent formats. I am planning to use this to configure erlide projects, which is better than today's use of Java properties files, but it would feel a lot better if something like this would become universal. Of course, everything would be much simpler if we could assume that all applications are standard OTP applications, but the reality is that we can't - as a tool provider we have to be able to support even these settings. -------------------------------------- This is just a first draft, I'm sure that I missed some details and others are suboptimal. Please let me know what you think. best regards, Vlad From francesco@REDACTED Tue Sep 14 17:20:56 2010 From: francesco@REDACTED (Francesco Cesarini) Date: Tue, 14 Sep 2010 16:20:56 +0100 Subject: O'Reilly's Erlang Programming $9.99 Ebook Offer (Today Only) Message-ID: <4C8F92D8.3090101@erlang-solutions.com> For those of you not on twitter (Which seems to have gone mad today, in a nice kind of way a quick note to say that O'Reilly has chosen Erlang Programming as the Ebook of the day deal. Today only, you can buy any of the e-book versions for $9.99 Use the code DDEPE when checking out. The URL is http://oreilly.com/catalog/9780596518189/ Cheers, F -- Erlang Solutions Ltd. http://www.erlang-solutions.com From zuav@REDACTED Tue Sep 14 17:43:16 2010 From: zuav@REDACTED (Alexander Zhukov) Date: Tue, 14 Sep 2010 19:43:16 +0400 Subject: Could not attach remote shell from within Emacs shell buffer Message-ID: <87mxrk5o17.fsf@yandex.ru> Hi! When I run the following command from Emacs shell buffer (M-x shell): erl -sname node2@REDACTED -remsh node1@REDACTED I could not connect to node1@REDACTED No messages. Silence. And no connection. But when I run the same command from xterm all goes well. I have absolutely no ideas what goes wrong with Emacs shell. Erlang R14A, Emacs 23.2, Debian Lenny. -- Alexander Zhukov From fritchie@REDACTED Tue Sep 14 18:45:43 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 14 Sep 2010 11:45:43 -0500 Subject: [erlang-questions] About behavior of OTP's supervisor-worker architecture In-Reply-To: Message of "Mon, 13 Sep 2010 23:24:18 +0200." Message-ID: <75035.1284482743@snookles.snookles.com> Hi Torben & Tushar & fellow hackers. Torben Hoffmann wrote: th> The OTP supervisor module does not support your desired behaviour, th> but you can use the supervisor and the monitor facility of Erlang to th> implement it. Torben's description is certainly one way to do it. It looks simpler than what I was going to describe ... and it'll probably work, depending on the nature of the application. You would probably put Torben's gen_server-based monitoring proc elsewhere in the supervisor hierarchy, e.g. top_sup | +---------+-----------+ | | static_sup dynamic_sup | | | | th_monitor_proc proc1 proc2 proc3 ... where 'th_monitor_proc' is the monitoring proc that Torben described. The first problem is that 'th_monitor_proc' will start before 'dynamic_sup' starts, because supervisors start their children from left-to-right, and a static supervisor won't finish its init until all of its children have initialized. One (usually unstated) reason supervisors are so useful is that they start everything in a deterministic, well-ordered manner ... that's very important when controlling hardware (which can be very fussy about the order of operations) or software with strict must-run-before dependencies. So, it would be better to flip the tree like this: top_sup | +---------+-----------+ | | dynamic_sup static_sup | th_monitor_proc Then we don't have who-started-first problems at startup. Then the 'th_monitor_proc' can start the dynamic children: top_sup | +---------+-----------+ | | dynamic_sup static_sup | | | | proc1 proc2 proc3 th_monitor_proc The 'top_sup' supervisor should use a one_for_all restart strategy. After all, the 'dynamic_sup' might fail ... unlikely ... but one of my favorite testing gimmicks is to run the 'appmon' application, get the tree view of supervisor & worker processes, then start clicking 'Kill' on random processes in the tree. :-) Try it. Watching the restarts is entertaining *and* instructive. And it helps demonstrate how the various restart strategies do. You probably do not want to do this: top_sup | +---------+-----------+ | | another_sup static_sup | | +--------+-----+ th_monitor_proc | | dynamic_sup other_whatever | | | proc1 proc2 proc3 ... especially if 'another_sup' has a different restart strategy. If it's possible for 'dynamic_sup' to be killed while 'th_monitor_proc' is alive, then the races where 'th_monitor_proc' tries to restart a child but fails becaues 'dynamic_sup' isn't alive ... that situation is best avoided. -Scott P.S. An astute reader might have a question about the tree below. The question is, "What if the 'top_sup' wants to shut down. Won't 'dynamic_sup' be killed first, and won't I still have the same problem of 'th_monitor_proc' is alive but 'dynamic_sup' is dead?" top_sup | +---------+-----------+ | | dynamic_sup static_sup | th_monitor_proc The answer is "No". The Design Principles guide says: 5.10 Stopping Since the supervisor is part of a supervision tree, it will automatically be terminated by its supervisor. When asked to shutdown, it will terminate all child processes in reversed start order according to the respective shutdown specifications, and then terminate itself. P.P.S. Another solution is to write a module R that calls module P's callbacks the first time that it's run and calls module Q's callbacks when restarted. The "problem" is then shifted to making R be able to figure out if a child is running the first time or it's been restarted. You could write a file in the file system, or keep something in Mnesia or a system-wide ETS table, or other stateful thing. Managing such state is usually costs more than its value, but it can be useful in some situations, especially if you already have to manage state like that for other parts of the app. -s From kenneth.lundin@REDACTED Tue Sep 14 19:22:01 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Tue, 14 Sep 2010 19:22:01 +0200 Subject: [erlang-questions] ets vs list In-Reply-To: References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> <4C8E75EF.7000702@amberbio.com> <4C8F26FD.8070201@erlang-solutions.com> Message-ID: On Tue, Sep 14, 2010 at 11:02 AM, Johan Montelius wrote: > While we're talking ets tables: > > Wouldn't it be nice to be able to send table references across nodes? Today > when you create a new tabel you get a table reference that is an integer. > The table can only be accessed by processes in the same node and god know > what will happen if you send the reference over to another node where there > is another table registered under the same number. >From the very beginning ets-tables almost in the way you suggest i.e the table identifier was location transparent and could be used in any process on any node. OTP R7B released 2000 was the last version that worked like that. Unfortunately I don't remember exactly why we removed the distribution support from ets except that it caused more problems than it solved. I also remember that it was no problem to remove this feature, i.e it was not much used. I will check why we removed it and come back. /Kenneth , Erlang/OTP Ericsson > > Since tables have a mutable state they should be modeled as processess. When > you create a table you get a process identifier. Locally on the node where > the table was created all operations are done as usual i.e. direct access to > the table without the overhead of sending messages. When you send the > process identifier over to another node however, a process identifier is > sent that refers to a proxy process on the originating node. > > Using the functions in the ets module using a regular process identifier > would result in messages being sent to that process and replies being > received. Just like accessing a local ets table (slower and we have to have > some error codes if the process is dead). > > One could the also open up for a asynchronous access to the table: > > ? Ref = ets:lookup_and_send_me_the_reply(Table, Key), > ? do_something(), > ? receive > ? ? ? ?{Ref, Value} -> > > How's that :-) > > ?Johan > > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From fritchie@REDACTED Tue Sep 14 20:21:41 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 14 Sep 2010 13:21:41 -0500 Subject: Erlang-DTrace status? Message-ID: <79781.1284488501@snookles.snookles.com> Howdy. A while back Garry Bulmer and Tim Becker were working on an Erlang-DTrace project. The ZIP file from 2009 that was alleged to be at Sun's OpenSolaris developer thingie is now missing from its successor? http://hub.opensolaris.org/bin/view/Project+erlang-dtrace/files ... now shows me zero downloadable files(*). Was the project source lost in the shuffle to opensolaris.org or the Oracle acquisition or none-of-the-above? -Scott (*) Hopefully I'm not having a bad browser day.... From Antonio.Musumeci@REDACTED Tue Sep 14 20:31:09 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Tue, 14 Sep 2010 14:31:09 -0400 Subject: mnesia:update_element Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3E1EE49@NYWEXMBX2129.msad.ms.com> I assume the reason mnesia has no update_element is due to dets not having it? Is there any specific reason why dets doesn't? From pascalchapier@REDACTED Wed Sep 15 10:44:27 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Wed, 15 Sep 2010 10:44:27 +0200 Subject: mnesia:update_element Message-ID: Hello, Concerning Mnesia there is a powerful update mechanism with: mnesia:transaction(UpdateFunc(Parameters)), I am not sure that one can find a simpler and more general update(Element) model for actions such as add to cart remove from cart apply 10% discount modify phone number... it can be faster in some cases: mnesia:dirty_read, updateFunc, mnesia:dirty_write if concurrency is not a problem, mnesia:dirty_write(#my_table{key = Key, item=Value}), if only item has to be overwritten and my_table is a set (concurrency ?). PC From Antonio.Musumeci@REDACTED Wed Sep 15 13:29:20 2010 From: Antonio.Musumeci@REDACTED (Musumeci, Antonio S) Date: Wed, 15 Sep 2010 07:29:20 -0400 Subject: [erlang-questions] mnesia:update_element In-Reply-To: References: Message-ID: <01489E237C2C8F45AF7898E135F48E8019A3E1EF0D@NYWEXMBX2129.msad.ms.com> Obviously there are ways to simulate it but there are situations where you don't care to read then write in a transaction. Where you could do a dirty read and then a dirty update_element. That's going to be faster than dirty read, transaction(wread + write). -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Pascal Chapier Sent: Wednesday, September 15, 2010 4:44 AM To: erlang-questions@REDACTED Subject: [erlang-questions] mnesia:update_element Hello, Concerning Mnesia there is a powerful update mechanism with: mnesia:transaction(UpdateFunc(Parameters)), I am not sure that one can find a simpler and more general update(Element) model for actions such as add to cart remove from cart apply 10% discount modify phone number... it can be faster in some cases: mnesia:dirty_read, updateFunc, mnesia:dirty_write if concurrency is not a problem, mnesia:dirty_write(#my_table{key = Key, item=Value}), if only item has to be overwritten and my_table is a set (concurrency ?). PC From xavier.erlang@REDACTED Wed Sep 15 14:26:12 2010 From: xavier.erlang@REDACTED (=?iso-8859-1?Q?Xavier_Gu=E9rin?=) Date: Wed, 15 Sep 2010 14:26:12 +0200 Subject: Minimal erlang kenel/compiler Message-ID: Hi list, I would like to be able to execute basic parallel erlang application on my home-made OS, designed for multicore embedded architectures. I tried to look around for minimal erlang configurations / embedded erlang / (put your own adjective here) erlang, without success. I also looked into the sources, and I could not get anything out of this investigation. Could anyone tell me how to configure such a minimal setting and how to port erlang to another OS ? Thank you for your help, Xavier ---- Xavier Gu?rin - Senior Research Engineer TIMA Laboratory, SLS Group 46, av. F?lix Viallet, 38000 GRENOBLE, FR Mail : xavier.guerin@REDACTED Phone : +33(0)4 76 57 47 59 Ad Astra Per Aspera Triste e' quel discepolo che non avanza il suo maestro Ubi patria, ibi bene From gustav.simonsson@REDACTED Wed Sep 15 14:59:01 2010 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Wed, 15 Sep 2010 12:59:01 +0000 (GMT) Subject: [erlang-questions] Minimal erlang kenel/compiler In-Reply-To: Message-ID: <1747496751.799811284555541698.JavaMail.root@zimbra> Hi! I took the liberty of forwarding your mail to erlang-embedded@REDACTED It's a mailing list for people interested in topics related to Erlang in embedded systems/applications. Regards, Gustav Simonsson. ----- Original Message ----- From: "Xavier Gu?rin" To: erlang-questions@REDACTED Sent: Wednesday, 15 September, 2010 14:26:12 GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: [erlang-questions] Minimal erlang kenel/compiler Hi list, I would like to be able to execute basic parallel erlang application on my home-made OS, designed for multicore embedded architectures. I tried to look around for minimal erlang configurations / embedded erlang / (put your own adjective here) erlang, without success. I also looked into the sources, and I could not get anything out of this investigation. Could anyone tell me how to configure such a minimal setting and how to port erlang to another OS ? Thank you for your help, Xavier ---- Xavier Gu?rin - Senior Research Engineer TIMA Laboratory, SLS Group 46, av. F?lix Viallet, 38000 GRENOBLE, FR Mail : xavier.guerin@REDACTED Phone : +33(0)4 76 57 47 59 Ad Astra Per Aspera Triste e' quel discepolo che non avanza il suo maestro Ubi patria, ibi bene From tiemen.meerman@REDACTED Wed Sep 15 14:54:34 2010 From: tiemen.meerman@REDACTED (Tiemen Meerman) Date: Wed, 15 Sep 2010 13:54:34 +0100 Subject: [erlang-questions] Minimal erlang kenel/compiler In-Reply-To: References: Message-ID: <5A53ABFB-7699-490E-B331-90C7BF194D64@phorm.com> Xavier, I have just started a similar research project, although my aim was to run Erlang *without* an underlying OS on the NetLogic XLR/XLP processor. I have the feeling the questions to be answered are not completely dissimilar to your project. I think the main areas of interest are making sure the scheduler and I/ O runs happily (requires either thread support or poll+select support). Any other implementation details depend on your architecture: filesystem availability, how you want to segment your multicore processor: 1 VM per chip, many VMs per chip, will a networking stack be available, etc. There are some other UNIX / Windows dependencies, but they are very much localized. -T On 15 Sep 2010, at 13:26, Xavier Gu?rin wrote: > Hi list, > > I would like to be able to execute basic parallel erlang application > on my home-made OS, designed for multicore embedded architectures. I > tried to look around for minimal erlang configurations / embedded > erlang / (put your own adjective here) erlang, without success. I > also looked into the sources, and I could not get anything out of > this investigation. > > Could anyone tell me how to configure such a minimal setting and how > to port erlang to another OS ? > > Thank you for your help, > Xavier > > ---- > Xavier Gu?rin - Senior Research Engineer > TIMA Laboratory, SLS Group > 46, av. F?lix Viallet, 38000 GRENOBLE, FR > Mail : xavier.guerin@REDACTED > Phone : +33(0)4 76 57 47 59 > Ad Astra Per Aspera > Triste e' quel discepolo che non avanza il suo maestro > Ubi patria, ibi bene > Tiemen Meerman Deployment Team Leader Phorm UK, Inc. Liberty House 222 Regent Street London W1B 5TR T: +44 (0)207 297 2347 M: +44 (0)7825 505627 tiemen.meerman@REDACTED http://www.phorm.com/email_confidentiality.html http://tiemen.org/phorm-pubkey.gpg From kenneth.lundin@REDACTED Wed Sep 15 16:22:23 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Wed, 15 Sep 2010 16:22:23 +0200 Subject: Erlang/OTP R14B has been released Message-ID: Erlang/OTP R14B has been released. This release is mainly a stabilization of the R14A release (which was a beta) but there are some new functionality also. You can find the README file for the release at http://www.erlang.org/download/otp_src_R14B.readme The source distribution and binary distribution for Windows can be downloaded from http://www.erlang.org/download/otp_src_R14B.tar.gz http://www.erlang.org/download/otp_win32_R14B.exe The distribution can also be downloaded using the BitTorrent protocol. Use the following torrent files to download the source distribution and binary distribution for Windows: http://www.erlang.org/download/otp_src_R14B.tar.gz.torrent http://www.erlang.org/download/otp_win32_R14B.exe.torrent Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please read the README file that is part of the distribution. The on-line documentation can be found at: http://www.erlang.org/doc/ You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_R14B.tar.gz http://www.erlang.org/download/otp_doc_man_R14B.tar.gz We also want to thank those that sent us patches, suggestions and bug reports, The OTP Team Kenneth Lundin, Erlang/OTP, Ericsson AB From boris.muehmer@REDACTED Wed Sep 15 19:20:06 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Wed, 15 Sep 2010 19:20:06 +0200 Subject: snmpwalk like function Message-ID: Is there a simple "snmpwalk" (http://net-snmp.sourceforge.net/wiki/index.php/TUT:snmpwalk) like function in Erlang/OTP, please? Or is there a snmp manager needed to get/collect data from other systems? Thanks in advance, - boris From zabrane3@REDACTED Wed Sep 15 22:30:36 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Wed, 15 Sep 2010 22:30:36 +0200 Subject: otp_src_R14B "Lock check assertion" problem Message-ID: Hi, I've updated my dev server to "otp_src_R14B". After compiling my application, trying to running it throws this error: ... beam/global.h:1323: Lock check assertion "erts_lc_is_port_locked(&erts_port[ix])" failed! Currently no locks are locked by the async 42 thread. aborted Please, advices? N.B: I'm under Fedora13, 64bits. The same error was thrown under "Snow Leopard" -- Regards Zabrane From zabrane3@REDACTED Wed Sep 15 23:40:23 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Wed, 15 Sep 2010 23:40:23 +0200 Subject: otp_src_R14B "Lock check assertion" problem In-Reply-To: References: Message-ID: Answering my own question. That was due to: --enable-lock-checking Could someone please tell me what this option is for? -- Regards Zabrane 2010/9/15 zabrane Mikael : > Hi, > > I've updated my dev server to "otp_src_R14B". > After compiling my application, trying to running it throws this error: > > ... > beam/global.h:1323: Lock check assertion > "erts_lc_is_port_locked(&erts_port[ix])" failed! > Currently no locks are locked by the async 42 thread. > aborted > > Please, advices? > > N.B: I'm under Fedora13, 64bits. The same error was thrown under "Snow Leopard" > -- > Regards > Zabrane > From rickard@REDACTED Thu Sep 16 00:13:09 2010 From: rickard@REDACTED (Rickard Green) Date: Thu, 16 Sep 2010 00:13:09 +0200 Subject: otp_src_R14B "Lock check assertion" problem Message-ID: <4C9144F5.8070401@erlang.org> zabrane Mikael wrote: > Answering my own question. That was due to: > --enable-lock-checking > > Could someone please tell me what this option is for? > > -- > Regards > Zabrane > > > 2010/9/15 zabrane Mikael < zabrane3 ? gmail ? com >: >> Hi, >> >> I've updated my dev server to "otp_src_R14B". >> After compiling my application, trying to running it throws this error: >> >> ... >> beam/global.h:1323: Lock check assertion >> "erts_lc_is_port_locked(&erts_port[ix])" failed! >> Currently no locks are locked by the async 42 thread. >> aborted >> >> Please, advices? >> >> N.B: I'm under Fedora13, 64bits. The same error was thrown under "Snow Leopard" >> -- >> Regards >> Zabrane >> Yes this was due to the --enable-lock-checking switch which is for debugging. However, the assertion should not have been triggered. It would be good to have a look at a core-dump or at least a stackdump. Enable core dumps either like this (bash, sh): $ ulimit -c unlimited or like this (csh, tcsh): > limit coredumpsize unlimited Run your app until the assertion is triggered. Print a stackdump (where is beam.smp or beam in non-smp case: $ gdb $ERL_TOP/bin//bin/ (gdb) bt ... (typing 'quit' at the gdb prompt will get you out of gdb) Mail us all that gdb printed. If you can provide us with the emulator and the coredump, do that too (preferably put them somewhere where we can download them). Regards, Rickard Green -- Rickard Green, Erlang/OTP, Ericsson AB. From rickard@REDACTED Thu Sep 16 00:17:29 2010 From: rickard@REDACTED (Rickard Green) Date: Thu, 16 Sep 2010 00:17:29 +0200 Subject: otp_src_R14B "Lock check assertion" problem In-Reply-To: <4C9144F5.8070401@erlang.org> References: <4C9144F5.8070401@erlang.org> Message-ID: <4C9145F9.9070100@erlang.org> Rickard Green wrote: > Enable core dumps either like this (bash, sh): > $ ulimit -c unlimited > or like this (csh, tcsh): > > limit coredumpsize unlimited > Do this on linux, not on macosx where it is messier than this. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. From kenji.rikitake@REDACTED Thu Sep 16 05:17:52 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Thu, 16 Sep 2010 12:17:52 +0900 Subject: R14B FreeBSD unofficial port Message-ID: <20100916031752.GA35455@k2r.org> The following tar.gz archive includes an unofficial port of Erlang R14B for FreeBSD (erlang-r14b,1). Note 1: R14B official port is under review by the port maintainer Giacomo Olgeni. This port is a patched version of his preliminary port. Note 2: Java functionality untested. Note 3: Compilation/installation tested on FreeBSD 7.3-RELEASE i386. http://www.ne.jp/asahi/bdx/info/software/erlang-r14b.k2r.20100916.tar.gz Regards, Kenji Rikitake From rzezeski@REDACTED Thu Sep 16 06:06:10 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Thu, 16 Sep 2010 00:06:10 -0400 Subject: R14B, ETS, read_concurrency Message-ID: After glancing at the R14B release notes I noticed the SMP related improvements that were made. I specifically find the new read_concurrency ETS opt interesting. I currently have an app that performs roughly 480 million lookups (in worst case) on multiple ETS tables, with some tables having up to 10 million entries. I perform these lookups in parallel using 8 processes (on a 16 core machine). This is the longest running part of my program with a runtime of > 3min. My point--I am doing a _huge_ amount of concurrent reads and every microsecond counts. I am curious if anyone could throw a minimum % increase of lookups per second I might expect? -Ryan From kenji.rikitake@REDACTED Thu Sep 16 06:31:22 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Thu, 16 Sep 2010 13:31:22 +0900 Subject: R14B FreeBSD unofficial port (revised) Message-ID: <20100916043122.GA63296@k2r.org> The following tar.gz archive includes an unofficial port of Erlang R14B for FreeBSD (erlang-r14b,1). The previous one had a wrong distinfo - this one has the right one. Note 1: R14B official port is under review by the port maintainer Giacomo Olgeni. This port is a patched version of his preliminary port. Note 2: Java functionality untested. Note 3: Compilation/installation tested on FreeBSD 7.3-RELEASE i386. http://www.ne.jp/asahi/bdx/info/software/erlang-r14b.k2r.20100916-2.tar.gz Regards, Kenji Rikitake From fdmanana@REDACTED Thu Sep 16 13:33:47 2010 From: fdmanana@REDACTED (Filipe David Manana) Date: Thu, 16 Sep 2010 12:33:47 +0100 Subject: weird badinfo error when using SSL socket Message-ID: Hello, does anyone ever had the following error when using SSL socket in passive mode ({active, once})? cheers [Thu, 16 Sep 2010 00:10:34 GMT] [error] [<0.604.0>] ** Generic server <0.604.0> terminating ** Last message in was {tcp,#Port<0.2288>, <<"\r\n6d\r\n,\n{\"seq\":70,\"id\":\"97b36d5003934d0c9dd58057b05fa167\",\"changes\":[{\"rev\":\"1-0d6deda5b380ae207ba87a7a3a32d0a1\"}]}\r\n6d\r\n,\n{\"seq\":71,\"id\":\"8a1c475b8dc5426e9172d6b970ae7c03\",\"changes\":[{\"rev\":\"1-72851f645fb6ab77f36866cbe505d82c\"}]}\r\n6d\r\n,\n{\"seq\":72,\"id\":\"fdb1d5b1c5b24ce481463ad668c13c40\",\"changes\":[{\"rev\":\"1-c37b5444eec8375631c326a0e77ca427\"}]}\r\n6d\r\n,\n{\"seq\":73,\"id\":\"b612465dafc44699b09d8bef5d4d4d8d\",\"changes\":[{\"rev\":\"1-be951f78ba830f5a1002abe0ce479c2d\"}]}\r\n6d\r\n,\n{\"seq\":74,\"id\":\"d2c2b5a771ef4b57b6d58fce2808cf7c\",\"changes\":[{\"rev\":\"1-c628443ff4dd7c3d9b4fd226727e2841\"}]}\r\n6d\r\n,\n{\"seq\":75,\"id\":\"8d669c377f08442981ce2d18a21d920b\",\"changes\":[{\"rev\":\"1-6db3a14c76701b87b0686412093ac103\"}]}\r\n6d\r\n,\n{\"seq\":76,\"id\":\"367bf0948d9d459582d187c9232844b8\",\"changes\":[{\"rev\":\"1-16ae7cf1c04c4f7c024493de1f18c8ed\"}]}\r\n6d\r\n,\n{\"seq\":77,\"id\":\"f2c805327ae740098e5db221c3f27b4b\",\"changes\":[{\"rev\":\"1-b22aa541f7e353a4cd430a9293239c77\"}]}\r\n6d\r\n,\n{\"seq\":78,\"id\":\"6ddf8033cec845c8986ee4bd03ff8ed6\",\"changes\":[{\"rev\":\"1-23f5957d250f5079277e6e4a86def1f1\"}]}\r\n6d\r\n,\n{\"seq\":79,\"id\":\"738365bd4fed44158516211847c13616\",\"changes\":[{\"rev\":\"1-6dcd375366f107fb2575c8eda6c6bdec\"}]}\r\n6d\r\n,\n{\"seq\":80,\"id\":\"2d66c797761b4506934d00b2fd260f90\",\"changes\":[{\"rev\":\"1-cc7dddd31fd753a9b4577607ce321cef\"}]}\r\n6d\r\n,\n{\"seq\":81,\"id\":\"0c01c012d4f540a3a015d57681a0af4f\",\"changes\":[{\"rev\":\"1-ff288fbba546fbfbf78c602e2fa39ea2\"}]}\r\n6d\r\n,\n{\"seq\":82,\"id\":\"dc8a7ff04d37428ea83c3515a801bd32\",\"changes\":[{\"rev\":\"1-2">>} ** When Server state == {st,connector,<0.119.0>,<0.603.0>,<0.603.0>,11,false, [{mode,binary}, {nodelay,true}, {active,once}, {packet,0}, {ip,{0,0,0,0}}, {verify,0}, {depth,1}], {sslsocket,11,<0.604.0>}, #Port<0.2288>,nil,open,false,false} ** Reason for termination == ** {error,{badinfo,{tcp,#Port<0.2288>, <<"\r\n6d\r\n,\n{\"seq\":70,\"id\":\"97b36d5003934d0c9dd58057b05fa167\",\"changes\":[{\"rev\":\"1-0d6deda5b380ae207ba87a7a3a32d0a1\"}]}\r\n6d\r\n,\n{\"seq\":71,\"id\":\"8a1c475b8dc5426e9172d6b970ae7c03\",\"changes\":[{\"rev\":\"1-72851f645fb6ab77f36866cbe505d82c\"}]}\r\n6d\r\n,\n{\"seq\":72,\"id\":\"fdb1d5b1c5b24ce481463ad668c13c40\",\"changes\":[{\"rev\":\"1-c37b5444eec8375631c326a0e77ca427\"}]}\r\n6d\r\n,\n{\"seq\":73,\"id\":\"b612465dafc44699b09d8bef5d4d4d8d\",\"changes\":[{\"rev\":\"1-be951f78ba830f5a1002abe0ce479c2d\"}]}\r\n6d\r\n,\n{\"seq\":74,\"id\":\"d2c2b5a771ef4b57b6d58fce2808cf7c\",\"changes\":[{\"rev\":\"1-c628443ff4dd7c3d9b4fd226727e2841\"}]}\r\n6d\r\n,\n{\"seq\":75,\"id\":\"8d669c377f08442981ce2d18a21d920b\",\"changes\":[{\"rev\":\"1-6db3a14c76701b87b0686412093ac103\"}]}\r\n6d\r\n,\n{\"seq\":76,\"id\":\"367bf0948d9d459582d187c9232844b8\",\"changes\":[{\"rev\":\"1-16ae7cf1c04c4f7c024493de1f18c8ed\"}]}\r\n6d\r\n,\n{\"seq\":77,\"id\":\"f2c805327ae740098e5db221c3f27b4b\",\"changes\":[{\"rev\":\"1-b22aa541f7e353a4cd430a9293239c77\"}]}\r\n6d\r\n,\n{\"seq\":78,\"id\":\"6ddf8033cec845c8986ee4bd03ff8ed6\",\"changes\":[{\"rev\":\"1-23f5957d250f5079277e6e4a86def1f1\"}]}\r\n6d\r\n,\n{\"seq\":79,\"id\":\"738365bd4fed44158516211847c13616\",\"changes\":[{\"rev\":\"1-6dcd375366f107fb2575c8eda6c6bdec\"}]}\r\n6d\r\n,\n{\"seq\":80,\"id\":\"2d66c797761b4506934d00b2fd260f90\",\"changes\":[{\"rev\":\"1-cc7dddd31fd753a9b4577607ce321cef\"}]}\r\n6d\r\n,\n{\"seq\":81,\"id\":\"0c01c012d4f540a3a015d57681a0af4f\",\"changes\":[{\"rev\":\"1-ff288fbba546fbfbf78c602e2fa39ea2\"}]}\r\n6d\r\n,\n{\"seq\":82,\"id\":\"dc8a7ff04d37428ea83c3515a801bd32\",\"changes\":[{\"rev\":\"1-2">>}}} [Thu, 16 Sep 2010 00:10:34 GMT] [error] [<0.604.0>] {error_report,<0.62.0>, {<0.604.0>,crash_report, [[{initial_call,{ssl_broker,init,['Argument__1']}}, {pid,<0.604.0>}, {registered_name,[]}, {error_info, {exit, {error, {badinfo, {tcp,#Port<0.2288>, <<"\r\n6d\r\n,\n{\"seq\":70,\"id\":\"97b36d5003934d0c9dd58057b05fa167\",\"changes\":[{\"rev\":\"1-0d6deda5b380ae207ba87a7a3a32d0a1\"}]}\r\n6d\r\n,\n{\"seq\":71,\"id\":\"8a1c475b8dc5426e9172d6b970ae7c03\",\"changes\":[{\"rev\":\"1-72851f645fb6ab77f36866cbe505d82c\"}]}\r\n6d\r\n,\n{\"seq\":72,\"id\":\"fdb1d5b1c5b24ce481463ad668c13c40\",\"changes\":[{\"rev\":\"1-c37b5444eec8375631c326a0e77ca427\"}]}\r\n6d\r\n,\n{\"seq\":73,\"id\":\"b612465dafc44699b09d8bef5d4d4d8d\",\"changes\":[{\"rev\":\"1-be951f78ba830f5a1002abe0ce479c2d\"}]}\r\n6d\r\n,\n{\"seq\":74,\"id\":\"d2c2b5a771ef4b57b6d58fce2808cf7c\",\"changes\":[{\"rev\":\"1-c628443ff4dd7c3d9b4fd226727e2841\"}]}\r\n6d\r\n,\n{\"seq\":75,\"id\":\"8d669c377f08442981ce2d18a21d920b\",\"changes\":[{\"rev\":\"1-6db3a14c76701b87b0686412093ac103\"}]}\r\n6d\r\n,\n{\"seq\":76,\"id\":\"367bf0948d9d459582d187c9232844b8\",\"changes\":[{\"rev\":\"1-16ae7cf1c04c4f7c024493de1f18c8ed\"}]}\r\n6d\r\n,\n{\"seq\":77,\"id\":\"f2c805327ae740098e5db221c3f27b4b\",\"changes\":[{\"rev\":\"1-b22aa541f7e353a4cd430a9293239c77\"}]}\r\n6d\r\n,\n{\"seq\":78,\"id\":\"6ddf8033cec845c8986ee4bd03ff8ed6\",\"changes\":[{\"rev\":\"1-23f5957d250f5079277e6e4a86def1f1\"}]}\r\n6d\r\n,\n{\"seq\":79,\"id\":\"738365bd4fed44158516211847c13616\",\"changes\":[{\"rev\":\"1-6dcd375366f107fb2575c8eda6c6bdec\"}]}\r\n6d\r\n,\n{\"seq\":80,\"id\":\"2d66c797761b4506934d00b2fd260f90\",\"changes\":[{\"rev\":\"1-cc7dddd31fd753a9b4577607ce321cef\"}]}\r\n6d\r\n,\n{\"seq\":81,\"id\":\"0c01c012d4f540a3a015d57681a0af4f\",\"changes\":[{\"rev\":\"1-ff288fbba546fbfbf78c602e2fa39ea2\"}]}\r\n6d\r\n,\n{\"seq\":82,\"id\":\"dc8a7ff04d37428ea83c3515a801bd32\",\"changes\":[{\"rev\":\"1-2">>}}}, [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, -- Filipe David Manana, fdmanana@REDACTED, fdmanana@REDACTED "Reasonable men adapt themselves to the world. ?Unreasonable men adapt the world to themselves. ?That's why all progress depends on unreasonable men." From rickard@REDACTED Thu Sep 16 14:13:54 2010 From: rickard@REDACTED (Rickard Green) Date: Thu, 16 Sep 2010 14:13:54 +0200 Subject: R14B, ETS, read_concurrency Message-ID: <4C920A02.8080506@erlang.org> Ryan Zezeski wrote: > After glancing at the R14B release notes I noticed the SMP related > improvements that were made. I specifically find the new read_concurrency > ETS opt interesting. I currently have an app that performs roughly 480 > million lookups (in worst case) on multiple ETS tables, with some tables > having up to 10 million entries. I perform these lookups in parallel using > 8 processes (on a 16 core machine). This is the longest running part of my > program with a runtime of > 3min. > > My point--I am doing a _huge_ amount of concurrent reads and every > microsecond counts. I am curious if anyone could throw a minimum % increase > of lookups per second I might expect? > > -Ryan What the effect will be on your application is impossible to guess. This depends on how writes are done in the tables using the read_concurrency option (if you have lots of write locked operations too you may even get a performance degradation), what else your processes do (also other processes not accessing the tables), what hardware you use, etc... In order to give some figures, though. When doing lots of ets lookups (only lookups) of small records on a public table from lots (hundreds) of processes I get a speedup of about 5 using read_concurrency compared to not using it on a dual quad-core machine we got, and about 2.5 on a quad-core machine we got. The only thing these processes do are repeated ets lookups. You should, however, not expect figures like these for normal apps, since they typically do other stuff than ets lookup too. The only way to know what effect it will have on your app is to try it out. Regards, Rickard Green -- Rickard Green, Erlang/OTP, Ericsson AB. From alexey.v.romanov@REDACTED Thu Sep 16 16:49:44 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Thu, 16 Sep 2010 18:49:44 +0400 Subject: Cross-compiling with OpenSSL Message-ID: It turned out I need the crypto application in my embedded Erlang system. So I've cross-compiled OpenSSL 1.0.0a using this approach: http://stuff.thatblogs.com/content/cross-compiling-openssl-how-cross-compile-openssl, copied apps/openssl, include/openssl/*, lib/libcrypto.a, and lib/libssl.a (is anything else needed, by the way?) to ~/sbctools/arm-2007q3/openssl/1.0.0a and issued this command: ./otp_build configure --host=arm-none-linux-gnueabi --build=i686-pc-linux-gnu --prefix=/opt/erlang erl_xcomp_sysroot=~/sbctools/arm-2007q3 --disable-smp --disable-megaco-flex-scanner-lineno --disable-megaco-reentrant-flex-scanner --without-termcap --without-javac --with-ssl=~/sbctools/arm-2007q3/openssl/1.0.0a Surprisingly, this gives configure: error: Invalid path to option --with-ssl=PATH (not a subdirectory to cross system root) Certainly ~/sbctools/arm-2007q3/openssl/1.0.0a is a subdirectory of ~/sbctools/arm-2007q3. I've tried to supply the path from the cross system root instead (i.e. --with-ssl=openssl/1.0.0a), but this doesn't work either. How can I fix this? Yours, Alexey Romanov From rzezeski@REDACTED Thu Sep 16 17:42:44 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Thu, 16 Sep 2010 11:42:44 -0400 Subject: R14B, ETS, read_concurrency In-Reply-To: <4C920A02.8080506@erlang.org> References: <4C920A02.8080506@erlang.org> Message-ID: On Thu, Sep 16, 2010 at 8:13 AM, Rickard Green wrote: > > The only way to know what effect it will have on your app is to try it > out. Rickard, you're exactly right. I should have worded my question differently. I'm interested in any results anyone has using this new feature, and that's exactly what you gave me, thanks. In my case, I have highly concurrent reads with _zero_ interrupting writes. The writes are always done in a sequential manner before the reads. Also, I'm not only doing reads in my tight loop, but it is the dominator in my profile. It seems my app may benefit from this new feature. I'll report back when I get some time to try it. From rickard.s.green@REDACTED Thu Sep 16 17:46:33 2010 From: rickard.s.green@REDACTED (Rickard Green) Date: Thu, 16 Sep 2010 17:46:33 +0200 Subject: Cross-compiling with OpenSSL Message-ID: <4C923BD9.2080501@ericsson.com> Alexey Romanov wrote: > It turned out I need the crypto application in my embedded Erlang > system. So I've cross-compiled OpenSSL 1.0.0a using this approach: > http://stuff.thatblogs.com/content/cross-compiling-openssl-how-cross-compile-openssl, > copied apps/openssl, include/openssl/*, lib/libcrypto.a, and > lib/libssl.a (is anything else needed, by the way?) to > ~/sbctools/arm-2007q3/openssl/1.0.0a and issued this command: > > ./otp_build configure --host=arm-none-linux-gnueabi > --build=i686-pc-linux-gnu --prefix=/opt/erlang > erl_xcomp_sysroot=~/sbctools/arm-2007q3 --disable-smp > --disable-megaco-flex-scanner-lineno > --disable-megaco-reentrant-flex-scanner --without-termcap > --without-javac --with-ssl=~/sbctools/arm-2007q3/openssl/1.0.0a > > Surprisingly, this gives > > configure: error: Invalid path to option --with-ssl=PATH (not a > subdirectory to cross system root) > > Certainly ~/sbctools/arm-2007q3/openssl/1.0.0a is a subdirectory of > ~/sbctools/arm-2007q3. I've tried to supply the path from the cross > system root instead (i.e. --with-ssl=openssl/1.0.0a), but this doesn't > work either. How can I fix this? > > Yours, Alexey Romanov otp_build expands tilde for erl_xcomp_sysroot, but not for --with-ssl, so if you use absolute paths beginning with / it should work. You seem to use an r13 since the ssl path isn't required to have any relation to the sysroot in r14. The problem should also go away if you use r14 instead of r13. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. From alexey.v.romanov@REDACTED Thu Sep 16 17:56:19 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Thu, 16 Sep 2010 19:56:19 +0400 Subject: Cross-compiling with OpenSSL In-Reply-To: <4C923BD9.2080501@ericsson.com> References: <4C923BD9.2080501@ericsson.com> Message-ID: On Thu, Sep 16, 2010 at 7:46 PM, Rickard Green wrote: > Alexey Romanov wrote: >> >> It turned out I need the crypto application in my embedded Erlang >> system. So I've cross-compiled OpenSSL 1.0.0a using this approach: >> >> http://stuff.thatblogs.com/content/cross-compiling-openssl-how-cross-compile-openssl, >> copied apps/openssl, include/openssl/*, lib/libcrypto.a, and >> lib/libssl.a (is anything else needed, by the way?) to >> ~/sbctools/arm-2007q3/openssl/1.0.0a and issued this command: >> >> ./otp_build configure --host=arm-none-linux-gnueabi >> --build=i686-pc-linux-gnu --prefix=/opt/erlang >> erl_xcomp_sysroot=~/sbctools/arm-2007q3 --disable-smp >> --disable-megaco-flex-scanner-lineno >> --disable-megaco-reentrant-flex-scanner --without-termcap >> --without-javac --with-ssl=~/sbctools/arm-2007q3/openssl/1.0.0a >> >> Surprisingly, this gives >> >> configure: error: Invalid path to option --with-ssl=PATH (not a >> subdirectory to cross system root) >> >> Certainly ~/sbctools/arm-2007q3/openssl/1.0.0a is a subdirectory of >> ~/sbctools/arm-2007q3. I've tried to supply the path from the cross >> system root instead (i.e. --with-ssl=openssl/1.0.0a), but this doesn't >> work either. How can I fix this? >> >> Yours, Alexey Romanov > > otp_build expands tilde for erl_xcomp_sysroot, but not for --with-ssl, so if > you use absolute paths beginning with / it should work. Thanks! > You seem to use an r13 since the ssl path isn't required to have any > relation to the sysroot in r14. The problem should also go away if you use > r14 instead of r13. Yes, I am currently on R13, was planning to switch soon. > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB. > From joe@REDACTED Thu Sep 16 18:55:35 2010 From: joe@REDACTED (Joe Williams) Date: Thu, 16 Sep 2010 09:55:35 -0700 Subject: undefined reference to 'erts_restore_fpu' valgrind debug build in R13B04 Message-ID: <344327EB-5424-45DC-8C40-1CC8699D0477@joetify.com> I am attempting to build (make valgrind FLAVOR=smp) a valgrind runtime and received the following error. Note that a normal debug build worked fine. obj/x86_64-unknown-linux-gnu/valgrind/smp/hipe_amd64_bifs.o: In function `nbif_handle_fp_exception': /home/joe/otp_src_R13B04/erts/emulator/x86_64-unknown-linux-gnu/valgrind/smp/hipe_amd64_bifs.S:287: undefined reference to `erts_restore_fpu' collect2: ld returned 1 exit status make[1]: *** [/home/joe/otp_src_R13B04/bin/x86_64-unknown-linux-gnu/beam.valgrind.smp] Error 1 make[1]: Leaving directory `/home/joe/otp_src_R13B04/erts/emulator' make: *** [valgrind] Error 2 In some old mailing list posts I found that it might be related to unreliable floating point support. Running configure in erts it looks good: checking for unreliable floating point execptions... reliable I assume it's related to hipe being enabled. Anyone else run into this? Thanks. -Joe Name: Joseph A. Williams Email: joe@REDACTED Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From joe@REDACTED Thu Sep 16 19:26:52 2010 From: joe@REDACTED (Joe Williams) Date: Thu, 16 Sep 2010 10:26:52 -0700 Subject: [erlang-questions] undefined reference to 'erts_restore_fpu' valgrind debug build in R13B04 In-Reply-To: <344327EB-5424-45DC-8C40-1CC8699D0477@joetify.com> References: <344327EB-5424-45DC-8C40-1CC8699D0477@joetify.com> Message-ID: <7538038D-C7E6-4ED9-BDCD-D418B6735224@joetify.com> FWIW, I did a make clean, configured without hipe, rebuilt and got the same result. -Joe On Sep 16, 2010, at 9:55 AM, Joe Williams wrote: > I am attempting to build (make valgrind FLAVOR=smp) a valgrind runtime and received the following error. Note that a normal debug build worked fine. > > > obj/x86_64-unknown-linux-gnu/valgrind/smp/hipe_amd64_bifs.o: In function `nbif_handle_fp_exception': > /home/joe/otp_src_R13B04/erts/emulator/x86_64-unknown-linux-gnu/valgrind/smp/hipe_amd64_bifs.S:287: undefined reference to `erts_restore_fpu' > collect2: ld returned 1 exit status > make[1]: *** [/home/joe/otp_src_R13B04/bin/x86_64-unknown-linux-gnu/beam.valgrind.smp] Error 1 > make[1]: Leaving directory `/home/joe/otp_src_R13B04/erts/emulator' > make: *** [valgrind] Error 2 > > In some old mailing list posts I found that it might be related to unreliable floating point support. Running configure in erts it looks good: > > > checking for unreliable floating point execptions... reliable > > > I assume it's related to hipe being enabled. Anyone else run into this? > > Thanks. > -Joe > > > Name: Joseph A. Williams > Email: joe@REDACTED > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > Name: Joseph A. Williams Email: joe@REDACTED Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From joe@REDACTED Thu Sep 16 19:47:02 2010 From: joe@REDACTED (Joe Williams) Date: Thu, 16 Sep 2010 10:47:02 -0700 Subject: [erlang-questions] undefined reference to 'erts_restore_fpu' valgrind debug build in R13B04 In-Reply-To: <7538038D-C7E6-4ED9-BDCD-D418B6735224@joetify.com> References: <344327EB-5424-45DC-8C40-1CC8699D0477@joetify.com> <7538038D-C7E6-4ED9-BDCD-D418B6735224@joetify.com> Message-ID: <11393459-F17E-4556-B416-392AE8307F8A@joetify.com> Scratch that, hipe wasn't actually disabled. It indeed does work without hipe enabled. -Joe On Sep 16, 2010, at 10:26 AM, Joe Williams wrote: > > FWIW, I did a make clean, configured without hipe, rebuilt and got the same result. > > -Joe > > > On Sep 16, 2010, at 9:55 AM, Joe Williams wrote: > >> I am attempting to build (make valgrind FLAVOR=smp) a valgrind runtime and received the following error. Note that a normal debug build worked fine. >> >> >> obj/x86_64-unknown-linux-gnu/valgrind/smp/hipe_amd64_bifs.o: In function `nbif_handle_fp_exception': >> /home/joe/otp_src_R13B04/erts/emulator/x86_64-unknown-linux-gnu/valgrind/smp/hipe_amd64_bifs.S:287: undefined reference to `erts_restore_fpu' >> collect2: ld returned 1 exit status >> make[1]: *** [/home/joe/otp_src_R13B04/bin/x86_64-unknown-linux-gnu/beam.valgrind.smp] Error 1 >> make[1]: Leaving directory `/home/joe/otp_src_R13B04/erts/emulator' >> make: *** [valgrind] Error 2 >> >> In some old mailing list posts I found that it might be related to unreliable floating point support. Running configure in erts it looks good: >> >> >> checking for unreliable floating point execptions... reliable >> >> >> I assume it's related to hipe being enabled. Anyone else run into this? >> >> Thanks. >> -Joe >> >> >> Name: Joseph A. Williams >> Email: joe@REDACTED >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > Name: Joseph A. Williams > Email: joe@REDACTED > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > Name: Joseph A. Williams Email: joe@REDACTED Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From kostis@REDACTED Thu Sep 16 23:54:15 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Fri, 17 Sep 2010 00:54:15 +0300 Subject: errors in gb_sets and gn_trees Message-ID: <4C929207.8080204@cs.ntua.gr> Modules gb_sets and gb_trees each define an insert/2 function whose documentation (for gb_sets) reads: insert(Element, Set1) -> Set2 Types: Element = term() Set1 = Set2 = gb_set() Returns a new gb_set formed from Set1 with Element inserted. Assumes that Element is not present in Set1. and whose code reads: insert(Key, Val, {S, T}) when is_integer(S) -> S1 = S+1, {S1, insert_1(Key, Val, T, ?pow(S1, ?p))}. insert_1(Key, Value, {Key1, V, Smaller, Bigger}, S) when Key < Key1 -> case insert_1(Key, Value, Smaller, ?div2(S)) of .... insert_1(Key, _, _, _) -> erlang:error({key_exists, Key}). Should the last line be a call to erlang:error/1 or to erlang:throw/1 ? Kostis From roberto@REDACTED Fri Sep 17 01:08:45 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 17 Sep 2010 01:08:45 +0200 Subject: R14B gen_tcp:recv/2,3 Message-ID: dear list, i read from R14B release's readme: http://erlang.org/download/otp_src_R14B.readme OTP-8831 For a socket in the HTTP packet mode, the return value from gen_tcp:recv/2,3 if there is an error in the header will be {ok,{http_error,String}} instead of {error,{http_error,String}} to be consistent with ssl:recv/2,3. i'm not really sure i did understand well: does this mean that all code which relies on gen_tcp:recv/2,3 needs to be rewritten? for instance, will this code generate a no matching clause in R14B in case an error is produced? case gen_tcp:recv(Sock, Len, RecvTimeout) of {ok, Bin} -> {ok, Bin}; {error, Reason} -> {error, Reason} end. i, for one, surely hope i've misinterpreted this statement.. r. From fdmanana@REDACTED Fri Sep 17 01:18:53 2010 From: fdmanana@REDACTED (Filipe David Manana) Date: Fri, 17 Sep 2010 00:18:53 +0100 Subject: new_ssl in R13B03 Message-ID: Hello, I'm trying to use the new_ssl module in R13B03. In the ssl:connect call I pass the options {ssl_imp, new} and {verify, 0}. However I get the error: =ERROR REPORT==== 16-Sep-2010::23:47:07 === SSL: certify_certificate: ./ssl_handshake.erl:486:Fatal error: handshake_failure ** exception error: no match of right hand side value {error,esslconnect} in function ssl_test:test/0 Tried as well passing {verify, verify_none} and {fail_if_no_peer_cert, false} instead of {verify, 0} (since they're mentioned in the man page for new_ssl module) but doesn't work either: ** exception exit: badarg in function gen_tcp:connect/4 in call from ssl:do_new_connect/4 in call from ssl_test:test/0 I need to get my code running on at least R13B03. The code is: -module(ssl_test). -compile(export_all). test() -> Body = iolist_to_binary([ "GET / HTTP/1.1\r\n", "Host: foo.bar.com\r\n", "Accept: */*\r\n", "Connection: close\r\n", "\r\n" ]), application:start(crypto), application:start(public_key), application:start(ssl), Options = [ {ssl_imp, new}, binary, {nodelay, true}, {active, false}, {verify, 0} % {reuse_sessions, false}, % {ciphers, ssl:cipher_suites(erlang)} ], {ok, S} = ssl:connect("foo.bar.com", 443, Options), ok = ssl:send(S, Body), loop(S), ssl:close(S). loop(S) -> ssl:setopts(S, [{active, once}]), receive {ssl, _, Data} -> io:format("received data: ~p~n", [Data]), loop(S); {ssl_closed, _} -> io:format("socket closed", []); {ssl_error, _, Error} -> io:format("socket error: ~p", [Error]) end. Am I missing something? cheers -- Filipe David Manana, fdmanana@REDACTED, fdmanana@REDACTED "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men." From rvirding@REDACTED Fri Sep 17 03:05:41 2010 From: rvirding@REDACTED (Robert Virding) Date: Fri, 17 Sep 2010 03:05:41 +0200 Subject: [erlang-questions] errors in gb_sets and gn_trees In-Reply-To: <4C929207.8080204@cs.ntua.gr> References: <4C929207.8080204@cs.ntua.gr> Message-ID: It should be erlang:error/1 as it is. Erlang:error/1 is intended for signaling errors, in this case a {key_exists,Key} error, while erlang:throw/1 was intended for throwing non-local returns not throwing errors. Throws and errors are handled differently in catch/try. They also result in different exit reasons if not caught at all, for erlang:error the exit value is the actual error reason together with the call stack, while for erlang:throw it is a 'nocatch' error. Robert 2010/9/16 Kostis Sagonas : > Modules gb_sets and gb_trees each define an insert/2 function whose > documentation (for gb_sets) reads: > > ?insert(Element, Set1) -> Set2 > ? ? Types: > ? ? ? Element = term() > ? ? ? Set1 = Set2 = gb_set() > ?Returns a new gb_set formed from Set1 with Element inserted. Assumes > ?that Element is not present in Set1. > > and whose code reads: > > ?insert(Key, Val, {S, T}) when is_integer(S) -> > ? ?S1 = S+1, > ? ?{S1, insert_1(Key, Val, T, ?pow(S1, ?p))}. > > ?insert_1(Key, Value, {Key1, V, Smaller, Bigger}, S) when Key < Key1 -> > ? ?case insert_1(Key, Value, Smaller, ?div2(S)) of > ?.... > ?insert_1(Key, _, _, _) -> > ? ?erlang:error({key_exists, Key}). > > Should the last line be a call to erlang:error/1 or to erlang:throw/1 ? > > Kostis > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From rzezeski@REDACTED Fri Sep 17 06:14:48 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Fri, 17 Sep 2010 00:14:48 -0400 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: References: Message-ID: On Mon, Sep 6, 2010 at 6:31 AM, Max Lapshin wrote: > > Yes, of course. > Do not forget about difference between f() and ?MODULE:f() > > Max, as a newb to Erlang, are you saying that without the module qualification and if you imported a function f/0 from another module it would instead call that? Actually, I just went ahead and tried that and the compiler gave me an error. So I ask, what are you referring to? -Ryan From max.lapshin@REDACTED Fri Sep 17 06:24:11 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 17 Sep 2010 08:24:11 +0400 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 8:14 AM, Ryan Zezeski wrote: > Max, as a newb to Erlang, are you saying that without the module > qualification and if you imported a function f/0 from another module it > would instead call that? ?Actually, I just went ahead and tried that and the > compiler gave me an error. ?So I ask, what are you referring to? > -Ryan > As for me, I was very inattentive reading erlang tutorial and it was a surprise for me, when following code didn't upgraded after code upgrade: f() -> io:format("Hi~n"), timer:sleep(500), f(). It is very important to use explicit module specification: f() -> io:format("Hi~n"), timer:sleep(500), ?MODULE:f(). ?MODULE is a compile time constant, referring to current module and this form of call will change code on runtime module update. From alexey.v.romanov@REDACTED Fri Sep 17 06:28:37 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Fri, 17 Sep 2010 08:28:37 +0400 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 8:14 AM, Ryan Zezeski wrote: > On Mon, Sep 6, 2010 at 6:31 AM, Max Lapshin wrote: > >> >> Yes, of course. >> Do not forget about difference between f() and ?MODULE:f() >> >> > Max, as a newb to Erlang, are you saying that without the module > qualification and if you imported a function f/0 from another module it > would instead call that? ?Actually, I just went ahead and tried that and the > compiler gave me an error. ?So I ask, what are you referring to? The difference is that if you change the code of your module while it's running, ?MODULE:f() will always call the function in the new version of the module, while f() will call the old version. See http://spawnlink.com/articles/rules-of-hot-code-swapping/ From boris.muehmer@REDACTED Fri Sep 17 06:36:03 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Fri, 17 Sep 2010 06:36:03 +0200 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: References: Message-ID: 2010/9/17 Ryan Zezeski : > On Mon, Sep 6, 2010 at 6:31 AM, Max Lapshin wrote: >> Do not forget about difference between f() and ?MODULE:f() > would instead call that? ?Actually, I just went ahead and tried that and the > compiler gave me an error. ?So I ask, what are you referring to? There is an "->" missing after the "after 1000", did You add it? Is "f/0" exported? The call to "?MODULE:f()" will fail, if "f/0" isn't in the export list. - boris From lemenkov@REDACTED Fri Sep 17 06:37:12 2010 From: lemenkov@REDACTED (Peter Lemenkov) Date: Fri, 17 Sep 2010 08:37:12 +0400 Subject: [erlang-questions] new_ssl in R13B03 In-Reply-To: References: Message-ID: Hello! 2010/9/17 Filipe David Manana : > Hello, > > I'm trying to use the new_ssl module in R13B03. There was something weird in ssl module prior to R13B04 (at least from R12B5) - it seems that it ignored 'noverify' option entirely throwing meaningless errors like yours. Perhaps there is something else. In my case (ignoring noveryfy) I ended up explicitly adding valid root certificates. See for example my patch to mochiweb for R12B5: http://github.com/lemenkov/mochiweb/commit/be2bc2ad -- With best regards, Peter Lemenkov. From bengt.kleberg@REDACTED Fri Sep 17 07:38:01 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 17 Sep 2010 07:38:01 +0200 Subject: [erlang-questions] R14B gen_tcp:recv/2,3 In-Reply-To: References: Message-ID: <1284701881.4807.3.camel@seasc1137> Greetings, If (I have not read the document) gen_tcp:recv/2,3 returns {ok, {http_error,String}} your have a matching clause. Unfortunately the match means that Bin will be {http_error,String}, which is probably not what you want. bengt On Fri, 2010-09-17 at 01:08 +0200, Roberto Ostinelli wrote: > dear list, > > i read from R14B release's readme: > http://erlang.org/download/otp_src_R14B.readme > > OTP-8831 For a socket in the HTTP packet mode, the return value from > gen_tcp:recv/2,3 if there is an error in the header will be > {ok,{http_error,String}} instead of {error,{http_error,String}} to be > consistent with ssl:recv/2,3. > > i'm not really sure i did understand well: does this mean that all > code which relies on gen_tcp:recv/2,3 needs to be rewritten? for > instance, will this code generate a no matching clause in R14B in case > an error is produced? > > case gen_tcp:recv(Sock, Len, RecvTimeout) of > {ok, Bin} -> {ok, Bin}; > {error, Reason} -> {error, Reason} > end. > > i, for one, surely hope i've misinterpreted this statement.. > > r. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From mabrek@REDACTED Fri Sep 17 07:09:59 2010 From: mabrek@REDACTED (mabrek) Date: Fri, 17 Sep 2010 09:09:59 +0400 Subject: how to get stacktraces of all processes? Message-ID: <4C92F827.6000203@gmail.com> Hello. I've got application that hangs on startup. I tried several alternatives. Appmon doesn't show unstarted applications. Pman shows only start function and current function for process, but I want full stacktraces. i() is similar to pman. erlang:process_display gives quite obscure output, process_info(.., [backtrace]) is the same. Is there a way to get full stacktraces of all processes in VM? Best regards, Anton Lebedevich. From alexey.v.romanov@REDACTED Fri Sep 17 08:40:59 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Fri, 17 Sep 2010 10:40:59 +0400 Subject: Can't build R14B from source Message-ID: When building R14B, I've got the following error: === Entering application kernel make[3]: Entering directory `/home/aromanov/otp_src/otp_src_R14B/lib/kernel/src' make -w RELEASE_PATH=/home/aromanov/otp_arm_compiled release_spec make[4]: Entering directory `/home/aromanov/otp_src/otp_src_R14B/lib/kernel/src' erlc -W +debug_info -I../include -o../ebin application.erl erlc -W +debug_info -I../include -o../ebin application_controller.erl erlc -W +debug_info -I../include -o../ebin application_master.erl erlc -W +debug_info -I../include -o../ebin application_starter.erl erlc -W +debug_info -I../include -o../ebin auth.erl erlc -W +debug_info -I../include -o../ebin code.erl erlc -W +debug_info -I../include -o../ebin code_server.erl erlc -W +debug_info -I../include -o../ebin disk_log.erl erlc -W +debug_info -I../include -o../ebin disk_log_1.erl erlc -W +debug_info -I../include -o../ebin disk_log_server.erl erlc -W +debug_info -I../include -o../ebin disk_log_sup.erl erlc -W +debug_info -I../include -o../ebin dist_ac.erl erlc -W +debug_info -I../include -o../ebin dist_util.erl erlc -W +debug_info -I../include -o../ebin erl_boot_server.erl erlc -W +debug_info -I../include -o../ebin erl_ddll.erl erlc -W +debug_info -I../include -o../ebin erl_distribution.erl erlc -W +debug_info -I../include -Depmd_port_no=4369 -Depmd_node_type=110 -Depmd_dist_low=5 -Depmd_dist_high=5 -Derlang_daemon_port=4369 -o../ebin erl_epmd.erl erlc -W +debug_info -I../include -o../ebin erl_reply.erl erlc -W +debug_info -I../include -o../ebin erts_debug.erl erlc -W +debug_info -I../include -o../ebin error_handler.erl erlc -W +debug_info -I../include -o../ebin error_logger.erl erlc -W +debug_info -I../include -o../ebin file.erl /home/aromanov/otp_src/otp_src_R14B/lib/kernel/src/../include/file.hrl:28: type date_time() undefined /home/aromanov/otp_src/otp_src_R14B/lib/kernel/src/../include/file.hrl:30: type date_time() undefined /home/aromanov/otp_src/otp_src_R14B/lib/kernel/src/../include/file.hrl:31: type date_time() undefined Comparison with R13B showed types date(), time(), and date_time() used to be defined in file.hrl and are now defined in file.erl, and moving the definition back allowed everything to compile. Did anybody else run into this error? Presumably if many people were, it would have been fixed long ago, but I can't see why it would only happen for me. Yours, Alexey Romanov From roberto@REDACTED Fri Sep 17 09:29:10 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 17 Sep 2010 09:29:10 +0200 Subject: [erlang-questions] R14B gen_tcp:recv/2,3 In-Reply-To: <1284701881.4807.3.camel@seasc1137> References: <1284701881.4807.3.camel@seasc1137> Message-ID: > Greetings, > > If (I have not read the document) gen_tcp:recv/2,3 returns {ok, > {http_error,String}} your have a matching clause. Unfortunately the > match means that Bin will be {http_error,String}, which is probably not > what you want. > hi bengt, my code was just making a statement and is indeed incorrect. i do have quite some code which relies on {packet, http} and passive mode, so i believe i might be confronted with adding code which handles running on different versions of the erlang VM. *if* i'm not misunderstanding, this does break backwards compatibility and i'm quite surprised, to the point i'm asking this list for confirmation. r. From bengt.kleberg@REDACTED Fri Sep 17 09:51:41 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 17 Sep 2010 09:51:41 +0200 Subject: [erlang-questions] R14B gen_tcp:recv/2,3 In-Reply-To: References: <1284701881.4807.3.camel@seasc1137> Message-ID: <1284709901.4807.24.camel@seasc1137> I read the README and it does say that backwards compatibility is broken for this case. That is very unusual and unfortunate for you, and others, using this. bengt On Fri, 2010-09-17 at 09:29 +0200, Roberto Ostinelli wrote: > > Greetings, > > > > If (I have not read the document) gen_tcp:recv/2,3 returns {ok, > > {http_error,String}} your have a matching clause. Unfortunately the > > match means that Bin will be {http_error,String}, which is probably not > > what you want. > > > > hi bengt, > > my code was just making a statement and is indeed incorrect. > > i do have quite some code which relies on {packet, http} and passive > mode, so i believe i might be confronted with adding code which > handles running on different versions of the erlang VM. *if* i'm > not misunderstanding, this does break backwards compatibility and i'm > quite surprised, to the point i'm asking this list for confirmation. > > r. From metalzong@REDACTED Fri Sep 17 09:51:50 2010 From: metalzong@REDACTED (metal zong) Date: Fri, 17 Sep 2010 15:51:50 +0800 Subject: problem on timeout setting in eunit tests Message-ID: Hello, Seems timeout mechanism of eunit can not work properly with test set which contains more than one test and also can not work with fixture properly. Please see following code for more information. -module(test). -export([run/0]). -include_lib("eunit/include/eunit.hrl"). setup() -> ?debugMsg("setup ..."), ok. teardown(S) -> ?debugFmt("teardown ... ~p~n", [S]), ok. test() -> timer:sleep(1000), ?debugMsg("test ..."), ok. test0() -> timer:sleep(6000), ?debugMsg("test0 ..."), ok. test1(P) -> timer:sleep(1000), ?debugFmt("test1 ... ~p~n", [P]), ok. test2(P) -> timer:sleep(6000), ?debugFmt("test2 ... ~p~n", [P]), ok. run() -> %% {timeout, 60, [fun test0/0]}. %% ok %% {timeout, 60, [fun test/0, fun test0/0]}. %% timeout %% {timeout, 60, {with, setup(), [fun test2/1]}}. %% ok %% {timeout, 60, {with, setup(), [fun test1/1, fun test2/1]}}. %% timeout %% {timeout, 60, {setup, fun setup/0, fun teardown/1, {with, [fun test2/1]}}}. %% timeout %% {timeout, 60, {setup, fun setup/0, fun teardown/1, {with, [fun test1/1, fun test2/1]}}}. %%timeout From bengt.kleberg@REDACTED Fri Sep 17 09:57:39 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 17 Sep 2010 09:57:39 +0200 Subject: [erlang-questions] R14B gen_tcp:recv/2,3 In-Reply-To: References: <1284701881.4807.3.camel@seasc1137> Message-ID: <1284710259.4807.28.camel@seasc1137> One more thing. I hope you are not contemplating having different versions of your code for different versions of the erlang VM. If you code like this: case gen_tcp:recv(Sock, Len, RecvTimeout) of {ok, {http_error,String}} -> {error, Reason} ; {ok, Bin} -> {ok, Bin} ; {error, Reason} -> {error, Reason} end. it will work on both versions of the erlang VM. bengt On Fri, 2010-09-17 at 09:29 +0200, Roberto Ostinelli wrote: > > Greetings, > > > > If (I have not read the document) gen_tcp:recv/2,3 returns {ok, > > {http_error,String}} your have a matching clause. Unfortunately the > > match means that Bin will be {http_error,String}, which is probably not > > what you want. > > > > hi bengt, > > my code was just making a statement and is indeed incorrect. > > i do have quite some code which relies on {packet, http} and passive > mode, so i believe i might be confronted with adding code which > handles running on different versions of the erlang VM. *if* i'm > not misunderstanding, this does break backwards compatibility and i'm > quite surprised, to the point i'm asking this list for confirmation. > > r. From metalzong@REDACTED Fri Sep 17 09:57:31 2010 From: metalzong@REDACTED (metal zong) Date: Fri, 17 Sep 2010 15:57:31 +0800 Subject: Fwd: problem on timeout setting in eunit tests In-Reply-To: References: Message-ID: erlang: erlang-r13b04 eunit: eunit-2.1.5 and latest code in svn ---------- Forwarded message ---------- From: metal zong Date: Fri, Sep 17, 2010 at 3:51 PM Subject: problem on timeout setting in eunit tests To: erlang-questions@REDACTED Hello, Seems timeout mechanism of eunit can not work properly with test set which contains more than one test and also can not work with fixture properly. Please see following code for more information. -module(test). -export([run/0]). -include_lib("eunit/include/eunit.hrl"). setup() -> ?debugMsg("setup ..."), ok. teardown(S) -> ?debugFmt("teardown ... ~p~n", [S]), ok. test() -> timer:sleep(1000), ?debugMsg("test ..."), ok. test0() -> timer:sleep(6000), ?debugMsg("test0 ..."), ok. test1(P) -> timer:sleep(1000), ?debugFmt("test1 ... ~p~n", [P]), ok. test2(P) -> timer:sleep(6000), ?debugFmt("test2 ... ~p~n", [P]), ok. run() -> %% {timeout, 60, [fun test0/0]}. %% ok %% {timeout, 60, [fun test/0, fun test0/0]}. %% timeout %% {timeout, 60, {with, setup(), [fun test2/1]}}. %% ok %% {timeout, 60, {with, setup(), [fun test1/1, fun test2/1]}}. %% timeout %% {timeout, 60, {setup, fun setup/0, fun teardown/1, {with, [fun test2/1]}}}. %% timeout %% {timeout, 60, {setup, fun setup/0, fun teardown/1, {with, [fun test1/1, fun test2/1]}}}. %%timeout From attila.r.nohl@REDACTED Fri Sep 17 11:29:46 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Fri, 17 Sep 2010 11:29:46 +0200 Subject: [erlang-questions] how to get stacktraces of all processes? In-Reply-To: <4C92F827.6000203@gmail.com> References: <4C92F827.6000203@gmail.com> Message-ID: 2010/9/17, mabrek : > Hello. > > I've got application that hangs on startup. > I tried several alternatives. Appmon doesn't show unstarted applications. > Pman shows only start function and current function for process, but I want > full stacktraces. i() is similar to pman. erlang:process_display gives quite > obscure output, process_info(.., [backtrace]) is the same. > Is there a way to get full stacktraces of all processes in VM? I don't know any way, but the stacktrace might not be that useful because tail recursive calls are missing from the trace. If I were you, I'd first check for processes which have non-empty message queues (that process probably got stuck somewhere and doesn't get to process the message) or which have high number and quickly increasing reductions (this indicates an infinite loop). Then I'd check the current function of the suspicious process and try to find out what happened. You can get this information from the process_info. You might also turn on tracing early on in the application startup using dbg, if you have an idea where things go wrong. From ingela@REDACTED Fri Sep 17 11:54:52 2010 From: ingela@REDACTED (Ingela Andin) Date: Fri, 17 Sep 2010 11:54:52 +0200 Subject: [erlang-questions] R14B gen_tcp:recv/2,3 In-Reply-To: <1284710259.4807.28.camel@seasc1137> References: <1284701881.4807.3.camel@seasc1137> <1284710259.4807.28.camel@seasc1137> Message-ID: Hi! This change does break backwards compatibility. The reason we changed it was that we want gen_tcp and ssl to behave the same, if possible. Ssl did behave according to the documentation and in the intended way, so the gen_tcp behavior is considered a bug. Either way to make them behave the same we had to break backwards compatibility for either gen_tcp or ssl. Also http-packet mode has not been supported very long, actually we made it supported for the sake of ssl so that new ssl could support http-packet mode and not break applications using it. As for supporting more than one OTP-version, it should be no problem having a error clause that will be dead code in R14 and do something in previous versions. The content of the http_error tuple should not be any different from before. Regards Ingela Erlang/OTP team - Ericsson AB 2010/9/17 Bengt Kleberg : > One more thing. I hope you are not contemplating having different > versions of your code for different versions of the erlang VM. > > If you code like this: > > case gen_tcp:recv(Sock, Len, RecvTimeout) > of {ok, {http_error,String}} -> {error, Reason} > ; {ok, Bin} -> {ok, Bin} > ; {error, Reason} -> {error, Reason} > end. > > it will work on both versions of the erlang VM. > > > bengt > > On Fri, 2010-09-17 at 09:29 +0200, Roberto Ostinelli wrote: >> > Greetings, >> > >> > If (I have not read the document) gen_tcp:recv/2,3 returns {ok, >> > {http_error,String}} your have a matching clause. Unfortunately the >> > match means that Bin will be {http_error,String}, which is probably not >> > what you want. >> > >> >> hi bengt, >> >> my code was just making a statement and is indeed incorrect. >> >> i do have quite some code which relies on {packet, http} and passive >> mode, so i believe i might be confronted with adding code which >> handles running on different versions of the erlang VM. *if* i'm >> not misunderstanding, this does break backwards compatibility and i'm >> quite surprised, to the point i'm asking this list for confirmation. >> >> r. > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From robert.virding@REDACTED Fri Sep 17 12:25:13 2010 From: robert.virding@REDACTED (Robert Virding) Date: Fri, 17 Sep 2010 10:25:13 +0000 (GMT) Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: <135744570.4331284719040369.JavaMail.root@zimbra> Message-ID: <373251123.4471284719113411.JavaMail.root@zimbra> Most of this has been said before here but to bring it all together and try to clarify it: - A call to a function without a qualifying module name, i.e. foo(...), is ALWAYS a call to a local function in the same module and the same version of that module. Always. (though see importing below) - A call to a function which is qualified with a module name, i.e. bar:foo(...), is ALWAYS a call to an exported function in the latest version of that module. Always, even if the module is the same as the one in which the call is made. This is the only way to call a function in another module. - There is nothing at all special about the predefined macro MODULE, nothing. So in the module 'bar' ?MODULE:foo(...) and bar:foo(...) are exactly the same. The only benefits may be that it is clearer that you are actually calling the same module, and it may save you some work if you change the module name. - Importing a function from a module provides absolutely no new functionality. It just saves you from having to explicitly qualify the function call with the module name. So if foo/1 is imported from module bar, -import(bar,[foo/1])., then foo(...) and bar:foo(...) are exactly equivalent, the compiler does the conversion internally. It saves you some writing. Many people frown on importing and say you shouldn't do it as it obscures what you are actually doing, I only use for importing functions from some common modules like lists. I think that's about it. Robert ----- Ursprungligt meddelande ----- Fr?n: "Ryan Zezeski" Till: "Max Lapshin" Kopia: "Zvi" , erlang-questions@REDACTED Skickat: fredag, 17 sep 2010 6:14:48 GMT +01:00 Amsterdam/Berlin/Bern/Rom/Stockholm/Wien ?mne: Re: [erlang-questions] Is this code tail recursive? On Mon, Sep 6, 2010 at 6:31 AM, Max Lapshin wrote: > > Yes, of course. > Do not forget about difference between f() and ?MODULE:f() > > Max, as a newb to Erlang, are you saying that without the module qualification and if you imported a function f/0 from another module it would instead call that? Actually, I just went ahead and tried that and the compiler gave me an error. So I ask, what are you referring to? -Ryan From gianfranco.alongi@REDACTED Fri Sep 17 12:28:21 2010 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Fri, 17 Sep 2010 10:28:21 +0000 (GMT) Subject: [erlang-questions] how to get stacktraces of all processes? In-Reply-To: <4C92F827.6000203@gmail.com> Message-ID: <270532205.4731284719301686.JavaMail.root@zimbra> I don't know if you have already tried, but you can try 1) Turning on sasl, 2) Using dbg to trace all calls made by your application. Cheers /G ----- Original Message ----- From: "mabrek" To: erlang-questions@REDACTED Sent: Friday, September 17, 2010 6:09:59 AM GMT +00:00 GMT Britain, Ireland, Portugal Subject: [erlang-questions] how to get stacktraces of all processes? Hello. I've got application that hangs on startup. I tried several alternatives. Appmon doesn't show unstarted applications. Pman shows only start function and current function for process, but I want full stacktraces. i() is similar to pman. erlang:process_display gives quite obscure output, process_info(.., [backtrace]) is the same. Is there a way to get full stacktraces of all processes in VM? Best regards, Anton Lebedevich. ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From ingela.andin@REDACTED Fri Sep 17 12:31:29 2010 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 17 Sep 2010 12:31:29 +0200 Subject: [erlang-questions] new_ssl in R13B03 In-Reply-To: References: Message-ID: Hi! New ssl is considerd exprimental until R14. In R14 there will be an certificate expired error for this particular case. There is now a discussion related to this on the erlang-bug list. Regards Ingela Erlang/OTP team - Ericsson AB 2010/9/17 Peter Lemenkov : > Hello! > > 2010/9/17 Filipe David Manana : >> Hello, >> >> I'm trying to use the new_ssl module in R13B03. > > There was something weird in ssl module prior to R13B04 (at least from > R12B5) - it seems that it ignored 'noverify' option entirely throwing > meaningless errors like yours. Perhaps there is something else. > > In my case (ignoring noveryfy) I ended up explicitly adding valid root > certificates. See for example my patch to mochiweb for R12B5: > > http://github.com/lemenkov/mochiweb/commit/be2bc2ad > > -- > With best regards, Peter Lemenkov. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From dmitrii@REDACTED Fri Sep 17 12:50:01 2010 From: dmitrii@REDACTED (Dmitrii Dimandt) Date: Fri, 17 Sep 2010 13:50:01 +0300 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: <373251123.4471284719113411.JavaMail.root@zimbra> References: <373251123.4471284719113411.JavaMail.root@zimbra> Message-ID: <24E9AEF6-FE7C-482B-80FE-76FC49E4CF51@gmail.com> I think this is worthy of being placed in the FAQ. > Most of this has been said before here but to bring it all together and try to clarify it: > > - A call to a function without a qualifying module name, i.e. foo(...), is ALWAYS a call to a local function in the same module and the same version of that module. Always. (though see importing below) > > - A call to a function which is qualified with a module name, i.e. bar:foo(...), is ALWAYS a call to an exported function in the latest version of that module. Always, even if the module is the same as the one in which the call is made. This is the only way to call a function in another module. > > - There is nothing at all special about the predefined macro MODULE, nothing. So in the module 'bar' ?MODULE:foo(...) and bar:foo(...) are exactly the same. The only benefits may be that it is clearer that you are actually calling the same module, and it may save you some work if you change the module name. > > - Importing a function from a module provides absolutely no new functionality. It just saves you from having to explicitly qualify the function call with the module name. So if foo/1 is imported from module bar, -import(bar,[foo/1])., then foo(...) and bar:foo(...) are exactly equivalent, the compiler does the conversion internally. It saves you some writing. Many people frown on importing and say you shouldn't do it as it obscures what you are actually doing, I only use for importing functions from some common modules like lists. > > I think that's about it. > > Robert > > ----- Ursprungligt meddelande ----- > Fr?n: "Ryan Zezeski" > Till: "Max Lapshin" > Kopia: "Zvi" , erlang-questions@REDACTED > Skickat: fredag, 17 sep 2010 6:14:48 GMT +01:00 Amsterdam/Berlin/Bern/Rom/Stockholm/Wien > ?mne: Re: [erlang-questions] Is this code tail recursive? > > On Mon, Sep 6, 2010 at 6:31 AM, Max Lapshin wrote: > >> >> Yes, of course. >> Do not forget about difference between f() and ?MODULE:f() >> >> > Max, as a newb to Erlang, are you saying that without the module > qualification and if you imported a function f/0 from another module it > would instead call that? Actually, I just went ahead and tried that and the > compiler gave me an error. So I ask, what are you referring to? > > -Ryan > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From bengt.kleberg@REDACTED Fri Sep 17 14:46:29 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 17 Sep 2010 14:46:29 +0200 Subject: [erlang-questions] R14B gen_tcp:recv/2,3 In-Reply-To: References: <1284701881.4807.3.camel@seasc1137> <1284710259.4807.28.camel@seasc1137> Message-ID: <1284727589.26053.1.camel@seasc1137> I like that Erlang/OTP is working towards consistency. bengt On Fri, 2010-09-17 at 11:54 +0200, Ingela Andin wrote: > Hi! > > This change does break backwards compatibility. The reason we changed > it was that we want gen_tcp and ssl to behave the same, if possible. > Ssl did behave according to the documentation and in the intended way, > so the gen_tcp behavior is considered a bug. > Either way to make them behave the same we had to break backwards compatibility > for either gen_tcp or ssl. Also http-packet mode has not been > supported very long, > actually we made it supported for the sake of ssl so that new ssl could > support http-packet mode and not break applications using it. > As for supporting more than one OTP-version, it should be no problem having a > error clause that will be dead code in R14 and do something in > previous versions. > The content of the http_error tuple should not be any different from before. > > Regards Ingela Erlang/OTP team - Ericsson AB > > > 2010/9/17 Bengt Kleberg : > > One more thing. I hope you are not contemplating having different > > versions of your code for different versions of the erlang VM. > > > > If you code like this: > > > > case gen_tcp:recv(Sock, Len, RecvTimeout) > > of {ok, {http_error,String}} -> {error, Reason} > > ; {ok, Bin} -> {ok, Bin} > > ; {error, Reason} -> {error, Reason} > > end. > > > > it will work on both versions of the erlang VM. > > > > > > bengt > > > > On Fri, 2010-09-17 at 09:29 +0200, Roberto Ostinelli wrote: > >> > Greetings, > >> > > >> > If (I have not read the document) gen_tcp:recv/2,3 returns {ok, > >> > {http_error,String}} your have a matching clause. Unfortunately the > >> > match means that Bin will be {http_error,String}, which is probably not > >> > what you want. > >> > > >> > >> hi bengt, > >> > >> my code was just making a statement and is indeed incorrect. > >> > >> i do have quite some code which relies on {packet, http} and passive > >> mode, so i believe i might be confronted with adding code which > >> handles running on different versions of the erlang VM. *if* i'm > >> not misunderstanding, this does break backwards compatibility and i'm > >> quite surprised, to the point i'm asking this list for confirmation. > >> > >> r. > > > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > From joelr1@REDACTED Fri Sep 17 15:23:38 2010 From: joelr1@REDACTED (Joel Reymont) Date: Fri, 17 Sep 2010 14:23:38 +0100 Subject: profiling network apps: fprof vs sequential tracing Message-ID: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> Suppose I'm trying to profile a web app running on top of mochiweb. The app starts processes for every request coming in and these talk to other processes that use the disk, etc. There are a few hundred requests in a single benchmarking run. I'm measuring disk write performance and see that it slows down to a crawl sometimes. Has anyone used fprof or sequential tracing to identify bottlenecks in this or similar scenario? The problem with fprof is that it aggregates by process. This means I have several hundred profiling sets, one per request, in a 79mb text file. It's very hard to find a bottleneck in such a huge file, specially when it's split per process. I think I'll need to further massage this file to establish a path a new request goes through, from the time it hits the server to the time output is sent. I could then analyze several hundred paths like this and look for statistical outliers. I'm wondering if it's better to use sequential tracing in this scenario. Creating a new token when the request hits the app should automatically establish the path I'm looking for and allow me to time each segment. Am I on the right track? Has anyone dealt with a similar situation? Thanks, Joel --- http://twitter.com/wagerlabs From joelr1@REDACTED Fri Sep 17 15:29:56 2010 From: joelr1@REDACTED (Joel Reymont) Date: Fri, 17 Sep 2010 14:29:56 +0100 Subject: graph database for trace analysis Message-ID: What do you folks think of using a graph database for trace analysis? Each trace converts into several graph nodes with attributes like source process, destination process, timestamp, etc. The benefit I see is being able to construct queries that time requests as they go through the system. Each request consists of several messages sent between different processes and results in a response. It seems near impossible to do this kind of analysis using available profiling tools. Interactive querying and analysis is definitely impossible at the moment. --- http://twitter.com/wagerlabs From egil@REDACTED Fri Sep 17 17:01:43 2010 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Fri, 17 Sep 2010 17:01:43 +0200 Subject: [erlang-questions] profiling network apps: fprof vs sequential tracing In-Reply-To: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> References: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> Message-ID: <4C9382D7.3010601@erix.ericsson.se> On 2010-09-17 15:23, Joel Reymont wrote: > Suppose I'm trying to profile a web app running on top of mochiweb. > > The app starts processes for every request coming in and these talk to other processes that use the disk, etc. There are a few hundred requests in a single benchmarking run. > > I'm measuring disk write performance and see that it slows down to a crawl sometimes. > > Has anyone used fprof or sequential tracing to identify bottlenecks in this or similar scenario? > > The problem with fprof is that it aggregates by process. This means I have several hundred profiling sets, one per request, in a 79mb text file. It's very hard to find a bottleneck in such a huge file, specially when it's split per process. I'm not a 100% sure on what you are after but there are a couple of alternatives to fprof for finding bottlenecks. You can use eprof to find processes + code that produces bottlenecks. It is somewhat limited compared to fprof (no callgraph for instance) but is is much much faster (especially after the rewrite to r14) and will likely identify problematic intersections of processes and code. For parallelism issues you can use lcnt. You need to recompile your erlang binary with lcnt enabled (not for production use). lcnt will pinpoint locks with long acquisition time and high collision rates, which is a signal of high contention for specific memory areas. Regards, Bj?rn-Egil Erlang/OTP From olivier.boudeville@REDACTED Fri Sep 17 17:39:23 2010 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Fri, 17 Sep 2010 17:39:23 +0200 Subject: Asynchronous error_logger:error_msg considered harmful? Message-ID: Hello, When displaying a message with error_logger:error_msg/2, apparently this is done asynchronously, the call returns before the message is actually displayed in the console. As a result, if in the next few instructions the VM is halted (these two events must be often correlated), there is a race condition between the displaying and the halting: sometimes the message shows up, sometimes not. Short of knowing a better solution, this was one of the very few places where a timer:sleep/1 call helped me, despite this is never a satisfying solution. And indeed it led to hiding a second race condition, this time in my code, which was difficult to track because of this awkward delay. Apparently there could be a better solution based on the use of sys:get_status(error_logger) which could allow to wait just the duration needed until the operation is done, thus allowing somewhat to emulate a synchronous error_logger:error_msg/2 (I have not figured out yet out to use it properly). I was wanting to advertise a bit this solution as it seems not very well-known. Besides, for error messages, wouldn't it be safer to be synchronous at the first place? Best regards, Olivier Boudeville. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. From mononcqc@REDACTED Fri Sep 17 20:25:21 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 17 Sep 2010 14:25:21 -0400 Subject: [erlang-questions] profiling network apps: fprof vs sequential tracing In-Reply-To: <4C9382D7.3010601@erix.ericsson.se> References: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> <4C9382D7.3010601@erix.ericsson.se> Message-ID: 2010/9/17 Bj?rn-Egil Dahlberg > > You can use eprof to find processes + code that produces bottlenecks. It is > somewhat limited compared to fprof (no callgraph for instance) but is is > much much faster (especially after the rewrite to r14) and will likely > identify problematic intersections of processes and code. > > > Regards, > Bj?rn-Egil > Erlang/OTP I'm not sure when the rewrite happened, but I just tested this on R14A (confirmed by joelr on R14B) and it seems like there is a bug with eprof:analyse/2. Whenever there is no option passed, the sort defaults to {sort, time}. However, whenever an option is passed (either {sort, calls} or {sort, time}), the output is sorted by the number of calls. From mark@REDACTED Fri Sep 17 20:43:08 2010 From: mark@REDACTED (Mark Scandariato) Date: Fri, 17 Sep 2010 14:43:08 -0400 Subject: R14B Illegal instruction? Message-ID: I just built R14B for Ubuntu 9.04 (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4)) and when I try to enable some async threads, I get an "Illegal instruction" fault. Here's a stack trace: erlang@REDACTED:~/otp_src_R14B$ bin/cerl -debug +A 4 Illegal instruction (core dumped) erlang@REDACTED:~/otp_src_R14B$ gdb --core=core GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". (no debugging symbols found) Core was generated by `/home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug -A 4 -- -root /h'. Program terminated with signal 4, Illegal instruction. [New process 28563] [New process 28587] #0 0x0823d51a in ?? () (gdb) file /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug Reading symbols from /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug...done. (gdb) where #0 ethr_event_reset__ (e=0xb769c030) at ../include/internal/pthread/ethr_event.h:94 #1 0x0823d723 in ethr_event_reset (e=0xb769c030) at pthread/ethr_event.c:200 #2 0x0823cf1c in ethr_thr_create (tid=0xb7580ca4, func=0x81515c6 , arg=0xb7580c48, opts=0xbfb85348) at pthread/ethread.c:256 #3 0x080cb160 in erts_thr_create (tid=0xb7580ca4, func=0x81515c6 , arg=0xb7580c48, opts=0xbfb85348) at beam/erl_threads.h:348 #4 0x081511bb in init_async (hndl=6) at beam/erl_async.c:143 #5 0x081c62ab in async_drv_start (port_num=1, name=0x825f3fa "async", opts=0xbfb85400) at sys/unix/sys.c:2375 #6 0x080e48d5 in erts_open_driver (driver=0xb769dbb8, pid=4294967291, name=0x825f3fa "async", opts=0xbfb85400, error_number_ptr=0x0) at beam/io.c:641 #7 0x081c66a5 in sys_init_io () at sys/unix/sys.c:2528 #8 0x080e60d3 in init_io () at beam/io.c:1304 #9 0x0809b41b in erl_init () at beam/erl_init.c:281 #10 0x0809dd25 in erl_start (argc=15, argv=0xbfb85704) at beam/erl_init.c:1389 #11 0x0807f126 in main (argc=4001536, argv=0x0) at sys/unix/erl_main.c:29 I can send a core file if anyone wants it. From rickard@REDACTED Fri Sep 17 21:52:37 2010 From: rickard@REDACTED (Rickard Green) Date: Fri, 17 Sep 2010 21:52:37 +0200 Subject: R14B Illegal instruction? Message-ID: <4C93C705.7010206@erlang.org> Mark Scandariato wrote: > just built R14B for Ubuntu 9.04 (gcc version 4.3.3 (Ubuntu > 4.3.3-5ubuntu4)) and when I try to enable some async threads, I get an > "Illegal instruction" fault. > > Here's a stack trace: > > erlang@REDACTED:~/otp_src_R14B$ bin/cerl -debug +A 4 > Illegal instruction (core dumped) > erlang@REDACTED:~/otp_src_R14B$ gdb --core=core > GNU gdb 6.8-debian > Copyright (C) 2008 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "i486-linux-gnu". > (no debugging symbols found) > Core was generated by > `/home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug -A 4 -- -root > /h'. > Program terminated with signal 4, Illegal instruction. > [New process 28563] > [New process 28587] > #0 0x0823d51a in ?? () > (gdb) file /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug > Reading symbols from > /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug...done. > (gdb) where > #0 ethr_event_reset__ (e=0xb769c030) at > ../include/internal/pthread/ethr_event.h:94 > #1 0x0823d723 in ethr_event_reset (e=0xb769c030) at > pthread/ethr_event.c:200 > #2 0x0823cf1c in ethr_thr_create (tid=0xb7580ca4, func=0x81515c6 > , arg=0xb7580c48, opts=0xbfb85348) at pthread/ethread.c:256 > #3 0x080cb160 in erts_thr_create (tid=0xb7580ca4, func=0x81515c6 > , arg=0xb7580c48, opts=0xbfb85348) at beam/erl_threads.h:348 > #4 0x081511bb in init_async (hndl=6) at beam/erl_async.c:143 > #5 0x081c62ab in async_drv_start (port_num=1, name=0x825f3fa "async", > opts=0xbfb85400) at sys/unix/sys.c:2375 > #6 0x080e48d5 in erts_open_driver (driver=0xb769dbb8, pid=4294967291, > name=0x825f3fa "async", opts=0xbfb85400, error_number_ptr=0x0) > at beam/io.c:641 > #7 0x081c66a5 in sys_init_io () at sys/unix/sys.c:2528 > #8 0x080e60d3 in init_io () at beam/io.c:1304 > #9 0x0809b41b in erl_init () at beam/erl_init.c:281 > #10 0x0809dd25 in erl_start (argc=15, argv=0xbfb85704) at > beam/erl_init.c:1389 > #11 0x0807f126 in main (argc=4001536, argv=0x0) at sys/unix/erl_main.c:29 > > > I can send a core file if anyone wants it. I suspect that you have an old x86 processor, and probably need to pass the --enable-ethread-pre-pentium4-compatibility command line argument to configure. See . However, I just noted that this flag is buggy and have no effect :( I'll fix that in the next release. As a workaround you can pass the following as arguments to configure: --enable-ethread-pre-pentium4-compatibility enable_ethread_pre_pentium4_compatibilit=yes It is important that you pass both of them otherwise it wont work. You can verify that it worked by verifying that the following line exists in both $ERL_TOP/erts/include/internal/i686-pc-linux-gnu/ethread_header_config.h and $ERL_TOP/erts/i686-pc-linux-gnu/config.h: #define ETHR_PRE_PENTIUM4_COMPAT 1 In order to avoid trouble, don't reuse the source tree you have already built in. Build from scratch in a new one. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. From mikpe@REDACTED Fri Sep 17 22:08:57 2010 From: mikpe@REDACTED (Mikael Pettersson) Date: Fri, 17 Sep 2010 22:08:57 +0200 Subject: [erlang-questions] R14B Illegal instruction? In-Reply-To: References: Message-ID: <19603.51929.485076.491881@pilspetsen.it.uu.se> Mark Scandariato writes: > I just built R14B for Ubuntu 9.04 (gcc version 4.3.3 (Ubuntu > 4.3.3-5ubuntu4)) and when I try to enable some async threads, I get an > "Illegal instruction" fault. > > Here's a stack trace: > > erlang@REDACTED:~/otp_src_R14B$ bin/cerl -debug +A 4 > Illegal instruction (core dumped) > erlang@REDACTED:~/otp_src_R14B$ gdb --core=core > GNU gdb 6.8-debian > Copyright (C) 2008 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "i486-linux-gnu". > (no debugging symbols found) > Core was generated by > `/home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug -A 4 -- -root > /h'. > Program terminated with signal 4, Illegal instruction. > [New process 28563] > [New process 28587] > #0 0x0823d51a in ?? () > (gdb) file /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug > Reading symbols from > /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug...done. > (gdb) where > #0 ethr_event_reset__ (e=0xb769c030) at > ../include/internal/pthread/ethr_event.h:94 > #1 0x0823d723 in ethr_event_reset (e=0xb769c030) at > pthread/ethr_event.c:200 > #2 0x0823cf1c in ethr_thr_create (tid=0xb7580ca4, func=0x81515c6 > , arg=0xb7580c48, opts=0xbfb85348) at pthread/ethread.c:256 > #3 0x080cb160 in erts_thr_create (tid=0xb7580ca4, func=0x81515c6 > , arg=0xb7580c48, opts=0xbfb85348) at beam/erl_threads.h:348 > #4 0x081511bb in init_async (hndl=6) at beam/erl_async.c:143 > #5 0x081c62ab in async_drv_start (port_num=1, name=0x825f3fa "async", > opts=0xbfb85400) at sys/unix/sys.c:2375 > #6 0x080e48d5 in erts_open_driver (driver=0xb769dbb8, pid=4294967291, > name=0x825f3fa "async", opts=0xbfb85400, error_number_ptr=0x0) > at beam/io.c:641 > #7 0x081c66a5 in sys_init_io () at sys/unix/sys.c:2528 > #8 0x080e60d3 in init_io () at beam/io.c:1304 > #9 0x0809b41b in erl_init () at beam/erl_init.c:281 > #10 0x0809dd25 in erl_start (argc=15, argv=0xbfb85704) at > beam/erl_init.c:1389 > #11 0x0807f126 in main (argc=4001536, argv=0x0) at sys/unix/erl_main.c:29 > > > I can send a core file if anyone wants it. Show us (a) the contents of the registers, and (b) a disassembly of the code surrounding the PC that faulted. From mark@REDACTED Fri Sep 17 23:04:45 2010 From: mark@REDACTED (Mark Scandariato) Date: Fri, 17 Sep 2010 17:04:45 -0400 Subject: [erlang-questions] R14B Illegal instruction? In-Reply-To: <19603.51929.485076.491881@pilspetsen.it.uu.se> References: <19603.51929.485076.491881@pilspetsen.it.uu.se> Message-ID: (gdb) disassemble Dump of assembler code for function ethr_event_reset__: 0x0823d501 : push %ebp 0x0823d502 : mov %esp,%ebp 0x0823d504 : sub $0x8,%esp 0x0823d507 : mov 0x8(%ebp),%eax 0x0823d50a : movl $0x77777777,0x4(%esp) 0x0823d512 : mov %eax,(%esp) 0x0823d515 : call 0x8238488 0x0823d51a : mfence 0x0823d51d : leave 0x0823d51e : ret End of assembler dump. (gdb) info all-registers eax 0x77777777 2004318071 ecx 0xb78084b4 -1216314188 edx 0xb75ce030 -1218650064 ebx 0xb776fff4 -1216937996 esp 0xbfcb3010 0xbfcb3010 ebp 0xbfcb3018 0xbfcb3018 esi 0x82427e0 136587232 edi 0x807f050 134738000 eip 0x823d51a 0x823d51a eflags 0x10282 [ SF IF RF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51 st0 0 (raw 0x00000000000000000000) st1 0 (raw 0x00000000000000000000) st2 0 (raw 0x00000000000000000000) st3 0 (raw 0x00000000000000000000) st4 0 (raw 0x00000000000000000000) st5 0 (raw 0x00000000000000000000) st6 0 (raw 0x00000000000000000000) st7 0 (raw 0x00000000000000000000) fctrl 0x37f 895 fstat 0x0 0 ftag 0xffff 65535 fiseg 0x0 0 fioff 0x0 0 foseg 0x0 0 fooff 0x0 0 fop 0x0 0 xmm0 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm1 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm2 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm3 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm4 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm5 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm6 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm7 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 }, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} mxcsr 0x1f80 [ IM DM ZM OM UM PM ] mm0 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm1 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm2 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm3 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm4 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm5 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm6 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm7 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} On Fri, Sep 17, 2010 at 4:08 PM, Mikael Pettersson wrote: > Mark Scandariato writes: > > I just built R14B for Ubuntu 9.04 (gcc version 4.3.3 (Ubuntu > > 4.3.3-5ubuntu4)) and when I try to enable some async threads, I get an > > "Illegal instruction" fault. > > > > Here's a stack trace: > > > > erlang@REDACTED:~/otp_src_R14B$ bin/cerl -debug +A 4 > > Illegal instruction (core dumped) > > erlang@REDACTED:~/otp_src_R14B$ gdb --core=core > > GNU gdb 6.8-debian > > Copyright (C) 2008 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html > > > > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > > and "show warranty" for details. > > This GDB was configured as "i486-linux-gnu". > > (no debugging symbols found) > > Core was generated by > > `/home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug -A 4 -- > -root > > /h'. > > Program terminated with signal 4, Illegal instruction. > > [New process 28563] > > [New process 28587] > > #0 0x0823d51a in ?? () > > (gdb) file /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug > > Reading symbols from > > /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug...done. > > (gdb) where > > #0 ethr_event_reset__ (e=0xb769c030) at > > ../include/internal/pthread/ethr_event.h:94 > > #1 0x0823d723 in ethr_event_reset (e=0xb769c030) at > > pthread/ethr_event.c:200 > > #2 0x0823cf1c in ethr_thr_create (tid=0xb7580ca4, func=0x81515c6 > > , arg=0xb7580c48, opts=0xbfb85348) at pthread/ethread.c:256 > > #3 0x080cb160 in erts_thr_create (tid=0xb7580ca4, func=0x81515c6 > > , arg=0xb7580c48, opts=0xbfb85348) at beam/erl_threads.h:348 > > #4 0x081511bb in init_async (hndl=6) at beam/erl_async.c:143 > > #5 0x081c62ab in async_drv_start (port_num=1, name=0x825f3fa "async", > > opts=0xbfb85400) at sys/unix/sys.c:2375 > > #6 0x080e48d5 in erts_open_driver (driver=0xb769dbb8, pid=4294967291, > > name=0x825f3fa "async", opts=0xbfb85400, error_number_ptr=0x0) > > at beam/io.c:641 > > #7 0x081c66a5 in sys_init_io () at sys/unix/sys.c:2528 > > #8 0x080e60d3 in init_io () at beam/io.c:1304 > > #9 0x0809b41b in erl_init () at beam/erl_init.c:281 > > #10 0x0809dd25 in erl_start (argc=15, argv=0xbfb85704) at > > beam/erl_init.c:1389 > > #11 0x0807f126 in main (argc=4001536, argv=0x0) at > sys/unix/erl_main.c:29 > > > > > > I can send a core file if anyone wants it. > > Show us (a) the contents of the registers, and (b) a disassembly > of the code surrounding the PC that faulted. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From mark@REDACTED Fri Sep 17 23:06:43 2010 From: mark@REDACTED (Mark Scandariato) Date: Fri, 17 Sep 2010 17:06:43 -0400 Subject: [erlang-questions] Re: R14B Illegal instruction? In-Reply-To: <4C93C705.7010206@erlang.org> References: <4C93C705.7010206@erlang.org> Message-ID: It is a 2005-vintage AMD Athlon XP (in a VMWare virtual machine). Configuring with "--enable-ethread-pre-pentium4-compatibility enable_ethread_pre_pentium4_compatibilit=yes" did the trick - Thanks! On Fri, Sep 17, 2010 at 3:52 PM, Rickard Green wrote: > > Mark Scandariato wrote: > >> just built R14B for Ubuntu 9.04 (gcc version 4.3.3 (Ubuntu >> 4.3.3-5ubuntu4)) and when I try to enable some async threads, I get an >> "Illegal instruction" fault. >> >> Here's a stack trace: >> >> erlang@REDACTED:~/otp_src_R14B$ bin/cerl -debug +A 4 >> Illegal instruction (core dumped) >> erlang@REDACTED:~/otp_src_R14B$ gdb --core=core >> GNU gdb 6.8-debian >> Copyright (C) 2008 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later < >> http://gnu.org/licenses/gpl.html >> >>> >>> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "i486-linux-gnu". >> (no debugging symbols found) >> Core was generated by >> `/home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug -A 4 -- -root >> /h'. >> Program terminated with signal 4, Illegal instruction. >> [New process 28563] >> [New process 28587] >> #0 0x0823d51a in ?? () >> (gdb) file /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug >> Reading symbols from >> /home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug...done. >> (gdb) where >> #0 ethr_event_reset__ (e=0xb769c030) at >> ../include/internal/pthread/ethr_event.h:94 >> #1 0x0823d723 in ethr_event_reset (e=0xb769c030) at >> pthread/ethr_event.c:200 >> #2 0x0823cf1c in ethr_thr_create (tid=0xb7580ca4, func=0x81515c6 >> , arg=0xb7580c48, opts=0xbfb85348) at pthread/ethread.c:256 >> #3 0x080cb160 in erts_thr_create (tid=0xb7580ca4, func=0x81515c6 >> , arg=0xb7580c48, opts=0xbfb85348) at beam/erl_threads.h:348 >> #4 0x081511bb in init_async (hndl=6) at beam/erl_async.c:143 >> #5 0x081c62ab in async_drv_start (port_num=1, name=0x825f3fa "async", >> opts=0xbfb85400) at sys/unix/sys.c:2375 >> #6 0x080e48d5 in erts_open_driver (driver=0xb769dbb8, pid=4294967291, >> name=0x825f3fa "async", opts=0xbfb85400, error_number_ptr=0x0) >> at beam/io.c:641 >> #7 0x081c66a5 in sys_init_io () at sys/unix/sys.c:2528 >> #8 0x080e60d3 in init_io () at beam/io.c:1304 >> #9 0x0809b41b in erl_init () at beam/erl_init.c:281 >> #10 0x0809dd25 in erl_start (argc=15, argv=0xbfb85704) at >> beam/erl_init.c:1389 >> #11 0x0807f126 in main (argc=4001536, argv=0x0) at sys/unix/erl_main.c:29 >> >> >> I can send a core file if anyone wants it. >> > > I suspect that you have an old x86 processor, and probably need to pass the > --enable-ethread-pre-pentium4-compatibility command line argument to > configure. See < > http://www.erlang.org/doc/installation_guide/INSTALL.html#How-to-Build-and-Install-ErlangOTP_A-Closer-Look-at-the-individual-Steps_Configuring > >. > > However, I just noted that this flag is buggy and have no effect :( I'll > fix that in the next release. As a workaround you can pass the following as > arguments to configure: > --enable-ethread-pre-pentium4-compatibility > enable_ethread_pre_pentium4_compatibilit=yes > > It is important that you pass both of them otherwise it wont work. You can > verify that it worked by verifying that the following line exists in both > $ERL_TOP/erts/include/internal/i686-pc-linux-gnu/ethread_header_config.h and > $ERL_TOP/erts/i686-pc-linux-gnu/config.h: > > #define ETHR_PRE_PENTIUM4_COMPAT 1 > > In order to avoid trouble, don't reuse the source tree you have already > built in. Build from scratch in a new one. > > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From mikpe@REDACTED Fri Sep 17 23:22:26 2010 From: mikpe@REDACTED (Mikael Pettersson) Date: Fri, 17 Sep 2010 23:22:26 +0200 Subject: [erlang-questions] R14B Illegal instruction? In-Reply-To: References: <19603.51929.485076.491881@pilspetsen.it.uu.se> Message-ID: <19603.56338.708189.958008@pilspetsen.it.uu.se> Mark Scandariato writes: > (gdb) disassemble > Dump of assembler code for function ethr_event_reset__: > 0x0823d501 : push %ebp > 0x0823d502 : mov %esp,%ebp > 0x0823d504 : sub $0x8,%esp > 0x0823d507 : mov 0x8(%ebp),%eax > 0x0823d50a : movl $0x77777777,0x4(%esp) > 0x0823d512 : mov %eax,(%esp) > 0x0823d515 : call 0x8238488 > 0x0823d51a : mfence > 0x0823d51d : leave > 0x0823d51e : ret > End of assembler dump. > (gdb) info all-registers ... > eip 0x823d51a 0x823d51a Sigh. See Rickard Green's reply. Builds on 32-bit x86 now use Pentium 4 instructions without prior runtime checks or explicit configure-time options. You now have to explicitly configure pre-Pentium 4 compatibility before the build. Needless to say I think they got the logic wrong. Defaults should be safe, with options for higher-performance (but potentially incompatible) code if needed. From tushar.erlang@REDACTED Fri Sep 17 23:52:29 2010 From: tushar.erlang@REDACTED (Tushar Deshpande) Date: Fri, 17 Sep 2010 17:52:29 -0400 Subject: Supervisor gets killed if a worker is killed Message-ID: Hi, I've just started with OTP. I wrote my first OTP application. I started a root supervisor and a worker process. The worker process performed as expected. But, when I killed the worker using Appmon, I observed that the supervisor too was killed. Why did it happen? Here's the childspec for the worker process {hsr_server, {hsr_server, start_link, []}, permanent, 2000, worker, [hsr_server]} hsr_server is the worker process I would very much appreciate any help/suggestions. Best Regards, Tushar Deshpande From steven.charles.davis@REDACTED Sat Sep 18 02:34:00 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 17 Sep 2010 17:34:00 -0700 (PDT) Subject: Basic lists(?) question Message-ID: What is the meaning of applying the | operator to a non-list tail? 1> A = [a|0]. [a|0] 2> is_list(A). true 3> length(A). ** exception error: bad argument in function length/1 called as length([a|0]) Sorry if this has been asked many times before. I kind of remember seeing this before, but I can't find it in a search. /s From steven.charles.davis@REDACTED Sat Sep 18 02:46:06 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 17 Sep 2010 17:46:06 -0700 (PDT) Subject: Basic lists(?) question In-Reply-To: References: Message-ID: <17e8af42-a9f1-4ce9-a735-831c94b18df8@l17g2000vbf.googlegroups.com> Agh. Found it using google just two minutes after asking.... http://groups.google.com/group/erlang-programming/browse_thread/thread/e50133061a822a85 On Sep 17, 7:34?pm, Steve Davis wrote: > What is the meaning of applying the | operator to a non-list tail? From xuxb1979@REDACTED Sat Sep 18 03:01:16 2010 From: xuxb1979@REDACTED (=?utf-8?Q?=E5=BE=90=E6=99=93=E6=BB=A8?=) Date: Sat, 18 Sep 2010 09:01:16 +0800 Subject: =?utf-8?Q?=E7=AD=94=E5=A4=8D:_Re:_[erlang-questions]_14A_wants_'fop',_wha?= =?utf-8?Q?t/where=3F?= Message-ID: <4c940f72.0882e50a.24ac.ffffd9fc@mx.google.com> Yg i7ytb ???T8588?? ----- ???? ----- ???: Michael Turner ????: 2010?9?10? 21:17 ???: Max Lapshin ??: Ryan Zezeski ; Richard O'Keefe ; Erlang Questions ??: Re: [erlang-questions] 14A wants 'fop', what/where? On Thu, Sep 9, 2010 at 3:05 PM, Max Lapshin wrote: > On Thu, Sep 9, 2010 at 9:42 AM, Michael Turner > wrote: > > this theory that PHP won out over a huge field of server-side scripting > > languages in the late 90s (some of them probably better) because it was > the > > easiest to install. > > > > Erlang in installable almost anywhere without any fop. It is optional > dependency > Yes, but ISTR when I last did an Erlang build for a machine with no pre-built executable anywhere, I found that it's an enabled option. I had to look pretty hard, then finally ask around, to figure out to disable it. Finally, I gave up on that, and figured out where to get FOP and build it. And I didn't even want the PDFs, I was just trying to get through the build. -michael turner From rzezeski@REDACTED Sat Sep 18 06:51:15 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Sat, 18 Sep 2010 00:51:15 -0400 Subject: [erlang-questions] node to node message passing In-Reply-To: <4C8CB006.6070908@amberbio.com> References: <4C8CB006.6070908@amberbio.com> Message-ID: On Sun, Sep 12, 2010 at 6:48 AM, Morten Krogh wrote: > > For communication between node A and node B, there is a process (send > process) on each node, that coordinates all messages. The send process > keeps queues of different priorities around, e.g., a high priority, medium > priority and low priority. Messages are split up into fragments of > a maximum size. The receiver(node B) send process assembles the fragments > into the original message and delivers it locally to the > right process. The fragments ensure that no single transfer will occupy the > connection for very long. > There will be a function send_priority where the user can specify a > priority. The usual send will default to medium, say. > Net tick will use high priority, of course. Small messages that are needed > to produce a web application response can have high priority. File transfers > for backup purposes can have low priority. > The send process then switches between the queues in some way, that could > be very similar to context switching priorities. > > What happens to the low priority, large binary msg when the system becomes loaded with a constant stream of smaller, high priority messages? It seems the low priority msg would become neglected and then you'll be complaining how the msg takes too long to arrive. I think in all but the most trivial cases you would have to understand the inner workings of the priority scheduler, if I may call it that, to understand how your messages will be delivered. To me, this sounds like a problem to be handled by the application. Another thought, why are you trying to pass such a large amount of data anyways? If you are truly constrained by such a slow network then why not do everything in your power to avoid passing large amounts of data? I'm thinking of Hadoop here--move the process to the data, not the other way around. -Ryan From rzezeski@REDACTED Sat Sep 18 07:24:25 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Sat, 18 Sep 2010 01:24:25 -0400 Subject: [erlang-questions] Supervisor gets killed if a worker is killed In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 5:52 PM, Tushar Deshpande wrote: > > The worker process performed as expected. But, when > I killed the worker using Appmon, I observed that the > supervisor too was killed. Why did it happen? > > Here's the childspec for the worker process > {hsr_server, {hsr_server, start_link, []}, permanent, 2000, worker, > [hsr_server]} > > hsr_server is the worker process > > In this case, the more important thing is your supervisor spec. I'm guessing the maximum restarts is set to 0. After the first restart the supervisor will have reached it's restart threshold, kill all worker processes and then itself. http://www.erlang.org/doc/design_principles/sup_princ.html#id66694 -Ryan From erlang@REDACTED Sat Sep 18 12:41:48 2010 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 18 Sep 2010 12:41:48 +0200 Subject: upnp device discovery Message-ID: Has anybody got some erlang code for upnp device disovery? /Joe From jesper.louis.andersen@REDACTED Sat Sep 18 13:27:03 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 18 Sep 2010 13:27:03 +0200 Subject: [erlang-questions] upnp device discovery In-Reply-To: References: Message-ID: On Sat, Sep 18, 2010 at 12:41 PM, Joe Armstrong wrote: > Has anybody got some erlang code for upnp device disovery? I have a keen interest in this as well. -- J. From max.lapshin@REDACTED Sat Sep 18 19:25:50 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Sat, 18 Sep 2010 21:25:50 +0400 Subject: [erlang-questions] ets vs list In-Reply-To: References: <4C8E3541.7050400@erlang-solutions.com> <4C8E3826.30004@amberbio.com> <4C8E4041.1080301@amberbio.com> <4C8E46AA.9050301@erlang-solutions.com> <4C8E4CED.4040602@amberbio.com> <4C8E508D.3040507@erlang-solutions.com> <4C8E5B52.9030506@amberbio.com> <4C8E6990.7000805@erlang-solutions.com> <4C8E75EF.7000702@amberbio.com> <4C8F26FD.8070201@erlang-solutions.com> Message-ID: Lionet (Lev Valkin) helped a lot with excelent solution: store pids in duplicate_bag with key by state. It happened to be very complicated solution, but fastest. From ulrich.moritz@REDACTED Sun Sep 19 00:12:39 2010 From: ulrich.moritz@REDACTED (Moritz Ulrich) Date: Sun, 19 Sep 2010 00:12:39 +0200 Subject: Emacs' Flymake for Erlang with Distel Message-ID: Hello, I'm using Emacs for my daily Erlang programming. One tool I really like is Flymake, which checks if the file compiles on-the-fly. It also embeds the errors in the source-buffer as hidden comments. The erlang distribution contains a file for this: erlang-flymake.el in the lib/tools-*/ directory. However, the implementation of flymake isn't perfect for the following reasons: - It spawns a second process (erlc) after every save (this can be tweaked in flymake, but better implementations would just make an rpc to a running node) - It isn't suited for self-containing erlang projects like nitrogen (It doesn't handle special paths) My idea for fixing both of these would be rather simple: Implement erlang-flymake with Distel (Distributed Emacs Lisp) Distel is an emacs package to make rpc calls to erlang nodes. It's rather advanced and I think it can be used to implement this behavior. I would like to implement this, but I need some starting points: How can I compile a file with just an erlang command and get the output erlc would give me? (erlang:compile(MyFile) or something like this) Are there pitfalls? Oh, and please tell me if someone already implemented this :-) Best regards, Moritz Ulrich -- Moritz Ulrich Programmer, Student, Almost normal Guy http://www.google.com/profiles/ulrich.moritz BB5F086F-C798-41D5-B742-494C1E9677E8 From tushar.erlang@REDACTED Sun Sep 19 01:19:55 2010 From: tushar.erlang@REDACTED (Tushar Deshpande) Date: Sat, 18 Sep 2010 19:19:55 -0400 Subject: [erlang-questions] Supervisor gets killed if a worker is killed In-Reply-To: References: Message-ID: Hi Ryan, You guessed it right. Here's the restart strategy that I used. RestartStrategy = {one_for_one, 0, 1}. I changed it to one_for_one, 10, 1} i.e. 10 restarts in 1 sec, and everything worked fine. Thanks for the help. Best Regards, Tushar Deshpande From tushar.erlang@REDACTED Sun Sep 19 01:37:03 2010 From: tushar.erlang@REDACTED (Tushar Deshpande) Date: Sat, 18 Sep 2010 19:37:03 -0400 Subject: Can OTP worker resume from same state after being restarted by supervisor? Message-ID: Hi, I've a question about OTP's restart strategy. I wrote a simple OTP application with just a root supervisor and a single worker. The worker has an internal state defined by a single integer (count). The 'count' is initialized to zero. Each call to the worker process increments this count by one. I called the worker process 5 times using gen_server:call. This updated current value of counter to be 5. Then, I opened the appmon and killed the worker process. OTP supervisor promptly restarted it. I thought that the process would be resume with the same state that it had at the time of crash. But, I observed that for the restarted process, the 'counter' was reset to zero. I would like the process to resume from the same state after it's restarted by supervisor. Is it possible to do this in OTP? Best Regards, Tushar Deshpande From essiene@REDACTED Sun Sep 19 02:00:22 2010 From: essiene@REDACTED (Essien Essien) Date: Sun, 19 Sep 2010 01:00:22 +0100 Subject: [erlang-questions] Can OTP worker resume from same state after being restarted by supervisor? In-Reply-To: References: Message-ID: Hi Tushar, On Sun, Sep 19, 2010 at 12:37 AM, Tushar Deshpande wrote: > Hi, > > I've a question about OTP's restart strategy. > > I wrote a simple OTP application with just a root supervisor > and a single worker. ?The worker has an internal state defined > by a single integer (count). ?The 'count' is initialized to zero. > Each call to the worker process increments this count by one. > I called the worker process 5 times using gen_server:call. > This updated current value of counter to be 5. > > Then, I opened the appmon and killed the worker process. > OTP supervisor promptly restarted it. ?I thought that the > process would be resume with the same state that it had at > the time of crash. ?But, I observed that for the restarted process, > the 'counter' was reset to zero. > > I would like the process to resume from the same state after > it's restarted by supervisor. You will have to maintain your own persistent state. Write to a file, update a DETS table, use mnesia... But you do have to do this yourself. Supervision tree is not about managing state, only about keeping the tree up and running. > > Is it possible to do this in OTP? > > > Best Regards, > > Tushar Deshpande > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From freza@REDACTED Sun Sep 19 02:21:46 2010 From: freza@REDACTED (Jachym Holecek) Date: Sun, 19 Sep 2010 01:21:46 +0100 Subject: [erlang-questions] Can OTP worker resume from same state after being restarted by supervisor? In-Reply-To: References: Message-ID: <20100919002146.GA24953@hanele> # Tushar Deshpande 2010-09-19: > I've a question about OTP's restart strategy. > > I wrote a simple OTP application with just a root supervisor > and a single worker. The worker has an internal state defined > by a single integer (count). The 'count' is initialized to zero. > Each call to the worker process increments this count by one. > I called the worker process 5 times using gen_server:call. > This updated current value of counter to be 5. > > Then, I opened the appmon and killed the worker process. > OTP supervisor promptly restarted it. I thought that the > process would be resume with the same state that it had at > the time of crash. But, I observed that for the restarted process, > the 'counter' was reset to zero. Right, this is the expected behaviour -- supervisor's children are restarted with the arguments given in that supervisor's child specs and these arguments are used by child's init/1 function to calculate initial state. Restarting with exactly the same internal state by default would be a bad idea -- if that state led to a crash, then there's really no reason at all to believe in it's integrity; better start all over again. > I would like the process to resume from the same state after > it's restarted by supervisor. > > Is it possible to do this in OTP? Yes. If you want to store state persistently, you'd probably go with Mnesia table, for transient storage (not preserved across application restarts) you could use a public ETS table owned by a do-nothing server process living alongside you child-supervisor. As to when to store worker process state snapshots -- you can either isolate known-safe spots in worker lifetime or trap exits and store state from terminate/2 callback. In more realistic scenarios, you'd probably offload this responsibility from workers altogether and come up with some kind of job manager server to deal with it instead. Other point is that typically you'd want to store only carefully chosen bits of worker state, instead of whole context structure -- that could get messy if you're keeping track of more dynamic things than an integer or two (think pids or ets ids or some transaction ids that could all be long gone by the time you restart & restore worker context). Lastly, for the particular case you're describing, you can remove the counter from server state and instead rely on ets:update_counter/N or mnesia:dirty_update_counter/N to manage it. HTH, -- Jachym From attila.r.nohl@REDACTED Sun Sep 19 10:14:45 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Sun, 19 Sep 2010 10:14:45 +0200 Subject: [erlang-questions] Emacs' Flymake for Erlang with Distel In-Reply-To: References: Message-ID: 2010/9/19, Moritz Ulrich : [...] > How can I compile a file with just an erlang command and get the > output erlc would give me? (erlang:compile(MyFile) or something like > this) compile:file/2 > Are there pitfalls? Make sure that the necessary behaviours are loaded both when compiling from emacs and compiling from the command line. From karol.skocik@REDACTED Sun Sep 19 10:43:28 2010 From: karol.skocik@REDACTED (karol skocik) Date: Sun, 19 Sep 2010 10:43:28 +0200 Subject: [erlang-questions] Emacs' Flymake for Erlang with Distel In-Reply-To: References: Message-ID: This serves me pretty well: http://github.com/ks/flymake-erl/blob/master/flymake-erl and solves >> It isn't suited for self-containing erlang projects like nitrogen >> (It doesn't handle special paths) by walking directories, locating Emakefiles and applying most specific Emakefile applicable for file to compile. It does not depend on Distel or anything else, but could be straightforwardly integrated. I didn't find spawning erlc a problem for me, but opinion on this matter is surely a matter of taste. The upside of spawning erlc is, that it's trivial and correct, delays between saving a file and seeing mistakes in Emacs buffer are short (couple hundreds of milisecs). Karol On Sun, Sep 19, 2010 at 12:12 AM, Moritz Ulrich wrote: > Hello, > > I'm using Emacs for my daily Erlang programming. One tool I really > like is Flymake, which checks if the file compiles on-the-fly. It also > embeds the errors in the source-buffer as hidden comments. > > The erlang distribution contains a file for this: erlang-flymake.el in > the lib/tools-*/ directory. > > However, the implementation of flymake isn't perfect for the following reasons: > - It spawns a second process (erlc) after every save (this can be > tweaked in flymake, but better implementations would just make an rpc > to a running node) > - It isn't suited for self-containing erlang projects like nitrogen > (It doesn't handle special paths) > > > My idea for fixing both of these would be rather simple: Implement > erlang-flymake with Distel (Distributed Emacs Lisp) > Distel is an emacs package to make rpc calls to erlang nodes. It's > rather advanced and I think it can be used to implement this behavior. > > I would like to implement this, but I need some starting points: > > How can I compile a file with just an erlang command and get the > output erlc would give me? (erlang:compile(MyFile) or something like > this) > Are there pitfalls? > > Oh, and please tell me if someone already implemented this :-) > > Best regards, > Moritz Ulrich > > -- > Moritz Ulrich > Programmer, Student, Almost normal Guy > > http://www.google.com/profiles/ulrich.moritz > BB5F086F-C798-41D5-B742-494C1E9677E8 > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From serge@REDACTED Sun Sep 19 13:34:26 2010 From: serge@REDACTED (Serge Aleynikov) Date: Sun, 19 Sep 2010 07:34:26 -0400 Subject: [erlang-questions] how to get stacktraces of all processes? In-Reply-To: <4C92F827.6000203@gmail.com> References: <4C92F827.6000203@gmail.com> Message-ID: <4C95F542.7030807@gmail.com> Anton, This type of behavior is usually indicative of the fact that your application has a deadlock. During startup application_controller starts all applications in your release file synchronously one by one. If any one application has a deadlock, it locks the application_controller. The most probable cause of a deadlock during startup of an application is when it tries to issue gen_server:call/3 call from within gen_server's init/1 callback. Check for that. Also make sure you included sasl application. What I normally do is start your emulator using "-boot start_sasl", and then start the debugger (debugger:start()), include the application's modules in the debugger's list of files to watch for, and set up a breakpoint in the supervisor's init call, so that I can walk through the application startup process. Then spotting the deadlock becomes quite straight forward. Serge On 9/17/2010 1:09 AM, mabrek wrote: > Hello. > > I've got application that hangs on startup. > I tried several alternatives. Appmon doesn't show unstarted > applications. Pman shows only start function and current function for > process, but I want full stacktraces. i() is similar to pman. > erlang:process_display gives quite obscure output, process_info(.., > [backtrace]) is the same. > Is there a way to get full stacktraces of all processes in VM? > > Best regards, > Anton Lebedevich. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ulf.wiger@REDACTED Sun Sep 19 14:14:54 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Sun, 19 Sep 2010 14:14:54 +0200 Subject: [erlang-questions] Supervisor gets killed if a worker is killed In-Reply-To: References: Message-ID: <4C95FEBE.8090400@erlang-solutions.com> On 09/19/2010 01:19 AM, Tushar Deshpande wrote: > Hi Ryan, > > You guessed it right. Here's the restart strategy that I used. > RestartStrategy = {one_for_one, 0, 1}. > > I changed it to one_for_one, 10, 1} i.e. 10 restarts in 1 sec, > and everything worked fine. It may well be that you have thought through the scenario where you'd reach the number 10, but it strikes me as a high number. In many cases, I'd say that 3 restarts in one second is more than enough. The thing to consider is how long it may take for a looping restart to... well, loop. If you pick too high a number for MaxR, you risk missing some looping restarts because they don't occur quickly enough for the limit to be reached. That, and detection time. If one second is a very long time in your application, and you can't imagine a situation where even 5 restarts within that time would be considered 'normal', there is no reason to set MaxR so high. In some cases, even something like {one_for_one, 3, 60} would be fine, but again, you need to think through the scenarios for your own application. BR, Ulf W From ulf.wiger@REDACTED Sun Sep 19 14:28:00 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Sun, 19 Sep 2010 14:28:00 +0200 Subject: [erlang-questions] Can OTP worker resume from same state after being restarted by supervisor? In-Reply-To: <20100919002146.GA24953@hanele> References: <20100919002146.GA24953@hanele> Message-ID: <4C9601D0.5040304@erlang-solutions.com> On 09/19/2010 02:21 AM, Jachym Holecek wrote: > > Yes. If you want to store state persistently, you'd probably go with > Mnesia table, for transient storage (not preserved across application > restarts) you could use a public ETS table owned by a do-nothing server > process living alongside you child-supervisor. An old trick is to create the ets table in the start function, before starting the child process. This way, the supervisor becomes the owner of the table, and it will survive child restarts. The drawback is that you are never allowed to execute code in the supervisor again (except at the next child restart), which makes it a bit awkward to perform certain upgrade functions, e.g. deleting the table (you can rename it, though). A newer trick is to specify an heir of the table (see the docs for ets:new/2). It would be possible to make the supervisor the heir of your table, in which case the supervisor will receive an unexpected message occasionally and complain to the error_logger. So the old trick could be enhanced by creating the ets table in the supervisor, then giving it to the newly started child, having set the heir to itself (as described in the docs for ets:give_away/2). These are tricks for convenience - not necessarily better than creating a table owner process. BR, Ulf W From erlang@REDACTED Sun Sep 19 19:44:00 2010 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 19 Sep 2010 19:44:00 +0200 Subject: mnesia web interface Message-ID: Is there a web interface to mnesia? /Joe From kenneth.lundin@REDACTED Sun Sep 19 21:08:31 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Sun, 19 Sep 2010 21:08:31 +0200 Subject: [erlang-questions] mnesia web interface In-Reply-To: References: Message-ID: I am not aware of any general web-interface towards Mnesia (or ets/dets). There is a graphical GS-based interface via the tv (Table Viewer) application. GS is on it's way to be deprecated so it would be good if we could replace tv with Web-based or Wx-based or maybe both types of interfaces. /Kenneth Erlang/OTP Ericsson On Sun, Sep 19, 2010 at 7:44 PM, Joe Armstrong wrote: > Is there a web interface to mnesia? > > /Joe > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From rpettit@REDACTED Sun Sep 19 21:27:28 2010 From: rpettit@REDACTED (Rick Pettit) Date: Sun, 19 Sep 2010 14:27:28 -0500 (CDT) Subject: [erlang-questions] mnesia web interface In-Reply-To: References: Message-ID: <57146.172.29.10.158.1284924448.squirrel@squirrelmail.vail> On Sun, September 19, 2010 12:44 pm, Joe Armstrong wrote: > Is there a web interface to mnesia? I haven't had a chance to use it yet, but there's a Yaws appmod named "ymnesia" which might be worth checking out. -Rick From ok@REDACTED Mon Sep 20 00:12:54 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 20 Sep 2010 10:12:54 +1200 Subject: [erlang-questions] local renaming with a module attribute In-Reply-To: References: Message-ID: On Sep 11, 2010, at 4:48 AM, Fred Hebert wrote: > I would like to discuss the idea of adding a new module attribute allowing > to rename a module when doing explicit external calls (local renaming, if I > can coin the term). The attribute -rename(long_module_name, name) would > allow the programmer to use 'name' instead of 'long_module_name' inside a > given module. This is pretty much the kind of thing we have with the current Java-wannabe dotted package names, and gets into the same kind of horrible messes. For a rather different kind of approach, which I think addresses the issues that Fred Hebert is concerned with, and more, see http://www.cs.otago.ac.nz/staffpriv/ok/childmod.htm which I see is now four and a half years old. The idea is to stand the problem on its head and say "why should an module have a really_long_multipart_name in the first place?" From wolfmanjm@REDACTED Mon Sep 20 02:33:18 2010 From: wolfmanjm@REDACTED (Jim Morris) Date: Sun, 19 Sep 2010 17:33:18 -0700 (PDT) Subject: running an Erlang OTP application on a production Linux server Message-ID: <76e44379-3472-4704-a517-43e72b0ffade@z30g2000prg.googlegroups.com> I recently needed to deploy a full blown OTP application on a production Linux server, with all the always up, auto restart if VM dies, start on server boot etc. After much Googling and reading I came up with my own (convoluted) solution which I hope others will find interesting. http://blog.wolfman.com/articles/2010/9/19/running-erlang-otp-applications-as-daemons-on-ubuntu-servers My take is that if Erlang were able to capture and handle Linux signals like TERM within an OTP application my job would have been a lot simpler :) From rapsey@REDACTED Mon Sep 20 06:08:11 2010 From: rapsey@REDACTED (Rapsey) Date: Mon, 20 Sep 2010 06:08:11 +0200 Subject: [erlang-questions] running an Erlang OTP application on a production Linux server In-Reply-To: <76e44379-3472-4704-a517-43e72b0ffade@z30g2000prg.googlegroups.com> References: <76e44379-3472-4704-a517-43e72b0ffade@z30g2000prg.googlegroups.com> Message-ID: Do you have upstart? This is what I use: start on filesystem and net-device-up stop on runlevel [06] env HOME=/home/ubuntu respawn script chdir path/to/ebin exec sudo su nobody -c "erl +Bd +K true +A 32 -noinput -mnesia dir '\"/path/to/mnesiadir\"' -eval \"application:start(myapp,permanent)\"" end script lp, Sergej On Mon, Sep 20, 2010 at 2:33 AM, Jim Morris wrote: > I recently needed to deploy a full blown OTP application on a > production Linux server, with all the always up, auto restart if VM > dies, start on server boot etc. > > After much Googling and reading I came up with my own (convoluted) > solution which I hope others will find interesting. > > > http://blog.wolfman.com/articles/2010/9/19/running-erlang-otp-applications-as-daemons-on-ubuntu-servers > > My take is that if Erlang were able to capture and handle Linux > signals like TERM within an OTP application my job would have been a > lot simpler :) > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From max.lapshin@REDACTED Mon Sep 20 07:16:44 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 09:16:44 +0400 Subject: [erlang-questions] running an Erlang OTP application on a production Linux server In-Reply-To: References: <76e44379-3472-4704-a517-43e72b0ffade@z30g2000prg.googlegroups.com> Message-ID: On Mon, Sep 20, 2010 at 8:08 AM, Rapsey wrote: > Do you have upstart? This is what I use: > But how to stop it? From rapsey@REDACTED Mon Sep 20 07:42:32 2010 From: rapsey@REDACTED (Rapsey) Date: Mon, 20 Sep 2010 07:42:32 +0200 Subject: [erlang-questions] running an Erlang OTP application on a production Linux server In-Reply-To: References: <76e44379-3472-4704-a517-43e72b0ffade@z30g2000prg.googlegroups.com> Message-ID: On Mon, Sep 20, 2010 at 7:16 AM, Max Lapshin wrote: > On Mon, Sep 20, 2010 at 8:08 AM, Rapsey wrote: > > Do you have upstart? This is what I use: > > > > But how to stop it? > sudo stop name_of_conf From max.lapshin@REDACTED Mon Sep 20 08:33:46 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 10:33:46 +0400 Subject: limited error_logger dump (vm crashes because of generating dump) Message-ID: For several times I meet bad situation, when BEAM crashes due to error_logger. It begins to dump laaarge state, take much more memory than state was due to list nature (once, I've seen error_logger eating 15GB of memory) and then crashes. I haven't yet looked at otp code, but I'm interested if it is possible to implement limited dump of state. Perhaps, I just need to add byte limit to ~p output in io_lib:format? If this would be the easiest solution, I can make a patch. I don't see any practical reason to dump more than 2-3 megabytes of state and messages and in any case it is a very bad situation, when error_logger halts and crashes VM. From max.lapshin@REDACTED Mon Sep 20 08:53:04 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 10:53:04 +0400 Subject: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: In fact gen_server:format_status doesn't solve the problem: gen_server.erl: format("** Generic server ~p terminating \n" "** Last message in was ~p~n" "** When Server state == ~p~n" "** Reason for termination == ~n** ~p~n", [Name, Msg, State, Reason1]), This formatting has problems: Reason1 maybe many megabytes big when state was big and some badmatch has met with it. Maybe it would be better to format status (change import from error_logger:format to error_logger_formatter:format) into limited-size binary in the dying process and only then send data to error_logger process. It will reduce memory consumption, lower bottleneck-ness of one-core error_logger. Speaking about detailed logging: in fact, it is usually interesting to know first 10-20 lines of termination Reason and process dictionay with each value, limited to 10-20 lines. Or 20-40. But not 400 000 lines of state dump. From max.lapshin@REDACTED Mon Sep 20 08:53:04 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 10:53:04 +0400 Subject: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: In fact gen_server:format_status doesn't solve the problem: gen_server.erl: format("** Generic server ~p terminating \n" "** Last message in was ~p~n" "** When Server state == ~p~n" "** Reason for termination == ~n** ~p~n", [Name, Msg, State, Reason1]), This formatting has problems: Reason1 maybe many megabytes big when state was big and some badmatch has met with it. Maybe it would be better to format status (change import from error_logger:format to error_logger_formatter:format) into limited-size binary in the dying process and only then send data to error_logger process. It will reduce memory consumption, lower bottleneck-ness of one-core error_logger. Speaking about detailed logging: in fact, it is usually interesting to know first 10-20 lines of termination Reason and process dictionay with each value, limited to 10-20 lines. Or 20-40. But not 400 000 lines of state dump. From egil@REDACTED Mon Sep 20 09:52:32 2010 From: egil@REDACTED (=?UTF-8?B?QmrDtnJuLUVnaWwgRGFobGJlcmc=?=) Date: Mon, 20 Sep 2010 09:52:32 +0200 Subject: [erlang-questions] profiling network apps: fprof vs sequential tracing In-Reply-To: References: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> <4C9382D7.3010601@erix.ericsson.se> Message-ID: <4C9712C0.2080202@erix.ericsson.se> On 2010-09-17 20:25, Fred Hebert wrote: > 2010/9/17 Bj?rn-Egil Dahlberg > >> >> You can use eprof to find processes + code that produces bottlenecks. It is >> somewhat limited compared to fprof (no callgraph for instance) but is is >> much much faster (especially after the rewrite to r14) and will likely >> identify problematic intersections of processes and code. >> >> >> Regards, >> Bj?rn-Egil >> Erlang/OTP > > > I'm not sure when the rewrite happened, but I just tested this on R14A > (confirmed by joelr on R14B) and it seems like there is a bug with > eprof:analyse/2. Whenever there is no option passed, the sort defaults to > {sort, time}. However, whenever an option is passed (either {sort, calls} or > {sort, time}), the output is sorted by the number of calls. > I will look into it. // Bj?rn-Egil From wolfmanjm@REDACTED Mon Sep 20 10:15:13 2010 From: wolfmanjm@REDACTED (Jim Morris) Date: Mon, 20 Sep 2010 01:15:13 -0700 (PDT) Subject: running an Erlang OTP application on a production Linux server In-Reply-To: References: <76e44379-3472-4704-a517-43e72b0ffade@z30g2000prg.googlegroups.com> Message-ID: Yes I have upstart and I mention that as an option in my article. However it does not handle logging the way I wanted it, and upstart seems to be in a state of flux so I didn't want to use it. It is also only available on the latest version of Ubuntu, and not all my servers run a version that has upstart. However I agree it can do the job, but you do need to add a pre-stop stanza if you want to shutdown gracefully, otherwise it does exactly what daemontools does, which was the whole point of my article :) On Sep 19, 10:16?pm, Max Lapshin wrote: > On Mon, Sep 20, 2010 at 8:08 AM, Rapsey wrote: > > Do you have upstart? This is what I use: > > But how to stop it? > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > Seehttp://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED From attila.r.nohl@REDACTED Mon Sep 20 12:18:39 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Mon, 20 Sep 2010 12:18:39 +0200 Subject: [erlang-questions] local renaming with a module attribute In-Reply-To: References: Message-ID: 2010/9/20, Richard O'Keefe : [...] > The idea is to stand the problem on its head and say "why should an module > have a really_long_multipart_name in the first place?" The project I'm working on has nearly 10000 modules. In order to avoid module name collision and to avoid module names like astfgl, the median module name length is 16 characters. If this is really long or just long - I'm not sure. From joelr1@REDACTED Mon Sep 20 13:57:48 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 12:57:48 +0100 Subject: [erlang-questions] profiling network apps: fprof vs sequential tracing In-Reply-To: <4C9382D7.3010601@erix.ericsson.se> References: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> <4C9382D7.3010601@erix.ericsson.se> Message-ID: On Sep 17, 2010, at 4:01 PM, Bj?rn-Egil Dahlberg wrote: > lcnt will pinpoint locks with long acquisition time and high collision rates, which is a signal of high contention for specific memory areas. Does this tell you anything? 6> lcnt:conflicts(). lock id #tries #collisions collisions [%] time [us] duration [%] ----- --- ------- ------------ --------------- ---------- ------------- alcu_allocator 26 95048664 317604 0.3341 3113330 1.8408 run_queue 4 10342831 210812 2.0382 2924435 1.7291 pix_lock 256 9795287 17111 0.1747 1416804 0.8377 pollset 1 992163 24805 2.5001 1350653 0.7986 proc_msgq 1816 7019209 67222 0.9577 628357 0.3715 proc_status 1816 15674155 92564 0.5906 592533 0.3503 proc_link 1816 4471858 9593 0.2145 574034 0.3394 port_lock 335 3263611 19866 0.6087 545129 0.3223 db_tab 51 7044068 2975 0.0422 476275 0.2816 timeofday 1 4417012 20162 0.4565 439839 0.2601 proc_main 1816 8389632 301788 3.5972 188551 0.1115 drv_ev_state 16 657921 811 0.1233 126814 0.0750 timer_wheel 1 1594264 4582 0.2874 89597 0.0530 make_ref 1 1093840 1314 0.1201 38031 0.0225 fix_alloc 11 4387004 15027 0.3425 30502 0.0180 run_queue_sleep_list 4 3562460 79 0.0022 12454 0.0074 inet_buffer_stack_lock 1 168484 117 0.0694 11138 0.0066 ptimer_pre_alloc_lock 4 828826 47 0.0057 8794 0.0052 message_pre_alloc_lock 4 5894368 2725 0.0462 7282 0.0043 driver_list 1 746783 127 0.0170 5112 0.0030 ok 7> Thanks, Joel --- http://twitter.com/wagerlabs From egil@REDACTED Mon Sep 20 14:28:20 2010 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Mon, 20 Sep 2010 14:28:20 +0200 Subject: [erlang-questions] profiling network apps: fprof vs sequential tracing In-Reply-To: References: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> <4C9382D7.3010601@erix.ericsson.se> Message-ID: <4C975364.9010804@erix.ericsson.se> On 2010-09-20 13:57, Joel Reymont wrote: > > On Sep 17, 2010, at 4:01 PM, Bj?rn-Egil Dahlberg wrote: > >> lcnt will pinpoint locks with long acquisition time and high collision rates, which is a signal of high contention for specific memory areas. > > Does this tell you anything? > > 6> lcnt:conflicts(). > lock id #tries #collisions collisions [%] time [us] duration [%] > ----- --- ------- ------------ --------------- ---------- ------------- > alcu_allocator 26 95048664 317604 0.3341 3113330 1.8408 > run_queue 4 10342831 210812 2.0382 2924435 1.7291 > pix_lock 256 9795287 17111 0.1747 1416804 0.8377 > pollset 1 992163 24805 2.5001 1350653 0.7986 > proc_msgq 1816 7019209 67222 0.9577 628357 0.3715 > proc_status 1816 15674155 92564 0.5906 592533 0.3503 > proc_link 1816 4471858 9593 0.2145 574034 0.3394 > port_lock 335 3263611 19866 0.6087 545129 0.3223 > db_tab 51 7044068 2975 0.0422 476275 0.2816 > timeofday 1 4417012 20162 0.4565 439839 0.2601 > proc_main 1816 8389632 301788 3.5972 188551 0.1115 > drv_ev_state 16 657921 811 0.1233 126814 0.0750 > timer_wheel 1 1594264 4582 0.2874 89597 0.0530 > make_ref 1 1093840 1314 0.1201 38031 0.0225 > fix_alloc 11 4387004 15027 0.3425 30502 0.0180 > run_queue_sleep_list 4 3562460 79 0.0022 12454 0.0074 > inet_buffer_stack_lock 1 168484 117 0.0694 11138 0.0066 > ptimer_pre_alloc_lock 4 828826 47 0.0057 8794 0.0052 > message_pre_alloc_lock 4 5894368 2725 0.0462 7282 0.0043 > driver_list 1 746783 127 0.0170 5112 0.0030 > ok > 7> Initially this looks good, however, you should examine the process locks more closely. Use, lcnt:swap_pid_keys() and lcnt:conflicts([{combine, false}]) to see the contention on individual locks. The 1816 processes has together 3.6% collision rate which suggests that some process has a lot of contention on its main_lock. Regards, Bj?rn-Egil From sverker@REDACTED Mon Sep 20 14:29:24 2010 From: sverker@REDACTED (Sverker Eriksson) Date: Mon, 20 Sep 2010 14:29:24 +0200 Subject: [erlang-questions] How to release term created by enif_make_copy In-Reply-To: References: Message-ID: <4C9753A4.3010202@erix.ericsson.se> Late answer after long vacation: caox wrote: > Hi > > In my experiment of erlang nif, I use an global env to store term received from erlang proc s via enif_make_copy(). I am wondering how could I free some specific terms created on this env but not all of them? > > I didn't find any lib funcs to do this job. Both enif_clear_env() and enif_free_env() do whole cleanup, and enif_free is annotated to be used only for memory allocated by enif_alloc. > There is no way to free the memory of an individual term while keeping other terms in the same environment. You would have to create the term with its own environment. /Sverker, Erlang/OTP From max.lapshin@REDACTED Mon Sep 20 14:29:56 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 16:29:56 +0400 Subject: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: Fresh stats from production: [{pid,<5525.6.0>}, {memory,1103578784}, {message_queue_len,6}, {total_heap_size,275894550}, {heap_size,47828850}, {stack_size,19121}, {dictionary,[{'$ancestors',[<5525.2.0>]}, {'$initial_call',{gen_event,init_it,6}}]}, {current_function,{io_lib,write_binary_body,2}}] In few seconds error_logger will shut down whole virtual machine. I've made the following to prevent it: http://github.com/erlyvideo/erlyvideo/blob/master/src/core/io_lib_pretty_limited.erl ?? this is a io_lib_pretty, adapted not to runaway out of memory http://github.com/erlyvideo/erlyvideo/blob/master/src/core/gen_server_ems.erl#L747 ? gen server with adaptations to use limited print. I have also removed sasl logger from error_logger handlers: http://github.com/erlyvideo/erlyvideo/blob/master/src/erlyvideo.erl#L84 http://github.com/erlyvideo/erlmedia/blob/master/src/ems_log.erl#L50 and still I'm getting failures. Seems, that I will have to replace error_logger with my own stuff =( From joelr1@REDACTED Mon Sep 20 14:37:00 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 13:37:00 +0100 Subject: percept issue Message-ID: Should I be concerned about this issue? Any suggestions on how to troubleshoot it? Thanks, Joel --- 13> percept:analyze("/Volumes/xxx/percept.dat"). Parsing: "/Volumes/xxx/percept.dat" check_activity_consistency, state flow invalid. --- http://twitter.com/wagerlabs From vinoski@REDACTED Mon Sep 20 14:55:32 2010 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 20 Sep 2010 08:55:32 -0400 Subject: [erlang-questions] Re: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 2:53 AM, Max Lapshin wrote: > In fact gen_server:format_status doesn't solve the problem: > > gen_server.erl: > ? ?format("** Generic server ~p terminating \n" > ? ? ? ? ? "** Last message in was ~p~n" > ? ? ? ? ? "** When Server state == ~p~n" > ? ? ? ? ? "** Reason for termination == ~n** ~p~n", > ? ? ? ? ? [Name, Msg, State, Reason1]), > > > This formatting has problems: Reason1 maybe many megabytes big when > state was big and some badmatch has met with it. > Maybe it would be better to format status (change import from > error_logger:format to error_logger_formatter:format) into > limited-size binary in the dying process and only then send data to > error_logger process. It will reduce memory consumption, lower > bottleneck-ness of one-core error_logger. > > Speaking about detailed logging: in fact, it is usually interesting to > know first 10-20 lines of termination Reason and process dictionay > with each value, limited to 10-20 lines. Or 20-40. But not 400 000 > lines of state dump. Perhaps you could provide your own format_status function for your gen_server: http://erlang.org/doc/man/gen_server.html#Module:format_status-2 See especially the final paragraph of the description there. --steve From egil@REDACTED Mon Sep 20 14:57:37 2010 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Mon, 20 Sep 2010 14:57:37 +0200 Subject: [erlang-questions] percept issue In-Reply-To: References: Message-ID: <4C975A41.1030302@erix.ericsson.se> On 2010-09-20 14:37, Joel Reymont wrote: > Should I be concerned about this issue? > > Any suggestions on how to troubleshoot it? > > Thanks, Joel > > --- > > 13> percept:analyze("/Volumes/xxx/percept.dat"). > Parsing: "/Volumes/xxx/percept.dat" > check_activity_consistency, state flow invalid. It is probably nothing to be concerned with. The warning notifies that there is a inactive / active mismatch when parsing the log. However, it is then sorted by timestamp when inserted to the db. system_profile messages *might* come out-of-order, it has been a bug there before, though that is taken care of during insert and consolidation. // Bj?rn-Egil From max.lapshin@REDACTED Mon Sep 20 15:00:35 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 17:00:35 +0400 Subject: [erlang-questions] Re: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 4:55 PM, Steve Vinoski wrote: > > Perhaps you could provide your own format_status function for your gen_server: > > http://erlang.org/doc/man/gen_server.html#Module:format_status-2 > Steve, it is now enough. gen_server is also printing last message (that may be VERY large) and reason for stop. This reason may also involve current status, bypassing format_status. And this is also not the end: proc_lib sends error info to error_logger, including reason. And it is another way of leaking gen_server status. From joelr1@REDACTED Mon Sep 20 15:02:27 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 14:02:27 +0100 Subject: [erlang-questions] percept issue In-Reply-To: <4C975A41.1030302@erix.ericsson.se> References: <4C975A41.1030302@erix.ericsson.se> Message-ID: <9A56E829-3901-4719-A3BE-791156AA9BFB@gmail.com> On Sep 20, 2010, at 1:57 PM, Bj?rn-Egil Dahlberg wrote: > system_profile messages *might* come out-of-order, it has been a bug there before, though that is taken care of during insert and consolidation. Fair enough. My issue at the moment is that my Percept DAT file is 21.79gb and I only have 8gb of physical memory in my laptop. I hope percept:analyze/1 can pull through! --- http://twitter.com/wagerlabs From gleber.p@REDACTED Mon Sep 20 15:34:11 2010 From: gleber.p@REDACTED (Gleb Peregud) Date: Mon, 20 Sep 2010 15:34:11 +0200 Subject: [erlang-questions] Re: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: That's the reason, if it is possible, why I prefer to use private ets for storing any big data structures in process's state. On Mon, Sep 20, 2010 at 15:00, Max Lapshin wrote: > On Mon, Sep 20, 2010 at 4:55 PM, Steve Vinoski wrote: >> >> Perhaps you could provide your own format_status function for your gen_server: >> >> http://erlang.org/doc/man/gen_server.html#Module:format_status-2 >> > > Steve, it is now enough. gen_server is also printing last message > (that may be VERY large) and reason for stop. This reason may also > involve current status, bypassing format_status. And this is also not > the end: proc_lib sends error info to error_logger, including reason. > And it is another way of leaking gen_server status. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From max.lapshin@REDACTED Mon Sep 20 15:35:57 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 20 Sep 2010 17:35:57 +0400 Subject: [erlang-questions] Re: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 5:34 PM, Gleb Peregud wrote: > That's the reason, if it is possible, why I prefer to use private ets > for storing any big data structures in process's state. > It is a workaround, but it isn't applicable for me. I'd better remove all logging from erlang, than use structure, which is several times slower. From joelr1@REDACTED Mon Sep 20 15:52:00 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 14:52:00 +0100 Subject: [erlang-questions] percept issue In-Reply-To: <4C975A41.1030302@erix.ericsson.se> References: <4C975A41.1030302@erix.ericsson.se> Message-ID: If I have a 21gb precept dat file, does it mean that 21gb or more of virtual memory will be used to process it? Thanks, Joel --- http://twitter.com/wagerlabs From egil@REDACTED Mon Sep 20 16:10:57 2010 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Mon, 20 Sep 2010 16:10:57 +0200 Subject: [erlang-questions] percept issue In-Reply-To: References: <4C975A41.1030302@erix.ericsson.se> Message-ID: <4C976B71.8080504@erix.ericsson.se> On 2010-09-20 15:52, Joel Reymont wrote: > If I have a 21gb precept dat file, does it mean that 21gb or more of virtual memory will be used to process it? Percept is not built to handle that amount of data. It is intended for small snapshots (i.e 20 - 30 seconds) of a system during heavy load. It will try to read everything into ets so your system will not be happy. // Bj?rn-Egil From joelr1@REDACTED Mon Sep 20 16:32:20 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 15:32:20 +0100 Subject: [erlang-questions] profiling network apps: fprof vs sequential tracing In-Reply-To: <4C975364.9010804@erix.ericsson.se> References: <4D183539-CA78-4694-B7DB-38048628DA98@gmail.com> <4C9382D7.3010601@erix.ericsson.se> <4C975364.9010804@erix.ericsson.se> Message-ID: Here's another try... It seems that run queue has the most collisions and so does the alcu (what's this?) allocator. Inspecting run_queue does not reveal much, same for port_lock. Inspecting alcu_allocator is a bit more informative (see bottom of this message). What is sl_alloc in alcu_allocator? What else can you tell from the output below? Thanks, Joel --- 4> lcnt:conflicts(). lock id #tries #collisions collisions [%] time [us] duration [%] ----- --- ------- ------------ --------------- ---------- ------------- run_queue 4 10487093 231528 2.2077 2938951 3.5419 alcu_allocator 26 97202142 351907 0.3620 2862881 3.4503 pix_lock 256 10015157 17007 0.1698 1354482 1.6324 pollset 1 1040351 25264 2.4284 994662 1.1987 timeofday 1 4047854 22195 0.5483 506510 0.6104 db_tab 51 7178614 2379 0.0331 141460 0.1705 timer_wheel 1 1353739 5456 0.4030 121754 0.1467 port_lock 6 174978 7006 4.0039 117303 0.1414 proc_msgq 118 2289433 4009 0.1751 69833 0.0842 proc_status 118 5119297 7218 0.1410 55563 0.0670 drv_ev_state 16 678154 576 0.0849 48590 0.0586 fix_alloc 11 4477982 15539 0.3470 36026 0.0434 proc_main 118 2193895 34621 1.5781 28330 0.0341 proc_link 118 1379852 2468 0.1789 28017 0.0338 make_ref 1 1117599 1310 0.1172 7732 0.0093 message_pre_alloc_lock 4 6024214 2838 0.0471 7478 0.0090 run_queue_sleep_list 4 3670712 152 0.0041 6047 0.0073 asyncq 4 2289643 99 0.0043 4476 0.0054 gc_info 1 495809 231 0.0466 2017 0.0024 driver_list 1 763813 126 0.0165 1243 0.0015 ok 5> lcnt:swap_pid_keys(). ok 6> lcnt:conflicts([{print, [name, tries, ratio, time]}]). lock #tries collisions [%] time [us] ----- ------- --------------- ---------- run_queue 10487093 2.2077 2938951 alcu_allocator 97202142 0.3620 2862881 pix_lock 10015157 0.1698 1354482 pollset 1040351 2.4284 994662 timeofday 4047854 0.5483 506510 db_tab 7178614 0.0331 141460 timer_wheel 1353739 0.4030 121754 #Port 171730 4.0360 117111 couch_server 1822575 0.4158 67427 couch_httpd_vhost 1661330 0.4809 53325 drv_ev_state 678154 0.0849 48590 fix_alloc 4477982 0.3470 36026 error_logger 2957617 0.8027 31324 1882295 0.4172 20830 make_ref 1117599 0.1172 7732 message_pre_alloc_lock 6024214 0.0471 7478 run_queue_sleep_list 3670712 0.0041 6047 code_server 738174 0.0646 5178 asyncq 2289643 0.0043 4476 user_drv 803014 0.0321 2586 ok 7> lcnt:conflicts([{combine, false}, {print, [name, id, tries, ratio, time]}]). lock id #tries collisions [%] time [us] ----- --- ------- --------------- ---------- alcu_allocator sl_alloc 22038182 1.0851 1713952 run_queue 1 2634529 2.5435 1070317 pollset undefined 1040351 2.4284 994662 pix_lock 155 2745777 0.5145 830022 run_queue 2 2588360 2.2875 764836 run_queue 3 2647046 2.0890 631155 timeofday undefined 4047854 0.5483 506510 run_queue 4 2617158 1.9110 472643 alcu_allocator sl_alloc 21125298 0.2046 310092 pix_lock 159 781355 0.1273 283931 alcu_allocator binary_alloc 1731685 0.3971 127924 timer_wheel undefined 1353739 0.4030 121754 alcu_allocator binary_alloc 1722667 0.3959 117874 alcu_allocator binary_alloc 1661971 0.3650 117487 #Port port_lock 171730 4.0360 117111 alcu_allocator binary_alloc 1708185 0.3724 106316 db_tab stats_hit_table 343547 0.6235 94918 alcu_allocator sl_alloc 21293832 0.0899 94616 alcu_allocator sl_alloc 20218634 0.0769 87777 alcu_allocator eheap_alloc 805310 0.5648 66938 ok 8> 10> lcnt:inspect(alcu_allocator, [{print, [name, id, tries, colls, ratio, duration]}]). lock id #tries #collisions collisions [%] duration [%] ----- --- ------- ------------ --------------- ------------- alcu_allocator sl_alloc 22038182 239134 1.0851 2.0656 alcu_allocator sl_alloc 21125298 43221 0.2046 0.3737 alcu_allocator binary_alloc 1731685 6876 0.3971 0.1542 alcu_allocator binary_alloc 1722667 6820 0.3959 0.1421 alcu_allocator binary_alloc 1661971 6067 0.3650 0.1416 alcu_allocator binary_alloc 1708185 6362 0.3724 0.1281 alcu_allocator sl_alloc 21293832 19142 0.0899 0.1140 alcu_allocator sl_alloc 20218634 15541 0.0769 0.1058 alcu_allocator eheap_alloc 805310 4548 0.5648 0.0807 alcu_allocator eheap_alloc 468005 970 0.2073 0.0473 alcu_allocator eheap_alloc 479774 1035 0.2157 0.0412 alcu_allocator eheap_alloc 476197 949 0.1993 0.0405 alcu_allocator driver_alloc 565863 771 0.1363 0.0081 alcu_allocator driver_alloc 398542 141 0.0354 0.0019 alcu_allocator driver_alloc 380875 133 0.0349 0.0014 alcu_allocator ets_alloc 86067 20 0.0232 0.0013 alcu_allocator ets_alloc 87458 22 0.0252 0.0009 alcu_allocator driver_alloc 396137 131 0.0331 0.0007 alcu_allocator ets_alloc 85320 8 0.0094 0.0007 alcu_allocator ets_alloc 88007 12 0.0136 0.0004 ok --- http://twitter.com/wagerlabs From joelr1@REDACTED Mon Sep 20 17:14:04 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 16:14:04 +0100 Subject: using percept Message-ID: What can you tell from this precept overview? http://skitch.com/joel.reymont/da1fa/percept Does it imply that there's very low concurrency in the application during the 16s run? What are the black and green bits? Why does the drop-down display Ports by default, has Processes greyed out and displays no information when I select Ports & Processes? Thanks, Joel --- http://twitter.com/wagerlabs From joelr1@REDACTED Mon Sep 20 18:28:07 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 17:28:07 +0100 Subject: [erlang-questions] percept issue In-Reply-To: <4C976B71.8080504@erix.ericsson.se> References: <4C975A41.1030302@erix.ericsson.se> <4C976B71.8080504@erix.ericsson.se> Message-ID: <2E438276-AA18-4742-A43C-4A1E2AE3AB42@gmail.com> On Sep 20, 2010, at 3:10 PM, Bj?rn-Egil Dahlberg wrote: > Percept is not built to handle that amount of data. It is intended for small snapshots (i.e 20 - 30 seconds) of a system during heavy load. There's a bug in percept, I think. I invoked percept:analyze/1 before percept:stop_profile and the DAT file just kept growing and growing and growing. I ended with 46gb+, I think. Stopping percept results in a very analyzable 300mb file. --- http://twitter.com/wagerlabs From joelr1@REDACTED Mon Sep 20 18:38:56 2010 From: joelr1@REDACTED (Joel Reymont) Date: Mon, 20 Sep 2010 17:38:56 +0100 Subject: using percept In-Reply-To: References: Message-ID: Here's a graph from a 70s run: http://skitch.com/joel.reymont/da1si/percept What can you tell by looking at it? Thanks, Joel --- http://twitter.com/wagerlabs From acton@REDACTED Mon Sep 20 18:57:39 2010 From: acton@REDACTED (Brian Acton) Date: Mon, 20 Sep 2010 09:57:39 -0700 Subject: corrupt external term Message-ID: Hi everyone, I'm running R13B04 on FreeBSD 7.3-stable. I have a heterogeneous cluster set up in a frontend/backend configuration with two mnesia nodes doing primary storage in the backend. The frontend nodes talk to the backend machines using the mnesia remote protocol. Recently, I've started to experience failure as seen by this 'frontend-node1' got a corrupted external term from 'backend-node1' on distribution channel 10173 <<...,104,2,114,0,3,82,0,3, ...>> ATOM_CACHE_REF translations: 0='frontend-node1', 1='', 2=xmlcdata, 3=xmlelement, 4=jid, 5=never, 6=offline_msg The binary above is truncated for brevity but I think everyone gets the gist of what is going on. A quick google search on the phrase yielded only one link. This link references a bug in the atom cache: http://groups.google.com/group/erlang-programming/browse_thread/thread/0d5bf2cbaa7c921a/5b59ee7d0800cde7?lnk=raot&fwc=2&pli=1 So several questions ensue: 1) is this just a simple matter of mnesia table corruption? 2) any recommendations on how to rebuild / resolve the corruption? 3) is it possible that this corruption is related to the aforementioned atom cache bug? 4) any tips on instrumentation / tracing / logging to further isolate this? Any tips / tricks / help, definitely appreciated. Thx, --b From anthonym@REDACTED Mon Sep 20 19:07:27 2010 From: anthonym@REDACTED (Anthony Molinaro) Date: Mon, 20 Sep 2010 10:07:27 -0700 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: <373251123.4471284719113411.JavaMail.root@zimbra> References: <135744570.4331284719040369.JavaMail.root@zimbra> <373251123.4471284719113411.JavaMail.root@zimbra> Message-ID: <20100920170727.GE48108@alumni.caltech.edu> On Fri, Sep 17, 2010 at 10:25:13AM +0000, Robert Virding wrote: > - Importing a function from a module provides absolutely no new functionality. It just saves you from having to explicitly qualify the function call with the module name. So if foo/1 is imported from module bar, -import(bar,[foo/1])., then foo(...) and bar:foo(...) are exactly equivalent, the compiler does the conversion internally. It saves you some writing. Many people frown on importing and say you shouldn't do it as it obscures what you are actually doing, I only use for importing functions from some common modules like lists. So for code change does this mean the new version of an imported module is called? Thanks, -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro From dmercer@REDACTED Mon Sep 20 19:33:55 2010 From: dmercer@REDACTED (David Mercer) Date: Mon, 20 Sep 2010 12:33:55 -0500 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: <20100920170727.GE48108@alumni.caltech.edu> References: <135744570.4331284719040369.JavaMail.root@zimbra> <373251123.4471284719113411.JavaMail.root@zimbra> <20100920170727.GE48108@alumni.caltech.edu> Message-ID: <006801cb58ea$01a1bec0$04e53c40$@com> On September 20, 2010, Anthony Molinaro wrote: > On Fri, Sep 17, 2010 at 10:25:13AM +0000, Robert Virding wrote: > > - Importing a function from a module provides absolutely no new > functionality. It just saves you from having to explicitly qualify the > function call with the module name. So if foo/1 is imported from module > bar, -import(bar,[foo/1])., then foo(...) and bar:foo(...) are exactly > equivalent, the compiler does the conversion internally. It saves you some > writing. Many people frown on importing and say you shouldn't do it as it > obscures what you are actually doing, I only use for importing functions > from some common modules like lists. > > So for code change does this mean the new version of an imported module is > called? Yes. From ulf.wiger@REDACTED Mon Sep 20 20:33:59 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 20 Sep 2010 20:33:59 +0200 Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: <006801cb58ea$01a1bec0$04e53c40$@com> References: <135744570.4331284719040369.JavaMail.root@zimbra> <373251123.4471284719113411.JavaMail.root@zimbra> <20100920170727.GE48108@alumni.caltech.edu> <006801cb58ea$01a1bec0$04e53c40$@com> Message-ID: <4C97A917.2070007@erlang-solutions.com> On 20/09/2010 19:33, David Mercer wrote: > On September 20, 2010, Anthony Molinaro wrote: >> So for code change does this mean the new version of an imported module is >> called? > > Yes. But there is one case that is still a bit tricky. Consider a module where you e.g. spawn some connector processes. These would do some initialization, then call a blocking function, placing the current module on the call stack until the function returns. If the module is purged during this time, the processes will be killed. Example: dies() -> spawn_link(fun() -> timer:sleep(10000), io:fwrite("still alive~n", []) end). Eshell V5.7.5 (abort with ^G) 1> c(mymod). {module, mymod} 2> mymod:dies(). <0.61.0> 3> l(mymod). {module,mymod} 4> l(mymod). ** exception exit: killed In http:github.com/esl/plain_fsm/ I recently added a special function to allow such blocking calls to be made in a tail_call fashion to allow for soft upgrade. Example (module tail_call_test): p() -> spawn_link(fun() -> wait(mystate) end). wait(S) -> ?tail_apply(fun() -> timer:sleep(10000) end, reply, S). reply(ok, R, S) -> io:fwrite("reply -> ~p (S=~p)~n", [R,S]). The macro expands to a slightly more complex function, plain_fsm:tail_apply/5. The first argument of the macro is the blocking function call. The second is the name of the function that gets called to handle the return value, and S is the "state" of the process. 5> tail_call_test:p(). <0.73.0> reply -> ok (S=mystate) 6> tail_call_test:p(). <0.79.0> 7> c(tail_call_test,[{i,"../include"}]). {ok,tail_call_test} 8> c(tail_call_test,[{i,"../include"}]). {ok,tail_call_test} 9> c(tail_call_test,[{i,"../include"}]). {ok,tail_call_test} reply -> ok (S=mystate) As you can tell, the process didn't die. It is also possible to add a function, data_vsn() -> any(), which is checked before and after the call. Plain_fsm will call Mod:code_change/3 if the value of data_vsn() has changed in the meantime. code_change(From,S,Xtra) -> io:fwrite("code_change(~p, ~p, ~p)~n", [From,S,Xtra]), {ok, {upgraded, S}}. 10> tail_call_test:p(). <0.270.0> 11> c(tail_call_test,[{i,"../include"}]). {ok,tail_call_test} 12> c(tail_call_test,[{i,"../include"}]). {ok,tail_call_test} code_change(0, mystate, tail_apply) reply -> ok (S={upgraded,mystate}) Plain_fsm does something similar with its own version of hibernate(), and includes a few other goodies that at least I find convenient. BR, Ulf W -- Ulf Wiger CTO, Erlang Solutions Ltd, formerly Erlang Training & Consulting Ltd http://www.erlang-solutions.com From robert.virding@REDACTED Mon Sep 20 22:22:13 2010 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 20 Sep 2010 20:22:13 +0000 (GMT) Subject: [erlang-questions] Is this code tail recursive? In-Reply-To: <20100920170727.GE48108@alumni.caltech.edu> Message-ID: <16959951.12011285014133720.JavaMail.root@zimbra> ----- "Anthony Molinaro" wrote: > On Fri, Sep 17, 2010 at 10:25:13AM +0000, Robert Virding wrote: > > - Importing a function from a module provides absolutely no new > functionality. It just saves you from having to explicitly qualify the > function call with the module name. So if foo/1 is imported from > module bar, -import(bar,[foo/1])., then foo(...) and bar:foo(...) are > exactly equivalent, the compiler does the conversion internally. It > saves you some writing. Many people frown on importing and say you > shouldn't do it as it obscures what you are actually doing, I only use > for importing functions from some common modules like lists. > > So for code change does this mean the new version of an imported > module is > called? Yes, importing functions is pure syntactic sugar. Robert From anton.krasovsky@REDACTED Mon Sep 20 23:05:49 2010 From: anton.krasovsky@REDACTED (Anton Krasovsky) Date: Mon, 20 Sep 2010 22:05:49 +0100 Subject: Erlang User Group in Ireland Message-ID: Guys, I wonder if there are any Erlang users in Ireland, particularly in Dublin? I think it would be a good idea to start an Erlang user group here. Please let me know if you're interested! Regards, Anton From rzezeski@REDACTED Wed Sep 22 02:34:43 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Tue, 21 Sep 2010 20:34:43 -0400 Subject: [erlang-questions] Re: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 9:35 AM, Max Lapshin wrote: > > It is a workaround, but it isn't applicable for me. I'd better remove > all logging from erlang, than use structure, which is several times > slower. > > > I've run into the same problem as you. In my case, I have a web application that has _very large_ files (> 700MB) posted to it. My application reads the entire request body as a binary and passes it around. While developing there would be causes where I would have a bad case match or some other error . This would cause SASL to attempt to print out the entire binary, which, obviously, did not go so well. It would cause a massive spike in memory and typically I would have to kill the VM. Eventually, I ironed out all the bugs (well, the ones I know about anyways :) ), and it's since been a distant memory. However, I remember thinking "I wish I had a way to tell SASL to abbreviate the output." The format_status callback looks useful for certain situations...I really need to sit down and read through the canonical docs. From max.lapshin@REDACTED Wed Sep 22 06:14:10 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 22 Sep 2010 08:14:10 +0400 Subject: [erlang-questions] Re: limited error_logger dump (vm crashes because of generating dump) In-Reply-To: References: Message-ID: At least, I've removed all native logging: SASL and builtin one. Changed gen_server to gen_server_ems, which is dumping state, reason and message not so beautiful, but with io_lib_pretty_limited, which is a safe rewrite of io_lib_pretty. Fixed log4erl error_handler to dump reports. Now logging looks much better for me: =INFO REPORT==== 22-Sep-2010::08:12:50 === Starting Erlyvideo ... Eshell V5.8 (abort with ^G) (ems@REDACTED)1> =INFO REPORT==== 22-Sep-2010::08:12:50 === Loading logger configuration from priv/log4erl.conf info 2010-09-22 08:12:50.992366 Starting crypto_sup:crypto_server <0.62.0> info 2010-09-22 08:12:50.994390 ===================== Started application: crypto ===================== info 2010-09-22 08:12:51.31351 Starting kernel_safe_sup:timer_server <0.74.0> info 2010-09-22 08:12:51.44217 Starting rtmp_sup:rtmpt_sessions_sup <0.79.0> info 2010-09-22 08:12:51.44646 Starting rtmp_sup:rtmpt_session_sup <0.80.0> info 2010-09-22 08:12:51.44996 Starting rtmp_sup:rtmp_socket_sup <0.81.0> info 2010-09-22 08:12:51.46619 Starting rtmp_sup:rtmp_stat_collector_sup <0.82.0> info 2010-09-22 08:12:51.46904 ===================== Started application: rtmp ===================== info 2010-09-22 08:12:51.49732 <0.2.0> {"Erlyvideo is loading config from file ~s~n",["priv/erlyvideo.conf"]} info 2010-09-22 08:12:51.247934 Starting ems_sup:ems_network_lag_monitor_sup <0.89.0> info 2010-09-22 08:12:51.252619 Starting ems_sup:shared_objects_sup <0.99.0> info 2010-09-22 08:12:51.252974 Starting ems_sup:shared_object_sup <0.100.0> info 2010-09-22 08:12:51.253304 Starting ems_sup:media_provider_sup_default <0.101.0> info 2010-09-22 08:12:51.253865 ===================== Started application: erlyvideo ===================== info 2010-09-22 08:12:51.265251 <0.2.0> {std_info,"Started Erlyvideo"} And erlyvideo can survive crash of process with 10-200 megabytes of used memory. Earlier it was a death to system. From vladdu55@REDACTED Wed Sep 22 10:32:10 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 22 Sep 2010 10:32:10 +0200 Subject: eunit status messages Message-ID: Hi! I checked the eunit code and want to make sure that I didn't miss anything: when starting a test run, there is an undocumented option 'report' that should allow to register a process to receive status messages. I tried it but I get strange bahaviour, maybe I'm doing something wrong? 1> eunit:test(mod, [{report, self()}]). Test passed. ok 2> receive X->X after 5 -> nothing end. {start,#Ref<0.0.0.67>} 6> receive X->X after 5 -> nothing end. nothing 7> eunit:test(mod, [{report, self()}]). Test passed. ok 8> receive X->X after 5 -> nothing end. nothing It works the first time, but not afterwards. Also I would have expected to see the status messages defined in eunit_proc... I will try to dig deeper into the code, but some pointers would be much appreciated. Would it be a problem to make this functionality public and documented? best regards, Vlad From vladdu55@REDACTED Wed Sep 22 10:48:24 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 22 Sep 2010 10:48:24 +0200 Subject: eunit status messages In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:32, Vlad Dumitrescu wrote: > I checked the eunit code and want to make sure that I didn't miss > anything: when starting a test run, there is an undocumented option > 'report' that should allow to register a process to receive status > messages. I tried it but I get strange bahaviour, maybe I'm doing > something wrong? Okay, scratch that... it just doesn't work with the shell process as receiver. When will I learn to not use it for anything important? :-) The question that remains is if I can count on this undocumented feature to remain available? regards, Vlad From ebegumisa@REDACTED Wed Sep 22 11:28:35 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 22 Sep 2010 19:28:35 +1000 Subject: UBF(A) vs ETF / UBF(C) vs gen_fsm Message-ID: Hello all, I've been looking at Dr. Joe Armstrong's UBF with interest, especially for IPC between two Erlang nodes over TCP (non-epmd) and possibly between an Erlang node and a XULRunner client (JavaScript/XPCOM). I have two questions for those with UBF experience... 1) Concerning UBF(A): Joe mentioned in his original paper that UBF(A) encoding was more upto 40% more compact than the VM External Term Format. But that was 8 years ago. I understand the external term format has improved since then. Is UBF(A) still more compact? 2) Concerning UBF(C): If one is using gen_fsm server-side together with UBF(A) over the wire, would I be correct in saying that the OTP behaviour provides the contract checking that UBF(C) would provide? That is, would using gen_fsm with UBF encoding for messages render the UBF(C) contract checker redundant? - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From erlang@REDACTED Wed Sep 22 12:29:30 2010 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 22 Sep 2010 12:29:30 +0200 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 11:28 AM, Edmond Begumisa wrote: > Hello all, > > I've been looking at Dr. Joe Armstrong's UBF with interest, especially for > IPC between two Erlang nodes over TCP (non-epmd) and possibly between an > Erlang node and a XULRunner client (JavaScript/XPCOM). I have two questions > for those with UBF experience... That sounds like a very good idea - I'd really like to see clear separation of interest - Erlang on one side of a communication channel, XULRunner on the other side with a clear specified interface between and no feature leakage between the two. You could try UBF for this - if it turned out that it wasn't good I have a few other tricks up my sleeve - how is the javascript/XULrunner side of things? - the erlang bit is easy - can you easily setup the underlying socket communication with XULrunner? > > 1) Concerning UBF(A): Joe mentioned in his original paper that UBF(A) > encoding was more upto 40% more compact than the VM External Term Format. > But that was 8 years ago. I understand the external term format has improved > since then. Is UBF(A) still more compact? I haven't measured but I suspect UBF will be more compact - the UBF caching optimization allows you to substitute any repeated term by a single character - so if you discover that some term is repeated, you can just replace it by a single character - this depends upon the encoder, which is up to you. > > 2) Concerning UBF(C): If one is using gen_fsm server-side together with > UBF(A) over the wire, would I be correct in saying that the OTP behaviour > provides the contract checking that UBF(C) would provide? That is, would > using gen_fsm with UBF encoding for messages render the UBF(C) contract > checker redundant? No - to do this you would have to compile a UBF(C) state machine to a gen_fsm state machine I see no advantage in this - it's essentially the same thing, but the UBF state checker is is more specialized than gen_fsm. gen_fsm is a generic machine. The UBF(C) checker is specialized for UBF contracts Another thought - the UBF type system would *easily* map to a JSON transport format I blogged this in 2009 http://armstrongonsoftware.blogspot.com/2009/02/json-protocols-part-1.html It's the same idea as UBF using JSON instead of UBF(A) - the encoding isn't as pretty but it makes interoperability easier - I guess *every* language has a JSON encoder/decoder so you don't have to write a UBF encoder/decoder. Having parsed the JSON (or UBF(A) ) the parse trees would be essentially the same thing so the contract checker would be easy. Cheers /Joe > > - Edmond - > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ebegumisa@REDACTED Wed Sep 22 14:37:01 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 22 Sep 2010 22:37:01 +1000 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: References: Message-ID: Thanks very much for your quick response... On Wed, 22 Sep 2010 20:29:30 +1000, Joe Armstrong wrote: > That sounds like a very good idea - I'd really like to see clear > separation of > interest - Erlang on one side of a communication channel, XULRunner on > the other side > with a clear specified interface between and no feature leakage between > the two. I've had some unpleasant experiences with informal home-grown/private protocols so the benefits of UBF for my purposes immediately jumped out at me. It's one of those things you can't appreciate unless you've been there. > You could try UBF for this - if it turned out that it wasn't good I > have a few other > tricks up my sleeve - how is the javascript/XULrunner side of things? > - the erlang bit > is easy - can you easily setup the underlying socket communication > with XULrunner? Promising but it's still early stages. One problem so far is that XULRunner is best suited for HTTP communication (raw TCP sockets are doable but a real pain). I'm currently trying to see if rather than going against the grain by wresting with Mozilla's lower level socket API's, it might be easier to have the Erlang side serve UBF over HTTP using say mochiweb (long-poll/keep-alive issues notwithstanding.) I'll be sure to post some code when I've got something working. I'm intrigued by those tricks you mention?!?! > I haven't measured but I suspect UBF will be more compact - the UBF > caching optimization > allows you to substitute any repeated term by a single character - so > if you discover that > some term is repeated, you can just replace it by a single character - > this depends upon the > encoder, which is up to you. Perhaps it's time for a UBF EEP! IMO, it would also make writing & debugging port drivers easier. > No - to do this you would have to compile a UBF(C) state machine to a > gen_fsm state machine > I see no advantage in this - it's essentially the same thing, but the > UBF state checker is > is more specialized than gen_fsm. gen_fsm is a generic machine. The > UBF(C) checker is > specialized for UBF contracts I see... I understand now. > Another thought - the UBF type system would *easily* map to a JSON > transport format > I blogged this in 2009 > > http://armstrongonsoftware.blogspot.com/2009/02/json-protocols-part-1.html > > It's the same idea as UBF using JSON instead of UBF(A) - the encoding > isn't as pretty > but it makes interoperability easier - I guess *every* language has a > JSON encoder/decoder > so you don't have to write a UBF encoder/decoder. > > Having parsed the JSON (or UBF(A) ) the parse trees would be > essentially the same thing > so the contract checker would be easy. Ditto! It's actually that blog entry that got me to look at UBF (what ever happened to part II of that BTW!?!) I've been contemplating using Gemini's UBF-JSON to make life easier on the XULRunner side. I don't know if you've had a look at/recommend their implementation... http://github.com/norton/ubf-jsonrpc Erlang and XULRunner could be a lethal combination. I'm sure I'm not the only one who has seen this. Thanks. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From magnus@REDACTED Wed Sep 22 14:38:57 2010 From: magnus@REDACTED (Magnus Henoch) Date: Wed, 22 Sep 2010 13:38:57 +0100 Subject: eunit status messages In-Reply-To: (Vlad Dumitrescu's message of "Wed, 22 Sep 2010 10:32:10 +0200") References: Message-ID: <84mxraymu6.fsf@linux-b2a3.site> Vlad Dumitrescu writes: > I checked the eunit code and want to make sure that I didn't miss > anything: when starting a test run, there is an undocumented option > 'report' that should allow to register a process to receive status > messages. I tried it but I get strange bahaviour, maybe I'm doing > something wrong? > > 1> eunit:test(mod, [{report, self()}]). > Test passed. > ok > 2> receive X->X after 5 -> nothing end. > {start,#Ref<0.0.0.67>} > 6> receive X->X after 5 -> nothing end. > nothing > 7> eunit:test(mod, [{report, self()}]). > Test passed. > ok > 8> receive X->X after 5 -> nothing end. > nothing > > It works the first time, but not afterwards. Also I would have > expected to see the status messages defined in eunit_proc... You're using the variable X more than once, so you're not seeing the messages as they are not identical to the first one. You can use the shell function flush() to print messages received by the shell: Eshell V5.7.5 (abort with ^G) 1> eunit:test(fun() -> ok end, [{report, self()}]). Test passed. ok 2> flush(). Shell got {start,#Ref<0.0.0.30>} Shell got {status,[], {progress,'begin', {group,[{desc,undefined}, {spawn,undefined}, {order,undefined}]}}} Shell got {status,[1], {progress,'begin', {test,[{desc,undefined}, {source,{erl_eval,expr,5}}, {line,0}]}}} Shell got {status,[1],{progress,'end',{ok,[{time,6},{output,<<>>}]}}} Shell got {status,[],{progress,'end',{1,[{time,21},{output,<<>>}]}}} Shell got {stop,#Ref<0.0.0.30>,<0.33.0>} ok -- Magnus Henoch, magnus@REDACTED Erlang Solutions http://www.erlang-solutions.com/ From vladdu55@REDACTED Wed Sep 22 14:48:29 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Wed, 22 Sep 2010 14:48:29 +0200 Subject: [erlang-questions] Re: eunit status messages In-Reply-To: <84mxraymu6.fsf@linux-b2a3.site> References: <84mxraymu6.fsf@linux-b2a3.site> Message-ID: On Wed, Sep 22, 2010 at 14:38, Magnus Henoch wrote: > You're using the variable X more than once, so you're not seeing the > messages as they are not identical to the first one. Riiiight, thanks for pointing that out! I used to have a routine of forgetting all unused bindings in the shell, but it's been a while since I needed to use it... regards, Vlad From max.lapshin@REDACTED Wed Sep 22 14:48:57 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 22 Sep 2010 16:48:57 +0400 Subject: Lock-free message queue Message-ID: Just a curiosity: currently there is some kind of locks in sending message. Have anybody tried to implement a lock-free linked list: http://www.amd64.org/fileadmin/user_upload/pub/epham08-asf-eval.pdf Or I'm just looking at wrong place and erts_smp_proc_lock is already using something like this? From egil@REDACTED Wed Sep 22 15:10:26 2010 From: egil@REDACTED (=?UTF-8?B?QmrDtnJuLUVnaWwgRGFobGJlcmc=?=) Date: Wed, 22 Sep 2010 15:10:26 +0200 Subject: [erlang-questions] Lock-free message queue In-Reply-To: References: Message-ID: <4C9A0042.7090206@erix.ericsson.se> On 2010-09-22 14:48, Max Lapshin wrote: > Just a curiosity: currently there is some kind of locks in sending message. > Have anybody tried to implement a lock-free linked list: > http://www.amd64.org/fileadmin/user_upload/pub/epham08-asf-eval.pdf > > Or I'm just looking at wrong place and erts_smp_proc_lock is already > using something like this? The message queue already has this, sort of. The process that owns the message box has an "inner box" that he has a lock on and an "outer box" that all senders compete for. So the lock contention is on the tail of the queue on the "outer box" when lots of processes sends to that process. The mail box owner is not concerned with it though. // Bj?rn-Egil From alexander.harju@REDACTED Wed Sep 22 15:14:48 2010 From: alexander.harju@REDACTED (Alexander Harju) Date: Wed, 22 Sep 2010 15:14:48 +0200 Subject: [erlang-questions] upnp device discovery Message-ID: > Has anybody got some erlang code for upnp device disovery? > > /Joe > Hi. I was just experimenting with last weekend. For discovery you just have to send the discovery message to multicast address 239.255.255.250:1900 -define(UPNP_DISCOVER, "M-SEARCH * HTTP/1.1\r\n" "Host: 239.255.255.250:1900\r\n" "Man: \"ssdp:discover\"\r\n" "ST: upnp:rootdevice\r\n" "MX: 3\r\n\r\n"). {ok, S} = gen_udp:open(1900, [{reuseaddr,true}]), gen_udp:send(S, {239,255,255,250}, 1900, ?UPNP_DISCOVER), receive {udp, S, _, _, Msg0} -> [_|Msg] = lists:reverse(Msg0), io:format("~s~n", [lists:reverse(Msg)]) after 5000 -> ok end. The rest is "just" handling the UPnP massive SOAP protocol. xmerl and inets or yaws, erlsom. You can also listen to UPnP multicasts like this: Opts = [{multicast_loop,false}, {multicast_if,{0,0,0,0}}, {multicast_ttl,4}], {ok, S} = gen_udp**:open(1900, Opts), inet:setopts(S, [{add_membership, {Ip, {0,0,0,0}}}]), receive {udp, S, ... BR // Alex From pascalchapier@REDACTED Wed Sep 22 16:11:14 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Wed, 22 Sep 2010 16:11:14 +0200 Subject: I have trouble with R14B/wx/Windows XP Message-ID: Hi list, I have trouble with the R14B release under Windows XP. It seems that the wx library does not work properly on my PC. I installed the R14B on my PC, in addition to the R14A which I use by default. If I launch werl from the erl5.8.1\bin directory, and try to launch an application which use wx, it crashes without any log or valuable information: werl session: Erlang R14B (erts-5.8.1) [smp:2:2] [rq:2] [async-threads:0] Execution des cmds erlang from "D:\documents and Settings\PChapier\.erlang "Nom de la machine: "WXFR88952L", cookie nocookie Eshell V5.8.1 (abort with ^G) 1> debugger:start(). crash message (translated from french): werl.exe - Application error The instruction at "0x7C911980" uses the address memory location "0x00000000". The memory cannot be read. The crash happens as soon as the wx:new() function is called (I made the call directly from the shell to check it). I changed the windows path to use the R14B by default: same behaviour. A very strange thing is that it has worked properly once with debbugger:start(), and once with wx:new() within more than 20 trials; but as soon as I stopped wx, I got a crash at the next call to wx:new/0. I have tried also the toolbar, tv,appmon... and some application without graphic or using gs: no problem except toolbar->debugger. It is ok with R14A. Any idea? BR Pascal. From mononcqc@REDACTED Wed Sep 22 16:31:19 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 22 Sep 2010 10:31:19 -0400 Subject: [erlang-questions] I have trouble with R14B/wx/Windows XP In-Reply-To: References: Message-ID: Have you tried it with erl.exe instead? There are problems documented with werl.exe and stuff that requires GUIs, as far as I know. The erl.exe shell is supposed to be alright. On Wed, Sep 22, 2010 at 10:11 AM, Pascal Chapier wrote: > > Hi list, > > I have trouble with the R14B release under Windows XP. > It seems that the wx library does not work properly on my PC. > > I installed the R14B on my PC, in addition to the R14A which I use by > default. > > If I launch werl from the erl5.8.1\bin directory, and try to launch an > application which use wx, it crashes > without any log or valuable information: > > werl session: > Erlang R14B (erts-5.8.1) [smp:2:2] [rq:2] [async-threads:0] > > Execution des cmds erlang from "D:\documents and Settings\PChapier\.erlang > "Nom de la machine: "WXFR88952L", cookie nocookie > Eshell V5.8.1 (abort with ^G) > 1> debugger:start(). > > crash message (translated from french): > werl.exe - Application error > The instruction at "0x7C911980" uses the address memory location > "0x00000000". The memory cannot be read. > > The crash happens as soon as the wx:new() function is called (I made the > call directly > from the shell to check it). > I changed the windows path to use the R14B by default: same behaviour. > A very strange thing is that it has worked properly once with > debbugger:start(), > and once with wx:new() within more than 20 trials; but as soon as I stopped > wx, I got a crash at the next call > to wx:new/0. > > I have tried also the toolbar, tv,appmon... and some application without > graphic or using gs: > no problem except toolbar->debugger. > > It is ok with R14A. > > Any idea? > > BR > > Pascal. > > From dangud@REDACTED Wed Sep 22 16:40:40 2010 From: dangud@REDACTED (Dan Gudmundsson) Date: Wed, 22 Sep 2010 16:40:40 +0200 Subject: [erlang-questions] I have trouble with R14B/wx/Windows XP In-Reply-To: References: Message-ID: Actually it's the other way around, werl should work, and erl have problems. It seems wxWidgets and erlang fights over stdio or something, we havn't been able to find the real issue. werl -detached works if you don't want a shell at all example: werl -detached -run wx demo /Dan On Wed, Sep 22, 2010 at 4:31 PM, Fred Hebert wrote: > Have you tried it with erl.exe instead? There are problems documented with > werl.exe and stuff that requires GUIs, as far as I know. The erl.exe shell > is supposed to be alright. > > On Wed, Sep 22, 2010 at 10:11 AM, Pascal Chapier > wrote: > >> >> Hi list, >> >> I have trouble with the R14B release under Windows XP. >> It seems that the wx library does not work properly on my PC. >> >> I installed the R14B on my PC, in addition to the R14A which I use by >> default. >> >> If I launch werl from the erl5.8.1\bin directory, and try to launch an >> application which use wx, it crashes >> without any log or valuable information: >> >> werl session: >> Erlang R14B (erts-5.8.1) [smp:2:2] [rq:2] [async-threads:0] >> >> Execution des cmds erlang from "D:\documents and Settings\PChapier\.erlang >> "Nom de la machine: "WXFR88952L", cookie nocookie >> Eshell V5.8.1 ?(abort with ^G) >> 1> debugger:start(). >> >> crash message (translated from french): >> werl.exe - Application error >> The instruction at "0x7C911980" uses the address memory location >> "0x00000000". The memory cannot be read. >> >> The crash happens as soon as the wx:new() function is called (I made the >> call directly >> from the shell to check it). >> I changed the windows path to use the R14B by default: same behaviour. >> A very strange thing is that it has worked properly once with >> debbugger:start(), >> and once with wx:new() within more than 20 trials; but as soon as I stopped >> wx, I got a crash at the next call >> to wx:new/0. >> >> I have tried also the toolbar, tv,appmon... and some application without >> graphic or using gs: >> no problem except toolbar->debugger. >> >> It is ok with R14A. >> >> Any idea? >> >> BR >> >> Pascal. >> >> > From giulio.petrucci@REDACTED Wed Sep 22 18:11:06 2010 From: giulio.petrucci@REDACTED (Giulio Petrucci) Date: Wed, 22 Sep 2010 18:11:06 +0200 Subject: Installing on cygwin Message-ID: Hi there, this is my first post in this mailing list so, first of all I'd like to say hello to everyone reading here. I'm trying playing with the ReverseHTTP project (http://github.com/tonyg/reversehttp/) and I need to run a web server written in Erlang but... I'm a totally newbie with this language! :-) Under Linux (the OS I use mostly for personal stuff) I didn't have to do anything special. Under Windows, which I use at work, I'm a little bit in trouble. I'd like to install Erlang on my Cygwin installation. I've just downloaded the source code... now what shall I do? I've tried to get into the source directory and type "make" on my shell but... nothing happened. :-) Could anyone help me? Thanks in advance, Giulio -- From mjtruog@REDACTED Wed Sep 22 20:26:20 2010 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 22 Sep 2010 11:26:20 -0700 Subject: [erlang-questions] Lock-free message queue In-Reply-To: References: Message-ID: <4C9A4A4C.70706@gmail.com> It probably has already been implemented here: http://tim.klingt.org/git?p=boost_lockfree.git;a=summary On 09/22/2010 05:48 AM, Max Lapshin wrote: > Just a curiosity: currently there is some kind of locks in sending message. > Have anybody tried to implement a lock-free linked list: > http://www.amd64.org/fileadmin/user_upload/pub/epham08-asf-eval.pdf > > Or I'm just looking at wrong place and erts_smp_proc_lock is already > using something like this? > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > From pascalchapier@REDACTED Wed Sep 22 21:10:54 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Wed, 22 Sep 2010 21:10:54 +0200 Subject: [erlang-questions] I have trouble with R14B/wx/Windows XP Message-ID: Thank you Dan, it works with -detached option. it also works with "werl -detached -run my_module start -run debugger start", so it is possible to debug my program. Maybe if I create a my_module.tool file, I will be able to start the debugger before the application. BR. Pascal. From pascalchapier@REDACTED Wed Sep 22 21:51:10 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Wed, 22 Sep 2010 21:51:10 +0200 Subject: Installing on cygwin Message-ID: Hi Giulio, I know I am not answering to your question, but why don't you use the Erlang Windows installation, I am doing the reverse compare to you: mostly windows and very few linux. I have no problem to exchange code on both system. Best regards, Pascal. From fritchie@REDACTED Wed Sep 22 22:43:36 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Wed, 22 Sep 2010 15:43:36 -0500 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: Message of "Wed, 22 Sep 2010 22:37:01 +1000." Message-ID: <56566.1285188216@snookles.snookles.com> Edmond Begumisa wrote: >> Having parsed the JSON (or UBF(A) ) the parse trees would be >> essentially the same thing so the contract checker would be easy. eb> Ditto! It's actually that blog entry that got me to look at UBF eb> (what ever happened to part II of that BTW!?!) I've been eb> contemplating using Gemini's UBF-JSON to make life easier on the eb> XULRunner side. I don't know if you've had a look at/recommend their eb> implementation... eb> http://github.com/norton/ubf-jsonrpc Yes, that's what the ubf-jsonrpc package does. There's also the option of using the http://github.com/norton/ubf stuff as-is to implement a "JSF" server, which is straight JSON across a TCP socket (i.e., without any JSON-RPC HTTP stuff). IIRC, it's just a server-side configuration option {proto, ubf} or {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. -Scott From dangud@REDACTED Wed Sep 22 23:18:05 2010 From: dangud@REDACTED (Dan Gudmundsson) Date: Wed, 22 Sep 2010 23:18:05 +0200 Subject: [erlang-questions] I have trouble with R14B/wx/Windows XP In-Reply-To: References: Message-ID: :-) That part was not a suggestion to you, it was an example of how to write a gui application without starting werl shell. Anyone else have wx issue's on windows? Any windows programmers out there who can help with debugging? /Dan On Wed, Sep 22, 2010 at 9:10 PM, Pascal Chapier wrote: > > Thank you Dan, > > it works with -detached option. > > it also works with "werl -detached -run my_module start -run debugger start", so it is possible to debug my program. Maybe if I create a my_module.tool file, I will be able to start the debugger before the application. > > BR. > > Pascal. > > From mikpe@REDACTED Wed Sep 22 23:36:24 2010 From: mikpe@REDACTED (Mikael Pettersson) Date: Wed, 22 Sep 2010 23:36:24 +0200 Subject: [erlang-questions] Lock-free message queue In-Reply-To: <4C9A0042.7090206@erix.ericsson.se> References: <4C9A0042.7090206@erix.ericsson.se> Message-ID: <19610.30424.441782.259817@pilspetsen.it.uu.se> =?UTF-8?B?QmrDtnJuLUVnaWwgRGFobGJlcmc=?= writes: > On 2010-09-22 14:48, Max Lapshin wrote: > > Just a curiosity: currently there is some kind of locks in sending message. > > Have anybody tried to implement a lock-free linked list: > > http://www.amd64.org/fileadmin/user_upload/pub/epham08-asf-eval.pdf > > > > Or I'm just looking at wrong place and erts_smp_proc_lock is already > > using something like this? > > The message queue already has this, sort of. The process that owns the > message box has an "inner box" that he has a lock on and an "outer box" > that all senders compete for. So the lock contention is on the tail of > the queue on the "outer box" when lots of processes sends to that > process. The mail box owner is not concerned with it though. A problem here is that people seem to equate "lock-free" with "better than locks". Both approaches ultimately have similar hardware costs, namely cache lines ping-ponging over a bus between caches belonging to whichever package/core/thread wants ownership at the moment. If you call it a lock or just do the appropriate memory barrier + CAS or LL/SC doesn't matter IMO. In addition, lock-free algorithms generally only work as far as modifying the shared data structure goes; then there's a very tricky memory management problem(*), which brings in a whole new set of problems and possible solutions. Having said that, I do think that the process in-mbox (the outer one) is simple and restricted enough that a lock-free solution should be possible. (*) When you delete a node from a shared data structure, _who_ exactly gets to free() that node and when? From chandrashekhar.mullaparthi@REDACTED Thu Sep 23 00:07:25 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 22 Sep 2010 23:07:25 +0100 Subject: ANNOUNCE: ibrowse-2.0.0 Message-ID: Hello everyone, New version of ibrowse released. Extract from the README follows: 22-09-2010 - * Added option preserve_chunked_encoding. This allows the caller to get the raw HTTP response when the Transfer-Encoding is Chunked. This feature was requested by Benoit Chesneau who wanted to write a HTTP proxy using ibrowse. * Fixed bug with the {stream_to, {Pid, once}} option. Bug report and lot of help from Filipe David Manana. Thank you Filipe. * The {error, conn_failed} and {error, send_failed} return values are now of the form {error, {conn_failed, Err}} and {error, {send_failed, Err}}. This is so that the specific socket error can be returned to the caller. I think it looks a bit ugly, but that is the best compromise I could come up with. * Added application configuration parameters default_max_sessions and default_max_pipeline_size. These were previously hard coded to 10. * Versioning of ibrowse now follows the Semantic Versioning principles. See http://semver.org. Thanks to Anthony Molinaro for nudging me in this direction. * The connect_timeout option now only applies to the connection setup phase. In previous versions, the time taken to setup the connection was deducted from the specified timeout value for the request. cheers Chandru From juanjo@REDACTED Thu Sep 23 00:15:02 2010 From: juanjo@REDACTED (Juan Jose Comellas) Date: Wed, 22 Sep 2010 19:15:02 -0300 Subject: gen_leader usage/pointers Message-ID: Can anyone who's ever used or (even better!) written one of the many gen_leader incarnations point me to some documentation/examples on how to use it? I have a scenario where I have to keep a master process (to be chosen randomly or according to load) with slaves that takeover when the master goes down. I'm not sure if gen_leader is the correct solution, as the master communicates to an external service and there will be more than one. Both the master and the slave have open sockets to the external service and the slave socket is "activated" as soon as the master goes down. I'd appreciate any pointers anybody might give me. Thanks, Juanjo From piotrek.kaleta@REDACTED Thu Sep 23 00:52:43 2010 From: piotrek.kaleta@REDACTED (Piotr Kaleta) Date: Thu, 23 Sep 2010 00:52:43 +0200 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: References: Message-ID: <4C9A88BB.5050002@gmail.com> First of all, few (if not all) versions of gen_leader implementations has been gathered on github here: http://github.com/KirinDave/gen_leader_revival. As README says, you should use the 'combined_version'. A good starting point with tutorial is located at: http://order1.blogspot.com/2007/10/scalable-data-structures-in-erlang-and.html However you should remember that current versions of gen_leader works in a way that you have to statically define the list of nodes that can become master in case of other nodes failure. Current implementations however, are incapable of adding nodes that might claim master role in future, at runtime. This makes gen_leader incapable of working in unstable environments when all of master nodes might go down for some reason. On 09/23/2010 12:15 AM, Juan Jose Comellas wrote: > Can anyone who's ever used or (even better!) written one of the many > gen_leader incarnations point me to some documentation/examples on how to > use it? I have a scenario where I have to keep a master process (to be > chosen randomly or according to load) with slaves that takeover when the > master goes down. I'm not sure if gen_leader is the correct solution, as the > master communicates to an external service and there will be more than one. > Both the master and the slave have open sockets to the external service and > the slave socket is "activated" as soon as the master goes down. > > I'd appreciate any pointers anybody might give me. > > Thanks, > > Juanjo > From andrew@REDACTED Thu Sep 23 01:02:03 2010 From: andrew@REDACTED (Andrew Thompson) Date: Wed, 22 Sep 2010 19:02:03 -0400 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: <4C9A88BB.5050002@gmail.com> References: <4C9A88BB.5050002@gmail.com> Message-ID: <20100922230202.GG10050@hijacked.us> On Thu, Sep 23, 2010 at 12:52:43AM +0200, Piotr Kaleta wrote: > First of all, few (if not all) versions of gen_leader implementations > has been gathered on github here: > http://github.com/KirinDave/gen_leader_revival. > As README says, you should use the 'combined_version'. > > A good starting point with tutorial is located at: > http://order1.blogspot.com/2007/10/scalable-data-structures-in-erlang-and.html > > However you should remember that current versions of gen_leader works in > a way that you have to statically define the list of nodes that can > become master in case of other nodes failure. Current implementations > however, are incapable of adding nodes that might claim master role in > future, at runtime. This makes gen_leader incapable of working in > unstable environments when all of master nodes might go down for some > reason. > KirinDave's fork might lack this functionality, but abecciu's fork supports adding candidates at runtime (as well as some additional fixes that Dave hasn't merged). We've also cleaned gen_leader up to be an actual erlang project (rebar/makefile, appfile, etc). Andrew From chandrashekhar.mullaparthi@REDACTED Thu Sep 23 01:36:23 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Thu, 23 Sep 2010 00:36:23 +0100 Subject: ANNOUNCE: ibrowse-2.0.0 In-Reply-To: References: Message-ID: Apologies for the badly formatted version. This should be more readable. * Added option preserve_chunked_encoding. This allows the caller to get the raw HTTP response when the Transfer-Encoding is Chunked. This feature was requested by Benoit Chesneau who wanted to write a HTTP proxy using ibrowse. * Fixed bug with the {stream_to, {Pid, once}} option. Bug report and lot of help from Filipe David Manana. Thank you Filipe. * The {error, conn_failed} and {error, send_failed} return values are now of the form {error, {conn_failed, Err}} and {error, {send_failed, Err}}. This is so that the specific socket error can be returned to the caller. I think it looks a bit ugly, but that is the best compromise I could come up with. * Added application configuration parameters default_max_sessions and default_max_pipeline_size. These were previously hard coded to 10. * Versioning of ibrowse now follows the Semantic Versioning principles. See http://semver.org. Thanks to Anthony Molinaro for nudging me in this direction. * The connect_timeout option now only applies to the connection setup phase. In previous versions, the time taken to setup the connection was deducted from the specified timeout value for the request. On 22 September 2010 23:07, Chandru wrote: > Hello everyone, > > New version of ibrowse released. Extract from the README follows: > > 22-09-2010 - * Added option preserve_chunked_encoding. This allows the > caller to get > ? ? ? ? ? ? ? the raw HTTP response when the Transfer-Encoding is > Chunked. This feature > ? ? ? ? ? ? ? was requested by Benoit Chesneau who wanted to write a > HTTP proxy using > ? ? ? ? ? ? ? ibrowse. > ? ? ? ? ? ? * Fixed bug with the {stream_to, {Pid, once}} option. Bug > report and lot > ? ? ? ? ? ? ? of help from Filipe David Manana. Thank you Filipe. > ? ? ? ? ? ? * The {error, conn_failed} and {error, send_failed} > return values are > ? ? ? ? ? ? ? now of the form {error, {conn_failed, Err}} and > ? ? ? ? ? ? ? {error, {send_failed, Err}}. This is so that the > specific socket error > ? ? ? ? ? ? ? can be returned to the caller. I think it looks a bit > ugly, but that > ? ? ? ? ? ? ? is the best compromise I could come up with. > ? ? ? ? ? ? * Added application configuration parameters > default_max_sessions and > ? ? ? ? ? ? ? default_max_pipeline_size. These were previously hard > coded to 10. > ? ? ? ? ? ? * Versioning of ibrowse now follows the Semantic > Versioning principles. > ? ? ? ? ? ? ? See http://semver.org. Thanks to Anthony Molinaro for > nudging me in > ? ? ? ? ? ? ? this direction. > ? ? ? ? ? ? * The connect_timeout option now only applies to the > connection setup > ? ? ? ? ? ? ? phase. In previous versions, the time taken to setup > the connection > ? ? ? ? ? ? ? was deducted from the specified timeout value for the request. > > cheers > Chandru > From augusto@REDACTED Thu Sep 23 03:25:13 2010 From: augusto@REDACTED (Augusto Becciu) Date: Wed, 22 Sep 2010 22:25:13 -0300 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: References: Message-ID: If you're planning to use gen_leader you definitely want to use my fork which contains many bug fixes and some new features like dynamic addition of candidate nodes. The code is at: http://github.com/abecciu/gen_leader_revival I'm working on a test suite for it which I'm going to release soon, and I'll probably write some doc too. Here are some examples of gen_leader usage: http://github.com/Vagabond/OpenACD/blob/master/src/queue_manager.erl http://github.com/uwiger/gproc/blob/master/src/gproc_dist.erl http://github.com/abecciu/gen_leader_revival/blob/master/examples/skeleton.erl Let me know if you have more questions. Augusto On Wed, Sep 22, 2010 at 7:15 PM, Juan Jose Comellas wrote: > Can anyone who's ever used or (even better!) written one of the many > gen_leader incarnations point me to some documentation/examples on how to > use it? I have a scenario where I have to keep a master process (to be > chosen randomly or according to load) with slaves that takeover when the > master goes down. I'm not sure if gen_leader is the correct solution, as > the > master communicates to an external service and there will be more than one. > Both the master and the slave have open sockets to the external service and > the slave socket is "activated" as soon as the master goes down. > > I'd appreciate any pointers anybody might give me. > > Thanks, > > Juanjo > From ulf.wiger@REDACTED Thu Sep 23 10:45:25 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Thu, 23 Sep 2010 10:45:25 +0200 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: <4C9A88BB.5050002@gmail.com> References: <4C9A88BB.5050002@gmail.com> Message-ID: <4C9B13A5.9070105@erlang-solutions.com> On 23/09/2010 00:52, Piotr Kaleta wrote: > > However you should remember that current versions of gen_leader works in > a way that you have to statically define the list of nodes that can > become master in case of other nodes failure. Current implementations > however, are incapable of adding nodes that might claim master role in > future, at runtime. This makes gen_leader incapable of working in > unstable environments when all of master nodes might go down for some > reason. There is a good reason for this limitation, and while I will not venture to say that all attempts to fix it are necessarily broken, the challenge lies in that the original gen_leader has been subjected to unusually rigorous analysis: http://publications.lib.chalmers.se/cpl/record/index.xsql?pubid=69328 The first version of gen_leader used a leader-election algorithm that wasn't a perfect fit for Erlang's semantics, but even that version successfully went through model checking. Once more advanced methods were developed, including a formal semantics for Distributed Erlang, it was found to be broken, and a new algorithm was selected. This algorithm was tested with model checking, abstract trace analysis and QuickCheck in order to verify the _core behaviour_ (note! this is not the same thing as shaking out all bugs, and although quite a few bugs have been fixed since then, they were not defects in the core implementation). Now the problem: In order to support the dynamic addition and removal of nodes, you either have to show how this is compatible with Stoller's algorithm (which is not trivial, but may well be possible), or invent, or select, a different leader election algorithm. Ideally, one should then embark on a fairly ambitious project (although it probably doesn't have to amount to an entire PhD thesis) in order to show that the added feature didn't in fact break the core function of the leader election behaviour. An alternative, YMMV, would be to accept that everything doesn't have to be unbounded in terms of flexibility and failure modes. I would go as far as saying that one of the most important parts of designing for high availability is to figure out ways to _simplify_ the design so that you have as few different failure modes as possible. Since the AXD 301 is still often used as a reference (the famous "99.9999999% availability"), I could point out that it had two master nodes running in an active-standby configuration. There were also "expansion processors" for scalability up to 32 nodes, but if both master nodes went down, the whole system was considered down. From many discussions I sense the assumption that you have to have redundancy to the nth degree in order to achieve high availability, but the fact is that the AXD 301 scored better than 99.999% service availability based on field reports, each month for years (as long as I was keeping track). With that sort of design, you would configure gen_leader to have both master nodes as leader candidates and the rest as worker nodes (which can more easily be handled dynamically). This is also one reason why the limitation exists in gen_leader in the first place, as the original source of inspiration was the rcmLocker - a distributed read-write locker component which was part of the AXD 301 cluster controller[1]. It had a fairly primitive leader election implementation, based on the global name server. Thomas Arts observed that the locker was difficult to model-check since it combined several difficult patterns in the same implementation, and we started writing a behaviour for leader election that would allow you to separate that logic. Problems that were difficult to solve, and were strictly not needed in the AXD 301 were naturally pushed into some unspecified future. BR, Ulf W [1] http://forum.trapexit.org/viewtopic.php?p=30186#30186 -- Ulf Wiger CTO, Erlang Solutions Ltd, formerly Erlang Training & Consulting Ltd http://www.erlang-solutions.com From timo.lindemann@REDACTED Thu Sep 23 12:18:15 2010 From: timo.lindemann@REDACTED (Timo Lindemann) Date: Thu, 23 Sep 2010 12:18:15 +0200 Subject: Interactive Shell & unicode Message-ID: <1285237095.9813.14.camel@oblivion> Hi, it seems that the erlang shell is somehow affecting Unicode output in the following situation: -module(test). -compile(export_all). start() -> ets:new(tbl, [named_table, set]), ets:insert(tbl, {1, "?"}), io:format("~p~n", ets:tab2list(tbl)). (I don't know if that only happens with ets or dets tables, however.) If I do > erl -run test -run init stop then the output is garbled: > {1,"??"} but if I disable the shell, the output is correct: > erl -run test -run init stop -noshell > {1,"?"} This is Erlang 64-bit on Linux, btw. On my colleague's Powerbook G4, both commands produce the output > {1,[195,131,194,164]} What am I doing wrong? Not that it overly impedes my work, but I spent hours, yesterday, in trying to produce right string output that apparently looked correct to everyone else. Any ideas? Greetings, -T. From juanjo@REDACTED Thu Sep 23 14:23:24 2010 From: juanjo@REDACTED (Juan Jose Comellas) Date: Thu, 23 Sep 2010 09:23:24 -0300 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: <4C9A88BB.5050002@gmail.com> References: <4C9A88BB.5050002@gmail.com> Message-ID: Thanks for the data. I'd really prefer to be able to dynamically assign or remove new nodes to the cluster. It's not entirely critical, but it would be very uncomfortable if I had to restart the cluster in order to modify the list of nodes (i.e. I don't want to wake up at 4:00 AM to do it). On Wed, Sep 22, 2010 at 7:52 PM, Piotr Kaleta wrote: > First of all, few (if not all) versions of gen_leader implementations > has been gathered on github here: > http://github.com/KirinDave/gen_leader_revival. > As README says, you should use the 'combined_version'. > > A good starting point with tutorial is located at: > > http://order1.blogspot.com/2007/10/scalable-data-structures-in-erlang-and.html > > However you should remember that current versions of gen_leader works in > a way that you have to statically define the list of nodes that can > become master in case of other nodes failure. Current implementations > however, are incapable of adding nodes that might claim master role in > future, at runtime. This makes gen_leader incapable of working in > unstable environments when all of master nodes might go down for some > reason. > > On 09/23/2010 12:15 AM, Juan Jose Comellas wrote: > > Can anyone who's ever used or (even better!) written one of the many > > gen_leader incarnations point me to some documentation/examples on how to > > use it? I have a scenario where I have to keep a master process (to be > > chosen randomly or according to load) with slaves that takeover when the > > master goes down. I'm not sure if gen_leader is the correct solution, as > the > > master communicates to an external service and there will be more than > one. > > Both the master and the slave have open sockets to the external service > and > > the slave socket is "activated" as soon as the master goes down. > > > > I'd appreciate any pointers anybody might give me. > > > > Thanks, > > > > Juanjo > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From juanjo@REDACTED Thu Sep 23 15:09:21 2010 From: juanjo@REDACTED (Juan Jose Comellas) Date: Thu, 23 Sep 2010 10:09:21 -0300 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: <20100922230202.GG10050@hijacked.us> References: <4C9A88BB.5050002@gmail.com> <20100922230202.GG10050@hijacked.us> Message-ID: Thanks, I'll definitely look at that version, as adding nodes during runtime would be a big plus. Are you already using this version in OpenACD? I need a master/slave setup so that one node can take over another node's mod_event_socket connection to FreeSWITCH when we have to take out the original node or when it crashes. On Wed, Sep 22, 2010 at 8:02 PM, Andrew Thompson wrote: > On Thu, Sep 23, 2010 at 12:52:43AM +0200, Piotr Kaleta wrote: > > First of all, few (if not all) versions of gen_leader implementations > > has been gathered on github here: > > http://github.com/KirinDave/gen_leader_revival. > > As README says, you should use the 'combined_version'. > > > > A good starting point with tutorial is located at: > > > http://order1.blogspot.com/2007/10/scalable-data-structures-in-erlang-and.html > > > > However you should remember that current versions of gen_leader works in > > a way that you have to statically define the list of nodes that can > > become master in case of other nodes failure. Current implementations > > however, are incapable of adding nodes that might claim master role in > > future, at runtime. This makes gen_leader incapable of working in > > unstable environments when all of master nodes might go down for some > > reason. > > > > KirinDave's fork might lack this functionality, but abecciu's fork > supports adding candidates at runtime (as well as some additional fixes > that Dave hasn't merged). We've also cleaned gen_leader up to be an > actual erlang project (rebar/makefile, appfile, etc). > > Andrew > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From erlang@REDACTED Thu Sep 23 15:14:05 2010 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 23 Sep 2010 15:14:05 +0200 Subject: [erlang-questions] upnp device discovery In-Reply-To: References: Message-ID: I came to the same conclusion - pretty easy really - now I just have to eat up the xml and figure out what it means. /Joe On Wed, Sep 22, 2010 at 3:14 PM, Alexander Harju wrote: >> Has anybody got some erlang code for upnp device disovery? >> >> /Joe >> > > Hi. > I was just experimenting with last weekend. > > For discovery you just have to send the discovery message to multicast > address 239.255.255.250:1900 > > -define(UPNP_DISCOVER, > ? ?"M-SEARCH * HTTP/1.1\r\n" > ? ?"Host: 239.255.255.250:1900\r\n" > ? ?"Man: \"ssdp:discover\"\r\n" > ? ?"ST: upnp:rootdevice\r\n" > ? ?"MX: 3\r\n\r\n"). > > > {ok, S} = gen_udp:open(1900, [{reuseaddr,true}]), > > gen_udp:send(S, {239,255,255,250}, 1900, ?UPNP_DISCOVER), > receive > ? ?{udp, S, _, _, Msg0} -> > ? ? ? ?[_|Msg] = lists:reverse(Msg0), > ? ? ? ?io:format("~s~n", [lists:reverse(Msg)]) > ? ?after 5000 -> > ? ? ? ? ? ok > end. > > The rest is "just" handling the UPnP massive SOAP protocol. xmerl and inets > or yaws, erlsom. > > > You can also listen to UPnP multicasts like this: > > Opts = [{multicast_loop,false}, > ? ? ? ? ? ? ?{multicast_if,{0,0,0,0}}, > ? ? ? ? ? ? ?{multicast_ttl,4}], > > {ok, S} = gen_udp**:open(1900, Opts), > inet:setopts(S, [{add_membership, {Ip, {0,0,0,0}}}]), > > receive > ? ?{udp, S, ... > > BR > // Alex > From ebegumisa@REDACTED Thu Sep 23 15:19:44 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 23 Sep 2010 23:19:44 +1000 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: <56566.1285188216@snookles.snookles.com> References: <56566.1285188216@snookles.snookles.com> Message-ID: Thanks Scott, I was trying to figure out how to do that... documentation links I was referred to are a bit scattered at the moment. I'm sure when they get round to working on them the Gemini UBF tools will spread. - Edmond - On Thu, 23 Sep 2010 06:43:36 +1000, Scott Lystig Fritchie wrote: > Edmond Begumisa wrote: > >>> Having parsed the JSON (or UBF(A) ) the parse trees would be >>> essentially the same thing so the contract checker would be easy. > > eb> Ditto! It's actually that blog entry that got me to look at UBF > eb> (what ever happened to part II of that BTW!?!) I've been > eb> contemplating using Gemini's UBF-JSON to make life easier on the > eb> XULRunner side. I don't know if you've had a look at/recommend their > eb> implementation... > > eb> http://github.com/norton/ubf-jsonrpc > > Yes, that's what the ubf-jsonrpc package does. There's also the option > of using the http://github.com/norton/ubf stuff as-is to implement a > "JSF" server, which is straight JSON across a TCP socket (i.e., without > any JSON-RPC HTTP stuff). > > IIRC, it's just a server-side configuration option {proto, ubf} or > {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), > "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. > > -Scott > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From stephen.cuzzort@REDACTED Thu Sep 23 16:31:53 2010 From: stephen.cuzzort@REDACTED (Stephen Cuzzort) Date: Thu, 23 Sep 2010 10:31:53 -0400 Subject: undefined functions when using custom NIF library Message-ID: I'm trying to create a custom NIF library to handle some number crunching, but I'm having trouble when I load the nif. I call erlang:load_nif("./nifmath", 0), which fails - it says enif_get_string is undefined. I'm using gcc with Ubuntu 10.4, and have added liberts to the library list and included the erlang include directory. Steve From sverker@REDACTED Thu Sep 23 17:14:47 2010 From: sverker@REDACTED (Sverker Eriksson) Date: Thu, 23 Sep 2010 17:14:47 +0200 Subject: [erlang-questions] undefined functions when using custom NIF library In-Reply-To: References: Message-ID: <4C9B6EE7.8080402@erix.ericsson.se> You don't need liberts or any other library (unless you use them yourself). All enif_* symbols are defined within the emulator executable binary (beam or beam.smp) and should be resolved when your NIF library is loaded. I guess you are not building your library correctly. This usually works: gcc -Wall -fPIC -shared -o niftest.so niftest.c -I $ERL_ROOT/usr/include/ /Sverker, Erlang/OTP Stephen Cuzzort wrote: > I'm trying to create a custom NIF library to handle some number crunching, > but I'm having trouble when I load the nif. I call > erlang:load_nif("./nifmath", 0), which fails - it says enif_get_string is > undefined. I'm using gcc with Ubuntu 10.4, and have added liberts to the > library list and included the erlang include directory. > > Steve > > From jesper.louis.andersen@REDACTED Thu Sep 23 19:43:38 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 23 Sep 2010 19:43:38 +0200 Subject: [erlang-questions] upnp device discovery In-Reply-To: References: Message-ID: On Thu, Sep 23, 2010 at 3:14 PM, Joe Armstrong wrote: > I came to the same conclusion - pretty easy really - now I just have > to eat up the xml and figure out what it means. SOAP is a creation from the minions in hell. I'd just cheat and act as if I could speak the SOAP protocol without making a full implementation. Enterprise at its worst. -- J. From hynek@REDACTED Fri Sep 24 09:23:08 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Fri, 24 Sep 2010 09:23:08 +0200 Subject: [erlang-questions] Interactive Shell & unicode In-Reply-To: <1285237095.9813.14.camel@oblivion> References: <1285237095.9813.14.camel@oblivion> Message-ID: On Thu, Sep 23, 2010 at 12:18 PM, Timo Lindemann wrote: > Hi, > > it seems that the erlang shell is somehow affecting Unicode output in > the following situation: > > -module(test). > -compile(export_all). > > start() -> > ? ?ets:new(tbl, [named_table, set]), > ? ?ets:insert(tbl, {1, "?"}), > ? ?io:format("~p~n", ets:tab2list(tbl)). > > (I don't know if that only happens with ets or dets tables, however.) > > If I do >> erl -run test -run init stop > then the output is garbled: >> {1,"??"} > > but if I disable the shell, the output is correct: >> erl -run test -run init stop -noshell >> {1,"?"} > > This is Erlang 64-bit on Linux, btw. On my colleague's Powerbook G4, > both commands produce the output >> {1,[195,131,194,164]} > > What am I doing wrong? Not that it overly impedes my work, but I spent > hours, yesterday, in trying to produce right string output that > apparently looked correct to everyone else. Any ideas? > > Greetings, > -T. There is not "?" in you test module but "??". There is not utf8 support for module source code. -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From alexander.harju@REDACTED Fri Sep 24 11:53:52 2010 From: alexander.harju@REDACTED (Alexander Harju) Date: Fri, 24 Sep 2010 11:53:52 +0200 Subject: [erlang-questions] upnp device discovery In-Reply-To: References: Message-ID: On Thu, Sep 23, 2010 at 7:43 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Thu, Sep 23, 2010 at 3:14 PM, Joe Armstrong wrote: > > I came to the same conclusion - pretty easy really - now I just have > > to eat up the xml and figure out what it means. > > SOAP is a creation from the minions in hell. I'd just cheat and act as > if I could speak the SOAP protocol without making a full > implementation. Enterprise at its worst. > > I couldn't agree more :) XML is neither human or computer readable. Programmers nightmare no. 1 > -- > J. > From zabrane3@REDACTED Fri Sep 24 12:55:27 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Fri, 24 Sep 2010 12:55:27 +0200 Subject: ERES new releases? Message-ID: Hi Corrado, ERES's source code link is broken: http://www.diit.unict.it/users/csanto/eres-0.0.2.alpha.zip Is there a new release out there? -- Regards Zabrane From johanmon@REDACTED Fri Sep 24 14:20:58 2010 From: johanmon@REDACTED (Johan Montelius) Date: Fri, 24 Sep 2010 14:20:58 +0200 Subject: [erlang-questions] gen_leader usage/pointers In-Reply-To: References: Message-ID: Hi, I played around with leader election and atomic multicast last spring and you will find some ideas of how things can be done and what to look out for. You can find it here: http://web.it.kth.se/~johanmon/tibidabo.html Johan On Thu, 23 Sep 2010 00:15:02 +0200, Juan Jose Comellas wrote: > Can anyone who's ever used or (even better!) written one of the many > gen_leader incarnations point me to some documentation/examples on how to > use it? I have a scenario where I have to keep a master process (to be > chosen randomly or according to load) with slaves that takeover when the > master goes down. I'm not sure if gen_leader is the correct solution, as > the > master communicates to an external service and there will be more than > one. > Both the master and the slave have open sockets to the external service > and > the slave socket is "activated" as soon as the master goes down. > > I'd appreciate any pointers anybody might give me. > > Thanks, > > Juanjo -- Associate Professor Dr. Johan Montelius KTH - Royal Institute of Technology Stockholm, Sweden From steven.charles.davis@REDACTED Fri Sep 24 15:41:55 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 24 Sep 2010 06:41:55 -0700 (PDT) Subject: upnp device discovery In-Reply-To: References: Message-ID: I believe that XML was originally invented at Microsoft (as a simplified form of SGML). You have to gasp in despair at how this simplified document markup language has became used for protocol definitions, and even worse as a wire transfer format; Tasks for which it is uniquely unsuited. From k.rock@REDACTED Fri Sep 24 18:36:44 2010 From: k.rock@REDACTED (k.rock@REDACTED) Date: Fri, 24 Sep 2010 18:36:44 +0200 Subject: Compiling Erlang Interface with VC++ 2010 - LIB Issues Message-ID: <000301cb5c06$addf68d0$099e3a70$@rock-technologies.com> Dear Erlang Questions, When compiling an Erlang Interface with VC++ 2010 I get the following compiler errors. 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol __imp__gethostbyname@REDACTED referenced in function _ei_gethostbyname 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol __imp__gethostbyaddr@REDACTED referenced in function _ei_gethostbyaddr 1>D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\sqlite_port.exe : fatal error LNK1120: 2 unresolved externals Path set correctly to C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\lib\ei.lib C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\include\ei.h What is going wrong here. Thank you. Rock InformationTechnologies (India) Pvt. Dipl.-Ing.(FH) Klaus Rock Bonhoefferstrasse 37 73432 Aalen - Germany Tel : ++49-7367-96500 Mail : k.rock@REDACTED From mikkelgj@REDACTED Fri Sep 24 18:40:53 2010 From: mikkelgj@REDACTED (Mikkel Jensen) Date: Fri, 24 Sep 2010 18:40:53 +0200 Subject: Common test issues - Possible bug? Message-ID: Hi list! I have been working on setting up a test case using Erlang Common Test. The test is pretty simple, it consists of a sequence of calls to ct_ssh:exec(). Running the test once works. Running the test a few times (using repeat), usually works. But if i let the test suite repeat a large number of times, it fails after successfully running the test multiple times (In my last attempt it completed 42 runs, no joke, but it seems like the repeated runs can stop at any time). The error i'm getting is: It is not possible to install CT while running in interactive mode. To exit this mode, run ct:stop_interactive(). To enter the interactive mode again, run ct:start_interactive() Does anyone know how i can avoid this, allowing me to run a large number of repeated tests? On a sidenote: I have noticed in the logs that ct_ssh will sometimes have lost its connection when trying to call ct_ssh:exec(), and thus will attempt to reconnect. This attempt will, however, always fail instantly. A workaround i have made is to manually connect on each call i make to ct_ssh:exec(). This seems less than ideal, is there a better way? Kind regards and thanks in advance, Mikkel From aj@REDACTED Fri Sep 24 23:21:34 2010 From: aj@REDACTED (AJ Heller) Date: Fri, 24 Sep 2010 14:21:34 -0700 Subject: Bundling an erlang/OTP runtime system with custom applications into standalone executables for multiple platforms Message-ID: I'm interested in being able to write and distribute standalone erlang applications (i.e. a minimal erlang/OTP runtime system would be bundled in with the application; end-users would not have to download and install an erlang/OTP distribution on their own). Has anyone successfully built and distributed erlang applications with all the required erlang/OTP software bundled in? Similarly, are there any documented/tested/supported ways of doing it? I found Joe Armstrong's standalone erlang for R9B-0. I didn't find any related projects (or processes for creating multi-platform standalone distributions) for R14B, or any of the R13 releases for that matter. Has anyone done work in this area? Thank you much, --aj From devangana@REDACTED Fri Sep 24 23:30:04 2010 From: devangana@REDACTED (Devangana Tarafdar) Date: Fri, 24 Sep 2010 16:30:04 -0500 Subject: erlang +R emulator flag question Message-ID: Hello, I had a confusion regarding the erlang emulator flag +R as given in the erlang man page for "erl". I am attempting to connect to a node that is running an older version of erlang, from an erlang node that I know is running Release R11B. As regards the version of the older erlang node, I am giving the results of the commands that I ran to find the version but I am still not quite clear what the version is. 1. Old node snapshot: $ erl -name X -setcookie mycookie Erlang (BEAM) emulator version 2002.03.23 [source] Eshell V2002.03.23 (abort with ^G) (X@REDACTED)1> erlang:system_info(version). "2002.03.23" (X@REDACTED)2> init:script_id(). {"OTP APN 181 01","P9"} (X@REDACTED)3> erlang:system_info(otp_release). =ERROR REPORT==== 24-Sep-2010::15:51:20 === Error in process <0.29.0> on node 'X@REDACTED' with exit value: {badarg,[{erlang,system_info,[otp_release]},{erl_eval,expr,3},{erl_eval,exprs,4},{shell,eval_loop,2}]} ** exited: {badarg,[{erlang,system_info,[otp_release]}, {erl_eval,expr,3}, {erl_eval,exprs,4}, {shell,eval_loop,2}]} ** (X@REDACTED)4> 2. Similar commands run from the new node; erl -name X -setcookie mycookie Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [kernel-poll:false] Eshell V5.5.5 (abort with ^G) (X@REDACTED)2> erlang:system_info(version). "5.5.5" (X@REDACTED)3> erlang:system_info(otp_release). "R11B" (X@REDACTED)4> init:script_id(). {"OTP APN 181 01","R11B"} My question is as follows: I attempted to successively connect to the older node by using the emulator flags +R 9, +R 8, +R 7 in turn . and I could successfully get a "pong" reply to the command net_adm:ping('X@REDACTED') on the newer node in all the 3 cases. The documentation states that "The release number must be in the range 7... This limits the emulator, making it possible for it to communicate with Erlang nodes (as well as C- and Java nodes) running that earlier release." but maybe I am missing something here. I am wondering which flag to use "safely". Any suggestions / corrections would be greatly appreciated. Regards, Devangana From ulrich.moritz@REDACTED Sat Sep 25 01:45:31 2010 From: ulrich.moritz@REDACTED (Moritz Ulrich) Date: Sat, 25 Sep 2010 01:45:31 +0200 Subject: [erlang-questions] Bundling an erlang/OTP runtime system with custom applications into standalone executables for multiple platforms In-Reply-To: References: Message-ID: Nitrogen (1) does this: If you download a prebuilt binary from the page, you get a complete stand-alone package with Erlang and everything you need to run a webpage with Nitrogen in one directory. (1) http://nitrogenproject.com/ On Fri, Sep 24, 2010 at 11:21 PM, AJ Heller wrote: > I'm interested in being able to write and distribute standalone erlang > applications (i.e. a minimal erlang/OTP runtime system would be > bundled in with the application; end-users would not have to download > and install an erlang/OTP distribution on their own). Has anyone > successfully built and distributed erlang applications with all the > required erlang/OTP software bundled in? Similarly, are there any > documented/tested/supported ways of doing it? > > I found Joe Armstrong's standalone erlang > for R9B-0. I didn't find any > related projects (or processes for creating multi-platform standalone > distributions) for R14B, or any of the R13 releases for that matter. > Has anyone done work in this area? > > Thank you much, > --aj > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Moritz Ulrich Programmer, Student, Almost normal Guy http://www.google.com/profiles/ulrich.moritz BB5F086F-C798-41D5-B742-494C1E9677E8 From klacke@REDACTED Sat Sep 25 02:09:54 2010 From: klacke@REDACTED (Claes Wikstrom) Date: Sat, 25 Sep 2010 02:09:54 +0200 Subject: [erlang-questions] mnesia web interface In-Reply-To: <57146.172.29.10.158.1284924448.squirrel@squirrelmail.vail> References: <57146.172.29.10.158.1284924448.squirrel@squirrelmail.vail> Message-ID: <4C9D3DD2.7020003@hyber.org> On 09/19/2010 09:27 PM, Rick Pettit wrote: > On Sun, September 19, 2010 12:44 pm, Joe Armstrong wrote: >> Is there a web interface to mnesia? > > I haven't had a chance to use it yet, but there's a Yaws appmod named > "ymnesia" which might be worth checking out. That's the one. It's exactly a webui towards mnesia, but if I recall correctly, it may need some tweaking ... or more, /klacke From klacke@REDACTED Sat Sep 25 02:20:54 2010 From: klacke@REDACTED (Claes Wikstrom) Date: Sat, 25 Sep 2010 02:20:54 +0200 Subject: [erlang-questions] node to node message passing In-Reply-To: <4C8E7E2E.708@amberbio.com> References: <4C8CB006.6070908@amberbio.com> <201009121951.20575.jh@sotun.de> <4C8E7E2E.708@amberbio.com> Message-ID: <4C9D4066.6020004@hyber.org> On 09/13/2010 09:40 PM, Morten Krogh wrote: > Hi Jan > > Thanks for your answer. I guess we really disagree on this:) > > I was stunned when I saw the nodes disconnecting in the middle of a large message passing, If this really is the case - I doubt - its a bug. /klacke From anders.wei@REDACTED Sat Sep 25 05:15:33 2010 From: anders.wei@REDACTED (Anders Wei) Date: Sat, 25 Sep 2010 11:15:33 +0800 Subject: [erlang-questions] Bundling an erlang/OTP runtime system with custom applications into standalone executables for multiple platforms In-Reply-To: References: Message-ID: Hi Check this: http://www.erlang.org/doc/man/systools.html -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of AJ Heller Sent: 2010?9?25? 5:22 To: Erlang Questions Subject: [erlang-questions] Bundling an erlang/OTP runtime system with custom applications into standalone executables for multiple platforms I'm interested in being able to write and distribute standalone erlang applications (i.e. a minimal erlang/OTP runtime system would be bundled in with the application; end-users would not have to download and install an erlang/OTP distribution on their own). Has anyone successfully built and distributed erlang applications with all the required erlang/OTP software bundled in? Similarly, are there any documented/tested/supported ways of doing it? I found Joe Armstrong's standalone erlang for R9B-0. I didn't find any related projects (or processes for creating multi-platform standalone distributions) for R14B, or any of the R13 releases for that matter. Has anyone done work in this area? Thank you much, --aj ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From chandrashekhar.mullaparthi@REDACTED Sat Sep 25 06:52:28 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Sat, 25 Sep 2010 05:52:28 +0100 Subject: ibrowse-2.0.1 (was Re: ANNOUNCE: ibrowse-2.0.0) Message-ID: As Joe kindly pointed out, I didn't mention in my previous announcement, what ibrowse was and where to get it from. So here is a proper announcement :-) ibrowse is a HTTP client written in erlang, an alternative to the built in HTTP client implementation in OTP. Get it from: http://github.com/cmullaparthi/ibrowse/ Here is an extract from the README: 24-09-2010 - v2.0.1 * Removed a spurious io:format statement Chandru From kenneth.lundin@REDACTED Sat Sep 25 08:51:42 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Sat, 25 Sep 2010 08:51:42 +0200 Subject: [erlang-questions] Bundling an erlang/OTP runtime system with custom applications into standalone executables for multiple platforms In-Reply-To: References: Message-ID: Reltool (in the distribution) supports the creation of standalone target systems just like you describe. http://www.erlang.org/doc/apps/reltool/index.html Reltool is quite new and created among other thing to support the scenario you describe. /Kenneth , Erlang/OTP Ericsson On Fri, Sep 24, 2010 at 11:21 PM, AJ Heller wrote: > I'm interested in being able to write and distribute standalone erlang > applications (i.e. a minimal erlang/OTP runtime system would be > bundled in with the application; end-users would not have to download > and install an erlang/OTP distribution on their own). Has anyone > successfully built and distributed erlang applications with all the > required erlang/OTP software bundled in? Similarly, are there any > documented/tested/supported ways of doing it? > > I found Joe Armstrong's standalone erlang > for R9B-0. I didn't find any > related projects (or processes for creating multi-platform standalone > distributions) for R14B, or any of the R13 releases for that matter. > Has anyone done work in this area? > > Thank you much, > --aj > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From csanto@REDACTED Sat Sep 25 08:56:16 2010 From: csanto@REDACTED (Corrado Santoro) Date: Sat, 25 Sep 2010 08:56:16 +0200 Subject: [erlang-questions] ERES new releases? In-Reply-To: References: Message-ID: Please find the new release at: http://eresye.sourceforge.net/ All the best, --Corrado > Hi Corrado, > > ERES's source code link is broken: > http://www.diit.unict.it/users/csanto/eres-0.0.2.alpha.zip > > Is there a new release out there? > > -- > Regards > Zabrane > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From boris.muehmer@REDACTED Sat Sep 25 10:25:15 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Sat, 25 Sep 2010 10:25:15 +0200 Subject: How to analyse why (w)erl.exe is not starting? Message-ID: I wanted to try out the R14B release on my (hosted) windows 2003 SP2 server. But when I try to start (w)erl using the icon or via command line, the process shows up in the task list and vanishes again after just a few secs, without any window popping up... I reinstalled it after a reboot => no change. I uninstalled R14B and installed R13B03 and R13B04: no problems at all. How can I check what is going wrong? There are no entires in the eventlog. Also there is no output when started from the command line. Thanks in advance - boris From zabrane3@REDACTED Sat Sep 25 11:04:19 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Sat, 25 Sep 2010 11:04:19 +0200 Subject: [erlang-questions] ERES new releases? In-Reply-To: References: Message-ID: Thanks guys. Exactly what I was looking after. -- Regards Zabrane 2010/9/25 Corrado Santoro : > Please find the new release at: > > http://eresye.sourceforge.net/ > > All the best, > --Corrado > > >> Hi Corrado, >> >> ERES's source code link is broken: >> http://www.diit.unict.it/users/csanto/eres-0.0.2.alpha.zip >> >> Is there a new release out there? >> >> -- >> Regards >> Zabrane >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > From zabrane3@REDACTED Sat Sep 25 11:11:08 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Sat, 25 Sep 2010 11:11:08 +0200 Subject: [erlang-questions] ERES new releases? In-Reply-To: References: Message-ID: Hi Corrado, Is there a way to get a version of ERESYE with an Apache2, MIT, or BSD license? -- Regards Zabrane 2010/9/25 zabrane Mikael : > Thanks guys. > Exactly what I was looking after. > > -- > Regards > Zabrane > > > 2010/9/25 Corrado Santoro : >> Please find the new release at: >> >> http://eresye.sourceforge.net/ >> >> All the best, >> --Corrado >> >> >>> Hi Corrado, >>> >>> ERES's source code link is broken: >>> http://www.diit.unict.it/users/csanto/eres-0.0.2.alpha.zip >>> >>> Is there a new release out there? >>> >>> -- >>> Regards >>> Zabrane >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >>> >> >> >> > From erlang@REDACTED Sat Sep 25 12:39:43 2010 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 25 Sep 2010 12:39:43 +0200 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: References: <56566.1285188216@snookles.snookles.com> Message-ID: I've been thinking ... It would be really great if I could send messages from a browser to erlang. I want to add an extra button to firefox that when pressed analyzes something about the current page and sends a message to an erlang server. I was reading about solvent http://simile.mit.edu/wiki/Solvent some quotes " Interactively highlight parts of the page you wish to scrape, directly in your browser, and obtain the right XPaths for them Edit and execute the scraper code directly in the browser, making the development cycle fast and incremental Save and publish the scraper with the required metadata, so that others can discover it ..." This is *exacty* what I want to do - highlight some text in the brower - as I release the mouse the highlighted text is sent to and erlang server to be analysed and stored. IMHO bookmarking a site is not interesting - it's some small fragment on a page that interest me - I want to: a) highlight it b) edit the highlighted bit (ie throw me into an editor) c) store the edited result (later I might even be able to "pre-read" a new page finding what might be interesting for me in the page :-) This is very generic - if I could isolate "analyze something about the current page" into a convenient js module then I could do all sort of fun things. This seems to require a firefox plugin - any idea how to write this? If I want to start a collaborative project where do I advertise for smart javascript programmers who know the firefox internals. The erlang stuff is easy :-) but the js is tricky - (and I guess this group is not the best lace to ask js questions?) (( where is the best group for this (ie js questions))) On Thu, Sep 23, 2010 at 3:19 PM, Edmond Begumisa wrote: > Thanks Scott, I was trying to figure out how to do that... documentation > links I was referred to are a bit scattered at the moment. I'm sure when > they get round to working on them the Gemini UBF tools will spread. > > - Edmond - > > On Thu, 23 Sep 2010 06:43:36 +1000, Scott Lystig Fritchie > wrote: > >> Edmond Begumisa wrote: >> >>>> Having parsed the JSON (or UBF(A) ) the parse trees would be >>>> essentially the same thing so the contract checker would be easy. >> >> eb> Ditto! It's actually that blog entry that got me to look at UBF >> eb> (what ever happened to part II of that BTW!?!) I've been >> eb> contemplating using Gemini's UBF-JSON to make life easier on the >> eb> XULRunner side. I don't know if you've had a look at/recommend their >> eb> implementation... >> >> eb> http://github.com/norton/ubf-jsonrpc >> >> Yes, that's what the ubf-jsonrpc package does. ?There's also the option >> of using the http://github.com/norton/ubf stuff as-is to implement a >> "JSF" server, which is straight JSON across a TCP socket (i.e., without >> any JSON-RPC HTTP stuff). >> >> IIRC, it's just a server-side configuration option {proto, ubf} or >> {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), >> "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. >> >> -Scott >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > From klacke@REDACTED Sat Sep 25 14:58:55 2010 From: klacke@REDACTED (Claes Wikstrom) Date: Sat, 25 Sep 2010 14:58:55 +0200 Subject: Yaws 1.89 Message-ID: <4C9DF20F.3010007@hyber.org> New release of Yaws called 1.89 is now available. Code, docs and relnotes as usual at http://yaws.hyber.org Enjoy, /klacke From bogus@does.not.exist.com Sat Sep 25 17:49:08 2010 From: bogus@does.not.exist.com () Date: Sun, 26 Sep 2010 01:49:08 +1000 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: References: <56566.1285188216@snookles.snookles.com> Message-ID: <2a5997$5chvk1@smtp05.syd.iprimus.net.au> Dr Armstrong, Some comments... On Sat, 25 Sep 2010 20:39:43 +1000, Joe Armstrong wrote: > I've been thinking ... > > It would be really great if I could send messages from a browser > to erlang. > > I want to add an extra button to firefox that when pressed > analyzes something about the current page and sends a message > to an erlang server. > > I was reading about solvent > > http://simile.mit.edu/wiki/Solvent > > some quotes > > " Interactively highlight parts of the page you wish to scrape, > directly in your browser, and obtain the right XPaths for them > > Edit and execute the scraper code directly in the browser, making > the development cycle fast and incremental > > Save and publish the scraper with the required metadata, so that > others can discover it > > ..." > > This is *exacty* what I want to do - highlight some text in the > brower - as I release the mouse the highlighted text is sent > to and erlang server to be analysed and stored. > > IMHO bookmarking a site is not interesting - it's some small fragment > on a page that interest me - I want to: > > a) highlight it > b) edit the highlighted bit (ie throw me into an editor) > c) store the edited result > > (later I might even be able to "pre-read" a new page finding > what might be interesting for me in the page :-) This sounds very interesting. If I've understood correctly, you want the results delivered to you on the Erlang side as HTML snippets or JSON with embedded HTML strings? Also, where you say "throw me into an editor" -- do you mean a editor in the word-processing sense or an editor in the html authoring sense (like in Solvent?) The former would be easy (Mozilla have that infrastructure already), the latter substantially more involved. > This is very generic - if I could isolate "analyze something about the > current page" into a convenient js module then I could do all sort of > fun things. > > This seems to require a Firefox plugin - any idea how to write this? Yes and No. No because I've never actually written a Firefox extension/add-on before. Yes because writing one is basically a subset/specialised form of a XULRunner/Mozilla Framework application which I do have knowledge of. If you want to have a crack at it yourself (you know the saying "if you want something done properly..."), you probably want to start with the XUL tutorial, then the Extension tutorial, then as a reference the official free e-book "Creating Applications with Mozilla" which goes through the Mozilla framework... https://developer.mozilla.org/en/XUL_Tutorial http://kb.mozillazine.org/Getting_started_with_extension_development https://developer.mozilla.org/en/Setting_up_extension_development_environment https://developer.mozilla.org/en/Building_an_Extension http://books.mozdev.org/html/index.html You might need the js debugger too... http://www.mozilla.org/projects/venkman/venkman-walkthrough.html > If I want to start a collaborative project where do I advertise for > smart javascript programmers who know the firefox internals. > I'm not sure where you'd advertise (Stackoverflow maybe), but I can advise on what to advertise for... You might want to specifically advertise for a programmer familiar with the Mozilla Application Framework -- of which JavaScript is just an element (XUL/XBL/XPCOM/RDF-XML being the major other bits.) This will weed out the hobbyists. Another thing to look out for is the fact that the Mozilla flavor of JavaScript is not quite the same as the typical JavaScript. That is, the js used in web-pages is "narrower" than the js used in Firefox extensions/XULRunner apps -- those familiar with the former (prototype/jquery types) are unlikely to be useful with the latter. For example, extension writers frequently access Mozilla C++ interfaces via js and navigate a different DOM (the internals you refer to.) Their code looks alien to web js programmers. So, familiarity with ns (netscape) interfaces and the Firefox/XUL DOM is something else you should specifically ask for. IMO, rather than advertise, your best bet is to look for a Firefox extension that has some characteristics of what you're looking for, then contact the author. You can browse/search for extensions here... https://addons.mozilla.org/en-US/firefox/ > The erlang stuff is easy :-) but the js is tricky - (and I guess this > group is > not the best lace to ask js questions?) > > (( where is the best group for this (ie js questions))) > The Mozilla add-on forum is where you'll find Firefox extension types who speak "Mozilla" JavaScript... https://developer.mozilla.org/forums/ucp.php?mode=login That said, I'm going to put a day aside and try to whip something up myself (assuming I've understood what you want.) I'm very interested in being part of this project -- especially to learn more about Erlang. It's also a good excuse to make the Mozilla Framework -> Firefox Extension migration I've been meaning to do. - Edmond - > > > > On Thu, Sep 23, 2010 at 3:19 PM, Edmond Begumisa > wrote: >> Thanks Scott, I was trying to figure out how to do that... documentation >> links I was referred to are a bit scattered at the moment. I'm sure when >> they get round to working on them the Gemini UBF tools will spread. >> >> - Edmond - >> >> On Thu, 23 Sep 2010 06:43:36 +1000, Scott Lystig Fritchie >> wrote: >> >>> Edmond Begumisa wrote: >>> >>>>> Having parsed the JSON (or UBF(A) ) the parse trees would be >>>>> essentially the same thing so the contract checker would be easy. >>> >>> eb> Ditto! It's actually that blog entry that got me to look at UBF >>> eb> (what ever happened to part II of that BTW!?!) I've been >>> eb> contemplating using Gemini's UBF-JSON to make life easier on the >>> eb> XULRunner side. I don't know if you've had a look at/recommend >>> their >>> eb> implementation... >>> >>> eb> http://github.com/norton/ubf-jsonrpc >>> >>> Yes, that's what the ubf-jsonrpc package does. There's also the option >>> of using the http://github.com/norton/ubf stuff as-is to implement a >>> "JSF" server, which is straight JSON across a TCP socket (i.e., without >>> any JSON-RPC HTTP stuff). >>> >>> IIRC, it's just a server-side configuration option {proto, ubf} or >>> {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), >>> "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. >>> >>> -Scott >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Sat Sep 25 18:12:57 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Sun, 26 Sep 2010 02:12:57 +1000 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: References: <56566.1285188216@snookles.snookles.com> Message-ID: Dr Armstrong, Some comments... On Sat, 25 Sep 2010 20:39:43 +1000, Joe Armstrong wrote: > I've been thinking ... > > It would be really great if I could send messages from a browser > to erlang. > > I want to add an extra button to firefox that when pressed > analyzes something about the current page and sends a message > to an erlang server. > > I was reading about solvent > > http://simile.mit.edu/wiki/Solvent > > some quotes > > " Interactively highlight parts of the page you wish to scrape, > directly in your browser, and obtain the right XPaths for them > > Edit and execute the scraper code directly in the browser, making > the development cycle fast and incremental > > Save and publish the scraper with the required metadata, so that > others can discover it > > ..." > > This is *exacty* what I want to do - highlight some text in the > brower - as I release the mouse the highlighted text is sent > to and erlang server to be analysed and stored. > > IMHO bookmarking a site is not interesting - it's some small fragment > on a page that interest me - I want to: > > a) highlight it > b) edit the highlighted bit (ie throw me into an editor) > c) store the edited result > > (later I might even be able to "pre-read" a new page finding > what might be interesting for me in the page :-) > This sounds very interesting. If I've understood correctly, you want the results delivered to you on the Erlang side as HTML snippets/JSON with embedded HTML strings? The clever parts will be on the Erlang side? Also, where you say "throw me into an editor" -- do you mean a editor in the word-processing sense or an editor in the html authoring sense (like in Solvent?) The former would be easy (Mozilla have that infrastructure already), the latter substantially more involved. > This is very generic - if I could isolate "analyze something about the > current page" into a convenient js module then I could do all sort of > fun things. > > This seems to require a firefox plugin Yes, you'd need to write a Firefox extension to do this. > - any idea how to write this? > Yes and No. No because I've never actually written a Firefox extension/add-on before. Yes because writing one is basically a subset/specialised form of a Mozilla Framework/XULRunner application which I do have experience with. If you want to have a crack at it yourself (you know the saying "if you want something done properly..."), you probably want to start with the XUL tutorial, then the extension tutorial, then as a reference the official free e-book "Creating Applications with Mozilla" which goes through the Mozilla Application Framework... https://developer.mozilla.org/en/XUL_Tutorial http://kb.mozillazine.org/Getting_started_with_extension_development https://developer.mozilla.org/en/Setting_up_extension_development_environment https://developer.mozilla.org/en/Building_an_Extension http://books.mozdev.org/html/index.html You might need the js debugger too... http://www.mozilla.org/projects/venkman/venkman-walkthrough.html > If I want to start a collaborative project where do I advertise for > smart javascript programmers who know the firefox internals. > I'm not sure where you'd advertise (Stackoverflow maybe), but I can advise on what skills to advertise for... You might want to specifically advertise for a programmer familiar with the Mozilla Application Framework -- of which JavaScript is just an element (XUL/XBL/XPCOM/RDF-XML being the major other bits.) This will weed out the hobbyists. Another thing to look out for is the fact that the Mozilla flavor of JavaScript is not quite the same as the typical JavaScript. That is, the js used in web-pages is "narrower" than the js used in Firefox extensions/XULRunner apps -- those familiar with the former (prototype/jquery types) are unlikely to be useful with the latter. For example, extension writers frequently access Mozilla C++ interfaces via js and navigate a different DOM (the internals you refer to.) Their code looks alien to web js programmers. So, familiarity with ns (netscape) interfaces and the Firefox/XUL DOM is something else you should specifically ask for. IMO, rather than advertise, your best bet is to look for a Firefox extension that has some characteristics of what you're looking for, then contact the author. You can browse/search for extensions here... https://addons.mozilla.org/en-US/firefox/ > The erlang stuff is easy :-) but the js is tricky - (and I guess this > group is > not the best lace to ask js questions?) > > (( where is the best group for this (ie js questions))) > > The Mozilla add-on forum is where you'll find Firefox extension types who speak "Mozilla JavaScript"... https://developer.mozilla.org/forums/ucp.php?mode=login That said, with your permission, I'd like to put a day aside and try to whip something up myself (assuming I've understood what you want.) I'm very interested in being part of this project -- especially to learn more about Erlang. It's also a good excuse to make the Mozilla Framework -> Firefox Extension migration I've been meaning to do. Is this something you intend on open-sourcing? - Edmond - > > > On Thu, Sep 23, 2010 at 3:19 PM, Edmond Begumisa > wrote: >> Thanks Scott, I was trying to figure out how to do that... documentation >> links I was referred to are a bit scattered at the moment. I'm sure when >> they get round to working on them the Gemini UBF tools will spread. >> >> - Edmond - >> >> On Thu, 23 Sep 2010 06:43:36 +1000, Scott Lystig Fritchie >> wrote: >> >>> Edmond Begumisa wrote: >>> >>>>> Having parsed the JSON (or UBF(A) ) the parse trees would be >>>>> essentially the same thing so the contract checker would be easy. >>> >>> eb> Ditto! It's actually that blog entry that got me to look at UBF >>> eb> (what ever happened to part II of that BTW!?!) I've been >>> eb> contemplating using Gemini's UBF-JSON to make life easier on the >>> eb> XULRunner side. I don't know if you've had a look at/recommend >>> their >>> eb> implementation... >>> >>> eb> http://github.com/norton/ubf-jsonrpc >>> >>> Yes, that's what the ubf-jsonrpc package does. There's also the option >>> of using the http://github.com/norton/ubf stuff as-is to implement a >>> "JSF" server, which is straight JSON across a TCP socket (i.e., without >>> any JSON-RPC HTTP stuff). >>> >>> IIRC, it's just a server-side configuration option {proto, ubf} or >>> {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), >>> "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. >>> >>> -Scott >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Sat Sep 25 18:40:25 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Sun, 26 Sep 2010 02:40:25 +1000 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm In-Reply-To: References: <56566.1285188216@snookles.snookles.com> Message-ID: My apologies, Ignore that first reply with a blank 'from' address -- that was an incomplete draft that my caffeinated fingers decided to send all on their own. - Edmond - On Sat, 25 Sep 2010 20:39:43 +1000, Joe Armstrong wrote: > I've been thinking ... > > It would be really great if I could send messages from a browser > to erlang. > > I want to add an extra button to firefox that when pressed > analyzes something about the current page and sends a message > to an erlang server. > > I was reading about solvent > > http://simile.mit.edu/wiki/Solvent > > some quotes > > " Interactively highlight parts of the page you wish to scrape, > directly in your browser, and obtain the right XPaths for them > > Edit and execute the scraper code directly in the browser, making > the development cycle fast and incremental > > Save and publish the scraper with the required metadata, so that > others can discover it > > ..." > > This is *exacty* what I want to do - highlight some text in the > brower - as I release the mouse the highlighted text is sent > to and erlang server to be analysed and stored. > > IMHO bookmarking a site is not interesting - it's some small fragment > on a page that interest me - I want to: > > a) highlight it > b) edit the highlighted bit (ie throw me into an editor) > c) store the edited result > > (later I might even be able to "pre-read" a new page finding > what might be interesting for me in the page :-) > > This is very generic - if I could isolate "analyze something about the > current page" into a convenient js module then I could do all sort of > fun things. > > This seems to require a firefox plugin - any idea how to write this? > > If I want to start a collaborative project where do I advertise for > smart javascript programmers who know the firefox internals. > > The erlang stuff is easy :-) but the js is tricky - (and I guess this > group is > not the best lace to ask js questions?) > > (( where is the best group for this (ie js questions))) > > > > > > On Thu, Sep 23, 2010 at 3:19 PM, Edmond Begumisa > wrote: >> Thanks Scott, I was trying to figure out how to do that... documentation >> links I was referred to are a bit scattered at the moment. I'm sure when >> they get round to working on them the Gemini UBF tools will spread. >> >> - Edmond - >> >> On Thu, 23 Sep 2010 06:43:36 +1000, Scott Lystig Fritchie >> wrote: >> >>> Edmond Begumisa wrote: >>> >>>>> Having parsed the JSON (or UBF(A) ) the parse trees would be >>>>> essentially the same thing so the contract checker would be easy. >>> >>> eb> Ditto! It's actually that blog entry that got me to look at UBF >>> eb> (what ever happened to part II of that BTW!?!) I've been >>> eb> contemplating using Gemini's UBF-JSON to make life easier on the >>> eb> XULRunner side. I don't know if you've had a look at/recommend >>> their >>> eb> implementation... >>> >>> eb> http://github.com/norton/ubf-jsonrpc >>> >>> Yes, that's what the ubf-jsonrpc package does. There's also the option >>> of using the http://github.com/norton/ubf stuff as-is to implement a >>> "JSF" server, which is straight JSON across a TCP socket (i.e., without >>> any JSON-RPC HTTP stuff). >>> >>> IIRC, it's just a server-side configuration option {proto, ubf} or >>> {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), >>> "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. >>> >>> -Scott >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From erlang@REDACTED Sat Sep 25 19:35:21 2010 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 25 Sep 2010 19:35:21 +0200 Subject: new release announcements ... please read Message-ID: Pleeeeeease start release announcements thusly: is a is available form Fill in and I've seen a releases of eres, yaws, ibrowse, announced in the last couple of days. They did not have the above information. I knew what *one* of these was. Now I know that the person posing the article knows what the thing is since they live breath and dream the code - but perhaps everybody else on the planet doesn't know - so please tell them. yaws - is a web server (I happened to know this) I didn't know what the other two were without googling You can't assume people who you've never met know what you're talking about. /Joe From ted.karmel@REDACTED Sat Sep 25 22:53:21 2010 From: ted.karmel@REDACTED (Ted Karmel) Date: Sat, 25 Sep 2010 20:53:21 +0000 Subject: [erlang-questions] UBF(A) vs ETF / UBF(C) vs gen_fsm Message-ID: Well though this is not the approriate forum for js, take a look at jetpack for easy firefox plugin scripting. Joe Armstrong wrote: > I've been thinking ... > >It would be really great if I could send messages from a browser >to erlang. > >I want to add an extra button to firefox that when pressed >analyzes something about the current page and sends a message >to an erlang server. > >I was reading about solvent > >http://simile.mit.edu/wiki/Solvent > >some quotes > >" Interactively highlight parts of the page you wish to scrape, >directly in your browser, and obtain the right XPaths for them > > Edit and execute the scraper code directly in the browser, making >the development cycle fast and incremental > > Save and publish the scraper with the required metadata, so that >others can discover it > > ..." > >This is *exacty* what I want to do - highlight some text in the >brower - as I release the mouse the highlighted text is sent >to and erlang server to be analysed and stored. > >IMHO bookmarking a site is not interesting - it's some small fragment >on a page that interest me - I want to: > > a) highlight it > b) edit the highlighted bit (ie throw me into an editor) > c) store the edited result > >(later I might even be able to "pre-read" a new page finding >what might be interesting for me in the page :-) > >This is very generic - if I could isolate "analyze something about the >current page" into a convenient js module then I could do all sort of >fun things. > >This seems to require a firefox plugin - any idea how to write this? > >If I want to start a collaborative project where do I advertise for >smart javascript programmers who know the firefox internals. > >The erlang stuff is easy :-) but the js is tricky - (and I guess this group is >not the best lace to ask js questions?) > >(( where is the best group for this (ie js questions))) > > > > > >On Thu, Sep 23, 2010 at 3:19 PM, Edmond Begumisa > wrote: >> Thanks Scott, I was trying to figure out how to do that... documentation >> links I was referred to are a bit scattered at the moment. I'm sure when >> they get round to working on them the Gemini UBF tools will spread. >> >> - Edmond - >> >> On Thu, 23 Sep 2010 06:43:36 +1000, Scott Lystig Fritchie >> wrote: >> >>> Edmond Begumisa wrote: >>> >>>>> Having parsed the JSON (or UBF(A) ) the parse trees would be >>>>> essentially the same thing so the contract checker would be easy. >>> >>> eb> Ditto! It's actually that blog entry that got me to look at UBF >>> eb> (what ever happened to part II of that BTW!?!) I've been >>> eb> contemplating using Gemini's UBF-JSON to make life easier on the >>> eb> XULRunner side. I don't know if you've had a look at/recommend their >>> eb> implementation... >>> >>> eb> http://github.com/norton/ubf-jsonrpc >>> >>> Yes, that's what the ubf-jsonrpc package does. ?There's also the option >>> of using the http://github.com/norton/ubf stuff as-is to implement a >>> "JSF" server, which is straight JSON across a TCP socket (i.e., without >>> any JSON-RPC HTTP stuff). >>> >>> IIRC, it's just a server-side configuration option {proto, ubf} or >>> {proto, jsf} or {proto, ebf} to have an Erlang server speak UBF(A), >>> "JSF", or UBF-terms-encoded-with-term_to_binary(), respectively. >>> >>> -Scott >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> > >________________________________________________________________ >erlang-questions (at) erlang.org mailing list. >See http://www.erlang.org/faq.html >To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From luismarianoguerra@REDACTED Sun Sep 26 06:57:49 2010 From: luismarianoguerra@REDACTED (Mariano Guerra) Date: Sun, 26 Sep 2010 01:57:49 -0300 Subject: erlang shell implementation question Message-ID: Hi, I'm trying to improve the implementation of efene's[0] shell. Right now it doesn't have readline-like capabilities, that is, the ability to edit the line before evaluating it, or having a history of previous commands. I was reading the shell.erl file and I couldn't find the place where the input is read and the features described above (plus shortcuts like ctrl+a) are implemented. can someone point me to the file and function where this is implemented? thanks! [0] http://github.com/marianoguerra/efene From luismarianoguerra@REDACTED Sun Sep 26 08:05:33 2010 From: luismarianoguerra@REDACTED (Mariano Guerra) Date: Sun, 26 Sep 2010 03:05:33 -0300 Subject: erlang shell implementation question In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 1:57 AM, Mariano Guerra wrote: > Hi, > > ?I'm trying to improve the implementation of efene's[0] shell. Right > now it doesn't have readline-like capabilities, that is, the ability > to edit the line before evaluating it, or having a history of previous > commands. > > I was reading the shell.erl file and I couldn't find the place where > the input is read and the features described above (plus shortcuts > like ctrl+a) are implemented. > > can someone point me to the file and function where this is implemented? I read a little more and now I'm looking at edlin.erl, group.erl and user_drv.erl, still can't find a simple way to ask for a character or something like that. http://github.com/erlang/otp/blob/dev/lib/stdlib/src/edlin.erl http://github.com/erlang/otp/blob/dev/lib/kernel/src/user_drv.erl http://github.com/erlang/otp/blob/dev/lib/kernel/src/group.erl From brooklynne@REDACTED Sun Sep 26 07:29:15 2010 From: brooklynne@REDACTED (brooklynne) Date: Sun, 26 Sep 2010 18:29:15 +1300 Subject: New here. Message-ID: <4C9EDA2B.4050102@brooklynnemichelle.com> I'm new here and learning Erlang via the website learnyousomeerlang.com It's also my first language (not for want of trying just the first one to actually make sense. Brooklynne From mazen.harake@REDACTED Sun Sep 26 08:33:25 2010 From: mazen.harake@REDACTED (Mazen Harake) Date: Sun, 26 Sep 2010 06:33:25 +0000 (GMT) Subject: [erlang-questions] new release announcements ... please read In-Reply-To: <77744776.29501285482580385.JavaMail.root@zimbra> Message-ID: <1855818699.29521285482805495.JavaMail.root@zimbra> I remember we discussed a list of "best practices" here on the mailing list at some point. This included how to tag announcements questions and discussions (such as [ANN] for announcements). Unfortunately I think this just died out but it would be really nice to have an Best Practices page perhaps by extending the FAQ[1] page which already exists? Personally it would be easier to find the topics one think are interested if this was in place. Any takers to start a thread and start creating these best practices? /M [1]: http://www.erlang.org/faq.html ----- Original Message ----- From: "Joe Armstrong" To: "Erlang" Sent: Saturday, 25 September, 2010 20:35:21 GMT +02:00 Harare / Pretoria Subject: [erlang-questions] new release announcements ... please read Pleeeeeease start release announcements thusly: ?? is a ?? is available form Fill in and I've seen a releases of eres, yaws, ibrowse, announced in the last couple of days. They did not have the above information. I knew what *one* of these was. Now I know that the person posing the article knows what the thing is since they live breath and dream the code - but perhaps everybody else on the planet doesn't know - so please tell them. yaws - is a web server (I happened to know this) I didn't know what the other two were without googling You can't assume people who you've never met know what you're talking about. /Joe ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From kchugalinskiy@REDACTED Sun Sep 26 10:56:37 2010 From: kchugalinskiy@REDACTED (Konstantin Chugalinskiy) Date: Sun, 26 Sep 2010 11:56:37 +0300 Subject: [erlang-questions] Installing on cygwin In-Reply-To: References: Message-ID: Hi, Giulio! First you must run configure script. Type ./configure ; make ; make install and everything will be fine :-) By the way, it is written in a file named "INSTALL.md". 2010/9/22 Giulio Petrucci > Hi there, > > this is my first post in this mailing list so, first of all I'd like > to say hello to everyone reading here. > I'm trying playing with the ReverseHTTP project > (http://github.com/tonyg/reversehttp/) and I need to run a web server > written in Erlang but... I'm a totally newbie with this language! :-) > > Under Linux (the OS I use mostly for personal stuff) I didn't have to > do anything special. Under Windows, which I use at work, I'm a little > bit in trouble. I'd like to install Erlang on my Cygwin installation. > I've just downloaded the source code... now what shall I do? I've > tried to get into the source directory and type "make" on my shell > but... nothing happened. :-) Could anyone help me? > > Thanks in advance, > Giulio > > -- > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- with best regards, Chugalinskiy Konstantin From gleber.p@REDACTED Sun Sep 26 15:03:45 2010 From: gleber.p@REDACTED (Gleb Peregud) Date: Sun, 26 Sep 2010 15:03:45 +0200 Subject: Rebar and EPM Message-ID: Hello Have authors of Rebar and/or EPM considered joining the efforts and including EPM into Rebar for dependencies handling and retrieval? Both projects share pretty the same ideology and it seems like a very good fit. Maybe there's a third-party effort for doing it. Have anyone approached this task? Best, Gleb Peregud From klacke@REDACTED Sun Sep 26 16:26:21 2010 From: klacke@REDACTED (Claes Wikstrom) Date: Sun, 26 Sep 2010 16:26:21 +0200 Subject: Work at Tail-f Message-ID: <4C9F580D.4030302@hyber.org> I'm looking for a master hacker that wants to work with me at Tail-f Systems. The official recruit URL is at http://www.tail-f.com/about-us/careers You'll be working together with a whole group of legendary Erlang hackers such as Martin Bj?rklund, Per Hedeland, Sebastian Stollo, Johan Bevemyr and .. well me. I'm looking for someone who is not only an Erlang hacker, but is also very good with Java. For the last 12 months I've been building a NETCONF (RFC 4741) manager and I need help now. The manager core is written in Erlang and it needs lots of more work. The user code is to be written in Java, and we're developing all kinds of interface libraries to facilitate the user code. You'd have to work in Stockholm, no remotes, but being able to speak Swedish is not a prerequisite. Reply to my tail-f address, klacke@REDACTED /klacke From dewa19@REDACTED Sun Sep 26 18:07:42 2010 From: dewa19@REDACTED (-sigit i.-) Date: Sun, 26 Sep 2010 18:07:42 +0200 Subject: Erlang user group in madrid? Message-ID: Hi all, Is anyone here have info on erlang user group in Madrid (their local maillist, email, etc)? I appreciate the info. Thxalot, -sigit i.- From luismarianoguerra@REDACTED Sun Sep 26 19:01:22 2010 From: luismarianoguerra@REDACTED (Mariano Guerra) Date: Sun, 26 Sep 2010 14:01:22 -0300 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 1:07 PM, -sigit i.- wrote: > Hi all, > > Is anyone here have info on erlang user group in Madrid (their local > maillist, email, etc)? > I appreciate the info. the http://www.erlang-ar.com.ar/ user group has some people from spain (and I think some from madrid) if you want to join :) as far as I remember erlar is the only spanish speaking user group I found when I created it. maybe we could create some subgroups inside the main group. hope that helps From antoine.koener@REDACTED Sun Sep 26 21:07:53 2010 From: antoine.koener@REDACTED (Koener Antoine) Date: Sun, 26 Sep 2010 21:07:53 +0200 Subject: [erlang-questions] How to analyse why (w)erl.exe is not starting? In-Reply-To: References: Message-ID: <2D16B184-02B1-4757-B14D-8E54172A398C@gmail.com> On Sep 25, 2010, at 10:25 , Boris M?hmer wrote: > I wanted to try out the R14B release on my (hosted) windows 2003 SP2 > server. > > But when I try to start (w)erl using the icon or via command line, the > process shows up in the task list and vanishes again after just a few > secs, without any window popping up... > > I reinstalled it after a reboot => no change. > I uninstalled R14B and installed R13B03 and R13B04: no problems at > all. > > How can I check what is going wrong? There are no entires in the > eventlog. Also there is no output when started from the command line. Look for 'depends.exe', this is the windows 'ldd' command. You'll spot which dll is missing if any. You can also try 'procmon' from sysinternals, this is the [sl]trace fro windows. > > > Thanks in advance > - boris > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From boris.muehmer@REDACTED Sun Sep 26 21:49:34 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Sun, 26 Sep 2010 21:49:34 +0200 Subject: [erlang-questions] How to analyse why (w)erl.exe is not starting? In-Reply-To: <2D16B184-02B1-4757-B14D-8E54172A398C@gmail.com> References: <2D16B184-02B1-4757-B14D-8E54172A398C@gmail.com> Message-ID: 2010/9/26 Koener Antoine : > Look for 'depends.exe', this is the windows 'ldd' command. > You'll spot which dll is missing if any. > [...] > You can also try 'procmon' from sysinternals, this is the [sl]trace fro > windows. I know both tools, but I didn't want to copy them over to that server. Somehow I hoped there would be an easier way to track down this misbehaviour... - boris From wglozer@REDACTED Sun Sep 26 22:59:09 2010 From: wglozer@REDACTED (Will) Date: Sun, 26 Sep 2010 13:59:09 -0700 Subject: announce: epgsql-1.3 Message-ID: Hello, In celebration of the recent release of PostgreSQL 9.0 I've released v1.3 of epgsql, a database driver for PostgreSQL. The project was also moved from bitbucket to github and thus from mercurial to git. You'll find it here: http://github.com/wg/epgsql In addition to the usual bug fixes, and support for PostgreSQL 9.0 features, this release also supports asynchronous notification of notice and warning messages from the server as well as the LISTEN/NOTIFY mechanism, including NOTIFY payloads (only supported by 9.0+). Enjoy! -Will From rzezeski@REDACTED Mon Sep 27 05:08:20 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Sun, 26 Sep 2010 23:08:20 -0400 Subject: calendar now_to_universal_time/1 vs. universal_time/0 Message-ID: What is different about these two functions? The only thing I can gather from the docs is that universal_time *may* return local time in certain circumstances but now_to_universal_time will not. In my applications, thus far, when I need UTC time I've been using universal_time. Is this correct? Should I use now_to_universal_time instead? I ask because I've been going through the rebar source code and it uses now_to_universal_time. -Ryan From ttmrichter@REDACTED Mon Sep 27 06:22:15 2010 From: ttmrichter@REDACTED (Michael Richter) Date: Mon, 27 Sep 2010 12:22:15 +0800 Subject: Debugger fault at startup. Message-ID: So, for the first time I find myself having to fire up the debugger in Erlang. Following the user manual, I do this: $ erl Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] > [async-threads:0] [hipe] [kernel-poll:true] > Eshell V5.8.1 (abort with ^G) 1> debugger:start(). beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) > (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct > malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= > (unsigned long)((((__builtin_offsetof (struct malloc_chunk, > fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - > 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == > 0)' failed. Aborted $ I don't even know where to begin here. Is there any expert on the debugger's inner workings that knows both what that means and what can be done about it? -- "Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot." --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. From ingela@REDACTED Mon Sep 27 09:29:09 2010 From: ingela@REDACTED (Ingela Anderton Andin) Date: Mon, 27 Sep 2010 09:29:09 +0200 Subject: Usage of new verify_fun ssl option In-Reply-To: <805D1C94-E010-423A-8B68-5B4360382158@semiocast.com> References: <805D1C94-E010-423A-8B68-5B4360382158@semiocast.com> Message-ID: <4CA047C5.1060406@erix.ericsson.se> Hi Paul! Paul Guyot wrote: > Hello, > > I am trying to port our previous code that used verify_fun and validate_extensions_fun SSL options. Our previous code simply checked that the peer certificate has a given extension (and this extension is both critical and under our own OID). The local (server) certificate does not have this extension. > > I thought I could note that the extension is present in the user state and then check the user state at the end (when Event is 'valid'), like this: > > -spec verify_fun(#'OTPCertificate'{}, {bad_cert, any()} | {extension, #'Extension'{}} | valid, [atom()]) -> {valid, [atom()]} | {fail, any()} | {unknown, [atom()]}. > verify_fun(_Certificate, {bad_cert, _} = Reason, _State) -> > {fail, Reason}; > verify_fun(_Certificate, {extension, #'Extension'{extnID = ?MY_OID, critical = true}}, State) -> > {valid, [critical_extension_found | State]}; > verify_fun(_Certificate, {extension, #'Extension'{}}, State) -> > {unknown, State}; > verify_fun(_Certificate, valid, State) -> > case lists:member(critical_extension_found, State) of > true -> {valid, State}; > false -> {fail, {bad_cert, extension_missing}} > end. > > However, I ended up having two issues with this approach: > - the verify_fun function is also passed the local (server) certificate, which does not have the critical extension. How can I distinguish the two certificates? > Good question, there are extensions that could be used if they are present ... maybe we should change the verify fun to be called with {valid, peer} for the peer certificate!? > - there is a badmatch when I return a state different from the initial state. > > > This is the match with UserState0 public_key:validate/2: > http://github.com/erlang/otp/blob/dev/lib/public_key/src/public_key.erl#L559 > > UserState0 is also in the second parameter of validate/2. Is this match really intended? > > No this looks really wrong. It ought to be UserState6 = pubkey_cert:validate_signature(OtpCert, DerCert, Key, KeyParams, UserState5, VerifyFun), UserState = pubkey_cert:verify_fun(OtpCert, valid, UserState6, VerifyFun), > BTW, there is a small typo in the documentation for the type of the verify function. The record is spelled there #'OtpCertificate' instead of #'OTPCertificate'. > > Paul > Thank you for noticing we will fix it. Regards Ingela Erlang/OTP-team - Ericsson AB From pguyot@REDACTED Mon Sep 27 09:37:08 2010 From: pguyot@REDACTED (Paul Guyot) Date: Mon, 27 Sep 2010 09:37:08 +0200 Subject: Usage of new verify_fun ssl option In-Reply-To: <4CA047C5.1060406@erix.ericsson.se> References: <805D1C94-E010-423A-8B68-5B4360382158@semiocast.com> <4CA047C5.1060406@erix.ericsson.se> Message-ID: <5A79FB7B-DBD2-45D8-9F64-DA77B47D7E0B@kallisys.net> Ingela, Thank you for your quick reply. >> - the verify_fun function is also passed the local (server) certificate, which does not have the critical extension. How can I distinguish the two certificates? >> > Good question, there are extensions that could be used if they are present ... maybe we should change > the verify fun to be called with {valid, peer} for the peer certificate!? That would work for us. Regards, Paul -- Semiocast http://semiocast.com/ +33.175000290 - 62 bis rue Gay-Lussac, 75005 Paris From olopierpa@REDACTED Mon Sep 27 10:31:53 2010 From: olopierpa@REDACTED (Pierpaolo Bernardi) Date: Mon, 27 Sep 2010 10:31:53 +0200 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 05:08, Ryan Zezeski wrote: > What is different about these two functions? ?The only thing I can gather > from the docs is that universal_time *may* return local time in certain > circumstances but now_to_universal_time will not. You mean universal_time/0 vs now_to_universal_time/1 ? :) They have different arity and specification. now_to_universal_time(now()) however should be equivalent to universal_time(), as far as I understand. P. From cbenac@REDACTED Mon Sep 27 10:48:08 2010 From: cbenac@REDACTED (Clara Benac Earle) Date: Mon, 27 Sep 2010 10:48:08 +0200 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: Message-ID: <4CA05A48.5030100@fi.upm.es> As far as I know, there is not an Erlang user group in Madrid, so let's start one! Those who have interest please send me an email, and I will organize a Madrid Erlounge asap. Nos vemos Clara -sigit i.- wrote: > Hi all, > > Is anyone here have info on erlang user group in Madrid (their local > maillist, email, etc)? > I appreciate the info. > > Thxalot, > -sigit i.- > > From ttmrichter@REDACTED Mon Sep 27 13:09:20 2010 From: ttmrichter@REDACTED (Michael Richter) Date: Mon, 27 Sep 2010 19:09:20 +0800 Subject: Debugger fault at startup. In-Reply-To: References: Message-ID: After reading Pascal Chapier's message, I tried something else. From a fresh, unaltered build of R14B on a pretty bog-standard Ubuntu 10.04 installation typing "wx:new()." in the shell generates precisely the same crash message. This leads me to believe that wx is badly broken in R14B. Trying either "wx:new()." or "debugger:start()." in R14B on a completely standard Windows XP system causes the VM to crash instantly, whether run as werl.exe or erl.exe. I happen to also have R13B02 on that machine and the same problem is exhibited there. It looks to me like wx support is really badly broken and has been for some time. On 27 September 2010 12:22, Michael Richter wrote: > So, for the first time I find myself having to fire up the debugger in > Erlang. Following the user manual, I do this: > > $ erl > > Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] >> [async-threads:0] [hipe] [kernel-poll:true] > > >> Eshell V5.8.1 (abort with ^G) > > 1> debugger:start(). > > beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) >> (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct >> malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= >> (unsigned long)((((__builtin_offsetof (struct malloc_chunk, >> fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - >> 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == >> 0)' failed. > > Aborted > > $ > > > I don't even know where to begin here. Is there any expert on the > debugger's inner workings that knows both what that means and what can be > done about it? > > -- > "Perhaps people don't believe this, but throughout all of the discussions > of entering China our focus has really been what's best for the Chinese > people. It's not been about our revenue or profit or whatnot." > --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. > -- "Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot." --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. From lukas@REDACTED Mon Sep 27 13:19:04 2010 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 27 Sep 2010 13:19:04 +0200 Subject: [erlang-questions] Common test issues - Possible bug? Message-ID: <1285586344.16569.27.camel@ancalagon.du.uab.ericsson.se> Hi Mikkel! What are you using to run your tests? The run_test command, or are you starting them from an erlang shell? Also have you seen this problem when not using ct_ssh? or is it only when you are doing tests with ssh calls in them? If you could provide me with a minimal test suite which reproduces the error you are seeing that would be great! Regards, Lukas, Erlang/OTP, Ericsson AB. On Fri, Sep 24, 2010 at 6:40 PM, Mikkel Jensen wrote: Hi list! I have been working on setting up a test case using Erlang Common Test. The test is pretty simple, it consists of a sequence of calls to ct_ssh:exec(). Running the test once works. Running the test a few times (using repeat), usually works. But if i let the test suite repeat a large number of times, it fails after successfully running the test multiple times (In my last attempt it completed 42 runs, no joke, but it seems like the repeated runs can stop at any time). The error i'm getting is: It is not possible to install CT while running in interactive mode. To exit this mode, run ct:stop_interactive(). To enter the interactive mode again, run ct:start_interactive() Does anyone know how i can avoid this, allowing me to run a large number of repeated tests? On a sidenote: I have noticed in the logs that ct_ssh will sometimes have lost its connection when trying to call ct_ssh:exec(), and thus will attempt to reconnect. This attempt will, however, always fail instantly. A workaround i have made is to manually connect on each call i make to ct_ssh:exec(). This seems less than ideal, is there a better way? Kind regards and thanks in advance, Mikkel From boris.muehmer@REDACTED Mon Sep 27 13:32:20 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Mon, 27 Sep 2010 13:32:20 +0200 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: 2010/9/27 Michael Richter : > It looks to me like wx support is really badly broken and has been for some > time. I use Ubuntu 10.04 on serveral machines (most of them are 64 Bit systems). I only use self-compiled Erlang versions. (Based on my blog article: http://borisukun.blogspot.com/2010/05/installing-erlangotp-on-ubuntu-from.html ) And I didn't have any problems with wx on R13B3, R13B4, and R14B. Actually I did many wx/XRC tests this weekend, and had no problems, except my own programming errors. Does "wx:demo()" work? - boris From attila.r.nohl@REDACTED Mon Sep 27 13:56:45 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Mon, 27 Sep 2010 13:56:45 +0200 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: I've just tested on R14B on SuSE 10 and the wx demo works fine... My gut feeling is that your build was not properly built. Or you might have a hardware error. 2010/9/27, Michael Richter : > After reading Pascal Chapier's message, I tried something else. From a > fresh, unaltered build of R14B on a pretty bog-standard Ubuntu 10.04 > installation typing "wx:new()." in the shell generates precisely the same > crash message. This leads me to believe that wx is badly broken in R14B. > Trying either "wx:new()." or "debugger:start()." in R14B on a completely > standard Windows XP system causes the VM to crash instantly, whether run as > werl.exe or erl.exe. I happen to also have R13B02 on that machine and the > same problem is exhibited there. > > It looks to me like wx support is really badly broken and has been for some > time. > > On 27 September 2010 12:22, Michael Richter wrote: > >> So, for the first time I find myself having to fire up the debugger in >> Erlang. Following the user manual, I do this: >> >> $ erl >> >> Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] >>> [async-threads:0] [hipe] [kernel-poll:true] >> >> >>> Eshell V5.8.1 (abort with ^G) >> >> 1> debugger:start(). >> >> beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) >>> (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct >>> malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= >>> (unsigned long)((((__builtin_offsetof (struct malloc_chunk, >>> fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - >>> 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == >>> 0)' failed. >> >> Aborted >> >> $ >> >> >> I don't even know where to begin here. Is there any expert on the >> debugger's inner workings that knows both what that means and what can be >> done about it? >> >> -- >> "Perhaps people don't believe this, but throughout all of the discussions >> of entering China our focus has really been what's best for the Chinese >> people. It's not been about our revenue or profit or whatnot." >> --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. >> > > > > -- > "Perhaps people don't believe this, but throughout all of the discussions of > entering China our focus has really been what's best for the Chinese people. > It's not been about our revenue or profit or whatnot." > --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. > From daniel.goertzen@REDACTED Mon Sep 27 16:15:00 2010 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Mon, 27 Sep 2010 09:15:00 -0500 Subject: non-trivial supervisor idioms? Message-ID: I've read the documentation on supervision and have seen a few tutorials, but they don't seem to move beyond the core concepts. For example, what happens if you want to check and optionally setup an mnesia schema during startup...where should this code go? In the supervisor init() or start_link() function? Should I have my supervisor create a worker process whole sole job is to do this kind of setup and then dynamically add other workers (or supervisors) to the supervisor with start_child()? I have this feeling there are already idiomatic ways of doing this, and I'm reinventing the wheel poorly and incompletely. Any advice appreciated. Thanks, Dan. From daniel.goertzen@REDACTED Mon Sep 27 16:31:16 2010 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Mon, 27 Sep 2010 09:31:16 -0500 Subject: [erlang-questions] Installing on cygwin In-Reply-To: References: Message-ID: This installation guide suggests that Erlang probably won't work on cygwin... http://www.erlang.org/doc/installation_guide/INSTALL-WIN32.html (see 4th question in faq) That said, I am happily using cygwin to invoke the native Windows version of Erlang. There's a bit of fussing to deal with Windows vs Unix paths, but it hasn't been a huge deal so far. (I am also having good luck using the native Windows version of Emacs instead of cygwin emacs.) Dan. On Wed, Sep 22, 2010 at 11:11 AM, Giulio Petrucci wrote: > Hi there, > > this is my first post in this mailing list so, first of all I'd like > to say hello to everyone reading here. > I'm trying playing with the ReverseHTTP project > (http://github.com/tonyg/reversehttp/) and I need to run a web server > written in Erlang but... I'm a totally newbie with this language! :-) > > Under Linux (the OS I use mostly for personal stuff) I didn't have to > do anything special. Under Windows, which I use at work, I'm a little > bit in trouble. I'd like to install Erlang on my Cygwin installation. > I've just downloaded the source code... now what shall I do? I've > tried to get into the source directory and type "make" on my shell > but... nothing happened. :-) Could anyone help me? > > Thanks in advance, > Giulio > > -- > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Daniel Goertzen ----------------- dang@REDACTED (work) daniel.goertzen@REDACTED (home) ----------------- 1 204 272 6149 (home/office) 1 204 470 8360 (mobile) ----------------- From ulf.wiger@REDACTED Mon Sep 27 16:31:47 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 27 Sep 2010 16:31:47 +0200 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: Message-ID: <4CA0AAD3.3030209@erlang-solutions.com> On 27/09/2010 16:15, Daniel Goertzen wrote: > I've read the documentation on supervision and have seen a few tutorials, > but they don't seem to move beyond the core concepts. For example, what > happens if you want to check and optionally setup an mnesia schema during > startup...where should this code go? In the supervisor init() or > start_link() function? Should I have my supervisor create a worker process > whole sole job is to do this kind of setup and then dynamically add other > workers (or supervisors) to the supervisor with start_child()? I strongly recommend doing that sort of thing in a separate procedure, rather than in the startup phase. If you want your application to be able to bootstrap itself, I would suggest that you either: - create a special application that runs before your other apps, and verifies that the installation is ok. To this end, it might be useful to know that you can pre-sort the .rel file. The systools lib will only change the sort order if needed to respect start dependencies. - Introduce start_phases, then do minimal work in the init function, and push the rest to functions that are called from start phase hooks. This also has the advantage that you know that your processes are all started and ready to respond during the init phase. Start phases are documented in http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3 BR, Ulf W -- Ulf Wiger CTO, Erlang Solutions Ltd, formerly Erlang Training & Consulting Ltd http://www.erlang-solutions.com From ebegumisa@REDACTED Mon Sep 27 16:40:48 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 00:40:48 +1000 Subject: [erlang-questions] Compiling Erlang Interface with VC++ 2010 - LIB Issues In-Reply-To: <000301cb5c06$addf68d0$099e3a70$@rock-technologies.com> References: <000301cb5c06$addf68d0$099e3a70$@rock-technologies.com> Message-ID: It looks to me like the VS linker is having trouble with winsock... Try adding Ws2_32.lib to the link command line. - Edmond - On Sat, 25 Sep 2010 02:36:44 +1000, wrote: > Dear Erlang Questions, > > > When compiling an Erlang Interface with VC++ 2010 I get the following > compiler errors. > > > 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol > __imp__gethostbyname@REDACTED referenced in function _ei_gethostbyname > > 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol > __imp__gethostbyaddr@REDACTED referenced in function _ei_gethostbyaddr > > 1>D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\sqlite_port.exe : > fatal > error LNK1120: 2 unresolved externals > > > Path set correctly to > > > C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\lib\ei.lib > > C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\include\ei.h > > > What is going wrong here. > > > Thank you. > > Rock InformationTechnologies (India) Pvt. > Dipl.-Ing.(FH) Klaus Rock > Bonhoefferstrasse 37 > 73432 Aalen - Germany > Tel : ++49-7367-96500 > Mail : k.rock@REDACTED > ompose?to=k.rock@REDACTED> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From chad@REDACTED Mon Sep 27 16:52:59 2010 From: chad@REDACTED (Chad DePue) Date: Mon, 27 Sep 2010 11:52:59 -0300 Subject: [erlang-questions] New here. In-Reply-To: <4C9EDA2B.4050102@brooklynnemichelle.com> References: <4C9EDA2B.4050102@brooklynnemichelle.com> Message-ID: Welcome! I wonder how often Erlang is someone's first computer language? That seems relatively unusual! On Sun, Sep 26, 2010 at 2:29 AM, brooklynne < brooklynne@REDACTED> wrote: > I'm new here and learning Erlang via the website learnyousomeerlang.comIt's also my first language (not for want of trying just the first one to > actually make sense. > > Brooklynne > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ebegumisa@REDACTED Mon Sep 27 17:24:48 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 01:24:48 +1000 Subject: [erlang-questions] How to analyse why (w)erl.exe is not starting? In-Reply-To: References: <2D16B184-02B1-4757-B14D-8E54172A398C@gmail.com> Message-ID: Is Windows Error Reporting enabled on the box? My guess is that werl.exe/erl.exe are crashing before they've had a chance to set up the emulator crash dump infrastructure (but hopefully after all the dll loading is done) in which case a minidump might help. If I'm not mistaken, the official windows OTP binary installer has a sub-directory with debug versions of werl.exe/erl.exe and corresponding symbol files (werl.pbg/erl.pdb). You could try copying these to your server, then triggering the creation of a minidump using WER (which you could copy from the temp directory before it's sent to Microsoft), then finally analyse the call-stack using VS. http://support.microsoft.com/kb/310414 (Don't know if WER configuration for win2k3 is the same as for XP) - Edmond - On Mon, 27 Sep 2010 05:49:34 +1000, Boris M?hmer wrote: > 2010/9/26 Koener Antoine : >> Look for 'depends.exe', this is the windows 'ldd' command. >> You'll spot which dll is missing if any. >> [...] >> You can also try 'procmon' from sysinternals, this is the [sl]trace fro >> windows. > > I know both tools, but I didn't want to copy them over to that server. > Somehow I hoped there would be an easier way to track down this > misbehaviour... > > > - boris > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ttmrichter@REDACTED Mon Sep 27 17:38:35 2010 From: ttmrichter@REDACTED (Michael Richter) Date: Mon, 27 Sep 2010 23:38:35 +0800 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: I would agree with this but for the fact that two pre-built Erlang for Windows (R14B and R13B02) exhibit similar behaviour (but without the "helpful" message) on a completely different machine. As for my build, it's a raw untar, ./configure & make & sudo make install. I'm not sure how I could possibly screw that up short of mis-spelling "configure". I even repeated the process today just to double-check. On 27 September 2010 19:56, Attila Rajmund Nohl wrote: > I've just tested on R14B on SuSE 10 and the wx demo works fine... My > gut feeling is that your build was not properly built. Or you might > have a hardware error. > > 2010/9/27, Michael Richter : > > After reading Pascal Chapier's message, I tried something else. From a > > fresh, unaltered build of R14B on a pretty bog-standard Ubuntu 10.04 > > installation typing "wx:new()." in the shell generates precisely the same > > crash message. This leads me to believe that wx is badly broken in R14B. > > Trying either "wx:new()." or "debugger:start()." in R14B on a completely > > standard Windows XP system causes the VM to crash instantly, whether run > as > > werl.exe or erl.exe. I happen to also have R13B02 on that machine and > the > > same problem is exhibited there. > > > > It looks to me like wx support is really badly broken and has been for > some > > time. > > > > On 27 September 2010 12:22, Michael Richter > wrote: > > > >> So, for the first time I find myself having to fire up the debugger in > >> Erlang. Following the user manual, I do this: > >> > >> $ erl > >> > >> Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] > >>> [async-threads:0] [hipe] [kernel-poll:true] > >> > >> > >>> Eshell V5.8.1 (abort with ^G) > >> > >> 1> debugger:start(). > >> > >> beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) > >>> (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct > >>> malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) > >= > >>> (unsigned long)((((__builtin_offsetof (struct malloc_chunk, > >>> fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) > - > >>> 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) > == > >>> 0)' failed. > >> > >> Aborted > >> > >> $ > >> > >> > >> I don't even know where to begin here. Is there any expert on the > >> debugger's inner workings that knows both what that means and what can > be > >> done about it? > >> > >> -- > >> "Perhaps people don't believe this, but throughout all of the > discussions > >> of entering China our focus has really been what's best for the Chinese > >> people. It's not been about our revenue or profit or whatnot." > >> --Sergey Brin, demonstrating the emptiness of the "don't be evil" > mantra. > >> > > > > > > > > -- > > "Perhaps people don't believe this, but throughout all of the discussions > of > > entering China our focus has really been what's best for the Chinese > people. > > It's not been about our revenue or profit or whatnot." > > --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. > > > -- "Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot." --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. From ttmrichter@REDACTED Mon Sep 27 17:38:35 2010 From: ttmrichter@REDACTED (Michael Richter) Date: Mon, 27 Sep 2010 23:38:35 +0800 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: I would agree with this but for the fact that two pre-built Erlang for Windows (R14B and R13B02) exhibit similar behaviour (but without the "helpful" message) on a completely different machine. As for my build, it's a raw untar, ./configure & make & sudo make install. I'm not sure how I could possibly screw that up short of mis-spelling "configure". I even repeated the process today just to double-check. On 27 September 2010 19:56, Attila Rajmund Nohl wrote: > I've just tested on R14B on SuSE 10 and the wx demo works fine... My > gut feeling is that your build was not properly built. Or you might > have a hardware error. > > 2010/9/27, Michael Richter : > > After reading Pascal Chapier's message, I tried something else. From a > > fresh, unaltered build of R14B on a pretty bog-standard Ubuntu 10.04 > > installation typing "wx:new()." in the shell generates precisely the same > > crash message. This leads me to believe that wx is badly broken in R14B. > > Trying either "wx:new()." or "debugger:start()." in R14B on a completely > > standard Windows XP system causes the VM to crash instantly, whether run > as > > werl.exe or erl.exe. I happen to also have R13B02 on that machine and > the > > same problem is exhibited there. > > > > It looks to me like wx support is really badly broken and has been for > some > > time. > > > > On 27 September 2010 12:22, Michael Richter > wrote: > > > >> So, for the first time I find myself having to fire up the debugger in > >> Erlang. Following the user manual, I do this: > >> > >> $ erl > >> > >> Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] > >>> [async-threads:0] [hipe] [kernel-poll:true] > >> > >> > >>> Eshell V5.8.1 (abort with ^G) > >> > >> 1> debugger:start(). > >> > >> beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) > >>> (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct > >>> malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) > >= > >>> (unsigned long)((((__builtin_offsetof (struct malloc_chunk, > >>> fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) > - > >>> 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) > == > >>> 0)' failed. > >> > >> Aborted > >> > >> $ > >> > >> > >> I don't even know where to begin here. Is there any expert on the > >> debugger's inner workings that knows both what that means and what can > be > >> done about it? > >> > >> -- > >> "Perhaps people don't believe this, but throughout all of the > discussions > >> of entering China our focus has really been what's best for the Chinese > >> people. It's not been about our revenue or profit or whatnot." > >> --Sergey Brin, demonstrating the emptiness of the "don't be evil" > mantra. > >> > > > > > > > > -- > > "Perhaps people don't believe this, but throughout all of the discussions > of > > entering China our focus has really been what's best for the Chinese > people. > > It's not been about our revenue or profit or whatnot." > > --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. > > > -- "Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot." --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. From ebegumisa@REDACTED Mon Sep 27 17:44:56 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 01:44:56 +1000 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: I observed haven't any problems with debugger either (R14B -- Mac OS X 10.5.8 and Windows 7) But I am forced to use the et_viewer in gs mode coz it hangs in wx mode on win7. So there might be a problem with wxErlang. - Edmond - On Mon, 27 Sep 2010 21:56:45 +1000, Attila Rajmund Nohl wrote: > I've just tested on R14B on SuSE 10 and the wx demo works fine... My > gut feeling is that your build was not properly built. Or you might > have a hardware error. > > 2010/9/27, Michael Richter : >> After reading Pascal Chapier's message, I tried something else. From a >> fresh, unaltered build of R14B on a pretty bog-standard Ubuntu 10.04 >> installation typing "wx:new()." in the shell generates precisely the >> same >> crash message. This leads me to believe that wx is badly broken in >> R14B. >> Trying either "wx:new()." or "debugger:start()." in R14B on a >> completely >> standard Windows XP system causes the VM to crash instantly, whether >> run as >> werl.exe or erl.exe. I happen to also have R13B02 on that machine and >> the >> same problem is exhibited there. >> >> It looks to me like wx support is really badly broken and has been for >> some >> time. >> >> On 27 September 2010 12:22, Michael Richter >> wrote: >> >>> So, for the first time I find myself having to fire up the debugger in >>> Erlang. Following the user manual, I do this: >>> >>> $ erl >>> >>> Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] >>>> [async-threads:0] [hipe] [kernel-poll:true] >>> >>> >>>> Eshell V5.8.1 (abort with ^G) >>> >>> 1> debugger:start(). >>> >>> beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) >>>> (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct >>>> malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >>>> >= >>>> (unsigned long)((((__builtin_offsetof (struct malloc_chunk, >>>> fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * >>>> (sizeof(size_t))) - >>>> 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & >>>> pagemask) == >>>> 0)' failed. >>> >>> Aborted >>> >>> $ >>> >>> >>> I don't even know where to begin here. Is there any expert on the >>> debugger's inner workings that knows both what that means and what can >>> be >>> done about it? >>> >>> -- >>> "Perhaps people don't believe this, but throughout all of the >>> discussions >>> of entering China our focus has really been what's best for the Chinese >>> people. It's not been about our revenue or profit or whatnot." >>> --Sergey Brin, demonstrating the emptiness of the "don't be evil" >>> mantra. >>> >> >> >> >> -- >> "Perhaps people don't believe this, but throughout all of the >> discussions of >> entering China our focus has really been what's best for the Chinese >> people. >> It's not been about our revenue or profit or whatnot." >> --Sergey Brin, demonstrating the emptiness of the "don't be evil" >> mantra. >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From mononcqc@REDACTED Mon Sep 27 17:45:01 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 27 Sep 2010 11:45:01 -0400 Subject: [erlang-questions] New here. In-Reply-To: <4C9EDA2B.4050102@brooklynnemichelle.com> References: <4C9EDA2B.4050102@brooklynnemichelle.com> Message-ID: Hi, I'm the guy behind learnyousomeerlang.com. I'm especially interested to know how well it works for people who have never programmed in any other language before (even though the book assumes some experience). Let me know if you have any comments or parts that you found particularly hard. From ebegumisa@REDACTED Mon Sep 27 17:53:54 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 01:53:54 +1000 Subject: [erlang-questions] Compiling Erlang Interface with VC++ 2010 - LIB Issues In-Reply-To: <000901cb5e58$ffb1b380$ff151a80$@rock-technologies.com> References: <000301cb5c06$addf68d0$099e3a70$@rock-technologies.com> <000901cb5e58$ffb1b380$ff151a80$@rock-technologies.com> Message-ID: You could just ignore those warnings (it's finally linking after all.) But to fix those (I hate long warnings too), you can try either... a) Turning off incremental linking and debugging b) Enabling the symbol server (so it can fetch those older VC pdb files) - Edmond - On Tue, 28 Sep 2010 01:31:02 +1000, wrote: > Hi Edmond, > > The Result. > > 1>erl_interface.lib(erl_error.o) : warning LNK4099: PDB 'vc80.pdb' was > not > found with 'erl_interface.lib(erl_error.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>erl_interface.lib(erl_eterm.o) : warning LNK4099: PDB 'vc80.pdb' was > not > found with 'erl_interface.lib(erl_eterm.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>erl_interface.lib(erl_fix_alloc.o) : warning LNK4099: PDB 'vc80.pdb' > was > not found with 'erl_interface.lib(erl_fix_alloc.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>erl_interface.lib(erl_malloc.o) : warning LNK4099: PDB 'vc80.pdb' was > not > found with 'erl_interface.lib(erl_malloc.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>erl_interface.lib(erl_marshal.o) : warning LNK4099: PDB 'vc80.pdb' was > not > found with 'erl_interface.lib(erl_marshal.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(ei_resolve.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with > 'ei.lib(ei_resolve.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(decode_big.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with > 'ei.lib(decode_big.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(decode_double.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with 'ei.lib(decode_double.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(decode_long.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with > 'ei.lib(decode_long.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(ei_malloc.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with > 'ei.lib(ei_malloc.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(ei_pthreads.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with > 'ei.lib(ei_pthreads.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(get_type.o) : warning LNK4099: PDB 'vc80.pdb' was not found with > 'ei.lib(get_type.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1>ei.lib(ei_compat.o) : warning LNK4099: PDB 'vc80.pdb' was not found > with > 'ei.lib(ei_compat.o)' or at > 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking > object as if no debug info > 1> sqlite_port.vcxproj -> > D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\sqlite_port.exe > ========== Rebuild All: 1 succeeded, 0 failed, 0 skipped ========== > > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On > Behalf Of Edmond Begumisa > Sent: Monday, September 27, 2010 4:41 PM > To: erlang-questions@REDACTED; k.rock@REDACTED > Subject: Re: [erlang-questions] Compiling Erlang Interface with VC++ > 2010 - > LIB Issues > > It looks to me like the VS linker is having trouble with winsock... > > Try adding Ws2_32.lib to the link command line. > > - Edmond - > > On Sat, 25 Sep 2010 02:36:44 +1000, wrote: > >> Dear Erlang Questions, >> >> >> When compiling an Erlang Interface with VC++ 2010 I get the following >> compiler errors. >> >> >> 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol >> __imp__gethostbyname@REDACTED referenced in function _ei_gethostbyname >> >> 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol >> __imp__gethostbyaddr@REDACTED referenced in function _ei_gethostbyaddr >> >> 1>D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\sqlite_port.exe : >> fatal >> error LNK1120: 2 unresolved externals >> >> >> Path set correctly to >> >> >> C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\lib\ei.lib >> >> C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\include\ei.h >> >> >> What is going wrong here. >> >> >> Thank you. >> >> Rock InformationTechnologies (India) Pvt. >> Dipl.-Ing.(FH) Klaus Rock >> Bonhoefferstrasse 37 >> 73432 Aalen - Germany >> Tel : ++49-7367-96500 >> Mail : k.rock@REDACTED >> > 0/mc/c ompose?to=k.rock@REDACTED> >> >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From daniel.goertzen@REDACTED Mon Sep 27 17:59:47 2010 From: daniel.goertzen@REDACTED (Daniel Goertzen) Date: Mon, 27 Sep 2010 10:59:47 -0500 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: <4CA0AAD3.3030209@erlang-solutions.com> References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: On Mon, Sep 27, 2010 at 9:31 AM, Ulf Wiger wrote: > On 27/09/2010 16:15, Daniel Goertzen wrote: > >> I've read the documentation on supervision and have seen a few tutorials, >> but they don't seem to move beyond the core concepts. For example, what >> happens if you want to check and optionally setup an mnesia schema during >> startup...where should this code go? In the supervisor init() or >> start_link() function? Should I have my supervisor create a worker >> process >> whole sole job is to do this kind of setup and then dynamically add other >> workers (or supervisors) to the supervisor with start_child()? >> > > I strongly recommend doing that sort of thing in a separate procedure, > rather than in the startup phase. > > If you want your application to be able to bootstrap itself, I would > suggest that you either: > > - create a special application that runs before your other apps, > and verifies that the installation is ok. To this end, it might be > useful to know that you can pre-sort the .rel file. The systools lib > will only change the sort order if needed to respect start > dependencies. > - Introduce start_phases, then do minimal work in the init function, > and push the rest to functions that are called from start phase > hooks. This also has the advantage that you know that your processes > are all started and ready to respond during the init phase. > > Start phases are documented in > http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3 > > BR, > Ulf W > > Thanks, I forgot about start phases. Makes perfect sense for mnesia init. Now how about this scenario: I am opening an ssh client connection which will have multiple channels. The channel processes require a connection reference, so I have to establish the connection before creating them. I will keep these channel processes under a supervisor. Now, I have the choice of establishing the connection in supervisor init() and then returning childspecs with the connection reference built-in, or creating a worker to establish the connection and then dynamically add channel processes to the supervisor afterwards. I have to think this kind of complex startup happens frequently enough that best practices have been established. Or perhaps these situations are actually rare? BTW, my gut feeling is that I should leave supervisor init() alone and have a worker perform complex startup. Thanks, Dan. From ebegumisa@REDACTED Mon Sep 27 18:14:39 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 02:14:39 +1000 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: <4CA0AAD3.3030209@erlang-solutions.com> References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: Ulf, I've been doing such initialisation in the init function of a worker manager process. Using Daniel's example, I might have a gen_server child of the main supervisor called db_mgr and set up the mnesia schema in db_mgr:init Have I been doing the 'wrong' thing OTP-wise? - Edmond - On Tue, 28 Sep 2010 00:31:47 +1000, Ulf Wiger wrote: > On 27/09/2010 16:15, Daniel Goertzen wrote: >> I've read the documentation on supervision and have seen a few >> tutorials, >> but they don't seem to move beyond the core concepts. For example, what >> happens if you want to check and optionally setup an mnesia schema >> during >> startup...where should this code go? In the supervisor init() or >> start_link() function? Should I have my supervisor create a worker >> process >> whole sole job is to do this kind of setup and then dynamically add >> other >> workers (or supervisors) to the supervisor with start_child()? > > I strongly recommend doing that sort of thing in a separate procedure, > rather than in the startup phase. > > If you want your application to be able to bootstrap itself, I would > suggest that you either: > > - create a special application that runs before your other apps, > and verifies that the installation is ok. To this end, it might be > useful to know that you can pre-sort the .rel file. The systools lib > will only change the sort order if needed to respect start > dependencies. > - Introduce start_phases, then do minimal work in the init function, > and push the rest to functions that are called from start phase > hooks. This also has the advantage that you know that your processes > are all started and ready to respond during the init phase. > > Start phases are documented in > http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3 > > BR, > Ulf W > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From rzezeski@REDACTED Mon Sep 27 18:21:09 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Mon, 27 Sep 2010 12:21:09 -0400 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 4:31 AM, Pierpaolo Bernardi wrote: > On Mon, Sep 27, 2010 at 05:08, Ryan Zezeski wrote: > > What is different about these two functions? The only thing I can gather > > from the docs is that universal_time *may* return local time in certain > > circumstances but now_to_universal_time will not. > > > now_to_universal_time(now()) however should be equivalent to > universal_time(), as far as I understand. > > This is, essentially, the question I'm asking. Should I prefer one over the other for *any* reason? Right now, I prefer universal_time because it's less typing :) -Ryan From olopierpa@REDACTED Mon Sep 27 18:25:46 2010 From: olopierpa@REDACTED (Pierpaolo Bernardi) Date: Mon, 27 Sep 2010 18:25:46 +0200 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 18:21, Ryan Zezeski wrote: > This is, essentially, the question I'm asking. ?Should I prefer one over the > other for *any* reason? ?Right now, I prefer universal_time because it's > less typing :) A perfectly good reason. It's also a tiny bit clearer. P. From essiene@REDACTED Mon Sep 27 18:38:23 2010 From: essiene@REDACTED (Essien Essien) Date: Mon, 27 Sep 2010 17:38:23 +0100 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: Hi Dan, > Thanks, I forgot about start phases. ?Makes perfect sense for mnesia init. > > Now how about this scenario: ?I am opening an ssh client connection which > will have multiple channels. ?The channel processes require a connection > reference, so I have to establish the connection before creating them. ?I > will keep these channel processes under a supervisor. ?Now, I have the > choice of establishing the connection in supervisor init() and then > returning ?childspecs with the connection reference built-in, or creating a > worker to establish the connection and then dynamically add channel > processes to the supervisor afterwards. I would create the connection in a connection object/process of some sort and that process/object would be responsible for calling supervisor:start_child(ChannelSup, [Connection]), when I needed a new channel on that connection. Roughly something like: {ok, Connection} = connection:create(Host, Port, ...), % this would create and return an SSH connection, a gen_server would do or something. {ok, ChanSup} = supervisor:start_link({local, chan_sup}, chan_sup_callback, []), {ok, Chan1} = supervisor:start_child(ChanSup, [Connection]), {ok, Chan2} = supervisor:start_child(ChanSup, [Connection]), ... Ofcourse ChanSup is a simple_one_for_one supervisor with an init/1 like: init([]) -> {ok, { {simple_one_for_one, 10, 10}, [{channel,{channel, start_link, []}, transient, 5000, worker, [channel]}]}}. I would also setup monitors b/w the Connection and the Channels, so if the connection dies, say if its closed, the channels would all die along with it. The connection's supervisor would be responsible for bringing up this entire tree after disaster has struck. > > I have to think this kind of complex startup happens frequently enough that > best practices have been established. ?Or perhaps these situations are > actually rare? Dunno ;) > > BTW, my gut feeling is that I should leave supervisor init() alone and have > a worker perform complex startup. I don't really do much in my supervisor inits, sometimes in my top level supervisors, I read config files, etc, but that's about it. Not much more. That's me though... others more experienced than me may have different ways of using supervisor:init/1 cheers, Essien > > Thanks, > Dan. > From ulf.wiger@REDACTED Mon Sep 27 19:00:13 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 27 Sep 2010 19:00:13 +0200 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On 27 Sep 2010, at 18:25, Pierpaolo Bernardi wrote: > On Mon, Sep 27, 2010 at 18:21, Ryan Zezeski wrote: > >> This is, essentially, the question I'm asking. Should I prefer one over the >> other for *any* reason? Right now, I prefer universal_time because it's >> less typing :) > > A perfectly good reason. It's also a tiny bit clearer. erlang:now() is actually a bit different from the calendar clock. It can, by definition, never jump backwards. It will also attempt not to make large adjustments of any kind, so if it detects that there is a large difference between the system clock and the erlang:now() clock, the now() clock will speed up or slow down 1% in order to converge with the actual time, without disturbing timeouts etc. which rely on now() to provide a smooth representation of system real-time. In other words, erlang:universal_time/0 will return the actual time, whereas erlang:now() may, under certain circumstances, differ quite significantly from actual time. BR, Ulf Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From ebegumisa@REDACTED Mon Sep 27 19:12:33 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 03:12:33 +1000 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: Hi Daniel, IMO, your first suggestion (having the supervisor create a connection in init()) I think would be a bad idea. My understanding of OTP design principles is that the supervisor itself shouldn't do any actual work, it should only keep an eye on others who do do the work. Your second suggestion would be much better. I'd approach it like Essien, but I'd also wrap the call supervisor:start_child(ChanSup, [Connection]) in a function exported from the supervisor module... new_child(ChanSup, [Connection]) supervisor:start_child(ChanSup, [Connection]). To reinforce what Essien said, your connection would 'ask' the supervisor to 'create and keep an eye on' a new channel. - Edmond - On Tue, 28 Sep 2010 02:38:23 +1000, Essien Essien wrote: > Hi Dan, > > > >> Thanks, I forgot about start phases. Makes perfect sense for mnesia >> init. >> >> Now how about this scenario: I am opening an ssh client connection >> which >> will have multiple channels. The channel processes require a connection >> reference, so I have to establish the connection before creating them. >> I >> will keep these channel processes under a supervisor. Now, I have the >> choice of establishing the connection in supervisor init() and then >> returning childspecs with the connection reference built-in,or >> creating a >> worker to establish the connection and then dynamically add channel >> processes to the supervisor afterwards. > > I would create the connection in a connection object/process of some > sort and that process/object would be responsible for calling > supervisor:start_child(ChannelSup, [Connection]), when I needed a new > channel on that connection. > > Roughly something like: > > > > {ok, Connection} = connection:create(Host, Port, ...), % this would > create and return an SSH connection, a gen_server would do or > something. > > {ok, ChanSup} = supervisor:start_link({local, chan_sup}, > chan_sup_callback, []), > > {ok, Chan1} = supervisor:start_child(ChanSup, [Connection]), > {ok, Chan2} = supervisor:start_child(ChanSup, [Connection]), > ... > > > > Ofcourse ChanSup is a simple_one_for_one supervisor with an init/1 like: > > > > init([]) -> > {ok, { > {simple_one_for_one, 10, 10}, > [{channel,{channel, start_link, []}, > transient, 5000, worker, [channel]}]}}. > > > > I would also setup monitors b/w the Connection and the Channels, so if > the connection dies, say if its closed, the channels would all die > along with it. The connection's supervisor would be responsible for > bringing up this entire tree after disaster has struck. > >> >> I have to think this kind of complex startup happens frequently enough >> that >> best practices have been established. Or perhaps these situations are >> actually rare? > > Dunno ;) > >> >> BTW, my gut feeling is that I should leave supervisor init() alone and >> have >> a worker perform complex startup. > > I don't really do much in my supervisor inits, sometimes in my top > level supervisors, I read config files, etc, but that's about it. Not > much more. That's me though... others more experienced than me may > have different ways of using supervisor:init/1 > > cheers, > Essien >> >> Thanks, >> Dan. >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ulf.wiger@REDACTED Mon Sep 27 19:19:29 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 27 Sep 2010 19:19:29 +0200 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: On 27 Sep 2010, at 18:14, Edmond Begumisa wrote: > Ulf, > > I've been doing such initialisation in the init function of a worker manager process. Using Daniel's example, I might have a gen_server child of the main supervisor called db_mgr and set up the mnesia schema in db_mgr:init > > Have I been doing the 'wrong' thing OTP-wise? Not necessarily, but my personal preference is to cleanly separate setup code from application startup. This is in part because I used to work on a very complex product, where the setup was decidedly non-trivial, and the startup process had to be optimised in several steps. Still, even there, I believe that the setup logic was bootstrapped into the startup phase, but the code was still kept cleanly separated. The only thing that was part of the startup was a simple check to see if the setup code had been run. BR, Ulf W > > - Edmond - > > On Tue, 28 Sep 2010 00:31:47 +1000, Ulf Wiger wrote: > >> On 27/09/2010 16:15, Daniel Goertzen wrote: >>> I've read the documentation on supervision and have seen a few tutorials, >>> but they don't seem to move beyond the core concepts. For example, what >>> happens if you want to check and optionally setup an mnesia schema during >>> startup...where should this code go? In the supervisor init() or >>> start_link() function? Should I have my supervisor create a worker process >>> whole sole job is to do this kind of setup and then dynamically add other >>> workers (or supervisors) to the supervisor with start_child()? >> >> I strongly recommend doing that sort of thing in a separate procedure, >> rather than in the startup phase. >> >> If you want your application to be able to bootstrap itself, I would >> suggest that you either: >> >> - create a special application that runs before your other apps, >> and verifies that the installation is ok. To this end, it might be >> useful to know that you can pre-sort the .rel file. The systools lib >> will only change the sort order if needed to respect start >> dependencies. >> - Introduce start_phases, then do minimal work in the init function, >> and push the rest to functions that are called from start phase >> hooks. This also has the advantage that you know that your processes >> are all started and ready to respond during the init phase. >> >> Start phases are documented in >> http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3 >> >> BR, >> Ulf W >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From ebegumisa@REDACTED Mon Sep 27 19:26:35 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 03:26:35 +1000 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: Oops, that code should have been... -module(chan_sup). ... new_child(Connection) -> supervisor:start_child(?MODULE, [Connection]). ... Or something like that. - Edmond - On Tue, 28 Sep 2010 03:12:33 +1000, Edmond Begumisa wrote: > Hi Daniel, > > IMO, your first suggestion (having the supervisor create a connection in > init()) I think would be a bad idea. My understanding of OTP design > principles is that the supervisor itself shouldn't do any actual work, > it should only keep an eye on others who do do the work. > > Your second suggestion would be much better. I'd approach it like > Essien, but I'd also wrap the call supervisor:start_child(ChanSup, > [Connection]) in a function exported from the supervisor module... > > new_child(ChanSup, [Connection]) > supervisor:start_child(ChanSup, [Connection]). > > To reinforce what Essien said, your connection would 'ask' the > supervisor to 'create and keep an eye on' a new channel. > > - Edmond - > > On Tue, 28 Sep 2010 02:38:23 +1000, Essien Essien > wrote: > >> Hi Dan, >> >> >> >>> Thanks, I forgot about start phases. Makes perfect sense for mnesia >>> init. >>> >>> Now how about this scenario: I am opening an ssh client connection >>> which >>> will have multiple channels. The channel processes require a >>> connection >>> reference, so I have to establish the connection before creating them. >>> I >>> will keep these channel processes under a supervisor. Now, I have the >>> choice of establishing the connection in supervisor init() and then >>> returning childspecs with the connection reference built-in,or >>> creating a >>> worker to establish the connection and then dynamically add channel >>> processes to the supervisor afterwards. >> >> I would create the connection in a connection object/process of some >> sort and that process/object would be responsible for calling >> supervisor:start_child(ChannelSup, [Connection]), when I needed a new >> channel on that connection. >> >> Roughly something like: >> >> >> >> {ok, Connection} = connection:create(Host, Port, ...), % this would >> create and return an SSH connection, a gen_server would do or >> something. >> >> {ok, ChanSup} = supervisor:start_link({local, chan_sup}, >> chan_sup_callback, []), >> >> {ok, Chan1} = supervisor:start_child(ChanSup, [Connection]), >> {ok, Chan2} = supervisor:start_child(ChanSup, [Connection]), >> ... >> >> >> >> Ofcourse ChanSup is a simple_one_for_one supervisor with an init/1 >> like: >> >> >> >> init([]) -> >> {ok, { >> {simple_one_for_one, 10, 10}, >> [{channel,{channel, start_link, []}, >> transient, 5000, worker, [channel]}]}}. >> >> >> >> I would also setup monitors b/w the Connection and the Channels, so if >> the connection dies, say if its closed, the channels would all die >> along with it. The connection's supervisor would be responsible for >> bringing up this entire tree after disaster has struck. >> >>> >>> I have to think this kind of complex startup happens frequently enough >>> that >>> best practices have been established. Or perhaps these situations are >>> actually rare? >> >> Dunno ;) >> >>> >>> BTW, my gut feeling is that I should leave supervisor init() alone and >>> have >>> a worker perform complex startup. >> >> I don't really do much in my supervisor inits, sometimes in my top >> level supervisors, I read config files, etc, but that's about it. Not >> much more. That's me though... others more experienced than me may >> have different ways of using supervisor:init/1 >> >> cheers, >> Essien >>> >>> Thanks, >>> Dan. >>> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From norton@REDACTED Mon Sep 27 16:28:02 2010 From: norton@REDACTED (Joseph Wayne Norton) Date: Mon, 27 Sep 2010 23:28:02 +0900 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: Message-ID: Dan - It depends on your application and needs, but I'd put this type of setup work in your application's start/2 callback before starting your application's supervisor(s). On Mon, 27 Sep 2010 23:15:00 +0900, Daniel Goertzen wrote: > I've read the documentation on supervision and have seen a few tutorials, > but they don't seem to move beyond the core concepts. For example, what > happens if you want to check and optionally setup an mnesia schema during > startup...where should this code go? In the supervisor init() or > start_link() function? Should I have my supervisor create a worker > process > whole sole job is to do this kind of setup and then dynamically add other > workers (or supervisors) to the supervisor with start_child()? > > I have this feeling there are already idiomatic ways of doing this, and > I'm > reinventing the wheel poorly and incompletely. > > Any advice appreciated. Thanks, > Dan. -- norton@REDACTED From ebegumisa@REDACTED Mon Sep 27 20:05:56 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 04:05:56 +1000 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: Hi again Ulf, It's great to get 'the guy' on this subject online so I'm going to take full advantage and ask two more questions that have been dogging me... Firstly, is there an open-source project you know of that uses included-applications and/or start phases properly that I could take a peek at? Maybe in OTP source itself? Secondly, I've always liked the idea of using included applications not necessarily for start phases but as a delayed/start-on-demand mechanism (taking advantage of the fact that included apps are automatically loaded but not started.) That is, manually calling application:start(foo) only if a particular feature of my app is used. But I have one query that made attempts for such use short-lived... the fact that an application can only be included by one other application. I think this limitation makes it harder to use included apps and start phases especially if you're using apps that are not in-house. For example, lets say CouchDB starts using mnesia (ok that's dumb but...) and decide to start it up using start phases (and therefore add it as an included application in couch.app) Then I have my FunkyApp that's been using mnesia too as included application. I then decide to use CouchDB for a new funky feature of FunkyApp. Now things break because mnesia is being used by both FunkyApp and CouchDB. To fix this, I not only have to modify my in-house app I have to modify the out-house CouchDB too. Is there an obvious fix to this I've been missing? - Edmond - On Tue, 28 Sep 2010 03:19:29 +1000, Ulf Wiger wrote: On 27 Sep 2010, at 18:14, Edmond Begumisa wrote: Ulf, I've been doing such initialisation in the init function of a worker manager process. Using Daniel's example, I might have a gen_server child of the main supervisor called db_mgr and set up the mnesia schema in db_mgr:init Have I been doing the 'wrong' thing OTP-wise? Not necessarily, but my personal preference is to cleanly separate setup code from application startup. This is in part because I used to work on a very complex product, where the setup was decidedly non-trivial, and the startup process had to be optimised in several steps. Still, even there, I believe that the setup logic was bootstrapped into the startup phase, but the code was still kept cleanly separated. The only thing that was part of the startup was a simple check to see if the setup code had been run. BR, Ulf W - Edmond - On Tue, 28 Sep 2010 00:31:47 +1000, Ulf Wiger wrote: On 27/09/2010 16:15, Daniel Goertzen wrote: I've read the documentation on supervision and have seen a few tutorials, but they don't seem to move beyond the core concepts. For example, what happens if you want to check and optionally setup an mnesia schema during startup...where should this code go? In the supervisor init() or start_link() function? Should I have my supervisor create a worker process whole sole job is to do this kind of setup and then dynamically add other workers (or supervisors) to the supervisor with start_child()? I strongly recommend doing that sort of thing in a separate procedure, rather than in the startup phase. If you want your application to be able to bootstrap itself, I would suggest that you either: - create a special application that runs before your other apps, and verifies that the installation is ok. To this end, it might be useful to know that you can pre-sort the .rel file. The systools lib will only change the sort order if needed to respect start dependencies. - Introduce start_phases, then do minimal work in the init function, and push the rest to functions that are called from start phase hooks. This also has the advantage that you know that your processes are all started and ready to respond during the init phase. Start phases are documented in http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3 BR, Ulf W -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Sep 27 20:15:13 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 28 Sep 2010 04:15:13 +1000 Subject: [erlang-questions] Compiling Erlang Interface with VC++ 2010 - LIB Issues In-Reply-To: <001901cb5e6e$071c74b0$15555e10$@rock-technologies.com> References: <000301cb5c06$addf68d0$099e3a70$@rock-technologies.com> <000901cb5e58$ffb1b380$ff151a80$@rock-technologies.com> <001901cb5e6e$071c74b0$15555e10$@rock-technologies.com> Message-ID: Well, I suppose that'll do it too... assuming you won't miss the debug symbols for you ei program. In which case you're a much better C/C++ programmer than I am! - Edmond - On Tue, 28 Sep 2010 04:01:34 +1000, wrote: > With setting Generate Debug Info to No under Project Propeties > > Configuration Properties > Linker > Debugging > > It works. > > Thank you so far. > > Klaus > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On > Behalf Of Edmond Begumisa > Sent: Monday, September 27, 2010 5:54 PM > To: k.rock@REDACTED > Cc: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Compiling Erlang Interface with VC++ > 2010 - > LIB Issues > > You could just ignore those warnings (it's finally linking after all.) > > But to fix those (I hate long warnings too), you can try either... > > a) Turning off incremental linking and debugging > b) Enabling the symbol server (so it can fetch those older VC pdb files) > > - Edmond - > > > On Tue, 28 Sep 2010 01:31:02 +1000, wrote: > >> Hi Edmond, >> >> The Result. >> >> 1>erl_interface.lib(erl_error.o) : warning LNK4099: PDB 'vc80.pdb' was >> not >> found with 'erl_interface.lib(erl_error.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>erl_interface.lib(erl_eterm.o) : warning LNK4099: PDB 'vc80.pdb' was >> not >> found with 'erl_interface.lib(erl_eterm.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>erl_interface.lib(erl_fix_alloc.o) : warning LNK4099: PDB 'vc80.pdb' >> was >> not found with 'erl_interface.lib(erl_fix_alloc.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>erl_interface.lib(erl_malloc.o) : warning LNK4099: PDB 'vc80.pdb' >> 1>was >> not >> found with 'erl_interface.lib(erl_malloc.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>erl_interface.lib(erl_marshal.o) : warning LNK4099: PDB 'vc80.pdb' >> 1>was >> not >> found with 'erl_interface.lib(erl_marshal.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(ei_resolve.o) : warning LNK4099: PDB 'vc80.pdb' was not found >> with >> 'ei.lib(ei_resolve.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(decode_big.o) : warning LNK4099: PDB 'vc80.pdb' was not found >> with >> 'ei.lib(decode_big.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(decode_double.o) : warning LNK4099: PDB 'vc80.pdb' was not >> 1>found >> with 'ei.lib(decode_double.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(decode_long.o) : warning LNK4099: PDB 'vc80.pdb' was not >> 1>found >> with >> 'ei.lib(decode_long.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(ei_malloc.o) : warning LNK4099: PDB 'vc80.pdb' was not found >> with >> 'ei.lib(ei_malloc.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(ei_pthreads.o) : warning LNK4099: PDB 'vc80.pdb' was not >> 1>found >> with >> 'ei.lib(ei_pthreads.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(get_type.o) : warning LNK4099: PDB 'vc80.pdb' was not found >> 1>with >> 'ei.lib(get_type.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1>ei.lib(ei_compat.o) : warning LNK4099: PDB 'vc80.pdb' was not found >> with >> 'ei.lib(ei_compat.o)' or at >> 'D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\vc80.pdb'; linking >> object as if no debug info >> 1> sqlite_port.vcxproj -> >> D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\sqlite_port.exe >> ========== Rebuild All: 1 succeeded, 0 failed, 0 skipped ========== >> >> >> -----Original Message----- >> From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] >> On Behalf Of Edmond Begumisa >> Sent: Monday, September 27, 2010 4:41 PM >> To: erlang-questions@REDACTED; k.rock@REDACTED >> Subject: Re: [erlang-questions] Compiling Erlang Interface with VC++ >> 2010 - >> LIB Issues >> >> It looks to me like the VS linker is having trouble with winsock... >> >> Try adding Ws2_32.lib to the link command line. >> >> - Edmond - >> >> On Sat, 25 Sep 2010 02:36:44 +1000, >> wrote: >> >>> Dear Erlang Questions, >>> >>> >>> When compiling an Erlang Interface with VC++ 2010 I get the >>> following compiler errors. >>> >>> >>> 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol >>> __imp__gethostbyname@REDACTED referenced in function _ei_gethostbyname >>> >>> 1>ei.lib(ei_resolve.o) : error LNK2019: unresolved external symbol >>> __imp__gethostbyaddr@REDACTED referenced in function _ei_gethostbyaddr >>> >>> 1>D:\Projects\C++_Exercises\SQLite\sqlite_port\Debug\sqlite_port.exe : >>> fatal >>> error LNK1120: 2 unresolved externals >>> >>> >>> Path set correctly to >>> >>> >>> C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\lib\ei.lib >>> >>> C:\Program Files (x86)\erl5.8.1\lib\erl_interface-3.7.1\include\ei.h >>> >>> >>> What is going wrong here. >>> >>> >>> Thank you. >>> >>> Rock InformationTechnologies (India) Pvt. >>> Dipl.-Ing.(FH) Klaus Rock >>> Bonhoefferstrasse 37 >>> 73432 Aalen - Germany >>> Tel : ++49-7367-96500 >>> Mail : k.rock@REDACTED >>> >> 0 0/mc/c ompose?to=k.rock@REDACTED> >>> >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From boris.muehmer@REDACTED Mon Sep 27 20:38:58 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Mon, 27 Sep 2010 20:38:58 +0200 Subject: [erlang-questions] How to analyse why (w)erl.exe is not starting? In-Reply-To: <2D16B184-02B1-4757-B14D-8E54172A398C@gmail.com> References: <2D16B184-02B1-4757-B14D-8E54172A398C@gmail.com> Message-ID: Well, thanks for all Your input... I just took some time, to have a closer look... 2010/9/26 Koener Antoine : > Look for 'depends.exe', this is the windows 'ldd' command. > You'll spot which dll is missing if any. Not a missing dll, but some interesting behaviour, when using the profile mode of Dependency Walker on "werl.exe": 1) R13B4/erl5.7.5: everything works (nothing new). 2) R14B/erl5.8.1/normal binary: profiling fails with MS/VC++: "The application has requested the Runtime to terminate in an unusual way." 3) R14B/erl5.8.1/debug binary (under erts-5.8.1/bin): profiling fails with Werl: "Unable to load emulator DLL (C:\PROGRA~2\ERL58~1.1\ERTS~1.1\ERTS-5~1.1\bin\beam.smp.dll)" 4) I reinstalled R14B using a short path (just under C:, nur under program files): no improvement I attached both the profile logs for WERL R13B4 and R14B. There are several interesting things in it: a) What's this (this also happens with the working R13B4): Loaded "NOT_AN_IMAGE" at address 0x7D4C0000. Cannot hook module. b) Another error (this also happens with the working R13B4): GetProcAddress(0x77BA0000 [MSVCRT.DLL], "_get_terminate") called from "MSVCR80.DLL" at address 0x7813447F and returned NULL. Error: The specified procedure could not be found (127). c) This is the death blow: DllMain(0x00A90000, DLL_PROCESS_ATTACH, 0x00000000) in "BEAM.SMP.DLL" returned 0 (0x0). d) This belogs to "c" I think LoadLibraryA("C:\PROGRA~2\ERL58~1.1\ERTS-5~1.1\bin\beam.smp.dll") returned NULL. Error: A dynamic link library (DLL) initialization routine failed (1114). I just wonder: what has changed from R13B4 to R14B to break it only on "that" system. "That" system is a "managed server" hosted by "1und1" using "Virtuozzo". Somehow I believe "Virtuozzo" may be the root of this problem. - boris -------------- next part -------------- A non-text attachment was scrubbed... Name: WERL-R13B4-profile.log Type: text/x-log Size: 57959 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: WERL-R14B-profile.log Type: text/x-log Size: 22844 bytes Desc: not available URL: From ivan060111ad@REDACTED Mon Sep 27 22:02:35 2010 From: ivan060111ad@REDACTED (Ivan Carmenates =?utf-8?Q?Garc=C3=ADa?=) Date: Mon, 27 Sep 2010 15:02:35 -0500 Subject: Hi all!, please take a look on this, OTP.Net and EVO (Extended Visual Otp) Message-ID: Hi all, my name is Ivan I?m from Cuba, I would like to promote Erlang in this country, I can say that we are so small community here, What I know we are just 3 persons that knows Erlang very much and it power, this is my second time I can join to this community, I love Erlang, actually I am developing a new component to communicate clients made in C# with Erlang servers, using the Otp.Net library of Vlad Dumitrescu but I had wrote new layers to make so more easy the programming, I call my component EVO (Extended Visual Otp)... a little example could be: this.erlangServerInterface.Connect(); With that we are connecting with the server in Erlang. The configuration is a visual one. There are no need to write code for that. And for requests you just have to write: this.erlangServerRequest.Request(?request description?, object message); The object message could be anything even an Otp.Erlang.Object object or c# object, for example: this.erlangServerRequest.Request(?say hello!!!?, new object[] {new Otp.Erlang.Atom("hello"), ?Oh definitely I want to say hello to this community?}); That?s the same to send a {hello, ?Oh definitely I want to say hello to this community?} message to Erlang. We can do any kind of combination for example: Person person1 = new Person(?Ivan?, ?26?, ?Cuba?) This.erlangServerRequest.Request(?insert person data?, new object[] {? ?insert_person? ?, person1} ); Is the same of {insert_person, person1} The class Person of course must be marked as [serializable] attribute But that is not all; to receive the replies the erlangServerRequest object has an OnReceive method. OnReceive(IServerReply _reply) { If (_reply.CSharpReply is Person) { MessageBox.Show((_reply.CSharpReply as Person).Name ); } } There are 3 kinds of request in EVO. A normal request that waits for request to be completed to return the control to the program, the async request that starts a new thread for the request then you can make others request at instant. And the sync request that waits for reply too. this.erlangServerRequest.RequestAsync(?description?, object msg); Or IRequestInfo req = this.erlangServerRequest.Request(?description?, object msg); IServerReply reply = req.WaitForReply(); You can even abort the request once it was made. For example: IRequestInfo req = this.erlangServerRequest.Request(?description?, object msg); req.AbortRequest (); That means that you have the power to destroy the process in the server that is resolving for your request. And of course ignore if there are any possibility of be too late to abort request in server, then ignore the reply on the client. The magic here is that you can have in the same client many connections to different servers and for each connection has many request handlers and for each request handlers has many request made of sync or async way. ErlangServerInterface interface1 = new ErlangServerInterface(); interface1.ServerAddress = ?localhost?; interface1.ServerPort = ?5800?; ? ? ? ErlangServerRequest requethandler1 = new ErlangServerRequest(); requesthandler1.ErlangServerInterface = interface1; requesthandler1.Request(?say hi?, ? ? hi? ?); requesthandler1.Request(?say hello?, ? ? hello? ?); you can make many request through the same requesthandler1 and have no confusion with the replies, because you can check that of this way: OnReceive(IServerReply _reply) { switch(_reply.Description) { case ?say hi?: // do what ever you want break; case ?say hello?: // do what ever you want here; break; } } In Erlang is so much easy and I had developed a super template to helps you write even more easy code that you can write in Erlang this could be: %% Request process code. loopRequest(Args)-> receive %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% USER FUNCTIONS TO MODIFY %% %% %% %% Request: {Msg, Pid, RequestInfo} %% %% Reply: {reply, Reply, RequestInfo} %% %% %% %% RequestInfo = {Description, Id, WasMade, Kind, RequestHandlerHash} %% %% %% %% NOTE: %% %% - You can use send_to_all(Description, Msg) function to send a message to all clients. %% %% - Do not use 'loopRequest(Args)' calls in any clause. %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% {hi, Pid, RequestInfo}-> io:format("message received: ~p~n", [Msg]), Pid ! {reply, "yeah thanks for ask", RequestInfo}; {{update_char, CharName, X, Y}, Pid, RequestInfo}-> send_to_all(update_char, {CharName, X, Y}); %%R = appname_db_module:update_char(CharName, X, Y); {get_chars, Pid, RequestInfo}-> Reply = appname_db_module:get_chars(), Pid ! {reply, Reply, RequestInfo}; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% TEMPLATE %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% {Msg, Pid, RequestInfo}-> Reply = %% Code Here. Msg, %% Replies to client. Pid ! {reply, Reply, RequestInfo}, %% If you want to send a message to all clients. send_to_all(description, Reply), %% Debug message. appname_debug_module:print(Args, reply, {Reply, Pid, RequestInfo}) end. That and so much more functionalities can be made with this component that I had called EVO. If there are any people interesting please can contact me in ivan060111ad@REDACTED From rzezeski@REDACTED Mon Sep 27 21:15:45 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Mon, 27 Sep 2010 15:15:45 -0400 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 1:00 PM, Ulf Wiger wrote: > > On 27 Sep 2010, at 18:25, Pierpaolo Bernardi wrote: > > > On Mon, Sep 27, 2010 at 18:21, Ryan Zezeski wrote: > > > >> This is, essentially, the question I'm asking. Should I prefer one over > the > >> other for *any* reason? Right now, I prefer universal_time because it's > >> less typing :) > > > > A perfectly good reason. It's also a tiny bit clearer. > > erlang:now() is actually a bit different from the calendar clock. > It can, by definition, never jump backwards. > > It will also attempt not to make large adjustments of any kind, so if it > detects > that there is a large difference between the system clock and the > erlang:now() > clock, the now() clock will speed up or slow down 1% in order to converge > with the actual time, without disturbing timeouts etc. which rely on now() > to > provide a smooth representation of system real-time. > > In other words, erlang:universal_time/0 will return the actual time, > whereas > erlang:now() may, under certain circumstances, differ quite significantly > from actual time. > > Thanks Ulf, but I was actually referring to calendar:universal_time/0. I did not know of the existence of erlang:universtaltime/0 until you pointed it out just now. Now I'm curious, what is the difference, if any, between these two? Is their existence just a case of history? Another question, both functions claim that they return UTC "if available" otherwise they return local time. In what case is UTC not available? -Ryan From pascalchapier@REDACTED Mon Sep 27 21:31:28 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Mon, 27 Sep 2010 21:31:28 +0200 Subject: [erlang-questions] Re: Debugger fault at startup. Message-ID: Hi Michael, I read this thread with interest. I noticed that the problem you have has not the same consequences I had: there is an assertion failed when I had a simple windows crash. I could be a different problem, or a different behavior due to the diverse environment (64bit vs 32, Hipe vs standard, Linux vs Windows ?). On my side i was able to launch the debugger and wxWindow using : "werl -detached -run my_module start -run debugger start" As it is not very convenient, I came back to the R14A which works fine on my system. I started to use wxWindow on this version and didn't tried another one: I am learning this environment, as a hobby, and also because I think it could be valuable in my company. But I'm still looking for an idea for a "punchy" demo :o) Maybe you could test these workarounds to confirm that it is the same problem. Best Regards Pascal. From zeno490@REDACTED Mon Sep 27 21:35:22 2010 From: zeno490@REDACTED (Nicholas Frechette) Date: Mon, 27 Sep 2010 15:35:22 -0400 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: While we are on the subject, I am in possession of a rare laptop. The motherboard is bad and while the computer is fully functional, the clock sometimes goes backwards (when running as well as when booting, or waking up from hibernation/sleep). I'm 100% positive it isn't a battery problem. If the clock goes backwards, could this affect adversely a running erlang application? You mentionned timers could get affected and erlang:now()? It can sometime reverse by a few minutes, sometimes hours... and one time it reversed several years (years before the physical manufacture of the laptop...). I've never really noticed bad behavior from winxp or ubuntu running on it aside from the few times where the time difference was so big, weird things would happen in respect with synchronization of applications. Now obviously I'm not running anything mission critical on that time traveling laptop but it would be nice to know the expected behavior for when I'm doing development work on it. On Mon, Sep 27, 2010 at 3:15 PM, Ryan Zezeski wrote: > On Mon, Sep 27, 2010 at 1:00 PM, Ulf Wiger > wrote: > > > > > On 27 Sep 2010, at 18:25, Pierpaolo Bernardi wrote: > > > > > On Mon, Sep 27, 2010 at 18:21, Ryan Zezeski > wrote: > > > > > >> This is, essentially, the question I'm asking. Should I prefer one > over > > the > > >> other for *any* reason? Right now, I prefer universal_time because > it's > > >> less typing :) > > > > > > A perfectly good reason. It's also a tiny bit clearer. > > > > erlang:now() is actually a bit different from the calendar clock. > > It can, by definition, never jump backwards. > > > > It will also attempt not to make large adjustments of any kind, so if it > > detects > > that there is a large difference between the system clock and the > > erlang:now() > > clock, the now() clock will speed up or slow down 1% in order to converge > > with the actual time, without disturbing timeouts etc. which rely on > now() > > to > > provide a smooth representation of system real-time. > > > > In other words, erlang:universal_time/0 will return the actual time, > > whereas > > erlang:now() may, under certain circumstances, differ quite significantly > > from actual time. > > > > > Thanks Ulf, but I was actually referring to calendar:universal_time/0. I > did not know of the existence of erlang:universtaltime/0 until you pointed > it out just now. Now I'm curious, what is the difference, if any, between > these two? Is their existence just a case of history? > > Another question, both functions claim that they return UTC "if available" > otherwise they return local time. In what case is UTC not available? > > -Ryan > From alexander.harju@REDACTED Mon Sep 27 22:50:52 2010 From: alexander.harju@REDACTED (Alexander Harju) Date: Mon, 27 Sep 2010 22:50:52 +0200 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 9:15 PM, Ryan Zezeski wrote: > On Mon, Sep 27, 2010 at 1:00 PM, Ulf Wiger > wrote: > > > > > On 27 Sep 2010, at 18:25, Pierpaolo Bernardi wrote: > > > > > On Mon, Sep 27, 2010 at 18:21, Ryan Zezeski > wrote: > > > > > >> This is, essentially, the question I'm asking. Should I prefer one > over > > the > > >> other for *any* reason? Right now, I prefer universal_time because > it's > > >> less typing :) > > > > > > A perfectly good reason. It's also a tiny bit clearer. > > > > erlang:now() is actually a bit different from the calendar clock. > > It can, by definition, never jump backwards. > > > > It will also attempt not to make large adjustments of any kind, so if it > > detects > > that there is a large difference between the system clock and the > > erlang:now() > > clock, the now() clock will speed up or slow down 1% in order to converge > > with the actual time, without disturbing timeouts etc. which rely on > now() > > to > > provide a smooth representation of system real-time. > > > > In other words, erlang:universal_time/0 will return the actual time, > > whereas > > erlang:now() may, under certain circumstances, differ quite significantly > > from actual time. > > > > > Thanks Ulf, but I was actually referring to calendar:universal_time/0. I > did not know of the existence of erlang:universtaltime/0 until you pointed > it out just now. Now I'm curious, what is the difference, if any, between > these two? Is their existence just a case of history? > > Another question, both functions claim that they return UTC "if available" > otherwise they return local time. In what case is UTC not available? > calendar:universal_time() calls erlang:universaltime() so the difference is just another function call. Regarding now(). You shouldn't call calendar:now_to_universaltime(now()). That doesn't make any sense at all. now() has to be unique, therefore it's MUCH slower. The only time it make sense to to use calendar:now_to_universaltime(...) is when you have a now-timestamp and want to convert it to universal-time. If you need a timestamp higher precision than seconds I recommend to write your own now-nif to get rid of the penalty of now(). // Alex > > -Ryan > From rzezeski@REDACTED Tue Sep 28 04:05:31 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Mon, 27 Sep 2010 22:05:31 -0400 Subject: quoted atoms in type specs Message-ID: Is there a reason to use a quoted atom rather than a bare atom in a type specification. E.g., in the reference manual is the following example: Atom :: atom() | Erlang_Atom %% 'foo', 'bar', ... I also notice this in a the module getopt in the rebar source code. -type arg_type() :: 'atom' | 'binary' | 'boolean' | 'float' | 'integer' | 'string'. Why are the atoms being quoted? I was able to compile a module where the type specification uses bare atoms. Why not declare it that way? -Ryan From ttmrichter@REDACTED Tue Sep 28 04:15:41 2010 From: ttmrichter@REDACTED (Michael Richter) Date: Tue, 28 Sep 2010 10:15:41 +0800 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: On 28 September 2010 03:31, Pascal Chapier wrote: > I read this thread with interest. > I noticed that the problem you have has not the same consequences I had: > there is an assertion failed when I had a simple windows crash. > I communicated poorly there. On LINUX I have the assertion failure show up. On Windows it just crashes exactly as you reported. > On my side i was able to launch the debugger and wxWindow using : > "werl -detached -run my_module start -run debugger start" > This did not work for me. Basically anything that uses wx crashes on impact in Windows and dies from an assertion failure in Linux. -- "Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot." --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. From steven.charles.davis@REDACTED Tue Sep 28 07:09:46 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Mon, 27 Sep 2010 22:09:46 -0700 (PDT) Subject: non-trivial supervisor idioms? In-Reply-To: <4CA0AAD3.3030209@erlang-solutions.com> References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: Hi Dan, Just some of my experiences that may help... I faced much the same issue(s), and also I wanted to keep application:start(myapp) and myapp:start() synonymous. I finalized on doing the startup validation in the app's start() function, then started the main supervisor only when the initialization/validation was completed. I defined mnesia/ crypto and other dependencies as included apps and started their own sups under the main app sup (as recommended http://www.erlang.org/doc/design_principles/included_applications.html). I experimented with start phases, but ran into issues when a phase failed during startup (http://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss: 1771:201003:maambghdccechjinphpl), so I abandoned that approach. Regards, /s From mazen.harake@REDACTED Tue Sep 28 08:05:17 2010 From: mazen.harake@REDACTED (Mazen Harake) Date: Tue, 28 Sep 2010 09:05:17 +0300 Subject: [erlang-questions] Re: non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: <4CA1859D.7030309@erlang-solutions.com> On 28/09/2010 08:09, Steve Davis wrote: > I defined mnesia/ > crypto and other dependencies as included apps and started their own > sups under the main app sup (as recommended > http://www.erlang.org/doc/design_principles/included_applications.html). I'm not sure this is a good idea imho, I think included applications are meant to be used for your own specific applications only and not the generic ones like mnesia, crypto etc. I could be wrong but doing it the way you described just feels wrong. The way I understand included applications is that lets say you build application A1.0 and you want to generalize some part of that and move that part into its own application B1.0. Application B in this case doesn't make sense on its own (I.e. you remove application A) but it does make sense in the way that if you want to build a new A, and make it A2.0 then B1.0 could still apply (perhaps it implements some core protocol or what ever). In THIS case it would make sense to have application B as an included application and not as a primary application. However, I usually start all applications as primary applications and specify dependencies instead. I make the assumption/argument that if Application B can't start up then there is no point for A to start thus the normal boot up sequence is most of the time (I'd say 98%) good enough. The way I usually handle the scenario OP mentioned: always make sure data, configuration, mnesia tables exist and are sane in the start function of my application (myapp:start/2). You can spread out the various test functions in their respective modules to encapsulate the functionality (E.g. myapp_data:check_fix(), myapp_conf:validate(), myapp_db:ensure_tables() etc...) but essentially it is all tested from the application's start function. I _never_ start other applications in my application... I believe this is unnecessary and that it makes the release centered around one application and I rarely write a system where I have a release with only one application. I have personally very little experience with start phases but I have never come across any situation that couldn't be solved in a (conceptually) equally good way. My 2 cents :) /Mazen From kostis@REDACTED Tue Sep 28 08:13:03 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 28 Sep 2010 09:13:03 +0300 Subject: [erlang-questions] quoted atoms in type specs In-Reply-To: References: Message-ID: <4CA1876F.7030701@cs.ntua.gr> Ryan Zezeski wrote: > Is there a reason to use a quoted atom rather than a bare atom in a type > specification. E.g., in the reference manual is the following example: > > Atom :: atom() > | Erlang_Atom %% 'foo', 'bar', ... > > I also notice this in a the module getopt in the rebar source code. > > -type arg_type() :: 'atom' | 'binary' | 'boolean' | 'float' | 'integer' | > 'string'. > > Why are the atoms being quoted? I was able to compile a module where the > type specification uses bare atoms. Why not declare it that way? There is no deep reason; it's just a convention. Personally, I've been bitten once too many by omitting parentheses, e.g. writing: -type my_type() :: atom | integer. when I should have written: -type my_type() :: atom() | integer(). If one _really_ wants the former, it's IMO better to write it as: -type my_type() :: 'atom' | 'integer'. which makes it crystal clear that the atoms are there for a reason and it's not just that parentheses are missing. But you are right: the first and third type declarations in my reply are equivalent. Kostis From kenneth.lundin@REDACTED Tue Sep 28 08:44:41 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Tue, 28 Sep 2010 08:44:41 +0200 Subject: [erlang-questions] calendar now_to_universal_time/1 vs. universal_time/0 In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 10:50 PM, Alexander Harju wrote: > On Mon, Sep 27, 2010 at 9:15 PM, Ryan Zezeski wrote: > >> On Mon, Sep 27, 2010 at 1:00 PM, Ulf Wiger >> wrote: >> >> > >> > On 27 Sep 2010, at 18:25, Pierpaolo Bernardi wrote: >> > >> > > On Mon, Sep 27, 2010 at 18:21, Ryan Zezeski >> wrote: >> > > >> > >> This is, essentially, the question I'm asking. ?Should I prefer one >> over >> > the >> > >> other for *any* reason? ?Right now, I prefer universal_time because >> it's >> > >> less typing :) >> > > >> > > A perfectly good reason. ?It's also a tiny bit clearer. >> > >> > erlang:now() is actually a bit different from the calendar clock. >> > It can, by definition, never jump backwards. >> > >> > It will also attempt not to make large adjustments of any kind, so if it >> > detects >> > that there is a large difference between the system clock and the >> > erlang:now() >> > clock, the now() clock will speed up or slow down 1% in order to converge >> > with the actual time, without disturbing timeouts etc. which rely on >> now() >> > to >> > provide a smooth representation of system real-time. >> > >> > In other words, erlang:universal_time/0 will return the actual time, >> > whereas >> > erlang:now() may, under certain circumstances, differ quite significantly >> > from actual time. >> > >> > >> Thanks Ulf, but I was actually referring to calendar:universal_time/0. ?I >> did not know of the existence of erlang:universtaltime/0 until you pointed >> it out just now. ?Now I'm curious, what is the difference, if any, between >> these two? ?Is their existence just a case of history? >> >> Another question, both functions claim that they return UTC "if available" >> otherwise they return local time. ?In what case is UTC not available? >> > > calendar:universal_time() calls erlang:universaltime() so the difference is > just another function call. > > Regarding now(). > You shouldn't call calendar:now_to_universaltime(now()). That doesn't make > any sense at all. > now() has to be unique, therefore it's MUCH slower. > The only time it make sense to to use calendar:now_to_universaltime(...) is > when you have a now-timestamp and want to convert it to universal-time. > > If you need a timestamp higher precision than seconds I recommend to write > your own now-nif to get rid of the penalty of now(). If you need a timestamp with the same precision as now() but without the penalty of now() always returning increasing unique times and the compensation for sudden jumps in time you can use os:timestamp() which returns a value with the same format as now() but directly from the OS. This can be valuable if you for example want to put timestamps in logs and later compare with timestamps generated by other programs on the same machine. So there is no need to write your own NIF. os:timestamp/0 was introduced in June 2009 in the R13B01 release. /Kenneth Erlang/OTP Ericsson. > > // Alex > > >> >> -Ryan >> > From ulf.wiger@REDACTED Tue Sep 28 09:07:16 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Tue, 28 Sep 2010 09:07:16 +0200 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: On 27 Sep 2010, at 20:05, Edmond Begumisa wrote: > Hi again Ulf, > > It's great to get 'the guy' on this subject online so I'm going to take full advantage and ask two more questions that have been dogging me... > > Firstly, is there an open-source project you know of that uses included-applications and/or start phases properly that I could take a peek at? Maybe in OTP source itself? Off the top of my head, I really can't think of any. :) The area where start phases really come in handy is when your application needs to support failover/takeover behaviour. This is also when the StartType argument is needed. One can implement takeover by writing a special start phase that instructs the processes to take over processing from the other side. In general, it is best to do this at a point where all processes have been started and are ready to process incoming requests. The initial reason for start phases was that the complex call-handling applications at Ericsson had some pretty horrendous dependencies to sort out before they could start accepting calls, and doing this work in the init function of the processes simply wasn't feasible. Also, when a process dies in the init function, this is interpreted as a start error, and the application start will fail, whereas individual processes have proper supervision while they are responding to requests from the start phase code (which runs in the application_starter process). Included applications were mainly introduced since the same call-handling applications needed to move as one during failover and takeover, and starting a dozen or so top applications made that much more difficult. It was just too much code and too many modules to integrate into one single application without one more structuring layer. Initially, I wrote some code that read .appSrc files in each sub-application and integrated them into one larger application, using a top-level resource file - I think it had the extension .appLm (as in load module - never mind; it made sense at Ericsson, and it was so long ago that I may be remembering wrong). This was later generalised by OTP into included_applications. The O&M applications also had a problem during takeover: The snmp code assumed that the snmp agent was locally registered on the same node, which wasn't necessarily the case during the transition - either on the node taking over or on the node where it ran before. We then created a wrapper application that included all the O&M applications, and called the individual start functions for each included app. Later, we moved away from that, as we had to also support applications that were written according to a different timeline, and therefore couldn't be integrated the same way as our other apps. I came up with a solution for starting and stopping included apps and plugging in their start phase hooks in the right places in the startup flow, but for some reason people found it complicated... :) The better solution was to make use of the fact that the application controller now had a message passing interface for controlling the starting and stopping of apps. We were already using this in our cluster controller, so we could extend it by specifying distributed start dependencies and which applications needed to do takeover in parallel. This way, the cluster controller knew in which order to move applications during takeover, and in which order to terminate them, once migrated. Unfortunately, all this code is proprietary. It's on my long list of things I'd like to do, but that list just keeps growing, without much ever being removed from it... A long time ago, I made a prototype (and sent to OTP) that introduced start phase dependencies. This would IMHO make it much easier to specify dependencies between applications. As an example, mnesia loads tables in the background, so when the application:start() function returns, one cannot assume that tables are loaded, and has to call mnesia:wait_for_tables() (which can time out, and has some corner cases where tables will never be loaded without intervention - not that the function itself will tell you when they occur). It might be better if mnesia had a load_tables start phase, which other applications could depend on. BR, Ulf W > > Secondly, I've always liked the idea of using included applications not necessarily for start phases but as a delayed/start-on-demand mechanism (taking advantage of the fact that included apps are automatically loaded but not started.) That is, manually calling application:start(foo) only if a particular feature of my app is used. But I have one query that made attempts for such use short-lived... the fact that an application can only be included by one other application. I think this limitation makes it harder to use included apps and start phases especially if you're using apps that are not in-house. > > For example, lets say CouchDB starts using mnesia (ok that's dumb but...) and decide to start it up using start phases (and therefore add it as an included application in couch.app) Then I have my FunkyApp that's been using mnesia too as included application. I then decide to use CouchDB for a new funky feature of FunkyApp. Now things break because mnesia is being used by both FunkyApp and CouchDB. To fix this, I not only have to modify my in-house app I have to modify the out-house CouchDB too. > > Is there an obvious fix to this I've been missing? > > - Edmond - > > On Tue, 28 Sep 2010 03:19:29 +1000, Ulf Wiger wrote: > > > On 27 Sep 2010, at 18:14, Edmond Begumisa wrote: > > Ulf, > > I've been doing such initialisation in the init function of a worker manager process. Using Daniel's example, I might have a gen_server child of the main supervisor called db_mgr and set up the mnesia schema in db_mgr:init > > Have I been doing the 'wrong' thing OTP-wise? > > Not necessarily, but my personal preference is to cleanly separate setup code > from application startup. This is in part because I used to work on a very complex > product, where the setup was decidedly non-trivial, and the startup process had > to be optimised in several steps. > > Still, even there, I believe that the setup logic was bootstrapped into the startup > phase, but the code was still kept cleanly separated. The only thing that was > part of the startup was a simple check to see if the setup code had been run. > > BR, > Ulf W > > > > - Edmond - > > On Tue, 28 Sep 2010 00:31:47 +1000, Ulf Wiger wrote: > > On 27/09/2010 16:15, Daniel Goertzen wrote: > I've read the documentation on supervision and have seen a few tutorials, > but they don't seem to move beyond the core concepts. For example, what > happens if you want to check and optionally setup an mnesia schema during > startup...where should this code go? In the supervisor init() or > start_link() function? Should I have my supervisor create a worker process > whole sole job is to do this kind of setup and then dynamically add other > workers (or supervisors) to the supervisor with start_child()? > > I strongly recommend doing that sort of thing in a separate procedure, > rather than in the startup phase. > > If you want your application to be able to bootstrap itself, I would > suggest that you either: > > - create a special application that runs before your other apps, > and verifies that the installation is ok. To this end, it might be > useful to know that you can pre-sort the .rel file. The systools lib > will only change the sort order if needed to respect start > dependencies. > - Introduce start_phases, then do minimal work in the init function, > and push the rest to functions that are called from start phase > hooks. This also has the advantage that you know that your processes > are all started and ready to respond during the init phase. > > Start phases are documented in > http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3 > > BR, > Ulf W > > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From dangud@REDACTED Tue Sep 28 09:16:12 2010 From: dangud@REDACTED (Dan Gudmundsson) Date: Tue, 28 Sep 2010 09:16:12 +0200 Subject: [erlang-questions] Re: Debugger fault at startup. In-Reply-To: References: Message-ID: Hi et_viewer might be another problem, I did some last minute patches to make it run at all on windows. It was taking some shortcuts that only worked on linux. It's not much I can do without some more info, wx works here, both on linux and windows. I have not made any changes that should make it start causing those troubles, I have added new functionality in R14B but that shouldn't be called until the user are using those functions from erlang, i.e. the initialization part is the same as before. /Dan On Mon, Sep 27, 2010 at 5:44 PM, Edmond Begumisa wrote: > I observed haven't any problems with debugger either (R14B -- Mac OS X > 10.5.8 and Windows 7) > > But I am forced to use the et_viewer in gs mode coz it hangs in wx mode on > win7. So there might be a problem with wxErlang. > > - Edmond - > > > On Mon, 27 Sep 2010 21:56:45 +1000, Attila Rajmund Nohl > wrote: > >> I've just tested on R14B on SuSE 10 and the wx demo works fine... My >> gut feeling is that your build was not properly built. Or you might >> have a hardware error. >> >> 2010/9/27, Michael Richter : >>> >>> After reading Pascal Chapier's message, I tried something else. ?From a >>> fresh, unaltered build of R14B on a pretty bog-standard Ubuntu 10.04 >>> installation typing "wx:new()." in the shell generates precisely the same >>> crash message. ?This leads me to believe that wx is badly broken in R14B. >>> ?Trying either "wx:new()." or "debugger:start()." in R14B on a completely >>> standard Windows XP system causes the VM to crash instantly, whether run >>> as >>> werl.exe or erl.exe. ?I happen to also have R13B02 on that machine and >>> the >>> same problem is exhibited there. >>> >>> It looks to me like wx support is really badly broken and has been for >>> some >>> time. >>> >>> On 27 September 2010 12:22, Michael Richter wrote: >>> >>>> So, for the first time I find myself having to fire up the debugger in >>>> Erlang. ?Following the user manual, I do this: >>>> >>>> $ erl >>>> >>>> Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] >>>>> >>>>> [async-threads:0] [hipe] [kernel-poll:true] >>>> >>>> >>>>> Eshell V5.8.1 ?(abort with ^G) >>>> >>>> 1> debugger:start(). >>>> >>>> beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) >>>>> >>>>> (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct >>>>> malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >>>>> >= >>>>> (unsigned long)((((__builtin_offsetof (struct malloc_chunk, >>>>> fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) >>>>> - >>>>> 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) >>>>> == >>>>> 0)' failed. >>>> >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Aborted >>>> >>>> $ >>>> >>>> >>>> I don't even know where to begin here. ?Is there any expert on the >>>> debugger's inner workings that knows both what that means and what can >>>> be >>>> done about it? >>>> >>>> -- >>>> "Perhaps people don't believe this, but throughout all of the >>>> discussions >>>> of entering China our focus has really been what's best for the Chinese >>>> people. It's not been about our revenue or profit or whatnot." >>>> --Sergey Brin, demonstrating the emptiness of the "don't be evil" >>>> mantra. >>>> >>> >>> >>> >>> -- >>> "Perhaps people don't believe this, but throughout all of the discussions >>> of >>> entering China our focus has really been what's best for the Chinese >>> people. >>> It's not been about our revenue or profit or whatnot." >>> --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. >>> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From krab@REDACTED Tue Sep 28 10:26:15 2010 From: krab@REDACTED (Kresten Krab Thorup) Date: Tue, 28 Sep 2010 10:26:15 +0200 Subject: [erlang-questions] non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> Message-ID: <5C96E84C-BC1C-486E-93C3-8BCE0E95D881@trifork.com> On Sep 28, 2010, at 9:07 , Ulf Wiger wrote: >> Firstly, is there an open-source project you know of that uses included-applications and/or start phases properly that I could take a peek at? Maybe in OTP source itself? > RabbitMQ inits/starts mnesia as part of it's startup, so perhaps you can draw inspiration from there. I have no idea if they do it "the right OTP way" though. Kresten Krab Thorup, CTO, Trifork From dionne@REDACTED Tue Sep 28 12:40:50 2010 From: dionne@REDACTED (Robert Dionne) Date: Tue, 28 Sep 2010 06:40:50 -0400 Subject: beam.smp crashes OS X Message-ID: I'm curious if others are seeing beam.smp crash OS X ? I'm running R13B04 on OSX 10.6 I thought it was sporadic at first and rare, but it turns out to be very reproducible. By killing a terminal window where an erl shell is running or Aquamacs it happens. I've sent the crash report to Apple. The obvious workaround is to make sure erl is exited. It's like slamming a car door on your finger, "well just don't do that" Cheers, Bob From dizzyd@REDACTED Tue Sep 28 14:09:22 2010 From: dizzyd@REDACTED (Dave Smith) Date: Tue, 28 Sep 2010 06:09:22 -0600 Subject: [erlang-questions] beam.smp crashes OS X In-Reply-To: References: Message-ID: Yup this is a known bug to Apple. Drives me nuts. D. On Tue, Sep 28, 2010 at 4:40 AM, Robert Dionne wrote: > I'm curious if others are seeing beam.smp crash OS X ? > > I'm running R13B04 on OSX 10.6 ?I thought it was sporadic at first and rare, but it turns out to be very reproducible. By killing a terminal window where an erl shell is running or Aquamacs it happens. > > I've sent the crash report to Apple. The obvious workaround is to make sure erl is exited. It's like slamming a car door on your finger, "well just don't do that" > > Cheers, > > Bob > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From steven.charles.davis@REDACTED Tue Sep 28 15:18:21 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Tue, 28 Sep 2010 06:18:21 -0700 (PDT) Subject: non-trivial supervisor idioms? In-Reply-To: <4CA1859D.7030309@erlang-solutions.com> References: <4CA0AAD3.3030209@erlang-solutions.com> <4CA1859D.7030309@erlang-solutions.com> Message-ID: Hi Mazen, I understand your rationale, but I'm not so sure it's a cause for concern. The node/application is entirely dependent on the availability of those "generic" apps. By putting all parts of the app under one supervisor I have a situation where any issues will allow the node to behave in a controlled way. I believe that the approach of a single top-level supervisor for the app and dependencies is definitely an "OTP" thing to do. Possibly the only drawback is that it uses undocumented APIs (the sups) from libraries in the platform. However, I'm interested in further thoughts from yourself or others on this point. Regards, Steve On Sep 28, 1:05?am, Mazen Harake wrote: > I'm not sure this is a good idea imho, I think included applications are > meant to be used for your own specific applications only and not the > generic ones like mnesia, crypto etc. I could be wrong but doing it the > way you described just feels wrong. From mark@REDACTED Tue Sep 28 19:44:12 2010 From: mark@REDACTED (Mark Scandariato) Date: Tue, 28 Sep 2010 13:44:12 -0400 Subject: VM segfault on exit with wx Message-ID: Seems like the VM segfaults on exit whenever I run anything that uses wx (this is a single processor system). Let me know if you need any other info (or a core file or whatever). Mark. erlang@REDACTED:~/otp_src_R14B$ uname -a Linux ubuntu 2.6.28-19-generic #65-Ubuntu SMP Thu Sep 16 14:14:28 UTC 2010 i686 GNU/Linux erlang@REDACTED:~/otp_src_R14B$ bin/cerl -debug -smp enable Erlang R14B (erts-5.8.1) [source] [smp:1:1] [rq:1] [async-threads:0] [hipe] [kernel-poll:false] [type-assertions] [debug-compiled] [lock-checking] Eshell V5.8.1 (abort with ^G) 1> wx:demo(). {wx_ref,35,wxFrame,<0.34.0>} 2> q(). ok 3> Segmentation fault (core dumped) erlang@REDACTED:~/otp_src_R14B$ gdb --core=core --se=bin/i686-pc-linux-gnu/beam.debug.smp GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu"... warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/tls/i686/cmov/libutil.so.1...done. Loaded symbols for /lib/tls/i686/cmov/libutil.so.1 Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libdl.so.2 Reading symbols from /lib/tls/i686/cmov/libm.so.6...done. Loaded symbols for /lib/tls/i686/cmov/libm.so.6 Reading symbols from /lib/libncurses.so.5...done. Loaded symbols for /lib/libncurses.so.5 Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done. Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0 Reading symbols from /lib/tls/i686/cmov/librt.so.1...done. Loaded symbols for /lib/tls/i686/cmov/librt.so.1 Reading symbols from /lib/tls/i686/cmov/libc.so.6...done. Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libz.so.1...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libgtk-x11-2.0.so.0...done. Loaded symbols for /usr/lib/libgtk-x11-2.0.so.0 Reading symbols from /usr/lib/libgdk-x11-2.0.so.0...done. Loaded symbols for /usr/lib/libgdk-x11-2.0.so.0 Reading symbols from /usr/lib/libatk-1.0.so.0...done. Loaded symbols for /usr/lib/libatk-1.0.so.0 Reading symbols from /usr/lib/libpangoft2-1.0.so.0...done. Loaded symbols for /usr/lib/libpangoft2-1.0.so.0 Reading symbols from /usr/lib/libgdk_pixbuf-2.0.so.0...done. Loaded symbols for /usr/lib/libgdk_pixbuf-2.0.so.0 Reading symbols from /usr/lib/libgio-2.0.so.0...done. Loaded symbols for /usr/lib/libgio-2.0.so.0 Reading symbols from /usr/lib/libpango-1.0.so.0...done. Loaded symbols for /usr/lib/libpango-1.0.so.0 Reading symbols from /usr/lib/libfreetype.so.6...done. Loaded symbols for /usr/lib/libfreetype.so.6 Reading symbols from /usr/lib/libfontconfig.so.1...done. Loaded symbols for /usr/lib/libfontconfig.so.1 Reading symbols from /usr/lib/libgobject-2.0.so.0...done. Loaded symbols for /usr/lib/libgobject-2.0.so.0 Reading symbols from /usr/lib/libgmodule-2.0.so.0...done. Loaded symbols for /usr/lib/libgmodule-2.0.so.0 Reading symbols from /usr/lib/libgthread-2.0.so.0...done. Loaded symbols for /usr/lib/libgthread-2.0.so.0 Reading symbols from /usr/lib/libglib-2.0.so.0...done. Loaded symbols for /usr/lib/libglib-2.0.so.0 Reading symbols from /usr/lib/libXinerama.so.1...done. Loaded symbols for /usr/lib/libXinerama.so.1 Reading symbols from /usr/lib/libpng12.so.0...done. Loaded symbols for /usr/lib/libpng12.so.0 Reading symbols from /usr/lib/libexpat.so.1...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /usr/lib/libXext.so.6...done. Loaded symbols for /usr/lib/libXext.so.6 Reading symbols from /usr/lib/libXrender.so.1...done. Loaded symbols for /usr/lib/libXrender.so.1 Reading symbols from /usr/lib/libXi.so.6...done. Loaded symbols for /usr/lib/libXi.so.6 Reading symbols from /usr/lib/libXrandr.so.2...done. Loaded symbols for /usr/lib/libXrandr.so.2 Reading symbols from /usr/lib/libXcursor.so.1...done. Loaded symbols for /usr/lib/libXcursor.so.1 Reading symbols from /usr/lib/libpangocairo-1.0.so.0...done. Loaded symbols for /usr/lib/libpangocairo-1.0.so.0 Reading symbols from /usr/lib/libX11.so.6...done. Loaded symbols for /usr/lib/libX11.so.6 Reading symbols from /usr/lib/libXcomposite.so.1...done. Loaded symbols for /usr/lib/libXcomposite.so.1 Reading symbols from /usr/lib/libXdamage.so.1...done. Loaded symbols for /usr/lib/libXdamage.so.1 Reading symbols from /usr/lib/libXfixes.so.3...done. Loaded symbols for /usr/lib/libXfixes.so.3 Reading symbols from /usr/lib/libcairo.so.2...done. Loaded symbols for /usr/lib/libcairo.so.2 Reading symbols from /lib/libpcre.so.3...done. Loaded symbols for /lib/libpcre.so.3 Reading symbols from /lib/libselinux.so.1...done. Loaded symbols for /lib/libselinux.so.1 Reading symbols from /usr/lib/libXau.so.6...done. Loaded symbols for /usr/lib/libXau.so.6 Reading symbols from /usr/lib/libxcb.so.1...done. Loaded symbols for /usr/lib/libxcb.so.1 Reading symbols from /usr/lib/libpixman-1.so.0...done. Loaded symbols for /usr/lib/libpixman-1.so.0 Reading symbols from /usr/lib/libdirectfb-1.0.so.0...done. Loaded symbols for /usr/lib/libdirectfb-1.0.so.0 Reading symbols from /usr/lib/libfusion-1.0.so.0...done. Loaded symbols for /usr/lib/libfusion-1.0.so.0 Reading symbols from /usr/lib/libdirect-1.0.so.0...done. Loaded symbols for /usr/lib/libdirect-1.0.so.0 Reading symbols from /usr/lib/libxcb-render-util.so.0...done. Loaded symbols for /usr/lib/libxcb-render-util.so.0 Reading symbols from /usr/lib/libxcb-render.so.0...done. Loaded symbols for /usr/lib/libxcb-render.so.0 Reading symbols from /usr/lib/libXdmcp.so.6...done. Loaded symbols for /usr/lib/libXdmcp.so.6 Reading symbols from /usr/lib/gconv/UTF-32.so...done. Loaded symbols for /usr/lib/gconv/UTF-32.so Reading symbols from /lib/tls/i686/cmov/libnss_compat.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_compat.so.2 Reading symbols from /lib/tls/i686/cmov/libnsl.so.1...done. Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1 Reading symbols from /lib/tls/i686/cmov/libnss_nis.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_nis.so.2 Reading symbols from /lib/tls/i686/cmov/libnss_files.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_files.so.2 Reading symbols from /usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so...done. Loaded symbols for /usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so Reading symbols from /usr/lib/libcanberra-gtk.so.0...done. Loaded symbols for /usr/lib/libcanberra-gtk.so.0 Reading symbols from /usr/lib/libcanberra.so.0...done. Loaded symbols for /usr/lib/libcanberra.so.0 Reading symbols from /usr/lib/libvorbisfile.so.3...done. Loaded symbols for /usr/lib/libvorbisfile.so.3 Reading symbols from /usr/lib/libvorbis.so.0...done. Loaded symbols for /usr/lib/libvorbis.so.0 Reading symbols from /usr/lib/libogg.so.0...done. Loaded symbols for /usr/lib/libogg.so.0 Reading symbols from /usr/lib/libtdb.so.1...done. Loaded symbols for /usr/lib/libtdb.so.1 Reading symbols from /usr/lib/libltdl.so.7...done. Loaded symbols for /usr/lib/libltdl.so.7 Reading symbols from /usr/lib/gtk-2.0/2.10.0/engines/libmurrine.so...done. Loaded symbols for /usr/lib/gtk-2.0/2.10.0/engines/libmurrine.so Reading symbols from /usr/lib/pango/1.6.0/modules/pango-basic-fc.so...done. Loaded symbols for /usr/lib/pango/1.6.0/modules/pango-basic-fc.so Core was generated by `/home/erlang/otp_src_R14B/bin/i686-pc-linux-gnu/beam.debug.smp -- -root /home/e'. Program terminated with signal 11, Segmentation fault. [New process 3898] [New process 3897] [New process 3869] [New process 3894] [New process 3895] [New process 3896] #0 0xb410c974 in ?? () from /lib/libselinux.so.1 (gdb) where #0 0xb410c974 in ?? () from /lib/libselinux.so.1 #1 0xb4105b8e in ?? () from /lib/libselinux.so.1 #2 0xb40fe1d8 in ?? () from /lib/libselinux.so.1 #3 0xb410e350 in _fini () from /lib/libselinux.so.1 #4 0xb7802a83 in ?? () from /lib/ld-linux.so.2 #5 0xb7628bb9 in exit () from /lib/tls/i686/cmov/libc.so.6 #6 0x0809f9b0 in erl_exit (n=0, fmt=0x827ea3b "") at beam/erl_init.c:1575 #7 0x080edb6b in halt_0 (A__p=0xb748b174) at beam/bif.c:3396 #8 0x081bf1a5 in process_main () at beam/beam_emu.c:2170 #9 0x0810b12d in sched_thread_func (vesdp=0xb73d5900) at beam/erl_process.c:3635 #10 0x0826b969 in thr_wrapper (vtwd=0xbffa7190) at pthread/ethread.c:106 #11 0xb776d4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #12 0xb76de5ee in clone () from /lib/tls/i686/cmov/libc.so.6 (gdb) From mazen.harake@REDACTED Wed Sep 29 08:25:14 2010 From: mazen.harake@REDACTED (Mazen Harake) Date: Wed, 29 Sep 2010 09:25:14 +0300 Subject: [erlang-questions] Re: non-trivial supervisor idioms? In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> <4CA1859D.7030309@erlang-solutions.com> Message-ID: <4CA2DBCA.6010503@erlang-solutions.com> Hi Steve, I haven't personally come across a situation where I thought to my self "I need to use included applications". After many discussions over these sort of things in the company (related to release handling really) I have more or less settled on one way of doing things. It seems to work in all situations so far and is well suited for maintenance, release upgrades and fail-over scenarios. Without going into a mile long post and forcing myself to write a TLDR in the end I usually do things this way: All applications are attached one level under the main top supervisor; no included applications, only dependencies. Each application is either a service, glue/logic or library application. A service application is mnesia for example; using mnesia you _request_ it to create a table... etc. The library applications are just that libraries, _usually_ static with no processes. Glue/Logic applications are the ones which are actually the crux here. I usually design the applications in a way that they are tolerant to restarts at any point I.e: You should be able to do application:stop(App) and the chain of events should stop waiting for the application to come back up again. Ok so with this in mind imagine this scenario: I start my application and it has mnesia in included_applications. my application then starts mnesia, mnesia initializes its tables and returns, my application then checks that all the tables are there and continue. In this case, if you do application:stop(App) you shut down mnesia. All this works if the release is centered around my application, that is I only have 1 which is the "master application" and that imo is wrong. Note that in any case you check that tables are available and not corrupt or what ever you need to check. Now if you want to build another application or extend/maintain/fix/whatever then you have to move mnesia back out again (because you can't have another application relying on my application to keep mnesia up). This in turns forces me to change my application to make sure to check for mnesia is running and have tables at start up (which I already have anyway). Now if I skip the first step and always do my design in the latter mentioned way (because change always comes) I will have saved a part of re-factoring (and also part of refactoring the design). I can add App2 in the next release without worrying too much about my first app being abused by to many changes. Also if let's say mnesia crashes, then both my apps crash, but since you design your apps to be able to crash at any point (Take inconsistent states into account when bouncing up) they should be fine because they will be prepared on startup to wait for resources/services (like mnesia). So in the end it is a win-win (imho) to never use included_applications :) I'm not claiming this is the *most* "OTP"-way to do it or that any other way is necessarily wrong but some things in OTP you don't use that often and I haven't found a need to use this. (btw I would love to hear from someone who has experience with distributed applications, how did that work out!?) You write that "I have a situation where any issues will allow the node to behave in a controlled way" but is it in a situation where the applications which are included are only used by your main application? I'm curious in what you see is the gain over having the application processes "flat" under the top application supervisor. /Mazen On 28/09/2010 16:18, Steve Davis wrote: > Hi Mazen, > > I understand your rationale, but I'm not so sure it's a cause for > concern. The node/application is entirely dependent on the > availability of those "generic" apps. By putting all parts of the app > under one supervisor I have a situation where any issues will allow > the node to behave in a controlled way. I believe that the approach of > a single top-level supervisor for the app and dependencies is > definitely an "OTP" thing to do. Possibly the only drawback is that it > uses undocumented APIs (the sups) from libraries in the platform. > However, I'm interested in further thoughts from yourself or others on > this point. > > Regards, > Steve > > On Sep 28, 1:05 am, Mazen Harake > wrote: >> I'm not sure this is a good idea imho, I think included applications are >> meant to be used for your own specific applications only and not the >> generic ones like mnesia, crypto etc. I could be wrong but doing it the >> way you described just feels wrong. > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From mikkelgj@REDACTED Wed Sep 29 11:13:52 2010 From: mikkelgj@REDACTED (Mikkel Jensen) Date: Wed, 29 Sep 2010 11:13:52 +0200 Subject: [erlang-questions] Common test issues - Possible bug? In-Reply-To: <1285586344.16569.27.camel@ancalagon.du.uab.ericsson.se> References: <1285586344.16569.27.camel@ancalagon.du.uab.ericsson.se> Message-ID: Hi Lukas First of all, thanks for your response! I have tried using both run_test and erlang shell to start the tests, both gives me error i described, although the erlang shell usually seems like its able to execute more repeated tests than the run_test command. This is the first test case i'm building with Common Test - I have not made any other test cases, so i have not experienced this in any other context. Only when using ct_ssh. I have been trying to narrow down the test case to find specificly what is causing the errors, but i have been unable to. However the kinds of tests im running are not too advanced. I have made a test-case which imitates the behavior of my "real" test case, just instead of using cnee for imitating user input, the test subject is a python script which can be controlled directly from the console. In this python-test, i have not been able to get the error, not even when repeating the test 500 times or more. I hope you can find the time to go through my not-so-minimal test case. As you can probably see from the code, i am not the most experienced erlang programmer, and the lack of websites/blogs describing Common Test best practices and examples makes it very hard to know what i'm doing wrong. The user guide is great, but can sometimes be a bit hard to understand. Real test case: http://ideone.com/v6vvu Python test case: http://ideone.com/GcxR2 %%% CODE FOR "REAL" TEST CASE %%% %%%%%%%%%%%%%%%%%%%%%%%% -module(migration_SUITE). -compile(export_all). -include_lib("common_test/include/ct.hrl"). % Require the following tags before executing the suite. % The tags are provided by a config file suite() -> [{require, hidenets}, {require, demo}, {require, node1}, {require, node2}, {require, testserver}]. init_per_suite(Config) -> % Start ssh connections {ok, Hidenets} = ct_ssh:connect(hidenets), {ok, Demo} = ct_ssh:connect(demo), {ok, Node1} = ct_ssh:connect(node1), {ok, Node2} = ct_ssh:connect(node2), {ok, Testserver} = ct_ssh:connect(testserver), % Append handles to config % OBS: This is not actually used, because of the disconnect issue [{ssh_handles,{Hidenets,Demo,Node1,Node2,Testserver}} | Config]. end_per_suite(Config) -> % Disconnect from all handles Handles = erlang:tuple_to_list(?config(ssh_handles, Config)), lists:foreach(fun(Handle) -> ct_ssh:disconnect(Handle) end, Handles). init_per_group(_GroupName, Config) -> % Get ssh handles from config % {Hidenets, Demo, Node1, Node2, Testserver} = ?config(ssh_handles, Config), % Start context agent on hidenets (simple console application) runcmd(hidenets, "./ContextAgentLinux/ContextAgent"), % Start GUI macro playback runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee.xns"), runcmd(demo, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee.xns"), runcmd(node1, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee.xns"), runcmd(node2, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee.xns"), Config. end_per_group(_GroupName, Config) -> % Get ssh handles from config % {Hidenets, Demo, Node1, Node2, Testserver} = ?config(ssh_handles, Config), % Stop GUY macro playback runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee-end.xns"), runcmd(demo, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee-end.xns"), runcmd(node1, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee-end.xns"), runcmd(node2, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee-end.xns"), % Stop context agent on hidenets runcmd(hidenets, "pkill Context"), ok. % Initialize the testcase init_per_testcase(_TestCase, Config) -> Config. % Execute the following after a test case is finished end_per_testcase(_TestCase, Config) -> ok. groups() -> [{migrateTest, [sequence], [migCase1, migCase2]}]. all() -> % Return a list of all test cases in the suite. [migrateTest]. migCase1() -> []. migCase2() -> []. migCase1(Config) -> %Hidenets = element(1, ?config(ssh_handles,Config)), % Arbitrary delay to ensure eclipse + CA is started up and running ok = timer:sleep(5000), % Trigger runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee-triggerzero.xns"), ok = timer:sleep(5000), ok. migCase2(Config) -> % Hidenets = element(1, ?config(ssh_handles,Config)), % Arbitrary delay to ensure eclipse + CA is started up and running ok = timer:sleep(5000), % Trigger runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f ~/ct/xnee/xnee-triggerone.xns"), ok = timer:sleep(5000), ok. % Workaround for disconnect issue runcmd(HandleCfg, Cmd) -> case ct_ssh:connect(HandleCfg) of {ok, Handle} -> case ct_ssh:exec(Handle, Cmd, 3000) of {ok, Data} -> ok; {error, _} -> runcmd(HandleCfg, Cmd); {_, _} -> ok end; {error, _} -> runcmd(HandleCfg, Cmd); _ -> runcmd(HandleCfg, Cmd) end. %%% CODE FOR PYTHON TEST %%% %%%%%%%%%%%%%%%%%%%%%% -module(python_migration_SUITE). -compile(export_all). -include_lib("common_test/include/ct.hrl"). suite() -> % Make SSH and Python settings from config files, sshconfig.cfg % and pyconfig.cfg, available in the suite [{require, server}, {require, client1}, {require, client2}, {require, serverSettings}, {require, client1Settings}, {require, client2Settings}]. init_per_suite(Config) -> % Start ssh connection to server, client1 and client2 {ok, Server} = ct_ssh:connect(server), {ok, Client1} = ct_ssh:connect(client1), {ok, Client2} = ct_ssh:connect(client2), % Append handles to config variable [{ssh_handles,{Server, Client1, Client2}} | Config]. end_per_suite(Config) -> % Get ssh handles from config and make them a traversable list Handles = erlang:tuple_to_list(?config(ssh_handles, Config)), % Disconnect any active ssh handle lists:foreach(fun(Handle) -> ct_ssh:disconnect(Handle) end, Handles). init_per_group(_GroupName, Config) -> % Get ssh handles from config {Server, Client1, Client2} = ?config(ssh_handles, Config), % Start python server on server and log stdout ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, serverpy}) ++ " -i " ++ ct:get_config({serverSettings, serverip}) ++ " -p " ++ ct:get_config({serverSettings, serverport}) ++ " >> " ++ ct:get_config({serverSettings, serverlog}), 2000), % Start python client on client1 and log stdout ct_ssh:exec(Client1, "python " ++ ct:get_config({client1Settings, client1py}) ++ " -i " ++ ct:get_config({client1Settings, client1ip}) ++ " -p " ++ ct:get_config({client1Settings, client1port}) ++ " > " ++ ct:get_config({client1Settings, client1log}), 2000), % Start python client on client2 and log stdout ct_ssh:exec(Client2, "python " ++ ct:get_config({client2Settings, client2py}) ++ " -i " ++ ct:get_config({client2Settings, client2ip}) ++ " -p " ++ ct:get_config({client2Settings, client2port}) ++ " > " ++ ct:get_config({client2Settings, client2log}), 2000), Config. end_per_group(_GroupName, Config) -> % Get ssh handles from config {Server, Client1, Client2} = ?config(ssh_handles, Config), % End python server on server ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, commandpy}) ++ " -x"), % End python client on client1 ct_ssh:exec(Client1, "pkill -f 'client.py'"), % End python client on client2 ct_ssh:exec(Client2, "pkill -f 'client.py'"), ok. init_per_testcase(_TestCase, Config) -> Config. end_per_testcase(_TestCase, _Config) -> ok. groups() -> % Define sequential group migrateC1, then migrate2 [{migrateTest, [sequence], [migrateC1, migrateC2]}]. all() -> % Run the whole group when the suite is run [migrateTest]. migrateC1() -> % Do nothing in this testcase info function []. migrateC2() -> % Do nothing in this testcase info function []. migrateC1(Config) -> % Get server handle as it is the only one used in the testcase Server = element(1, ?config(ssh_handles,Config)), % Initial delay 2 seconds to assure that the server has been started properly ok = timer:sleep(2000), % Redirect server stream to client1 ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, commandpy}) ++ " -i " ++ ct:get_config({client1Settings, client1ip}) ++ " -p " ++ ct:get_config({client1Settings, client1port})), % Delay 7 seconds ok = timer:sleep(7000), ok. migrateC2(Config) -> % Get server handle as it is the only one used in the testcase Server = element(1, ?config(ssh_handles,Config)), % Redirect server stream to client2 ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, commandpy}) ++ " -i " ++ ct:get_config({client2Settings, client2ip}) ++ " -p " ++ ct:get_config({client2Settings, client2port})), % Delay 7 seconds ok = timer:sleep(7000), ok. 2010/9/27 Lukas Larsson > Hi Mikkel! > > What are you using to run your tests? The run_test command, or are you > starting them from an erlang shell? > > Also have you seen this problem when not using ct_ssh? or is it only > when you are doing tests with ssh calls in them? > > If you could provide me with a minimal test suite which reproduces the > error you are seeing that would be great! > > Regards, > Lukas, Erlang/OTP, Ericsson AB. > > On Fri, Sep 24, 2010 at 6:40 PM, Mikkel Jensen > wrote: > Hi list! > > I have been working on setting up a test case using Erlang > Common Test. The > test is pretty simple, it consists of a sequence of calls to > ct_ssh:exec(). > > Running the test once works. Running the test a few times (using > repeat), > usually works. But if i let the test suite repeat a large number > of times, > it fails after successfully running the test multiple times (In > my last > attempt it completed 42 runs, no joke, but it seems like the > repeated runs > can stop at any time). > > The error i'm getting is: > It is not possible to install CT while running in interactive > mode. > To exit this mode, run ct:stop_interactive(). > To enter the interactive mode again, run ct:start_interactive() > > Does anyone know how i can avoid this, allowing me to run a > large number of > repeated tests? > > On a sidenote: I have noticed in the logs that ct_ssh will > sometimes have > lost its connection when trying to call ct_ssh:exec(), and thus > will attempt > to reconnect. This attempt will, however, always fail instantly. > A > workaround i have made is to manually connect on each call i > make to > ct_ssh:exec(). This seems less than ideal, is there a better > way? > > Kind regards and thanks in advance, > Mikkel > > > > From ttmrichter@REDACTED Thu Sep 30 06:35:49 2010 From: ttmrichter@REDACTED (Michael Richter) Date: Thu, 30 Sep 2010 12:35:49 +0800 Subject: wx and the problem that won't go away. Message-ID: I have now tested Erlang/OTP R14B on (literally) a dozen different Windows XP machines as well as my Ubuntu 10.04 system. For the Windows machines, ten of which were 100% Erlang-clean (i.e. had never even once had Erlang installed), I followed this procedure: 1. Install Erlang/OTP R14B from the supplied binaries without touching a single option in the installer. 2. Reboot the machine. 3. Open a command line prompt. 4. Run *werl*. 5. At the shell prompt execute wx:demo(). The result, as reported before: on 12 out of the 12 machines werl crashes hard and Windows pops up a dialog box asking if I want to send Microsoft information on the crash so they can ignore it in perpetuity. (Seriously: has ANYBODY ever seen ANYTHING out of Microsoft from one of these crash reports?) For the Ubuntu 10.04 system I followed this procedure: 1. *rm -fR /usr/local/lib/erlang.* 2. Delete all of the items in /usr/local/bin that point to anything Erlang-related. 3. *find / -name "*erl*"* 4. Found out that 10.04 installs Erlang/OTP for some of the desktop utilities (!). 5. *sudo aptitude purge* ** 6. *sudo aptitude purge *** 7. *sudo aptitude install* ** 8. *tar xvf otp_src_R14B.tar.gz* 9. *cd otp_src_R14B* 10. *./configure* (no errors here) 11. *make* (no errors here) 12. *sudo make install* (no errors here) 13. Reboot the system. 14. Open up a command prompt. 15. *wx:demo().* The result, as before: $ erl Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] > [async-threads:0] [hipe] [kernel-poll:true] > Eshell V5.8.1 (abort with ^G) 1> wx:demo(). beam.smp: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) > (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct > malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= > (unsigned long)((((__builtin_offsetof (struct malloc_chunk, > fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - > 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == > 0)' failed. Aborted I'm completely at wit's end here. I have done nothing unusual on any of the machines I've tested this on. Out of the 12 Windows XP machines, 3 were brand new installs -- Windows XP SP3 + downloaded security updates *and nothing else*. On my Ubuntu machine I've tried my best to ensure there was nothing unusual lurking in any dark corners, but I'm not in a position to do a full-on, ground-up reinstallation. The result in all cases has been unqualified failure: Windows crashes the VM hard, Linux prints that assertion failure and then crashes the VM hard. Can anybody -- preferably someone who knows about wx and its implementation -- provide me with some clues as to which direction I should be stepping now to get this working? (This means "it works for me" is specifically *not* welcome unless it's paired with some feasible tips to investigate, incidentally.) Is there anybody who can look at that above assertion and get a feel for what may have got wrong where and how to perhaps fix it? As things stand now, there's a number of tools in the Erlang/OT distribution that are 100% useless to me because they rely on wx. This includes the debugger. I'd kind of like to see this situation end. * *-- "Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot." --Sergey Brin, demonstrating the emptiness of the "don't be evil" mantra. From sam@REDACTED Thu Sep 30 06:43:35 2010 From: sam@REDACTED (Sam Bobroff) Date: Thu, 30 Sep 2010 14:43:35 +1000 Subject: List comprehension filter expressions not throwing exceptions Message-ID: <4CA41577.5020107@m5net.com> Hi all, I've discovered (after some rather painful debugging) that badrecord exceptions (and others, actually) are sometimes silently converted to false when they're generated inside a list comprehension filter expression (as if they were a guard?). I couldn't find any reference to this behaviour in the manual, so perhaps it's a bug or perhaps there needs to be a note in the manual. It certainly surprised me. Here's some code to reproduce the behaviour: -module(lc). -export([a/0, b/0]). -record(rec, {x}). a() -> [E || E <- [1], E#rec.x =:= 1]. b() -> [E || E <- [1], check_element(E)]. check_element(E) -> E#rec.x =:= 1. Tested on R13B04, R14A or R14B, I get: $ erl Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.8 (abort with ^G) 1> c(lc). {ok,lc} 2> lc:a(). [] 3> lc:b(). ** exception error: {badrecord,rec} in function lc:check_element/1 in call from lc:'-b/0-lc$^0/1-0-'/1 4> Where I would expect both a() and b() to crash with badrecord. As the code shows, executing the same expression wrapped in a function call *does* cause the exception to be raised, so it's not as simple as all exceptions being converted. Peace, Sam. -- Sam Bobroff | sam@REDACTED | M5 Networks Why does my email have those funny headers? Because I use PGP to sign my email (and you should too!): that's how you know it's really from me. See: http://en.wikipedia.org/wiki/Pretty_Good_Privacy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: OpenPGP digital signature URL: From gonzalo@REDACTED Thu Sep 30 10:05:29 2010 From: gonzalo@REDACTED (Gonzalo) Date: Thu, 30 Sep 2010 10:05:29 +0200 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: <4CA05A48.5030100@fi.upm.es> References: <4CA05A48.5030100@fi.upm.es> Message-ID: Hi Clara, It would be great to have an Erlang user group in Madrid. I have little experience with the language/platform but sure I would attend the Madrid Erlounge event. Saludos. Gonzalo. On Mon, Sep 27, 2010 at 10:48 AM, Clara Benac Earle wrote: > As far as I know, there is not an Erlang user group in Madrid, so let's > start one! > > Those who have interest please send me an email, and I will organize a > Madrid Erlounge asap. > > Nos vemos > Clara > > > -sigit i.- wrote: >> >> Hi all, >> >> Is anyone here have info on erlang user group in Madrid (their local >> maillist, email, etc)? >> I appreciate the info. >> >> Thxalot, >> -sigit i.- >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From steve@REDACTED Thu Sep 30 10:12:56 2010 From: steve@REDACTED (Steve Strong) Date: Thu, 30 Sep 2010 10:12:56 +0200 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: <4CA05A48.5030100@fi.upm.es> Message-ID: Madrid would probably be my closest Erlang user group, but it's still a 6 hour drive. I'm guessing that the likelihood of an English-speaking Erlang usergroup in the Marbella / Malaga area is pretty slim? Cheers, Steve On Thu, Sep 30, 2010 at 10:05 AM, Gonzalo wrote: > Hi Clara, > > It would be great to have an Erlang user group in Madrid. I have > little experience with the language/platform but sure I would attend > the Madrid Erlounge event. > > Saludos. > Gonzalo. > > > > > On Mon, Sep 27, 2010 at 10:48 AM, Clara Benac Earle > wrote: > > As far as I know, there is not an Erlang user group in Madrid, so let's > > start one! > > > > Those who have interest please send me an email, and I will organize a > > Madrid Erlounge asap. > > > > Nos vemos > > Clara > > > > > > -sigit i.- wrote: > >> > >> Hi all, > >> > >> Is anyone here have info on erlang user group in Madrid (their local > >> maillist, email, etc)? > >> I appreciate the info. > >> > >> Thxalot, > >> -sigit i.- > >> > >> > > > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From max.lapshin@REDACTED Thu Sep 30 10:14:59 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 30 Sep 2010 12:14:59 +0400 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: <4CA05A48.5030100@fi.upm.es> Message-ID: On Thu, Sep 30, 2010 at 12:12 PM, Steve Strong wrote: > Madrid would probably be my closest Erlang user group, but it's still a 6 > hour drive. ?I'm guessing that the likelihood of an English-speaking Erlang > usergroup in the Marbella / Malaga area is pretty slim? > What erlang group may be near excelent surf spot? =)) From hynek@REDACTED Thu Sep 30 10:44:12 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Thu, 30 Sep 2010 10:44:12 +0200 Subject: [erlang-questions] List comprehension filter expressions not throwing exceptions In-Reply-To: <4CA41577.5020107@m5net.com> References: <4CA41577.5020107@m5net.com> Message-ID: On Thu, Sep 30, 2010 at 6:43 AM, Sam Bobroff wrote: > Hi all, > > I've discovered (after some rather painful debugging) that badrecord > exceptions (and others, actually) are sometimes silently converted to > false when they're generated inside a list comprehension filter > expression (as if they were a guard?). > > I couldn't find any reference to this behaviour in the manual, so > perhaps it's a bug or perhaps there needs to be a note in the manual. It > certainly surprised me. > > Here's some code to reproduce the behaviour: > > -module(lc). > -export([a/0, b/0]). > -record(rec, {x}). > > a() -> > ? ? ? ?[E || E <- [1], E#rec.x =:= 1]. > > b() -> > ? ? ? ?[E || E <- [1], check_element(E)]. > > check_element(E) -> > ? ? ? ?E#rec.x =:= 1. > > Tested on R13B04, R14A or R14B, I get: > > $ erl > Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] > [hipe] [kernel-poll:false] > > Eshell V5.8 ?(abort with ^G) > 1> c(lc). > {ok,lc} > 2> lc:a(). > [] > 3> lc:b(). > ** exception error: {badrecord,rec} > ? ? in function ?lc:check_element/1 > ? ? in call from lc:'-b/0-lc$^0/1-0-'/1 > 4> > > Where I would expect both a() and b() to crash with badrecord. As the > code shows, executing the same expression wrapped in a function call > *does* cause the exception to be raised, so it's not as simple as all > exceptions being converted. > > Peace, > Sam. Hi Sam, I think it is expected behavior that lists comprehension tests should behave same as 'when' clause in function or 'case' clauses. I.e. it is intended to handle exceptions as false. Call to check_element/1 cant be converted to 'when' so this doesn't catch exception. > -- > Sam Bobroff | sam@REDACTED | M5 Networks > Why does my email have those funny headers? Because I use PGP to sign > my email (and you should too!): that's how you know it's really from me. > See: http://en.wikipedia.org/wiki/Pretty_Good_Privacy > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From gonzalo@REDACTED Thu Sep 30 10:46:19 2010 From: gonzalo@REDACTED (Gonzalo) Date: Thu, 30 Sep 2010 10:46:19 +0200 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: <4CA05A48.5030100@fi.upm.es> Message-ID: Hi, For the next Erlang Madrid event I would suggest to have some hands-on intro conferences so novice developers are introduced with the basic concepts of Erlang/OTP. Best regards, Gon. On Thu, Sep 30, 2010 at 10:14 AM, Max Lapshin wrote: > On Thu, Sep 30, 2010 at 12:12 PM, Steve Strong wrote: >> Madrid would probably be my closest Erlang user group, but it's still a 6 >> hour drive. ?I'm guessing that the likelihood of an English-speaking Erlang >> usergroup in the Marbella / Malaga area is pretty slim? >> > > What erlang group may be near excelent surf spot? =)) > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From joelr1@REDACTED Thu Sep 30 10:53:38 2010 From: joelr1@REDACTED (Joel Reymont) Date: Thu, 30 Sep 2010 09:53:38 +0100 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: <4CA05A48.5030100@fi.upm.es> Message-ID: <5B77804E-F0F2-4AA5-9E48-862CBC4F0F4E@gmail.com> Tenerife would be very decent! Sent from my iPhone in Tenerife On 30/09/2010, at 09:14, Max Lapshin wrote: > On Thu, Sep 30, 2010 at 12:12 PM, Steve Strong wrote: >> Madrid would probably be my closest Erlang user group, but it's still a 6 >> hour drive. I'm guessing that the likelihood of an English-speaking Erlang >> usergroup in the Marbella / Malaga area is pretty slim? >> > > What erlang group may be near excelent surf spot? =)) > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From kostis@REDACTED Thu Sep 30 11:55:56 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Thu, 30 Sep 2010 12:55:56 +0300 Subject: [erlang-questions] List comprehension filter expressions not throwing exceptions In-Reply-To: <4CA41577.5020107@m5net.com> References: <4CA41577.5020107@m5net.com> Message-ID: <4CA45EAC.5010604@cs.ntua.gr> Sam Bobroff wrote: > Hi all, > > I've discovered (after some rather painful debugging) that badrecord > exceptions (and others, actually) are sometimes silently converted to > false when they're generated inside a list comprehension filter > expression (as if they were a guard?). > > I couldn't find any reference to this behaviour in the manual, so > perhaps it's a bug or perhaps there needs to be a note in the manual. It > certainly surprised me. > > Here's some code to reproduce the behaviour: > > -module(lc). > -export([a/0, b/0]). > -record(rec, {x}). > > a() -> > [E || E <- [1], E#rec.x =:= 1]. > > b() -> > [E || E <- [1], check_element(E)]. > > check_element(E) -> > E#rec.x =:= 1. > > Tested on R13B04, R14A or R14B, I get: > > $ erl > Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] > [hipe] [kernel-poll:false] > > Eshell V5.8 (abort with ^G) > 1> c(lc). > {ok,lc} > 2> lc:a(). > [] > 3> lc:b(). > ** exception error: {badrecord,rec} > in function lc:check_element/1 > in call from lc:'-b/0-lc$^0/1-0-'/1 > 4> > > Where I would expect both a() and b() to crash with badrecord. As the > code shows, executing the same expression wrapped in a function call > *does* cause the exception to be raised, so it's not as simple as all > exceptions being converted. Your expectations are wrong. It's not the function call that is to blame here. The problem is that your b+check_element code is not equivalent to the code of the a() function. The equivalent code is: b() -> [E || E <- [1], check_element(E)]. check_element(E) when E#rec.x =:= 1 -> true; check_element(_) -> false. which does not raise an exception either. Kostis From avtobiff@REDACTED Thu Sep 30 14:45:24 2010 From: avtobiff@REDACTED (Per Andersson) Date: Thu, 30 Sep 2010 14:45:24 +0200 Subject: [erlang-questions] running an Erlang OTP application on a production Linux server In-Reply-To: References: Message-ID: Hi! Have a look at how ejabberd and the (Debian only?) ejabberd init script does it. -- Per From max.lapshin@REDACTED Thu Sep 30 16:20:31 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 30 Sep 2010 18:20:31 +0400 Subject: R14 speedup: thanks Message-ID: On R13 erlyvideo used 0,25% of one xeon core to serve a client, now on R14 it has fallen to 0,17% of core per client, so R14 is 30% faster on my task. I want to thank erlang team for your work. Hope, I will help you with my commits =) From nirth.furzahad@REDACTED Thu Sep 30 17:37:36 2010 From: nirth.furzahad@REDACTED (David Sergey) Date: Thu, 30 Sep 2010 18:37:36 +0300 Subject: Implementation of Fact (Belief) Base Message-ID: Hello Everybody I'm researching ways in Erlang to create Prolog (or any other logical language like) fact/belif base. Point is ? some data structures should be stored on process for later evaluation. Best Regards David From zabrane3@REDACTED Thu Sep 30 17:45:09 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Thu, 30 Sep 2010 17:45:09 +0200 Subject: [erlang-questions] Implementation of Fact (Belief) Base In-Reply-To: References: Message-ID: Have a look to ERES : http://groups.google.com/group/erlang-programming/browse_thread/thread/6ea7e7de37274523 -- Regards Zabrane 2010/9/30 David Sergey : > Hello Everybody > > I'm researching ways in Erlang to create Prolog (or any other logical language like) fact/belif base. Point is ? some data structures should be stored on process for later evaluation. > > Best Regards > David > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From pablo.platt@REDACTED Thu Sep 30 17:38:14 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Thu, 30 Sep 2010 08:38:14 -0700 (PDT) Subject: valid use for gen_server? Message-ID: <253554.95925.qm@web112607.mail.gq1.yahoo.com> Hi, I'm building a simple IM server and to make it modular I want to let plugins subscribe handlers to messages (ejabberd hooks, django signals, js listeners). If the handler is sync it can modify the message and return it or return false to prevent further processing of the message. If the handler is async it receives the message but doesn't effect it or other handlers. I see two options to implement the above, gen_event or custom event module. What bothers me in gen_event is that it executes handlers in its own context. If events are rare that shouldn't be a problem but if each IM message will fire several events wouldn't this flood the gen_event server? I can use a custom event module that saves events and handlers in ETS table and call functions dynamically with Module:Function(Args). This also doesn't feel right because each IM message will require several ETS lookups and dynamic functions calls which according to the efficiency guide are are 6 times slower than normal calls. Can you suggest which one is better? Thanks From fernando.benavides@REDACTED Thu Sep 30 19:00:41 2010 From: fernando.benavides@REDACTED (Fernando Benavides) Date: Thu, 30 Sep 2010 14:00:41 -0300 Subject: Dialyzer and parameterized modules Message-ID: <1285866041.1877.10.camel@army-desktop> Hi all, I wrote a post in TrapExit OTP/Erlang Forums about this issue. It says something like this: I know that since v2.2.0, dialyzer has support for parameterized modules, but somehow it's not working for me. Lets say that, for some unknown reason (o_O), I want a parameterized module for atom queues. I write this tiny parameterized module then: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -module(atom_queue, [Queue]). -export([push/1, pop/0]). pop() -> case Queue of [] -> {undefined, atom_queue:new([])}; [X|Rest] -> {X, atom_queue:new(Rest)} end. push(X) -> atom_queue:new(lists:reverse([X|Queue])). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% If I run dialyzer on it, no warnings are reported. But it has no specs and I like to compile my code with warn_missing_spec. The compiler says that the specs for pop/0 and push/1 are missing. So, I add the specs... %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -module(atom_queue, [Queue]). -export([push/1, pop/0]). -spec pop() -> {undefined | atom(), {?MODULE, [atom()]}}. pop() -> case Queue of [] -> {undefined, atom_queue:new([])}; [X|Rest] -> {X, atom_queue:new(Rest)} end. -spec push(atom()) -> {?MODULE, [atom()]}. push(X) -> atom_queue:new(lists:reverse([X|Queue])). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Compiler is not complaining anymore, which is great. But when I try to dialyze the module, I get: Contract for function that does not exist: atom_queue:pop/0 Contract for function that does not exist: atom_queue:push/1 I change the specs to satisfy dialyzer, like this: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -module(atom_queue, [Queue]). -export([push/1, pop/0]). -spec pop({?MODULE, [atom()]}) -> {undefined | atom(), {?MODULE, [atom()]}}. pop() -> case Queue of [] -> {undefined, atom_queue:new([])}; [X|Rest] -> {X, atom_queue:new(Rest)} end. -spec push(atom(), {?MODULE, [atom()]}) -> {?MODULE, [atom()]}. push(X) -> atom_queue:new(lists:reverse([X|Queue])). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% But, of course, I can't compile that code: spec for undefined function atom_queue:pop/1 spec for undefined function atom_queue:push/2 So, basically the question is: How can I write a parameterized module with specs that satisfies both dialyzer and erlc at the same time? ________________________________________________________________________ Fernando Benavides fernando@REDACTED From mpquique@REDACTED Thu Sep 30 19:06:36 2010 From: mpquique@REDACTED (Enrique Marcote) Date: Thu, 30 Sep 2010 19:06:36 +0200 Subject: [erlang-questions] Erlang user group in madrid? In-Reply-To: References: <4CA05A48.5030100@fi.upm.es> Message-ID: <2A5063DC-84D1-4B78-B6AC-96B02927F6AD@gmail.com> Erlanger and surfer? I thought I was the only one ;-) Quique El 30/09/2010, a las 10:14, Max Lapshin escribi?: > On Thu, Sep 30, 2010 at 12:12 PM, Steve Strong wrote: >> Madrid would probably be my closest Erlang user group, but it's still a 6 >> hour drive. I'm guessing that the likelihood of an English-speaking Erlang >> usergroup in the Marbella / Malaga area is pretty slim? >> > > What erlang group may be near excelent surf spot? =)) > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ivan060111ad@REDACTED Thu Sep 30 22:46:28 2010 From: ivan060111ad@REDACTED (Ivan Carmenates =?iso-8859-1?Q?Garc=EDa?=) Date: Thu, 30 Sep 2010 15:46:28 -0500 Subject: please take a look on that!!! (ExtVisualOtp) Message-ID: Hi all, please take a look on this. It's is a pdf file with the documentation of the component I had developped, ExtVisualOtp. Best, Regards... [file:///F:/new/EVO%2030-Sep-2010%20%20%2014-04/ExtVisualOtp/ExtendedVisualOtp%20(English).pdf] -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Thu Sep 30 22:05:41 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Thu, 30 Sep 2010 22:05:41 +0200 Subject: [erlang-questions] please take a look on that!!! (ExtVisualOtp) In-Reply-To: References: Message-ID: There's no attachement Ivan! -- Regards Zabrane 2010/9/30 Ivan Carmenates Garc?a : > Hi all, please take a look on this. > It's is a pdf file with the documentation of the component I had developped, > ExtVisualOtp. > Best, Regards... > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ivan060111ad@REDACTED Thu Sep 30 23:14:05 2010 From: ivan060111ad@REDACTED (Ivan Carmenates =?iso-8859-1?Q?Garc=EDa?=) Date: Thu, 30 Sep 2010 16:14:05 -0500 Subject: ExtVisualOtp documentation Message-ID: Hi all!!!, I will try again, but I did it with attachement I though this mail list do not allow attachements. -------------- next part -------------- An HTML attachment was scrubbed... URL: