From sean@REDACTED Mon Sep 2 08:00:01 2013 From: sean@REDACTED (Functional Jobs) Date: Mon, 2 Sep 2013 02:00:01 -0400 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <5224297882266@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Erlang Developer at Voalte, Inc http://functionaljobs.com/jobs/8644-erlang-developer-at-voalte-inc Cheers, Sean Murphy FunctionalJobs.com From bengt.kleberg@REDACTED Mon Sep 2 10:22:54 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Mon, 2 Sep 2013 10:22:54 +0200 Subject: [erlang-questions] clarify: lists:seq/3 Failure: If To Greetings, The documentation (http://www.erlang.org/doc/man/lists.html#seq-3) says "Failure: If To lists:seq(10, 6, 4). [] Should that not fail? bengt From n.oxyde@REDACTED Mon Sep 2 10:26:15 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 2 Sep 2013 10:26:15 +0200 Subject: [erlang-questions] clarify: lists:seq/3 Failure: If To References: <1378110174.4784.9.camel@sekic1152.release> Message-ID: Why would it fail? 6 is not strictly less than 10-4. -- Anthony Ramine Le 2 sept. 2013 ? 10:22, Bengt Kleberg a ?crit : > Greetings, > > The documentation (http://www.erlang.org/doc/man/lists.html#seq-3) says > "Failure: If To> lists:seq(10, 6, 4). > [] > > Should that not fail? > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From tang.binbin@REDACTED Mon Sep 2 08:02:41 2013 From: tang.binbin@REDACTED (=?UTF-8?B?5ZSQ5a6+5a6+?=) Date: Mon, 02 Sep 2013 14:02:41 +0800 Subject: [erlang-questions] How to use ssh:connect? Message-ID: <52242A01.5020905@sincetimes.com> hi all: I have done the following test: on client A : $ ssh-keygen -t rsa -f /opt/ssh_daemon/ssh_host_rsa_key $ ssh-keygen -t rsa -f /opt/tangbinbin/.ssh/id_rsa Eshell V5.10.2 (abort with ^G) 1> crypto:start(). ok 2> ssh:start(). ok 3> {ok, Sshd} = ssh:daemon(8989, [{system_dir, "/opt/ssh_daemon"}, {user_dir, "/opt/tangbinbin/.ssh"}]). {ok,<0.52.0>} $netstat -lntp | grep 8989 tcp6 0 0 :::8989 :::* LISTEN 6792/beam.smp on server B Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> crypto:start(). ok 2> ssh:start(). ok 3> ssh:connect("HostA", 8989, [{user_dir, "/opt/tangbinbin/.ssh"}]). New host 192.168.0.203 accept [y/n]?y {error,{"Key exchange failed",{error,eacces}}} How to connnet with A? From ahe.sanath@REDACTED Mon Sep 2 12:02:57 2013 From: ahe.sanath@REDACTED (Sanath Prasanna) Date: Mon, 2 Sep 2013 15:32:57 +0530 Subject: [erlang-questions] MySQL with emysql Issue Message-ID: Hi all, I used emysql to working with mysql in my Erlang Application. Sometimes, follwoing error come & need to restart application to working properly. Else set_connection {{'EXIT',{failed_to_connect_to_database,emfile}}, =ERROR REPORT==== 2-Sep-2013::12:49:21 === File operation error: emfile. Target: ./lib.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 2-Sep-2013::12:49:21 === File operation error: emfile. Target: /opt/test_app/system/lib/kernel-2.14.1/ebin/lib.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 2-Sep-2013::12:49:21 === File operation error: emfile. Target: /opt/test_app/system/lib/stdlib-1.17.1/ebin/lib.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 2-Sep-2013::12:49:21 === File operation error: emfile. Target: /opt/test_app/system/lib/sasl-2.1.9.2/ebin/lib.beam. Function: get_file. Process: code_server. =ERROR REPORT==== 2-Sep-2013::12:49:21 === File operation error: emfile. Target: /opt/test_app/system/lib/compiler-4.7.1/ebin/lib.beam. Function: get_file. Process: code_server. Pls help to solve this. Br, Sanath code : case catch emysql:add_pool(ConnPoolId, Config#db_conn_data.pool_size, Config#db_conn_data.username, Config#db_conn_data.password, Config#db_conn_data.host, Config#db_conn_data.port, DbName, utf8) of ok -> {ok, ConnPoolId}; {error, pool_already_exists} -> already_exist; {'EXIT',{failed_to_connect_to_database, emfile}} -> io:fwrite("Else set_connection ~p~n",[{{failed_to_connect_to_database, emfile}, DbName, ConnPoolId}]), catch spawn(?MODULE, update_status,["failed_to_connect_to_database"]), error; Else -> catch spawn(?MODULE, update_status,["database_error"]), io:fwrite("Else set_connection ~p~n",[{Else, DbName, ConnPoolId}]), error end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Sep 2 12:10:39 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 2 Sep 2013 12:10:39 +0200 Subject: [erlang-questions] MySQL with emysql Issue In-Reply-To: References: Message-ID: On Mon, Sep 2, 2013 at 12:02 PM, Sanath Prasanna wrote: > Else set_connection {{'EXIT',{failed_to_connect_to_database,emfile}}, > =ERROR REPORT==== 2-Sep-2013::12:49:21 === > File operation error: emfile. Target: ./lib.beam. Function: get_file. > Process: code_server. > The `emfile` POSIX error code would suggest you are running out of file descriptors from the operating system. Do you have any knowledge of a leak in the system and to which subsystem. Try asking the OS about open files so you can get an idea about what is wrong here. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Mon Sep 2 15:34:32 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Mon, 2 Sep 2013 15:34:32 +0200 Subject: [erlang-questions] mnesia file locations Message-ID: <006001cea7e1$29604be0$7c20e3a0$@gmail.com> All, is it possible to specify different locations for mnesia log files and data files? Thanks, Janos -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Mon Sep 2 18:04:04 2013 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 2 Sep 2013 18:04:04 +0200 (CEST) Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> Message-ID: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> I will be in NYC 10-15 sept so I will do my best to attend. Erlounges are great. Robert ----- Original Message ----- > From: "Bob Ippolito" > To: "Roberto Ostinelli" > Cc: "Erlang" > Sent: Friday, 30 August, 2013 5:17:52 AM > Subject: Re: [erlang-questions] erlounge in nyc > I'm in NYC from Sep 5-13 so I could probably also attend > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > > I moved to NYC. If this happens I'm in too. > > > r. > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies < heinz@REDACTED > > > > > wrote: > > > > I'll do my best to make it too :) > > > > > > Cheers, > > > > > > Heinz > > > > > > On Aug 29, 2013, at 22:16, Garry Hodgson < garry@REDACTED > > > > wrote: > > > > > > > i would be love to get together with some other > > > > > > > erlang folks, if this is still on the table. > > > > > > > > > > > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > > > > > > >> Hello, > > > > > > >> > > > > > > >> Somebody, and I forget who asked me, and I forget when, to drop them a > > > >> line > > > > > > >> if I was every in New York. > > > > > > >> > > > > > > >> I'll be in New York from 10 - 14 sept, if anybody is interested in a > > > >> meetup. > > > > > > >> > > > > > > >> /Joe > > > > > > > > > > > > > > -- > > > > > > > Garry Hodgson > > > > > > > AT&T Chief Security Office (CSO) > > > > > > > > > > > > > > "This e-mail and any files transmitted with it are AT&T property, are > > > > confidential, and are intended solely for the use of the individual or > > > > entity to whom this e-mail is addressed. If you are not one of the > > > > named > > > > recipient(s) or otherwise have reason to believe that you have received > > > > this message in error, please notify the sender and delete this message > > > > immediately from your computer. Any other use, retention, > > > > dissemination, > > > > forwarding, printing, or copying of this e-mail is strictly > > > > prohibited." > > > > > > > > > > > > > > _______________________________________________ > > > > > > > erlang-questions mailing list > > > > > > > erlang-questions@REDACTED > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > > > > > erlang-questions mailing list > > > > > > erlang-questions@REDACTED > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Tue Sep 3 14:26:44 2013 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 3 Sep 2013 14:26:44 +0200 Subject: [erlang-questions] mnesia file locations In-Reply-To: <006001cea7e1$29604be0$7c20e3a0$@gmail.com> References: <006001cea7e1$29604be0$7c20e3a0$@gmail.com> Message-ID: Hi Janos, Yes, you can use -mnesia dir when starting your system http://www.erlang.org/doc/apps/mnesia/Mnesia_chap2.html#id61530 Best Regards, Ahmed On Mon, Sep 2, 2013 at 3:34 PM, Janos Hary wrote: > All,**** > > ** ** > > is it possible to specify different locations for mnesia log files and > data files?**** > > ** ** > > Thanks,**** > > Janos**** > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Tue Sep 3 14:32:46 2013 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Tue, 3 Sep 2013 14:32:46 +0200 Subject: [erlang-questions] =?iso-8859-1?q?Erlang/Elixir_User_Group_in_the?= =?iso-8859-1?q?_=28=D6=7C=D8=29resund_region?= Message-ID: Hi, How many would be interested in an Erlang/Elixir User Group with meetings in Copenhagen and Malm?? Cheers, __ /orben -- http://www.linkedin.com/in/torbenhoffmann @LeHoff -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Tue Sep 3 14:35:53 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 3 Sep 2013 14:35:53 +0200 Subject: [erlang-questions] =?utf-8?q?Erlang/Elixir_User_Group_in_the_=28?= =?utf-8?b?w5Z8w5gpcmVzdW5kIHJlZ2lvbg==?= In-Reply-To: References: Message-ID: +1 :) On Tue, Sep 3, 2013 at 2:32 PM, Torben Hoffmann wrote: > Hi, > > How many would be interested in an Erlang/Elixir User Group with meetings > in Copenhagen and Malm?? > > Cheers, > __ > /orben > > -- > http://www.linkedin.com/in/torbenhoffmann > @LeHoff > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From no-reply@REDACTED Tue Sep 3 15:00:20 2013 From: no-reply@REDACTED (Dropbox) Date: Tue, 03 Sep 2013 13:00:20 +0000 Subject: [erlang-questions] =?utf-8?q?JOSHUA_MUZAAYA_invited_you_to_check_?= =?utf-8?q?out_Dropbox?= Message-ID: <20130903130020.2C7FD5C22D6@sjc-batch6.sjc.dropbox.com> Hi there, JOSHUA MUZAAYA wants you to try Dropbox! Dropbox lets you bring all your photos, docs and videos with you anywhere and share them easily. Get started here. https://www.dropbox.com/l/GWRLe8i4XIZWHbn35o5jBT1 Thanks! - The Dropbox Team ____________________________________________________ To stop receiving invites from Dropbox, please go to https://www.dropbox.com/l/pFadD5GOuLQE6N9bu6qt4V1 Dropbox, Inc., PO Box 77767, San Francisco, CA 94107 -------------- next part -------------- An HTML attachment was scrubbed... URL: From yoursurrogategod@REDACTED Tue Sep 3 22:09:34 2013 From: yoursurrogategod@REDACTED (Yves S. Garret) Date: Tue, 3 Sep 2013 16:09:34 -0400 Subject: [erlang-questions] Debugger hangs Message-ID: Hello, I'm running this on a Mac OS X 10.7.5. I have release 16B01. Whenever I start up the debugger and then increase the size of the windows (by dragging the borders), the whole setup just hangs. It doesn't come back to me, but sits there and doesn't do anything. Has anyone else had this problem? How did you get around it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Sep 3 23:09:18 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 4 Sep 2013 00:09:18 +0300 Subject: [erlang-questions] dets disk i/o Message-ID: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@gmail.com> Hello, Does any one run a disk i/o analysis on dets tables? I am very curious about bag case, write scenario. Let's assume that I have a key which is associated with multiple entries. What is dets behaviour when I am writing a new entry to same key. Does it make append-only to data slot? or Does it read/write whole slot? Best Regards, Dmitry From fritchie@REDACTED Wed Sep 4 05:48:05 2013 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 03 Sep 2013 22:48:05 -0500 Subject: [erlang-questions] Pre-party for the ACM Erlang Workshop in Boston Message-ID: <35827.1378266485@snookles.snookles.com> Basho will be hosting a pre-party for the Erlang Workshop co-hosted event at ICFP this year at the Cambridge office. Anyone in the area should attend to talk Erlang. We hope that the planets will align so that Kostis Sagonas can present his latest work on concurrent testing. When: Friday, September 27 at 8:00pm Details: see https://guestlistapp.com/events/178131 Many thanks to Chris Meiklejohn for shepherding this event. -Scott From jesper.louis.andersen@REDACTED Wed Sep 4 13:42:28 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 4 Sep 2013 13:42:28 +0200 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: On Tue, Aug 27, 2013 at 6:26 PM, Pablo Polvorin < pablo.polvorin@REDACTED> wrote: > It is focused on queries returning large result sets, that was our > original problem. There are more things done but not yet had time > to complete a writting about it. Note this isn't on oficial emysql > repo, Jesper did try to include some of the changes, but at that time > he found some problems (thanks!), solved now on our repo HEAD. > And the changes have now been merged into the official Emysql driver. Thanks! I still need some help with Garrett Smith's patches. There is a branch https://github.com/Eonblast/Emysql/commits/jl/gar1t/master with the changes, but I have yet to get those in. I don't know how cleanly they are merging at the moment however. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Wed Sep 4 13:56:08 2013 From: g@REDACTED (Garrett Smith) Date: Wed, 4 Sep 2013 06:56:08 -0500 Subject: [erlang-questions] [HEADS UP] Emysql driver In-Reply-To: References: Message-ID: Okay, okay :) Let me set some time aside today for this. Sorry for the feet dragging! On Wed, Sep 4, 2013 at 6:42 AM, Jesper Louis Andersen wrote: > > On Tue, Aug 27, 2013 at 6:26 PM, Pablo Polvorin > wrote: >> >> It is focused on queries returning large result sets, that was our >> original problem. There are more things done but not yet had time >> to complete a writting about it. Note this isn't on oficial emysql >> repo, Jesper did try to include some of the changes, but at that time >> he found some problems (thanks!), solved now on our repo HEAD. > > > And the changes have now been merged into the official Emysql driver. > Thanks! > > I still need some help with Garrett Smith's patches. There is a branch > > https://github.com/Eonblast/Emysql/commits/jl/gar1t/master > > with the changes, but I have yet to get those in. I don't know how cleanly > they are merging at the moment however. > > > -- > J. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From chauhan.kuldeep.sinh@REDACTED Wed Sep 4 13:58:15 2013 From: chauhan.kuldeep.sinh@REDACTED (KuldeepSinh Chauhan) Date: Wed, 4 Sep 2013 13:58:15 +0200 Subject: [erlang-questions] erlang-questions Digest, Vol 129, Issue 3 In-Reply-To: References: Message-ID: Hi Torben, I am interested in an Erlang/Elixir User Group with meetings in Copenhagen and Malmo. Thanks for your initiative. In case, if you need any volunteer, count me in too. Regards, Kuldeep. On Wed, Sep 4, 2013 at 12:00 PM, wrote: > Send erlang-questions mailing list submissions to > erlang-questions@REDACTED > > To subscribe or unsubscribe via the World Wide Web, visit > http://erlang.org/mailman/listinfo/erlang-questions > or, via email, send a message with subject or body 'help' to > erlang-questions-request@REDACTED > > You can reach the person managing the list at > erlang-questions-owner@REDACTED > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of erlang-questions digest..." > > > Today's Topics: > > 1. Re: mnesia file locations (Ahmed Omar) > 2. Erlang/Elixir User Group in the (?|?)resund region > (Torben Hoffmann) > 3. Re: Erlang/Elixir User Group in the (?|?)resund region > (Jesper Louis Andersen) > 4. JOSHUA MUZAAYA invited you to check out Dropbox (Dropbox) > 5. Debugger hangs (Yves S. Garret) > 6. dets disk i/o (Dmitry Kolesnikov) > 7. Pre-party for the ACM Erlang Workshop in Boston > (Scott Lystig Fritchie) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 3 Sep 2013 14:26:44 +0200 > From: Ahmed Omar > To: Janos Hary > Cc: "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] mnesia file locations > Message-ID: > AiHEeQdhnyG2OjQ@REDACTED> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Janos, > Yes, you can use -mnesia dir when starting your system > http://www.erlang.org/doc/apps/mnesia/Mnesia_chap2.html#id61530 > > Best Regards, > Ahmed > > > On Mon, Sep 2, 2013 at 3:34 PM, Janos Hary wrote: > > > All,**** > > > > ** ** > > > > is it possible to specify different locations for mnesia log files and > > data files?**** > > > > ** ** > > > > Thanks,**** > > > > Janos**** > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20130903/d0b242c5/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Tue, 3 Sep 2013 14:32:46 +0200 > From: Torben Hoffmann > To: erlang-questions > Subject: [erlang-questions] Erlang/Elixir User Group in the > (?|?)resund region > Message-ID: > zQmH7GjRG9zhoWvVJKR_xc34ideRKF0T3Tai5UrazBg@REDACTED> > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > How many would be interested in an Erlang/Elixir User Group with meetings > in Copenhagen and Malm?? > > Cheers, > __ > /orben > > -- > http://www.linkedin.com/in/torbenhoffmann > @LeHoff > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20130903/bcdd9372/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Tue, 3 Sep 2013 14:35:53 +0200 > From: Jesper Louis Andersen > > To: Torben Hoffmann > Cc: erlang-questions > Subject: Re: [erlang-questions] Erlang/Elixir User Group in the > (?|?)resund region > Message-ID: > < > CAGrdgiVsXSiP2UA-oWS6JEVGKzuQ_OtaqK_Fkejt-cJGTdsmDg@REDACTED> > Content-Type: text/plain; charset="utf-8" > > +1 :) > > > On Tue, Sep 3, 2013 at 2:32 PM, Torben Hoffmann >wrote: > > > Hi, > > > > How many would be interested in an Erlang/Elixir User Group with meetings > > in Copenhagen and Malm?? > > > > Cheers, > > __ > > /orben > > > > -- > > http://www.linkedin.com/in/torbenhoffmann > > @LeHoff > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20130903/13161753/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Tue, 03 Sep 2013 13:00:20 +0000 > From: Dropbox > To: erlang-questions@REDACTED > Subject: [erlang-questions] JOSHUA MUZAAYA invited you to check out > Dropbox > Message-ID: <20130903130020.2C7FD5C22D6@REDACTED> > Content-Type: text/plain; charset="utf-8" > > Hi there, > > JOSHUA MUZAAYA wants you to try Dropbox! Dropbox lets you bring all your > photos, docs and videos with you anywhere and share them easily. > > Get started here. > https://www.dropbox.com/l/GWRLe8i4XIZWHbn35o5jBT1 > > Thanks! > - The Dropbox Team > > ____________________________________________________ > To stop receiving invites from Dropbox, please go to > https://www.dropbox.com/l/pFadD5GOuLQE6N9bu6qt4V1 > Dropbox, Inc., PO Box 77767, San Francisco, CA 94107 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20130903/8fc1aad7/attachment-0001.html > > > > ------------------------------ > > Message: 5 > Date: Tue, 3 Sep 2013 16:09:34 -0400 > From: "Yves S. Garret" > To: "erlang-questions@REDACTED" > Subject: [erlang-questions] Debugger hangs > Message-ID: > Q4w@REDACTED> > Content-Type: text/plain; charset="iso-8859-1" > > Hello, > > I'm running this on a Mac OS X 10.7.5. I have release 16B01. Whenever I > start up the debugger > and then increase the size of the windows (by dragging the borders), the > whole setup just hangs. It > doesn't come back to me, but sits there and doesn't do anything. > > Has anyone else had this problem? How did you get around it? > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://erlang.org/pipermail/erlang-questions/attachments/20130903/d701ce2a/attachment-0001.html > > > > ------------------------------ > > Message: 6 > Date: Wed, 4 Sep 2013 00:09:18 +0300 > From: Dmitry Kolesnikov > To: erlang-questions Questions > Subject: [erlang-questions] dets disk i/o > Message-ID: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@REDACTED> > Content-Type: text/plain; charset=us-ascii > > Hello, > > Does any one run a disk i/o analysis on dets tables? > I am very curious about bag case, write scenario. > > Let's assume that I have a key which is associated with multiple entries. > What is dets behaviour when I am writing a new entry to same key. > Does it make append-only to data slot? or Does it read/write whole slot? > > Best Regards, > Dmitry > > > ------------------------------ > > Message: 7 > Date: Tue, 03 Sep 2013 22:48:05 -0500 > From: Scott Lystig Fritchie > To: erlang-questions@REDACTED > Subject: [erlang-questions] Pre-party for the ACM Erlang Workshop in > Boston > Message-ID: <35827.1378266485@REDACTED> > > Basho will be hosting a pre-party for the Erlang Workshop co-hosted > event at ICFP this year at the Cambridge office. Anyone in the area > should attend to talk Erlang. We hope that the planets will align so > that Kostis Sagonas can present his latest work on concurrent testing. > > When: Friday, September 27 at 8:00pm > Details: see https://guestlistapp.com/events/178131 > > Many thanks to Chris Meiklejohn for shepherding this event. > > -Scott > > > ------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > End of erlang-questions Digest, Vol 129, Issue 3 > ************************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Wed Sep 4 14:24:51 2013 From: spawn.think@REDACTED (Ahmed Omar) Date: Wed, 4 Sep 2013 14:24:51 +0200 Subject: [erlang-questions] mnesia silent failure and table corruption In-Reply-To: <004f01cea336$fa6b2390$ef416ab0$@gmail.com> References: <004f01cea336$fa6b2390$ef416ab0$@gmail.com> Message-ID: Hi Janos, Indeed, in that case transactions could be considered succeeded while actual storage operation failed... The thing is, If mnesia needs to write data to a store (in this case dets), it doesn't check for results for this operation. Actually, it does even catch exceptions. (mnesia_tm:do_update/4) As dets have a maximum of 2GB for file size, I believe the table gets corrupted after that. (somebody correct me please) So any operation on that dets table will return the bad_object error. So the commit is considered succeeded, mnesia logs it, transaction returns {atomic, ok} sync_transaction won't help, because it just waits for the commit to be synced/logged, but also doesn't check results of the store operation. so you could end up in situation where dirty write operation returns an error, and in a transaction returns {atomic, ok} > dets:insert(table1, SomeKey, SomeVal). {error,{{bad_object,read_buckets}, "/tmp/table1.DAT"}} > mnesia:dirty_write(SomeRecord). {error,{{bad_object,read_buckets}, "/tmp/table1.DAT"}} > mnesia:transaction(fun()-> mnesia:write(SomeRecord) end). {atomic,ok} You could try to detect that in your transaction by adding an operation that will fail (like read, first, last, ..etc) but that won't protect you against corrupting the file. The question is, should mnesia detect this kind of errors and if yes, what to do about it? Also, can dets protect against the corruption somehow? (reject the operation that could lead to size> 2GB?) Best Regards, Ahmed On Tue, Aug 27, 2013 at 5:06 PM, Janos Hary wrote: > Thanks, but no luck. It behaves the same with sync_transaction. > > Janos > > -----Original Message----- > From: hawk.mattsson@REDACTED [mailto:hawk.mattsson@REDACTED] On Behalf > Of > H?kan Mattsson > Sent: Tuesday, August 27, 2013 4:02 PM > To: Janos Hary > Cc: erlang-questions > Subject: Re: [erlang-questions] mnesia silent failure and table corruption > > On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary > wrote: > > All, > > > > I started to write records into a table from a simple test function. > > When the table size reaches 2GB write transactions still report > > success, but they are silently ignored. The table became unusable, > > read operations return error, and mnesia cannot recover the table. How > > shall I avoid such situation? > > Try using mnesia:sync_transaction() instead of mnesia:transaction(). > Then Mnesia will use syncronized calls to disk_log (and syncronized commits > when several nodes are involved). > > /H?kan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Wed Sep 4 16:19:08 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Wed, 4 Sep 2013 16:19:08 +0200 Subject: [erlang-questions] mnesia silent failure and table corruption In-Reply-To: References: <004f01cea336$fa6b2390$ef416ab0$@gmail.com> Message-ID: <001e01cea979$b8dcd420$2a967c60$@gmail.com> Ahmed, Thanks for the detailed explanation. As per http://www.erlang.org/doc/apps/mnesia/Mnesia_chap4.html : ?Durability. This transaction property ensures that changes made to the DBMS by a transaction are permanent. Once a transaction has been committed, all changes made to the database are durable - i.e. they are written safely to disc and will not be corrupted or disappear.? I would definitely expect {aborted, Reason} in case of a storage operation failure. Since then I found another occurrence of this problem. If I?m precautious and put my data in fragmented table, but have a secondary index, that index will still kept in a single dets file. In my case I tried to index SHA hashes and the index grows pretty fast. Thanks again for your answer, even if it was a bad news for me. Mnesia looks to be a great database, but I think it was a mistake from my side to try using it as a complete RDBMS replacement. Best Regards, Tamas From: Ahmed Omar [mailto:spawn.think@REDACTED] Sent: Wednesday, September 04, 2013 2:25 PM To: Janos Hary Cc: erlang-questions; H?kan Mattsson Subject: Re: [erlang-questions] mnesia silent failure and table corruption Hi Janos, Indeed, in that case transactions could be considered succeeded while actual storage operation failed... The thing is, If mnesia needs to write data to a store (in this case dets), it doesn't check for results for this operation. Actually, it does even catch exceptions. (mnesia_tm:do_update/4) As dets have a maximum of 2GB for file size, I believe the table gets corrupted after that. (somebody correct me please) So any operation on that dets table will return the bad_object error. So the commit is considered succeeded, mnesia logs it, transaction returns {atomic, ok} sync_transaction won't help, because it just waits for the commit to be synced/logged, but also doesn't check results of the store operation. so you could end up in situation where dirty write operation returns an error, and in a transaction returns {atomic, ok} > dets:insert(table1, SomeKey, SomeVal). {error,{{bad_object,read_buckets}, "/tmp/table1.DAT"}} > mnesia:dirty_write(SomeRecord). {error,{{bad_object,read_buckets}, "/tmp/table1.DAT"}} > mnesia:transaction(fun()-> mnesia:write(SomeRecord) end). {atomic,ok} You could try to detect that in your transaction by adding an operation that will fail (like read, first, last, ..etc) but that won't protect you against corrupting the file. The question is, should mnesia detect this kind of errors and if yes, what to do about it? Also, can dets protect against the corruption somehow? (reject the operation that could lead to size> 2GB?) Best Regards, Ahmed On Tue, Aug 27, 2013 at 5:06 PM, Janos Hary wrote: Thanks, but no luck. It behaves the same with sync_transaction. Janos -----Original Message----- From: hawk.mattsson@REDACTED [mailto:hawk.mattsson@REDACTED] On Behalf Of H?kan Mattsson Sent: Tuesday, August 27, 2013 4:02 PM To: Janos Hary Cc: erlang-questions Subject: Re: [erlang-questions] mnesia silent failure and table corruption On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary wrote: > All, > > I started to write records into a table from a simple test function. > When the table size reaches 2GB write transactions still report > success, but they are silently ignored. The table became unusable, > read operations return error, and mnesia cannot recover the table. How > shall I avoid such situation? Try using mnesia:sync_transaction() instead of mnesia:transaction(). Then Mnesia will use syncronized calls to disk_log (and syncronized commits when several nodes are involved). /H?kan _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Wed Sep 4 16:53:28 2013 From: spawn.think@REDACTED (Ahmed Omar) Date: Wed, 4 Sep 2013 16:53:28 +0200 Subject: [erlang-questions] mnesia silent failure and table corruption In-Reply-To: <001e01cea979$b8dcd420$2a967c60$@gmail.com> References: <004f01cea336$fa6b2390$ef416ab0$@gmail.com> <001e01cea979$b8dcd420$2a967c60$@gmail.com> Message-ID: Hi Janos, In my view, the main purpose for mnesia is being in-memory distributed database, not a complete RDBMS replacement. There are some projects to user different backends than DETS for mnesia (for persistence data). One project was to a framework for mnesia pluggable backends (based on earlier work of Ulf Wiger on mnesia_ext_filesystem). The first plugin was based on LevelDB. It was presented in EUC 2012 http://www.erlang-factory.com/conference/ErlangUserConference2012/speakers/RolandKarlsson I'm not sure about the project status right now. Best Regards, Ahmed On Wed, Sep 4, 2013 at 4:19 PM, Janos Hary wrote: > Ahmed,**** > > ** ** > > Thanks for the detailed explanation. **** > > ** ** > > As per http://www.erlang.org/doc/apps/mnesia/Mnesia_chap4.html :**** > > ?Durability. This transaction property ensures that changes made to the > DBMS by a transaction are permanent. Once a transaction has been committed, > all changes made to the database are durable - i.e. they are written safely > to disc and will not be corrupted or disappear.?**** > > ** ** > > I would definitely expect {aborted, Reason} in case of a storage operation > failure. **** > > ** ** > > Since then I found another occurrence of this problem. If I?m precautious > and put my data in fragmented table, but have a secondary index, that index > will still kept in a single dets file. In my case I tried to index SHA > hashes and the index grows pretty fast.**** > > ** ** > > Thanks again for your answer, even if it was a bad news for me. Mnesia > looks to be a great database, but I think it was a mistake from my side to > try using it as a complete RDBMS replacement.**** > > ** ** > > Best Regards,**** > > Tamas**** > > ** ** > > *From:* Ahmed Omar [mailto:spawn.think@REDACTED] > *Sent:* Wednesday, September 04, 2013 2:25 PM > *To:* Janos Hary > *Cc:* erlang-questions; H?kan Mattsson > > *Subject:* Re: [erlang-questions] mnesia silent failure and table > corruption**** > > ** ** > > Hi Janos, **** > > ** ** > > Indeed, in that case transactions could be considered succeeded while > actual storage operation failed...**** > > ** ** > > The thing is, If mnesia needs to write data to a store (in this case > dets), it doesn't check for results for this operation. Actually, it does > even catch exceptions. (mnesia_tm:do_update/4)**** > > ** ** > > As dets have a maximum of 2GB for file size, I believe the table gets > corrupted after that. (somebody correct me please)**** > > So any operation on that dets table will return the bad_object error.**** > > ** ** > > So the commit is considered succeeded, mnesia logs it, transaction returns > {atomic, ok}**** > > ** ** > > sync_transaction won't help, because it just waits for the commit to be > synced/logged, but also doesn't check results of the store operation.**** > > ** ** > > so you could end up in situation where dirty write operation returns an > error, and in a transaction returns {atomic, ok}**** > > ** ** > > > dets:insert(table1, SomeKey, SomeVal).**** > > {error,{{bad_object,read_buckets},**** > > "/tmp/table1.DAT"}}**** > > > mnesia:dirty_write(SomeRecord).**** > > {error,{{bad_object,read_buckets},**** > > "/tmp/table1.DAT"}}**** > > > mnesia:transaction(fun()-> mnesia:write(SomeRecord) end). **** > > {atomic,ok}**** > > ** ** > > You could try to detect that in your transaction by adding an operation > that will fail (like read, first, last, ..etc) but that won't protect you > against corrupting the file.**** > > ** ** > > ** ** > > The question is, should mnesia detect this kind of errors and if yes, what > to do about it?**** > > Also, can dets protect against the corruption somehow? (reject the > operation that could lead to size> 2GB?)**** > > ** ** > > ** ** > > Best Regards, **** > > Ahmed**** > > ** ** > > ** ** > > On Tue, Aug 27, 2013 at 5:06 PM, Janos Hary > wrote:**** > > Thanks, but no luck. It behaves the same with sync_transaction. > > Janos**** > > > -----Original Message----- > From: hawk.mattsson@REDACTED [mailto:hawk.mattsson@REDACTED] On Behalf > Of > H?kan Mattsson > Sent: Tuesday, August 27, 2013 4:02 PM > To: Janos Hary > Cc: erlang-questions > Subject: Re: [erlang-questions] mnesia silent failure and table corruption > > On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary > wrote: > > All, > > > > I started to write records into a table from a simple test function. > > When the table size reaches 2GB write transactions still report > > success, but they are silently ignored. The table became unusable, > > read operations return error, and mnesia cannot recover the table. How > > shall I avoid such situation? > > Try using mnesia:sync_transaction() instead of mnesia:transaction(). > Then Mnesia will use syncronized calls to disk_log (and syncronized commits > when several nodes are involved). > > /H?kan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbothma@REDACTED Thu Sep 5 09:46:11 2013 From: jbothma@REDACTED (JD Bothma) Date: Thu, 5 Sep 2013 09:46:11 +0200 Subject: [erlang-questions] eaddrinuse despite {reuseaddr, true} Message-ID: Why can this sometimes exit with exception error: no match of right hand side value {error,eaddrinuse} ? As far as I know, nothing's connecting to the socket in this time. [ begin {ok, LSock} = gen_tcp:listen(12345, [{reuseaddr, true}]), {error, timeout} = gen_tcp:accept(LSock, 0), ok = gen_tcp:close(LSock) end || _X <- lists:seq(1, 100000) ]. This is on R14 and 15 on linux and R16 on OS X. BTW it doesn't seem to happen if I don't accept. I'm debugging this behaviour in yaws with SSL but it also seems to happen with plain gen_tcp. JD -------------- next part -------------- An HTML attachment was scrubbed... URL: From mangalaman93@REDACTED Thu Sep 5 09:58:52 2013 From: mangalaman93@REDACTED (aman mangal) Date: Thu, 5 Sep 2013 13:28:52 +0530 Subject: [erlang-questions] eaddrinuse despite {reuseaddr, true} In-Reply-To: References: Message-ID: Hi, The error is thrown in the first line. It is returning *{error, ** eaddrinuse}* (where you are expeciting *{ok, LSock}*) because the port is already being used by some different process. Two process cannot listen on the same port at a time. Aman Mangal 4th year Undergraduate Student Department of Computer Science & Engineering IIT Bombay www.cse.iitb.ac.in/~amanmangal On Thu, Sep 5, 2013 at 1:16 PM, JD Bothma wrote: > Why can this sometimes exit with exception error: no match of right hand > side value {error,eaddrinuse} ? > > As far as I know, nothing's connecting to the socket in this time. > > [ begin {ok, LSock} = gen_tcp:listen(12345, [{reuseaddr, true}]), > {error, timeout} = gen_tcp:accept(LSock, 0), > ok = gen_tcp:close(LSock) > end > || _X <- lists:seq(1, 100000) ]. > > This is on R14 and 15 on linux and R16 on OS X. > > BTW it doesn't seem to happen if I don't accept. I'm debugging this > behaviour in yaws with SSL but it also seems to happen with plain gen_tcp. > > JD > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbothma@REDACTED Thu Sep 5 10:27:59 2013 From: jbothma@REDACTED (JD Bothma) Date: Thu, 5 Sep 2013 10:27:59 +0200 Subject: [erlang-questions] eaddrinuse despite {reuseaddr, true} In-Reply-To: References: Message-ID: On 5 September 2013 09:58, aman mangal wrote: > > The error is thrown in the first line. It is returning *{error, ** > eaddrinuse}* (where you are expeciting *{ok, LSock}*) because the port > is already being used by some different process. Two process cannot listen > on the same port at a time. > It runs for a while, and sometimes gets through all bind/closes, but sometimes crashes mid-way. reuseaddr should be telling the OS that the closing socket may be reused when this process tries to listen on the same port again. No process was listening on this port before the test. This code should be ensuring that the OS allows listening on the socket even when the last close hasn't completely released the socket. Without reuseaddr, we can sometimes expect eaddrinuse because close returns before the socket is guaranteed to be completely released. That's if I haven't misunderstood something. If this code is correct, either something's happening on my machines that I'm not controlling properly or there's a synchronisation bug in the erlang networking code that I'd like to help identify and solve. > On Thu, Sep 5, 2013 at 1:16 PM, JD Bothma wrote: > >> Why can this sometimes exit with exception error: no match of right hand >> side value {error,eaddrinuse} ? >> >> As far as I know, nothing's connecting to the socket in this time. >> >> [ begin {ok, LSock} = gen_tcp:listen(12345, [{reuseaddr, true}]), >> {error, timeout} = gen_tcp:accept(LSock, 0), >> ok = gen_tcp:close(LSock) >> end >> || _X <- lists:seq(1, 100000) ]. >> >> This is on R14 and 15 on linux and R16 on OS X. >> >> BTW it doesn't seem to happen if I don't accept. I'm debugging this >> behaviour in yaws with SSL but it also seems to happen with plain gen_tcp. >> >> JD >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yash_ganthe@REDACTED Thu Sep 5 12:16:11 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Thu, 5 Sep 2013 10:16:11 +0000 Subject: [erlang-questions] Getting rebar to compile a single module Message-ID: Hi, My application app1 has modules a, b and c. Another application app2 needs just one module from my application. In the apps's rebar.config, I have added app1 as a subdir so that app1 gets compiled. This causes all modules of app1 to get compiled. Is there a way to as rebar to compile only a single module? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Thu Sep 5 13:56:35 2013 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 5 Sep 2013 13:56:35 +0200 Subject: [erlang-questions] eaddrinuse despite {reuseaddr, true} In-Reply-To: References: Message-ID: Hi JD, I think you are maybe confusing reuseadrr or (SO_REUSEADDR) usage in general. AFAIK, In most sockets implementations, you can't bind two sockets with the same exact source(port/IP)/protocol combination. That's not specific to Erlang, but to most used operating systems. Here's a good read on the topic (specially the table for possible combination) http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t A simple example to show one scenario of using reuseaddr: %Listening on a specific IP and on a wildcard > {ok, LSock1} = gen_tcp:listen(12345, [{reuseaddr, true},{ip,{127,0,0,1}}]). {ok,#Port<0.816>} > {ok, LSock2} = gen_tcp:listen(12345, [{reuseaddr, true},{ip,{0,0,0,0}}]). {ok,#Port<0.817>} > %Same without reuseaddr: > > f(). ok > {ok, LSock1} = gen_tcp:listen(12346, [{ip,{127,0,0,1}}]). {ok,#Port<0.818>} > {ok, LSock2} = gen_tcp:listen(12346, [{ip,{0,0,0,0}}]). ** exception error: no match of right hand side value {error,eaddrinuse} So in your case, sometimes it will succeed (if socket got actually closed before the next gen_tcp:listen call, and in other times will fail because the socket was not yet closed. Does this answer your question? Best Regards, Ahmed Omar On Thu, Sep 5, 2013 at 10:27 AM, JD Bothma wrote: > On 5 September 2013 09:58, aman mangal wrote: > >> >> The error is thrown in the first line. It is returning *{error, ** >> eaddrinuse}* (where you are expeciting *{ok, LSock}*) because the port >> is already being used by some different process. Two process cannot listen >> on the same port at a time. >> > > It runs for a while, and sometimes gets through all bind/closes, but > sometimes crashes mid-way. > > reuseaddr should be telling the OS that the closing socket may be reused > when this process tries to listen on the same port again. > > No process was listening on this port before the test. This code should be > ensuring that the OS allows listening on the socket even when the last > close hasn't completely released the socket. Without reuseaddr, we can > sometimes expect eaddrinuse because close returns before the socket is > guaranteed to be completely released. > > That's if I haven't misunderstood something. If this code is correct, > either something's happening on my machines that I'm not controlling > properly or there's a synchronisation bug in the erlang networking code > that I'd like to help identify and solve. > > >> On Thu, Sep 5, 2013 at 1:16 PM, JD Bothma wrote: >> >>> Why can this sometimes exit with exception error: no match of right hand >>> side value {error,eaddrinuse} ? >>> >>> As far as I know, nothing's connecting to the socket in this time. >>> >>> [ begin {ok, LSock} = gen_tcp:listen(12345, [{reuseaddr, true}]), >>> {error, timeout} = gen_tcp:accept(LSock, 0), >>> ok = gen_tcp:close(LSock) >>> end >>> || _X <- lists:seq(1, 100000) ]. >>> >>> This is on R14 and 15 on linux and R16 on OS X. >>> >>> BTW it doesn't seem to happen if I don't accept. I'm debugging this >>> behaviour in yaws with SSL but it also seems to happen with plain gen_tcp. >>> >>> JD >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Sep 5 21:19:17 2013 From: g@REDACTED (Garrett Smith) Date: Thu, 5 Sep 2013 14:19:17 -0500 Subject: [erlang-questions] [ANN] Chicago Erlang Factory Lite - Oct 4 Message-ID: If you can make it to Chicago on Oct 4, we're hosting a Factory Lite! http://www.chicagoerlang.com This is a one day, eight session mini conference diving into Erlang super powers. Some topics: - Erlang/OTP essentials - Erlang in the multicore era - High volume messaging - Laws of fault tolerance - Erlang and web scale To say this will be epic is an understatement. Epicly epic perhaps. More like stupendously epicly epic! If you can join us, take advantage of the early bird pricing, which ends this weekend. Garrett From jbothma@REDACTED Fri Sep 6 11:44:34 2013 From: jbothma@REDACTED (JD Bothma) Date: Fri, 6 Sep 2013 11:44:34 +0200 Subject: [erlang-questions] eaddrinuse despite {reuseaddr, true} In-Reply-To: References: Message-ID: On 5 September 2013 13:56, Ahmed Omar wrote: > Hi JD, > I think you are maybe confusing reuseadrr or (SO_REUSEADDR) usage in > general. > Yeah, only BSD worked the way I thought and still can't guarantee it completely. Thanks for the clear response and link! AFAIK, In most sockets implementations, you can't bind two sockets with the > same exact source(port/IP)/protocol combination. > That's not specific to Erlang, but to most used operating systems. > > Here's a good read on the topic (specially the table for possible > combination) > > http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t > If this is correct (and it looks pretty well-thought-through) the BSD implementation should do what I want with SO_REUSEADDR but the linux implementation won't. "BSD:... However, if SO_REUSEADDR is set for the socket you are trying to bind, another socket bound to the same address and port in state TIME_WAIT is simply ignored, after all its already "half dead", and your socket can bind to exactly the same address without any problem." "Linux:... This option behaves generally the as in BSD with two important exceptions. One exception is that a if a listening (server) TCP socket is already bound to a wildcard IP address and a specific port, no other TCP socket can be bound to the same port, regardless whether either one or both sockets have this flag set. Not even if it would use a more specific address (as is allowed in case of BSD)." I can achieve what I want with Linux kernel 3.9+ and SO_REUSEPORT if OTP supporteds it, but that's irrelevant for my current kernel. Lukas also mentioned that it could be that the emulator returns on the port close before the kernel actually puts the socket in TIME_WAIT (if I understood correctly) and SMP makes it tricky to get serial guarantees on things like this. Both the BSD and Linux 3.9 would have potentially confusing behaviour since the old socket might still be listening when the new socket starts. In the end, for my application, it's best accept that bind can fail and keep trying when the application is designed to rebind the port it had recently and the environment is designed for nothing else to be binding there. JD > > A simple example to show one scenario of using reuseaddr: > %Listening on a specific IP and on a wildcard > > > {ok, LSock1} = gen_tcp:listen(12345, [{reuseaddr, > true},{ip,{127,0,0,1}}]). > {ok,#Port<0.816>} > > {ok, LSock2} = gen_tcp:listen(12345, [{reuseaddr, > true},{ip,{0,0,0,0}}]). > {ok,#Port<0.817>} > > > > %Same without reuseaddr: > > > > > f(). > ok > > {ok, LSock1} = gen_tcp:listen(12346, [{ip,{127,0,0,1}}]). > > {ok,#Port<0.818>} > > {ok, LSock2} = gen_tcp:listen(12346, [{ip,{0,0,0,0}}]). > ** exception error: no match of right hand side value {error,eaddrinuse} > > So in your case, sometimes it will succeed (if socket got actually closed > before the next gen_tcp:listen call, and in other times will fail because > the socket was not yet closed. > > Does this answer your question? > > Best Regards, > Ahmed Omar > > > > On Thu, Sep 5, 2013 at 10:27 AM, JD Bothma wrote: > >> On 5 September 2013 09:58, aman mangal wrote: >> >>> >>> The error is thrown in the first line. It is returning *{error, ** >>> eaddrinuse}* (where you are expeciting *{ok, LSock}*) because the port >>> is already being used by some different process. Two process cannot listen >>> on the same port at a time. >>> >> >> It runs for a while, and sometimes gets through all bind/closes, but >> sometimes crashes mid-way. >> >> reuseaddr should be telling the OS that the closing socket may be reused >> when this process tries to listen on the same port again. >> >> No process was listening on this port before the test. This code should >> be ensuring that the OS allows listening on the socket even when the last >> close hasn't completely released the socket. Without reuseaddr, we can >> sometimes expect eaddrinuse because close returns before the socket is >> guaranteed to be completely released. >> >> That's if I haven't misunderstood something. If this code is correct, >> either something's happening on my machines that I'm not controlling >> properly or there's a synchronisation bug in the erlang networking code >> that I'd like to help identify and solve. >> >> >>> On Thu, Sep 5, 2013 at 1:16 PM, JD Bothma wrote: >>> >>>> Why can this sometimes exit with exception error: no match of right >>>> hand side value {error,eaddrinuse} ? >>>> >>>> As far as I know, nothing's connecting to the socket in this time. >>>> >>>> [ begin {ok, LSock} = gen_tcp:listen(12345, [{reuseaddr, true}]), >>>> {error, timeout} = gen_tcp:accept(LSock, 0), >>>> ok = gen_tcp:close(LSock) >>>> end >>>> || _X <- lists:seq(1, 100000) ]. >>>> >>>> This is on R14 and 15 on linux and R16 on OS X. >>>> >>>> BTW it doesn't seem to happen if I don't accept. I'm debugging this >>>> behaviour in yaws with SSL but it also seems to happen with plain gen_tcp. >>>> >>>> JD >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Fri Sep 6 18:00:19 2013 From: robert.virding@REDACTED (Robert Virding) Date: Fri, 6 Sep 2013 18:00:19 +0200 (CEST) Subject: [erlang-questions] erlounge in nyc In-Reply-To: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> Message-ID: <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> Are people interested in a NYC erlounge? I will be there next week and am free all evenings except Wednesday. Someone suggest when and where. Robert ----- Original Message ----- > From: "Robert Virding" > To: "Bob Ippolito" > Cc: "Erlang" , "Roberto Ostinelli" > > Sent: Monday, 2 September, 2013 6:04:04 PM > Subject: Re: [erlang-questions] erlounge in nyc > I will be in NYC 10-15 sept so I will do my best to attend. Erlounges are > great. > Robert > ----- Original Message ----- > > From: "Bob Ippolito" > > > To: "Roberto Ostinelli" > > > Cc: "Erlang" > > > Sent: Friday, 30 August, 2013 5:17:52 AM > > > Subject: Re: [erlang-questions] erlounge in nyc > > > I'm in NYC from Sep 5-13 so I could probably also attend > > > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > > > > I moved to NYC. If this happens I'm in too. > > > > > > r. > > > > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies < > > > heinz@REDACTED > > > > > > > wrote: > > > > > > > I'll do my best to make it too :) > > > > > > > > > > Cheers, > > > > > > > > > > Heinz > > > > > > > > > > On Aug 29, 2013, at 22:16, Garry Hodgson < garry@REDACTED > > > > > wrote: > > > > > > > > > > > i would be love to get together with some other > > > > > > > > > > > erlang folks, if this is still on the table. > > > > > > > > > > > > > > > > > > > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > > > > > > > > > > >> Hello, > > > > > > > > > > >> > > > > > > > > > > >> Somebody, and I forget who asked me, and I forget when, to drop them > > > > >> a > > > > >> line > > > > > > > > > > >> if I was every in New York. > > > > > > > > > > >> > > > > > > > > > > >> I'll be in New York from 10 - 14 sept, if anybody is interested in a > > > > >> meetup. > > > > > > > > > > >> > > > > > > > > > > >> /Joe > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Garry Hodgson > > > > > > > > > > > AT&T Chief Security Office (CSO) > > > > > > > > > > > > > > > > > > > > > > "This e-mail and any files transmitted with it are AT&T property, are > > > > > confidential, and are intended solely for the use of the individual > > > > > or > > > > > entity to whom this e-mail is addressed. If you are not one of the > > > > > named > > > > > recipient(s) or otherwise have reason to believe that you have > > > > > received > > > > > this message in error, please notify the sender and delete this > > > > > message > > > > > immediately from your computer. Any other use, retention, > > > > > dissemination, > > > > > forwarding, printing, or copying of this e-mail is strictly > > > > > prohibited." > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > erlang-questions mailing list > > > > > > > > > > > erlang-questions@REDACTED > > > > > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > _______________________________________________ > > > > > > > > > > erlang-questions mailing list > > > > > > > > > > erlang-questions@REDACTED > > > > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Fri Sep 6 18:54:42 2013 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 6 Sep 2013 12:54:42 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> Message-ID: AFA the 'when': I believe the matching dates for everyone would have to be either Thursday 12th or Friday 13th. I'm new to NY so I'm not in a position to suggest the 'when' part. :) r. On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding < robert.virding@REDACTED> wrote: > Are people interested in a NYC erlounge? I will be there next week and am > free all evenings except Wednesday. Someone suggest when and where. > > Robert > > > ------------------------------ > > *From: *"Robert Virding" > *To: *"Bob Ippolito" > *Cc: *"Erlang" , "Roberto Ostinelli" < > roberto@REDACTED> > *Sent: *Monday, 2 September, 2013 6:04:04 PM > > *Subject: *Re: [erlang-questions] erlounge in nyc > > I will be in NYC 10-15 sept so I will do my best to attend. Erlounges are > great. > > Robert > > ------------------------------ > > *From: *"Bob Ippolito" > *To: *"Roberto Ostinelli" > *Cc: *"Erlang" > *Sent: *Friday, 30 August, 2013 5:17:52 AM > *Subject: *Re: [erlang-questions] erlounge in nyc > > I'm in NYC from Sep 5-13 so I could probably also attend > > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > >> I moved to NYC. If this happens I'm in too. >> >> r. >> >> >> On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies > > wrote: >> >>> I'll do my best to make it too :) >>> >>> Cheers, >>> Heinz >>> On Aug 29, 2013, at 22:16, Garry Hodgson wrote: >>> >>> > i would be love to get together with some other >>> > erlang folks, if this is still on the table. >>> > >>> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >>> >> Hello, >>> >> >>> >> Somebody, and I forget who asked me, and I forget when, to drop them >>> a line >>> >> if I was every in New York. >>> >> >>> >> I'll be in New York from 10 - 14 sept, if anybody is interested in a >>> meetup. >>> >> >>> >> /Joe >>> > >>> > -- >>> > Garry Hodgson >>> > AT&T Chief Security Office (CSO) >>> > >>> > "This e-mail and any files transmitted with it are AT&T property, are >>> confidential, and are intended solely for the use of the individual or >>> entity to whom this e-mail is addressed. If you are not one of the named >>> recipient(s) or otherwise have reason to believe that you have received >>> this message in error, please notify the sender and delete this message >>> immediately from your computer. Any other use, retention, dissemination, >>> forwarding, printing, or copying of this e-mail is strictly prohibited." >>> > >>> > _______________________________________________ >>> > erlang-questions mailing list >>> > erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Fri Sep 6 18:55:25 2013 From: roberto@REDACTED (Roberto Ostinelli) Date: Fri, 6 Sep 2013 12:55:25 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> Message-ID: *where* I'm new to NY so I'm not in a position to suggest the 'when' part. :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siraaj@REDACTED Fri Sep 6 19:57:32 2013 From: siraaj@REDACTED (Siraaj Khandkar) Date: Fri, 06 Sep 2013 13:57:32 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> Message-ID: <522A178C.6060300@khandkar.net> Count me in! I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for that reason :)). On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: > AFA the 'when': I believe the matching dates for everyone would have to > be either Thursday 12th or Friday 13th. > > I'm new to NY so I'm not in a position to suggest the 'when' part. :) > > r. > > > On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding > > wrote: > > Are people interested in a NYC erlounge? I will be there next week > and am free all evenings except Wednesday. Someone suggest when and > where. > > Robert > > > ------------------------------------------------------------------------ > > *From: *"Robert Virding" > > *To: *"Bob Ippolito" > > *Cc: *"Erlang" >, "Roberto Ostinelli" > > > *Sent: *Monday, 2 September, 2013 6:04:04 PM > > *Subject: *Re: [erlang-questions] erlounge in nyc > > I will be in NYC 10-15 sept so I will do my best to attend. > Erlounges are great. > > Robert > > ------------------------------------------------------------------------ > > *From: *"Bob Ippolito" > > *To: *"Roberto Ostinelli" > > *Cc: *"Erlang" > > *Sent: *Friday, 30 August, 2013 5:17:52 AM > *Subject: *Re: [erlang-questions] erlounge in nyc > > I'm in NYC from Sep 5-13 so I could probably also attend > > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > > I moved to NYC. If this happens I'm in too. > > r. > > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies > wrote: > > I'll do my best to make it too :) > > Cheers, > Heinz > On Aug 29, 2013, at 22:16, Garry Hodgson > wrote: > > > i would be love to get together with some other > > erlang folks, if this is still on the table. > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > >> Hello, > >> > >> Somebody, and I forget who asked me, and I > forget when, to drop them a line > >> if I was every in New York. > >> > >> I'll be in New York from 10 - 14 sept, if > anybody is interested in a meetup. > >> > >> /Joe > > > > -- > > Garry Hodgson > > AT&T Chief Security Office (CSO) > > > > "This e-mail and any files transmitted with it > are AT&T property, are confidential, and are > intended solely for the use of the individual or > entity to whom this e-mail is addressed. If you are > not one of the named recipient(s) or otherwise have > reason to believe that you have received this > message in error, please notify the sender and > delete this message immediately from your computer. > Any other use, retention, dissemination, forwarding, > printing, or copying of this e-mail is strictly > prohibited." > > From bob@REDACTED Fri Sep 6 20:57:19 2013 From: bob@REDACTED (Bob Ippolito) Date: Fri, 6 Sep 2013 11:57:19 -0700 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <522A178C.6060300@khandkar.net> References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> Message-ID: I can do Thursday the 12th, but not Friday (flying out that day). On Fri, Sep 6, 2013 at 10:57 AM, Siraaj Khandkar wrote: > Count me in! > > I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for > that reason :)). > > > > On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: > >> AFA the 'when': I believe the matching dates for everyone would have to >> be either Thursday 12th or Friday 13th. >> >> I'm new to NY so I'm not in a position to suggest the 'when' part. :) >> >> r. >> >> >> On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding >> >> >> >> wrote: >> >> Are people interested in a NYC erlounge? I will be there next week >> and am free all evenings except Wednesday. Someone suggest when and >> where. >> >> Robert >> >> >> ------------------------------**------------------------------** >> ------------ >> >> *From: *"Robert Virding" >> >> >> >> *To: *"Bob Ippolito" > >> *Cc: *"Erlang" > >>, >> "Roberto Ostinelli" >> > >> *Sent: *Monday, 2 September, 2013 6:04:04 PM >> >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> >> I will be in NYC 10-15 sept so I will do my best to attend. >> Erlounges are great. >> >> Robert >> >> ------------------------------**------------------------------** >> ------------ >> >> *From: *"Bob Ippolito" > >> >> *To: *"Roberto Ostinelli" > > >> *Cc: *"Erlang" > >> >> >> *Sent: *Friday, 30 August, 2013 5:17:52 AM >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> >> I'm in NYC from Sep 5-13 so I could probably also attend >> >> On Thursday, August 29, 2013, Roberto Ostinelli wrote: >> >> I moved to NYC. If this happens I'm in too. >> >> r. >> >> >> >> On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies >> wrote: >> >> I'll do my best to make it too :) >> >> Cheers, >> Heinz >> On Aug 29, 2013, at 22:16, Garry Hodgson >> wrote: >> >> > i would be love to get together with some other >> > erlang folks, if this is still on the table. >> > >> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >> >> Hello, >> >> >> >> Somebody, and I forget who asked me, and I >> forget when, to drop them a line >> >> if I was every in New York. >> >> >> >> I'll be in New York from 10 - 14 sept, if >> anybody is interested in a meetup. >> >> >> >> /Joe >> > >> > -- >> > Garry Hodgson >> > AT&T Chief Security Office (CSO) >> > >> > "This e-mail and any files transmitted with it >> are AT&T property, are confidential, and are >> intended solely for the use of the individual or >> entity to whom this e-mail is addressed. If you are >> not one of the named recipient(s) or otherwise have >> reason to believe that you have received this >> message in error, please notify the sender and >> delete this message immediately from your computer. >> Any other use, retention, dissemination, forwarding, >> printing, or copying of this e-mail is strictly >> prohibited." >> > >> > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahesh@REDACTED Fri Sep 6 21:05:12 2013 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Fri, 6 Sep 2013 15:05:12 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <522A178C.6060300@khandkar.net> References: <52200081.3070302@research.att.com> <79D98322-2A7E-4347-AB7D-C5ACE41AF5A7@licenser.net> <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> Message-ID: Thursday works quite well for me too. How about at Biblio (in Williamsburg) Excellent hipster bar with great beers ( hipsters and beer. Well duh) and excellent munchies... http://bibliobrooklyn.com/2013/06/05/beer-menu/ Sent via smoke signals... Mahesh Paolini-Subramanya That tall bald Indian guy... On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" wrote: > Count me in! > > I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for > that reason :)). > > > On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: > >> AFA the 'when': I believe the matching dates for everyone would have to >> be either Thursday 12th or Friday 13th. >> >> I'm new to NY so I'm not in a position to suggest the 'when' part. :) >> >> r. >> >> >> On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding >> >> >> >> wrote: >> >> Are people interested in a NYC erlounge? I will be there next week >> and am free all evenings except Wednesday. Someone suggest when and >> where. >> >> Robert >> >> >> ------------------------------**------------------------------** >> ------------ >> >> *From: *"Robert Virding" >> >> >> >> *To: *"Bob Ippolito" > >> *Cc: *"Erlang" > >>, >> "Roberto Ostinelli" >> > >> *Sent: *Monday, 2 September, 2013 6:04:04 PM >> >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> I will be in NYC 10-15 sept so I will do my best to attend. >> Erlounges are great. >> >> Robert >> >> ------------------------------**------------------------------** >> ------------ >> >> *From: *"Bob Ippolito" > >> >> *To: *"Roberto Ostinelli" > > >> *Cc: *"Erlang" > >> >> >> *Sent: *Friday, 30 August, 2013 5:17:52 AM >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> I'm in NYC from Sep 5-13 so I could probably also attend >> >> On Thursday, August 29, 2013, Roberto Ostinelli wrote: >> >> I moved to NYC. If this happens I'm in too. >> >> r. >> >> >> >> On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies >> wrote: >> >> I'll do my best to make it too :) >> >> Cheers, >> Heinz >> On Aug 29, 2013, at 22:16, Garry Hodgson >> wrote: >> >> > i would be love to get together with some other >> > erlang folks, if this is still on the table. >> > >> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >> >> Hello, >> >> >> >> Somebody, and I forget who asked me, and I >> forget when, to drop them a line >> >> if I was every in New York. >> >> >> >> I'll be in New York from 10 - 14 sept, if >> anybody is interested in a meetup. >> >> >> >> /Joe >> > >> > -- >> > Garry Hodgson >> > AT&T Chief Security Office (CSO) >> > >> > "This e-mail and any files transmitted with it >> are AT&T property, are confidential, and are >> intended solely for the use of the individual or >> entity to whom this e-mail is addressed. If you are >> not one of the named recipient(s) or otherwise have >> reason to believe that you have received this >> message in error, please notify the sender and >> delete this message immediately from your computer. >> Any other use, retention, dissemination, forwarding, >> printing, or copying of this e-mail is strictly >> prohibited." >> > >> > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Sat Sep 7 04:42:56 2013 From: colanderman@REDACTED (Chris King) Date: Fri, 06 Sep 2013 22:42:56 -0400 Subject: [erlang-questions] NIF bitstrings? Message-ID: I cannot find a way to work with bitstrings (i.e. non-multiple-of-8-sized binaries) in NIFs. i.e. I see no equivalent to erlang:bit_size/1. What am I missing? From vinoski@REDACTED Sat Sep 7 14:00:21 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Sat, 7 Sep 2013 08:00:21 -0400 Subject: [erlang-questions] NIF bitstrings? In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 10:42 PM, Chris King wrote: > I cannot find a way to work with bitstrings (i.e. non-multiple-of-8-sized > binaries) in NIFs. i.e. I see no equivalent to erlang:bit_size/1. What am > I missing? > According to http://www.erlang.org/doc/man/erl_nif.html: "Bitstrings with an arbitrary bit length have no support yet." --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Sat Sep 7 14:59:40 2013 From: colanderman@REDACTED (Chris King) Date: Sat, 07 Sep 2013 08:59:40 -0400 Subject: [erlang-questions] NIF bitstrings? In-Reply-To: References: Message-ID: On Sat, 07 Sep 2013 08:00:21 -0400, Steve Vinoski wrote: > According to http://www.erlang.org/doc/man/erl_nif.html: > > "Bitstrings with an arbitrary bit length have no support yet." Ah I missed that. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Sep 7 23:22:58 2013 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 7 Sep 2013 23:22:58 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: <20130827031421.GB10745@hijacked.us> References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> <20130827031421.GB10745@hijacked.us> Message-ID: <2EC4CE9C-DCCA-4DA3-BE9B-AA757B23FAE8@feuerlabs.com> If library applications are listed in reltool.config as {App, load}, they will not be started. Also, as regards dependencies for parse transforms, I recently wrote a fairly complex parse transform that I thought is both well-contained and fairly easy to understand and maintain. https://github.com/uwiger/locks/blob/master/src/locks_watcher.erl A brief explanation. In the module being transformed, a pseudo-call to locks_watcher/1 is replaced by code returning {erl_eval, exprs, [Es,Bs]}, which is then used in spawn/4, in order to launch a script on a remote node. watch_node(N) -> {M, F, A} = locks_watcher(self()), % expanded through parse_transform spawn(N, M, F, A). The parse_transform function actually lifts the abstract code from its own debug info, and also inlines any local calls made from the lifted function. The transform and inlining code is only ca 80 lines of code and should be fairly reusable (sans the slight absurdity of generating abstract abstract code). So in the linked module above, lines 22-65 contain the code that's lifted into the transformed module. It's plain Erlang, with the only real complication that recursion is done using funs. BR, Ulf W On 27 Aug 2013, at 05:14, Andrew Thompson wrote: > On Mon, Aug 26, 2013 at 09:56:55PM -0500, Paul Davis wrote: >> Andrew, >> >> Can you explain the reticence about having compiler and syntax_tools as >> dependencies? At first I was mostly assuming you wanted lager to handle >> compiler logging or some such so it was a question of startup ordering. >> Given goldrush I'm wondering if it were a more a general concern about just >> removing dependencies for file sizes and so on. > > It was mainly the complexity incurred for application startup by > depending on those applications, even though they're only library > applications, the application controller insists they be 'started'. > > That complexity seems to be mostly handled by lager now, so the original > reasons may not have any grounding anymore. > > Andrew > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From ulf@REDACTED Sun Sep 8 00:08:31 2013 From: ulf@REDACTED (Ulf Wiger) Date: Sun, 8 Sep 2013 00:08:31 +0200 Subject: [erlang-questions] lager and dynamic log cutting via compiled module In-Reply-To: <2EC4CE9C-DCCA-4DA3-BE9B-AA757B23FAE8@feuerlabs.com> References: <2B0B4A59-285E-4300-8297-AA7E6EAB1DCA@feuerlabs.com> <20130826171136.GA10745@hijacked.us> <20130827031421.GB10745@hijacked.us> <2EC4CE9C-DCCA-4DA3-BE9B-AA757B23FAE8@feuerlabs.com> Message-ID: I'm not sure my example was really relevant to this thread, but I still think it's pretty cool. :) /Ulf W Ulf Wiger, Feuerlabs, Inc. http://feuerlabs.com 7 sep 2013 kl. 23:22 skrev Ulf Wiger : > > If library applications are listed in reltool.config as {App, load}, they will not be started. > > Also, as regards dependencies for parse transforms, I recently wrote a fairly > complex parse transform that I thought is both well-contained and fairly > easy to understand and maintain. > > https://github.com/uwiger/locks/blob/master/src/locks_watcher.erl > > A brief explanation. In the module being transformed, a pseudo-call to > locks_watcher/1 is replaced by code returning {erl_eval, exprs, [Es,Bs]}, > which is then used in spawn/4, in order to launch a script on a remote > node. > > watch_node(N) -> > {M, F, A} = > locks_watcher(self()), % expanded through parse_transform > spawn(N, M, F, A). > > The parse_transform function actually lifts the abstract code from its own > debug info, and also inlines any local calls made from the lifted function. > The transform and inlining code is only ca 80 lines of code and should > be fairly reusable (sans the slight absurdity of generating abstract abstract > code). > > So in the linked module above, lines 22-65 contain the code that's lifted > into the transformed module. It's plain Erlang, with the only real complication > that recursion is done using funs. > > BR, > Ulf W > > On 27 Aug 2013, at 05:14, Andrew Thompson wrote: > >> On Mon, Aug 26, 2013 at 09:56:55PM -0500, Paul Davis wrote: >>> Andrew, >>> >>> Can you explain the reticence about having compiler and syntax_tools as >>> dependencies? At first I was mostly assuming you wanted lager to handle >>> compiler logging or some such so it was a question of startup ordering. >>> Given goldrush I'm wondering if it were a more a general concern about just >>> removing dependencies for file sizes and so on. >> >> It was mainly the complexity incurred for application startup by >> depending on those applications, even though they're only library >> applications, the application controller insists they be 'started'. >> >> That complexity seems to be mostly handled by lager now, so the original >> reasons may not have any grounding anymore. >> >> Andrew >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From dmkolesnikov@REDACTED Sun Sep 8 21:08:13 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 8 Sep 2013 22:08:13 +0300 Subject: [erlang-questions] dets disk i/o In-Reply-To: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@gmail.com> References: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@gmail.com> Message-ID: <640DA627-EE35-4CB2-8927-0F40309A79D4@gmail.com> ping! It looks that I should measure it by myself. Best Regards, Dmitry On Sep 4, 2013, at 12:09 AM, Dmitry Kolesnikov wrote: > Hello, > > Does any one run a disk i/o analysis on dets tables? > I am very curious about bag case, write scenario. > > Let's assume that I have a key which is associated with multiple entries. > What is dets behaviour when I am writing a new entry to same key. > Does it make append-only to data slot? or Does it read/write whole slot? > > Best Regards, > Dmitry From robert.virding@REDACTED Sun Sep 8 21:57:40 2013 From: robert.virding@REDACTED (Robert Virding) Date: Sun, 8 Sep 2013 21:57:40 +0200 (CEST) Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> Message-ID: <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> That sounds very reasonable, and near the subway too. :-) Any time after 6pm suits me. Robert ----- Original Message ----- > From: "Mahesh Paolini-Subramanya" > To: "Siraaj Khandkar" > Cc: "Erlang" , "Roberto Ostinelli" > > Sent: Friday, 6 September, 2013 9:05:12 PM > Subject: Re: [erlang-questions] erlounge in nyc > Thursday works quite well for me too. > How about at Biblio (in Williamsburg) Excellent hipster bar with great beers > ( hipsters and beer. Well duh) and excellent munchies... > http://bibliobrooklyn.com/2013/06/05/beer-menu/ > Sent via smoke signals... > Mahesh Paolini-Subramanya > That tall bald Indian guy... > On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" < siraaj@REDACTED > wrote: > > Count me in! > > > I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for > > that reason :)). > > > On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: > > > > AFA the 'when': I believe the matching dates for everyone would have to > > > > > > be either Thursday 12th or Friday 13th. > > > > > > I'm new to NY so I'm not in a position to suggest the 'when' part. :) > > > > > > r. > > > > > > On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding > > > > > > < robert.virding@REDACTED solutions.com > > > > > > > wrote: > > > > > > Are people interested in a NYC erlounge? I will be there next week > > > > > > and am free all evenings except Wednesday. Someone suggest when and > > > > > > where. > > > > > > Robert > > > > > > ------------------------------ ------------------------------ > > > ------------ > > > > > > *From: *"Robert Virding" < robert.virding@REDACTED solutions.com > > > > > > > > > > > > > *To: *"Bob Ippolito" < bob@REDACTED > > > > > > > *Cc: *"Erlang" < erlang-questions@REDACTED > > > > > > >, "Roberto Ostinelli" > > > > > > < roberto@REDACTED > > > > > > > *Sent: *Monday, 2 September, 2013 6:04:04 PM > > > > > > *Subject: *Re: [erlang-questions] erlounge in nyc > > > > > > I will be in NYC 10-15 sept so I will do my best to attend. > > > > > > Erlounges are great. > > > > > > Robert > > > > > > ------------------------------ ------------------------------ > > > ------------ > > > > > > *From: *"Bob Ippolito" < bob@REDACTED > > > > > > > *To: *"Roberto Ostinelli" < roberto@REDACTED > > > > > > > > > > > > > *Cc: *"Erlang" < erlang-questions@REDACTED > > > > > > > > > > > > > *Sent: *Friday, 30 August, 2013 5:17:52 AM > > > > > > *Subject: *Re: [erlang-questions] erlounge in nyc > > > > > > I'm in NYC from Sep 5-13 so I could probably also attend > > > > > > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > > > > > > I moved to NYC. If this happens I'm in too. > > > > > > r. > > > > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies > > > > > > < heinz@REDACTED > wrote: > > > > > > I'll do my best to make it too :) > > > > > > Cheers, > > > > > > Heinz > > > > > > On Aug 29, 2013, at 22:16, Garry Hodgson > > > > > > < garry@REDACTED > wrote: > > > > > > > i would be love to get together with some other > > > > > > > erlang folks, if this is still on the table. > > > > > > > > > > > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > > > > > > >> Hello, > > > > > > >> > > > > > > >> Somebody, and I forget who asked me, and I > > > > > > forget when, to drop them a line > > > > > > >> if I was every in New York. > > > > > > >> > > > > > > >> I'll be in New York from 10 - 14 sept, if > > > > > > anybody is interested in a meetup. > > > > > > >> > > > > > > >> /Joe > > > > > > > > > > > > > > -- > > > > > > > Garry Hodgson > > > > > > > AT&T Chief Security Office (CSO) > > > > > > > > > > > > > > "This e-mail and any files transmitted with it > > > > > > are AT&T property, are confidential, and are > > > > > > intended solely for the use of the individual or > > > > > > entity to whom this e-mail is addressed. If you are > > > > > > not one of the named recipient(s) or otherwise have > > > > > > reason to believe that you have received this > > > > > > message in error, please notify the sender and > > > > > > delete this message immediately from your computer. > > > > > > Any other use, retention, dissemination, forwarding, > > > > > > printing, or copying of this e-mail is strictly > > > > > > prohibited." > > > > > > > > > > > > ______________________________ _________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/ listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Sun Sep 8 22:03:10 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Sun, 8 Sep 2013 22:03:10 +0200 Subject: [erlang-questions] dets disk i/o In-Reply-To: <640DA627-EE35-4CB2-8927-0F40309A79D4@gmail.com> References: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@gmail.com> <640DA627-EE35-4CB2-8927-0F40309A79D4@gmail.com> Message-ID: <002b01ceacce$728b3810$57a1a830$@gmail.com> Dimitry, Not exactly what you asked, but I measured the write performance of mnesia disc_only bag table, which is backed by dets. I used the following function to generate records, where I can set the amount of repeating keys by ?REPCOUNT: fill_modulo(_, Tab, 0, _, _, AccessMod) -> {ok, tab_size(Tab, AccessMod)}; fill_modulo(run, Tab, N, Max, T, AccessMod) -> T1 = progress(N, Max, T), R = #trec{ dbid = N rem ?REPCOUNT, data1 = <<"Hello World!">>, data2 = Max - N }, case write_transaction(Tab, R, AccessMod) of ok -> fill_modulo(run, Tab, N-1, Max, T1, AccessMod); error -> {error, Max-N} end. This is the result of inserting 4M records with ?REPCOUNT set to 10000 (each line shows the running time of the given 5% of the whole task): ......... 11.01s at 5% ......... 11.44s at 10% ......... 12.39s at 15% ......... 12.37s at 20% ......... 12.89s at 25% ......... 13.53s at 30% ......... 14.09s at 35% ......... 14.04s at 40% ......... 15.07s at 45% ......... 14.84s at 50% ......... 15.52s at 55% ......... 15.49s at 60% ......... 16.57s at 65% ......... 16.25s at 70% ......... 17.60s at 75% ......... 16.89s at 80% ......... 17.41s at 85% ......... 18.63s at 90% ......... 17.77s at 95% ......... tab7 -> (ok) modulo insert into bag 4000000 of 4000000 in 306.931000s And this is the result of the same test ran with ?REPCOUNT set to 100: ......... 30.34s at 5% ......... 70.68s at 10% ......... 117.20s at 15% ......... 163.29s at 20% ......... 203.08s at 25% ......... 246.20s at 30% ... and giving up waiting. Since then I learned some other problems using mnesia/dets in this manner (see my post 'mnesia silent failure and table corruption' and Ahmed Omar's reply), and I stopped further investigation. Now I'm evaluating Berkely DB for the task. Best Regards, Janos -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Dmitry Kolesnikov Sent: Sunday, September 08, 2013 9:08 PM To: erlang-questions Questions Subject: Re: [erlang-questions] dets disk i/o ping! It looks that I should measure it by myself. Best Regards, Dmitry On Sep 4, 2013, at 12:09 AM, Dmitry Kolesnikov wrote: > Hello, > > Does any one run a disk i/o analysis on dets tables? > I am very curious about bag case, write scenario. > > Let's assume that I have a key which is associated with multiple entries. > What is dets behaviour when I am writing a new entry to same key. > Does it make append-only to data slot? or Does it read/write whole slot? > > Best Regards, > Dmitry _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From overminddl1@REDACTED Sun Sep 8 22:05:20 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Sun, 8 Sep 2013 14:05:20 -0600 Subject: [erlang-questions] dets disk i/o In-Reply-To: <002b01ceacce$728b3810$57a1a830$@gmail.com> References: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@gmail.com> <640DA627-EE35-4CB2-8927-0F40309A79D4@gmail.com> <002b01ceacce$728b3810$57a1a830$@gmail.com> Message-ID: I would recommend leveldb if you need local and fast storage, there is a great erlang library for it too. On Sep 8, 2013 2:03 PM, "Janos Hary" wrote: > Dimitry, > > Not exactly what you asked, but I measured the write performance of mnesia > disc_only bag table, which is backed by dets. I used the following function > to generate records, where I can set the amount of repeating keys by > ?REPCOUNT: > > fill_modulo(_, Tab, 0, _, _, AccessMod) -> > {ok, tab_size(Tab, AccessMod)}; > fill_modulo(run, Tab, N, Max, T, AccessMod) -> > T1 = progress(N, Max, T), > R = #trec{ dbid = N rem ?REPCOUNT, data1 = <<"Hello World!">>, > data2 > = Max - N }, > case write_transaction(Tab, R, AccessMod) of > ok -> > fill_modulo(run, Tab, N-1, Max, T1, AccessMod); > error -> > {error, Max-N} > end. > > This is the result of inserting 4M records with ?REPCOUNT set to 10000 > (each > line shows the running time of the given 5% of the whole task): > ......... 11.01s at 5% > ......... 11.44s at 10% > ......... 12.39s at 15% > ......... 12.37s at 20% > ......... 12.89s at 25% > ......... 13.53s at 30% > ......... 14.09s at 35% > ......... 14.04s at 40% > ......... 15.07s at 45% > ......... 14.84s at 50% > ......... 15.52s at 55% > ......... 15.49s at 60% > ......... 16.57s at 65% > ......... 16.25s at 70% > ......... 17.60s at 75% > ......... 16.89s at 80% > ......... 17.41s at 85% > ......... 18.63s at 90% > ......... 17.77s at 95% > ......... > tab7 -> (ok) modulo insert into bag 4000000 of 4000000 in 306.931000s > > And this is the result of the same test ran with ?REPCOUNT set to 100: > ......... 30.34s at 5% > ......... 70.68s at 10% > ......... 117.20s at 15% > ......... 163.29s at 20% > ......... 203.08s at 25% > ......... 246.20s at 30% > ... and giving up waiting. > > Since then I learned some other problems using mnesia/dets in this manner > (see my post 'mnesia silent failure and table corruption' and Ahmed Omar's > reply), and I stopped further investigation. Now I'm evaluating Berkely DB > for the task. > > Best Regards, > Janos > > -----Original Message----- > From: erlang-questions-bounces@REDACTED > [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Dmitry > Kolesnikov > Sent: Sunday, September 08, 2013 9:08 PM > To: erlang-questions Questions > Subject: Re: [erlang-questions] dets disk i/o > > ping! > It looks that I should measure it by myself. > > Best Regards, > Dmitry > > On Sep 4, 2013, at 12:09 AM, Dmitry Kolesnikov > wrote: > > > Hello, > > > > Does any one run a disk i/o analysis on dets tables? > > I am very curious about bag case, write scenario. > > > > Let's assume that I have a key which is associated with multiple entries. > > What is dets behaviour when I am writing a new entry to same key. > > Does it make append-only to data slot? or Does it read/write whole slot? > > > > Best Regards, > > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sun Sep 8 22:17:42 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 8 Sep 2013 23:17:42 +0300 Subject: [erlang-questions] dets disk i/o In-Reply-To: References: <1129337E-4FFF-4CDA-8E12-FAF23772C7F4@gmail.com> <640DA627-EE35-4CB2-8927-0F40309A79D4@gmail.com> <002b01ceacce$728b3810$57a1a830$@gmail.com> Message-ID: <46668F46-288D-4D46-87E5-9E2AF2D8B6CA@gmail.com> leveldb uses LSM-tree while dets linear hash like schema. I do agree about leveldb usage in general. However, I am looking for explicit linear hashing storage. - Dmitry On Sep 8, 2013, at 11:05 PM, OvermindDL1 wrote: > I would recommend leveldb if you need local and fast storage, there is a great erlang library for it too. > > On Sep 8, 2013 2:03 PM, "Janos Hary" wrote: > Dimitry, > > Not exactly what you asked, but I measured the write performance of mnesia > disc_only bag table, which is backed by dets. I used the following function > to generate records, where I can set the amount of repeating keys by > ?REPCOUNT: > > fill_modulo(_, Tab, 0, _, _, AccessMod) -> > {ok, tab_size(Tab, AccessMod)}; > fill_modulo(run, Tab, N, Max, T, AccessMod) -> > T1 = progress(N, Max, T), > R = #trec{ dbid = N rem ?REPCOUNT, data1 = <<"Hello World!">>, data2 > = Max - N }, > case write_transaction(Tab, R, AccessMod) of > ok -> > fill_modulo(run, Tab, N-1, Max, T1, AccessMod); > error -> > {error, Max-N} > end. > > This is the result of inserting 4M records with ?REPCOUNT set to 10000 (each > line shows the running time of the given 5% of the whole task): > ......... 11.01s at 5% > ......... 11.44s at 10% > ......... 12.39s at 15% > ......... 12.37s at 20% > ......... 12.89s at 25% > ......... 13.53s at 30% > ......... 14.09s at 35% > ......... 14.04s at 40% > ......... 15.07s at 45% > ......... 14.84s at 50% > ......... 15.52s at 55% > ......... 15.49s at 60% > ......... 16.57s at 65% > ......... 16.25s at 70% > ......... 17.60s at 75% > ......... 16.89s at 80% > ......... 17.41s at 85% > ......... 18.63s at 90% > ......... 17.77s at 95% > ......... > tab7 -> (ok) modulo insert into bag 4000000 of 4000000 in 306.931000s > > And this is the result of the same test ran with ?REPCOUNT set to 100: > ......... 30.34s at 5% > ......... 70.68s at 10% > ......... 117.20s at 15% > ......... 163.29s at 20% > ......... 203.08s at 25% > ......... 246.20s at 30% > ... and giving up waiting. > > Since then I learned some other problems using mnesia/dets in this manner > (see my post 'mnesia silent failure and table corruption' and Ahmed Omar's > reply), and I stopped further investigation. Now I'm evaluating Berkely DB > for the task. > > Best Regards, > Janos > > -----Original Message----- > From: erlang-questions-bounces@REDACTED > [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Dmitry Kolesnikov > Sent: Sunday, September 08, 2013 9:08 PM > To: erlang-questions Questions > Subject: Re: [erlang-questions] dets disk i/o > > ping! > It looks that I should measure it by myself. > > Best Regards, > Dmitry > > On Sep 4, 2013, at 12:09 AM, Dmitry Kolesnikov > wrote: > > > Hello, > > > > Does any one run a disk i/o analysis on dets tables? > > I am very curious about bag case, write scenario. > > > > Let's assume that I have a key which is associated with multiple entries. > > What is dets behaviour when I am writing a new entry to same key. > > Does it make append-only to data slot? or Does it read/write whole slot? > > > > Best Regards, > > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ababkin@REDACTED Sun Sep 8 23:14:50 2013 From: ababkin@REDACTED (Alex Babkin) Date: Sun, 8 Sep 2013 17:14:50 -0400 Subject: [erlang-questions] squeezing performance out of cowboy - how? Message-ID: Hi Was trying to load test my cowboy based websever with tsung and httperf, and it seems like i'm unable to squeeze out more than 200 req/s from even the rest_hello_world example app. Are there any resources on how erlang/cowboy/linux should be configured for optimal req/s numbers? i tried ubuntu 12.04, 13.04 erlang vers R15B01, R16B01 cowboy master and 0.8 tried it on laptop (older macbook pro) and a powerful core7 blade and still, get timeout errors in httperf or errors like below in tsung when trying to pound the server with as low as 300 req/s: stats: error_connect_eaddrinuse 1320 10836 stats: error_abort_max_conn_retries 378 378 also tried playing around with settings like these while starting the server, with no apparent difference: +K true -smp enable +swt low +spp true +P 512000 -env ERL_MAX_PORTS 1000000 Thanks in advance Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From ababkin@REDACTED Sun Sep 8 23:19:52 2013 From: ababkin@REDACTED (Alex Babkin) Date: Sun, 8 Sep 2013 17:19:52 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> References: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> Message-ID: will see you guys there On Sun, Sep 8, 2013 at 3:57 PM, Robert Virding < robert.virding@REDACTED> wrote: > That sounds very reasonable, and near the subway too. :-) Any time after > 6pm suits me. > > Robert > > ------------------------------ > > *From: *"Mahesh Paolini-Subramanya" > *To: *"Siraaj Khandkar" > *Cc: *"Erlang" , "Roberto Ostinelli" < > roberto@REDACTED> > *Sent: *Friday, 6 September, 2013 9:05:12 PM > > *Subject: *Re: [erlang-questions] erlounge in nyc > > Thursday works quite well for me too. > How about at Biblio (in Williamsburg) Excellent hipster bar with great > beers ( hipsters and beer. Well duh) and excellent munchies... > > http://bibliobrooklyn.com/2013/06/05/beer-menu/ > > Sent via smoke signals... > Mahesh Paolini-Subramanya > That tall bald Indian guy... > On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" wrote: > >> Count me in! >> >> I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for >> that reason :)). >> >> >> On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: >> >>> AFA the 'when': I believe the matching dates for everyone would have to >>> be either Thursday 12th or Friday 13th. >>> >>> I'm new to NY so I'm not in a position to suggest the 'when' part. :) >>> >>> r. >>> >>> >>> On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding >>> >> > wrote: >>> >>> Are people interested in a NYC erlounge? I will be there next week >>> and am free all evenings except Wednesday. Someone suggest when and >>> where. >>> >>> Robert >>> >>> >>> ------------------------------------------------------------ >>> ------------ >>> >>> *From: *"Robert Virding" >> > >>> *To: *"Bob Ippolito" > >>> *Cc: *"Erlang" >> >, "Roberto Ostinelli" >>> > >>> *Sent: *Monday, 2 September, 2013 6:04:04 PM >>> >>> *Subject: *Re: [erlang-questions] erlounge in nyc >>> >>> I will be in NYC 10-15 sept so I will do my best to attend. >>> Erlounges are great. >>> >>> Robert >>> >>> ------------------------------------------------------------ >>> ------------ >>> >>> *From: *"Bob Ippolito" >> bob@REDACTED>> >>> *To: *"Roberto Ostinelli" >> > >>> *Cc: *"Erlang" >> > >>> *Sent: *Friday, 30 August, 2013 5:17:52 AM >>> *Subject: *Re: [erlang-questions] erlounge in nyc >>> >>> I'm in NYC from Sep 5-13 so I could probably also attend >>> >>> On Thursday, August 29, 2013, Roberto Ostinelli wrote: >>> >>> I moved to NYC. If this happens I'm in too. >>> >>> r. >>> >>> >>> >>> On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies >>> wrote: >>> >>> I'll do my best to make it too :) >>> >>> Cheers, >>> Heinz >>> On Aug 29, 2013, at 22:16, Garry Hodgson >>> wrote: >>> >>> > i would be love to get together with some other >>> > erlang folks, if this is still on the table. >>> > >>> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >>> >> Hello, >>> >> >>> >> Somebody, and I forget who asked me, and I >>> forget when, to drop them a line >>> >> if I was every in New York. >>> >> >>> >> I'll be in New York from 10 - 14 sept, if >>> anybody is interested in a meetup. >>> >> >>> >> /Joe >>> > >>> > -- >>> > Garry Hodgson >>> > AT&T Chief Security Office (CSO) >>> > >>> > "This e-mail and any files transmitted with it >>> are AT&T property, are confidential, and are >>> intended solely for the use of the individual or >>> entity to whom this e-mail is addressed. If you are >>> not one of the named recipient(s) or otherwise have >>> reason to believe that you have received this >>> message in error, please notify the sender and >>> delete this message immediately from your computer. >>> Any other use, retention, dissemination, forwarding, >>> printing, or copying of this e-mail is strictly >>> prohibited." >>> > >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Sep 8 23:25:11 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Sun, 08 Sep 2013 23:25:11 +0200 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: References: Message-ID: <522CEB37.5080507@ninenines.eu> Works fine for me. Quick testing on my laptop with 'connection: close': % siege -b -c 500 -t 5s http://localhost:8080 ** SIEGE 3.0.1 ** Preparing 500 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 24906 hits Availability: 100.00 % Elapsed time: 4.20 secs Data transferred: 3.23 MB Response time: 0.08 secs Transaction rate: 5930.00 trans/sec Throughput: 0.77 MB/sec Concurrency: 453.53 Successful transactions: 24906 Failed transactions: 0 Longest transaction: 3.09 Shortest transaction: 0.00 Did you increase the maximum number of fd on your system? Considering your errors it's what comes to mind first. This series of tutorials go through configuring this and tweaking your system some more with sysctl: http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 If sysctl doesn't do it, look at ulimit also. If that's not it, check that you haven't modified the number of acceptors to some huge number that doesn't make any sense, leave it at around 100. On 09/08/2013 11:14 PM, Alex Babkin wrote: > Hi > > Was trying to load test my cowboy based websever with tsung and httperf, > and it seems like i'm unable to squeeze out more than 200 req/s from > even the rest_hello_world example app. > > Are there any resources on how erlang/cowboy/linux should be configured > for optimal req/s numbers? > > i tried ubuntu 12.04, 13.04 > erlang vers R15B01, R16B01 > cowboy master and 0.8 > tried it on laptop (older macbook pro) and a powerful core7 blade > > and still, get timeout errors in httperf or errors like below in tsung > when trying to pound the server with as low as 300 req/s: > stats: error_connect_eaddrinuse 1320 10836 > stats: error_abort_max_conn_retries 378 378 > > also tried playing around with settings like these while starting the > server, with no apparent difference: > +K true -smp enable +swt low +spp true +P 512000 -env ERL_MAX_PORTS > 1000000 > > Thanks in advance > Alex > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From colanderman@REDACTED Mon Sep 9 00:58:04 2013 From: colanderman@REDACTED (Chris King) Date: Sun, 08 Sep 2013 18:58:04 -0400 Subject: [erlang-questions] better HTTP client? Message-ID: Hi all, Due to various architectural bugs in the inets:httpc module (which I have reported) related to queueing multiple requests to a single server, I'm looking for a replacement for my application. The use case requires queuing many GET requests to a small number of servers; taking advantage of keep-alive/pipelining when available; and allowing outstanding unserviced requests to be canceled. I don't care about streaming or writing to disk or any sort of upload capability. Any suggestions or should I roll my own? (The work required to re-architect inets:httpc compared to writing a simple server that meets exactly my needs is too much for me right now.) Thanks, Chris From bob@REDACTED Mon Sep 9 01:00:20 2013 From: bob@REDACTED (Bob Ippolito) Date: Sun, 8 Sep 2013 16:00:20 -0700 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: It's been a long time since I looked under the hood, but have you looked at lhttpc? It worked well for us when inets httpc did not. On Sunday, September 8, 2013, Chris King wrote: > Hi all, > > Due to various architectural bugs in the inets:httpc module (which I have > reported) related to queueing multiple requests to a single server, I'm > looking for a replacement for my application. > > The use case requires queuing many GET requests to a small number of > servers; taking advantage of keep-alive/pipelining when available; and > allowing outstanding unserviced requests to be canceled. I don't care > about streaming or writing to disk or any sort of upload capability. Any > suggestions or should I roll my own? > > (The work required to re-architect inets:httpc compared to writing a > simple server that meets exactly my needs is too much for me right now.) > > Thanks, > Chris > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Mon Sep 9 01:20:54 2013 From: colanderman@REDACTED (Chris King) Date: Sun, 08 Sep 2013 19:20:54 -0400 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: On Sun, 08 Sep 2013 19:00:20 -0400, Bob Ippolito wrote: > It's been a long time since I looked under the hood, but have you looked > at lhttpc? It worked well for us when inets >httpc did not. Hm, that looks good, but does not support asynchronous cancelable requests. (I do lots of speculative prefetching; I want to be able to cancel not-yet-scheduled requests if they are no longer needed.) I will keep it in mind though; thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ababkin@REDACTED Mon Sep 9 01:29:12 2013 From: ababkin@REDACTED (Alex Babkin) Date: Sun, 8 Sep 2013 19:29:12 -0400 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: <522CEB37.5080507@ninenines.eu> References: <522CEB37.5080507@ninenines.eu> Message-ID: Thanks for reply Loic it seems that in my case i simply run out of num of open connections limit, and this happens after about 5-10 sec of running at 1000 req/s. I tried running as root and this seemed to gave me higher limit of connections, so i am able to sustain 500 req/s for at least 15 min I haven't tried siege yet, but can you actually sustain this rate of ~5000 req/s for around 15 min and not just 5 sec? (just curious) Alex On Sun, Sep 8, 2013 at 5:25 PM, Lo?c Hoguin wrote: > Works fine for me. Quick testing on my laptop with 'connection: close': > > % siege -b -c 500 -t 5s http://localhost:8080 > ** SIEGE 3.0.1 > ** Preparing 500 concurrent users for battle. > The server is now under siege... > Lifting the server siege... done. > > Transactions: 24906 hits > Availability: 100.00 % > Elapsed time: 4.20 secs > Data transferred: 3.23 MB > Response time: 0.08 secs > Transaction rate: 5930.00 trans/sec > Throughput: 0.77 MB/sec > Concurrency: 453.53 > Successful transactions: 24906 > Failed transactions: 0 > Longest transaction: 3.09 > Shortest transaction: 0.00 > > Did you increase the maximum number of fd on your system? Considering your > errors it's what comes to mind first. > > This series of tutorials go through configuring this and tweaking your > system some more with sysctl: http://www.metabrew.com/** > article/a-million-user-comet-**application-with-mochiweb-**part-1 > > If sysctl doesn't do it, look at ulimit also. > > If that's not it, check that you haven't modified the number of acceptors > to some huge number that doesn't make any sense, leave it at around 100. > > > On 09/08/2013 11:14 PM, Alex Babkin wrote: > >> Hi >> >> Was trying to load test my cowboy based websever with tsung and httperf, >> and it seems like i'm unable to squeeze out more than 200 req/s from >> even the rest_hello_world example app. >> >> Are there any resources on how erlang/cowboy/linux should be configured >> for optimal req/s numbers? >> >> i tried ubuntu 12.04, 13.04 >> erlang vers R15B01, R16B01 >> cowboy master and 0.8 >> tried it on laptop (older macbook pro) and a powerful core7 blade >> >> and still, get timeout errors in httperf or errors like below in tsung >> when trying to pound the server with as low as 300 req/s: >> stats: error_connect_eaddrinuse 1320 10836 >> stats: error_abort_max_conn_retries 378 378 >> >> also tried playing around with settings like these while starting the >> server, with no apparent difference: >> +K true -smp enable +swt low +spp true +P 512000 -env ERL_MAX_PORTS >> 1000000 >> >> Thanks in advance >> Alex >> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> >> > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Mon Sep 9 01:33:05 2013 From: bob@REDACTED (Bob Ippolito) Date: Sun, 8 Sep 2013 16:33:05 -0700 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: On Sunday, September 8, 2013, Chris King wrote: > ** > On Sun, 08 Sep 2013 19:00:20 -0400, Bob Ippolito > > wrote: > > It's been a long time since I looked under the hood, but have you looked > at lhttpc? It worked well for us when inets httpc did not. > > > Hm, that looks good, but does not support asynchronous cancelable > requests. (I do lots of speculative prefetching; I want to be able to > cancel not-yet-scheduled requests if they are no longer needed.) I will > keep it in mind though; thanks. > Async cancelable and synchronous are basically isomorphic in Erlang, just start a process to wrap the sync call and kill it if you want to cancel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Mon Sep 9 01:45:08 2013 From: colanderman@REDACTED (Chris King) Date: Sun, 08 Sep 2013 19:45:08 -0400 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito wrote: > Async cancelable and synchronous are basically isomorphic in Erlang, > just start a process to wrap the sync call and kill >it if you want to > cancel. Not quite -- the HTTP request remains outstanding and blocks the request queue. I need to clear those requests out, as I need to process new requests ASAP. This should be handled by the HTTP client itself, as only it knows how many requests it has actually sent to the server. (The alternative would be for the HTTP client to expose a crediting mechanism; though this would complicate the application.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Mon Sep 9 02:34:37 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Mon, 9 Sep 2013 02:34:37 +0200 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: On Sep 9, 2013 2:33 AM, "Benoit Chesneau" wrote: > > On Sep 9, 2013 12:57 AM, "Chris King" wrote: > > > > Hi all, > > > > Due to various architectural bugs in the inets:httpc module (which I > have reported) related to queueing multiple requests to a single server, > I'm looking for a replacement for my application. > > > > The use case requires queuing many GET requests to a small number of > servers; taking advantage of keep-alive/pipelining when available; and > allowing outstanding unserviced requests to be canceled. I don't care > about streaming or writing to disk or any sort of upload capability. Any > suggestions or should I roll my own? > > > > (The work required to re-architect inets:httpc compared to writing a > simple server that meets exactly my needs is too much for me right now.) > > > > Thanks, > > Chris > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > Maybe hackney can answer to your needs: > > http://github.com/benoitc/hackney > > Pipelines requests aren't supported yet but it should be easy tonadd them. > > - benoit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Mon Sep 9 03:44:31 2013 From: bob@REDACTED (Bob Ippolito) Date: Sun, 8 Sep 2013 18:44:31 -0700 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: Are you really sure about that? lhttpc:request/9 uses spawn_link, so it should abort the request as soon as the callee dies. https://github.com/esl/lhttpc/blob/master/src/lhttpc.erl#L431 On Sun, Sep 8, 2013 at 4:45 PM, Chris King wrote: > ** > On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito wrote: > > Async cancelable and synchronous are basically isomorphic in Erlang, just > start a process to wrap the sync call and kill it if you want to cancel. > > > Not quite -- the HTTP request remains outstanding and blocks the request > queue. I need to clear those requests out, as I need to process new > requests ASAP. This should be handled by the HTTP client itself, as only > it knows how many requests it has actually sent to the server. (The > alternative would be for the HTTP client to expose a crediting mechanism; > though this would complicate the application.) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Mon Sep 9 03:54:40 2013 From: colanderman@REDACTED (Chris King) Date: Sun, 08 Sep 2013 21:54:40 -0400 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: On Sun, 08 Sep 2013 21:44:31 -0400, Bob Ippolito wrote: > Are you really sure about that? lhttpc:request/9 uses spawn_link, so it > should abort the request as soon as the callee >dies. > https://github.com/esl/lhttpc/blob/master/src/lhttpc.erl#L431 Which, depending on the architecture, would either end the HTTP session (undesirable), or would leave the HTTP session in an undefined state (also undesirable). Of course, it's possible to handle such a forced exit gracefully, but looking at lhttpc_client I see no evidence that it does. -------------- next part -------------- An HTML attachment was scrubbed... URL: From colanderman@REDACTED Mon Sep 9 04:10:41 2013 From: colanderman@REDACTED (Chris King) Date: Sun, 08 Sep 2013 22:10:41 -0400 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: On Sun, 08 Sep 2013 20:34:37 -0400, Benoit Chesneau wrote: > > Maybe hackney can answer to your needs: > > http://github.com/benoitc/hackney > > Pipelines requests aren't supported yet but it should be easy tonadd > them. Thanks; it appears hackney doesn't support request canceling a queued request however. That's necessary for my application. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Mon Sep 9 08:22:34 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Mon, 9 Sep 2013 09:22:34 +0300 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: <3DF1E5AF-C55B-40EE-A510-D3204FF71652@gmail.com> Hello Chris, I am afraid that your requirements goes beyond http client spec. You are looking for pool of http client workers accomplished with async queue in front. I do not think that you will find a client http library that fulfills it. I would integrate queue, pool and light http client by myself. It would take more time but you will be sure about your needs. Best Regards, Dmitry >-|-|-*> On 9.9.2013, at 1.58, "Chris King" wrote: > Hi all, > > Due to various architectural bugs in the inets:httpc module (which I have reported) related to queueing multiple requests to a single server, I'm looking for a replacement for my application. > > The use case requires queuing many GET requests to a small number of servers; taking advantage of keep-alive/pipelining when available; and allowing outstanding unserviced requests to be canceled. I don't care about streaming or writing to disk or any sort of upload capability. Any suggestions or should I roll my own? > > (The work required to re-architect inets:httpc compared to writing a simple server that meets exactly my needs is too much for me right now.) > > Thanks, > Chris > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Mon Sep 9 09:03:54 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Mon, 09 Sep 2013 09:03:54 +0200 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: References: <522CEB37.5080507@ninenines.eu> Message-ID: <522D72DA.4060708@ninenines.eu> Yes it can. And at a higher rate too if I close transmission. httperf gives similar results. On 09/09/2013 01:29 AM, Alex Babkin wrote: > Thanks for reply Loic > > it seems that in my case i simply run out of num of open connections > limit, and this happens after about 5-10 sec of running at 1000 req/s. I > tried running as root and this seemed to gave me higher limit of > connections, so i am able to sustain 500 req/s for at least 15 min > > I haven't tried siege yet, but can you actually sustain this rate of > ~5000 req/s for around 15 min and not just 5 sec? (just curious) > > Alex > > > > On Sun, Sep 8, 2013 at 5:25 PM, Lo?c Hoguin > wrote: > > Works fine for me. Quick testing on my laptop with 'connection: close': > > % siege -b -c 500 -t 5s http://localhost:8080 > ** SIEGE 3.0.1 > ** Preparing 500 concurrent users for battle. > The server is now under siege... > Lifting the server siege... done. > > Transactions: 24906 hits > Availability: 100.00 % > Elapsed time: 4.20 secs > Data transferred: 3.23 MB > Response time: 0.08 secs > Transaction rate: 5930.00 trans/sec > Throughput: 0.77 MB/sec > Concurrency: 453.53 > Successful transactions: 24906 > Failed transactions: 0 > Longest transaction: 3.09 > Shortest transaction: 0.00 > > Did you increase the maximum number of fd on your system? > Considering your errors it's what comes to mind first. > > This series of tutorials go through configuring this and tweaking > your system some more with sysctl: > http://www.metabrew.com/__article/a-million-user-comet-__application-with-mochiweb-__part-1 > > > If sysctl doesn't do it, look at ulimit also. > > If that's not it, check that you haven't modified the number of > acceptors to some huge number that doesn't make any sense, leave it > at around 100. > > > On 09/08/2013 11:14 PM, Alex Babkin wrote: > > Hi > > Was trying to load test my cowboy based websever with tsung and > httperf, > and it seems like i'm unable to squeeze out more than 200 req/s from > even the rest_hello_world example app. > > Are there any resources on how erlang/cowboy/linux should be > configured > for optimal req/s numbers? > > i tried ubuntu 12.04, 13.04 > erlang vers R15B01, R16B01 > cowboy master and 0.8 > tried it on laptop (older macbook pro) and a powerful core7 blade > > and still, get timeout errors in httperf or errors like below in > tsung > when trying to pound the server with as low as 300 req/s: > stats: error_connect_eaddrinuse 1320 10836 > stats: error_abort_max_conn_retries 378 378 > > also tried playing around with settings like these while > starting the > server, with no apparent difference: > +K true -smp enable +swt low +spp true +P 512000 -env > ERL_MAX_PORTS > 1000000 > > Thanks in advance > Alex > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From mononcqc@REDACTED Mon Sep 9 14:14:36 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 9 Sep 2013 08:14:36 -0400 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: <20130909121435.GB13437@ferdmbp.local> You might be interested in my fork of lhttpc: https://github.com/ferd/lhttpc It's pretty much regular lhttpc, but I added sharded connection pools to it. It has been especially optimized for doing a lot of requests to a limited set of endpoints, and was developed during my stay at AdGear for real time bidding purposes. It's also used at other real time bidding shops now. It will not do request cancelling the way you mention it. It has a central load-balancer per endpoint, where you ask for sockets. It's easy to cancel a task once you have the connection yourself, but not during the time it's in the queue to the manager. If you kill the process that made the request, the manager will find out once it tries to give away the process and keep going until it hits a request that works (https://github.com/ferd/lhttpc/blob/master/src/lhttpc_lb.erl#L98) That might be suboptimal behavior in your case. I also have a fork of that very load balancer done using dispcount called dlhttpc: https://github.com/ferd/dlhttpc Dispcount is a low-latency pool (https://github.com/ferd/dispcount) that basically will tell you right away if you can have a resource or not. There will be no such thing as a blocking request using dlhttpc because the pooling model doesn't allow for an explicit request queue. It instead uses a pool of workers that may or may not be connected (in the case of dlhttpc, each worker will only connect once asked for it, but processes will be started already), and assumes you will retry until you hit one you need. It's been designed with cases of constant overload in mind, where you may arbitrarily decide to drop a request from the call-site. Depending on how flexible your requirements are, I'd try at least one of these libraries to see how they fare. lhttpc has been in production for a long while now and has been fairly stable as far as I can tell, everywhere it's been deployed. dlhttpc has seen no real production use, but dispcount has, so it's not too likely to be a terrible thing to deploy. Regards, Fred. On 09/08, Chris King wrote: > On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito wrote: > > >Async cancelable and synchronous are basically isomorphic in > >Erlang, just start a process to wrap the sync call and kill >it if > >you want to cancel. > > Not quite -- the HTTP request remains outstanding and blocks the > request queue. I need to clear those requests out, as I need to > process new requests ASAP. This should be handled by the HTTP > client itself, as only it knows how many requests it has actually > sent to the server. (The alternative would be for the HTTP client > to expose a crediting mechanism; though this would complicate the > application.) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From emile@REDACTED Mon Sep 9 14:24:10 2013 From: emile@REDACTED (Emile Joubert) Date: Mon, 09 Sep 2013 13:24:10 +0100 Subject: [erlang-questions] 1PB of arg_reg - wait what? In-Reply-To: References: <4FA92DF3.7090000@rabbitmq.com> Message-ID: <522DBDEA.2040409@rabbitmq.com> On 08/05/12 20:31, Max Lapshin wrote: > 99% that some error in process with many terms leads to error_logger > eating all possible CPU and memory. A RabbitMQ customer claims to have reproduced this error even after disabling logging. (The thread from the rabbitmq-discuss mailing list starts here: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-September/029829.html) The error is: =erl_crash_dump:0.2 Mon Sep 9 12:01:31 2013 Slogan: std_alloc: Cannot allocate 1121150602465920 bytes of memory (of type "arg_reg"). System version: Erlang R16B01 (erts-5.10.2) [source] [64-bit] [async-threads:30] [hipe] [kernel-poll:true] Compiled: Sun Jul 7 05:35:15 2013 Taints: Atoms: 27430 Is there anything that can be done to avoid errors like this? Is there any way of determining what the original error was that triggered the crash? -Emile From ababkin@REDACTED Mon Sep 9 18:37:52 2013 From: ababkin@REDACTED (Alex Babkin) Date: Mon, 9 Sep 2013 12:37:52 -0400 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: <522D72DA.4060708@ninenines.eu> References: <522CEB37.5080507@ninenines.eu> <522D72DA.4060708@ninenines.eu> Message-ID: <77281187-46B7-46C9-867A-CBEEB554A173@gmail.com> Hi Loic, it would be great to reproduce your req/s numbers. Do you mind sharing your sysctl.config? What distro/version of linux are you using? Thanks for the link, the setup directions in the article are a bit outdated i think Sent from my iPhone On Sep 9, 2013, at 3:03 AM, Lo?c Hoguin wrote: > Yes it can. And at a higher rate too if I close transmission. httperf gives similar results. > > On 09/09/2013 01:29 AM, Alex Babkin wrote: >> Thanks for reply Loic >> >> it seems that in my case i simply run out of num of open connections >> limit, and this happens after about 5-10 sec of running at 1000 req/s. I >> tried running as root and this seemed to gave me higher limit of >> connections, so i am able to sustain 500 req/s for at least 15 min >> >> I haven't tried siege yet, but can you actually sustain this rate of >> ~5000 req/s for around 15 min and not just 5 sec? (just curious) >> >> Alex >> >> >> >> On Sun, Sep 8, 2013 at 5:25 PM, Lo?c Hoguin > > wrote: >> >> Works fine for me. Quick testing on my laptop with 'connection: close': >> >> % siege -b -c 500 -t 5s http://localhost:8080 >> ** SIEGE 3.0.1 >> ** Preparing 500 concurrent users for battle. >> The server is now under siege... >> Lifting the server siege... done. >> >> Transactions: 24906 hits >> Availability: 100.00 % >> Elapsed time: 4.20 secs >> Data transferred: 3.23 MB >> Response time: 0.08 secs >> Transaction rate: 5930.00 trans/sec >> Throughput: 0.77 MB/sec >> Concurrency: 453.53 >> Successful transactions: 24906 >> Failed transactions: 0 >> Longest transaction: 3.09 >> Shortest transaction: 0.00 >> >> Did you increase the maximum number of fd on your system? >> Considering your errors it's what comes to mind first. >> >> This series of tutorials go through configuring this and tweaking >> your system some more with sysctl: >> http://www.metabrew.com/__article/a-million-user-comet-__application-with-mochiweb-__part-1 >> >> >> If sysctl doesn't do it, look at ulimit also. >> >> If that's not it, check that you haven't modified the number of >> acceptors to some huge number that doesn't make any sense, leave it >> at around 100. >> >> >> On 09/08/2013 11:14 PM, Alex Babkin wrote: >> >> Hi >> >> Was trying to load test my cowboy based websever with tsung and >> httperf, >> and it seems like i'm unable to squeeze out more than 200 req/s from >> even the rest_hello_world example app. >> >> Are there any resources on how erlang/cowboy/linux should be >> configured >> for optimal req/s numbers? >> >> i tried ubuntu 12.04, 13.04 >> erlang vers R15B01, R16B01 >> cowboy master and 0.8 >> tried it on laptop (older macbook pro) and a powerful core7 blade >> >> and still, get timeout errors in httperf or errors like below in >> tsung >> when trying to pound the server with as low as 300 req/s: >> stats: error_connect_eaddrinuse 1320 10836 >> stats: error_abort_max_conn_retries 378 378 >> >> also tried playing around with settings like these while >> starting the >> server, with no apparent difference: >> +K true -smp enable +swt low +spp true +P 512000 -env >> ERL_MAX_PORTS >> 1000000 >> >> Thanks in advance >> Alex >> >> >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu From sverker.eriksson@REDACTED Mon Sep 9 19:12:16 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 9 Sep 2013 19:12:16 +0200 Subject: [erlang-questions] 1PB of arg_reg - wait what? In-Reply-To: <4FA92DF3.7090000@rabbitmq.com> References: <4FA92DF3.7090000@rabbitmq.com> Message-ID: <522E0170.8040406@erix.ericsson.se> On 05/08/2012 04:30 PM, Simon MacMullen wrote: > We've just had a report of a Rabbit broker dying trying to allocate > approximately 1PB of memory: > > http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-May/019896.html > > Slogan: std_alloc: Cannot allocate 1124304355803072 bytes of memory > (of type "arg_reg"). > > Which is unfortunate enough in itself - but arg_reg? My understanding > of beam internals is not great, but that looks to be something like a > stack frame. A 1PB stack frame? > > As far as I can see "arg_reg" memory is only allocated in one place, > around line 3100 of beam_emu.c: > > https://github.com/erlang/otp/blob/maint/erts/emulator/beam/beam_emu.c#L3157 > > > ...where size is set to c_p->arity * sizeof(c_p->arg_reg[0]). > > process.arg_reg is an Eterm*, so sizeof(Eterm) *cannot* be huge, so it > must be the arity which is ~10^15. We have some excessive arities in > RabbitMQ, but not quite to that extent. > > Does anyone have any idea how this could arise, or can it just be > chalked up to cosmic rays? > c_p->arity in this case is supposed to be the number of arguments to some function or fun. The allocation is made to save the function arguments before the process is scheduled out. I seems the c_p->arity has got an insanely large value for some reason. A core dump would be more helpful to track down this bug. Command line option +d can be used to skip the erl_crash_dump and get a more usable core dump. /Sverker Erlang/OTP Ericsson From essen@REDACTED Mon Sep 9 19:24:25 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Mon, 09 Sep 2013 19:24:25 +0200 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: <77281187-46B7-46C9-867A-CBEEB554A173@gmail.com> References: <522CEB37.5080507@ninenines.eu> <522D72DA.4060708@ninenines.eu> <77281187-46B7-46C9-867A-CBEEB554A173@gmail.com> Message-ID: <522E0449.4090309@ninenines.eu> That's archlinux with no change to sysctl (other than swappiness but that has no impact on that test at all) with the erlang from archlinux packages (which is a little more optimized than what you get by default when compiling from source, but not *that* much, it's closer to +25%). Try it I guess, but I'm confident you can do the same for Ubuntu. The only real difference is up to date packages and next to no distro specific patches on archlinux. On 09/09/2013 06:37 PM, Alex Babkin wrote: > Hi Loic, it would be great to reproduce your req/s numbers. > Do you mind sharing your sysctl.config? > What distro/version of linux are you using? > Thanks for the link, the setup directions in the article are a bit outdated i think > > Sent from my iPhone > > On Sep 9, 2013, at 3:03 AM, Lo?c Hoguin wrote: > >> Yes it can. And at a higher rate too if I close transmission. httperf gives similar results. >> >> On 09/09/2013 01:29 AM, Alex Babkin wrote: >>> Thanks for reply Loic >>> >>> it seems that in my case i simply run out of num of open connections >>> limit, and this happens after about 5-10 sec of running at 1000 req/s. I >>> tried running as root and this seemed to gave me higher limit of >>> connections, so i am able to sustain 500 req/s for at least 15 min >>> >>> I haven't tried siege yet, but can you actually sustain this rate of >>> ~5000 req/s for around 15 min and not just 5 sec? (just curious) >>> >>> Alex >>> >>> >>> >>> On Sun, Sep 8, 2013 at 5:25 PM, Lo?c Hoguin >> > wrote: >>> >>> Works fine for me. Quick testing on my laptop with 'connection: close': >>> >>> % siege -b -c 500 -t 5s http://localhost:8080 >>> ** SIEGE 3.0.1 >>> ** Preparing 500 concurrent users for battle. >>> The server is now under siege... >>> Lifting the server siege... done. >>> >>> Transactions: 24906 hits >>> Availability: 100.00 % >>> Elapsed time: 4.20 secs >>> Data transferred: 3.23 MB >>> Response time: 0.08 secs >>> Transaction rate: 5930.00 trans/sec >>> Throughput: 0.77 MB/sec >>> Concurrency: 453.53 >>> Successful transactions: 24906 >>> Failed transactions: 0 >>> Longest transaction: 3.09 >>> Shortest transaction: 0.00 >>> >>> Did you increase the maximum number of fd on your system? >>> Considering your errors it's what comes to mind first. >>> >>> This series of tutorials go through configuring this and tweaking >>> your system some more with sysctl: >>> http://www.metabrew.com/__article/a-million-user-comet-__application-with-mochiweb-__part-1 >>> >>> >>> If sysctl doesn't do it, look at ulimit also. >>> >>> If that's not it, check that you haven't modified the number of >>> acceptors to some huge number that doesn't make any sense, leave it >>> at around 100. >>> >>> >>> On 09/08/2013 11:14 PM, Alex Babkin wrote: >>> >>> Hi >>> >>> Was trying to load test my cowboy based websever with tsung and >>> httperf, >>> and it seems like i'm unable to squeeze out more than 200 req/s from >>> even the rest_hello_world example app. >>> >>> Are there any resources on how erlang/cowboy/linux should be >>> configured >>> for optimal req/s numbers? >>> >>> i tried ubuntu 12.04, 13.04 >>> erlang vers R15B01, R16B01 >>> cowboy master and 0.8 >>> tried it on laptop (older macbook pro) and a powerful core7 blade >>> >>> and still, get timeout errors in httperf or errors like below in >>> tsung >>> when trying to pound the server with as low as 300 req/s: >>> stats: error_connect_eaddrinuse 1320 10836 >>> stats: error_abort_max_conn_retries 378 378 >>> >>> also tried playing around with settings like these while >>> starting the >>> server, with no apparent difference: >>> +K true -smp enable +swt low +spp true +P 512000 -env >>> ERL_MAX_PORTS >>> 1000000 >>> >>> Thanks in advance >>> Alex >>> >>> >>> _________________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/__listinfo/erlang-questions >>> >>> >>> >>> >>> -- >>> Lo?c Hoguin >>> Erlang Cowboy >>> Nine Nines >>> http://ninenines.eu >> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From n.oxyde@REDACTED Mon Sep 9 19:26:15 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 9 Sep 2013 19:26:15 +0200 Subject: [erlang-questions] 1PB of arg_reg - wait what? In-Reply-To: <522E0170.8040406@erix.ericsson.se> References: <4FA92DF3.7090000@rabbitmq.com> <522E0170.8040406@erix.ericsson.se> Message-ID: <6454B4B0-F6A1-48D1-8C02-31DBF672CE84@gmail.com> Out of curiosity, why is arity an Uint and not a byte? Erlang functions can't have more than 255 arguments, right? Le 9 sept. 2013 ? 19:12, Sverker Eriksson a ?crit : > I seems the c_p->arity has got an insanely large value for some reason. From max.lapshin@REDACTED Mon Sep 9 19:32:54 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 9 Sep 2013 21:32:54 +0400 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: <522E0449.4090309@ninenines.eu> References: <522CEB37.5080507@ninenines.eu> <522D72DA.4060708@ninenines.eu> <77281187-46B7-46C9-867A-CBEEB554A173@gmail.com> <522E0449.4090309@ninenines.eu> Message-ID: Alex, maybe you have just met a common problem when testing tool cannot produce required load to benchmark server? Look at CPU and network load for server and testing tool when you run benchmarks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ababkin@REDACTED Tue Sep 10 04:01:00 2013 From: ababkin@REDACTED (Alex Babkin) Date: Mon, 9 Sep 2013 22:01:00 -0400 Subject: [erlang-questions] squeezing performance out of cowboy - how? In-Reply-To: References: <522CEB37.5080507@ninenines.eu> <522D72DA.4060708@ninenines.eu> <77281187-46B7-46C9-867A-CBEEB554A173@gmail.com> <522E0449.4090309@ninenines.eu> Message-ID: Sorry for the spam, (what a crappy email client!) On Mon, Sep 9, 2013 at 9:58 PM, Alex Babkin wrote: > Thanks all for replies! > > After trying various suggestions i've realized that it was tsung that was > bottlenecking on me, looks like it cannot support more than 800 req/s or so > on a single node (or at least this is what i see) > > Looks like my server can support comfortably (cpu load of around 30% over > all cores) around 1,300 req/s with 3 instances of tsung generating traffic > - 2 dedicated nodes and 3rd tsung running on the same node as the server > > So it looks like i either need to run more nodes to generate more traffic > or tune tsung somehow > > > Alex > > On Sep 9, 2013, at 1:32 PM, Max Lapshin wrote: > > Alex, maybe you have just met a common problem when testing tool cannot > produce required load to benchmark server? > > Look at CPU and network load for server and testing tool when you run > benchmarks. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From omer.kilic@REDACTED Tue Sep 10 18:49:07 2013 From: omer.kilic@REDACTED (Omer Kilic) Date: Tue, 10 Sep 2013 17:49:07 +0100 Subject: [erlang-questions] [ANN] erlang-mini, a minimal Erlang package for small systems Message-ID: <522F4D83.3040300@erlang-solutions.com> Hi folks, We?re pleased to announce the availability of erlang-mini, a minimal version of the Erlang packages that we distribute on our downloads page[1], designed specifically with embedded devices in mind (no GUI related dependencies etc, weighing in under 20 megabytes). Installing erlang-mini on a freshly imaged Raspberry Pi or a Parallella board is a very simple processes. Using your favourite text editor, add the following line to your /etc/apt/sources.list For Raspberry Pi: deb http://packages.erlang-solutions.com/debian wheezy contrib For Parallella: deb http://packages.erlang-solutions.com/debian oneiric contrib Import the Erlang Solutions public key: wget http://packages.erlang-solutions.com/debian/erlang_solutions.asc sudo apt-key add erlang_solutions.asc Update the package database: sudo apt-get update Install erlang-mini: sudo apt-get install erlang-mini We are focusing our efforts on Raspberry Pi and Parallella at the moment but do get in touch if you're interested in seeing erlang-mini on other platforms as well. Cheers, Omer. [1] https://www.erlang-solutions.com/downloads/download-erlang-otp From fhamilton@REDACTED Tue Sep 10 19:24:28 2013 From: fhamilton@REDACTED (Felix Hamilton) Date: Tue, 10 Sep 2013 10:24:28 -0700 Subject: [erlang-questions] [ANN] erlang-mini, a minimal Erlang package for small systems In-Reply-To: <522F4D83.3040300@erlang-solutions.com> References: <522F4D83.3040300@erlang-solutions.com> Message-ID: Omer - This looks like a great project, but it might be nice if you posted the project home page for erlang-mini (if there is one) as well in your announcement. I did the google thing, no luck, and did a search for erlang-mini on the erlang solutions page, also no luck. Just FYI. ;-) /Felix On Tue, Sep 10, 2013 at 9:49 AM, Omer Kilic wrote: > > Hi folks, > > We?re pleased to announce the availability of erlang-mini, a minimal > version of the Erlang packages that we distribute on our downloads page[1], > designed specifically with embedded devices in mind (no GUI related > dependencies etc, weighing in under 20 megabytes). > > Installing erlang-mini on a freshly imaged Raspberry Pi or a Parallella > board is a very simple processes. Using your favourite text editor, add the > following line to your /etc/apt/sources.list > > For Raspberry Pi: > deb http://packages.erlang-**solutions.com/debianwheezy contrib > > For Parallella: > deb http://packages.erlang-**solutions.com/debianoneiric contrib > > Import the Erlang Solutions public key: > wget http://packages.erlang-**solutions.com/debian/erlang_** > solutions.asc > sudo apt-key add erlang_solutions.asc > > Update the package database: > sudo apt-get update > > Install erlang-mini: > sudo apt-get install erlang-mini > > > We are focusing our efforts on Raspberry Pi and Parallella at the moment > but do get in touch if you're interested in seeing erlang-mini on other > platforms as well. > > > Cheers, > Omer. > > > [1] https://www.erlang-solutions.**com/downloads/download-erlang-**otp > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.chamard@REDACTED Tue Sep 10 19:28:54 2013 From: david.chamard@REDACTED (David Chamard) Date: Tue, 10 Sep 2013 13:28:54 -0400 Subject: [erlang-questions] Mnesia Cluster splits Message-ID: <28782DC2867DA947B19FA7519EE8FCDEAF8AD7F1C3@MDC-MAIL-CMS01.ubisoft.org> Hi Everyone, I was wondering if there is the way to detect a network split of mnesia cluster beside the log file? The goal would be to monitor the state of the replication and status of the mnesia cluster. I tried with mnesia info but it does not seem(Or I do not understand the stats properly) what is needed to see if there is a network split of if a node is behind. Thank you, David From omer.kilic@REDACTED Tue Sep 10 19:31:30 2013 From: omer.kilic@REDACTED (Omer Kilic) Date: Tue, 10 Sep 2013 18:31:30 +0100 Subject: [erlang-questions] [ANN] erlang-mini, a minimal Erlang package for small systems In-Reply-To: References: <522F4D83.3040300@erlang-solutions.com> Message-ID: <522F5772.8080107@erlang-solutions.com> Hi Felix, On 10/09/13 18:24, Felix Hamilton wrote: > This looks like a great project, but it might be nice if you posted the > project home page for erlang-mini (if there is one) as well in your > announcement. I did the google thing, no luck, and did a search for > erlang-mini on the erlang solutions page, also no luck. Just FYI. ;-) Thanks for the suggestions. Details will be up on the ESL downloads page[1] soon and a copy of this announcement went live on [2] a couple of minutes ago, Google will probably pick it up shortly. Cheers, Omer. [1] https://www.erlang-solutions.com/downloads/download-erlang-otp [2] http://www.erlang-embedded.com/2013/09/new-erlang-package-for-small-devices-erlang-mini/ > On Tue, Sep 10, 2013 at 9:49 AM, Omer Kilic > > wrote: > > > Hi folks, > > We?re pleased to announce the availability of erlang-mini, a minimal > version of the Erlang packages that we distribute on our downloads > page[1], designed specifically with embedded devices in mind (no GUI > related dependencies etc, weighing in under 20 megabytes). > > Installing erlang-mini on a freshly imaged Raspberry Pi or a > Parallella board is a very simple processes. Using your favourite > text editor, add the following line to your /etc/apt/sources.list > > For Raspberry Pi: > deb http://packages.erlang-__solutions.com/debian > wheezy contrib > > For Parallella: > deb http://packages.erlang-__solutions.com/debian > oneiric contrib > > Import the Erlang Solutions public key: > wget > http://packages.erlang-__solutions.com/debian/erlang___solutions.asc > > sudo apt-key add erlang_solutions.asc > > Update the package database: > sudo apt-get update > > Install erlang-mini: > sudo apt-get install erlang-mini > > > We are focusing our efforts on Raspberry Pi and Parallella at the > moment but do get in touch if you're interested in seeing > erlang-mini on other platforms as well. > > > Cheers, > Omer. > > > [1] > https://www.erlang-solutions.__com/downloads/download-erlang-__otp > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > From desired.mta@REDACTED Tue Sep 10 21:23:17 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Tue, 10 Sep 2013 21:23:17 +0200 Subject: [erlang-questions] Mnesia Cluster splits In-Reply-To: <28782DC2867DA947B19FA7519EE8FCDEAF8AD7F1C3@MDC-MAIL-CMS01.ubisoft.org> References: <28782DC2867DA947B19FA7519EE8FCDEAF8AD7F1C3@MDC-MAIL-CMS01.ubisoft.org> Message-ID: On Tue, Sep 10, 2013 at 7:28 PM, David Chamard wrote: > Hi Everyone, > > I was wondering if there is the way to detect a network split of mnesia cluster beside the log file? > The goal would be to monitor the state of the replication and status of the mnesia cluster. > I tried with mnesia info but it does not seem(Or I do not understand the stats properly) what is needed to see if there is a network split of if a node is behind. The answer is somewhere here (this might as well be exactly what you need): https://github.com/uwiger/unsplit/ Motiejus From mallen@REDACTED Tue Sep 10 21:34:38 2013 From: mallen@REDACTED (Mark Allen) Date: Tue, 10 Sep 2013 14:34:38 -0500 Subject: [erlang-questions] Time range event filtering Message-ID: At $DAYJOB, my team is working on a project where we need to compute a time range (like 9am to 9pm in a specified timezone) and see if erlang:now() is in that time range and take a different action if the bool is true or false. Are there any good libraries for working with time ranges like this? I wanted to ask before we re-invent the wheel. I didn't find anything especially appropriate on a cursory web search. Thanks. Mark Mark Allen Sr Software Developer Alert Logic E: mallen@REDACTED From serge@REDACTED Tue Sep 10 21:46:20 2013 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 10 Sep 2013 15:46:20 -0400 Subject: [erlang-questions] Time range event filtering In-Reply-To: References: Message-ID: It should be fairly trivial to first convert the range to the UTC, and then check if erlang:now() fits in that range. Perhaps you can reuse of the functions in this project: https://github.com/saleyn/gen_timed_server which implements a scheduling supervisor that runs a child process on given days / time of day ranges. Serge On Tue, Sep 10, 2013 at 3:34 PM, Mark Allen wrote: > At $DAYJOB, my team is working on a project where we need to compute a > time range (like 9am to 9pm in a specified timezone) and see if > erlang:now() is in that time range and take a different action if the bool > is true or false. > > Are there any good libraries for working with time ranges like this? I > wanted to ask before we re-invent the wheel. I didn't find anything > especially appropriate on a cursory web search. > > Thanks. > > Mark > > Mark Allen > Sr Software Developer > Alert Logic > > E: mallen@REDACTED > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gomoripeti@REDACTED Tue Sep 10 21:51:45 2013 From: gomoripeti@REDACTED (=?ISO-8859-1?Q?Peti_G=F6m=F6ri?=) Date: Tue, 10 Sep 2013 20:51:45 +0100 Subject: [erlang-questions] [erlang-patches] cover & export_all In-Reply-To: References: <51DD171D.2090802@erlang.org> Message-ID: Hi Yuri, apparently the fix got a bit forgotten because of my typo, I created the below pull request to avoid further confusion https://github.com/erlang/otp/pull/74 Fredrik, could you consider it for inclusion? thanks Peter On Fri, Aug 30, 2013 at 7:50 AM, Yuri Lukyanov wrote: > Hi Peti, > > Did you have a chance to push your fix to the main repo? > > On Wed, Jul 10, 2013 at 12:34 PM, Peti G?m?ri > wrote: > > Hi Fredrik, > > > > sorry, what about this one (maybe it was a wrong ?) > > > > git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all > > > > On Wed, Jul 10, 2013 at 9:11 AM, Fredrik wrote: > >> > >> On 07/10/2013 03:19 AM, Peti G?m?ri wrote: > >> > >> Hi OTP team, > >> > >> here is a patch that addresses the problem Yuri described. > >> (I haven't added any tests or docs though) > >> > >> is it true that now you also accept pull requests? > >> > >> > >> > >> git fetch git://github.com/gomoripeti/otp.git pg?-cover-export-all > >> > >> > >> > https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all > >> > >> > https://github.com/gomoripeti/otp/compare/erlang:maint...pg?-cover-export-all.patch > >> > >> > >> br > >> Peter > >> > >> On Wed, Jul 10, 2013 at 12:34 AM, Peti G?m?ri > >> wrote: > >>> > >>> Hi Yuri, > >>> > >>> You are right, while cover compiling from source works (you can use > this > >>> as a workaround): > >>> > cover:compile(cover_test, UserOptions = [export_all, debug_info]). > >>> {ok,cover_test} > >>> > cover_test:test(). > >>> ok > >>> > >>> (because UserOptions from the arguments is taken when cover recompiles > >>> the instrumented forms) > >>> when cover compiling from beam UserOptions = [] is taken. > >>> This could be fixed in cover by taking the compile options from the > beam > >>> file as you assumed how it works. > >>> > >>> Actually the documentation still says that "Only options defining > include > >>> file directories and macros are passed to compile:file/2, everything > else is > >>> ignored." apparently a patch from Tobias Schlager added export_all to > the > >>> allowed options of cover:compile for the exact same use case as you > had. But > >>> this is missing from cover:compile_beam. > >>> > >>> May be I try to come up with a patch > >>> > >>> br > >>> Peter > >>> > >>> > >>> > >>> On Tue, Jul 9, 2013 at 11:56 AM, Yuri Lukyanov > wrote: > >>>> > >>>> It seems that there is no way to cover-compile modules with > export_all. > >>>> Here is a simple example: > >>>> > >>>> cover_test.erl: > >>>> > >>>> -module(cover_test). > >>>> test() -> ok. > >>>> > >>>> > >>>> $ erl > >>>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4] > >>>> [async-threads:0] [hipe] [kernel-poll:false] > >>>> > >>>> Eshell V5.9.1 (abort with ^G) > >>>> 1> c(cover_test, [debug_info,export_all]). > >>>> {ok,cover_test} > >>>> 2> cover_test:test(). > >>>> ok > >>>> 3> cover:compile_beam(cover_test). > >>>> {ok,cover_test} > >>>> 4> cover_test:test(). > >>>> ** exception error: undefined function cover_test:test/0 > >>>> 5> > >>>> > >>>> > >>>> Could someone explain why it is like this? Is it done on purpose? > >>>> Maybe it's a bug? > >>>> The reason I want modules to be cover-compiled with +export_all is > >>>> that it is sometimes convinient to have unit tests outside of a > >>>> module. Before unit tests are run the modules get compiled with > >>>> +export_all for tests to be able to access private functions. But the > >>>> situation is that it is not possibe in this case to enable coverage > >>>> analysys. > >>>> _______________________________________________ > >>>> erlang-questions mailing list > >>>> erlang-questions@REDACTED > >>>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >>> > >> > >> > >> > >> _______________________________________________ > >> erlang-patches mailing list > >> erlang-patches@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-patches > >> > >> Hello Peti, > >> I am getting, fatal: Couldn't find remote ref pg?-cover-export-all when > I > >> am trying to fetch. > >> Yes pull requests are accepted. > >> > >> -- > >> > >> BR Fredrik Gustafsson > >> Erlang OTP Team > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gumm@REDACTED Tue Sep 10 22:22:14 2013 From: gumm@REDACTED (Jesse Gumm) Date: Tue, 10 Sep 2013 15:22:14 -0500 Subject: [erlang-questions] Time range event filtering In-Reply-To: References: Message-ID: Hi Mark, You could give qdate[1] a whirl. While it doesn't yet have date/time comparison functions built-in yet[2], implementing those comparisons is relatively trivial in your app. It can convert from several formats and take into account timezones either in strings ("2013-09-10 9pm CDT") or as a tuple ({SomeDate, Timezone}). TZ = "CDT", >From = {{2013,9,10},{9,0,0}}, To = {{2013,9,10},{21,0,0}}, FromTS = qdate:to_now({From, TZ}), ToTS = qdate:to_now({To, TZ}), Now = erlang:now(), IsInRange = Now >= FromTS andalso Now =< ToTS. -Jesse [1] https://github.com/choptastic/qdate [2] https://github.com/choptastic/qdate/issues/2 On Tue, Sep 10, 2013 at 2:34 PM, Mark Allen wrote: > At $DAYJOB, my team is working on a project where we need to compute a time range (like 9am to 9pm in a specified timezone) and see if erlang:now() is in that time range and take a different action if the bool is true or false. > > Are there any good libraries for working with time ranges like this? I wanted to ask before we re-invent the wheel. I didn't find anything especially appropriate on a cursory web search. > > Thanks. > > Mark > > Mark Allen > Sr Software Developer > Alert Logic > > E: mallen@REDACTED > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm From bob@REDACTED Tue Sep 10 22:27:01 2013 From: bob@REDACTED (Bob Ippolito) Date: Tue, 10 Sep 2013 13:27:01 -0700 Subject: [erlang-questions] Time range event filtering In-Reply-To: References: Message-ID: Sounds like you should try and find an interval tree implementation for Erlang, one probably exists somewhere by now. The fact that you're working with times is basically irrelevant to the actual problem at hand. On Tue, Sep 10, 2013 at 12:34 PM, Mark Allen wrote: > At $DAYJOB, my team is working on a project where we need to compute a > time range (like 9am to 9pm in a specified timezone) and see if > erlang:now() is in that time range and take a different action if the bool > is true or false. > > Are there any good libraries for working with time ranges like this? I > wanted to ask before we re-invent the wheel. I didn't find anything > especially appropriate on a cursory web search. > > Thanks. > > Mark > > Mark Allen > Sr Software Developer > Alert Logic > > E: mallen@REDACTED > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sedrik@REDACTED Wed Sep 11 08:35:43 2013 From: sedrik@REDACTED (Fredrik Andersson) Date: Wed, 11 Sep 2013 08:35:43 +0200 Subject: [erlang-questions] Time range event filtering In-Reply-To: References: Message-ID: Dont use now() for dates. os:timestamp() is what you should use. On Tue, Sep 10, 2013 at 10:27 PM, Bob Ippolito wrote: > Sounds like you should try and find an interval tree implementation for > Erlang, one probably exists somewhere by now. The fact that you're working > with times is basically irrelevant to the actual problem at hand. > > > On Tue, Sep 10, 2013 at 12:34 PM, Mark Allen wrote: > >> At $DAYJOB, my team is working on a project where we need to compute a >> time range (like 9am to 9pm in a specified timezone) and see if >> erlang:now() is in that time range and take a different action if the bool >> is true or false. >> >> Are there any good libraries for working with time ranges like this? I >> wanted to ask before we re-invent the wheel. I didn't find anything >> especially appropriate on a cursory web search. >> >> Thanks. >> >> Mark >> >> Mark Allen >> Sr Software Developer >> Alert Logic >> >> E: mallen@REDACTED >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spawn.think@REDACTED Wed Sep 11 13:40:34 2013 From: spawn.think@REDACTED (Ahmed Omar) Date: Wed, 11 Sep 2013 13:40:34 +0200 Subject: [erlang-questions] Mnesia Cluster splits In-Reply-To: References: <28782DC2867DA947B19FA7519EE8FCDEAF8AD7F1C3@MDC-MAIL-CMS01.ubisoft.org> Message-ID: Hi David, The other way beside log file, is to subscribe to mnesia system events. http://www.erlang.org/doc/apps/mnesia/Mnesia_chap5.html#id76675 In case of an inconsistent database (usually happens after a split), your process will receive the event (message) {inconsistent_database, Context, Node} Mnesia won't do any automatic healing, you will have to decide what to do. (some options are offered in the documentation) As pointed earlier by Motiejus, unsplit is a framework (written by Ulf Wiger) to help resolving this kind of situations. Best Regards, - Ahmed Omar http://about.me/spawn.think/ On Tue, Sep 10, 2013 at 9:23 PM, Motiejus Jak?tys wrote: > On Tue, Sep 10, 2013 at 7:28 PM, David Chamard > wrote: > > Hi Everyone, > > > > I was wondering if there is the way to detect a network split of mnesia > cluster beside the log file? > > The goal would be to monitor the state of the replication and status of > the mnesia cluster. > > I tried with mnesia info but it does not seem(Or I do not understand the > stats properly) what is needed to see if there is a network split of if a > node is behind. > > The answer is somewhere here (this might as well be exactly what you > need): https://github.com/uwiger/unsplit/ > > Motiejus > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillips@REDACTED Wed Sep 11 15:59:32 2013 From: phillips@REDACTED (Phillip Sibanda) Date: Wed, 11 Sep 2013 15:59:32 +0200 Subject: [erlang-questions] DETS - {error,{not_closed, }} Message-ID: <52307744.70207@pharos-avantgard.com> Hello Everyone, I have a Proc A which has an open Dets table and is doing a high rate of read and writes to the Dets table. I would like Proc B to have read-only access to same Dets table. An attempt to open the file with read-only access by Proc B results in {error,{not_closed, }} as follows > dets:open_file(tab, [{access, read}, {file, }]). {error,{not_closed, }} However when Proc B attempts to open the table in read-write(default) mode, the table is opened successfully. From my understanding the default option works because it will result in an attempt to repair the table before it is opened. I'm bit wary of this because I want only Process A to make any changes to the table. My questions: 1. Is there an alternative way to allow Process B read-only access to the table? - preferably without Process A closing the table 2. Do I understand correctly that the default option works because the table is repaired? Regards, Phillip -- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pharos_signature.jpg Type: image/jpeg Size: 22822 bytes Desc: not available URL: From olivier.boudeville@REDACTED Wed Sep 11 16:37:01 2013 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Wed, 11 Sep 2013 16:37:01 +0200 Subject: [erlang-questions] [OT] List of companies providing Erlang development services? Message-ID: Hi all, (I hope that this kind of messages abides by the mailing list rules) We will request in the next few weeks a development service, and I would like to ensure that our purchasing department does not miss any potential subcontractor of interest. I am not sure that such a list exists, so any company offering Erlang development services may send me an off-list e-mail with its full coordinates, so that it can be included among the candidates that will be contacted. Only companies may apply, and purchasing departments tend to frown upon smaller structures. The work would be done on Sim-Diasca and WOOPER, would amount to a few months worth of effort, and would deal mostly with parse transforms, build tool, integration and tuning; detailed requirements and other information will be sent to the short list that will be selected by our purchasing department. On a side note, wouldn't it be neat if there were some place where Erlang subcontractors could easily subscribe and be found? Thanks in advance for all answers (and sorry for the noise), Best regards, Olivier Boudeville. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihai@REDACTED Wed Sep 11 17:33:00 2013 From: mihai@REDACTED (Mihai Balea) Date: Wed, 11 Sep 2013 11:33:00 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> Message-ID: <4AC76AE5-E7D6-4DA0-8DF1-8D306081B87C@hates.ms> I'm in, probably be there around 7pm Mihai On Sep 8, 2013, at 5:19 PM, Alex Babkin wrote: > will see you guys there > > > On Sun, Sep 8, 2013 at 3:57 PM, Robert Virding wrote: > That sounds very reasonable, and near the subway too. :-) Any time after 6pm suits me. > > Robert > > From: "Mahesh Paolini-Subramanya" > To: "Siraaj Khandkar" > Cc: "Erlang" , "Roberto Ostinelli" > Sent: Friday, 6 September, 2013 9:05:12 PM > > Subject: Re: [erlang-questions] erlounge in nyc > > Thursday works quite well for me too. > How about at Biblio (in Williamsburg) Excellent hipster bar with great beers ( hipsters and beer. Well duh) and excellent munchies... > > http://bibliobrooklyn.com/2013/06/05/beer-menu/ > Sent via smoke signals... > Mahesh Paolini-Subramanya > That tall bald Indian guy... > > On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" wrote: > Count me in! > > I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for that reason :)). > > > On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: > AFA the 'when': I believe the matching dates for everyone would have to > be either Thursday 12th or Friday 13th. > > I'm new to NY so I'm not in a position to suggest the 'when' part. :) > > r. > > > On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding > > wrote: > > Are people interested in a NYC erlounge? I will be there next week > and am free all evenings except Wednesday. Someone suggest when and > where. > > Robert > > > ------------------------------------------------------------------------ > > *From: *"Robert Virding" > > *To: *"Bob Ippolito" > > *Cc: *"Erlang" >, "Roberto Ostinelli" > > > *Sent: *Monday, 2 September, 2013 6:04:04 PM > > *Subject: *Re: [erlang-questions] erlounge in nyc > > I will be in NYC 10-15 sept so I will do my best to attend. > Erlounges are great. > > Robert > > ------------------------------------------------------------------------ > > *From: *"Bob Ippolito" > > *To: *"Roberto Ostinelli" > > *Cc: *"Erlang" > > *Sent: *Friday, 30 August, 2013 5:17:52 AM > *Subject: *Re: [erlang-questions] erlounge in nyc > > I'm in NYC from Sep 5-13 so I could probably also attend > > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > > I moved to NYC. If this happens I'm in too. > > r. > > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies > wrote: > > I'll do my best to make it too :) > > Cheers, > Heinz > On Aug 29, 2013, at 22:16, Garry Hodgson > wrote: > > > i would be love to get together with some other > > erlang folks, if this is still on the table. > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > >> Hello, > >> > >> Somebody, and I forget who asked me, and I > forget when, to drop them a line > >> if I was every in New York. > >> > >> I'll be in New York from 10 - 14 sept, if > anybody is interested in a meetup. > >> > >> /Joe > > > > -- > > Garry Hodgson > > AT&T Chief Security Office (CSO) > > > > "This e-mail and any files transmitted with it > are AT&T property, are confidential, and are > intended solely for the use of the individual or > entity to whom this e-mail is addressed. If you are > not one of the named recipient(s) or otherwise have > reason to believe that you have received this > message in error, please notify the sender and > delete this message immediately from your computer. > Any other use, retention, dissemination, forwarding, > printing, or copying of this e-mail is strictly > prohibited." > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Wed Sep 11 18:35:25 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Wed, 11 Sep 2013 18:35:25 +0200 Subject: [erlang-questions] Getting rebar to compile a single module In-Reply-To: References: Message-ID: On Thu, Sep 5, 2013 at 12:16 PM, Yash Ganthe wrote: > Hi, > > My application app1 has modules a, b and c. Another application app2 > needs just one module from my application. In the apps's > rebar.config, I have added app1 as a subdir so that app1 gets > compiled. This causes all modules of app1 to get compiled. > > Is there a way to as rebar to compile only a single module? No, there isn't because 'rebar compile' operates on projects. What is the use case or motivation? From serge@REDACTED Wed Sep 11 18:58:56 2013 From: serge@REDACTED (Serge Aleynikov) Date: Wed, 11 Sep 2013 12:58:56 -0400 Subject: [erlang-questions] pid representation in external term format Message-ID: Presently the PID representation in external term format is limited to the following: http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 ./erts/emulator/beam/erl_node_container_utils.h:#define ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) ./erts/emulator/beam/erl_term.h:#define _PID_NUM_SIZE 15 ID is limited to 15 bits Serial is limited to 3 bits So in total a PID consists of 18 bits, and therefore it seems that the number of pids on any remote node cannot exceed 2^18 (262144). While it may seem like a large number, when creating a node in other languages that implement Erlang distributed transport (e.g. C/C#/Java) and create/destroy mailboxes, the local pid counter used to create unique Pids can easily go over that limit. The work-around is to cache freed local pids and resurrect them when pid counter wraps around 2^18 boundary. This brings the question of whether that limitation is still necessary in the current version of distribution. Internally Pids use a wider representation (is it 28 bits?), so aside for supporting older versions of beam (which can be worked around through flags in distributed transport) is there any valid reason not to increase the pid maximum numbering limit? BTW, as a side note, how is the same problem addressed in the beam when the pid ID counter reaches that limit? Does it make it possible that a newly assigned Pid becomes non-unique? (I.e. if some entity still maintains a reference to an old Pid that died, and later after the pid ID counter wrapped around, a new Pid was assigned the same Pid ID number of a previously dead Pid, then the entity that had the reference to the old Pid with the same ID, could send a message to it that would not be valid for the new Pid.) Serge -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry@REDACTED Wed Sep 11 21:43:52 2013 From: garry@REDACTED (Garry Hodgson) Date: Wed, 11 Sep 2013 15:43:52 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <4AC76AE5-E7D6-4DA0-8DF1-8D306081B87C@hates.ms> References: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> <4AC76AE5-E7D6-4DA0-8DF1-8D306081B87C@hates.ms> Message-ID: <5230C7F8.80007@research.att.com> dang. it looks like i have a conflict, and cannot make it after all. oh well, i guess i'll catch the next one. On 9/11/13 11:33 AM, Mihai Balea wrote: > I'm in, probably be there around 7pm > > Mihai > > On Sep 8, 2013, at 5:19 PM, Alex Babkin wrote: > >> will see you guys there >> >> >> On Sun, Sep 8, 2013 at 3:57 PM, Robert Virding >> > > wrote: >> >> That sounds very reasonable, and near the subway too. :-) Any >> time after 6pm suits me. >> >> Robert >> >> ------------------------------------------------------------------------ >> >> *From: *"Mahesh Paolini-Subramanya" >> > >> *To: *"Siraaj Khandkar" > > >> *Cc: *"Erlang" > >, "Roberto Ostinelli" >> > >> *Sent: *Friday, 6 September, 2013 9:05:12 PM >> >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> Thursday works quite well for me too. >> How about at Biblio (in Williamsburg) Excellent hipster bar >> with great beers ( hipsters and beer. Well duh) and excellent >> munchies... >> >> http://bibliobrooklyn.com/2013/06/05/beer-menu/ >> >> Sent via smoke signals... >> Mahesh Paolini-Subramanya >> That tall bald Indian guy... >> >> On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" >> > wrote: >> >> Count me in! >> >> I'm good for Thursday the 12th, but I cannot do Fri the >> 13th (no, not for that reason :)). >> >> >> On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: >> >> AFA the 'when': I believe the matching dates for >> everyone would have to >> be either Thursday 12th or Friday 13th. >> >> I'm new to NY so I'm not in a position to suggest the >> 'when' part. :) >> >> r. >> >> >> On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding >> > >> > >> wrote: >> >> Are people interested in a NYC erlounge? I will >> be there next week >> and am free all evenings except Wednesday. >> Someone suggest when and >> where. >> >> Robert >> >> >> >> ------------------------------------------------------------------------ >> >> *From: *"Robert Virding" >> > >> > >> >> *To: *"Bob Ippolito" > > >> >> *Cc: *"Erlang" > >> > >>, "Roberto >> Ostinelli" >> > >> > >> >> *Sent: *Monday, 2 September, 2013 6:04:04 PM >> >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> I will be in NYC 10-15 sept so I will do my >> best to attend. >> Erlounges are great. >> >> Robert >> >> >> ------------------------------------------------------------------------ >> >> *From: *"Bob Ippolito" > > >> >> *To: *"Roberto Ostinelli" >> >> > >> >> *Cc: *"Erlang" >> > >> > >> >> *Sent: *Friday, 30 August, 2013 5:17:52 AM >> *Subject: *Re: [erlang-questions] >> erlounge in nyc >> >> I'm in NYC from Sep 5-13 so I could >> probably also attend >> >> On Thursday, August 29, 2013, Roberto >> Ostinelli wrote: >> >> I moved to NYC. If this happens I'm >> in too. >> >> r. >> >> >> >> On Thu, Aug 29, 2013 at 10:39 PM, >> Heinz Nikolaus Gies >> > > wrote: >> >> I'll do my best to make it too :) >> >> Cheers, >> Heinz >> On Aug 29, 2013, at 22:16, Garry >> Hodgson >> > > wrote: >> >> > i would be love to get >> together with some other >> > erlang folks, if this is still >> on the table. >> > >> > On 04/10/2013 02:59 PM, Joe >> Armstrong wrote: >> >> Hello, >> >> >> >> Somebody, and I forget who >> asked me, and I >> forget when, to drop them a line >> >> if I was every in New York. >> >> >> >> I'll be in New York from 10 - >> 14 sept, if >> anybody is interested in a meetup. >> >> >> >> /Joe >> > >> > -- >> > Garry Hodgson >> > AT&T Chief Security Office (CSO) >> > >> > "This e-mail and any files >> transmitted with it >> are AT&T property, are >> confidential, and are >> intended solely for the use of >> the individual or >> entity to whom this e-mail is >> addressed. If you are >> not one of the named recipient(s) >> or otherwise have >> reason to believe that you have >> received this >> message in error, please notify >> the sender and >> delete this message immediately >> from your computer. >> Any other use, retention, >> dissemination, forwarding, >> printing, or copying of this >> e-mail is strictly >> prohibited." >> > >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dattabora20@REDACTED Wed Sep 11 13:43:54 2013 From: dattabora20@REDACTED (Dattatraya) Date: Wed, 11 Sep 2013 17:13:54 +0530 Subject: [erlang-questions] Cross compiling erlang megaco module for ARM platform Message-ID: Hi I am new to this erlang/OTP. I am interest for megaco application which is part of erlang OTP. Is it possible to cross compile erlang megaco module for ARM platform Can you please guide us where I can found the detailed step involved in this. Regards Dattatraya -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Thu Sep 12 07:59:04 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 12 Sep 2013 07:59:04 +0200 Subject: [erlang-questions] Cross compiling erlang megaco module for ARM platform In-Reply-To: References: Message-ID: <1378965545.4808.2.camel@sekic1152.release> Greetings, There is nothing in the Erlang Megaco documentation (http://www.erlang.org/doc/apps/megaco/users_guide.html) that indicates that the Megaco module is not written in Erlang. BEAM files are cross platform and should run on an ARM Erlang/OTP system. bengt On Wed, 2013-09-11 at 17:13 +0530, Dattatraya wrote: > Hi > > > I am new to this erlang/OTP. I am interest for megaco application > which is part of erlang OTP. > > > Is it possible to cross compile erlang megaco module for ARM platform > > > Can you please guide us where I can found the detailed step involved > in this. > > > Regards > Dattatraya > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From evrenweb@REDACTED Thu Sep 12 08:13:19 2013 From: evrenweb@REDACTED (Evren Bayraktar) Date: Thu, 12 Sep 2013 06:13:19 +0000 (UTC) Subject: [erlang-questions] =?utf-8?b?TGlua2VkSW4nZGUgYmHEn2xhbnTEsSBrdXJt?= =?utf-8?q?a_daveti?= Message-ID: <64866537.111500509.1378966399248.JavaMail.app@ela4-app0130.prod> LinkedIn ------------ Sizi LinkedIn'deki profesyonel a??ma eklemek istiyorum. - Evren Bayraktar Evren Bayraktar BerilTech ?irketinde Web Developer pozisyonunda Ankara, T?rkiye Evren Bayraktar adl? ki?iyi tan?d???n?z? onaylay?n: https://www.linkedin.com/e/9bf4ye-hlhkznch-2z/isd/16496631028/FoM6bRGQ/?hs=false&tok=1A9Ao5bZ5RI5U1 -- Ba?lant? kurmak i?in davet e-postalar? al?yorsunuz. Aboneli?i iptal etmek i?in t?klay?n: http://www.linkedin.com/e/9bf4ye-hlhkznch-2z/XTlnFE8abyJd_kB1UIDLL9FatMerCISLwklnKzCOoQ/goo/erlang-questions%40erlang%2Eorg/20061/I5485925991_1/?hs=false&tok=3YBlZT_H5RI5U1 (c) 2012 LinkedIn Corporation 2029 Stierlin Ct., Mountain View, CA 94043 USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillips@REDACTED Thu Sep 12 09:17:05 2013 From: phillips@REDACTED (Phillip Sibanda) Date: Thu, 12 Sep 2013 09:17:05 +0200 Subject: [erlang-questions] DETS - {error,{not_closed, }} Message-ID: <52316A71.6080209@pharos-avantgard.com> Hello Everyone, I have a Proc A which has an open Dets table and is doing a high rate of read and writes to the Dets table. I would like Proc B to have read-only access to same Dets table. An attempt to open the file with read-only access by Proc B results in {error,{not_closed, }} as follows > dets:open_file(tab, [{access, read}, {file, }]). {error,{not_closed, }} Proc A and Proc B are not running within the same ERTS. However when Proc B attempts to open the table in read-write(default) mode, the table is opened successfully. From my understanding the default option works because it will result in an attempt to repair the table before it is opened. I'm bit wary of this because I want only Process A to make any changes to the table. My questions: 1. Is there an alternative way to allow Process B read-only access to the table? - preferably without Process A closing the table 2. Do I understand correctly that the default option works because the table is repaired? Regards, Phillip -- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pharos_signature.jpg Type: image/jpeg Size: 22822 bytes Desc: not available URL: From spawn.think@REDACTED Thu Sep 12 10:15:23 2013 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 12 Sep 2013 10:15:23 +0200 Subject: [erlang-questions] DETS - {error,{not_closed, }} In-Reply-To: <52307744.70207@pharos-avantgard.com> References: <52307744.70207@pharos-avantgard.com> Message-ID: Hi Philip, 1) Yes, you can. You need to make sure the table is flushed to disk, not to require repair. So your options are a) call dets:sync on your table from process A, and then start process B from there b) maintain a low auto_save value. I would recommend you open the file, do sync, and then wait for B to start before writing to the table. 2) Yes, because repair is done then. Note: I don't think it's a good idea to post same question twice on the list (with one different condition). Maybe it's better you reply to the original one with extra cases. Best Regards, - Ahmed Omar http://about.me/spawn.think/ On Wed, Sep 11, 2013 at 3:59 PM, Phillip Sibanda < phillips@REDACTED> wrote: > Hello Everyone, > > I have a Proc A which has an open Dets table and is doing a high rate of > read and writes to the Dets table. > > I would like Proc B to have read-only access to same Dets table. An > attempt to open the file with read-only access by Proc B results in {error,{not_closed, > }} as follows > > > dets:open_file(tab, [{access, read}, {file, }]). > {error,{not_closed, }} > > However when Proc B attempts to open the table in read-write(default) > mode, the table is opened successfully. From my understanding the default > option works because it will result in an attempt to repair the table > before it is opened. I'm bit wary of this because I want only Process A to > make any changes to the table. > > My questions: > > 1. Is there an alternative way to allow Process B read-only access to the > table? > - preferably without Process A closing the table > > 2. Do I understand correctly that the default option works because the > table is repaired? > > Regards, > Phillip > > > -- > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pharos_signature.jpg Type: image/jpeg Size: 22822 bytes Desc: not available URL: From bayinamine@REDACTED Thu Sep 12 06:52:17 2013 From: bayinamine@REDACTED (Michael Scofield) Date: Thu, 12 Sep 2013 12:52:17 +0800 Subject: [erlang-questions] Why can an unresolved function be called in lager? Message-ID: Hi all: I am new to Erlang, and now using lager as my logging tools. I went through the lager.erl file, and found there's no exported function called info/1. However, I can call lager:info/1 from the shell, it works fine. How can this possible? Thx! -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Sep 12 18:33:50 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 12 Sep 2013 20:33:50 +0400 Subject: [erlang-questions] Why can an unresolved function be called in lager? In-Reply-To: References: Message-ID: because there is no lager:info/1 or lager:info/2 functions. You need to add lager parse_transform as in documentation. On Thu, Sep 12, 2013 at 8:52 AM, Michael Scofield wrote: > Hi all: > I am new to Erlang, and now using lager as > my logging tools. > I went through the lager.erl file, and found there's no exported function > called info/1. However, I can call lager:info/1 from the shell, it works > fine. How can this possible? > Thx! > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ludovic@REDACTED Thu Sep 12 23:59:22 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Thu, 12 Sep 2013 23:59:22 +0200 Subject: [erlang-questions] Desing of MVC models Message-ID: Hi, I would like to ask a few questions about models (in an MVC sense) in Erlang. I come from a web software development background (PHP/js) and try to adapt myself to Erlang philosophy but it's not always easy. So, I'm building a game server for a browser game. I choose to stick with MVC as it is what I know best. In my game, a user can manage one many ships (3~5 at max I would say, and they act more like boats except it's not on sea ...). Many ships can be engaged in one fight, and if you control the too_big_to_not_kill ship, your ship will be in trouble: the record representing your ship in the database may have a lot of updates in a small range of time. (Each fire shot causing damages for example). Sooo (sorry this is long), there are other types of objects that i want to handle in a similar way : craft production lines inside ships, player bases, things that can be updated frequently and concurrently. I decided to build a small model lib based on a gen_server that could handle all theese requests one by one to avoid race conditions. Because pulling a record from a db and decrasing health points in two processes at the same time (user http requests) then updating the records doesn't work, then i need transactions on database, i need to lock the table, i don't like it. First question : is that wrong ? There it is : with the help of exprecs & parse_trans, i define a record, say 'ship' and a ship.erl module wich uses exprecs. Then i wrote a gen_server which, from a type (like 'ship') and an ID, can Create/Read/Update/Delete from the DB and keeps in its state my record. I made use of gproc to map ID to pids. Then, this gen_server has functions to work with the record. I know this is kind of OO, and, to minimize messaging, i use a simple pattern to update my records from a controller : i wrote a 'proc' function ("procedure") in my gen_server API which handles all interactions in a sort of "transaction". ... Controller calculations ... DamgeDone = calculate_damage_done(...) Conn = nemo:get_connection(testdba), M = nemo:load(ship,,Conn), Proc = [{get,health_points}, fun(HP,Ship) -> Ship#ship{health_points = HP- DamgeDone} end, {get,health_points}], NewHealthPoints = m:proc(M,Proc) This "kind of object" has methods such as get, set, and can handle funs which are passed the previous result and have to return the updated record, or {Reply,Record} (to avoid re-calling get health_points above). Or returning {error,Reason} which cancels any previous modifications of the record in the same 'proc' call. Then i plan to write "methods" in the ship module to be able to send {method, handle_damage, Args} to my proc instead of funs. Using records, i can use mnesia in a regular way and work with other libs that use Mnesia. Records being abstracted, i can easily write an adapter for PostgreSQL. Records are fast and don't need to be known by my model code thanks to exprecs. This is not too bad. But i would like to know what you now, because i'm not experienced. Does exprecs negates the performance of records ? Should i avoid to use a process for each model, because this is slow ? Then how to easily work with race conditions ? What do you use ? I thought about a few things : - linking model process to every caller and closing where there's no more caller alive. The main thing is to have only one entry point to a record in the db, but not to have all models processes running everytime. - Handling mutiple models in a same process ... What do you think ? Many thanks for reading this long, Cheers, Ludovic (Sorry for bad english if any mistakes) From mjtruog@REDACTED Fri Sep 13 00:22:12 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 12 Sep 2013 15:22:12 -0700 Subject: [erlang-questions] pid representation in external term format In-Reply-To: References: Message-ID: <52323E94.2010302@gmail.com> On 09/11/2013 09:58 AM, Serge Aleynikov wrote: > Presently the PID representation in external term format is limited to the following: > > http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 > > ./erts/emulator/beam/erl_node_container_utils.h:#define ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) > ./erts/emulator/beam/erl_term.h:#define _PID_NUM_SIZE 15 > > ID is limited to 15 bits > Serial is limited to 3 bits > > So in total a PID consists of 18 bits, and therefore it seems that the number of pids on any remote node cannot exceed 2^18 (262144). While it may seem like a large number, when creating a node in other languages that implement Erlang distributed transport (e.g. C/C#/Java) and create/destroy mailboxes, the local pid counter used to create unique Pids can easily go over that limit. The work-around is to cache freed local pids and resurrect them when pid counter wraps around 2^18 boundary. > > This brings the question of whether that limitation is still necessary in the current version of distribution. Internally Pids use a wider representation (is it 28 bits?), so aside for supporting older versions of beam (which can be worked around through flags in distributed transport) is there any valid reason not to increase the pid maximum numbering limit? > > BTW, as a side note, how is the same problem addressed in the beam when the pid ID counter reaches that limit? Does it make it possible that a newly assigned Pid becomes non-unique? (I.e. if some entity still maintains a reference to an old Pid that died, and later after the pid ID counter wrapped around, a new Pid was assigned the same Pid ID number of a previously dead Pid, then the entity that had the reference to the old Pid with the same ID, could send a message to it that would not be valid for the new Pid.) > > Serge > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Why wouldn't the Erlang VM be changed to use 4 bits for the Serial and 4 bits for the Id, so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Fri Sep 13 01:05:18 2013 From: serge@REDACTED (Serge Aleynikov) Date: Thu, 12 Sep 2013 19:05:18 -0400 Subject: [erlang-questions] pid representation in external term format In-Reply-To: <52323E94.2010302@gmail.com> References: <52323E94.2010302@gmail.com> Message-ID: It seems to me that you were mistaken as the 4 (id) and 4 (serial) in the External Term Format specification (*) indicate the sizes in bytes rather than in bits. (*) http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 On Thu, Sep 12, 2013 at 6:22 PM, Michael Truog wrote: > On 09/11/2013 09:58 AM, Serge Aleynikov wrote: > > Presently the PID representation in external term format is limited to > the following: > > http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 > > ./erts/emulator/beam/erl_node_container_utils.h:#define > ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) > ./erts/emulator/beam/erl_term.h:#define _PID_NUM_SIZE 15 > > ID is limited to 15 bits > Serial is limited to 3 bits > > So in total a PID consists of 18 bits, and therefore it seems that the > number of pids on any remote node cannot exceed 2^18 (262144). While it > may seem like a large number, when creating a node in other languages that > implement Erlang distributed transport (e.g. C/C#/Java) and create/destroy > mailboxes, the local pid counter used to create unique Pids can easily go > over that limit. The work-around is to cache freed local pids and > resurrect them when pid counter wraps around 2^18 boundary. > > This brings the question of whether that limitation is still necessary > in the current version of distribution. Internally Pids use a wider > representation (is it 28 bits?), so aside for supporting older versions of > beam (which can be worked around through flags in distributed transport) is > there any valid reason not to increase the pid maximum numbering limit? > > BTW, as a side note, how is the same problem addressed in the beam when > the pid ID counter reaches that limit? Does it make it possible that a > newly assigned Pid becomes non-unique? (I.e. if some entity still maintains > a reference to an old Pid that died, and later after the pid ID counter > wrapped around, a new Pid was assigned the same Pid ID number of a > previously dead Pid, then the entity that had the reference to the old Pid > with the same ID, could send a message to it that would not be valid for > the new Pid.) > > Serge > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > Why wouldn't the Erlang VM be changed to use 4 bits for the Serial and 4 > bits for the Id, so that the sizes match what the External Term Format > allows? Is this change on the Erlang VM roadmap? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Fri Sep 13 01:49:47 2013 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 12 Sep 2013 19:49:47 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: <5230C7F8.80007@research.att.com> References: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> <4AC76AE5-E7D6-4DA0-8DF1-8D306081B87C@hates.ms> <5230C7F8.80007@research.att.com> Message-ID: I've been blocked too. A cheers to you all. On Sep 11, 2013, at 3:43 PM, Garry Hodgson wrote: > dang. it looks like i have a conflict, and cannot > make it after all. oh well, i guess i'll catch the next one. > > > On 9/11/13 11:33 AM, Mihai Balea wrote: >> I'm in, probably be there around 7pm >> >> Mihai >> >> On Sep 8, 2013, at 5:19 PM, Alex Babkin wrote: >> >>> will see you guys there >>> >>> >>> On Sun, Sep 8, 2013 at 3:57 PM, Robert Virding wrote: >>> That sounds very reasonable, and near the subway too. :-) Any time after 6pm suits me. >>> >>> Robert >>> >>> From: "Mahesh Paolini-Subramanya" >>> To: "Siraaj Khandkar" >>> Cc: "Erlang" , "Roberto Ostinelli" >>> Sent: Friday, 6 September, 2013 9:05:12 PM >>> >>> Subject: Re: [erlang-questions] erlounge in nyc >>> >>> Thursday works quite well for me too. >>> How about at Biblio (in Williamsburg) Excellent hipster bar with great beers ( hipsters and beer. Well duh) and excellent munchies... >>> >>> http://bibliobrooklyn.com/2013/06/05/beer-menu/ >>> Sent via smoke signals... >>> Mahesh Paolini-Subramanya >>> That tall bald Indian guy... >>> >>> On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" wrote: >>> Count me in! >>> >>> I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not for that reason :)). >>> >>> >>> On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: >>> AFA the 'when': I believe the matching dates for everyone would have to >>> be either Thursday 12th or Friday 13th. >>> >>> I'm new to NY so I'm not in a position to suggest the 'when' part. :) >>> >>> r. >>> >>> >>> On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding >>> >> > wrote: >>> >>> Are people interested in a NYC erlounge? I will be there next week >>> and am free all evenings except Wednesday. Someone suggest when and >>> where. >>> >>> Robert >>> >>> >>> ------------------------------------------------------------------------ >>> >>> *From: *"Robert Virding" >> > >>> *To: *"Bob Ippolito" > >>> *Cc: *"Erlang" >> >, "Roberto Ostinelli" >>> > >>> *Sent: *Monday, 2 September, 2013 6:04:04 PM >>> >>> *Subject: *Re: [erlang-questions] erlounge in nyc >>> >>> I will be in NYC 10-15 sept so I will do my best to attend. >>> Erlounges are great. >>> >>> Robert >>> >>> ------------------------------------------------------------------------ >>> >>> *From: *"Bob Ippolito" > >>> *To: *"Roberto Ostinelli" >> > >>> *Cc: *"Erlang" >> > >>> *Sent: *Friday, 30 August, 2013 5:17:52 AM >>> *Subject: *Re: [erlang-questions] erlounge in nyc >>> >>> I'm in NYC from Sep 5-13 so I could probably also attend >>> >>> On Thursday, August 29, 2013, Roberto Ostinelli wrote: >>> >>> I moved to NYC. If this happens I'm in too. >>> >>> r. >>> >>> >>> >>> On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies >>> wrote: >>> >>> I'll do my best to make it too :) >>> >>> Cheers, >>> Heinz >>> On Aug 29, 2013, at 22:16, Garry Hodgson >>> wrote: >>> >>> > i would be love to get together with some other >>> > erlang folks, if this is still on the table. >>> > >>> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >>> >> Hello, >>> >> >>> >> Somebody, and I forget who asked me, and I >>> forget when, to drop them a line >>> >> if I was every in New York. >>> >> >>> >> I'll be in New York from 10 - 14 sept, if >>> anybody is interested in a meetup. >>> >> >>> >> /Joe >>> > >>> > -- >>> > Garry Hodgson >>> > AT&T Chief Security Office (CSO) >>> > >>> > "This e-mail and any files transmitted with it >>> are AT&T property, are confidential, and are >>> intended solely for the use of the individual or >>> entity to whom this e-mail is addressed. If you are >>> not one of the named recipient(s) or otherwise have >>> reason to believe that you have received this >>> message in error, please notify the sender and >>> delete this message immediately from your computer. >>> Any other use, retention, dissemination, forwarding, >>> printing, or copying of this e-mail is strictly >>> prohibited." >>> > >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahesh@REDACTED Fri Sep 13 01:52:23 2013 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Thu, 12 Sep 2013 19:52:23 -0400 Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <963893582.35163.1378137844988.JavaMail.zimbra@erlang-solutions.com> <1168270964.115810.1378483219505.JavaMail.zimbra@erlang-solutions.com> <522A178C.6060300@khandkar.net> <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> <4AC76AE5-E7D6-4DA0-8DF1-8D306081B87C@hates.ms> <5230C7F8.80007@research.att.com> Message-ID: We miss you... Sent via smoke signals... Mahesh Paolini-Subramanya That tall bald Indian guy... On Sep 12, 2013 7:49 PM, "Roberto Ostinelli" wrote: > I've been blocked too. A cheers to you all. > > On Sep 11, 2013, at 3:43 PM, Garry Hodgson wrote: > > dang. it looks like i have a conflict, and cannot > make it after all. oh well, i guess i'll catch the next one. > > > On 9/11/13 11:33 AM, Mihai Balea wrote: > > I'm in, probably be there around 7pm > > Mihai > > On Sep 8, 2013, at 5:19 PM, Alex Babkin wrote: > > will see you guys there > > > On Sun, Sep 8, 2013 at 3:57 PM, Robert Virding < > robert.virding@REDACTED> wrote: > >> That sounds very reasonable, and near the subway too. :-) Any time >> after 6pm suits me. >> >> Robert >> >> ------------------------------ >> >> *From: *"Mahesh Paolini-Subramanya" >> *To: *"Siraaj Khandkar" >> *Cc: *"Erlang" , "Roberto Ostinelli" < >> roberto@REDACTED> >> *Sent: *Friday, 6 September, 2013 9:05:12 PM >> >> *Subject: *Re: [erlang-questions] erlounge in nyc >> >> Thursday works quite well for me too. >> How about at Biblio (in Williamsburg) Excellent hipster bar with great >> beers ( hipsters and beer. Well duh) and excellent munchies... >> >> http://bibliobrooklyn.com/2013/06/05/beer-menu/ >> >> Sent via smoke signals... >> Mahesh Paolini-Subramanya >> That tall bald Indian guy... >> On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" wrote: >> >>> Count me in! >>> >>> I'm good for Thursday the 12th, but I cannot do Fri the 13th (no, not >>> for that reason :)). >>> >>> >>> On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: >>> >>>> AFA the 'when': I believe the matching dates for everyone would have to >>>> be either Thursday 12th or Friday 13th. >>>> >>>> I'm new to NY so I'm not in a position to suggest the 'when' part. :) >>>> >>>> r. >>>> >>>> >>>> On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding >>>> >>> > wrote: >>>> >>>> Are people interested in a NYC erlounge? I will be there next week >>>> and am free all evenings except Wednesday. Someone suggest when and >>>> where. >>>> >>>> Robert >>>> >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> *From: *"Robert Virding" >>> > >>>> *To: *"Bob Ippolito" > >>>> *Cc: *"Erlang" >>> >, "Roberto Ostinelli" >>>> > >>>> *Sent: *Monday, 2 September, 2013 6:04:04 PM >>>> >>>> *Subject: *Re: [erlang-questions] erlounge in nyc >>>> >>>> I will be in NYC 10-15 sept so I will do my best to attend. >>>> Erlounges are great. >>>> >>>> Robert >>>> >>>> ------------------------------------------------------------ >>>> ------------ >>>> >>>> *From: *"Bob Ippolito" >>> bob@REDACTED>> >>>> *To: *"Roberto Ostinelli" >>> > >>>> *Cc: *"Erlang" >>> > >>>> *Sent: *Friday, 30 August, 2013 5:17:52 AM >>>> *Subject: *Re: [erlang-questions] erlounge in nyc >>>> >>>> I'm in NYC from Sep 5-13 so I could probably also attend >>>> >>>> On Thursday, August 29, 2013, Roberto Ostinelli wrote: >>>> >>>> I moved to NYC. If this happens I'm in too. >>>> >>>> r. >>>> >>>> >>>> >>>> On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies >>>> wrote: >>>> >>>> I'll do my best to make it too :) >>>> >>>> Cheers, >>>> Heinz >>>> On Aug 29, 2013, at 22:16, Garry Hodgson >>>> wrote: >>>> >>>> > i would be love to get together with some other >>>> > erlang folks, if this is still on the table. >>>> > >>>> > On 04/10/2013 02:59 PM, Joe Armstrong wrote: >>>> >> Hello, >>>> >> >>>> >> Somebody, and I forget who asked me, and I >>>> forget when, to drop them a line >>>> >> if I was every in New York. >>>> >> >>>> >> I'll be in New York from 10 - 14 sept, if >>>> anybody is interested in a meetup. >>>> >> >>>> >> /Joe >>>> > >>>> > -- >>>> > Garry Hodgson >>>> > AT&T Chief Security Office (CSO) >>>> > >>>> > "This e-mail and any files transmitted with it >>>> are AT&T property, are confidential, and are >>>> intended solely for the use of the individual or >>>> entity to whom this e-mail is addressed. If you are >>>> not one of the named recipient(s) or otherwise have >>>> reason to believe that you have received this >>>> message in error, please notify the sender and >>>> delete this message immediately from your computer. >>>> Any other use, retention, dissemination, forwarding, >>>> printing, or copying of this e-mail is strictly >>>> prohibited." >>>> > >>>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Sep 13 02:44:39 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 12 Sep 2013 17:44:39 -0700 Subject: [erlang-questions] pid representation in external term format In-Reply-To: References: <52323E94.2010302@gmail.com> Message-ID: <52325FF7.8070709@gmail.com> Yes, what I meant was: Why wouldn't the Erlang VM be changed to use 4 bytes (32 bits) for the Serial and 4 bytes (32 bits) for the Id (giving us 46 more bits), so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? On 09/12/2013 04:05 PM, Serge Aleynikov wrote: > It seems to me that you were mistaken as the 4 (id) and 4 (serial) in the External Term Format specification (*) indicate the sizes in bytes rather than in bits. > > (*) http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 > > > On Thu, Sep 12, 2013 at 6:22 PM, Michael Truog > wrote: > > On 09/11/2013 09:58 AM, Serge Aleynikov wrote: >> Presently the PID representation in external term format is limited to the following: >> >> http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 >> >> ./erts/emulator/beam/erl_node_container_utils.h:#define ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) >> ./erts/emulator/beam/erl_term.h:#define _PID_NUM_SIZE 15 >> >> ID is limited to 15 bits >> Serial is limited to 3 bits >> >> So in total a PID consists of 18 bits, and therefore it seems that the number of pids on any remote node cannot exceed 2^18 (262144). While it may seem like a large number, when creating a node in other languages that implement Erlang distributed transport (e.g. C/C#/Java) and create/destroy mailboxes, the local pid counter used to create unique Pids can easily go over that limit. The work-around is to cache freed local pids and resurrect them when pid counter wraps around 2^18 boundary. >> >> This brings the question of whether that limitation is still necessary in the current version of distribution. Internally Pids use a wider representation (is it 28 bits?), so aside for supporting older versions of beam (which can be worked around through flags in distributed transport) is there any valid reason not to increase the pid maximum numbering limit? >> >> BTW, as a side note, how is the same problem addressed in the beam when the pid ID counter reaches that limit? Does it make it possible that a newly assigned Pid becomes non-unique? (I.e. if some entity still maintains a reference to an old Pid that died, and later after the pid ID counter wrapped around, a new Pid was assigned the same Pid ID number of a previously dead Pid, then the entity that had the reference to the old Pid with the same ID, could send a message to it that would not be valid for the new Pid.) >> >> Serge >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > Why wouldn't the Erlang VM be changed to use 4 bits for the Serial and 4 bits for the Id, so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Sep 13 09:36:27 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 13 Sep 2013 09:36:27 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: Message-ID: <5232C07B.7000306@ninenines.eu> On 09/12/2013 11:59 PM, Ludovic Demblans wrote: > Hi, > > I would like to ask a few questions about models (in an MVC sense) in > Erlang. > > I come from a web software development background (PHP/js) and try to > adapt myself to Erlang philosophy but it's not always easy. > > So, I'm building a game server for a browser game. I choose to stick > with MVC as it is what I know best. Sorry for not answering your question, I'd just like to give you an alternative instead. You most likely don't need to store all of this in a database. Why? Because a lot of this information is only useful while the player is active. I do not know the exact rules of the game you're building, but common things that generally do not need to be stored in the DB include for example position or current health, that is, what I call "live data". This live data is the data you would reset when loading up a new map/area, when the player starts playing or any similar situations. Don't put these in the DB. Where should you put them then? You create processes that contains these live data in their state. You create functions to perform operations on these processes, having the logic not in the "controller" but in these processes, the "controller" only needing to tell the process that something happened and what. This can be done using gen_server or gen_fsm (the latter is harder to learn but probably more adequate for many things). The process will also save to the DB the data that needs to be stored permanently, but it doesn't have to store it every time something happens, it can do it at regular intervals and on process termination. These processes are essentially what you call a model in MVC. You can simply have one process per "object" in the game. The cool thing however is that by doing the above your model is completely separate from your web layer, which means it's easy to plug a different layer (for mobile gaming for example) and more importantly that it's easy to test. It's also generally worth separating your game into its own OTP app separate from the web layer because it's now entirely independent, and there's no point in shutting everything down immediately if the web layer goes down and back up a second later. I often recommend people to write their game logic first without any graphics or database component, as it makes it easier to forget the bad habbits they had from other languages, and then, when the game works in tests (first manual, and then common_test automated testing), to add the DB (only for what you really need to store) and the graphics layer (in a separate OTP app). I hope that despite me not answering your question I could be useful to you. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From watson.timothy@REDACTED Fri Sep 13 11:46:29 2013 From: watson.timothy@REDACTED (QDev) Date: Fri, 13 Sep 2013 10:46:29 +0100 Subject: [erlang-questions] Question about basic use of diameter Message-ID: Hi - I'm a bit stuck using the diameter application. Do the Origin-Host and Origin-Realm have to reflect the real host and domain name(s) of the machine(s) in question? After attempts to write and test a basic relay kept failing, I tried using the server.erl and client.erl from the diameter application's ./examples/code directory and things don't seem to be working properly: Erlang R16B01 (erts-5.10.2) [source] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> diameter:start(). ok 2> server:start(). ok 3> client:start(). ok 4> client:connect(). ** exception error: undefined function client:connect/0 5> client:connect(tcp). {ok,#Ref<0.0.0.68>} 6> client:call(). {error,no_connection} I get the same problem if I run the client in a separate shell, but the diameter service *is* up an running: $ netstat -a | grep diameter tcp4 0 0 localhost.diameter *.* LISTEN Whilst testing my own code, instead of {error, no_connection} I get {error,no_service}} back from `diameter:call(?NAME, ?APP_ALIAS, Req, [])' - can anyone suggest what I might be missing here? The tutorial at http://vas.io/blog/2012/05/13/erlang-real-time-server-part-1-diameter-server/ doesn't work for me either - perhaps there is some environment configuration I'm missing? Q From jesper.louis.andersen@REDACTED Fri Sep 13 11:59:19 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 13 Sep 2013 11:59:19 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: <5232C07B.7000306@ninenines.eu> References: <5232C07B.7000306@ninenines.eu> Message-ID: The MVC pattern is prominent in other languages due to a few facts: * Shared Nothing - The only way to persist data over multiple independent connections is to push it down to a DB layer so you can communicate data from A to B. * Safety - If you want your data to survive a crash it *must* be put on stable storage all the time It is quite shocking how much of our typical infrastructure relies on massive databases and big caches in order to be able to run individual processes on a UNIX system. In Erlang, you have options. Safety is had due to isolation, so you don't have to push data onto stable storage as often. And you can quickly communicate in the same memory space between processes, so you don't need a mediating DB layer. Many game servers written in Erlang just keeps most of the game state in a process concurrent to the incoming HTTP requests. And then they use something like gproc to find the game and update its internal state logic. As Loic suggests, writing the game server should be possible without DB access, nor with HTTP access. On Fri, Sep 13, 2013 at 9:36 AM, Lo?c Hoguin wrote: > On 09/12/2013 11:59 PM, Ludovic Demblans wrote: > >> Hi, >> >> I would like to ask a few questions about models (in an MVC sense) in >> Erlang. >> >> I come from a web software development background (PHP/js) and try to >> adapt myself to Erlang philosophy but it's not always easy. >> >> So, I'm building a game server for a browser game. I choose to stick >> with MVC as it is what I know best. >> > > Sorry for not answering your question, I'd just like to give you an > alternative instead. > > You most likely don't need to store all of this in a database. > > Why? Because a lot of this information is only useful while the player is > active. I do not know the exact rules of the game you're building, but > common things that generally do not need to be stored in the DB include for > example position or current health, that is, what I call "live data". This > live data is the data you would reset when loading up a new map/area, when > the player starts playing or any similar situations. Don't put these in the > DB. > > Where should you put them then? You create processes that contains these > live data in their state. You create functions to perform operations on > these processes, having the logic not in the "controller" but in these > processes, the "controller" only needing to tell the process that something > happened and what. This can be done using gen_server or gen_fsm (the latter > is harder to learn but probably more adequate for many things). > > The process will also save to the DB the data that needs to be stored > permanently, but it doesn't have to store it every time something happens, > it can do it at regular intervals and on process termination. > > These processes are essentially what you call a model in MVC. You can > simply have one process per "object" in the game. The cool thing however is > that by doing the above your model is completely separate from your web > layer, which means it's easy to plug a different layer (for mobile gaming > for example) and more importantly that it's easy to test. > > It's also generally worth separating your game into its own OTP app > separate from the web layer because it's now entirely independent, and > there's no point in shutting everything down immediately if the web layer > goes down and back up a second later. > > I often recommend people to write their game logic first without any > graphics or database component, as it makes it easier to forget the bad > habbits they had from other languages, and then, when the game works in > tests (first manual, and then common_test automated testing), to add the DB > (only for what you really need to store) and the graphics layer (in a > separate OTP app). > > I hope that despite me not answering your question I could be useful to > you. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ferenc.holzhauser@REDACTED Fri Sep 13 12:41:17 2013 From: ferenc.holzhauser@REDACTED (Ferenc Holzhauser) Date: Fri, 13 Sep 2013 12:41:17 +0200 Subject: [erlang-questions] Question about basic use of diameter In-Reply-To: References: Message-ID: Hi Tim, For {error, no_service} of your own code: what does diameter:services() say? If diameter:start_service(...) worked then you should have something in that list that is called ?NAME. If it is not the case then check if you called start_service with the correct data. If you certainly did then I'm guessing rb is going to help out with some errors to debug. Generally diameter:service_info(yourservicenamehere, all) gives good information for troubleshooting when your service is running. BR, Ferenc On 13 September 2013 11:46, QDev wrote: > Hi - I'm a bit stuck using the diameter application. Do the Origin-Host > and Origin-Realm have to reflect the real host and domain name(s) of the > machine(s) in question? After attempts to write and test a basic relay kept > failing, I tried using the server.erl and client.erl from the diameter > application's ./examples/code directory and things don't seem to be working > properly: > > Erlang R16B01 (erts-5.10.2) [source] [smp:4:4] [async-threads:10] [hipe] > [kernel-poll:false] > > Eshell V5.10.2 (abort with ^G) > 1> diameter:start(). > ok > 2> server:start(). > ok > 3> client:start(). > ok > 4> client:connect(). > ** exception error: undefined function client:connect/0 > 5> client:connect(tcp). > {ok,#Ref<0.0.0.68>} > 6> client:call(). > {error,no_connection} > > I get the same problem if I run the client in a separate shell, but the > diameter service *is* up an running: > > $ netstat -a | grep diameter > tcp4 0 0 localhost.diameter *.* LISTEN > > Whilst testing my own code, instead of {error, no_connection} I get > {error,no_service}} back from `diameter:call(?NAME, ?APP_ALIAS, Req, [])' - > can anyone suggest what I might be missing here? The tutorial at > http://vas.io/blog/2012/05/13/erlang-real-time-server-part-1-diameter-server/doesn't work for me either - perhaps there is some environment > configuration I'm missing? > > Q > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Fri Sep 13 13:01:55 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 13 Sep 2013 14:01:55 +0300 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: <5232C07B.7000306@ninenines.eu> Message-ID: <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Hello, On Sep 13, 2013, at 12:59 PM, Jesper Louis Andersen wrote: > In Erlang, you have options. Safety is had due to isolation, so you don't have to push data onto stable storage as often. And you can quickly communicate in the same memory space between processes, so you don't need a mediating DB layer. This is precisely true? However, if we are talking about MMO gaming I see one potential issue which requires some clarification. Often, the data availability is important. Player would not forgive you if they "almost" win the battle but they status rolled back due to server crash or other failure that impact your "live" data. What would be possibilities to achive desired HA for transient data in Erlang and recover node failures. HA is required for millions of processes (e.g. process per user, process per ship, etc). * Keep copies of same data on two distinct nodes. Basically you spawn two "ship" management processes on node A and B. However, you as app developer is responsible to implement state conflict resolution, rollback, recovery, etc. * Use Mnesia, Riak as HA transient storage but this does not differ for my taste as any other mediating DB layer at some respect. * Use external DB layer but make asynchronous checkpoint of process status Did I miss something? Best Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.otp@REDACTED Fri Sep 13 13:02:47 2013 From: anders.otp@REDACTED (Anders Svensson) Date: Fri, 13 Sep 2013 13:02:47 +0200 Subject: [erlang-questions] Question about basic use of diameter In-Reply-To: References: Message-ID: On Fri, Sep 13, 2013 at 11:46 AM, QDev wrote: > > Hi - I'm a bit stuck using the diameter application. Do the Origin-Host and Origin-Realm have to reflect the real host and domain name(s) of the machine(s) in question? The diameter application doesn't care. All that RFC 6733 requires of Origin-Host is that it be unique within a single host (6.3), while Origin-Realm is expected to be related to the DNS domain name (6.4, 1.2; could be clearer). Using the physical host/domain is fairly typical it seems. > After attempts to write and test a basic relay kept failing, I tried using the server.erl and client.erl from the diameter application's ./examples/code directory and things don't seem to be working properly: > > Erlang R16B01 (erts-5.10.2) [source] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] > > Eshell V5.10.2 (abort with ^G) > 1> diameter:start(). > ok > 2> server:start(). > ok > 3> client:start(). > ok > 4> client:connect(). > ** exception error: undefined function client:connect/0 > 5> client:connect(tcp). > {ok,#Ref<0.0.0.68>} > 6> client:call(). > {error,no_connection} You haven't told the example server to start listening: you need to call server:listen(tcp) as well. Here's how it should look: Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10.2 (abort with ^G) 1> diameter:start(). ok 2> server:start(). ok 3> server:listen(tcp). {ok,#Ref<0.0.0.55>} 4> client:start(). ok 5> client:connect(tcp). {ok,#Ref<0.0.0.101>} 6> up: <0.56.0> client:call(). {ok,{diameter_base_RAA,"client;1440573120;1;nonode@REDACTED", 2001,"server.example.com","example.com",[],[],[],[],[], [],[],[],[],[]}} > I get the same problem if I run the client in a separate shell, but the diameter service *is* up an running: > > $ netstat -a | grep diameter > tcp4 0 0 localhost.diameter *.* LISTEN > > Whilst testing my own code, instead of {error, no_connection} I get {error,no_service}} back from `diameter:call(?NAME, ?APP_ALIAS, Req, [])' - can anyone suggest what I might be missing here? no_connection means that there's no appropriate transport connection, as determined by diameter and/or your pick_peer callback. no_service mean that you haven't configured a service (ie. diameter:start_service/2) with the specified name. > The tutorial at http://vas.io/blog/2012/05/13/erlang-real-time-server-part-1-diameter-server/ doesn't work for me either - perhaps there is some environment configuration I'm missing? Haven't seen this one but give the above a try for starters. /Anders, Erlang/OTP > > Q > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Fri Sep 13 13:37:20 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 13 Sep 2013 13:37:20 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: On Fri, Sep 13, 2013 at 1:01 PM, Dmitry Kolesnikov wrote: > * Use external DB layer but make asynchronous checkpoint of process status One way is to sync checkpoint at "big moments" in game. Say you've just slain Giant Behemoth Kraken Lurker In The Sky (GBKLITS). At this point, it makes sense to write your state back. After all, GBKLITS is hard to kill. In turn, you need a database with a very high write speed, since you will mostly be checkpointing data back to disk and very rarely read in state. In principle, you could just scan a disk_log file at startup, but this is unfeasible for searching. Good stores for this kind of data are either column stores (Kdb, Vertica, Cassandra), "SSTable"-variants (LevelDB), or certain "NewSQL" stores (VoltDB comes to mind). Given enough cluster size and enough sharding the write speed of a VoltDB cluster approached a million writes a second, albeit for rather simple stores/updates. But this means you can keep up, even for very big MMO-worlds with this solution. Mnesia could be used to hold certain in-memory state which has to be shared amongst users. But as a write-heavy store, I would avoid it[0] [0] Incidentally, I am currently looking at a mnesia db taking some 7000 writes per second, but I AM NOT RECOMMENDING YOU DO THAT :P -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Sep 13 15:43:12 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 13 Sep 2013 14:43:12 +0100 Subject: [erlang-questions] Question about basic use of diameter In-Reply-To: References: Message-ID: <92066D13-27E8-4552-8139-856CA71E8F43@gmail.com> Hi Ferenc, Thanks for the quick reply. On 13 Sep 2013, at 11:41, Ferenc Holzhauser wrote: > For {error, no_service} of your own code: what does diameter:services() say? > If diameter:start_service(...) worked then you should have something in that list that is called ?NAME. > If it is not the case then check if you called start_service with the correct data. > If you certainly did then I'm guessing rb is going to help out with some errors to debug. > > Generally diameter:service_info(yourservicenamehere, all) gives good information for troubleshooting when your service is running. > Except that I can't tell what I'm supposed to be looking for. I'm guessing some AVP isn't set right, but I'm not sure which. Both the client and the server nodes seem to contain the relevant [client] and [XX_service] entries when the tests execute, but I still get no_service when attempting to call. The test suite is executing this: Sid = diameter:session_id(atom_to_list(client)), ACR = #diameter_base_ACR{ 'Session-Id' = Sid, 'Accounting-Record-Type' = ?'DIAMETER_BASE_ACCOUNTING-RECORD-TYPE_EVENT_RECORD', 'Accounting-Record-Number' = 1 }, {ok, #diameter_base_ACA{}} = diameter:call(demo_diameter_test_client, client, ACR, []), The client has the following callback enabled: prepare_request(#diameter_packet{ msg = Req }, _, {_, #diameter_caps{ origin_host = {OH, DH}, origin_realm = {OR, DR} }}) -> {send, Req#diameter_base_ACR{ 'Origin-Host' = OH, 'Origin-Realm' = OR, 'Destination-Realm' = DR }}. If I start the server and client manually, instead of the no_service error, I get {error, no_connection}: (demo@REDACTED)1> (demo@REDACTED)1> diameter:service_info(demo_diameter_service, all). [{capabilities,[{'Origin-Host',"service.example.com"}, {'Origin-Realm',"example.com"}, {'Vendor-Id',193}, {'Product-Name',"Server"}, {'Origin-State-Id',[]}, {'Host-IP-Address',[]}, {'Supported-Vendor-Id',[]}, {'Auth-Application-Id',[0]}, {'Inband-Security-Id',[]}, {'Acct-Application-Id',[]}, {'Vendor-Specific-Application-Id',[]}, {'Firmware-Revision',[]}]}, {applications,[[{alias,demo_diameter_service}, {dictionary,diameter_gen_base_rfc3588}, {module,[diameter_callback, {diameter_callback,undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined,undefined,[]}]}, {init_state,demo_diameter_service}, {id,0}, {mutable,false}, {options,[{answer_errors,report}, {request_errors,answer_3xxx}]}]]}, {transport,[[{ref,#Ref<0.0.0.769>}, {type,listen}, {options,[{transport_module,diameter_tcp}, {transport_config,[{reuseaddr,true}, {ip,{127,0,0,1}}, {port,3868}]}]}, {accept,[[{watchdog,{<0.245.0>, {1379,78445,351797}, initial}}]]}, {statistics,[]}]]}, {pending,[]}, {options,[{sequence,{0,32}}, {share_peers,false}, {use_shared_peers,false}, {restrict_connections,nodes}]}] (demo@REDACTED)2> code:add_patha("/Users/t4/work/spikes/diameter-demo/test/ebin"). true (demo@REDACTED)3> demo_diameter_test_client:start(). ok (demo@REDACTED)4> diameter:services(). [demo_diameter_service,client] (demo@REDACTED)5> diameter:service_info(client, all). [{capabilities,[{'Origin-Host',"client.example.com"}, {'Origin-Realm',"example.com"}, {'Vendor-Id',0}, {'Product-Name',"Client"}, {'Origin-State-Id',[]}, {'Host-IP-Address',[]}, {'Supported-Vendor-Id',[]}, {'Auth-Application-Id',[0]}, {'Inband-Security-Id',[]}, {'Acct-Application-Id',[]}, {'Vendor-Specific-Application-Id',[]}, {'Firmware-Revision',[]}]}, {applications,[[{alias,client}, {dictionary,diameter_gen_base_rfc3588}, {module,[demo_diameter_test_client]}, {init_state,client}, {id,0}, {mutable,false}, {options,[{answer_errors,report}, {request_errors,answer_3xxx}]}]]}, {transport,[]}, {pending,[]}, {options,[{sequence,{0,32}}, {share_peers,false}, {use_shared_peers,false}, {restrict_connections,nodes}]}] (demo@REDACTED)6 demo_diameter_test_client:connect(). {ok,#Ref<0.0.0.1340>} (demo@REDACTED)7> diameter:service_info(client, all). [{capabilities,[{'Origin-Host',"client.example.com"}, {'Origin-Realm',"example.com"}, {'Vendor-Id',0}, {'Product-Name',"Client"}, {'Origin-State-Id',[]}, {'Host-IP-Address',[]}, {'Supported-Vendor-Id',[]}, {'Auth-Application-Id',[0]}, {'Inband-Security-Id',[]}, {'Acct-Application-Id',[]}, {'Vendor-Specific-Application-Id',[]}, {'Firmware-Revision',[]}]}, {applications,[[{alias,client}, {dictionary,diameter_gen_base_rfc3588}, {module,[demo_diameter_test_client]}, {init_state,client}, {id,0}, {mutable,false}, {options,[{answer_errors,report}, {request_errors,answer_3xxx}]}]]}, {transport,[[{ref,#Ref<0.0.0.1340>}, {type,connect}, {options,[{reconnect_timer,5000}, {transport_module,diameter_tcp}, {transport_config,[{raddr,{127,0,0,1}}, {rport,3868}, {reuseaddr,true}, {ip,{127,0,0,1}}]}]}, {watchdog,{<0.303.0>,{1379,78639,682458},okay}}, {peer,{<0.304.0>,{1379,78639,686181}}}, {apps,[{0,client}]}, {caps,[{origin_host,{"client.example.com", "service.example.com"}}, {origin_realm,{"example.com","example.com"}}, {host_ip_address,{[{127,0,0,1}],[{127,0,0,1}]}}, {vendor_id,{0,193}}, {product_name,{"Client","Server"}}, {origin_state_id,{[],[]}}, {supported_vendor_id,{[],[]}}, {auth_application_id,{[0],[0]}}, {inband_security_id,{[],[]}}, {acct_application_id,{[],[]}}, {vendor_specific_application_id,{[],...}}, {firmware_revision,{...}}, {avp,...}]}, {port,[{owner,<0.306.0>}, {module,diameter_tcp}, {socket,{{127,0,0,1},53123}}, {peer,{{127,0,0,1},3868}}, {statistics,[{recv_oct,136}, {recv_cnt,1}, {recv_max,136}, {recv_avg,136}, {recv_dvi,...}, {...}|...]}]}, {statistics,[{{{0,257,0},recv},1},{{{0,257,1},send},1}]}]]}, {pending,[]}, {options,[{sequence,{0,32}}, {share_peers,false}, {use_shared_peers,false}, {restrict_connections,nodes}]}] (demo@REDACTED> diameter:service_info(demo_diameter_service, all). [{capabilities,[{'Origin-Host',"service.example.com"}, {'Origin-Realm',"example.com"}, {'Vendor-Id',193}, {'Product-Name',"Server"}, {'Origin-State-Id',[]}, {'Host-IP-Address',[]}, {'Supported-Vendor-Id',[]}, {'Auth-Application-Id',[0]}, {'Inband-Security-Id',[]}, {'Acct-Application-Id',[]}, {'Vendor-Specific-Application-Id',[]}, {'Firmware-Revision',[]}]}, {applications,[[{alias,demo_diameter_service}, {dictionary,diameter_gen_base_rfc3588}, {module,[diameter_callback, {diameter_callback,undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined,undefined,[]}]}, {init_state,demo_diameter_service}, {id,0}, {mutable,false}, {options,[{answer_errors,report}, {request_errors,answer_3xxx}]}]]}, {transport,[[{ref,#Ref<0.0.0.769>}, {type,listen}, {options,[{transport_module,diameter_tcp}, {transport_config,[{reuseaddr,true}, {ip,{127,0,0,1}}, {port,3868}]}]}, {accept,[[{watchdog,{<0.308.0>, {1379,78639,684566}, initial}}], [{watchdog,{<0.245.0>,{1379,78445,351797},okay}}, {peer,{<0.246.0>,{1379,78639,685678}}}, {apps,[{0,demo_diameter_service}]}, {caps,[{origin_host,{"service.example.com", "client.example.com"}}, {origin_realm,{"example.com","example.com"}}, {host_ip_address,{[{127,0,...}],[{127,...}]}}, {vendor_id,{193,0}}, {product_name,{"Server",[...]}}, {origin_state_id,{[],...}}, {supported_vendor_id,{...}}, {auth_application_id,...}, {...}|...]}, {port,[{owner,<0.255.0>}, {module,diameter_tcp}, {socket,{{127,0,...},3868}}, {peer,{{127,...},53123}}, {statistics,[{...}|...]}]}]]}, {statistics,[{{{0,257,0},send},1},{{{0,257,1},recv},1}]}]]}, {pending,[]}, {options,[{sequence,{0,32}}, {share_peers,false}, {use_shared_peers,false}, {restrict_connections,nodes}]}] (demo@REDACTED)9> demo_diameter_test_client:call(). {error,no_connection} From ferenc.holzhauser@REDACTED Fri Sep 13 16:42:28 2013 From: ferenc.holzhauser@REDACTED (Ferenc Holzhauser) Date: Fri, 13 Sep 2013 16:42:28 +0200 Subject: [erlang-questions] Question about basic use of diameter In-Reply-To: <92066D13-27E8-4552-8139-856CA71E8F43@gmail.com> References: <92066D13-27E8-4552-8139-856CA71E8F43@gmail.com> Message-ID: Tim, I didn't spot anything crazy by quickly scanning through these. How does your pick_peer callback look like? Ferenc On 13 September 2013 15:43, Tim Watson wrote: > Hi Ferenc, > > Thanks for the quick reply. > > On 13 Sep 2013, at 11:41, Ferenc Holzhauser wrote: > > For {error, no_service} of your own code: what does diameter:services() > say? > > If diameter:start_service(...) worked then you should have something in > that list that is called ?NAME. > > If it is not the case then check if you called start_service with the > correct data. > > If you certainly did then I'm guessing rb is going to help out with some > errors to debug. > > > > Generally diameter:service_info(yourservicenamehere, all) gives good > information for troubleshooting when your service is running. > > > > Except that I can't tell what I'm supposed to be looking for. I'm guessing > some AVP isn't set right, but I'm not sure which. Both the client and the > server nodes seem to contain the relevant [client] and [XX_service] entries > when the tests execute, but I still get no_service when attempting to call. > The test suite is executing this: > > Sid = diameter:session_id(atom_to_list(client)), > ACR = #diameter_base_ACR{ 'Session-Id' = Sid, > 'Accounting-Record-Type' = > ?'DIAMETER_BASE_ACCOUNTING-RECORD-TYPE_EVENT_RECORD', > 'Accounting-Record-Number' = 1 }, > {ok, #diameter_base_ACA{}} = diameter:call(demo_diameter_test_client, > client, ACR, []), > > The client has the following callback enabled: > > prepare_request(#diameter_packet{ msg = Req }, _, > {_, #diameter_caps{ origin_host = {OH, DH}, > origin_realm = {OR, DR} }}) -> > {send, Req#diameter_base_ACR{ 'Origin-Host' = OH, > 'Origin-Realm' = OR, > 'Destination-Realm' = DR }}. > > If I start the server and client manually, instead of the no_service > error, I get {error, no_connection}: > > (demo@REDACTED)1> > (demo@REDACTED)1> diameter:service_info(demo_diameter_service, all). > [{capabilities,[{'Origin-Host',"service.example.com"}, > {'Origin-Realm',"example.com"}, > {'Vendor-Id',193}, > {'Product-Name',"Server"}, > {'Origin-State-Id',[]}, > {'Host-IP-Address',[]}, > {'Supported-Vendor-Id',[]}, > {'Auth-Application-Id',[0]}, > {'Inband-Security-Id',[]}, > {'Acct-Application-Id',[]}, > {'Vendor-Specific-Application-Id',[]}, > {'Firmware-Revision',[]}]}, > {applications,[[{alias,demo_diameter_service}, > {dictionary,diameter_gen_base_rfc3588}, > {module,[diameter_callback, > > {diameter_callback,undefined,undefined,undefined,undefined, > > undefined,undefined,undefined,undefined,undefined,[]}]}, > {init_state,demo_diameter_service}, > {id,0}, > {mutable,false}, > {options,[{answer_errors,report}, > {request_errors,answer_3xxx}]}]]}, > {transport,[[{ref,#Ref<0.0.0.769>}, > {type,listen}, > {options,[{transport_module,diameter_tcp}, > {transport_config,[{reuseaddr,true}, > {ip,{127,0,0,1}}, > {port,3868}]}]}, > {accept,[[{watchdog,{<0.245.0>, > {1379,78445,351797}, > initial}}]]}, > {statistics,[]}]]}, > {pending,[]}, > {options,[{sequence,{0,32}}, > {share_peers,false}, > {use_shared_peers,false}, > {restrict_connections,nodes}]}] > (demo@REDACTED)2> > code:add_patha("/Users/t4/work/spikes/diameter-demo/test/ebin"). > true > (demo@REDACTED)3> demo_diameter_test_client:start(). > ok > (demo@REDACTED)4> diameter:services(). > [demo_diameter_service,client] > (demo@REDACTED)5> diameter:service_info(client, all). > [{capabilities,[{'Origin-Host',"client.example.com"}, > {'Origin-Realm',"example.com"}, > {'Vendor-Id',0}, > {'Product-Name',"Client"}, > {'Origin-State-Id',[]}, > {'Host-IP-Address',[]}, > {'Supported-Vendor-Id',[]}, > {'Auth-Application-Id',[0]}, > {'Inband-Security-Id',[]}, > {'Acct-Application-Id',[]}, > {'Vendor-Specific-Application-Id',[]}, > {'Firmware-Revision',[]}]}, > {applications,[[{alias,client}, > {dictionary,diameter_gen_base_rfc3588}, > {module,[demo_diameter_test_client]}, > {init_state,client}, > {id,0}, > {mutable,false}, > {options,[{answer_errors,report}, > {request_errors,answer_3xxx}]}]]}, > {transport,[]}, > {pending,[]}, > {options,[{sequence,{0,32}}, > {share_peers,false}, > {use_shared_peers,false}, > {restrict_connections,nodes}]}] > (demo@REDACTED)6 demo_diameter_test_client:connect(). > {ok,#Ref<0.0.0.1340>} > (demo@REDACTED)7> diameter:service_info(client, all). > [{capabilities,[{'Origin-Host',"client.example.com"}, > {'Origin-Realm',"example.com"}, > {'Vendor-Id',0}, > {'Product-Name',"Client"}, > {'Origin-State-Id',[]}, > {'Host-IP-Address',[]}, > {'Supported-Vendor-Id',[]}, > {'Auth-Application-Id',[0]}, > {'Inband-Security-Id',[]}, > {'Acct-Application-Id',[]}, > {'Vendor-Specific-Application-Id',[]}, > {'Firmware-Revision',[]}]}, > {applications,[[{alias,client}, > {dictionary,diameter_gen_base_rfc3588}, > {module,[demo_diameter_test_client]}, > {init_state,client}, > {id,0}, > {mutable,false}, > {options,[{answer_errors,report}, > {request_errors,answer_3xxx}]}]]}, > {transport,[[{ref,#Ref<0.0.0.1340>}, > {type,connect}, > {options,[{reconnect_timer,5000}, > {transport_module,diameter_tcp}, > {transport_config,[{raddr,{127,0,0,1}}, > {rport,3868}, > {reuseaddr,true}, > {ip,{127,0,0,1}}]}]}, > {watchdog,{<0.303.0>,{1379,78639,682458},okay}}, > {peer,{<0.304.0>,{1379,78639,686181}}}, > {apps,[{0,client}]}, > {caps,[{origin_host,{"client.example.com", > "service.example.com"}}, > {origin_realm,{"example.com","example.com"}}, > {host_ip_address,{[{127,0,0,1}],[{127,0,0,1}]}}, > {vendor_id,{0,193}}, > {product_name,{"Client","Server"}}, > {origin_state_id,{[],[]}}, > {supported_vendor_id,{[],[]}}, > {auth_application_id,{[0],[0]}}, > {inband_security_id,{[],[]}}, > {acct_application_id,{[],[]}}, > {vendor_specific_application_id,{[],...}}, > {firmware_revision,{...}}, > {avp,...}]}, > {port,[{owner,<0.306.0>}, > {module,diameter_tcp}, > {socket,{{127,0,0,1},53123}}, > {peer,{{127,0,0,1},3868}}, > {statistics,[{recv_oct,136}, > {recv_cnt,1}, > {recv_max,136}, > {recv_avg,136}, > {recv_dvi,...}, > {...}|...]}]}, > {statistics,[{{{0,257,0},recv},1},{{{0,257,1},send},1}]}]]}, > {pending,[]}, > {options,[{sequence,{0,32}}, > {share_peers,false}, > {use_shared_peers,false}, > {restrict_connections,nodes}]}] > (demo@REDACTED> diameter:service_info(demo_diameter_service, all). > [{capabilities,[{'Origin-Host',"service.example.com"}, > {'Origin-Realm',"example.com"}, > {'Vendor-Id',193}, > {'Product-Name',"Server"}, > {'Origin-State-Id',[]}, > {'Host-IP-Address',[]}, > {'Supported-Vendor-Id',[]}, > {'Auth-Application-Id',[0]}, > {'Inband-Security-Id',[]}, > {'Acct-Application-Id',[]}, > {'Vendor-Specific-Application-Id',[]}, > {'Firmware-Revision',[]}]}, > {applications,[[{alias,demo_diameter_service}, > {dictionary,diameter_gen_base_rfc3588}, > {module,[diameter_callback, > > {diameter_callback,undefined,undefined,undefined,undefined, > > undefined,undefined,undefined,undefined,undefined,[]}]}, > {init_state,demo_diameter_service}, > {id,0}, > {mutable,false}, > {options,[{answer_errors,report}, > {request_errors,answer_3xxx}]}]]}, > {transport,[[{ref,#Ref<0.0.0.769>}, > {type,listen}, > {options,[{transport_module,diameter_tcp}, > {transport_config,[{reuseaddr,true}, > {ip,{127,0,0,1}}, > {port,3868}]}]}, > {accept,[[{watchdog,{<0.308.0>, > {1379,78639,684566}, > initial}}], > [{watchdog,{<0.245.0>,{1379,78445,351797},okay}}, > {peer,{<0.246.0>,{1379,78639,685678}}}, > {apps,[{0,demo_diameter_service}]}, > {caps,[{origin_host,{"service.example.com", > "client.example.com"}}, > {origin_realm,{"example.com","example.com > "}}, > > {host_ip_address,{[{127,0,...}],[{127,...}]}}, > {vendor_id,{193,0}}, > {product_name,{"Server",[...]}}, > {origin_state_id,{[],...}}, > {supported_vendor_id,{...}}, > {auth_application_id,...}, > {...}|...]}, > {port,[{owner,<0.255.0>}, > {module,diameter_tcp}, > {socket,{{127,0,...},3868}}, > {peer,{{127,...},53123}}, > {statistics,[{...}|...]}]}]]}, > {statistics,[{{{0,257,0},send},1},{{{0,257,1},recv},1}]}]]}, > {pending,[]}, > {options,[{sequence,{0,32}}, > {share_peers,false}, > {use_shared_peers,false}, > {restrict_connections,nodes}]}] > (demo@REDACTED)9> demo_diameter_test_client:call(). > {error,no_connection} > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Fri Sep 13 16:42:37 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Fri, 13 Sep 2013 15:42:37 +0100 Subject: [erlang-questions] Question about basic use of diameter In-Reply-To: References: Message-ID: <7253BA12-AAE8-4888-B413-BE9839159A87@gmail.com> Hi Anders, On 13 Sep 2013, at 12:02, Anders Svensson wrote: > > The diameter application doesn't care. All that RFC 6733 requires of > Origin-Host is that it be unique within a single host (6.3), while > Origin-Realm is expected to be related to the DNS domain name (6.4, > 1.2; could be clearer). Using the physical host/domain is fairly > typical it seems. > Ok fair enough. >> After attempts to write and test a basic relay kept failing, I tried using the server.erl and client.erl from the diameter application's ./examples/code directory and things don't seem to be working properly: > [snip] > You haven't told the example server to start listening: you need to > call server:listen(tcp) as well. Here's how it should look: > > 6> up: <0.56.0> > client:call(). > {ok,{diameter_base_RAA,"client;1440573120;1;nonode@REDACTED", > 2001,"server.example.com","example.com",[],[],[],[],[], > [],[],[],[],[]}} > [snip] Oops, I forgot about that (when using the example client/server code), but the peer always goes down immediately after coming up: 1> diameter:start(). ok 2> server:start(). ok 3> server:listen(tcp). {ok,#Ref<0.0.0.53>} 4> client:start(). ok 5> client:connect(tcp). {ok,#Ref<0.0.0.99>} 6> up: <0.55.0> down: <0.55.0> 6> 6> It doesn't do that when I start the client from another shell though - then it connects fine! - but the ACR being passed across the wire doesn't seem to encode properly - I assumed that because both the client and server application's are configured with `{dictionary, ?DIAMETER_DICT_COMMON}' that it wouldn't be necessary to do anything "special" to get them to encode/decode properly... 1> diameter:start(). ok 2> client:start(). ok 3> client:connect(tcp). {ok,#Ref<0.0.0.53>} 4> client:call(). =ERROR REPORT==== 13-Sep-2013::15:02:02 === Error in process <0.63.0> with exit value: {{badrecord,diameter_base_ACR},[{client_cb,prepare_request,3,[{file,"src/client_cb.erl"},{line,55}]},{diameter_traffic,send_request,6,[{file,"base/diameter_traffic.erl"},{line,1195}]},{diameter_traffic,'-send_request/4-fun-0-'... {error,encode} > > Haven't seen this one but give the above a try for starters. > Just to clarify, this is an untouched version of server.erl, server_cb.erl, client.erl and client_cb.erl that I'm experimenting with - just to try and figure out what's going awry with my own code. In my own tests, calls to diameter:service/0 show the client and server configured on their respective nodes, and the server has run diameter:add_transport(Name, {listen, server(T)}) (and the client, connect(...) and so on) during initialisation. I shall try to make a bit more progress before taking up any more of your time anyway. Thanks! Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From seth@REDACTED Fri Sep 13 16:47:01 2013 From: seth@REDACTED (Seth Falcon) Date: Fri, 13 Sep 2013 07:47:01 -0700 Subject: [erlang-questions] better HTTP client? In-Reply-To: <20130909121435.GB13437@ferdmbp.local> References: <20130909121435.GB13437@ferdmbp.local> Message-ID: We've taken a similar approach, inspired by Fred's fork of lhttpc, but built on top of ibrowse. For us as well, the use case is handling many connections to a small set of servers. https://github.com/opscode/opscoderl_httpc On Mon, Sep 9, 2013 at 5:14 AM, Fred Hebert wrote: > You might be interested in my fork of lhttpc: > https://github.com/ferd/lhttpc > > It's pretty much regular lhttpc, but I added sharded connection pools to > it. It has been especially optimized for doing a lot of requests to a > limited set of endpoints, and was developed during my stay at AdGear for > real time bidding purposes. It's also used at other real time bidding > shops now. > > It will not do request cancelling the way you mention it. It has a > central load-balancer per endpoint, where you ask for sockets. It's easy > to cancel a task once you have the connection yourself, but not during > the time it's in the queue to the manager. > > If you kill the process that made the request, the manager will find out > once it tries to give away the process and keep going until it hits a > request that works > (https://github.com/ferd/lhttpc/blob/master/src/lhttpc_lb.erl#L98) > > That might be suboptimal behavior in your case. > > I also have a fork of that very load balancer done using dispcount > called dlhttpc: https://github.com/ferd/dlhttpc > > Dispcount is a low-latency pool (https://github.com/ferd/dispcount) that > basically will tell you right away if you can have a resource or not. > There will be no such thing as a blocking request using dlhttpc because > the pooling model doesn't allow for an explicit request queue. It > instead uses a pool of workers that may or may not be connected (in the > case of dlhttpc, each worker will only connect once asked for it, but > processes will be started already), and assumes you will retry until you > hit one you need. It's been designed with cases of constant overload in > mind, where you may arbitrarily decide to drop a request from the > call-site. > > Depending on how flexible your requirements are, I'd try at least one of > these libraries to see how they fare. > > lhttpc has been in production for a long while now and has been fairly > stable as far as I can tell, everywhere it's been deployed. dlhttpc has > seen no real production use, but dispcount has, so it's not too likely > to be a terrible thing to deploy. > > Regards, > Fred. > > On 09/08, Chris King wrote: > > On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito wrote: > > > > >Async cancelable and synchronous are basically isomorphic in > > >Erlang, just start a process to wrap the sync call and kill >it if > > >you want to cancel. > > > > Not quite -- the HTTP request remains outstanding and blocks the > > request queue. I need to clear those requests out, as I need to > > process new requests ASAP. This should be handled by the HTTP > > client itself, as only it knows how many requests it has actually > > sent to the server. (The alternative would be for the HTTP client > > to expose a crediting mechanism; though this would complicate the > > application.) > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Seth Falcon | @sfalcon | http://userprimary.net/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From anders.otp@REDACTED Fri Sep 13 17:27:34 2013 From: anders.otp@REDACTED (Anders Svensson) Date: Fri, 13 Sep 2013 17:27:34 +0200 Subject: [erlang-questions] Question about basic use of diameter Message-ID: [Adding the list Cc I bungled in my reply below.] One more comment ... On Fri, Sep 13, 2013 at 5:13 PM, Anders Svensson wrote: > Hi Tim. > > On Fri, Sep 13, 2013 at 4:42 PM, Tim Watson wrote: >> Hi Anders, >> >> On 13 Sep 2013, at 12:02, Anders Svensson wrote: > >> You haven't told the example server to start listening: you need to >> call server:listen(tcp) as well. Here's how it should look: >> >> 6> up: <0.56.0> >> client:call(). >> {ok,{diameter_base_RAA,"client;1440573120;1;nonode@REDACTED", >> >> 2001,"server.example.com","example.com",[],[],[],[],[], >> [],[],[],[],[]}} >> >> [snip] >> >> Oops, I forgot about that (when using the example client/server code), but >> the peer always goes down immediately after coming up: >> >> 1> diameter:start(). >> ok >> 2> server:start(). >> ok >> 3> server:listen(tcp). >> {ok,#Ref<0.0.0.53>} >> 4> client:start(). >> ok >> 5> client:connect(tcp). >> {ok,#Ref<0.0.0.99>} >> 6> up: <0.55.0> >> down: <0.55.0> >> >> 6> >> 6> >> >> It doesn't do that when I start the client from another shell though - then >> it connects fine! - but the ACR being passed across the wire doesn't seem to >> encode properly - I assumed that because both the client and server >> application's are configured with `{dictionary, ?DIAMETER_DICT_COMMON}' that >> it wouldn't be necessary to do anything "special" to get them to >> encode/decode properly... >> >> 1> diameter:start(). >> ok >> 2> client:start(). >> ok >> 3> client:connect(tcp). >> {ok,#Ref<0.0.0.53>} >> 4> client:call(). >> >> =ERROR REPORT==== 13-Sep-2013::15:02:02 === >> Error in process <0.63.0> with exit value: >> {{badrecord,diameter_base_ACR},[{client_cb,prepare_request,3,[{file,"src/client_cb.erl"},{line,55}]},{diameter_traffic,send_request,6,[{file,"base/diameter_traffic.erl"},{line,1195}]},{diameter_traffic,'-send_request/4-fun-0-'... > > Your compilation of the example code looks to have included a > different version of the diameter hrls than the diameter itself: > diameter and the client have different ideas of the definition of > record diameter_base_ACR. Could you have compiled the example code > with something other than the R16B01 you've run it with? Another explanation would be having included hrls you've generated from a modified dictionary. Anders > > This error is what causes the connection to be dropped. > > Anders > > > > >> {error,encode} >> >> >> >> Haven't seen this one but give the above a try for starters. >> >> >> Just to clarify, this is an untouched version of server.erl, server_cb.erl, >> client.erl and client_cb.erl that I'm experimenting with - just to try and >> figure out what's going awry with my own code. In my own tests, calls to >> diameter:service/0 show the client and server configured on their respective >> nodes, and the server has run diameter:add_transport(Name, {listen, >> server(T)}) (and the client, connect(...) and so on) during initialisation. >> I shall try to make a bit more progress before taking up any more of your >> time anyway. >> >> Thanks! >> >> Tim >> From rickard@REDACTED Fri Sep 13 18:27:30 2013 From: rickard@REDACTED (Rickard Green) Date: Fri, 13 Sep 2013 18:27:30 +0200 Subject: [erlang-questions] pid representation in external term format Message-ID: <52333CF2.6030104@erlang.org> Up to somewhere around the R9 release pids were composed by a 15 bit index into the process table and a 3 bit serial. After that we use 28 bits, i.e., all bits available in an immediate term on a 32-bit machine. Number of bits used as index is determined at boot-time instead of at compile-time. This can, quite easily, be extended to 60 bits on 64-bit machines when it comes to the VM. I've already implemented this once in the VM, but dropped it since it required backward incompatible changes and a lot of work in erl_interface/ei. This made us settle for 28 bits on all machines (unfortunately). The _PID_NUM_SIZE of 15 bits is more or less only still there in order to make the textual representation of pids look (somewhat) the same as they used to before R9. Since 2^28 is a quite small number, pids may be reused quite fast in a system that spawns a lot of processes. All of the 28 bits of the pid is passed over the distribution. Regards, Rickard Green > Yes, what I meant was: > Why wouldn't the Erlang VM be changed to use 4 bytes (32 bits) for the Serial and 4 bytes (32 bits) for the Id (giving us 46 more bits), so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? > > On 09/12/2013 04:05 PM, Serge Aleynikov wrote: >> It seems to me that you were mistaken as the 4 (id) and 4 (serial) in the External Term Format specification (*) indicate the sizes in bytes rather than in bits. >> >> (*) http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 >> >> >> On Thu, Sep 12, 2013 at 6:22 PM, Michael Truog < > wrote: >> >> On 09/11/2013 09:58 AM, Serge Aleynikov wrote: >>> Presently the PID representation in external term format is limited to the following: >>> >>> http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 >>> >>> ./erts/emulator/beam/erl_node_container_utils.h:#define ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) >>> ./erts/emulator/beam/erl_term.h:#define _PID_NUM_SIZE 15 >>> >>> ID is limited to 15 bits >>> Serial is limited to 3 bits >>> >>> So in total a PID consists of 18 bits, and therefore it seems that the number of pids on any remote node cannot exceed 2^18 (262144). While it may seem like a large number, when creating a node in other languages that implement Erlang distributed transport (e.g. C/C#/Java) and create/destroy mailboxes, the local pid counter used to create unique Pids can easily go over that limit. The work-around is to cache freed local pids and resurrect them when pid counter wraps around 2^18 boundary. >>> >>> This brings the question of whether that limitation is still necessary in the current version of distribution. Internally Pids use a wider representation (is it 28 bits?), so aside for supporting older versions of beam (which can be worked around through flags in distributed transport) is there any valid reason not to increase the pid maximum numbering limit? >>> >>> BTW, as a side note, how is the same problem addressed in the beam when the pid ID counter reaches that limit? Does it make it possible that a newly assigned Pid becomes non-unique? (I.e. if some entity still maintains a reference to an old Pid that died, and later after the pid ID counter wrapped around, a new Pid was assigned the same Pid ID number of a previously dead Pid, then the entity that had the reference to the old Pid with the same ID, could send a message to it that would not be valid for the new Pid.) >>> >>> Serge >>> >>> >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> >>> http://erlang.org/mailman/listinfo/erlang-questions >> Why wouldn't the Erlang VM be changed to use 4 bits for the Serial and 4 bits for the Id, so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? >> >> >> -- Rickard Green, Erlang/OTP, Ericsson AB. From serge@REDACTED Fri Sep 13 20:00:09 2013 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 13 Sep 2013 14:00:09 -0400 Subject: [erlang-questions] pid representation in external term format In-Reply-To: <52333CF2.6030104@erlang.org> References: <52333CF2.6030104@erlang.org> Message-ID: A h, I was looking at stale source code where Serial was 3 bits rather than 13. Still 28 bits is a pretty small number when it comes to other languages interfacing with Erlang over distributed transport where a new Pid can be used for each RPC call. Finding an available Pid number that could be reused after the counter wrap-around wouldn't be as cheap as merely incrementing a counter. When the pid counter wraps around, how does the VM ensure that the incremented Pid value doesn't collide with an existing Pid? It doesn't seem like upgrading erl_interface to 64 bit representation of pids would be a huge matter, since pid structure already has two 32-bit integers (id + serial), so it would just be necessary to adjust the masks when encoding/decoding values. So, if there's a way to resurrect this issue and increase the Pid representation to 60bit in a 64bit environment, this would really be helpful. On Fri, Sep 13, 2013 at 12:27 PM, Rickard Green wrote: > Up to somewhere around the R9 release pids were composed by a 15 bit index > into the process table and a 3 bit serial. After that we use 28 bits, i.e., > all bits available in an immediate term on a 32-bit machine. Number of bits > used as index is determined at boot-time instead of at compile-time. > > This can, quite easily, be extended to 60 bits on 64-bit machines when it > comes to the VM. I've already implemented this once in the VM, but dropped > it since it required backward incompatible changes and a lot of work in > erl_interface/ei. This made us settle for 28 bits on all machines > (unfortunately). > > The _PID_NUM_SIZE of 15 bits is more or less only still there in order to > make the textual representation of pids look (somewhat) the same as they > used to before R9. > > Since 2^28 is a quite small number, pids may be reused quite fast in a > system that spawns a lot of processes. All of the 28 bits of the pid is > passed over the distribution. > > Regards, > Rickard Green > > Yes, what I meant was: >> Why wouldn't the Erlang VM be changed to use 4 bytes (32 bits) for the >> Serial and 4 bytes (32 bits) for the Id (giving us 46 more bits), so that >> the sizes match what the External Term Format allows? Is this change on >> the Erlang VM roadmap? >> >> On 09/12/2013 04:05 PM, Serge Aleynikov wrote: >> >>> It seems to me that you were mistaken as the 4 (id) and 4 (serial) in >>> the External Term Format specification (*) indicate the sizes in bytes >>> rather than in bits. >>> >>> (*) http://www.erlang.org/doc/**apps/erts/erl_ext_dist.html#**id87011 >>> >>> >>> On Thu, Sep 12, 2013 at 6:22 PM, Michael Truog < > wrote: >>> >>> On 09/11/2013 09:58 AM, Serge Aleynikov wrote: >>> >>>> Presently the PID representation in external term format is limited >>>> to the following: >>>> >>>> http://www.erlang.org/doc/**apps/erts/erl_ext_dist.html#**id87011 >>>> >>>> ./erts/emulator/beam/erl_node_**container_utils.h:#define >>>> ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) >>>> ./erts/emulator/beam/erl_term.**h:#define _PID_NUM_SIZE >>>> 15 >>>> >>>> ID is limited to 15 bits >>>> Serial is limited to 3 bits >>>> >>>> So in total a PID consists of 18 bits, and therefore it seems that >>>> the number of pids on any remote node cannot exceed 2^18 (262144). While >>>> it may seem like a large number, when creating a node in other languages >>>> that implement Erlang distributed transport (e.g. C/C#/Java) and >>>> create/destroy mailboxes, the local pid counter used to create unique Pids >>>> can easily go over that limit. The work-around is to cache freed local >>>> pids and resurrect them when pid counter wraps around 2^18 boundary. >>>> >>>> This brings the question of whether that limitation is still >>>> necessary in the current version of distribution. Internally Pids use a >>>> wider representation (is it 28 bits?), so aside for supporting older >>>> versions of beam (which can be worked around through flags in distributed >>>> transport) is there any valid reason not to increase the pid maximum >>>> numbering limit? >>>> >>>> BTW, as a side note, how is the same problem addressed in the beam >>>> when the pid ID counter reaches that limit? Does it make it possible that a >>>> newly assigned Pid becomes non-unique? (I.e. if some entity still maintains >>>> a reference to an old Pid that died, and later after the pid ID counter >>>> wrapped around, a new Pid was assigned the same Pid ID number of a >>>> previously dead Pid, then the entity that had the reference to the old Pid >>>> with the same ID, could send a message to it that would not be valid for >>>> the new Pid.) >>>> >>>> Serge >>>> >>>> >>>> >>>> ______________________________**_________________ >>>> erlang-questions mailing list >>>> >>>> http://erlang.org/mailman/**listinfo/erlang-questions >>>> >>> Why wouldn't the Erlang VM be changed to use 4 bits for the Serial >>> and 4 bits for the Id, so that the sizes match what the External Term >>> Format allows? Is this change on the Erlang VM roadmap? >>> >>> >>> >>> > -- > Rickard Green, Erlang/OTP, Ericsson AB. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickard@REDACTED Fri Sep 13 23:48:51 2013 From: rickard@REDACTED (Rickard Green) Date: Fri, 13 Sep 2013 23:48:51 +0200 Subject: [erlang-questions] pid representation in external term format In-Reply-To: References: <52333CF2.6030104@erlang.org> Message-ID: <51510008-90C3-4C11-9E05-4B9C449C08B3@erlang.org> On Sep 13, 2013, at 8:00 PM, Serge Aleynikov wrote: > Ah, I was looking at stale source code where Serial was 3 bits rather than 13. > > Still 28 bits is a pretty small number when it comes to other languages interfacing with Erlang over distributed transport where a new Pid can be used for each RPC call. Finding an available Pid number that could be reused after the counter wrap-around wouldn't be as cheap as merely incrementing a counter. > Since there only 2^28 unique pids per node I don't think it is a good idea to use a new pid in each rpc-call. I agree that it would be nice with 60-bit pids, for several reasons, but this currently has a very low priority compared to other things. > When the pid counter wraps around, how does the VM ensure that the incremented Pid value doesn't collide with an existing Pid? > The old algorithm scanned the process table for an unused id, the new algorithm introduced in R16B01 (somewhat buggy which has been fixed in R16B02) has a buffer of unused ids which it (more or less) just picks the next id from (see erl_ptab.c). > It doesn't seem like upgrading erl_interface to 64 bit representation of pids would be a huge matter, since pid structure already has two 32-bit integers (id + serial), so it would just be necessary to adjust the masks when encoding/decoding values. > At a first glance it might seem like an easy task to enable 60-bit pids in erl_interface/ei, but it is much more work than it first seems like as well as backward compatibility issues that have to be addressed. I don't remember the details since it was nearly ten years ago I looked at it. When we get a fully working, redesigned, and implemented open source patch for 60-bit pids in erl_interface/ei that also addresses the backward compatibility issues, I'll gladly implement 60-bit pids in the 64-bit VM and jinterface on my spare time :-) Regards, Rickard > So, if there's a way to resurrect this issue and increase the Pid representation to 60bit in a 64bit environment, this would really be helpful. > > > On Fri, Sep 13, 2013 at 12:27 PM, Rickard Green wrote: > Up to somewhere around the R9 release pids were composed by a 15 bit index into the process table and a 3 bit serial. After that we use 28 bits, i.e., all bits available in an immediate term on a 32-bit machine. Number of bits used as index is determined at boot-time instead of at compile-time. > > This can, quite easily, be extended to 60 bits on 64-bit machines when it comes to the VM. I've already implemented this once in the VM, but dropped it since it required backward incompatible changes and a lot of work in erl_interface/ei. This made us settle for 28 bits on all machines (unfortunately). > > The _PID_NUM_SIZE of 15 bits is more or less only still there in order to make the textual representation of pids look (somewhat) the same as they used to before R9. > > Since 2^28 is a quite small number, pids may be reused quite fast in a system that spawns a lot of processes. All of the 28 bits of the pid is passed over the distribution. > > Regards, > Rickard Green > > Yes, what I meant was: > Why wouldn't the Erlang VM be changed to use 4 bytes (32 bits) for the Serial and 4 bytes (32 bits) for the Id (giving us 46 more bits), so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? > > On 09/12/2013 04:05 PM, Serge Aleynikov wrote: > It seems to me that you were mistaken as the 4 (id) and 4 (serial) in the External Term Format specification (*) indicate the sizes in bytes rather than in bits. > > (*) http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 > > > On Thu, Sep 12, 2013 at 6:22 PM, Michael Truog < > wrote: > > On 09/11/2013 09:58 AM, Serge Aleynikov wrote: > Presently the PID representation in external term format is limited to the following: > > http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id87011 > > ./erts/emulator/beam/erl_node_container_utils.h:#define ERTS_MAX_PID_NUMBER ((1 << _PID_NUM_SIZE) - 1) > ./erts/emulator/beam/erl_term.h:#define _PID_NUM_SIZE 15 > > ID is limited to 15 bits > Serial is limited to 3 bits > > So in total a PID consists of 18 bits, and therefore it seems that the number of pids on any remote node cannot exceed 2^18 (262144). While it may seem like a large number, when creating a node in other languages that implement Erlang distributed transport (e.g. C/C#/Java) and create/destroy mailboxes, the local pid counter used to create unique Pids can easily go over that limit. The work-around is to cache freed local pids and resurrect them when pid counter wraps around 2^18 boundary. > > This brings the question of whether that limitation is still necessary in the current version of distribution. Internally Pids use a wider representation (is it 28 bits?), so aside for supporting older versions of beam (which can be worked around through flags in distributed transport) is there any valid reason not to increase the pid maximum numbering limit? > > BTW, as a side note, how is the same problem addressed in the beam when the pid ID counter reaches that limit? Does it make it possible that a newly assigned Pid becomes non-unique? (I.e. if some entity still maintains a reference to an old Pid that died, and later after the pid ID counter wrapped around, a new Pid was assigned the same Pid ID number of a previously dead Pid, then the entity that had the reference to the old Pid with the same ID, could send a message to it that would not be valid for the new Pid.) > > Serge > > > > _______________________________________________ > erlang-questions mailing list > > http://erlang.org/mailman/listinfo/erlang-questions > Why wouldn't the Erlang VM be changed to use 4 bits for the Serial and 4 bits for the Id, so that the sizes match what the External Term Format allows? Is this change on the Erlang VM roadmap? > > > > > -- > Rickard Green, Erlang/OTP, Ericsson AB. > From pablo.platt@REDACTED Sat Sep 14 04:09:33 2013 From: pablo.platt@REDACTED (pablo platt) Date: Sat, 14 Sep 2013 05:09:33 +0300 Subject: [erlang-questions] STUN server In-Reply-To: References: <51A94A2A.2020303@gmail.com> <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> Message-ID: Hi Benoit, Did you release your ice client and stun/turn server? Thanks On Thu, Aug 1, 2013 at 1:15 AM, pablo platt wrote: > Great news Benoit. > > Thanks > > > On Thu, Aug 1, 2013 at 1:13 AM, Benoit Chesneau wrote: > >> >> >> >> On Tue, Jul 30, 2013 at 6:39 PM, wrote: >> >>> benoit, did you make any progress with your ICE server? >>> >>> I'm testing the processone server but I need ICE (ice-lite) and there >>> are few things missing like the priority, use-candidate, ice-controlled and >>> ice-controlling attributes. >>> I've added the missing attributes but still can't respond to ICE >>> connectivity checks. >>> >>> I've found two more erlang implementations but they seems incomplete: >>> https://github.com/kevinlynx/icerl >>> https://github.com/kalta/nksip/blob/master/nksip/src/nksip_stun.erl >> >> >> Hi pablo, >> >> Yes I will have an ice client and stun/turn server available next week. >> >> The stun/turn server is working over TCP/TLS and udp. for tcp/tls it is >> using barrel. the stun/turn client is able to provide a socket linked to >> the public port. >> >> The ice client/server is able to decde ice messages and eventually use >> them. >> >> - benoit >> >> >> >>> >>> >>> On Sunday, July 7, 2013 5:59:40 AM UTC+3, Benoit Chesneau wrote: >>> >>>> >>>> >>>> >>>> On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov wrote: >>>> >>>>> On 01.06.2013 01:00, Igor Karymov wrote: >>>>> >>>>>> You may take a look here: >>>>>> https://github.com/processone/****ejabberd/tree/master/src/stun >>>>>> >>>>>> >>>>> It's a separate project now: https://github.com/processone/****stun >>>>> >>>>> -- >>>>> >>>>> >>>> I'm just back on that topic. >>>> >>>> Thanks for the code. Unfortunately the licensee is too restrictive for >>>> my project so I will have to find another way to do that. Anyway just >>>> wanted to say thanks it can be useful for some. >>>> >>>> - benoit >>>> >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sat Sep 14 05:59:15 2013 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 13 Sep 2013 20:59:15 -0700 Subject: [erlang-questions] STUN server In-Reply-To: References: <51A94A2A.2020303@gmail.com> <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> Message-ID: <5233DF13.1080209@gmail.com> https://github.com/lemenkov/rtplib/blob/master/src/stun.erl is BSD licensed and seems better than the links listed below. The basic stun server using it is here: https://github.com/lemenkov/erlstund This code could help create an ICE server. On 09/13/2013 07:09 PM, pablo platt wrote: > Hi Benoit, > > Did you release your ice client and stun/turn server? > > Thanks > > > On Thu, Aug 1, 2013 at 1:15 AM, pablo platt > wrote: > > Great news Benoit. > > Thanks > > > On Thu, Aug 1, 2013 at 1:13 AM, Benoit Chesneau > wrote: > > > > > On Tue, Jul 30, 2013 at 6:39 PM, > wrote: > > benoit, did you make any progress with your ICE server? > > I'm testing the processone server but I need ICE (ice-lite) and there are few things missing like the priority, use-candidate, ice-controlled and ice-controlling attributes. > I've added the missing attributes but still can't respond to ICE connectivity checks. > > I've found two more erlang implementations but they seems incomplete: > https://github.com/kevinlynx/icerl > https://github.com/kalta/nksip/blob/master/nksip/src/nksip_stun.erl > > > Hi pablo, > > Yes I will have an ice client and stun/turn server available next week. > > The stun/turn server is working over TCP/TLS and udp. for tcp/tls it is using barrel. the stun/turn client is able to provide a socket linked to the public port. > > The ice client/server is able to decde ice messages and eventually use them. > > - benoit > > > > > > On Sunday, July 7, 2013 5:59:40 AM UTC+3, Benoit Chesneau wrote: > > > > > On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov wrote: > > On 01.06.2013 01:00, Igor Karymov wrote: > > You may take a look here: > https://github.com/processone/ejabberd/tree/master/src/stun > > > It's a separate project now: https://github.com/processone/stun > > -- > > > I'm just back on that topic. > > Thanks for the code. Unfortunately the licensee is too restrictive for my project so I will have to find another way to do that. Anyway just wanted to say thanks it can be useful for some. > > - benoit > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Sat Sep 14 17:48:51 2013 From: pablo.platt@REDACTED (pablo platt) Date: Sat, 14 Sep 2013 18:48:51 +0300 Subject: [erlang-questions] STUN server In-Reply-To: <5233DF13.1080209@gmail.com> References: <51A94A2A.2020303@gmail.com> <455fff1c-2133-462f-a0b2-3d0f9e6ac6c5@googlegroups.com> <5233DF13.1080209@gmail.com> Message-ID: The message integrity fails for me https://github.com/lemenkov/rtplib/blob/master/src/stun.erl#L321 Did you mange to use this module for ICE with message-integrity? I'm still interested to try Benoit's implementation if available and open sourced. On Sat, Sep 14, 2013 at 6:59 AM, Michael Truog wrote: > https://github.com/lemenkov/rtplib/blob/master/src/stun.erl is BSD > licensed and seems better than the links listed below. The basic stun > server using it is here: > https://github.com/lemenkov/erlstund > > This code could help create an ICE server. > > > On 09/13/2013 07:09 PM, pablo platt wrote: > > Hi Benoit, > > Did you release your ice client and stun/turn server? > > Thanks > > > On Thu, Aug 1, 2013 at 1:15 AM, pablo platt wrote: > >> Great news Benoit. >> >> Thanks >> >> >> On Thu, Aug 1, 2013 at 1:13 AM, Benoit Chesneau wrote: >> >>> >>> >>> >>> On Tue, Jul 30, 2013 at 6:39 PM, wrote: >>> >>>> benoit, did you make any progress with your ICE server? >>>> >>>> I'm testing the processone server but I need ICE (ice-lite) and there >>>> are few things missing like the priority, use-candidate, ice-controlled and >>>> ice-controlling attributes. >>>> I've added the missing attributes but still can't respond to ICE >>>> connectivity checks. >>>> >>>> I've found two more erlang implementations but they seems incomplete: >>>> https://github.com/kevinlynx/icerl >>>> https://github.com/kalta/nksip/blob/master/nksip/src/nksip_stun.erl >>> >>> >>> Hi pablo, >>> >>> Yes I will have an ice client and stun/turn server available next >>> week. >>> >>> The stun/turn server is working over TCP/TLS and udp. for tcp/tls it >>> is using barrel. the stun/turn client is able to provide a socket linked to >>> the public port. >>> >>> The ice client/server is able to decde ice messages and eventually use >>> them. >>> >>> - benoit >>> >>> >>> >>>> >>>> >>>> On Sunday, July 7, 2013 5:59:40 AM UTC+3, Benoit Chesneau wrote: >>>> >>>>> >>>>> >>>>> >>>>> On Sat, Jun 1, 2013 at 3:11 AM, Evgeniy Khramtsov wrote: >>>>> >>>>>> On 01.06.2013 01:00, Igor Karymov wrote: >>>>>> >>>>>>> You may take a look here: >>>>>>> https://github.com/processone/ejabberd/tree/master/src/stun >>>>>>> >>>>>>> >>>>>> It's a separate project now: https://github.com/processone/stun >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>> I'm just back on that topic. >>>>> >>>>> Thanks for the code. Unfortunately the licensee is too restrictive >>>>> for my project so I will have to find another way to do that. Anyway just >>>>> wanted to say thanks it can be useful for some. >>>>> >>>>> - benoit >>>>> >>>>> >>> >> > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Sun Sep 15 11:57:52 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Sun, 15 Sep 2013 11:57:52 +0200 Subject: [erlang-questions] better HTTP client? In-Reply-To: References: Message-ID: Hi! On the branch http-client-cancel-request in my git repository I have addressed the issues you reported on request cancellation and pipelining. Alas it was not in time for R16B02 but will probably be merged to the maint branch soon after ... https://github.com/IngelaAndin/otp/compare/ia;inets;http-client-cancel-request Of course it passes our tests but all extra testing of this is welcome. Regards Ingela Erlang/OTP team - Ericsson AB 2013/9/9 Chris King > Hi all, > > Due to various architectural bugs in the inets:httpc module (which I have > reported) related to queueing multiple requests to a single server, I'm > looking for a replacement for my application. > > The use case requires queuing many GET requests to a small number of > servers; taking advantage of keep-alive/pipelining when available; and > allowing outstanding unserviced requests to be canceled. I don't care > about streaming or writing to disk or any sort of upload capability. Any > suggestions or should I roll my own? > > (The work required to re-architect inets:httpc compared to writing a > simple server that meets exactly my needs is too much for me right now.) > > Thanks, > Chris > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lee.sylvester@REDACTED Sun Sep 15 15:44:41 2013 From: lee.sylvester@REDACTED (Lee Sylvester) Date: Sun, 15 Sep 2013 14:44:41 +0100 Subject: [erlang-questions] Monitoring Message-ID: <75296695-39DB-4782-A970-268F80212313@gmail.com> Hey guys, So, I want to monitor my Erlang apps. Currently, I'm using Monit to monitor my other (non-Erlang) apps. Can I hook Monit up to monitor my Erlang apps, too? If so, what is the best to get the PID of the running Erlang app, so I can monitor it? I've tried to do this with a service script, but querying $! reveals nothing and there are just too many apps supporting my Erlang app to keep track of. If Monit is NOT the best option, then what is? Thanks for all the help Cheers, Lee From vinoski@REDACTED Sun Sep 15 16:31:03 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Sun, 15 Sep 2013 10:31:03 -0400 Subject: [erlang-questions] Monitoring In-Reply-To: <75296695-39DB-4782-A970-268F80212313@gmail.com> References: <75296695-39DB-4782-A970-268F80212313@gmail.com> Message-ID: On Sun, Sep 15, 2013 at 9:44 AM, Lee Sylvester wrote: > Hey guys, > > So, I want to monitor my Erlang apps. Currently, I'm using Monit to > monitor my other (non-Erlang) apps. Can I hook Monit up to monitor my > Erlang apps, too? If so, what is the best to get the PID of the running > Erlang app, so I can monitor it? I've tried to do this with a service > script, but querying $! reveals nothing and there are just too many apps > supporting my Erlang app to keep track of. If Monit is NOT the best > option, then what is? > Perhaps you can write the output of os:getpid() somewhere for your monitoring tool to find it? http://erlang.org/doc/man/os.html#getpid-0 --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sun Sep 15 16:37:28 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 15 Sep 2013 17:37:28 +0300 Subject: [erlang-questions] Monitoring In-Reply-To: <75296695-39DB-4782-A970-268F80212313@gmail.com> References: <75296695-39DB-4782-A970-268F80212313@gmail.com> Message-ID: <2DEDBCFA-2114-478C-8335-9AF00A172B18@gmail.com> Hello, I am using "heart" (erlang native) to monitor node and repair it damage. It works very well in our production, I am very positive about it. Of course it do not help on host crash. The host health is monitored using state-of-ther art technology (e.g. ZenOss) - Dmitry On Sep 15, 2013, at 4:44 PM, Lee Sylvester wrote: > Hey guys, > > So, I want to monitor my Erlang apps. Currently, I'm using Monit to monitor my other (non-Erlang) apps. Can I hook Monit up to monitor my Erlang apps, too? If so, what is the best to get the PID of the running Erlang app, so I can monitor it? I've tried to do this with a service script, but querying $! reveals nothing and there are just too many apps supporting my Erlang app to keep track of. If Monit is NOT the best option, then what is? > > Thanks for all the help > > Cheers, > Lee > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lee.sylvester@REDACTED Sun Sep 15 16:38:42 2013 From: lee.sylvester@REDACTED (Lee Sylvester) Date: Sun, 15 Sep 2013 15:38:42 +0100 Subject: [erlang-questions] Monitoring In-Reply-To: <2DEDBCFA-2114-478C-8335-9AF00A172B18@gmail.com> References: <75296695-39DB-4782-A970-268F80212313@gmail.com> <2DEDBCFA-2114-478C-8335-9AF00A172B18@gmail.com> Message-ID: <740FF892-BBA1-4843-9D77-6A5CC8251BD0@gmail.com> Thanks guys. I've just discovered erld. I'm going to give it a whirl, but will fall back to heart if it doesn't fulfil my needs ;-) Regards, Lee On 15 Sep 2013, at 15:37, Dmitry Kolesnikov wrote: > Hello, > > I am using "heart" (erlang native) to monitor node and repair it damage. It works very well in our production, I am very positive about it. Of course it do not help on host crash. The host health is monitored using state-of-ther art technology (e.g. ZenOss) > > - Dmitry > > On Sep 15, 2013, at 4:44 PM, Lee Sylvester wrote: > >> Hey guys, >> >> So, I want to monitor my Erlang apps. Currently, I'm using Monit to monitor my other (non-Erlang) apps. Can I hook Monit up to monitor my Erlang apps, too? If so, what is the best to get the PID of the running Erlang app, so I can monitor it? I've tried to do this with a service script, but querying $! reveals nothing and there are just too many apps supporting my Erlang app to keep track of. If Monit is NOT the best option, then what is? >> >> Thanks for all the help >> >> Cheers, >> Lee >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From g@REDACTED Sun Sep 15 19:55:27 2013 From: g@REDACTED (Garrett Smith) Date: Sun, 15 Sep 2013 12:55:27 -0500 Subject: [erlang-questions] Monitoring In-Reply-To: <740FF892-BBA1-4843-9D77-6A5CC8251BD0@gmail.com> References: <75296695-39DB-4782-A970-268F80212313@gmail.com> <2DEDBCFA-2114-478C-8335-9AF00A172B18@gmail.com> <740FF892-BBA1-4843-9D77-6A5CC8251BD0@gmail.com> Message-ID: Hi Lee, I put this into a general systems management category -- so whatever you do for other daemons/services you might want to do for Erlang. I have had success with runit and systemd -- using these for whole system init and supervision. One of the problems with pidfiles is that they eventually lie to you, which is hurtful. There's a school of though that says always run a service as a foreground process -- never in the background. runit e.g. prefers this approach. I use run_erl for this: http://www.erlang.org/doc/man/run_erl.html There are some here who will tell you not to use run_erl. Though I believe it's safe to ignore their advice on this point :) Garrett On Sun, Sep 15, 2013 at 9:38 AM, Lee Sylvester wrote: > Thanks guys. I've just discovered erld. I'm going to give it a whirl, but will fall back to heart if it doesn't fulfil my needs ;-) > > Regards, > Lee > > > > On 15 Sep 2013, at 15:37, Dmitry Kolesnikov wrote: > >> Hello, >> >> I am using "heart" (erlang native) to monitor node and repair it damage. It works very well in our production, I am very positive about it. Of course it do not help on host crash. The host health is monitored using state-of-ther art technology (e.g. ZenOss) >> >> - Dmitry >> >> On Sep 15, 2013, at 4:44 PM, Lee Sylvester wrote: >> >>> Hey guys, >>> >>> So, I want to monitor my Erlang apps. Currently, I'm using Monit to monitor my other (non-Erlang) apps. Can I hook Monit up to monitor my Erlang apps, too? If so, what is the best to get the PID of the running Erlang app, so I can monitor it? I've tried to do this with a service script, but querying $! reveals nothing and there are just too many apps supporting my Erlang app to keep track of. If Monit is NOT the best option, then what is? >>> >>> Thanks for all the help >>> >>> Cheers, >>> Lee >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From lee.sylvester@REDACTED Sun Sep 15 20:21:33 2013 From: lee.sylvester@REDACTED (Lee Sylvester) Date: Sun, 15 Sep 2013 19:21:33 +0100 Subject: [erlang-questions] Monitoring In-Reply-To: References: <75296695-39DB-4782-A970-268F80212313@gmail.com> <2DEDBCFA-2114-478C-8335-9AF00A172B18@gmail.com> <740FF892-BBA1-4843-9D77-6A5CC8251BD0@gmail.com> Message-ID: Thanks, Garrett. I'll give that a try. Having no luck with erld. I keep getting "Already running on pid 0" Cheers, Lee On 15 Sep 2013, at 18:55, Garrett Smith wrote: > Hi Lee, > > I put this into a general systems management category -- so whatever > you do for other daemons/services you might want to do for Erlang. > > I have had success with runit and systemd -- using these for whole > system init and supervision. > > One of the problems with pidfiles is that they eventually lie to you, > which is hurtful. > > There's a school of though that says always run a service as a > foreground process -- never in the background. runit e.g. prefers this > approach. I use run_erl for this: > > http://www.erlang.org/doc/man/run_erl.html > > There are some here who will tell you not to use run_erl. Though I > believe it's safe to ignore their advice on this point :) > > Garrett > > On Sun, Sep 15, 2013 at 9:38 AM, Lee Sylvester wrote: >> Thanks guys. I've just discovered erld. I'm going to give it a whirl, but will fall back to heart if it doesn't fulfil my needs ;-) >> >> Regards, >> Lee >> >> >> >> On 15 Sep 2013, at 15:37, Dmitry Kolesnikov wrote: >> >>> Hello, >>> >>> I am using "heart" (erlang native) to monitor node and repair it damage. It works very well in our production, I am very positive about it. Of course it do not help on host crash. The host health is monitored using state-of-ther art technology (e.g. ZenOss) >>> >>> - Dmitry >>> >>> On Sep 15, 2013, at 4:44 PM, Lee Sylvester wrote: >>> >>>> Hey guys, >>>> >>>> So, I want to monitor my Erlang apps. Currently, I'm using Monit to monitor my other (non-Erlang) apps. Can I hook Monit up to monitor my Erlang apps, too? If so, what is the best to get the PID of the running Erlang app, so I can monitor it? I've tried to do this with a service script, but querying $! reveals nothing and there are just too many apps supporting my Erlang app to keep track of. If Monit is NOT the best option, then what is? >>>> >>>> Thanks for all the help >>>> >>>> Cheers, >>>> Lee >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From ludovic@REDACTED Sun Sep 15 22:30:46 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Sun, 15 Sep 2013 22:30:46 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: Thank you all for your answers. Dmitry i dont understand what you mean by "HA" .. I'm not really into ditributed Erlang at the moment but i should definitely make somes tests. For now, if my lonely node crashes, as data is saved often to disc (e.g. in Postgres), it should be fresh. I just hope thant the entire server doesn't burn... Lo?c, you said : """You most likely don't need to store all of this in a database.""" Actually, this is a browser game server, the kind of you refresh a few times in a day to update things (buildings, productions, skills, ...), all the data must be saved often. And as people aren't connected continuously, i don't want all my data living in RAM all of the time. (BTW i should upgrade my hosting, but for now i am quite limited on RAM) Jesper said that "you can quickly communicate in the same memory space between processes, so you don't need a mediating DB layer". Yup, the database is only there to be a persistance layer, not to allow "things" to communicate. The models are there to bring the data to the application, then to store this data if changed. My code example was bad. When you update a record in my models, they are not saved each time, you must call a 'store' function on the model. Thats why i did the 'proc' function : first you load the data from the database, then the controller can .. control things and finally the record is updated and saved once. ... Controller calculations ... DamgeDone = calculate_damage_done(...) Conn = nemo:get_connection(testdba), M = nemo:load(ship,,Conn), Proc = [{get,health_points}, fun(HP,Ship) -> Ship#ship{health_points = HP- DamgeDone} end, {store}], %% <--- triggers the database update NewHealthPoints = m:proc(M,Proc) Plus, i wanted to build a model which could be used for any data in a more common web application : pages, articles, slidshows, todos, etc. Something more general as BossDB does. With the use of 'methods' in my models, written in the relevant 'type module' like ship.erl, i'm not so far from a regular gen_server. THe truth is i was fed up writing gen_servers again and again to do theese kind of things. But in the end it's barely the same thing. But i will follow your advice, i will use my code to build prototypes of servers and when things are kind of stable, turn theese models in regular gen_server instances. This should speed up code execution. And yes, my application is standalone, it just setup cowboy or yaws to be accessible through the web. Thanks you all again, this helps Le Fri, 13 Sep 2013 13:01:55 +0200, Dmitry Kolesnikov a ?crit: > Hello, > > On Sep 13, 2013, at 12:59 PM, Jesper Louis Andersen > wrote: > >> In Erlang, you have options. Safety is had due to isolation, so you >> don't have to push data onto stable storage as often. And you can >> quickly communicate in the same memory space between processes, so you >> don't need a mediating DB layer. > > This is precisely true? However, if we are talking about MMO gaming I > see one potential issue which requires some clarification. Often, the > data availability is important. Player would not forgive you if they > "almost" win the battle but they status rolled back due to server crash > or other failure that impact your "live" data. > > What would be possibilities to achive desired HA for transient data in > Erlang and recover node failures. HA is required for millions of > processes (e.g. process per user, process per ship, etc). > > * Keep copies of same data on two distinct nodes. Basically you spawn > two "ship" management processes on node A and B. However, you as app > developer is responsible to implement state conflict resolution, > rollback, recovery, etc. > > * Use Mnesia, Riak as HA transient storage but this does not differ for > my taste as any other mediating DB layer at some respect. > * Use external DB layer but make asynchronous checkpoint of process > status > > Did I miss something? > > Best Regards, > Dmitry -- Utilisant le logiciel de courrier r?volutionnaire d'Opera : http://www.opera.com/mail/ From ludovic@REDACTED Sun Sep 15 22:44:17 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Sun, 15 Sep 2013 22:44:17 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: Hi Jesper, i don't really understand why you advise me to use a very fast DB, (like voltDB) because your previous point was that if things are kept in state, storing it to the database should only be done from time to time (like just after killing a big boss). I want my models transient (even implemented as regular gen_servers), but i don't want to brutal-kill them : I just ask them gently to terminate and to store their data before. So they have time to do so. Or maybe you advised to not save data but, as precise moments, backup all the live data of the server, thus needing a very fast database ? Thank you, I Hope your mnesia holds it out :) Ludovic Le Fri, 13 Sep 2013 13:37:20 +0200, Jesper Louis Andersen a ?crit: > On Fri, Sep 13, 2013 at 1:01 PM, Dmitry Kolesnikov > wrote: > >> * Use external DB layer but make asynchronous checkpoint of process >> status > > > One way is to sync checkpoint at "big moments" in game. Say you've just > slain Giant Behemoth Kraken Lurker In The Sky (GBKLITS). At this point, > it > makes sense to write your state back. After all, GBKLITS is hard to kill. > In turn, you need a database with a very high write speed, since you will > mostly be checkpointing data back to disk and very rarely read in state. > In > principle, you could just scan a disk_log file at startup, but this is > unfeasible for searching. > > Good stores for this kind of data are either column stores (Kdb, Vertica, > Cassandra), "SSTable"-variants (LevelDB), or certain "NewSQL" stores > (VoltDB comes to mind). Given enough cluster size and enough sharding the > write speed of a VoltDB cluster approached a million writes a second, > albeit for rather simple stores/updates. But this means you can keep up, > even for very big MMO-worlds with this solution. > > Mnesia could be used to hold certain in-memory state which has to be > shared > amongst users. But as a write-heavy store, I would avoid it[0] > > [0] Incidentally, I am currently looking at a mnesia db taking some 7000 > writes per second, but I AM NOT RECOMMENDING YOU DO THAT :P -- Utilisant le logiciel de courrier r?volutionnaire d'Opera : http://www.opera.com/mail/ From essen@REDACTED Sun Sep 15 22:58:16 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Sun, 15 Sep 2013 22:58:16 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: <52361F68.5030208@ninenines.eu> On 09/15/2013 10:30 PM, Ludovic Demblans wrote: > Lo?c, you said : > > """You most likely don't need to store all of this in a database.""" > > Actually, this is a browser game server, the kind of you refresh a few > times in a day to update things (buildings, productions, skills, ...), > all the data must be saved often. And as people aren't connected > continuously, i don't want all my data living in RAM all of the time. > (BTW i should upgrade my hosting, but for now i am quite limited on > RAM) It makes no difference. When the player gets back online, you create your processes and load the state. What happens from that point on depends on the game of course, but for example you do not have to worry about saving the state if nobody else in the world can see what changed. Your player state is likely just a few kilobytes of RAM at most. With today's hosting that means a lot of players even on the smallest servers. (Unless if you use cloud stuff I suppose.) But this is the less interesting part of what I said. What will make you start thinking in Erlang is if you design your game with just processes, with no database and no browser. Just processes. Then you can add the browser and save the data and anything fancy you wish. But first you need to forget MVC ever existed, and start seeing the M in MVC as an actual live application you'll be talking to directly. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From ludovic@REDACTED Sun Sep 15 23:18:05 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Sun, 15 Sep 2013 23:18:05 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: <52361F68.5030208@ninenines.eu> References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> <52361F68.5030208@ninenines.eu> Message-ID: > When the player gets back online, you create your processes and load the > state. What happens from that point on depends on the game of course, > but for example you do not have to worry about saving the state if > nobody else in the world can see what changed. But in this case, i should never save state because all the data people can see come from theese states. But i have to save. > > Your player state is likely just a few kilobytes of RAM at most. With > today's hosting that means a lot of players even on the smallest > servers. (Unless if you use cloud stuff I suppose.) > > But this is the less interesting part of what I said. What will make you > start thinking in Erlang is if you design your game with just processes, > with no database and no browser. Just processes. Then you can add the > browser and save the data and anything fancy you wish. But first you > need to forget MVC ever existed, and start seeing the M in MVC as an > actual live application you'll be talking to directly. > Well ok, I will focus on building game mechanics and processes interactions. That will be way more fun anyway :) Thank you Ludovic From klas.johansson@REDACTED Sun Sep 15 23:42:24 2013 From: klas.johansson@REDACTED (Klas Johansson) Date: Sun, 15 Sep 2013 23:42:24 +0200 Subject: [erlang-questions] dialyzer and #record vs tuple In-Reply-To: <2417658.JFCBhiOC2b@moltowork> References: <3822354.5RfTPR0ESt@moltowork> <51B1A302.40406@cs.ntua.gr> <2417658.JFCBhiOC2b@moltowork> Message-ID: Hi, I've seen a similar situation. There was a module with the same name as a record. The record had two fields. module: x.erl record: -record(x, {name, opts}). This module happened to be a process which was started as a supervisor child with a "StartFunc" like this: X = #x{name=Name, opts=Opts}, ... StartFunc = {x, start_link, [X]}, ... The StartFunc was (of course) also mistaken for the record, since they look the same on the tuple level. /Klas On Fri, Jun 7, 2013 at 11:23 AM, Vincent de Phily < vincent.dephily@REDACTED> wrote: > On Friday 07 June 2013 12:08:18 Kostis Sagonas wrote: > > There is no perfect `solution' to this . > > > > Dialyzer works at a level where records have been expanded into tuples > > and thus records and tuples are indistinguishable between them. > > Furthermore, it makes the assumption that within a module any tuple with > > a record tag and with a size that matches some typed record declaration > > refers to the declared types. So you have two options: > > > > 1. Use different tags for records and tuples e.g. foo_rec vs. foo (as > > you write) > > > > 2. Do not declare types for records that for some reason or another > > you also want to have similar looking tuples (as you write, declaring > > the "union" of the types does not make much sense). > > > > From the two options, I personally very much prefer the first. > > Thanks for confirming this, pity there's no nice way of handling that > problem > but I'll stick with option 1 for now. > > Maybe when frames arrive in erlang it'll fix that issue :) > > -- > Vincent de Phily > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Sep 16 06:21:14 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 16 Sep 2013 08:21:14 +0400 Subject: [erlang-questions] Monitoring In-Reply-To: References: <75296695-39DB-4782-A970-268F80212313@gmail.com> <2DEDBCFA-2114-478C-8335-9AF00A172B18@gmail.com> <740FF892-BBA1-4843-9D77-6A5CC8251BD0@gmail.com> Message-ID: If you want to monitor, I advise always to check not pid, but some http port for 200 response. Your process may be running, but frozen or something else. Better monitor service that it provides. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rc.china@REDACTED Mon Sep 16 07:50:11 2013 From: rc.china@REDACTED (r c) Date: Mon, 16 Sep 2013 05:50:11 +0000 (UTC) Subject: [erlang-questions] Invitation to connect on LinkedIn Message-ID: <1806920947.8987535.1379310611942.JavaMail.app@ela4-bed80.prod> LinkedIn ------------ I'd like to add you to my professional network on LinkedIn. - r r c Software Engineer at Nokia Siemens Networks China Confirm that you know r c: https://www.linkedin.com/e/9bf4ye-hln9xbk4-6u/isd/16576216888/nnnlAuCx/?hs=false&tok=3OPbQiNyPuNBU1 -- You are receiving Invitation to Connect emails. Click to unsubscribe: http://www.linkedin.com/e/9bf4ye-hln9xbk4-6u/XTlnFE8abyJd_kB1UIDLL9FatMerCISLwklnKzCOoQ/goo/erlang-questions%40erlang%2Eorg/20061/I5510475408_1/?hs=false&tok=0lAH6s0WPuNBU1 (c) 2012 LinkedIn Corporation. 2029 Stierlin Ct, Mountain View, CA 94043, USA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Mon Sep 16 09:03:15 2013 From: ivan@REDACTED (Ivan uemlianin) Date: Mon, 16 Sep 2013 08:03:15 +0100 Subject: [erlang-questions] Invitation to connect on LinkedIn In-Reply-To: <1806920947.8987535.1379310611942.JavaMail.app@ela4-bed80.prod> References: <1806920947.8987535.1379310611942.JavaMail.app@ela4-bed80.prod> Message-ID: <06EB36AD-E102-4306-A95B-7B9FE0DEE1F9@llaisdy.com> All of us? -- festina lente On 16 Sep 2013, at 06:50, r c wrote: > > > > > > > > From r c > > Software Engineer at Nokia Siemens Networks > China > > > > > > > > I'd like to add you to my professional network on LinkedIn. > > - r > > > > > Confirm that you know r > > > > You are receiving Invitation to Connect emails. Unsubscribe > ? 2012, LinkedIn Corporation. 2029 Stierlin Ct. Mountain View, CA 94043, USA > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From tino.breddin@REDACTED Mon Sep 16 09:28:03 2013 From: tino.breddin@REDACTED (Tino Breddin) Date: Mon, 16 Sep 2013 09:28:03 +0200 Subject: [erlang-questions] Hidden Memory Hog In-Reply-To: <4E3994F3.1070407@2600hz.com> References: <4E35E5C9.90108@2600hz.com> <4E36CC75.8020409@2600hz.com> <723FDC86-0FC6-48EE-B012-05C57F9ECF25@hates.ms> <4E3994F3.1070407@2600hz.com> Message-ID: Hi James, I'm seeing similar issues on a cluster installation using R15B01. Did you get anywhere and figure out the issue? Cheers, Tino On Wed, Aug 3, 2011 at 8:35 PM, James Aimonetti wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > So I started a VM with the +Mim true flag and wondered if folks can help > narrow down where I should be looking for possible offenders? > > Using this in the shell: > > f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted = > lists:reverse(lists:keysort(3,Mem)). > [{heap,140578906112088,41084720,<0.5051.56>}, > {old_heap,140578980069464,32867800,<0.5051.56>}, > {old_heap,140579035463768,10770152,<0.8797.57>}, > {heap,140578978496600,1571344,<0.8797.57>}, > {heap,140579077124184,971144,<0.25.0>}, > {old_heap,140579095326808,971144,<0.455.0>}, > {old_heap,140579078697048,971144,<0.25.0>}, > {old_heap,140579090284632,600200,<0.404.0>}, > {timer_wheel,140579109355368,524288,undefined}, > {port_tab,140579107569760,466944,undefined}, > {code,140579086084496,326762,<0.25.0>}, > {scheduler_data,140579110159688,287680,undefined}, > {proc_tab,140579109892896,262144,undefined}, > {code,140579070881888,251787,<0.25.0>}, > {code,140579106199176,239377,<0.0.0>}, > {old_heap,140579074121920,229256,<0.396.0>}, > {code,140579082245832,193718,<0.25.0>}, > {code,140579082439600,187723,<0.25.0>}, > {code,140579075601296,183984,<0.25.0>}, > {code,140579085875512,183888,<0.25.0>}, > |...] > > The first four lines are the shell and a process spawned by the shell. > > If I run: > > lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted). > 112277446 > > Is that number valid? Should I be looking at lines with type heap or > old_heap only? > > The memory output from /proc/PID/status: > > VmPeak: 728864 kB > VmSize: 335188 kB > VmLck: 0 kB > VmHWM: 421760 kB > VmRSS: 250352 kB > VmData: 306400 kB > VmStk: 136 kB > VmExe: 1800 kB > VmLib: 3748 kB > VmPTE: 680 kB > VmSwap: 1520 kB > > Any direction for where to continue searching? > > Thanks for the help so far, > > James > > On 08/01/2011 10:32 AM, Mihai Balea wrote: > > That looks pretty benign. I suspect the OS reported memory usage for > the beam process is vastly higher, correct? > > Do you run any native code? NIFs, linked-in drivers? Maybe there's a > leak in there. > > > > On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote: > > > > No, no outliers. > > > > 1> erlang:memory(). > > [{total,29393336}, > > {processes,18646408}, > > {processes_used,18560824}, > > {system,10746928}, > > {atom,658865}, > > {atom_used,652857}, > > {binary,1059104}, > > {code,6608997}, > > {ets,434848}] > > > > Unless I'm reading that output incorrectly? > > > > On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote: > >>>> What does erlang:memory() show? Binaries, processes or ets uses that > >>>> much memory? > >>>> > >>>> 2011/8/1, James Aimonetti : > >>>> List, > >>>> > >>>> I'm at my wits end (they're short, no doubt, but still). I have a VM > >>>> running with several OTP applications that we've written. We see, over > >>>> the course of the day, memory consumption go up until its taking ~30% > >>>> system memory and 300-500MB of swap. Iterating over all processes and > >>>> checking their total_heap_size, no single process was greater than ~1 > >>>> MB, with ~300 processes runnning. Forcing garbage collection on all > >>>> processes released maybe 15MB of system memory. > >>>> > >>>> erlang:memory() lists total memory around 18-20MB; ets tables were all > >>>> minimal, no dets tables. No processes has a message_queue_len > 0. No > >>>> processes that we maintain use the process dictionary. > >>>> > >>>> I started a second VM, migrating our OTP apps to the second VM, one > at a > >>>> time, hoping that stopping one would cause a release of lots of memory > >>>> on the first VM. No such luck; it released a couple more MBs but > still a > >>>> large presence and no change in swap usage. I then stopped everything > I > >>>> could until only stdlib and kernel were running on the first VM, but > to > >>>> no avail. > >>>> > >>>> Each application has 1-3 long running gen_servers that listen on an > AMQP > >>>> queue and spawn worker processes to handle messages. They carry almost > >>>> no state. Some of the handlers will read binary blobs, open a socket, > >>>> wait for a connection, then transmit the blobs, closing the socket and > >>>> dying afterwards (they timeout after 5 minutes if no connection is > >>>> attempted, and are in a supervision tree so I know there aren't a mass > >>>> of them lying around). The blobs are between 20KB and 2MB. The rest of > >>>> the workers are similarly ephemeral. > >>>> > >>>> What other options are there to find where memory has been allocated? > We > >>>> use binaries almost exclusively for strings, don't construct large > >>>> lists. I've poured over the efficiency guide looking for ideas of > where > >>>> we've gone astray but am coming up blank so far. > >>>> > >>>> The system is idle; the only processes should be (and are, as far as I > >>>> can tell) the gen_servers meant to run for a long time. > >>>> > >>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4] > >>>> [async-threads:8] [kernel-poll:true] > >>>> > >>>> Any ideas are welcome, > >>>> > >>>> James > >>>> > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > >>>>> > >>>> _______________________________________________ > >>>> erlang-questions mailing list > >>>> erlang-questions@REDACTED > >>>> http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > - -- > James Aimonetti > Distributed Systems Engineer / DJ MC_ > > 2600hz | http://2600hz.com > sip:james@REDACTED > tel: 415.886.7905 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJOOZTzAAoJENc77s1OYoGgNSQH+QGcD5WDBSJp+z6RCjAqcRS+ > TM0fxxLolnWN9wdeje/v91ZA+G+usz/LbR0LhctwD7gJk29Ni56CO2cNP1osw5/c > VWMSvxllDqn6ZmqAGl+h0lpYH9ZLWiZNZ1f4BjihHLjyglhdAKdRYfbQSzZsilQA > Zu2AcfBvSQ564ByOEnU4ZEqrVaY9+EnI7WOjfd9HMHG2GBPgpDJauUGWfF21Qglf > w7FESB/Gv6TMUHgdBEpBXmi1A1XgJFlQh8kK0xz6rNEp7GbHHrzk9Nl/644HzwGJ > jsMpmBf6r+NXDwlR4Brd7eIdoIcnJ7DFIGEfUHFS73zkFAGxQdOoDCDGgnTSwSc= > =jp4S > -----END PGP SIGNATURE----- > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Sep 16 13:24:47 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 16 Sep 2013 13:24:47 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: On Sun, Sep 15, 2013 at 10:44 PM, Ludovic Demblans wrote: > Hi Jesper, i don't really understand why you advise me to use a very > fast DB, (like voltDB) because your previous point was that if things > are kept in state, storing it to the database should only be done from > time to time (like just after killing a big boss). > > I want my models transient (even implemented as regular gen_servers), > but i don't want to brutal-kill them : I just ask them gently to > terminate and to store their data before. So they have time to do so. > I am just trying to say that this model will scale quite well. Since you will have relatively few write-backs to stable storage. Also note you will have very few reads compared to writes in this model. So write-centric database systems can also do well in this scheme. Cassandra comes to mind, but I don't like it too much. I would probably just have a timeout on the gen_servers and if they are not being used for some grace-period, they write their state back onto stable storage and terminates. If a player then comes back, you can rejuvenate the process. Eventually by replaying all moves from the start. This means the capacity of the machine is limited by number of players currently playing, which is probably a much lower number than the amount of players in the database systems. -- J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tino.breddin@REDACTED Mon Sep 16 13:40:17 2013 From: tino.breddin@REDACTED (Tino Breddin) Date: Mon, 16 Sep 2013 13:40:17 +0200 Subject: [erlang-questions] Problem with Beam process not freeing memory Message-ID: Hi list, I'm experiencing issues with a couple of Beam nodes where I see a huge gap between the node's reported memory usage and the underlying Linux Kernel's view. This is using R15B01. As a start an application in such a node stores a lot of tuples (containing atoms and binary data) in ETS tables. That proceeds until a point where memory usage is 70% (6GB) of the available memory. At that point erlang:memory() and top (or /proc/PID/status) agree roughly on the memory usage. Then an internal cleanup task is performed, which clears obsolete records from the ETS tables. Afterwards, erlang:memory() reports an expected low value of roughly 60MB memory usage. (This includes binary data). However, the kernel still reports the high memory usage values (both VmRss and VmTotal) for the node. The kernel's usage view will stay stable until the ETS tables are filled to a point where the real memory usage exceeds the kernel's view, then the kernel reported usage will grow as well. Now having checked the node in some details I'm wondering what causes this difference between the BEAM's view and the Kernel's view on memory usage. I have 2 ideas which I'm checking right now. (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because the original dispatcher process which it was passed through before being stored in an ETS table is still alive. Thus there is still some reference to it? However, this would not explain why erlang:memory() reports a very low value for used memory for binaries. (2) low-level memory leak: Some driver or NIF leaking memory, which would obviously not be reported by erlang:memory(). However, then it surprises me that the Kernel's view stays stable while the BEAM's actual memory usage is still below the Kernel's view. It should be continuously growing in this case imho. I'd appreciate if anyone has some more insight or experience with such a behaviour, while I'm further digging into this. Cheers, Tino -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn@REDACTED Mon Sep 16 13:51:24 2013 From: masklinn@REDACTED (Masklinn) Date: Mon, 16 Sep 2013 13:51:24 +0200 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: References: Message-ID: <5B7BC4F6-8A3D-4D4A-9DA1-2C8976BD1396@masklinn.net> On 2013-09-16, at 13:40 , Tino Breddin wrote: > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because > the original dispatcher process which it was passed through before being > stored in an ETS table is still alive. Thus there is still some reference > to it? However, this would not explain why erlang:memory() reports a very > low value for used memory for binaries. > > (2) low-level memory leak: Some driver or NIF leaking memory, which would > obviously not be reported by erlang:memory(). However, then it surprises me > that the Kernel's view stays stable while the BEAM's actual memory usage is > still below the Kernel's view. It should be continuously growing in this > case imho. > > I'd appreciate if anyone has some more insight or experience with such a > behaviour, while I'm further digging into this. I have no info, but I'd suggest #3: the runtime does not release memory back to the OS under the assumption that the high-water mark probably isn't a unique event; or there is some delay between the moment where memory is released to the runtime and where memory is released by the runtime to the kernel (IIRC CPython has such a behavior with multi-stage allocations, internal pools and freelists[-1] and that's a fairly recent evolution, before 2006 CPython *never* released memory to the kernel[0]). [-1] this means memory released by a Python program may or may not be immediately free()'d back to the kernel, and due to release being predicated upon arena emptying[0] and ignoring freelists it will not happen for all internally released memory, at least not without an explicit full GC and maybe even with that [0] http://bugs.python.org/issue1123430 From pablo.platt@REDACTED Mon Sep 16 14:10:48 2013 From: pablo.platt@REDACTED (pablo platt) Date: Mon, 16 Sep 2013 15:10:48 +0300 Subject: [erlang-questions] Not getting exit_status when using open_port with docker Message-ID: Hi, I'm calling "docker run" from Erlang. http://docs.docker.io/en/latest/commandline/command/run/ I'm using open_port instead of os:cmd to be able to limit the execution time. I'm getting the output of the command but the do_read just keep going and I'm not able to get an exit status from the command. When running "echo 'test'" I'm getting {exit_status, 0}. When running the "docker run" command in the shell, I'm getting the output and it exits. When running any other docker command like "docker ps" I'm getting the exit signal. Is this a problem with how I'm using open_port or something wrong with the "docker run" command? -module(test). -export([run/0]). run() -> Cmd = "sudo docker run ubuntu echo \"test\"", Port = open_port({spawn, Cmd}, [binary, exit_status]), do_read(Port, <<>>). do_read(Port, Data) -> receive {Port,{data, NewData}} -> io:format("data ~p~n", [NewData]), do_read(Port, <>); {Port,{exit_status, 0}} -> io:format("exit~n"), case Data of <<"">> -> <<"ok">>; _ -> Data end; E -> io:format("Something: ~p~n", [E]) end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim@REDACTED Mon Sep 16 14:26:28 2013 From: maxim@REDACTED (Maxim Sokhatsky) Date: Mon, 16 Sep 2013 15:26:28 +0300 Subject: [erlang-questions] Not getting exit_status when using open_port with docker In-Reply-To: References: Message-ID: First of all you do it wrong just because of two things :) The first thing is erlsh library:?https://github.com/proger/erlsh And the second is erldocker library:?https://github.com/proger/erldocker Moreover if you are interesting in your own Erlang PaaS you could read the sources of the first Erlang PaaS: https://github.com/voxoz Hope this save your time :) /maxim ________________________________ > Date: Mon, 16 Sep 2013 15:10:48 +0300 > From: pablo.platt@REDACTED > To: erlang-questions@REDACTED > Subject: [erlang-questions] Not getting exit_status when using > open_port with docker > > Hi, > > I'm calling "docker run" from Erlang. > http://docs.docker.io/en/latest/commandline/command/run/ > I'm using open_port instead of os:cmd to be able to limit the execution time. > I'm getting the output of the command > but the do_read just keep going and I'm not able to get an exit status > from the command. > > When running "echo 'test'" I'm getting {exit_status, 0}. > When running the "docker run" command in the shell, I'm getting the > output and it exits. > When running any other docker command like "docker ps" I'm getting the > exit signal. > > Is this a problem with how I'm using open_port or something wrong with > the "docker run" command? > > -module(test). > > -export([run/0]). > > run() -> > Cmd = "sudo docker run ubuntu echo \"test\"", > Port = open_port({spawn, Cmd}, [binary, exit_status]), > do_read(Port, <<>>). > > do_read(Port, Data) -> > receive > {Port,{data, NewData}} -> > io:format("data ~p~n", [NewData]), > do_read(Port, <>); > {Port,{exit_status, 0}} -> > io:format("exit~n"), > case Data of > <<"">> -> <<"ok">>; > _ -> Data > end; > E -> > io:format("Something: ~p~n", [E]) > end. > > > _______________________________________________ erlang-questions > mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From pablo.platt@REDACTED Mon Sep 16 15:30:51 2013 From: pablo.platt@REDACTED (pablo platt) Date: Mon, 16 Sep 2013 16:30:51 +0300 Subject: [erlang-questions] Not getting exit_status when using open_port with docker In-Reply-To: References: Message-ID: I looked at erlsh. It uses open_port with a loop like I'm using in my code so my question still stands. Why I'm not getting the exit message from the "docker run" command? erldocker looks nice but I couldn't find the equivalent of "docker run" without starting, attaching, detaching and destroying a container. I'm not sure how to do it manually because I need to know when the container exits which docker run does. On Mon, Sep 16, 2013 at 3:26 PM, Maxim Sokhatsky wrote: > First of all you do it wrong just because of two things :) > > The first thing is erlsh library: https://github.com/proger/erlsh > And the second is erldocker library: https://github.com/proger/erldocker > > Moreover if you are interesting in your own Erlang PaaS > you could read the sources of the first Erlang PaaS: > > https://github.com/voxoz > > Hope this save your time :) > > /maxim > > ________________________________ > > Date: Mon, 16 Sep 2013 15:10:48 +0300 > > From: pablo.platt@REDACTED > > To: erlang-questions@REDACTED > > Subject: [erlang-questions] Not getting exit_status when using > > open_port with docker > > > > Hi, > > > > I'm calling "docker run" from Erlang. > > http://docs.docker.io/en/latest/commandline/command/run/ > > I'm using open_port instead of os:cmd to be able to limit the execution > time. > > I'm getting the output of the command > > but the do_read just keep going and I'm not able to get an exit status > > from the command. > > > > When running "echo 'test'" I'm getting {exit_status, 0}. > > When running the "docker run" command in the shell, I'm getting the > > output and it exits. > > When running any other docker command like "docker ps" I'm getting the > > exit signal. > > > > Is this a problem with how I'm using open_port or something wrong with > > the "docker run" command? > > > > -module(test). > > > > -export([run/0]). > > > > run() -> > > Cmd = "sudo docker run ubuntu echo \"test\"", > > Port = open_port({spawn, Cmd}, [binary, exit_status]), > > do_read(Port, <<>>). > > > > do_read(Port, Data) -> > > receive > > {Port,{data, NewData}} -> > > io:format("data ~p~n", [NewData]), > > do_read(Port, <>); > > {Port,{exit_status, 0}} -> > > io:format("exit~n"), > > case Data of > > <<"">> -> <<"ok">>; > > _ -> Data > > end; > > E -> > > io:format("Something: ~p~n", [E]) > > end. > > > > > > _______________________________________________ erlang-questions > > mailing list erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From james@REDACTED Mon Sep 16 16:07:40 2013 From: james@REDACTED (James Aimonetti) Date: Mon, 16 Sep 2013 07:07:40 -0700 Subject: [erlang-questions] Hidden Memory Hog In-Reply-To: References: <4E35E5C9.90108@2600hz.com> <4E36CC75.8020409@2600hz.com> <723FDC86-0FC6-48EE-B012-05C57F9ECF25@hates.ms> <4E3994F3.1070407@2600hz.com> Message-ID: <523710AC.9080703@2600hz.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 It ended up being a port driver that wasn't freeing memory in the C code (an older version of the erlang-syslog lib). But this was two years ago, so details are fuzzy for me :) James On 09/16/2013 12:28 AM, Tino Breddin wrote: > Hi James, > > I'm seeing similar issues on a cluster installation using R15B01. > Did you get anywhere and figure out the issue? > > Cheers, Tino > > > On Wed, Aug 3, 2011 at 8:35 PM, James Aimonetti > wrote: > > So I started a VM with the +Mim true flag and wondered if folks can > help narrow down where I should be looking for possible offenders? > > Using this in the shell: > > f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted = > lists:reverse(lists:keysort(3,Mem)). > [{heap,140578906112088,41084720,<0.5051.56>}, > {old_heap,140578980069464,32867800,<0.5051.56>}, > {old_heap,140579035463768,10770152,<0.8797.57>}, > {heap,140578978496600,1571344,<0.8797.57>}, > {heap,140579077124184,971144,<0.25.0>}, > {old_heap,140579095326808,971144,<0.455.0>}, > {old_heap,140579078697048,971144,<0.25.0>}, > {old_heap,140579090284632,600200,<0.404.0>}, > {timer_wheel,140579109355368,524288,undefined}, > {port_tab,140579107569760,466944,undefined}, > {code,140579086084496,326762,<0.25.0>}, > {scheduler_data,140579110159688,287680,undefined}, > {proc_tab,140579109892896,262144,undefined}, > {code,140579070881888,251787,<0.25.0>}, > {code,140579106199176,239377,<0.0.0>}, > {old_heap,140579074121920,229256,<0.396.0>}, > {code,140579082245832,193718,<0.25.0>}, > {code,140579082439600,187723,<0.25.0>}, > {code,140579075601296,183984,<0.25.0>}, > {code,140579085875512,183888,<0.25.0>}, |...] > > The first four lines are the shell and a process spawned by the > shell. > > If I run: > > lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted). > 112277446 > > Is that number valid? Should I be looking at lines with type heap > or old_heap only? > > The memory output from /proc/PID/status: > > VmPeak: 728864 kB VmSize: 335188 kB VmLck: 0 kB VmHWM: > 421760 kB VmRSS: 250352 kB VmData: 306400 kB VmStk: 136 > kB VmExe: 1800 kB VmLib: 3748 kB VmPTE: 680 kB > VmSwap: 1520 kB > > Any direction for where to continue searching? > > Thanks for the help so far, > > James > > On 08/01/2011 10:32 AM, Mihai Balea wrote: >>>> That looks pretty benign. I suspect the OS reported memory >>>> usage for > the beam process is vastly higher, correct? >>>> Do you run any native code? NIFs, linked-in drivers? Maybe >>>> there's a > leak in there. >>>> >>>> On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote: >>>> >>>> No, no outliers. >>>> >>>> 1> erlang:memory(). [{total,29393336}, {processes,18646408}, >>>> {processes_used,18560824}, {system,10746928}, {atom,658865}, >>>> {atom_used,652857}, {binary,1059104}, {code,6608997}, >>>> {ets,434848}] >>>> >>>> Unless I'm reading that output incorrectly? >>>> >>>> On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote: >>>>>>> What does erlang:memory() show? Binaries, processes or >>>>>>> ets uses that much memory? >>>>>>> >>>>>>> 2011/8/1, James Aimonetti : List, >>>>>>> >>>>>>> I'm at my wits end (they're short, no doubt, but >>>>>>> still). I have a VM running with several OTP >>>>>>> applications that we've written. We see, over the >>>>>>> course of the day, memory consumption go up until its >>>>>>> taking ~30% system memory and 300-500MB of swap. >>>>>>> Iterating over all processes and checking their >>>>>>> total_heap_size, no single process was greater than ~1 >>>>>>> MB, with ~300 processes runnning. Forcing garbage >>>>>>> collection on all processes released maybe 15MB of >>>>>>> system memory. >>>>>>> >>>>>>> erlang:memory() lists total memory around 18-20MB; ets >>>>>>> tables were all minimal, no dets tables. No processes >>>>>>> has a message_queue_len > 0. No processes that we >>>>>>> maintain use the process dictionary. >>>>>>> >>>>>>> I started a second VM, migrating our OTP apps to the >>>>>>> second VM, one > at a >>>>>>> time, hoping that stopping one would cause a release of >>>>>>> lots of memory on the first VM. No such luck; it >>>>>>> released a couple more MBs but > still a >>>>>>> large presence and no change in swap usage. I then >>>>>>> stopped everything > I >>>>>>> could until only stdlib and kernel were running on the >>>>>>> first VM, but > to >>>>>>> no avail. >>>>>>> >>>>>>> Each application has 1-3 long running gen_servers that >>>>>>> listen on an > AMQP >>>>>>> queue and spawn worker processes to handle messages. >>>>>>> They carry almost no state. Some of the handlers will >>>>>>> read binary blobs, open a socket, wait for a >>>>>>> connection, then transmit the blobs, closing the socket >>>>>>> and dying afterwards (they timeout after 5 minutes if >>>>>>> no connection is attempted, and are in a supervision >>>>>>> tree so I know there aren't a mass of them lying >>>>>>> around). The blobs are between 20KB and 2MB. The rest >>>>>>> of the workers are similarly ephemeral. >>>>>>> >>>>>>> What other options are there to find where memory has >>>>>>> been allocated? > We >>>>>>> use binaries almost exclusively for strings, don't >>>>>>> construct large lists. I've poured over the efficiency >>>>>>> guide looking for ideas of > where >>>>>>> we've gone astray but am coming up blank so far. >>>>>>> >>>>>>> The system is idle; the only processes should be (and >>>>>>> are, as far as I can tell) the gen_servers meant to run >>>>>>> for a long time. >>>>>>> >>>>>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] >>>>>>> [rq:4] [async-threads:8] [kernel-poll:true] >>>>>>> >>>>>>> Any ideas are welcome, >>>>>>> >>>>>>> James >>>>>>> >>>> _______________________________________________ >>>> erlang-questions mailing list erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> erlang-questions mailing list >>>>>>> erlang-questions@REDACTED >>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> > _______________________________________________ erlang-questions > mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > >> _______________________________________________ erlang-questions >> mailing list erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > - -- James Aimonetti Lead Systems Architect "I thought I fixed that" 2600Hz | http://2600hz.com sip:james@REDACTED tel:415.886.7905 irc:mc_ @ freenode -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlI3EKwACgkQ54NxaUq7OmASeAEAqxRKZyBrbsWUPBCezGiaQeRb PTDtNXnCEM5lq2gnBAcBALcISObSHxtSVHN/xI/sZ8GDzaUHIrddoZvy2AAG1Stl =Jxd5 -----END PGP SIGNATURE----- From g@REDACTED Mon Sep 16 16:23:02 2013 From: g@REDACTED (Garrett Smith) Date: Mon, 16 Sep 2013 09:23:02 -0500 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: On Mon, Sep 16, 2013 at 6:24 AM, Jesper Louis Andersen wrote: > > On Sun, Sep 15, 2013 at 10:44 PM, Ludovic Demblans > wrote: >> >> Hi Jesper, i don't really understand why you advise me to use a very >> fast DB, (like voltDB) because your previous point was that if things >> are kept in state, storing it to the database should only be done from >> time to time (like just after killing a big boss). >> >> I want my models transient (even implemented as regular gen_servers), >> but i don't want to brutal-kill them : I just ask them gently to >> terminate and to store their data before. So they have time to do so. It's hard to guarantee a graceful shutdown of processes, servers, etc. so if you can avoid relying on a shutdown message, it's a good idea. E.g. Jesper's suggestion of write backs on idle (or interval) is the right thinking IMO. > I am just trying to say that this model will scale quite well. Since you > will have relatively few write-backs to stable storage. Also note you will > have very few reads compared to writes in this model. So write-centric > database systems can also do well in this scheme. Cassandra comes to mind, > but I don't like it too much. One of the nice things about the process oriented approach that Erlang encourages is you can start thinking about federating data storage -- something that's closer to 1:1 with your process state. E.g. if a process is managing a few MB of state (a lot for a single user in most applications) it's trivial to push this data around a network. I've used something as simple as writing to a local dets file and then pushing that over a network for backup / restore. Heck, gziping term_to_binary(State) is probably enough in many cases. Smart people will probably criticize this as "rolling your own distributed database" -- which is fair. But I like the idea of making each process responsible for its state, both in loading and persisting [1]. Whether you use something naive like a local file and push to a remote location or something like Riak - it's the same idea. Garrett [1] The actual responsibility of loading/saving might be best implemented by *another* process -- but the idea here is that each process is responsible for its own state, whether it does so itself or uses a helper. From tino.breddin@REDACTED Mon Sep 16 16:53:07 2013 From: tino.breddin@REDACTED (Tino Breddin) Date: Mon, 16 Sep 2013 16:53:07 +0200 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: <5B7BC4F6-8A3D-4D4A-9DA1-2C8976BD1396@masklinn.net> References: <5B7BC4F6-8A3D-4D4A-9DA1-2C8976BD1396@masklinn.net> Message-ID: Hi, I don't think the runtime (BEAM) is at fault here since the stable memory usage reported by the kernel can go on for a week, depending how fast the real memory usage catches up. Tino On Mon, Sep 16, 2013 at 1:51 PM, Masklinn wrote: > On 2013-09-16, at 13:40 , Tino Breddin wrote: > > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because > > the original dispatcher process which it was passed through before being > > stored in an ETS table is still alive. Thus there is still some reference > > to it? However, this would not explain why erlang:memory() reports a very > > low value for used memory for binaries. > > > > (2) low-level memory leak: Some driver or NIF leaking memory, which would > > obviously not be reported by erlang:memory(). However, then it surprises > me > > that the Kernel's view stays stable while the BEAM's actual memory usage > is > > still below the Kernel's view. It should be continuously growing in this > > case imho. > > > > I'd appreciate if anyone has some more insight or experience with such a > > behaviour, while I'm further digging into this. > > I have no info, but I'd suggest #3: the runtime does not release memory > back to the OS under the assumption that the high-water mark probably > isn't a unique event; or there is some delay between the moment where > memory is released to the runtime and where memory is released by the > runtime to the kernel (IIRC CPython has such a behavior with multi-stage > allocations, internal pools and freelists[-1] and that's a fairly recent > evolution, before 2006 CPython *never* released memory to the kernel[0]). > > [-1] this means memory released by a Python program may or may not be > immediately free()'d back to the kernel, and due to release being > predicated upon arena emptying[0] and ignoring freelists it will > not happen for all internally released memory, at least not without > an explicit full GC and maybe even with that > [0] http://bugs.python.org/issue1123430 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo@REDACTED Mon Sep 16 15:16:34 2013 From: hugo@REDACTED (Hugo Mills) Date: Mon, 16 Sep 2013 14:16:34 +0100 Subject: [erlang-questions] file descriptors and ioctls Message-ID: <20130916131634.GA22835@carfax.org.uk> Is there a way of either turning a POSIX file descriptor into an erlang (raw) file descriptor or vice-versa? Alternatively, is there an ioctl interface for erlang that will allow me to call ioctls on an open file? I'm trying to talk to some hardware (it's a DVB receiver) -- most of the time, I can just read from an open file (e.g. device node) to get data from it, but occasionally I need to be able to call an ioctl to change its configuration. The most attractive approach I can see here is to write a NIF which will perform the ioctl(s) on an already-opened file. Clearly, if I use file:open(..., [raw]), it's dealing with something close to a traditional POSIX filehandle -- but the returned file:fd() is opaque in the latest documentation, so it's not obvious how, if at all, I can go from that to a POSIX fd reliably and with a reasonable degree of future-proofing. Going the other way (opening the file in the NIF and passing back a POSIX fd, to convert to a raw file:fd()) is also not clear to me whether it can be done. I note that open_port({fd, In, Out}) exists, but it looks like it probably shouldn't be used, from the slightly off-putting description in the manual. The final option, which will definitely work, is to write the whole thing in C -- open the file, handle the ioctls and read the data -- and run it as a port, but that puts most of that part of the application in C instead of erlang, and I'd quite like to avoid that, given that my main goal here is to learn how to write erlang programs. :) Can anyone offer me any advice or comment on the above options and observations? Thanks, Hugo. -- === Hugo Mills: hugo@REDACTED carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Attempted murder, now honestly, what is that? Do they give a --- Nobel Prize for attempted chemistry? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 828 bytes Desc: Digital signature URL: From max.lapshin@REDACTED Mon Sep 16 17:16:55 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 16 Sep 2013 19:16:55 +0400 Subject: [erlang-questions] file descriptors and ioctls In-Reply-To: <20130916131634.GA22835@carfax.org.uk> References: <20130916131634.GA22835@carfax.org.uk> Message-ID: I think that you are making something like this: https://github.com/erlyvideo/dvbs2 but better What for do you want DVB fd as an erlang file? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo@REDACTED Mon Sep 16 18:07:56 2013 From: hugo@REDACTED (Hugo Mills) Date: Mon, 16 Sep 2013 17:07:56 +0100 Subject: [erlang-questions] file descriptors and ioctls In-Reply-To: References: <20130916131634.GA22835@carfax.org.uk> Message-ID: <20130916160756.GC22835@carfax.org.uk> On Mon, Sep 16, 2013 at 07:16:55PM +0400, Max Lapshin wrote: > I think that you are making something like this: > https://github.com/erlyvideo/dvbs2 but better Something like that, indeed. I was going to start with DVB-T rather than DVB-S2, since that's what I've got, but that's a trivial change. I don't (yet) understand what erlang drivers actually do. Looks like I've got some reading to do this evening. > What for do you want DVB fd as an erlang file? So I can read from it in erlang without proxying the reads through some other language -- if I didn't need to configure the tuner, I could just open the device and read data from it, so that felt like a good approach. After that, the problem reduces to working out how to do the tuning... which then led to my original question. Hugo. -- === Hugo Mills: hugo@REDACTED carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Mixing mathematics and alcohol is dangerous. Don't --- drink and derive. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 828 bytes Desc: Digital signature URL: From max.lapshin@REDACTED Mon Sep 16 18:14:40 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 16 Sep 2013 20:14:40 +0400 Subject: [erlang-questions] file descriptors and ioctls In-Reply-To: <20130916160756.GC22835@carfax.org.uk> References: <20130916131634.GA22835@carfax.org.uk> <20130916160756.GC22835@carfax.org.uk> Message-ID: If you have FTA channels, you can take my code and adapt it for your needs. If you need, I can add README If you have scrambled channels, than forget about erlang and take astra or getstream_a84 software and use them. And you definitely need to read about erlang drivers. They are the way to get data from non-standard source into erlang system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nem@REDACTED Mon Sep 16 20:02:12 2013 From: nem@REDACTED (Geoff Cant) Date: Mon, 16 Sep 2013 11:02:12 -0700 Subject: [erlang-questions] file descriptors and ioctls In-Reply-To: <20130916131634.GA22835@carfax.org.uk> References: <20130916131634.GA22835@carfax.org.uk> Message-ID: <0EA0C652-899A-49DF-8BDE-586206B05292@erlang.geek.nz> You can open existing file descriptors as an erlang port by either doing erlang:open_port({fd, InFD, OutFD}, [...]), or you could use something like msantos' procket library ( https://github.com/msantos/procket ). Procket is probably more appropriate as it has ioctl support. Cheers, -- Geoff On 2013-09-16, at 06:16 , Hugo Mills wrote: > Is there a way of either turning a POSIX file descriptor into an > erlang (raw) file descriptor or vice-versa? Alternatively, is there an > ioctl interface for erlang that will allow me to call ioctls on an > open file? > > I'm trying to talk to some hardware (it's a DVB receiver) -- most > of the time, I can just read from an open file (e.g. device node) to > get data from it, but occasionally I need to be able to call an ioctl > to change its configuration. The most attractive approach I can see > here is to write a NIF which will perform the ioctl(s) on an > already-opened file. > > Clearly, if I use file:open(..., [raw]), it's dealing with > something close to a traditional POSIX filehandle -- but the returned > file:fd() is opaque in the latest documentation, so it's not obvious > how, if at all, I can go from that to a POSIX fd reliably and with a > reasonable degree of future-proofing. > > Going the other way (opening the file in the NIF and passing back a > POSIX fd, to convert to a raw file:fd()) is also not clear to me > whether it can be done. > > I note that open_port({fd, In, Out}) exists, but it looks like it > probably shouldn't be used, from the slightly off-putting description > in the manual. > > The final option, which will definitely work, is to write the whole > thing in C -- open the file, handle the ioctls and read the data -- > and run it as a port, but that puts most of that part of the > application in C instead of erlang, and I'd quite like to avoid that, > given that my main goal here is to learn how to write erlang programs. :) > > Can anyone offer me any advice or comment on the above options and > observations? > > Thanks, > Hugo. From max.lapshin@REDACTED Mon Sep 16 20:08:23 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 16 Sep 2013 22:08:23 +0400 Subject: [erlang-questions] file descriptors and ioctls In-Reply-To: <0EA0C652-899A-49DF-8BDE-586206B05292@erlang.geek.nz> References: <20130916131634.GA22835@carfax.org.uk> <0EA0C652-899A-49DF-8BDE-586206B05292@erlang.geek.nz> Message-ID: I think that it may be a bad idea to try to fill such structure like struct dvb_frontend_info fe_info; dynamically via erlang library. Also don't forget about very complicated reading DVB data in linux: https://github.com/erlyvideo/dvbs2/blob/master/c_src/dvbs2.c#L280 -------------- next part -------------- An HTML attachment was scrubbed... URL: From overminddl1@REDACTED Mon Sep 16 22:08:07 2013 From: overminddl1@REDACTED (OvermindDL1) Date: Mon, 16 Sep 2013 14:08:07 -0600 Subject: [erlang-questions] Not getting exit_status when using open_port with docker In-Reply-To: References: Message-ID: Look at erlexec instead. open_port is not designed for lifetime management like that. On Sep 16, 2013 7:31 AM, "pablo platt" wrote: > I looked at erlsh. It uses open_port with a loop like I'm using in my code > so my question still stands. > Why I'm not getting the exit message from the "docker run" command? > > erldocker looks nice but I couldn't find the equivalent of "docker run" > without starting, attaching, detaching and destroying a container. > I'm not sure how to do it manually because I need to know when the > container exits which docker run does. > > > On Mon, Sep 16, 2013 at 3:26 PM, Maxim Sokhatsky wrote: > >> First of all you do it wrong just because of two things :) >> >> The first thing is erlsh library: https://github.com/proger/erlsh >> And the second is erldocker library: https://github.com/proger/erldocker >> >> Moreover if you are interesting in your own Erlang PaaS >> you could read the sources of the first Erlang PaaS: >> >> https://github.com/voxoz >> >> Hope this save your time :) >> >> /maxim >> >> ________________________________ >> > Date: Mon, 16 Sep 2013 15:10:48 +0300 >> > From: pablo.platt@REDACTED >> > To: erlang-questions@REDACTED >> > Subject: [erlang-questions] Not getting exit_status when using >> > open_port with docker >> > >> > Hi, >> > >> > I'm calling "docker run" from Erlang. >> > http://docs.docker.io/en/latest/commandline/command/run/ >> > I'm using open_port instead of os:cmd to be able to limit the execution >> time. >> > I'm getting the output of the command >> > but the do_read just keep going and I'm not able to get an exit status >> > from the command. >> > >> > When running "echo 'test'" I'm getting {exit_status, 0}. >> > When running the "docker run" command in the shell, I'm getting the >> > output and it exits. >> > When running any other docker command like "docker ps" I'm getting the >> > exit signal. >> > >> > Is this a problem with how I'm using open_port or something wrong with >> > the "docker run" command? >> > >> > -module(test). >> > >> > -export([run/0]). >> > >> > run() -> >> > Cmd = "sudo docker run ubuntu echo \"test\"", >> > Port = open_port({spawn, Cmd}, [binary, exit_status]), >> > do_read(Port, <<>>). >> > >> > do_read(Port, Data) -> >> > receive >> > {Port,{data, NewData}} -> >> > io:format("data ~p~n", [NewData]), >> > do_read(Port, <>); >> > {Port,{exit_status, 0}} -> >> > io:format("exit~n"), >> > case Data of >> > <<"">> -> <<"ok">>; >> > _ -> Data >> > end; >> > E -> >> > io:format("Something: ~p~n", [E]) >> > end. >> > >> > >> > _______________________________________________ erlang-questions >> > mailing list erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.simonsson@REDACTED Mon Sep 16 22:23:57 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Mon, 16 Sep 2013 22:23:57 +0200 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: References: Message-ID: Could be the memory previously allocated by the BEAM is still counted as VmRss after it has freed the memory back to Linux. Comparing /proc/meminfo before and after the internal cleanup task in your Erlang app has run might give some ideas. Also, what happens with reported values after manually dropping the page cache [1]? Cheers, Gustav [?] http://linux-mm.org/Drop_Caches On Mon, Sep 16, 2013 at 1:40 PM, Tino Breddin wrote: > Hi list, > > I'm experiencing issues with a couple of Beam nodes where I see a huge gap > between the node's reported memory usage and the underlying Linux Kernel's > view. > > This is using R15B01. > > As a start an application in such a node stores a lot of tuples > (containing atoms and binary data) in ETS tables. That proceeds until a > point where memory usage is 70% (6GB) of the available memory. At that > point erlang:memory() and top (or /proc/PID/status) agree roughly on the > memory usage. Then an internal cleanup task is performed, which clears > obsolete records from the ETS tables. Afterwards, erlang:memory() reports > an expected low value of roughly 60MB memory usage. (This includes binary > data). However, the kernel still reports the high memory usage values (both > VmRss and VmTotal) for the node. The kernel's usage view will stay stable > until the ETS tables are filled to a point where the real memory usage > exceeds the kernel's view, then the kernel reported usage will grow as well. > > Now having checked the node in some details I'm wondering what causes this > difference between the BEAM's view and the Kernel's view on memory usage. I > have 2 ideas which I'm checking right now. > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because > the original dispatcher process which it was passed through before being > stored in an ETS table is still alive. Thus there is still some reference > to it? However, this would not explain why erlang:memory() reports a very > low value for used memory for binaries. > > (2) low-level memory leak: Some driver or NIF leaking memory, which would > obviously not be reported by erlang:memory(). However, then it surprises me > that the Kernel's view stays stable while the BEAM's actual memory usage is > still below the Kernel's view. It should be continuously growing in this > case imho. > > I'd appreciate if anyone has some more insight or experience with such a > behaviour, while I'm further digging into this. > > Cheers, > Tino > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Tue Sep 17 00:47:26 2013 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 17 Sep 2013 01:47:26 +0300 Subject: [erlang-questions] Not getting exit_status when using open_port with docker In-Reply-To: References: Message-ID: I'll try erlexec. Thanks On Mon, Sep 16, 2013 at 11:08 PM, OvermindDL1 wrote: > Look at erlexec instead. open_port is not designed for lifetime > management like that. > On Sep 16, 2013 7:31 AM, "pablo platt" wrote: > >> I looked at erlsh. It uses open_port with a loop like I'm using in my >> code so my question still stands. >> Why I'm not getting the exit message from the "docker run" command? >> >> erldocker looks nice but I couldn't find the equivalent of "docker run" >> without starting, attaching, detaching and destroying a container. >> I'm not sure how to do it manually because I need to know when the >> container exits which docker run does. >> >> >> On Mon, Sep 16, 2013 at 3:26 PM, Maxim Sokhatsky wrote: >> >>> First of all you do it wrong just because of two things :) >>> >>> The first thing is erlsh library: https://github.com/proger/erlsh >>> And the second is erldocker library: https://github.com/proger/erldocker >>> >>> Moreover if you are interesting in your own Erlang PaaS >>> you could read the sources of the first Erlang PaaS: >>> >>> https://github.com/voxoz >>> >>> Hope this save your time :) >>> >>> /maxim >>> >>> ________________________________ >>> > Date: Mon, 16 Sep 2013 15:10:48 +0300 >>> > From: pablo.platt@REDACTED >>> > To: erlang-questions@REDACTED >>> > Subject: [erlang-questions] Not getting exit_status when using >>> > open_port with docker >>> > >>> > Hi, >>> > >>> > I'm calling "docker run" from Erlang. >>> > http://docs.docker.io/en/latest/commandline/command/run/ >>> > I'm using open_port instead of os:cmd to be able to limit the >>> execution time. >>> > I'm getting the output of the command >>> > but the do_read just keep going and I'm not able to get an exit status >>> > from the command. >>> > >>> > When running "echo 'test'" I'm getting {exit_status, 0}. >>> > When running the "docker run" command in the shell, I'm getting the >>> > output and it exits. >>> > When running any other docker command like "docker ps" I'm getting the >>> > exit signal. >>> > >>> > Is this a problem with how I'm using open_port or something wrong with >>> > the "docker run" command? >>> > >>> > -module(test). >>> > >>> > -export([run/0]). >>> > >>> > run() -> >>> > Cmd = "sudo docker run ubuntu echo \"test\"", >>> > Port = open_port({spawn, Cmd}, [binary, exit_status]), >>> > do_read(Port, <<>>). >>> > >>> > do_read(Port, Data) -> >>> > receive >>> > {Port,{data, NewData}} -> >>> > io:format("data ~p~n", [NewData]), >>> > do_read(Port, <>); >>> > {Port,{exit_status, 0}} -> >>> > io:format("exit~n"), >>> > case Data of >>> > <<"">> -> <<"ok">>; >>> > _ -> Data >>> > end; >>> > E -> >>> > io:format("Something: ~p~n", [E]) >>> > end. >>> > >>> > >>> > _______________________________________________ erlang-questions >>> > mailing list erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.platt@REDACTED Tue Sep 17 00:42:51 2013 From: pablo.platt@REDACTED (pablo platt) Date: Tue, 17 Sep 2013 01:42:51 +0300 Subject: [erlang-questions] Not getting exit_status when using open_port with docker In-Reply-To: References: Message-ID: I'll try eerlexec. Thanks On Mon, Sep 16, 2013 at 11:08 PM, OvermindDL1 wrote: > Look at erlexec instead. open_port is not designed for lifetime > management like that. > On Sep 16, 2013 7:31 AM, "pablo platt" wrote: > >> I looked at erlsh. It uses open_port with a loop like I'm using in my >> code so my question still stands. >> Why I'm not getting the exit message from the "docker run" command? >> >> erldocker looks nice but I couldn't find the equivalent of "docker run" >> without starting, attaching, detaching and destroying a container. >> I'm not sure how to do it manually because I need to know when the >> container exits which docker run does. >> >> >> On Mon, Sep 16, 2013 at 3:26 PM, Maxim Sokhatsky wrote: >> >>> First of all you do it wrong just because of two things :) >>> >>> The first thing is erlsh library: https://github.com/proger/erlsh >>> And the second is erldocker library: https://github.com/proger/erldocker >>> >>> Moreover if you are interesting in your own Erlang PaaS >>> you could read the sources of the first Erlang PaaS: >>> >>> https://github.com/voxoz >>> >>> Hope this save your time :) >>> >>> /maxim >>> >>> ________________________________ >>> > Date: Mon, 16 Sep 2013 15:10:48 +0300 >>> > From: pablo.platt@REDACTED >>> > To: erlang-questions@REDACTED >>> > Subject: [erlang-questions] Not getting exit_status when using >>> > open_port with docker >>> > >>> > Hi, >>> > >>> > I'm calling "docker run" from Erlang. >>> > http://docs.docker.io/en/latest/commandline/command/run/ >>> > I'm using open_port instead of os:cmd to be able to limit the >>> execution time. >>> > I'm getting the output of the command >>> > but the do_read just keep going and I'm not able to get an exit status >>> > from the command. >>> > >>> > When running "echo 'test'" I'm getting {exit_status, 0}. >>> > When running the "docker run" command in the shell, I'm getting the >>> > output and it exits. >>> > When running any other docker command like "docker ps" I'm getting the >>> > exit signal. >>> > >>> > Is this a problem with how I'm using open_port or something wrong with >>> > the "docker run" command? >>> > >>> > -module(test). >>> > >>> > -export([run/0]). >>> > >>> > run() -> >>> > Cmd = "sudo docker run ubuntu echo \"test\"", >>> > Port = open_port({spawn, Cmd}, [binary, exit_status]), >>> > do_read(Port, <<>>). >>> > >>> > do_read(Port, Data) -> >>> > receive >>> > {Port,{data, NewData}} -> >>> > io:format("data ~p~n", [NewData]), >>> > do_read(Port, <>); >>> > {Port,{exit_status, 0}} -> >>> > io:format("exit~n"), >>> > case Data of >>> > <<"">> -> <<"ok">>; >>> > _ -> Data >>> > end; >>> > E -> >>> > io:format("Something: ~p~n", [E]) >>> > end. >>> > >>> > >>> > _______________________________________________ erlang-questions >>> > mailing list erlang-questions@REDACTED >>> > http://erlang.org/mailman/listinfo/erlang-questions >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Sep 17 02:39:41 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 16 Sep 2013 20:39:41 -0400 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: References: Message-ID: <20130917003940.GD46479@ferdmbp.local> I've recently run in similar issues and have received a bit of help from Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me, I tried to put into the recon library, currently on a different branch awaiting review and prone to change: https://github.com/ferd/recon/tree/allocators 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)` where `N` is the number of 'highest results' you want. The function will take a snapshot of the number of binary refs in your processes, then GC the node entirely, and then take another snapshot, make diff, and return the N biggest deltas in binaries. This will let you know what processes hold the most references to stale refc binaries. I recommend hibernation as a first way to try and fix this if it is the problem. 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)` where `Arg` can be: - `used` for the memory actively used (i.e. erlang:memory(total)) - `allocated` for the memory reserved by individual allocators (sum) - `usage` for the percentage. If the result of `allocated` is close to what the OS reports, you probably have fragmentation issues. If not, you may have a NIF or driver that allocates data outside of the ERTS allocators 3. check individual allocator usage levels with `recon_alloc:fragmentation(current)`. It will return usage percentages for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes by default. When the data allocated is too large (> than the single block carrier threshold [sbct]), it goes into its own block. Compare the results with what you get with `recon_alloc:fragmentation(max)`. If the current values have very low usage but the max ones have large ones, you may have lingering data, possibly held in long-term references or whatever that blocks deallocation of specific carriers. Different carrier strategies can help, which we can dive into if you see a problem with this. Feel free to read the comments in `recon_alloc` until I actually merge it in master, they contain some of the details about what to do or look for. Lukas may want me to correct me on the content of this post. I'm going from the limited knowledge he transmitted to me here, or rather, my limited understanding of it :) Regards, Fred. On 09/16, Tino Breddin wrote: > Hi list, > > I'm experiencing issues with a couple of Beam nodes where I see a huge gap > between the node's reported memory usage and the underlying Linux Kernel's > view. > > This is using R15B01. > > As a start an application in such a node stores a lot of tuples (containing > atoms and binary data) in ETS tables. That proceeds until a point where > memory usage is 70% (6GB) of the available memory. At that point > erlang:memory() and top (or /proc/PID/status) agree roughly on the memory > usage. Then an internal cleanup task is performed, which clears obsolete > records from the ETS tables. Afterwards, erlang:memory() reports an > expected low value of roughly 60MB memory usage. (This includes binary > data). However, the kernel still reports the high memory usage values (both > VmRss and VmTotal) for the node. The kernel's usage view will stay stable > until the ETS tables are filled to a point where the real memory usage > exceeds the kernel's view, then the kernel reported usage will grow as well. > > Now having checked the node in some details I'm wondering what causes this > difference between the BEAM's view and the Kernel's view on memory usage. I > have 2 ideas which I'm checking right now. > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because > the original dispatcher process which it was passed through before being > stored in an ETS table is still alive. Thus there is still some reference > to it? However, this would not explain why erlang:memory() reports a very > low value for used memory for binaries. > > (2) low-level memory leak: Some driver or NIF leaking memory, which would > obviously not be reported by erlang:memory(). However, then it surprises me > that the Kernel's view stays stable while the BEAM's actual memory usage is > still below the Kernel's view. It should be continuously growing in this > case imho. > > I'd appreciate if anyone has some more insight or experience with such a > behaviour, while I'm further digging into this. > > Cheers, > Tino > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From robert.virding@REDACTED Tue Sep 17 16:08:04 2013 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 17 Sep 2013 16:08:04 +0200 (CEST) Subject: [erlang-questions] erlounge in nyc In-Reply-To: References: <320991995.126835.1378670260349.JavaMail.zimbra@erlang-solutions.com> <4AC76AE5-E7D6-4DA0-8DF1-8D306081B87C@hates.ms> <5230C7F8.80007@research.att.com> Message-ID: <2005002675.179247.1379426884438.JavaMail.zimbra@erlang-solutions.com> Thanks to all who could make it, I enjoyed myself. And it felt good to be a cool hipster for a while. :-) And the beer was good. And the erlang community IS very friendly and nice to meet. Robert ----- Original Message ----- > From: "Mahesh Paolini-Subramanya" > To: "Roberto Ostinelli" > Cc: erlang-questions@REDACTED > Sent: Friday, 13 September, 2013 1:52:23 AM > Subject: Re: [erlang-questions] erlounge in nyc > We miss you... > Sent via smoke signals... > Mahesh Paolini-Subramanya > That tall bald Indian guy... > On Sep 12, 2013 7:49 PM, "Roberto Ostinelli" < roberto@REDACTED > wrote: > > I've been blocked too. A cheers to you all. > > > On Sep 11, 2013, at 3:43 PM, Garry Hodgson < garry@REDACTED > > > wrote: > > > > dang. it looks like i have a conflict, and cannot > > > > > > make it after all. oh well, i guess i'll catch the next one. > > > > > > On 9/11/13 11:33 AM, Mihai Balea wrote: > > > > > > > I'm in, probably be there around 7pm > > > > > > > > > > Mihai > > > > > > > > > > On Sep 8, 2013, at 5:19 PM, Alex Babkin wrote: > > > > > > > > > > > will see you guys there > > > > > > > > > > > > > > > On Sun, Sep 8, 2013 at 3:57 PM, Robert Virding < > > > > > robert.virding@REDACTED > wrote: > > > > > > > > > > > > > > > > That sounds very reasonable, and near the subway too. :-) Any time > > > > > > after > > > > > > 6pm > > > > > > suits me. > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > From: "Mahesh Paolini-Subramanya" < mahesh@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To: "Siraaj Khandkar" < siraaj@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cc: "Erlang" < erlang-questions@REDACTED >, "Roberto Ostinelli" > > > > > > > < > > > > > > > roberto@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent: Friday, 6 September, 2013 9:05:12 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject: Re: [erlang-questions] erlounge in nyc > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thursday works quite well for me too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > How about at Biblio (in Williamsburg) Excellent hipster bar with > > > > > > > great > > > > > > > beers > > > > > > > ( hipsters and beer. Well duh) and excellent munchies... > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://bibliobrooklyn.com/2013/06/05/beer-menu/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent via smoke signals... > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mahesh Paolini-Subramanya > > > > > > > > > > > > > > > > > > > > > > > > > > > > That tall bald Indian guy... > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 6, 2013 1:57 PM, "Siraaj Khandkar" < siraaj@REDACTED > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Count me in! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm good for Thursday the 12th, but I cannot do Fri the 13th > > > > > > > > (no, > > > > > > > > not > > > > > > > > for > > > > > > > > that reason :)). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/06/2013 12:54 PM, Roberto Ostinelli wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AFA the 'when': I believe the matching dates for everyone > > > > > > > > > would > > > > > > > > > have > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > be either Thursday 12th or Friday 13th. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm new to NY so I'm not in a position to suggest the 'when' > > > > > > > > > part. > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > r. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 6, 2013 at 12:00 PM, Robert Virding > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < robert.virding@REDACTED solutions.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Are people interested in a NYC erlounge? I will be there next > > > > > > > > > week > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and am free all evenings except Wednesday. Someone suggest > > > > > > > > > when > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > where. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------ ------------------------------ > > > > > > > > > ------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *From: *"Robert Virding" < robert.virding@REDACTED > > > > > > > > > solutions.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *To: *"Bob Ippolito" < bob@REDACTED > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Cc: *"Erlang" < erlang-questions@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >, "Roberto Ostinelli" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < roberto@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Sent: *Monday, 2 September, 2013 6:04:04 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Subject: *Re: [erlang-questions] erlounge in nyc > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I will be in NYC 10-15 sept so I will do my best to attend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Erlounges are great. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------ ------------------------------ > > > > > > > > > ------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *From: *"Bob Ippolito" < bob@REDACTED > > > > > > > > bob@REDACTED > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *To: *"Roberto Ostinelli" < roberto@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Cc: *"Erlang" < erlang-questions@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Sent: *Friday, 30 August, 2013 5:17:52 AM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Subject: *Re: [erlang-questions] erlounge in nyc > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm in NYC from Sep 5-13 so I could probably also attend > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, August 29, 2013, Roberto Ostinelli wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I moved to NYC. If this happens I'm in too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > r. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 29, 2013 at 10:39 PM, Heinz Nikolaus Gies > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < heinz@REDACTED > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll do my best to make it too :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Heinz > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Aug 29, 2013, at 22:16, Garry Hodgson > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < garry@REDACTED > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > i would be love to get together with some other > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > erlang folks, if this is still on the table. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/10/2013 02:59 PM, Joe Armstrong wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Somebody, and I forget who asked me, and I > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > forget when, to drop them a line > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> if I was every in New York. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> I'll be in New York from 10 - 14 sept, if > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > anybody is interested in a meetup. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> /Joe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Garry Hodgson > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AT&T Chief Security Office (CSO) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "This e-mail and any files transmitted with it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > are AT&T property, are confidential, and are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > intended solely for the use of the individual or > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > entity to whom this e-mail is addressed. If you are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not one of the named recipient(s) or otherwise have > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > reason to believe that you have received this > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > message in error, please notify the sender and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > delete this message immediately from your computer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any other use, retention, dissemination, forwarding, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > printing, or copying of this e-mail is strictly > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > prohibited." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________ _________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > erlang-questions mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > erlang-questions@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://erlang.org/mailman/ listinfo/erlang-questions > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > erlang-questions mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > erlang-questions@REDACTED > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > erlang-questions mailing list > > > > > > > > > > > > > > > > > > > > > erlang-questions@REDACTED > > > > > > > > > > > > > > > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > erlang-questions mailing list > > > > > > > > > > > > > > > erlang-questions@REDACTED > > > > > > > > > > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > erlang-questions mailing list erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > > _______________________________________________ > > > > > > erlang-questions mailing list > > > > > > erlang-questions@REDACTED > > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Tue Sep 17 16:36:19 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 17 Sep 2013 10:36:19 -0400 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> Message-ID: <20130917143618.GA13826@ferdair.local> Hi Robert, I CC'd the mailing list on this post, because I felt it could be interesting to share with everyone. To get into some more details, the difference between a garbage-collection and hibernating is that hibernation forces a full-sweep and does compaction work. It is more likely to actually remove old refs to binaries. The tricky part about binary leaks in these cases is that if the process you're garbage collecting holds some very long-lived references (or takes a long while before enabling them), you will move the references to the old heap, if my understanding is correct. Now if your process that leaks resources is busy or bogged down by some task, there are chances that manually calling GC at higher frequencies will force short-lived references to the old heap. Eventually, most of the subsequent GCs are done for no good reason until there's a full sweep to free the references, if my understanding is right. In comparison, some hibernations may turn out to be beneficial due to how they do the full sweep, especially on less active processes, without changing spawn_opt values or whatever. Both cases are not necessarily great, and I don't think there's one easy way to deal with it. One thing I've had in mind to try for a while was to run a function a bit like: -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer(). gc_count(PreviousCounter, Bin) -> case byte_size(Bin) of N when N >= 64 -> % refc binary Count = N + PreviousCounter case NewCount >= ?THRESHOLD of true -> erlang:garbage_collect(), 0; false -> NewCount end; N -> % heap binary PreviousCounter+N end. that could alternatively force some hibernation instead of GC'ing. This one could basically track the size of all binaries seen manually and force something when you go over a certain amount. It sucks, though, because that's basically manually doing your collection, and it doesn't mean that because you have seen a binary, it's ready to be GC'd. I've thus avoided trying it in the real world for now. In practice, at Heroku, we've decided to go for a hybrid approach in logplex. We force hibernation on some important events that interrupt our work flow no matter what (such as a socket disconnection, or long periods of time [seconds] without activity for a process), and have put a workaround in place to force VM-wide GCs when we're reaching critical amounts of memory: https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl The objective was to use global GC as a last measure in case individual (unobtrusive) hibernates were not enough to save a node. This later on prompted for exploring the allocators of the VM -- the value used (erlang:memory(total)) didn't represent the OS-imposed limits on the VM: nodes would be killed by going out of memory without first having had the chance to run the global GC. This lead to discovering things about fragmentation and characterizing our workloads to pick better allocation strategies that seem to work decently so far, so that erlang:memory(total), for one, has the right values, and also that we have a better time releasing allocated blocks of memory when most binaries vanish. I hope we can remove both the artificial hibernation calls and the workarounds to force some global GCs in the near future. Ideally, it sounds like the VM should possibly do more when it comes to the weight of refc binaries to individual processes' memory for GC, but I don't have a good idea of how this should be done in practice without having elephant-sized assumptions and holes in the solution without adding more knobs to the VM to configure things. Plus I'd have no idea on how to actually implement it. Regards, Fred. On 09/17, Robert Virding wrote: > Hi Fred, > > You recommend hibernating a process. Do you think this is better than calling the garbage collector in a process? I have no idea but hibernating seems more drastic, especially if the process is "in use"? > > Robert > > ----- Original Message ----- > > From: "Fred Hebert" > > To: "Tino Breddin" > > Cc: "Erlang Questions Mailing List" > > Sent: Tuesday, 17 September, 2013 2:39:41 AM > > Subject: Re: [erlang-questions] Problem with Beam process not freeing memory > > > > I've recently run in similar issues and have received a bit of help from > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me, > > I tried to put into the recon library, currently on a different branch > > awaiting review and prone to change: > > https://github.com/ferd/recon/tree/allocators > > > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)` > > where `N` is the number of 'highest results' you want. The function will > > take a snapshot of the number of binary refs in your processes, then GC > > the node entirely, and then take another snapshot, make diff, and return > > the N biggest deltas in binaries. This will let you know what processes > > hold the most references to stale refc binaries. I recommend hibernation > > as a first way to try and fix this if it is the problem. > > > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)` > > where `Arg` can be: > > - `used` for the memory actively used (i.e. erlang:memory(total)) > > - `allocated` for the memory reserved by individual allocators (sum) > > - `usage` for the percentage. > > If the result of `allocated` is close to what the OS reports, you > > probably have fragmentation issues. If not, you may have a NIF or driver > > that allocates data outside of the ERTS allocators > > > > 3. check individual allocator usage levels with > > `recon_alloc:fragmentation(current)`. It will return usage percentages > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes > > by default. When the data allocated is too large (> than the single > > block carrier threshold [sbct]), it goes into its own block. Compare the > > results with what you get with `recon_alloc:fragmentation(max)`. If the > > current values have very low usage but the max ones have large ones, you > > may have lingering data, possibly held in long-term references or > > whatever that blocks deallocation of specific carriers. Different > > carrier strategies can help, which we can dive into if you see a problem > > with this. > > > > Feel free to read the comments in `recon_alloc` until I actually merge > > it in master, they contain some of the details about what to do or look > > for. > > > > Lukas may want me to correct me on the content of this post. I'm going > > from the limited knowledge he transmitted to me here, or rather, my > > limited understanding of it :) > > > > Regards, > > Fred. > > > > On 09/16, Tino Breddin wrote: > > > Hi list, > > > > > > I'm experiencing issues with a couple of Beam nodes where I see a huge gap > > > between the node's reported memory usage and the underlying Linux Kernel's > > > view. > > > > > > This is using R15B01. > > > > > > As a start an application in such a node stores a lot of tuples (containing > > > atoms and binary data) in ETS tables. That proceeds until a point where > > > memory usage is 70% (6GB) of the available memory. At that point > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the memory > > > usage. Then an internal cleanup task is performed, which clears obsolete > > > records from the ETS tables. Afterwards, erlang:memory() reports an > > > expected low value of roughly 60MB memory usage. (This includes binary > > > data). However, the kernel still reports the high memory usage values (both > > > VmRss and VmTotal) for the node. The kernel's usage view will stay stable > > > until the ETS tables are filled to a point where the real memory usage > > > exceeds the kernel's view, then the kernel reported usage will grow as > > > well. > > > > > > Now having checked the node in some details I'm wondering what causes this > > > difference between the BEAM's view and the Kernel's view on memory usage. I > > > have 2 ideas which I'm checking right now. > > > > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because > > > the original dispatcher process which it was passed through before being > > > stored in an ETS table is still alive. Thus there is still some reference > > > to it? However, this would not explain why erlang:memory() reports a very > > > low value for used memory for binaries. > > > > > > (2) low-level memory leak: Some driver or NIF leaking memory, which would > > > obviously not be reported by erlang:memory(). However, then it surprises me > > > that the Kernel's view stays stable while the BEAM's actual memory usage is > > > still below the Kernel's view. It should be continuously growing in this > > > case imho. > > > > > > I'd appreciate if anyone has some more insight or experience with such a > > > behaviour, while I'm further digging into this. > > > > > > Cheers, > > > Tino > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > From robert.virding@REDACTED Tue Sep 17 16:53:40 2013 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 17 Sep 2013 16:53:40 +0200 (CEST) Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: <20130917143618.GA13826@ferdair.local> References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> Message-ID: <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> OK, I meant doing an explicit garbage collectionen and was under the impression that calling erlang:garbage_collect() actually did a full sweep of the process which is why hibernating seemed a bit of an overkill. Rober ----- Original Message ----- > From: "Fred Hebert" > To: "Robert Virding" > Cc: "Erlang Questions Mailing List" > Sent: Tuesday, 17 September, 2013 4:36:19 PM > Subject: Re: [erlang-questions] Problem with Beam process not freeing memory > > Hi Robert, > > I CC'd the mailing list on this post, because I felt it could be > interesting to share with everyone. > > To get into some more details, the difference between a > garbage-collection and hibernating is that hibernation forces a > full-sweep and does compaction work. It is more likely to actually > remove old refs to binaries. > > The tricky part about binary leaks in these cases is that if the process > you're garbage collecting holds some very long-lived references (or > takes a long while before enabling them), you will move the references > to the old heap, if my understanding is correct. Now if your process > that leaks resources is busy or bogged down by some task, there are > chances that manually calling GC at higher frequencies will force > short-lived references to the old heap. Eventually, most of the > subsequent GCs are done for no good reason until there's a full sweep to > free the references, if my understanding is right. > > In comparison, some hibernations may turn out to be beneficial due to > how they do the full sweep, especially on less active processes, without > changing spawn_opt values or whatever. > > Both cases are not necessarily great, and I don't think there's one easy > way to deal with it. > > One thing I've had in mind to try for a while was to run a function a > bit like: > > -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer(). > gc_count(PreviousCounter, Bin) -> > case byte_size(Bin) of > N when N >= 64 -> % refc binary > Count = N + PreviousCounter > case NewCount >= ?THRESHOLD of > true -> > erlang:garbage_collect(), > 0; > false -> > NewCount > end; > N -> % heap binary > PreviousCounter+N > end. > > that could alternatively force some hibernation instead of GC'ing. This > one could basically track the size of all binaries seen manually and > force something when you go over a certain amount. It sucks, though, > because that's basically manually doing your collection, and it doesn't > mean that because you have seen a binary, it's ready to be GC'd. I've > thus avoided trying it in the real world for now. > > In practice, at Heroku, we've decided to go for a hybrid approach in > logplex. We force hibernation on some important events that interrupt > our work flow no matter what (such as a socket disconnection, or long > periods of time [seconds] without activity for a process), and have put > a workaround in place to force VM-wide GCs when we're reaching critical > amounts of memory: > https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl > > The objective was to use global GC as a last measure in case individual > (unobtrusive) hibernates were not enough to save a node. > > This later on prompted for exploring the allocators of the VM -- the > value used (erlang:memory(total)) didn't represent the OS-imposed limits > on the VM: nodes would be killed by going out of memory without first > having had the chance to run the global GC. This lead to discovering > things about fragmentation and characterizing our workloads to pick > better allocation strategies that seem to work decently so far, so that > erlang:memory(total), for one, has the right values, and also that we > have a better time releasing allocated blocks of memory when most > binaries vanish. > > I hope we can remove both the artificial hibernation calls and the > workarounds to force some global GCs in the near future. Ideally, it > sounds like the VM should possibly do more when it comes to the weight > of refc binaries to individual processes' memory for GC, but I don't > have a good idea of how this should be done in practice without having > elephant-sized assumptions and holes in the solution without adding more > knobs to the VM to configure things. Plus I'd have no idea on how to > actually implement it. > > Regards, > Fred. > > > On 09/17, Robert Virding wrote: > > Hi Fred, > > > > You recommend hibernating a process. Do you think this is better than > > calling the garbage collector in a process? I have no idea but hibernating > > seems more drastic, especially if the process is "in use"? > > > > Robert > > > > ----- Original Message ----- > > > From: "Fred Hebert" > > > To: "Tino Breddin" > > > Cc: "Erlang Questions Mailing List" > > > Sent: Tuesday, 17 September, 2013 2:39:41 AM > > > Subject: Re: [erlang-questions] Problem with Beam process not freeing > > > memory > > > > > > I've recently run in similar issues and have received a bit of help from > > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me, > > > I tried to put into the recon library, currently on a different branch > > > awaiting review and prone to change: > > > https://github.com/ferd/recon/tree/allocators > > > > > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)` > > > where `N` is the number of 'highest results' you want. The function will > > > take a snapshot of the number of binary refs in your processes, then GC > > > the node entirely, and then take another snapshot, make diff, and return > > > the N biggest deltas in binaries. This will let you know what processes > > > hold the most references to stale refc binaries. I recommend hibernation > > > as a first way to try and fix this if it is the problem. > > > > > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)` > > > where `Arg` can be: > > > - `used` for the memory actively used (i.e. erlang:memory(total)) > > > - `allocated` for the memory reserved by individual allocators (sum) > > > - `usage` for the percentage. > > > If the result of `allocated` is close to what the OS reports, you > > > probably have fragmentation issues. If not, you may have a NIF or driver > > > that allocates data outside of the ERTS allocators > > > > > > 3. check individual allocator usage levels with > > > `recon_alloc:fragmentation(current)`. It will return usage percentages > > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes > > > by default. When the data allocated is too large (> than the single > > > block carrier threshold [sbct]), it goes into its own block. Compare the > > > results with what you get with `recon_alloc:fragmentation(max)`. If the > > > current values have very low usage but the max ones have large ones, you > > > may have lingering data, possibly held in long-term references or > > > whatever that blocks deallocation of specific carriers. Different > > > carrier strategies can help, which we can dive into if you see a problem > > > with this. > > > > > > Feel free to read the comments in `recon_alloc` until I actually merge > > > it in master, they contain some of the details about what to do or look > > > for. > > > > > > Lukas may want me to correct me on the content of this post. I'm going > > > from the limited knowledge he transmitted to me here, or rather, my > > > limited understanding of it :) > > > > > > Regards, > > > Fred. > > > > > > On 09/16, Tino Breddin wrote: > > > > Hi list, > > > > > > > > I'm experiencing issues with a couple of Beam nodes where I see a huge > > > > gap > > > > between the node's reported memory usage and the underlying Linux > > > > Kernel's > > > > view. > > > > > > > > This is using R15B01. > > > > > > > > As a start an application in such a node stores a lot of tuples > > > > (containing > > > > atoms and binary data) in ETS tables. That proceeds until a point where > > > > memory usage is 70% (6GB) of the available memory. At that point > > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the > > > > memory > > > > usage. Then an internal cleanup task is performed, which clears > > > > obsolete > > > > records from the ETS tables. Afterwards, erlang:memory() reports an > > > > expected low value of roughly 60MB memory usage. (This includes binary > > > > data). However, the kernel still reports the high memory usage values > > > > (both > > > > VmRss and VmTotal) for the node. The kernel's usage view will stay > > > > stable > > > > until the ETS tables are filled to a point where the real memory usage > > > > exceeds the kernel's view, then the kernel reported usage will grow as > > > > well. > > > > > > > > Now having checked the node in some details I'm wondering what causes > > > > this > > > > difference between the BEAM's view and the Kernel's view on memory > > > > usage. I > > > > have 2 ideas which I'm checking right now. > > > > > > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed > > > > because > > > > the original dispatcher process which it was passed through before > > > > being > > > > stored in an ETS table is still alive. Thus there is still some > > > > reference > > > > to it? However, this would not explain why erlang:memory() reports a > > > > very > > > > low value for used memory for binaries. > > > > > > > > (2) low-level memory leak: Some driver or NIF leaking memory, which > > > > would > > > > obviously not be reported by erlang:memory(). However, then it > > > > surprises me > > > > that the Kernel's view stays stable while the BEAM's actual memory > > > > usage is > > > > still below the Kernel's view. It should be continuously growing in > > > > this > > > > case imho. > > > > > > > > I'd appreciate if anyone has some more insight or experience with such > > > > a > > > > behaviour, while I'm further digging into this. > > > > > > > > Cheers, > > > > Tino > > > > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > From garazdawi@REDACTED Tue Sep 17 17:03:53 2013 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 17 Sep 2013 17:03:53 +0200 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> Message-ID: as far as I can see erlang:garbage_collect/0,1 does do fullsweeps [1]. [1]: https://github.com/erlang/otp/blob/maint/erts/emulator/beam/bif.c#L3770 On Tue, Sep 17, 2013 at 4:53 PM, Robert Virding < robert.virding@REDACTED> wrote: > OK, I meant doing an explicit garbage collectionen and was under the > impression that calling erlang:garbage_collect() actually did a full sweep > of the process which is why hibernating seemed a bit of an overkill. > > Rober > > ----- Original Message ----- > > From: "Fred Hebert" > > To: "Robert Virding" > > Cc: "Erlang Questions Mailing List" > > Sent: Tuesday, 17 September, 2013 4:36:19 PM > > Subject: Re: [erlang-questions] Problem with Beam process not freeing > memory > > > > Hi Robert, > > > > I CC'd the mailing list on this post, because I felt it could be > > interesting to share with everyone. > > > > To get into some more details, the difference between a > > garbage-collection and hibernating is that hibernation forces a > > full-sweep and does compaction work. It is more likely to actually > > remove old refs to binaries. > > > > The tricky part about binary leaks in these cases is that if the process > > you're garbage collecting holds some very long-lived references (or > > takes a long while before enabling them), you will move the references > > to the old heap, if my understanding is correct. Now if your process > > that leaks resources is busy or bogged down by some task, there are > > chances that manually calling GC at higher frequencies will force > > short-lived references to the old heap. Eventually, most of the > > subsequent GCs are done for no good reason until there's a full sweep to > > free the references, if my understanding is right. > > > > In comparison, some hibernations may turn out to be beneficial due to > > how they do the full sweep, especially on less active processes, without > > changing spawn_opt values or whatever. > > > > Both cases are not necessarily great, and I don't think there's one easy > > way to deal with it. > > > > One thing I've had in mind to try for a while was to run a function a > > bit like: > > > > -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer(). > > gc_count(PreviousCounter, Bin) -> > > case byte_size(Bin) of > > N when N >= 64 -> % refc binary > > Count = N + PreviousCounter > > case NewCount >= ?THRESHOLD of > > true -> > > erlang:garbage_collect(), > > 0; > > false -> > > NewCount > > end; > > N -> % heap binary > > PreviousCounter+N > > end. > > > > that could alternatively force some hibernation instead of GC'ing. This > > one could basically track the size of all binaries seen manually and > > force something when you go over a certain amount. It sucks, though, > > because that's basically manually doing your collection, and it doesn't > > mean that because you have seen a binary, it's ready to be GC'd. I've > > thus avoided trying it in the real world for now. > > > > In practice, at Heroku, we've decided to go for a hybrid approach in > > logplex. We force hibernation on some important events that interrupt > > our work flow no matter what (such as a socket disconnection, or long > > periods of time [seconds] without activity for a process), and have put > > a workaround in place to force VM-wide GCs when we're reaching critical > > amounts of memory: > > https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl > > > > The objective was to use global GC as a last measure in case individual > > (unobtrusive) hibernates were not enough to save a node. > > > > This later on prompted for exploring the allocators of the VM -- the > > value used (erlang:memory(total)) didn't represent the OS-imposed limits > > on the VM: nodes would be killed by going out of memory without first > > having had the chance to run the global GC. This lead to discovering > > things about fragmentation and characterizing our workloads to pick > > better allocation strategies that seem to work decently so far, so that > > erlang:memory(total), for one, has the right values, and also that we > > have a better time releasing allocated blocks of memory when most > > binaries vanish. > > > > I hope we can remove both the artificial hibernation calls and the > > workarounds to force some global GCs in the near future. Ideally, it > > sounds like the VM should possibly do more when it comes to the weight > > of refc binaries to individual processes' memory for GC, but I don't > > have a good idea of how this should be done in practice without having > > elephant-sized assumptions and holes in the solution without adding more > > knobs to the VM to configure things. Plus I'd have no idea on how to > > actually implement it. > > > > Regards, > > Fred. > > > > > > On 09/17, Robert Virding wrote: > > > Hi Fred, > > > > > > You recommend hibernating a process. Do you think this is better than > > > calling the garbage collector in a process? I have no idea but > hibernating > > > seems more drastic, especially if the process is "in use"? > > > > > > Robert > > > > > > ----- Original Message ----- > > > > From: "Fred Hebert" > > > > To: "Tino Breddin" > > > > Cc: "Erlang Questions Mailing List" > > > > Sent: Tuesday, 17 September, 2013 2:39:41 AM > > > > Subject: Re: [erlang-questions] Problem with Beam process not freeing > > > > memory > > > > > > > > I've recently run in similar issues and have received a bit of help > from > > > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught > me, > > > > I tried to put into the recon library, currently on a different > branch > > > > awaiting review and prone to change: > > > > https://github.com/ferd/recon/tree/allocators > > > > > > > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)` > > > > where `N` is the number of 'highest results' you want. The function > will > > > > take a snapshot of the number of binary refs in your processes, then > GC > > > > the node entirely, and then take another snapshot, make diff, and > return > > > > the N biggest deltas in binaries. This will let you know what > processes > > > > hold the most references to stale refc binaries. I recommend > hibernation > > > > as a first way to try and fix this if it is the problem. > > > > > > > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)` > > > > where `Arg` can be: > > > > - `used` for the memory actively used (i.e. erlang:memory(total)) > > > > - `allocated` for the memory reserved by individual allocators (sum) > > > > - `usage` for the percentage. > > > > If the result of `allocated` is close to what the OS reports, you > > > > probably have fragmentation issues. If not, you may have a NIF or > driver > > > > that allocates data outside of the ERTS allocators > > > > > > > > 3. check individual allocator usage levels with > > > > `recon_alloc:fragmentation(current)`. It will return usage > percentages > > > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data > goes > > > > by default. When the data allocated is too large (> than the single > > > > block carrier threshold [sbct]), it goes into its own block. Compare > the > > > > results with what you get with `recon_alloc:fragmentation(max)`. If > the > > > > current values have very low usage but the max ones have large ones, > you > > > > may have lingering data, possibly held in long-term references or > > > > whatever that blocks deallocation of specific carriers. Different > > > > carrier strategies can help, which we can dive into if you see a > problem > > > > with this. > > > > > > > > Feel free to read the comments in `recon_alloc` until I actually > merge > > > > it in master, they contain some of the details about what to do or > look > > > > for. > > > > > > > > Lukas may want me to correct me on the content of this post. I'm > going > > > > from the limited knowledge he transmitted to me here, or rather, my > > > > limited understanding of it :) > > > > > > > > Regards, > > > > Fred. > > > > > > > > On 09/16, Tino Breddin wrote: > > > > > Hi list, > > > > > > > > > > I'm experiencing issues with a couple of Beam nodes where I see a > huge > > > > > gap > > > > > between the node's reported memory usage and the underlying Linux > > > > > Kernel's > > > > > view. > > > > > > > > > > This is using R15B01. > > > > > > > > > > As a start an application in such a node stores a lot of tuples > > > > > (containing > > > > > atoms and binary data) in ETS tables. That proceeds until a point > where > > > > > memory usage is 70% (6GB) of the available memory. At that point > > > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the > > > > > memory > > > > > usage. Then an internal cleanup task is performed, which clears > > > > > obsolete > > > > > records from the ETS tables. Afterwards, erlang:memory() reports an > > > > > expected low value of roughly 60MB memory usage. (This includes > binary > > > > > data). However, the kernel still reports the high memory usage > values > > > > > (both > > > > > VmRss and VmTotal) for the node. The kernel's usage view will stay > > > > > stable > > > > > until the ETS tables are filled to a point where the real memory > usage > > > > > exceeds the kernel's view, then the kernel reported usage will > grow as > > > > > well. > > > > > > > > > > Now having checked the node in some details I'm wondering what > causes > > > > > this > > > > > difference between the BEAM's view and the Kernel's view on memory > > > > > usage. I > > > > > have 2 ideas which I'm checking right now. > > > > > > > > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed > > > > > because > > > > > the original dispatcher process which it was passed through before > > > > > being > > > > > stored in an ETS table is still alive. Thus there is still some > > > > > reference > > > > > to it? However, this would not explain why erlang:memory() reports > a > > > > > very > > > > > low value for used memory for binaries. > > > > > > > > > > (2) low-level memory leak: Some driver or NIF leaking memory, which > > > > > would > > > > > obviously not be reported by erlang:memory(). However, then it > > > > > surprises me > > > > > that the Kernel's view stays stable while the BEAM's actual memory > > > > > usage is > > > > > still below the Kernel's view. It should be continuously growing in > > > > > this > > > > > case imho. > > > > > > > > > > I'd appreciate if anyone has some more insight or experience with > such > > > > > a > > > > > behaviour, while I'm further digging into this. > > > > > > > > > > Cheers, > > > > > Tino > > > > > > > > > _______________________________________________ > > > > > erlang-questions mailing list > > > > > erlang-questions@REDACTED > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Tue Sep 17 17:31:32 2013 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 17 Sep 2013 08:31:32 -0700 Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> Message-ID: <523875D4.6030206@gmail.com> erlang:garbage_collect/0,1 existing is nice, but using it seems to only form a kludge. It seems much better to focus on having short-lived Erlang processes to capture the last reference to generated binaries (that are large enough to be reference counted, so greater than 64 bytes). The pid options for fullsweep, hibernate, erts command line options, and perhaps erlang:garbage_collect/0,1 usage, seems more like tuning for performance based on usage. Using short-lived processes in Erlang is using Erlang for what it is good at.... why would you fight that by having code blow-up memory-wise by leaving long-lived Erlang processes in your source code? This usage of short-lived processes is what concerns me when we talk about how Erlang pids are limited to 2^28 unique pids per node (i.e., 28 bits, not yet changed to be 60 bits), but it is good that the new algorithm for pid reuse in R16B01 will support this approach (i.e., the Erlang pid per rpc-call approach). (details are from the "pid representation in external term format" email thread) On 09/17/2013 08:03 AM, Lukas Larsson wrote: > as far as I can see erlang:garbage_collect/0,1 does do fullsweeps [1]. > > [1]: https://github.com/erlang/otp/blob/maint/erts/emulator/beam/bif.c#L3770 > > > On Tue, Sep 17, 2013 at 4:53 PM, Robert Virding > wrote: > > OK, I meant doing an explicit garbage collectionen and was under the impression that calling erlang:garbage_collect() actually did a full sweep of the process which is why hibernating seemed a bit of an overkill. > > Rober > > ----- Original Message ----- > > From: "Fred Hebert" > > > To: "Robert Virding" > > > Cc: "Erlang Questions Mailing List" > > > Sent: Tuesday, 17 September, 2013 4:36:19 PM > > Subject: Re: [erlang-questions] Problem with Beam process not freeing memory > > > > Hi Robert, > > > > I CC'd the mailing list on this post, because I felt it could be > > interesting to share with everyone. > > > > To get into some more details, the difference between a > > garbage-collection and hibernating is that hibernation forces a > > full-sweep and does compaction work. It is more likely to actually > > remove old refs to binaries. > > > > The tricky part about binary leaks in these cases is that if the process > > you're garbage collecting holds some very long-lived references (or > > takes a long while before enabling them), you will move the references > > to the old heap, if my understanding is correct. Now if your process > > that leaks resources is busy or bogged down by some task, there are > > chances that manually calling GC at higher frequencies will force > > short-lived references to the old heap. Eventually, most of the > > subsequent GCs are done for no good reason until there's a full sweep to > > free the references, if my understanding is right. > > > > In comparison, some hibernations may turn out to be beneficial due to > > how they do the full sweep, especially on less active processes, without > > changing spawn_opt values or whatever. > > > > Both cases are not necessarily great, and I don't think there's one easy > > way to deal with it. > > > > One thing I've had in mind to try for a while was to run a function a > > bit like: > > > > -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer(). > > gc_count(PreviousCounter, Bin) -> > > case byte_size(Bin) of > > N when N >= 64 -> % refc binary > > Count = N + PreviousCounter > > case NewCount >= ?THRESHOLD of > > true -> > > erlang:garbage_collect(), > > 0; > > false -> > > NewCount > > end; > > N -> % heap binary > > PreviousCounter+N > > end. > > > > that could alternatively force some hibernation instead of GC'ing. This > > one could basically track the size of all binaries seen manually and > > force something when you go over a certain amount. It sucks, though, > > because that's basically manually doing your collection, and it doesn't > > mean that because you have seen a binary, it's ready to be GC'd. I've > > thus avoided trying it in the real world for now. > > > > In practice, at Heroku, we've decided to go for a hybrid approach in > > logplex. We force hibernation on some important events that interrupt > > our work flow no matter what (such as a socket disconnection, or long > > periods of time [seconds] without activity for a process), and have put > > a workaround in place to force VM-wide GCs when we're reaching critical > > amounts of memory: > > https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl > > > > The objective was to use global GC as a last measure in case individual > > (unobtrusive) hibernates were not enough to save a node. > > > > This later on prompted for exploring the allocators of the VM -- the > > value used (erlang:memory(total)) didn't represent the OS-imposed limits > > on the VM: nodes would be killed by going out of memory without first > > having had the chance to run the global GC. This lead to discovering > > things about fragmentation and characterizing our workloads to pick > > better allocation strategies that seem to work decently so far, so that > > erlang:memory(total), for one, has the right values, and also that we > > have a better time releasing allocated blocks of memory when most > > binaries vanish. > > > > I hope we can remove both the artificial hibernation calls and the > > workarounds to force some global GCs in the near future. Ideally, it > > sounds like the VM should possibly do more when it comes to the weight > > of refc binaries to individual processes' memory for GC, but I don't > > have a good idea of how this should be done in practice without having > > elephant-sized assumptions and holes in the solution without adding more > > knobs to the VM to configure things. Plus I'd have no idea on how to > > actually implement it. > > > > Regards, > > Fred. > > > > > > On 09/17, Robert Virding wrote: > > > Hi Fred, > > > > > > You recommend hibernating a process. Do you think this is better than > > > calling the garbage collector in a process? I have no idea but hibernating > > > seems more drastic, especially if the process is "in use"? > > > > > > Robert > > > > > > ----- Original Message ----- > > > > From: "Fred Hebert" > > > > > To: "Tino Breddin" > > > > > Cc: "Erlang Questions Mailing List" > > > > > Sent: Tuesday, 17 September, 2013 2:39:41 AM > > > > Subject: Re: [erlang-questions] Problem with Beam process not freeing > > > > memory > > > > > > > > I've recently run in similar issues and have received a bit of help from > > > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me, > > > > I tried to put into the recon library, currently on a different branch > > > > awaiting review and prone to change: > > > > https://github.com/ferd/recon/tree/allocators > > > > > > > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)` > > > > where `N` is the number of 'highest results' you want. The function will > > > > take a snapshot of the number of binary refs in your processes, then GC > > > > the node entirely, and then take another snapshot, make diff, and return > > > > the N biggest deltas in binaries. This will let you know what processes > > > > hold the most references to stale refc binaries. I recommend hibernation > > > > as a first way to try and fix this if it is the problem. > > > > > > > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)` > > > > where `Arg` can be: > > > > - `used` for the memory actively used (i.e. erlang:memory(total)) > > > > - `allocated` for the memory reserved by individual allocators (sum) > > > > - `usage` for the percentage. > > > > If the result of `allocated` is close to what the OS reports, you > > > > probably have fragmentation issues. If not, you may have a NIF or driver > > > > that allocates data outside of the ERTS allocators > > > > > > > > 3. check individual allocator usage levels with > > > > `recon_alloc:fragmentation(current)`. It will return usage percentages > > > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes > > > > by default. When the data allocated is too large (> than the single > > > > block carrier threshold [sbct]), it goes into its own block. Compare the > > > > results with what you get with `recon_alloc:fragmentation(max)`. If the > > > > current values have very low usage but the max ones have large ones, you > > > > may have lingering data, possibly held in long-term references or > > > > whatever that blocks deallocation of specific carriers. Different > > > > carrier strategies can help, which we can dive into if you see a problem > > > > with this. > > > > > > > > Feel free to read the comments in `recon_alloc` until I actually merge > > > > it in master, they contain some of the details about what to do or look > > > > for. > > > > > > > > Lukas may want me to correct me on the content of this post. I'm going > > > > from the limited knowledge he transmitted to me here, or rather, my > > > > limited understanding of it :) > > > > > > > > Regards, > > > > Fred. > > > > > > > > On 09/16, Tino Breddin wrote: > > > > > Hi list, > > > > > > > > > > I'm experiencing issues with a couple of Beam nodes where I see a huge > > > > > gap > > > > > between the node's reported memory usage and the underlying Linux > > > > > Kernel's > > > > > view. > > > > > > > > > > This is using R15B01. > > > > > > > > > > As a start an application in such a node stores a lot of tuples > > > > > (containing > > > > > atoms and binary data) in ETS tables. That proceeds until a point where > > > > > memory usage is 70% (6GB) of the available memory. At that point > > > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the > > > > > memory > > > > > usage. Then an internal cleanup task is performed, which clears > > > > > obsolete > > > > > records from the ETS tables. Afterwards, erlang:memory() reports an > > > > > expected low value of roughly 60MB memory usage. (This includes binary > > > > > data). However, the kernel still reports the high memory usage values > > > > > (both > > > > > VmRss and VmTotal) for the node. The kernel's usage view will stay > > > > > stable > > > > > until the ETS tables are filled to a point where the real memory usage > > > > > exceeds the kernel's view, then the kernel reported usage will grow as > > > > > well. > > > > > > > > > > Now having checked the node in some details I'm wondering what causes > > > > > this > > > > > difference between the BEAM's view and the Kernel's view on memory > > > > > usage. I > > > > > have 2 ideas which I'm checking right now. > > > > > > > > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed > > > > > because > > > > > the original dispatcher process which it was passed through before > > > > > being > > > > > stored in an ETS table is still alive. Thus there is still some > > > > > reference > > > > > to it? However, this would not explain why erlang:memory() reports a > > > > > very > > > > > low value for used memory for binaries. > > > > > > > > > > (2) low-level memory leak: Some driver or NIF leaking memory, which > > > > > would > > > > > obviously not be reported by erlang:memory(). However, then it > > > > > surprises me > > > > > that the Kernel's view stays stable while the BEAM's actual memory > > > > > usage is > > > > > still below the Kernel's view. It should be continuously growing in > > > > > this > > > > > case imho. > > > > > > > > > > I'd appreciate if anyone has some more insight or experience with such > > > > > a > > > > > behaviour, while I'm further digging into this. > > > > > > > > > > Cheers, > > > > > Tino > > > > > > > > > _______________________________________________ > > > > > erlang-questions mailing list > > > > > erlang-questions@REDACTED > > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmakarlsson@REDACTED Tue Sep 17 17:46:24 2013 From: jmakarlsson@REDACTED (J K) Date: Tue, 17 Sep 2013 08:46:24 -0700 (PDT) Subject: [erlang-questions] Problem with Beam process not freeing memory In-Reply-To: References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> Message-ID: <1379432784.58826.YahooMailNeo@web162805.mail.bf1.yahoo.com> Hi, I think you need to set the fullsweep_after option to 0, for example: ?spawn_opt(Node,?M,?F,?[Args,self(),File],[{fullsweep_after,0}]) for it to happen immediately. Otherwise it will happen some time later. Regarding: ?nodes would be killed by going out of memory without first > having had the chance to run the global GC. I also ran into this problem and had to manually insert erlang:garbage_collect() at strategic places in my code for my application to work. I would prefer an option to say that "aggressive" GC should be done before terminating the node. JOhan >________________________________ > From: Lukas Larsson >To: Robert Virding >Cc: Erlang Questions Mailing List >Sent: Tuesday, September 17, 2013 5:03 PM >Subject: Re: [erlang-questions] Problem with Beam process not freeing memory > > > >as far as I can see erlang:garbage_collect/0,1 does do fullsweeps [1]. > >?? [1]: https://github.com/erlang/otp/blob/maint/erts/emulator/beam/bif.c#L3770 > > > > >On Tue, Sep 17, 2013 at 4:53 PM, Robert Virding wrote: > >OK, I meant doing an explicit garbage collectionen and was under the impression that calling erlang:garbage_collect() actually did a full sweep of the process which is why hibernating seemed a bit of an overkill. >> >>Rober >> >> >>----- Original Message ----- >>> From: "Fred Hebert" >> >>> To: "Robert Virding" >>> Cc: "Erlang Questions Mailing List" >> >>> Sent: Tuesday, 17 September, 2013 4:36:19 PM >>> Subject: Re: [erlang-questions] Problem with Beam process not freeing memory >>> >>> Hi Robert, >>> >>> I CC'd the mailing list on this post, because I felt it could be >>> interesting to share with everyone. >>> >>> To get into some more details, the difference between a >>> garbage-collection and hibernating is that hibernation forces a >>> full-sweep and does compaction work. It is more likely to actually >>> remove old refs to binaries. >>> >>> The tricky part about binary leaks in these cases is that if the process >>> you're garbage collecting holds some very long-lived references (or >>> takes a long while before enabling them), you will move the references >>> to the old heap, if my understanding is correct. Now if your process >>> that leaks resources is busy or bogged down by some task, there are >>> chances that manually calling GC at higher frequencies will force >>> short-lived references to the old heap. ?Eventually, most of the >>> subsequent GCs are done for no good reason until there's a full sweep to >>> free the references, if my understanding is right. >>> >>> In comparison, some hibernations may turn out to be beneficial due to >>> how they do the full sweep, especially on less active processes, without >>> changing spawn_opt values or whatever. >>> >>> Both cases are not necessarily great, and I don't think there's one easy >>> way to deal with it. >>> >>> One thing I've had in mind to try for a while was to run a function a >>> bit like: >>> >>> -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer(). >>> gc_count(PreviousCounter, Bin) -> >>> ? ? case byte_size(Bin) of >>> ? ? ? ? N when N >= 64 -> % refc binary >>> ? ? ? ? ? ? Count = N + PreviousCounter >>> ? ? ? ? ? ? case NewCount >= ?THRESHOLD of >>> ? ? ? ? ? ? ? ? true -> >>> ? ? ? ? ? ? ? ? ? ? erlang:garbage_collect(), >>> ? ? ? ? ? ? ? ? ? ? 0; >>> ? ? ? ? ? ? ? ?false -> >>> ? ? ? ? ? ? ? ? ? ?NewCount >>> ? ? ? ? ? ? end; >>> ? ? ? ? N -> % heap binary >>> ? ? ? ? ? ? PreviousCounter+N >>> ? ? end. >>> >>> that could alternatively force some hibernation instead of GC'ing. This >>> one could basically track the size of all binaries seen manually and >>> force something when you go over a certain amount. It sucks, though, >>> because that's basically manually doing your collection, and it doesn't >>> mean that because you have seen a binary, it's ready to be GC'd. I've >>> thus avoided trying it in the real world for now. >>> >>> In practice, at Heroku, we've decided to go for a hybrid approach in >>> logplex. We force hibernation on some important events that interrupt >>> our work flow no matter what (such as a socket disconnection, or long >>> periods of time [seconds] without activity for a process), and have put >>> a workaround in place to force VM-wide GCs when we're reaching critical >>> amounts of memory: >>> https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl >>> >>> The objective was to use global GC as a last measure in case individual >>> (unobtrusive) hibernates were not enough to save a node. >>> >>> This later on prompted for exploring the allocators of the VM -- the >>> value used (erlang:memory(total)) didn't represent the OS-imposed limits >>> on the VM: nodes would be killed by going out of memory without first >>> having had the chance to run the global GC. This lead to discovering >>> things about fragmentation and characterizing our workloads to pick >>> better allocation strategies that seem to work decently so far, so that >>> erlang:memory(total), for one, has the right values, and also that we >>> have a better time releasing allocated blocks of memory when most >>> binaries vanish. >>> >>> I hope we can remove both the artificial hibernation calls and the >>> workarounds to force some global GCs in the near future. Ideally, it >>> sounds like the VM should possibly do more when it comes to the weight >>> of refc binaries to individual processes' memory for GC, but I don't >>> have a good idea of how this should be done in practice without having >>> elephant-sized assumptions and holes in the solution without adding more >>> knobs to the VM to configure things. Plus I'd have no idea on how to >>> actually implement it. >>> >>> Regards, >>> Fred. >>> >>> >>> On 09/17, Robert Virding wrote: >>> > Hi Fred, >>> > >>> > You recommend hibernating a process. Do you think this is better than >>> > calling the garbage collector in a process? I have no idea but hibernating >>> > seems more drastic, especially if the process is "in use"? >>> > >>> > Robert >>> > >>> > ----- Original Message ----- >>> > > From: "Fred Hebert" >>> > > To: "Tino Breddin" >>> > > Cc: "Erlang Questions Mailing List" >>> > > Sent: Tuesday, 17 September, 2013 2:39:41 AM >>> > > Subject: Re: [erlang-questions] Problem with Beam process not freeing >>> > > memory >>> > > >>> > > I've recently run in similar issues and have received a bit of help from >>> > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me, >>> > > I tried to put into the recon library, currently on a different branch >>> > > awaiting review and prone to change: >>> > > https://github.com/ferd/recon/tree/allocators >>> > > >>> > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)` >>> > > where `N` is the number of 'highest results' you want. The function will >>> > > take a snapshot of the number of binary refs in your processes, then GC >>> > > the node entirely, and then take another snapshot, make diff, and return >>> > > the N biggest deltas in binaries. This will let you know what processes >>> > > hold the most references to stale refc binaries. I recommend hibernation >>> > > as a first way to try and fix this if it is the problem. >>> > > >>> > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)` >>> > > where `Arg` can be: >>> > > ?- `used` for the memory actively used (i.e. erlang:memory(total)) >>> > > ?- `allocated` for the memory reserved by individual allocators (sum) >>> > > ?- `usage` for the percentage. >>> > > If the result of `allocated` is close to what the OS reports, you >>> > > probably have fragmentation issues. If not, you may have a NIF or driver >>> > > that allocates data outside of the ERTS allocators >>> > > >>> > > 3. check individual allocator usage levels with >>> > > `recon_alloc:fragmentation(current)`. It will return usage percentages >>> > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes >>> > > by default. When the data allocated is too large (> than the single >>> > > block carrier threshold [sbct]), it goes into its own block. Compare the >>> > > results with what you get with `recon_alloc:fragmentation(max)`. If the >>> > > current values have very low usage but the max ones have large ones, you >>> > > may have lingering data, possibly held in long-term references or >>> > > whatever that blocks deallocation of specific carriers. Different >>> > > carrier strategies can help, which we can dive into if you see a problem >>> > > with this. >>> > > >>> > > Feel free to read the comments in `recon_alloc` until I actually merge >>> > > it in master, they contain some of the details about what to do or look >>> > > for. >>> > > >>> > > Lukas may want me to correct me on the content of this post. I'm going >>> > > from the limited knowledge he transmitted to me here, or rather, my >>> > > limited understanding of it :) >>> > > >>> > > Regards, >>> > > Fred. >>> > > >>> > > On 09/16, Tino Breddin wrote: >>> > > > Hi list, >>> > > > >>> > > > I'm experiencing issues with a couple of Beam nodes where I see a huge >>> > > > gap >>> > > > between the node's reported memory usage and the underlying Linux >>> > > > Kernel's >>> > > > view. >>> > > > >>> > > > This is using R15B01. >>> > > > >>> > > > As a start an application in such a node stores a lot of tuples >>> > > > (containing >>> > > > atoms and binary data) in ETS tables. That proceeds until a point where >>> > > > memory usage is 70% (6GB) of the available memory. At that point >>> > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the >>> > > > memory >>> > > > usage. Then an internal cleanup task is performed, which clears >>> > > > obsolete >>> > > > records from the ETS tables. Afterwards, erlang:memory() reports an >>> > > > expected low value of roughly 60MB memory usage. (This includes binary >>> > > > data). However, the kernel still reports the high memory usage values >>> > > > (both >>> > > > VmRss and VmTotal) for the node. The kernel's usage view will stay >>> > > > stable >>> > > > until the ETS tables are filled to a point where the real memory usage >>> > > > exceeds the kernel's view, then the kernel reported usage will grow as >>> > > > well. >>> > > > >>> > > > Now having checked the node in some details I'm wondering what causes >>> > > > this >>> > > > difference between the BEAM's view and the Kernel's view on memory >>> > > > usage. I >>> > > > have 2 ideas which I'm checking right now. >>> > > > >>> > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed >>> > > > because >>> > > > the original dispatcher process which it was passed through before >>> > > > being >>> > > > stored in an ETS table is still alive. Thus there is still some >>> > > > reference >>> > > > to it? However, this would not explain why erlang:memory() reports a >>> > > > very >>> > > > low value for used memory for binaries. >>> > > > >>> > > > (2) low-level memory leak: Some driver or NIF leaking memory, which >>> > > > would >>> > > > obviously not be reported by erlang:memory(). However, then it >>> > > > surprises me >>> > > > that the Kernel's view stays stable while the BEAM's actual memory >>> > > > usage is >>> > > > still below the Kernel's view. It should be continuously growing in >>> > > > this >>> > > > case imho. >>> > > > >>> > > > I'd appreciate if anyone has some more insight or experience with such >>> > > > a >>> > > > behaviour, while I'm further digging into this. >>> > > > >>> > > > Cheers, >>> > > > Tino >>> > > >>> > > > _______________________________________________ >>> > > > erlang-questions mailing list >>> > > > erlang-questions@REDACTED >>> > > > http://erlang.org/mailman/listinfo/erlang-questions >>> > > >>> > > _______________________________________________ >>> > > erlang-questions mailing list >>> > > erlang-questions@REDACTED >>> > > http://erlang.org/mailman/listinfo/erlang-questions >>> > > >>> >>_______________________________________________ >>erlang-questions mailing list >>erlang-questions@REDACTED >>http://erlang.org/mailman/listinfo/erlang-questions >> > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.l.france@REDACTED Tue Sep 17 18:10:10 2013 From: aaron.l.france@REDACTED (Aaron France) Date: Tue, 17 Sep 2013 18:10:10 +0200 Subject: [erlang-questions] PropEr and the symbolic variables In-Reply-To: <1379432784.58826.YahooMailNeo@web162805.mail.bf1.yahoo.com> References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> <1379432784.58826.YahooMailNeo@web162805.mail.bf1.yahoo.com> Message-ID: <52387EE2.7000205@gmail.com> Hi all, I'm creating a statem for a stateful system and am trying to model the various API calls. Essentially the system is a PUT/DELETE rest service, you can PUT data which is forevermore available via a further GET call and you can also DELETE said data and subsequent GETs will fail. This works well, mostly. However, I am not able to satisfactorily maintain what the state of the system is due to receiving symbolic variables in my next state function. Basically I am recording the state as: state{ dict() } and the dict contains: { username => boolean() }. Where the boolean represents whether that user has data which is in the system. The problem comes from the fact that my API calls return a status code for how the API call succeeded. Imagine deleting an already deleted piece of data, it would return e.g. a 404 instead of a 204. I model the state accordingly. When I receive the `Res` argument to my next_state/3 function, it is only sometimes a concrete value which I need. However, I need to change the state conditionally according to what that `Res` value is. Any ideas? Aaron From magnus@REDACTED Tue Sep 17 18:52:54 2013 From: magnus@REDACTED (Magnus Henoch) Date: Tue, 17 Sep 2013 17:52:54 +0100 Subject: [erlang-questions] PropEr and the symbolic variables In-Reply-To: <52387EE2.7000205@gmail.com> (Aaron France's message of "Tue, 17 Sep 2013 18:10:10 +0200") References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> <1379432784.58826.YahooMailNeo@web162805.mail.bf1.yahoo.com> <52387EE2.7000205@gmail.com> Message-ID: <87hadj9yqh.fsf@mia.config> Aaron France writes: > Hi all, > > I'm creating a statem for a stateful system and am trying to model the > various API calls. > > Essentially the system is a PUT/DELETE rest service, you can PUT data > which is forevermore available via a further GET call and you can also > DELETE said data and subsequent GETs will fail. > > This works well, mostly. However, I am not able to satisfactorily > maintain what the state of the system is due to receiving symbolic > variables in my next state function. > > Basically I am recording the state as: > > state{ dict() } > > and the dict contains: > > { username => boolean() }. Where the boolean represents whether that > user has data which is in the system. > > The problem comes from the fact that my API calls return a status code > for how the API call succeeded. Imagine deleting an already deleted > piece of data, it would return e.g. a 404 instead of a 204. I model > the state accordingly. > > When I receive the `Res` argument to my next_state/3 function, it is > only sometimes a concrete value which I need. However, I need to > change the state conditionally according to what that `Res` value is. > > Any ideas? The basic idea is that since you're keeping track of the state in your property, you know whether the API call should succeed or fail even before you make the call. If the item you are deleting was in the dictionary before the call, then you should get a 204, otherwise you should get a 404. Something like: next_state(State = #state{existing_usernames = Usernames}, _Result, {call, _, delete, [Username]}) -> case lists:member(Username, Usernames) of true -> %% The user existed before deletion State#state{existing_usernames = Usernames -- [Username]}; false -> %% The user didn't exist before deletion State end. (In fact, in the example above you could unconditionally remove the user, since the -- operator does nothing if what you're trying to remove doesn't exist.) You only really care about the status code in the postcondition function, which is guaranteed to always get real return values, never symbolic ones: postcondition(#state{existing_usernames = Usernames}, {call, _, delete, [Username]}, StatusCode) -> case lists:member(Username, Usernames) of true -> StatusCode =:= 204; false -> StatusCode =:= 404 end. Hope this helps, Magnus From norton@REDACTED Tue Sep 17 19:01:40 2013 From: norton@REDACTED (Joseph Wayne Norton) Date: Wed, 18 Sep 2013 02:01:40 +0900 Subject: [erlang-questions] PropEr and the symbolic variables In-Reply-To: <52387EE2.7000205@gmail.com> References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> <1379432784.58826.YahooMailNeo@web162805.mail.bf1.yahoo.com> <52387EE2.7000205@gmail.com> Message-ID: Aaron - Here is a demonstration quickcheck statem test for a subset of the ets API - https://github.com/norton/qc/blob/master/test/demo/qc_statem_ets.erl The example is a demonstration of a wrapper over QuickCheck. This wrapper helps automate some common features and use cases for statem-style tests. Fortunately, the callback API is almost nearly the same as the QuickCheck callback API and the PropEr callback API. This example might be of help to you. Joe N. On 2013/09/18, at 1:10, Aaron France wrote: > Hi all, > > I'm creating a statem for a stateful system and am trying to model the various API calls. > > Essentially the system is a PUT/DELETE rest service, you can PUT data which is forevermore available via a further GET call and you can also DELETE said data and subsequent GETs will fail. > > This works well, mostly. However, I am not able to satisfactorily maintain what the state of the system is due to receiving symbolic variables in my next state function. > > Basically I am recording the state as: > > state{ dict() } > > and the dict contains: > > { username => boolean() }. Where the boolean represents whether that user has data which is in the system. > > The problem comes from the fact that my API calls return a status code for how the API call succeeded. Imagine deleting an already deleted piece of data, it would return e.g. a 404 instead of a 204. I model the state accordingly. > > When I receive the `Res` argument to my next_state/3 function, it is only sometimes a concrete value which I need. However, I need to change the state conditionally according to what that `Res` value is. > > Any ideas? > > Aaron > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From aaron.l.france@REDACTED Tue Sep 17 19:55:21 2013 From: aaron.l.france@REDACTED (Aaron France) Date: Tue, 17 Sep 2013 19:55:21 +0200 Subject: [erlang-questions] PropEr and the symbolic variables In-Reply-To: <87hadj9yqh.fsf@mia.config> References: <20130917003940.GD46479@ferdmbp.local> <2065671555.178907.1379426378461.JavaMail.zimbra@erlang-solutions.com> <20130917143618.GA13826@ferdair.local> <1102226760.180050.1379429620736.JavaMail.zimbra@erlang-solutions.com> <1379432784.58826.YahooMailNeo@web162805.mail.bf1.yahoo.com> <52387EE2.7000205@gmail.com> <87hadj9yqh.fsf@mia.config> Message-ID: <52389789.8050409@gmail.com> Ah yes! That's a great conceptual help. It's very much more 'Zen' to trust in the postcondition to do the right thing and to assume in the next_state function. Thanks! On 17/09/13 18:52, Magnus Henoch wrote: > Aaron France writes: > >> Hi all, >> >> I'm creating a statem for a stateful system and am trying to model the >> various API calls. >> >> Essentially the system is a PUT/DELETE rest service, you can PUT data >> which is forevermore available via a further GET call and you can also >> DELETE said data and subsequent GETs will fail. >> >> This works well, mostly. However, I am not able to satisfactorily >> maintain what the state of the system is due to receiving symbolic >> variables in my next state function. >> >> Basically I am recording the state as: >> >> state{ dict() } >> >> and the dict contains: >> >> { username => boolean() }. Where the boolean represents whether that >> user has data which is in the system. >> >> The problem comes from the fact that my API calls return a status code >> for how the API call succeeded. Imagine deleting an already deleted >> piece of data, it would return e.g. a 404 instead of a 204. I model >> the state accordingly. >> >> When I receive the `Res` argument to my next_state/3 function, it is >> only sometimes a concrete value which I need. However, I need to >> change the state conditionally according to what that `Res` value is. >> >> Any ideas? > The basic idea is that since you're keeping track of the state in your > property, you know whether the API call should succeed or fail even > before you make the call. If the item you are deleting was in the > dictionary before the call, then you should get a 204, otherwise you > should get a 404. Something like: > > next_state(State = #state{existing_usernames = Usernames}, > _Result, > {call, _, delete, [Username]}) -> > case lists:member(Username, Usernames) of > true -> > %% The user existed before deletion > State#state{existing_usernames = Usernames -- [Username]}; > false -> > %% The user didn't exist before deletion > State > end. > > (In fact, in the example above you could unconditionally remove the > user, since the -- operator does nothing if what you're trying to remove > doesn't exist.) > > You only really care about the status code in the postcondition > function, which is guaranteed to always get real return values, never > symbolic ones: > > postcondition(#state{existing_usernames = Usernames}, > {call, _, delete, [Username]}, > StatusCode) -> > case lists:member(Username, Usernames) of > true -> > StatusCode =:= 204; > false -> > StatusCode =:= 404 > end. > > Hope this helps, > Magnus From matt@REDACTED Wed Sep 18 01:33:50 2013 From: matt@REDACTED (Matt Lewandowsky) Date: Tue, 17 Sep 2013 16:33:50 -0700 Subject: [erlang-questions] Cross-compiling tests with quoted ENV vars Message-ID: Hello, I am attempting to cross-compile OTP for a new platform, with a new $ERL_TOP/xcomp/erl-xcomp-*.conf file. Unfortunately, I've hit an interesting issue and it appears the only other xcomp platform which would encounter it is VxWorks, which I do not have access to. In my xcomp conf file, I use quoted values for CFLAGS and other environment variables. These get re-double-quoted by ts_install.erl as such: Env: [{"CFLAGS", "\"-mcpu=cortex-a9 -Wno-psabi -fstack-protector-all -D_FORTIFY_SOURCE=2 -O2 -Wformat -Wformat-security -Werror=format-security\""}, As the VxWorks target also has quoted variables, how do the tests get compiled for that platform since the documented procedure in the "Testing the cross compiled system" section of the User's Guide *should* fail, per the results I'm seeing? Are tests not cross-compiled by anyone, ever, for that platform? Is there some undocumented magic I'm missing? Is this a legitimate bug I should fix while I trod down this obviously lightly-used path? Thanks, --Matt P.S. As there doesn't seem to be any naming convention for configuration files in the xcomp directory and not much to work from, are there recommendations for the configuration file I'm working on? The host triplet is long, so would "something appropriate" be a good enough choice? I expect a > 40 character filename would be balked at. :) -- Matt Lewandowsky Big Geek Greenviolet matt@REDACTED http://www.greenviolet.net +1 415 578 5782 (US) +44 844 484 8254 (UK) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6621 bytes Desc: not available URL: From karol.urbanski@REDACTED Wed Sep 18 13:33:15 2013 From: karol.urbanski@REDACTED (Karol Urbanski) Date: Wed, 18 Sep 2013 13:33:15 +0200 Subject: [erlang-questions] R16B02 binary packages for CentOS, Debian, Mac, Ubuntu and Windows Message-ID: <20130918113315.GA1449@dex.erlang-solutions.com> Hello, You can find our binary packages for the newest Erlang release (R16B02) at http://www.erlang-solutions.com/downloads/download-erlang-otp The systems we provide the packages for are CentOS 6, Mac OS X Snow Leopard and newer, Debian 6.0.3 and 7.0.0, Ubuntu 13.04 + 12.10 + 12.04 and Windows. More to come soon! Best regards, Karol Urbanski From kenneth.lundin@REDACTED Wed Sep 18 16:46:01 2013 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Wed, 18 Sep 2013 16:46:01 +0200 Subject: [erlang-questions] OTP R16B02 has been released Message-ID: Erlang/OTP R16B02 has been released. OTP R16B02 is a service release with mostly a number of small corrections and user contributions. But there are some new functions worth mentioning as well, here are some of them: - New erl option +SP to set schedulers by percentages. - A new test scope has been added which is meant to be used for making sure that the Erlang VM and other platform specific libraries have been compiled correctly for your system. - Added application:ensure_all_started/1-2. - ~40 open source contributions You can find the README file with more detailed info at http://www.erlang.org/download/otp_src_R16B02.readme You can download the full source distribution from http://www.erlang.org/download/otp_src_R16B02.tar.gz http://www.erlang.org/download/otp_src_R16B02.readme (this file) Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please read the README that is part of the distribution. You can also find this release at the official Erlang/OTP Git-repository at Github here: https://github.com/erlang/otp tagged *OTP_R16B02* The Windows binary distribution can be downloaded from http://www.erlang.org/download/otp_win32_R16B02.exe http://www.erlang.org/download/otp_win64_R16B02.exe On-line documentation can be found at http://www.erlang.org/doc/. You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_R16B02.tar.gz http://www.erlang.org/download/otp_doc_man_R16B02.tar.gz We also want to thank those that sent us patches, suggestions and bug reports, The Erlang/OTP Team at Ericsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin@REDACTED Thu Sep 19 06:44:43 2013 From: kevin@REDACTED (kevin@REDACTED) Date: Thu, 19 Sep 2013 00:44:43 -0400 Subject: [erlang-questions] Software Estimation and Progress Tracking Message-ID: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> Hello All, I recently wrote a blog post on software estimation and progress tracking for functional languages. Unfortunately, the post was not terribly substantive and posed more questions than it answered. Since the Erlang community seems to have a lot of good practical experience on various sizes of commercial projects, I wondered if anyone here had experience with estimation and progress, particularly as it differs (or perhaps doesn't) when using Erlang as opposed to imperative languages. If you have any thoughts, please respond on this list or better yet, just add a comment to the blog post: http://funceng.com/2013/09/18/estimation-and-progress/ Thanks, Kevin From bengt.kleberg@REDACTED Thu Sep 19 08:32:39 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 19 Sep 2013 08:32:39 +0200 Subject: [erlang-questions] how: xmerl export without " Message-ID: <1379572360.4801.8.camel@sekic1152.release> Greetings, I would like to get rid of " when xmerl export a structure. I have the following structure: [{bike, [{year, 2000}], [{name, ["Suzuki"]}]}] When I do lists:flatten( xmerl:export_simple_content(Bike, xmerl_xml) ). the result is "Suzuki" How should I do to get "Suzuki" bengt From lukas@REDACTED Thu Sep 19 09:23:00 2013 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 19 Sep 2013 09:23:00 +0200 Subject: [erlang-questions] Cross-compiling tests with quoted ENV vars In-Reply-To: References: Message-ID: Hello Matt, As you have guessed the cross compile support for tests is indeed a very lightly-used path and also quite new. Right now we run nightly cross-compile tests for x86_64-saf-linux-gnu and powerpc-dso-linux-gnu (from Linux x86) and have not attempted to get it to work on any other. As neither of these use quoted CFLAGS, that specific option has slipped through and what you have found is indeed a bug. A patch that fixes this would be very appreciated! Lukas On Wed, Sep 18, 2013 at 1:33 AM, Matt Lewandowsky wrote: > Hello, > > I am attempting to cross-compile OTP for a new platform, with a new > $ERL_TOP/xcomp/erl-xcomp-*.conf file. Unfortunately, I've hit an > interesting > issue and it appears the only other xcomp platform which would encounter > it is > VxWorks, which I do not have access to. > > In my xcomp conf file, I use quoted values for CFLAGS and other environment > variables. These get re-double-quoted by ts_install.erl as such: > > Env: [{"CFLAGS", > "\"-mcpu=cortex-a9 -Wno-psabi -fstack-protector-all > -D_FORTIFY_SOURCE=2 > -O2 -Wformat -Wformat-security -Werror=format-security\""}, > > As the VxWorks target also has quoted variables, how do the tests get > compiled > for that platform since the documented procedure in the "Testing the cross > compiled system" section of the User's Guide *should* fail, per the results > I'm seeing? Are tests not cross-compiled by anyone, ever, for that > platform? > Is there some undocumented magic I'm missing? Is this a legitimate bug I > should fix while I trod down this obviously lightly-used path? > > Thanks, > > --Matt > > P.S. As there doesn't seem to be any naming convention for configuration > files > in the xcomp directory and not much to work from, are there recommendations > for the configuration file I'm working on? The host triplet is long, so > would > "something appropriate" be a good enough choice? I expect a > 40 character > filename would be balked at. :) > > -- > Matt Lewandowsky > Big Geek > Greenviolet > matt@REDACTED http://www.greenviolet.net > +1 415 578 5782 (US) +44 844 484 8254 (UK) > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cvi@REDACTED Thu Sep 19 09:34:19 2013 From: cvi@REDACTED (=?ISO-8859-1?Q?Christoffer_Vikstr=F6m?=) Date: Thu, 19 Sep 2013 09:34:19 +0200 Subject: [erlang-questions] how: xmerl export without " In-Reply-To: <1379572360.4801.8.camel@sekic1152.release> References: <1379572360.4801.8.camel@sekic1152.release> Message-ID: As far as i know the xml standard doesn't allow for unquoted attribute values, which is probably why xmerl outputs it that way. No idea if you can game xmerl to output non-standard xml though. /Christoffer On Thu, Sep 19, 2013 at 8:32 AM, Bengt Kleberg wrote: > Greetings, > > I would like to get rid of " when xmerl export a structure. > > I have the following structure: > [{bike, [{year, 2000}], [{name, ["Suzuki"]}]}] > > When I do > lists:flatten( xmerl:export_simple_content(Bike, xmerl_xml) ). > > the result is > "Suzuki" > > How should I do to get > "Suzuki" > > > bengt > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samuelrivas@REDACTED Thu Sep 19 09:35:02 2013 From: samuelrivas@REDACTED (Samuel) Date: Thu, 19 Sep 2013 09:35:02 +0200 Subject: [erlang-questions] dialyzer issue with opaque types Message-ID: Hi, I have next two (trimmed down) modules I want to dialyze (attached as well) : -module(foo). -export([foo/0]). -spec foo() -> bar:id(). foo() -> bar:id_undef(). -------------------------------------------------- -module(bar). -export([id_undef/0]). -opaque undef() :: -1. -opaque id() :: undef(). -export_type([id/0, undef/0]). -spec id_undef() -> id(). id_undef() -> -1. ----------------------------------------------- Running dialyzer on those fails like this: $ erlc +debug_info *.erl && dialyzer *.beam Checking whether the PLT /home/samuel/local/var/dialyzer/plts/R16B02-src/otp.plt is up-to-date... yes Proceeding with analysis... foo.erl:5: Invalid type specification for function foo:foo/0. The success typing is () -> bar:id() done in 0m3.02s done (warnings were emitted) That is, dialyzer thinks that bar:id() is the bad result type and that I should be using bar:id() instead :) Making undef() not opaque seems to solve the issue. I have verified this in both R15B03 and R16B02 -- Samuel -------------- next part -------------- A non-text attachment was scrubbed... Name: foo.erl Type: application/octet-stream Size: 84 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bar.erl Type: application/octet-stream Size: 164 bytes Desc: not available URL: From hugo@REDACTED Thu Sep 19 09:42:02 2013 From: hugo@REDACTED (Hugo Mills) Date: Thu, 19 Sep 2013 08:42:02 +0100 Subject: [erlang-questions] how: xmerl export without " In-Reply-To: <1379572360.4801.8.camel@sekic1152.release> References: <1379572360.4801.8.camel@sekic1152.release> Message-ID: <20130919074202.GA15356@carfax.org.uk> On Thu, Sep 19, 2013 at 08:32:39AM +0200, Bengt Kleberg wrote: > Greetings, > > I would like to get rid of " when xmerl export a structure. > > I have the following structure: > [{bike, [{year, 2000}], [{name, ["Suzuki"]}]}] > > When I do > lists:flatten( xmerl:export_simple_content(Bike, xmerl_xml) ). > > the result is > "Suzuki" > > How should I do to get > "Suzuki" That's not well-formed XML -- XML *requires* the use of the quotes containing attribute values. I would be surprised if you could make a conformant XML processor do what you're asking. If you need to know that the value of the "year" attribute is an integer (or an xsd:gYear), then you should either hard-code that knowledge in your consumer (with an appropriate human-readable standard for anyone else reading the data format), or use a schema document to formalise it. Hugo. -- === Hugo Mills: hugo@REDACTED carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- The most exciting phrase to hear in science, the one that --- heralds new discoveries, is not "Eureka!", but "That's funny..." -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 828 bytes Desc: Digital signature URL: From mjtruog@REDACTED Thu Sep 19 10:06:44 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 19 Sep 2013 01:06:44 -0700 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> Message-ID: <523AB094.7030100@gmail.com> Some links that seem relevant are: http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for-motorola-telecoms-software http://www.erlang.org/faq/how_do_i.html#id55052 On 09/18/2013 09:44 PM, kevin@REDACTED wrote: > Hello All, > > I recently wrote a blog post on software estimation and progress tracking for functional languages. Unfortunately, the post was not terribly substantive and posed more questions than it answered. Since the Erlang community seems to have a lot of good practical experience on various sizes of commercial projects, I wondered if anyone here had experience with estimation and progress, particularly as it differs (or perhaps doesn't) when using Erlang as opposed to imperative languages. > > If you have any thoughts, please respond on this list or better yet, just add a comment to the blog post: > > http://funceng.com/2013/09/18/estimation-and-progress/ > > Thanks, > > Kevin > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ulf@REDACTED Thu Sep 19 10:31:20 2013 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 19 Sep 2013 10:31:20 +0200 Subject: [erlang-questions] how: xmerl export without " In-Reply-To: References: <1379572360.4801.8.camel@sekic1152.release> Message-ID: <7FEFF8A3-19E8-4FA3-A535-3E40883A431D@feuerlabs.com> On 19 Sep 2013, at 09:34, Christoffer Vikstr?m wrote: > As far as i know the xml standard doesn't allow for unquoted attribute values, which is probably why xmerl outputs it that way. No idea if you can game xmerl to output non-standard xml though. > > /Christoffer The xmerl export requires only that the output is text, and one can always write an export callback that modifies an existing callback. Here's one that seems to do roughly what Bengt wants (keeping the prolog unchanged, which may not matter, since the body is no longer proper XML): -module(xml_wo_quotes). -export(['#xml-inheritance#'/0]). -export(['#element#'/5]). '#xml-inheritance#'() -> [xmerl_xml]. '#element#'(Tag, Data, Attrs, Parents, E) -> strip(xmerl_xml:'#element#'(Tag, Data, Attrs, Parents, E)). strip([["<", Tag|T]|Rest]) -> ["<", Tag, strip_(T) | Rest]; strip(["<", Tag|T]) -> ["<", Tag | strip_(T)]; strip([H]) when is_list(H) -> strip(H). strip_(T) -> {Attrs, Rest} = take_attrs(T), [remove_quotes(Attrs), Rest]. take_attrs(L) -> take_attrs(L, []). take_attrs([H|_] = L, Acc) when H==">"; H=="/>" -> {lists:reverse(Acc), L}; take_attrs([H|T], Acc) -> take_attrs(T, [H|Acc]). remove_quotes(L) -> re:replace(L, "\\\"", "", [global]). Example: Eshell V5.9.2 (abort with ^G) 1> c(xml_wo_quotes). {ok,xml_wo_quotes} 2> xmerl_scan:string(""). {{xmlElement,foo,foo,[], {xmlNamespace,[],[]}, [],1, [{xmlAttribute,a,[],[],[],[{foo,1}],1,[],"1",false}, {xmlAttribute,b,[],[],[],[{foo,1}],2,[],"2",false}], [{xmlElement,bar,bar,[], {xmlNamespace,[],[]}, [{foo,1}], 1, [{xmlAttribute,a,[],[],[], [{bar,1},{foo,...}], 1,[], [...],...}], [],[],"/Users/uwiger/src",undeclared}], [],"/Users/uwiger/src",undeclared}, []} 3> xmerl:export([element(1,v(2))], xml_wo_quotes). ["", [["<","foo", [[<<" a=">>,[],<<"1">>,[],<<" b=">>,[],<<"2">>,[]],[">"]], [["<","bar",[<<" a=">>,[],<<"1">>,[]],["/>"]]], [""]]]] 4> io:fwrite("~s~n", [v(3)]). ok BR, Ulf Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From jesper.louis.andersen@REDACTED Thu Sep 19 11:59:00 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 19 Sep 2013 11:59:00 +0200 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> Message-ID: On Thu, Sep 19, 2013 at 6:44 AM, wrote: > I recently wrote a blog post on software estimation and progress tracking > for functional languages. Unfortunately, the post was not terribly > substantive and posed more questions than it answered. Since the Erlang > community seems to have a lot of good practical experience on various sizes > of commercial projects, I wondered if anyone here had experience with > estimation and progress, particularly as it differs (or perhaps doesn't) > when using Erlang as opposed to imperative languages. The underlying problem of software estimation is this: "Please estimate how many years it takes to make a fusion reactor from now, solving all power problems on the earth in one fell swoop." The thing is, you can't. People understand this to be research and hence they have no qualms at throwing effort at the project in the hope it brings you closer to the ultimate goal, albeit slowly. Software is not "construction". It turns out to be "research". Regarding it as "construction" by imposing estimates and progress tracking is usually futile due to this dichotomy and the fact people missed it was "research". So, will the research happen faster in Erlang than in other languages? This is hard to answer without being concrete. My loose experience is that languages working at higher level in general provides shorter paths to enlightenment[0]. Since functional languages are often working at a higher level there is a considerable overlap. But don't underestimate something like Python, Ruby or Javascript which also works at the same high level. It is not always the case that you will be faster due to the trait of being functional. Compared to "traditional" languages like C or C++, the lack of proper garbage collection is the key thing which make you research faster. Compared to Java, it is the succinctness of Erlang. Compared to many other systems, like Python or Ruby, it is the toolset in Erlang to describe concurrency and the ability to utilize the machine via parallelism. Compared to Golang, it is probably the process isolation and the ability to run large complex systems with hundreds of applications in the same memory space without panics[1]. Bringing it back to estimation, I fear that we will just adapt and expect more features to delivered in shorter time, so our estimation of progress and effort will still be off, even if we get things done faster. For some reason, there is a bias toward "fast delivery" with no regard toward "quality of product". This is a depressing work environment, though: every deliverable will be too late, always. And this leads to a psychological phobia of estimation :) [0] Since software is research, the goal is to seek enlightenment and understand something deeper than before. [1] Though this is a hunch, not experience. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Sep 19 12:19:16 2013 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 19 Sep 2013 12:19:16 +0200 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> Message-ID: <6A9675D6-9999-4E31-B06B-759715CF8097@feuerlabs.com> On 19 Sep 2013, at 11:59, Jesper Louis Andersen wrote: > Software is not "construction". It turns out to be "research". Regarding it as "construction" by imposing estimates and progress tracking is usually futile due to this dichotomy and the fact people missed it was "research". Yes, but in my experience, this is how most research funding is determined as well: "We estimate that we will invent X in month 47 of the project, after which we will discover Y based on X in month 64, having spent N number of man-months in the process." BR, Ulf Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Thu Sep 19 12:22:17 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 19 Sep 2013 12:22:17 +0200 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: <6A9675D6-9999-4E31-B06B-759715CF8097@feuerlabs.com> References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> <6A9675D6-9999-4E31-B06B-759715CF8097@feuerlabs.com> Message-ID: On Thu, Sep 19, 2013 at 12:19 PM, Ulf Wiger wrote: > Yes, but in my experience, this is how most research funding is determined > as well: "We estimate that we will invent X in month 47 of the project, > after which we will discover Y based on X in month 64, having spent N > number of man-months in the process." Indeed. Which is why researchers do the research first and put it in a drawer. Then go seek the funding for the research. Thus they hit a 100% success rate, which is impressive. And it makes it consistent so they will get more funding for more research. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v@REDACTED Thu Sep 19 13:54:31 2013 From: v@REDACTED (Valentin Micic) Date: Thu, 19 Sep 2013 13:54:31 +0200 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> Message-ID: <6C9655B6-84AB-4390-B53C-24EFF3061A10@pharos-avantgard.com> Well, if you do not know what you're doing, then you can just as well call it a research. On 19 Sep 2013, at 11:59 AM, Jesper Louis Andersen wrote: > > On Thu, Sep 19, 2013 at 6:44 AM, wrote: > I recently wrote a blog post on software estimation and progress tracking for functional languages. Unfortunately, the post was not terribly substantive and posed more questions than it answered. Since the Erlang community seems to have a lot of good practical experience on various sizes of commercial projects, I wondered if anyone here had experience with estimation and progress, particularly as it differs (or perhaps doesn't) when using Erlang as opposed to imperative languages. > > The underlying problem of software estimation is this: > > "Please estimate how many years it takes to make a fusion reactor from now, solving all power problems on the earth in one fell swoop." > > The thing is, you can't. People understand this to be research and hence they have no qualms at throwing effort at the project in the hope it brings you closer to the ultimate goal, albeit slowly. > > Software is not "construction". It turns out to be "research". Regarding it as "construction" by imposing estimates and progress tracking is usually futile due to this dichotomy and the fact people missed it was "research". > > So, will the research happen faster in Erlang than in other languages? This is hard to answer without being concrete. My loose experience is that languages working at higher level in general provides shorter paths to enlightenment[0]. Since functional languages are often working at a higher level there is a considerable overlap. But don't underestimate something like Python, Ruby or Javascript which also works at the same high level. It is not always the case that you will be faster due to the trait of being functional. > > Compared to "traditional" languages like C or C++, the lack of proper garbage collection is the key thing which make you research faster. Compared to Java, it is the succinctness of Erlang. Compared to many other systems, like Python or Ruby, it is the toolset in Erlang to describe concurrency and the ability to utilize the machine via parallelism. Compared to Golang, it is probably the process isolation and the ability to run large complex systems with hundreds of applications in the same memory space without panics[1]. > > Bringing it back to estimation, I fear that we will just adapt and expect more features to delivered in shorter time, so our estimation of progress and effort will still be off, even if we get things done faster. For some reason, there is a bias toward "fast delivery" with no regard toward "quality of product". This is a depressing work environment, though: every deliverable will be too late, always. And this leads to a psychological phobia of estimation :) > > [0] Since software is research, the goal is to seek enlightenment and understand something deeper than before. > [1] Though this is a hunch, not experience. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Sep 19 14:03:52 2013 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 19 Sep 2013 14:03:52 +0200 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> Message-ID: Once upon a very long time ago we did a project to compare the efficiency of Erlang to PLEX. We implemented "the same things" (TM) in Erlang and PLEX and counted total man hours We did this for several different things. Erlang was "better" by a factor of 3 or 25 (in total man hours) - the weighted average was a factor 8 They asked "what is the smart programmer effect" We said "we don't know" We revised the figure 8 down to 3 to allow for "the smart programmer effect" - this was too high to be credible, so we revised it down to 1.6. (the factors 3 and 1.6 where just plucked out of the air with no justification) Experiments that show that Erlang is N times better than "something else" won't be believed if N is too high. The second point to remember is that you *never* implement exactly the same thing in two different languages (or very rarely) - the second time you do something you have presumably learnt from the mistakes made the first time you do something. If you implement the same thing N times in the same language, each implementation should take less effort and code than the last time you did it. What can you learn from this? The difference in programmer productivity can vary by a factor of 80 - (really it's infinity, because some programmers *never* get some code right, so the factor 80 discounts the totally failed efforts) - So given a productivity factor you have to normalize it by a factor that depends upon the skill and experience of the programmer. There are people who claim that they can make models estimating how long a software projects take. But even they say that such models have to be tuned, and are only applicable to projects which are broadly similar. After you've done almost the same thing half a dozen times it might be possible to estimate how long a similar project might take. The problem is we don't do similar things over and over again. Each new unsolved problem is precisely that, a new unsolved problem. Most time isn't spent programming anyway - programmer time is spent: a) fixing broken stuff that should not be broken b) trying to figure out what problem the customer actually wants solving c) writing experimental code to test some idea d) googling for some obscure fact that is needed to solve a) or b) e) writing and testing production code e) is actually pretty easy once a) - d) are fixed. But most measurements of productivity only measure lines of code in e) and man hours. I've been in this game for many years now, and I have the impression that a) is taking a larger and larger percentage of my time. 30 years ago there was far less software, but the software there was usually worked without any problems - the code was a lot smaller and consequently easier to understand. Again in the last 30 years programs have got hundreds to thousands of times larger (in terms of code lines) but programming languages haven't got that much better and our brains have not gotten any smarter. So the gap between what we can build and what we can understand is growing rapidly. Extrapolating a bit I guess a) is going to increase - so in a few years we'll have incredibly smart devices which almost work, and when broke nobody will able to fix, and programmers will spend 100% of their time fixing broken stuff that should not be broken. And no I have to figure out why firefox has suddenly stopped working - something is broken ... Cheers /Joe On Thu, Sep 19, 2013 at 6:44 AM, wrote: > Hello All, > > I recently wrote a blog post on software estimation and progress tracking > for functional languages. Unfortunately, the post was not terribly > substantive and posed more questions than it answered. Since the Erlang > community seems to have a lot of good practical experience on various sizes > of commercial projects, I wondered if anyone here had experience with > estimation and progress, particularly as it differs (or perhaps doesn't) > when using Erlang as opposed to imperative languages. > > If you have any thoughts, please respond on this list or better yet, just > add a comment to the blog post: > > http://funceng.com/2013/09/18/**estimation-and-progress/ > > Thanks, > > Kevin > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahesh@REDACTED Thu Sep 19 15:30:42 2013 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Thu, 19 Sep 2013 06:30:42 -0700 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: <6A9675D6-9999-4E31-B06B-759715CF8097@feuerlabs.com> References: <20a1b0d33a7f851ce08974ce8e4842a5@funceng.com> <6A9675D6-9999-4E31-B06B-759715CF8097@feuerlabs.com> Message-ID: <7ACE140E-F254-497B-928D-82324B15D504@dieswaytoofast.com> There are - at least - four orthogonal areas in which your software gets developed, each of which has different metrics for estimation, tracking, progress, etc., etc., etc. To throw some semantics at this 1) Technical When the specifics of the solution are clear, and it pretty much boils down to implementation. "I need to replace the valve on my Hot Tub" (With the same model, I've already done it once before, etc., etc.) 2) Engineering When you need to solve the problem first, before implementing the solution. There _is_ a body of kno "I need to install my Hot Tub" (Hmmm. There is no water line going out there. How do I get one? What about the electrical circuit? Do I need architectural permission? etc.) 3) Science You need to invent a new class of solutions for the problem at hand. "I need to install my Hot Tub in an underground bunker in the marshlands of Florida" (How do I build an underground bunker in the marsh? Maybe I can freeze the ground and pour concrete? How do I keep the concrete from sinking? Hmmm. Time to start running experiments?" 4) Art You need to get the intangibles correct, viz., is it maintainable? Supportable? Documented? Elegant? "Will some future Significant Other like my paranoid underground-bunker hot-tub?" Note that each of the above are _different_. - Its fairly easy to Manage / Maintain / Monitor "Technical" work. - There is a reasonable body of knowledge that helps in doing the same for "Engineering" stuff. - For Science, its all pretty clearly made-up (the fusion reactor!), - For "Art", well, it really _is_ in the eye of the beholder. (Good Documentation? For whom? What do you mean by "Good"? *I* understand it! And so does Jane!) Which brings me back to the original point - much as we would like it to be that way, software pretty much never fits neatly into one of the buckets above - it is some combination of the four, with different parameters for {T, E, S, A}. Whats worse, there is a time-variant aspect to this too - and the parameters are inter-related. e.g., different "Engineering" solutions have different "Technical" impacts. In short, your development process is actually f(T, E, S, A, Time) All this being said, there is a time-honored Academic way of solving f(T, E, S, A, Time), which basically consists of wishing-away the unknowns (or make unrealistic assumptions about them) and then spend an in-ordinate amount of time on the remaining parameters. With some appropriate tweaking of parameters, one can quite successfully make this match some "real-world" results, which can then be trumpeted widely. Any failures can be blamed on the actors (ho-ho. "Actors". In an erlang post. I crack myself up.) who were clearly not qualified.. This may very well be the best option? Cheers Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Google+ | Blog | Twitter | LinkedIn On Sep 19, 2013, at 3:19 AM, Ulf Wiger wrote: > > On 19 Sep 2013, at 11:59, Jesper Louis Andersen wrote: > >> Software is not "construction". It turns out to be "research". Regarding it as "construction" by imposing estimates and progress tracking is usually futile due to this dichotomy and the fact people missed it was "research". > > Yes, but in my experience, this is how most research funding is determined as well: "We estimate that we will invent X in month 47 of the project, after which we will discover Y based on X in month 64, having spent N number of man-months in the process." > > BR, > Ulf > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Sep 19 16:36:04 2013 From: g@REDACTED (Garrett Smith) Date: Thu, 19 Sep 2013 09:36:04 -0500 Subject: [erlang-questions] Suggested articles / blog posts on "Essential Erlang" Message-ID: I'm preparing a talk for the Chicago Erlang conference [1] and I'm looking for a bit of inspiration. The topic is "Essential Erlang". I've got my own ideas here, but I'd like to conduct a survey of excellent articles, blog posts, etc. that others have seen. I'm looking for content that gets to the heart of a particular Erlang topic or the Tao of Erlang and its ecosystem (not, e.g. a language tutorial or reference). If you can point me to something, I'd be most appreciative. As a returned favor, I'll compile a final list of suggestions and post it for posterity. Garrett [1] http://www.chicagoerlang.com From mononcqc@REDACTED Thu Sep 19 16:58:50 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Thu, 19 Sep 2013 10:58:50 -0400 Subject: [erlang-questions] Suggested articles / blog posts on "Essential Erlang" In-Reply-To: References: Message-ID: <20130919145849.GD13826@ferdair.local> I think Jesper (jlouis) has plenty of great articles and blog posts that I would categorize as very useful when it comes to "thinking in Erlang": http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-scheduling.html http://jlouisramblings.blogspot.com/2012/06/protocols-in-kingdom-of-distribution.html http://jlouisramblings.blogspot.com/2012/05/more-on-erlang-and-state.html http://jlouisramblings.blogspot.com/2011/07/erlangs-parallelism-is-not-parallelism.html http://jlouisramblings.blogspot.com/2010/11/on-erlang-state-and-crashes.html I also tend to like reading Paolo D'Incau's blog on all things Erlang, and relatively frequent interviews with members of the community: http://pdincau.wordpress.com/ There's plenty of other blogs, but those are the one that came quickly to mind outside of some list I once had written somewhere. On 09/19, Garrett Smith wrote: > I'm preparing a talk for the Chicago Erlang conference [1] and I'm > looking for a bit of inspiration. The topic is "Essential Erlang". > I've got my own ideas here, but I'd like to conduct a survey of > excellent articles, blog posts, etc. that others have seen. > > I'm looking for content that gets to the heart of a particular Erlang > topic or the Tao of Erlang and its ecosystem (not, e.g. a language > tutorial or reference). > > If you can point me to something, I'd be most appreciative. > > As a returned favor, I'll compile a final list of suggestions and post > it for posterity. > > Garrett > > [1] http://www.chicagoerlang.com > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mihai@REDACTED Thu Sep 19 17:04:50 2013 From: mihai@REDACTED (Mihai Balea) Date: Thu, 19 Sep 2013 11:04:50 -0400 Subject: [erlang-questions] R16B01 monitor delivery bug fixed in R16B02 Message-ID: <2CD6A1F4-A238-4E4F-A0B7-589A0A484017@hates.ms> Hi all, A while ago there was a short dialog on this list about a bug in monitor delivery in R16B01: http://erlang.org/pipermail/erlang-bugs/2013-July/003670.html Was that fixed in R16B02? I couldn't find anything explicit in the release notes, the one that comes closest is: OTP-11225 Fix serious race bug in R16B01 that could cause PID mix-ups when a lot of processes were spawned and terminated in a very rapid pace on an SMP emulator with at least two scheduler threads. Thank you, Mihai From sverker.eriksson@REDACTED Thu Sep 19 17:10:55 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 19 Sep 2013 17:10:55 +0200 Subject: [erlang-questions] R16B01 monitor delivery bug fixed in R16B02 In-Reply-To: <2CD6A1F4-A238-4E4F-A0B7-589A0A484017@hates.ms> References: <2CD6A1F4-A238-4E4F-A0B7-589A0A484017@hates.ms> Message-ID: <523B13FF.5080703@erix.ericsson.se> Yes, that's the fix. /Sverker Erlang/OTP, Ericsson On 09/19/2013 05:04 PM, Mihai Balea wrote: > Hi all, > > A while ago there was a short dialog on this list about a bug in monitor delivery in R16B01: > > http://erlang.org/pipermail/erlang-bugs/2013-July/003670.html > > Was that fixed in R16B02? I couldn't find anything explicit in the release notes, the one that comes closest is: > > OTP-11225 Fix serious race bug in R16B01 that could cause PID > mix-ups when a lot of processes were spawned and terminated > in a very rapid pace on an SMP emulator with at least two > scheduler threads. > > Thank you, > Mihai > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From sevenjp@REDACTED Thu Sep 19 19:56:23 2013 From: sevenjp@REDACTED (=?UTF-8?Q?Jo=C3=A3o_Neves?=) Date: Thu, 19 Sep 2013 19:56:23 +0200 Subject: [erlang-questions] [ANN] iota - a tool for enforcing clean separation of responsibilities Message-ID: Did you ever wish you could tell someone "just because I'm exporting this it doesn't mean you should be using it" through your code? iota is a tool (and a library) that uses Erlang attributes to allow you to specify application-level APIs implicitly by declaring what exported functions in each module compose the overall API external applications should use to interact with yours, with more checks coming soon(TM). iota was mostly inspired by the Kappa tool used internally at Klarna and mentioned by Cons T ?hs in his latest talk at the EUC in Stockholm. Kappa is used to make some sense of the free-for-all Erlang allows by having only a broad concept of exported and non-exported functions and performs a number of tasks and techniques that are in part an artifact of the development environment of Klarna and are probably not appropriate for most other people. With iota, you have a more generic tool that allows you to keep some sanity regarding the interactions between OTP applications while allowing you to export the functions you need inside your own application more freely. By using Erlang attributes we make these checks entirely up to you, so if you feel you must ignore them or don't want to use iota, no one is going to stop you! iota is hopefully also ready to be turned into rebar or EDTS plugins for a more streamlined integration into your workflows, like as a post-compilation hook. But enough with the chit-chat! You can get it (and more details on how to use it) here: https://github.com/jpgneves/iota If you find it useful (or not), let me know! Any comments, questions and pull requests are welcome! Many thanks to Samuel Rivas, H?kan Nilsson and Thomas J?rvstrand for providing feedback and useful libraries to build this with, and to Klarna Engineering in general for providing the necessary inspiration and ideas for this initial release. -- Jo?o Neves -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Fri Sep 20 11:29:58 2013 From: kostis@REDACTED (Kostis Sagonas) Date: Fri, 20 Sep 2013 11:29:58 +0200 Subject: [erlang-questions] dialyzer issue with opaque types In-Reply-To: References: Message-ID: <523C1596.206@cs.ntua.gr> On 09/19/2013 09:35 AM, Samuel wrote: > Hi, > > I have next two (trimmed down) modules I want to dialyze (attached as well) : > > -module(foo). > > -export([foo/0]). > > -spec foo() -> bar:id(). > foo() -> bar:id_undef(). > > -------------------------------------------------- > > -module(bar). > > -export([id_undef/0]). > > -opaque undef() :: -1. > -opaque id() :: undef(). > > -export_type([id/0, undef/0]). > > -spec id_undef() -> id(). > id_undef() -> -1. > > ----------------------------------------------- > > Running dialyzer on those fails like this: > > $ erlc +debug_info *.erl && dialyzer *.beam > Checking whether the PLT > /home/samuel/local/var/dialyzer/plts/R16B02-src/otp.plt is > up-to-date... > yes > Proceeding with analysis... > foo.erl:5: Invalid type specification for function foo:foo/0. The > success typing is () -> bar:id() > done in 0m3.02s > done (warnings were emitted) > > That is, dialyzer thinks that bar:id() is the bad result type and that > I should be using bar:id() instead :) I can very well see why some would classify this as a dialyzer bug (I see that you were careful to use the word "issue" in the subject header) but I am afraid that it is more of a limitation (or is "misunderstanding" a better term here?) of how opaques work. Dialyzer expects that the opaque declaration is something of the form: -opaque :: . where describes the structure of terms that belong to the opaque type. Ideally, the structure of these terms should be clearly distinguishable from that of all other type names that the module contains, hence we are prohibiting declarations of the form: -opaque :: term(). As a convenience to the programmer, we have so far allowed that type declarations can contain names of other type names where term structures are expected, i.e. declarations of the form :: as in your program. I am afraid this is problematic in the case of opaques, because the analysis is unable to classify on which opaque sort terms belong in this case. Is a -1 term an id() or an undef() in your example? OK, you have the spec declaration, but should dialyzer trust it? What I am trying to write is that general confusion can arise in these cases. This is what happens here, hence the confusing warning. To make the long story short, you can fix the immediate problem of this example at least by changing the declarations to: -opaque undef() :: -1. -opaque id() :: -1. but whether this "solution" extends to the untrimmed program or not, I do not know. As I explained, it's problematic to have terms of the same structure belonging to different sorts of opaque types. I'll look into this issue more closely. Thanks for the report! Kostis From varuna@REDACTED Fri Sep 20 12:56:15 2013 From: varuna@REDACTED (Varuna) Date: Fri, 20 Sep 2013 16:26:15 +0530 Subject: [erlang-questions] Software Estimation and Progress Tracking In-Reply-To: References: Message-ID: <523C29CF.5090602@eudaemonicsystems.net> On Thu, Sep 19, 2013 at 6:44 AM, wrote: > I recently wrote a blog post on software estimation and progress tracking for > functional languages. Unfortunately, the post was not terribly substantive and > ...... > Bringing it back to estimation, I fear that we will just adapt and expect more > features to delivered in shorter time, so our estimation of progress and > effort will still be off, even if we get things done faster. For some reason, > there is a bias toward "fast delivery" with no regard toward "quality of > product". This is a depressing work environment, though: every deliverable > will be too late, always. And this leads to a psychological phobia of > estimation :) > ... Not sure if SWEBOK Guide at (http://www.swebok.org) would be of any help, nevertheless, there is good information under the sections Software Engineering Economics, Software Engineering Management, and Software Engineering Models and Methods. With regards, Varuna From akonsu@REDACTED Fri Sep 20 17:08:20 2013 From: akonsu@REDACTED (akonsu) Date: Fri, 20 Sep 2013 11:08:20 -0400 Subject: [erlang-questions] Terminate process if it is slow at receiving messages Message-ID: Hello, I have posted this question to stackoverflow ( http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) and received one helpful response, but I am still not clear about the right way to do it. I also have seen this thread: http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, which is somewhat similar but it still did not hep me. I would appreciate more help. Here is the question: I have a data source process that sends messages to worker processes. To keep memory consumption under control, I need to terminate the workers that are slow retrieving their messages from their mailboxes. I am new to Erlang, I would appreciate any pointers. If this is difficult to accomplish with Erlang messages, maybe I can use sockets? If so, are there examples? EDIT: I have a registered process that reads from the web and generates a lot of data. It sends these data to all the "subscribed" processes using Erlang messages. For each particular piece of data, it sends the same message to all subscribers. I also have a web server that streams the data that the registered process reads. So, when a http client connects, the web server creates a process and this process subscribes to the registered process and starts receiving its messages. The registered process uses monitors to monitor subscribers. The subscribers are controlled by the web server, and when a connection is closed, the process that were serving this connection, dies. There is no acknowledgement, that is, subscribers do not respond when a message is sent to them. Although I can program them this way, but I think it is too much traffic. Basically I want to close the connection if a http client is too slow. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbothma@REDACTED Fri Sep 20 19:40:46 2013 From: jbothma@REDACTED (JD Bothma) Date: Fri, 20 Sep 2013 19:40:46 +0200 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: You could tag messages with a timestamp, and each time a client proceses a message, check difference between the message and current timestamp. Processes that get more behind than some threshold can close the connection and exit. Remember to find the cheap timestamp function - search around here - people have benchmarked them and explained why some are faster and some are slower. IIRC it was something to do with guarantees about increasing values at very high rates on SMP. Also as mentioned on stack-overflow, ref-counted binaries are held-onto until all processes which have seen them have GC'd so you might need to manually GC on long-lived processes every now and then. On 20 September 2013 17:08, akonsu wrote: > Hello, > > I have posted this question to stackoverflow ( > http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) > and received one helpful response, but I am still not clear about the right > way to do it. > > I also have seen this thread: > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, which > is somewhat similar but it still did not hep me. > > I would appreciate more help. Here is the question: > > > > I have a data source process that sends messages to worker processes. To > keep memory consumption under control, I need to terminate the workers that > are slow retrieving their messages from their mailboxes. > > I am new to Erlang, I would appreciate any pointers. If this is difficult > to accomplish with Erlang messages, maybe I can use sockets? If so, are > there examples? > > EDIT: > > I have a registered process that reads from the web and generates a lot of > data. It sends these data to all the "subscribed" processes using Erlang > messages. For each particular piece of data, it sends the same message to > all subscribers. > > I also have a web server that streams the data that the registered process > reads. So, when a http client connects, the web server creates a process > and this process subscribes to the registered process and starts receiving > its messages. > > The registered process uses monitors to monitor subscribers. The > subscribers are controlled by the web server, and when a connection is > closed, the process that were serving this connection, dies. > > There is no acknowledgement, that is, subscribers do not respond when a > message is sent to them. Although I can program them this way, but I think > it is too much traffic. > > Basically I want to close the connection if a http client is too slow. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Sep 20 19:42:34 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 20 Sep 2013 21:42:34 +0400 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: I would advise you to use gen_server:call and control sender this way. I've removed almost all async message sendings in erlyvideo and changed them to gen_server:call. But if you still want message sending, you can for example, kill all children with message_queue_len > 500 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Sep 20 20:14:37 2013 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 20 Sep 2013 11:14:37 -0700 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: <523C908D.5030903@gmail.com> Polling the message_queue_len of an Erlang pid is sort of like trying to catch a falling knife, and it is wasteful, processing time-wise. If you could set a limit on the message queue size of an Erlang pid, that might help, but it doesn't really help you handle error conditions. The best solution is to queue the messages you care about, within the Erlang pid's data, if they can't be handled immediately, so it becomes a queueing problem. To handle slow clients, you just give each client a timeout value. The OTP/Erlang source code doesn't decrement timeout values as time elapses, so it is possible that a timeout doesn't terminate a message call path in a way consistent with the timeout value. The top-level function call throws an exception, but other pids may continue to execute despite the timeout expiration. Often, people don't care about the timeout value not handling cumulative delay. If you do care about handling cumulative delay, you should look at my open source project here http://cloudi.org , since CloudI services handle the timeout decrements and the queueing (if you would want the most accurate timeout adjustments possible, there are CloudI service options for that here: https://github.com/okeuday/CloudI/blob/master/src/lib/cloudi_core/src/cloudi_configuration.hrl#L78). On 09/20/2013 08:08 AM, akonsu wrote: > Hello, > > I have posted this question to stackoverflow (http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) and received one helpful response, but I am still not clear about the right way to do it. > > I also have seen this thread: http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, which is somewhat similar but it still did not hep me. > > I would appreciate more help. Here is the question: > > > > I have a data source process that sends messages to worker processes. To keep memory consumption under control, I need to terminate the workers that are slow retrieving their messages from their mailboxes. > > I am new to Erlang, I would appreciate any pointers. If this is difficult to accomplish with Erlang messages, maybe I can use sockets? If so, are there examples? > > EDIT: > > I have a registered process that reads from the web and generates a lot of data. It sends these data to all the "subscribed" processes using Erlang messages. For each particular piece of data, it sends the same message to all subscribers. > > I also have a web server that streams the data that the registered process reads. So, when a http client connects, the web server creates a process and this process subscribes to the registered process and starts receiving its messages. > > The registered process uses monitors to monitor subscribers. The subscribers are controlled by the web server, and when a connection is closed, the process that were serving this connection, dies. > > There is no acknowledgement, that is, subscribers do not respond when a message is sent to them. Although I can program them this way, but I think it is too much traffic. > > Basically I want to close the connection if a http client is too slow. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mloftis@REDACTED Fri Sep 20 18:31:11 2013 From: mloftis@REDACTED (Michael Loftis) Date: Fri, 20 Sep 2013 09:31:11 -0700 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: Pretty simple actually...send them a message, perhaps with a timestamp (os:timestamp()) - the expectation is that the process responds back. If it does not at all within your window, kill it. If it does, but is unacceptably slow, kill it. Not at all difficult. On Fri, Sep 20, 2013 at 8:08 AM, akonsu wrote: > Hello, > > I have posted this question to stackoverflow > (http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) > and received one helpful response, but I am still not clear about the right > way to do it. > > I also have seen this thread: > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, which is > somewhat similar but it still did not hep me. > > I would appreciate more help. Here is the question: > > > > I have a data source process that sends messages to worker processes. To > keep memory consumption under control, I need to terminate the workers that > are slow retrieving their messages from their mailboxes. > > I am new to Erlang, I would appreciate any pointers. If this is difficult to > accomplish with Erlang messages, maybe I can use sockets? If so, are there > examples? > > EDIT: > > I have a registered process that reads from the web and generates a lot of > data. It sends these data to all the "subscribed" processes using Erlang > messages. For each particular piece of data, it sends the same message to > all subscribers. > > I also have a web server that streams the data that the registered process > reads. So, when a http client connects, the web server creates a process and > this process subscribes to the registered process and starts receiving its > messages. > > The registered process uses monitors to monitor subscribers. The subscribers > are controlled by the web server, and when a connection is closed, the > process that were serving this connection, dies. > > There is no acknowledgement, that is, subscribers do not respond when a > message is sent to them. Although I can program them this way, but I think > it is too much traffic. > > Basically I want to close the connection if a http client is too slow. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler From akonsu@REDACTED Fri Sep 20 18:38:42 2013 From: akonsu@REDACTED (akonsu) Date: Fri, 20 Sep 2013 12:38:42 -0400 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: thanks. but this would generate twice as much traffic because of the acknowledgement responses. Maybe instead of erlang messages I can use sockets to communicate? 2013/9/20 Michael Loftis > Pretty simple actually...send them a message, perhaps with a timestamp > (os:timestamp()) - the expectation is that the process responds back. > If it does not at all within your window, kill it. If it does, but is > unacceptably slow, kill it. > > Not at all difficult. > > On Fri, Sep 20, 2013 at 8:08 AM, akonsu wrote: > > Hello, > > > > I have posted this question to stackoverflow > > ( > http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages > ) > > and received one helpful response, but I am still not clear about the > right > > way to do it. > > > > I also have seen this thread: > > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, > which is > > somewhat similar but it still did not hep me. > > > > I would appreciate more help. Here is the question: > > > > > > > > I have a data source process that sends messages to worker processes. To > > keep memory consumption under control, I need to terminate the workers > that > > are slow retrieving their messages from their mailboxes. > > > > I am new to Erlang, I would appreciate any pointers. If this is > difficult to > > accomplish with Erlang messages, maybe I can use sockets? If so, are > there > > examples? > > > > EDIT: > > > > I have a registered process that reads from the web and generates a lot > of > > data. It sends these data to all the "subscribed" processes using Erlang > > messages. For each particular piece of data, it sends the same message to > > all subscribers. > > > > I also have a web server that streams the data that the registered > process > > reads. So, when a http client connects, the web server creates a process > and > > this process subscribes to the registered process and starts receiving > its > > messages. > > > > The registered process uses monitors to monitor subscribers. The > subscribers > > are controlled by the web server, and when a connection is closed, the > > process that were serving this connection, dies. > > > > There is no acknowledgement, that is, subscribers do not respond when a > > message is sent to them. Although I can program them this way, but I > think > > it is too much traffic. > > > > Basically I want to close the connection if a http client is too slow. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > -- > > "Genius might be described as a supreme capacity for getting its possessors > into trouble of all kinds." > -- Samuel Butler > -------------- next part -------------- An HTML attachment was scrubbed... URL: From akonsu@REDACTED Fri Sep 20 20:17:56 2013 From: akonsu@REDACTED (akonsu) Date: Fri, 20 Sep 2013 14:17:56 -0400 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: thanks everyone. Max, would you please explain what you mean by controlling the sender? Who would control the sender? why? Are you referring to the idea of sending an acknowledgement message and are proposing to use gen_server:call instead? Also, if I still want messages, the only way I know to get the queue length is to use process_info(self(),message_queue_len). Do you know if this call is expensive? Can I call this function often? I read that it is not advisable to call process_info/1 except for debugging, but it seems that process_info/2 is designed to be used in production. Konstantin 2013/9/20 Max Lapshin > I would advise you to use gen_server:call and control sender this way. > > I've removed almost all async message sendings in erlyvideo and changed > them to gen_server:call. > > But if you still want message sending, you can for example, kill all > children with message_queue_len > 500 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mloftis@REDACTED Fri Sep 20 20:29:09 2013 From: mloftis@REDACTED (Michael Loftis) Date: Fri, 20 Sep 2013 11:29:09 -0700 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: I never said go synchronous/use acks. Just send a completely seperate timing message occasionally. The mailbox queue is going to normally (well, depends a little on your code) be worked FIFO. So throwing a timing message at the process (possibly even from a completely different process) occasionally to check it's health. On Fri, Sep 20, 2013 at 9:38 AM, akonsu wrote: > thanks. but this would generate twice as much traffic because of the > acknowledgement responses. Maybe instead of erlang messages I can use > sockets to communicate? > > > 2013/9/20 Michael Loftis >> >> Pretty simple actually...send them a message, perhaps with a timestamp >> (os:timestamp()) - the expectation is that the process responds back. >> If it does not at all within your window, kill it. If it does, but is >> unacceptably slow, kill it. >> >> Not at all difficult. >> >> On Fri, Sep 20, 2013 at 8:08 AM, akonsu wrote: >> > Hello, >> > >> > I have posted this question to stackoverflow >> > >> > (http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) >> > and received one helpful response, but I am still not clear about the >> > right >> > way to do it. >> > >> > I also have seen this thread: >> > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, >> > which is >> > somewhat similar but it still did not hep me. >> > >> > I would appreciate more help. Here is the question: >> > >> > >> > >> > I have a data source process that sends messages to worker processes. To >> > keep memory consumption under control, I need to terminate the workers >> > that >> > are slow retrieving their messages from their mailboxes. >> > >> > I am new to Erlang, I would appreciate any pointers. If this is >> > difficult to >> > accomplish with Erlang messages, maybe I can use sockets? If so, are >> > there >> > examples? >> > >> > EDIT: >> > >> > I have a registered process that reads from the web and generates a lot >> > of >> > data. It sends these data to all the "subscribed" processes using Erlang >> > messages. For each particular piece of data, it sends the same message >> > to >> > all subscribers. >> > >> > I also have a web server that streams the data that the registered >> > process >> > reads. So, when a http client connects, the web server creates a process >> > and >> > this process subscribes to the registered process and starts receiving >> > its >> > messages. >> > >> > The registered process uses monitors to monitor subscribers. The >> > subscribers >> > are controlled by the web server, and when a connection is closed, the >> > process that were serving this connection, dies. >> > >> > There is no acknowledgement, that is, subscribers do not respond when a >> > message is sent to them. Although I can program them this way, but I >> > think >> > it is too much traffic. >> > >> > Basically I want to close the connection if a http client is too slow. >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> >> >> >> -- >> >> "Genius might be described as a supreme capacity for getting its >> possessors >> into trouble of all kinds." >> -- Samuel Butler > > -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler From robert.virding@REDACTED Fri Sep 20 21:07:05 2013 From: robert.virding@REDACTED (Robert Virding) Date: Fri, 20 Sep 2013 21:07:05 +0200 (CEST) Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: Message-ID: <1261717861.54348.1379704025869.JavaMail.zimbra@erlang-solutions.com> If you are going to use timestamps the timer call you want to avoid erlang:now() as its guarantees that each call generates a larger value than the previous which requires synchronisation between all the schedulers in Erlang VM. Somewhere I have a pathological example which goes slower the more cores you use because of this. IIRC doing process_info(message-queue_len) also has it s problems as the process does not keep a count of messages but counts them each time you call. IIRC! An alternative solution is one presented at an Erlang Factory in San Francisco by Fred Hebert used by the company he was working for. They didn't want to kill the process but efficiently throw away messages that had timed out. Their system did allowed them to process messages not in the order they arrived and throw away messages that had timed out. Fred will have to correct me here if I am wrong. The problem with having timestamps and receiving down the message queue checking for timed out messages is that this takes time so you will lose more messages because of this. Their solution was to push the messages on a stack when they arrived then when you want to process a message you just take the top one off the stack. If you get a timed out message then you know that all those further down the stack are older so you can throw then away without checking. IIRC even more they bounded the stack. You could augment this with a stack count so easily keep track if the process is falling behind or not. You could use this to either kill the process or just throw away messages. Pretty nifty I think as it reverses how you normally expect to do this. Robert ----- Original Message ----- > From: "Michael Loftis" > To: "akonsu" > Cc: erlang-questions@REDACTED > Sent: Friday, 20 September, 2013 8:29:09 PM > Subject: Re: [erlang-questions] Terminate process if it is slow at receiving messages > > I never said go synchronous/use acks. Just send a completely seperate > timing message occasionally. The mailbox queue is going to normally > (well, depends a little on your code) be worked FIFO. So throwing a > timing message at the process (possibly even from a completely > different process) occasionally to check it's health. > > On Fri, Sep 20, 2013 at 9:38 AM, akonsu wrote: > > thanks. but this would generate twice as much traffic because of the > > acknowledgement responses. Maybe instead of erlang messages I can use > > sockets to communicate? > > > > > > 2013/9/20 Michael Loftis > >> > >> Pretty simple actually...send them a message, perhaps with a timestamp > >> (os:timestamp()) - the expectation is that the process responds back. > >> If it does not at all within your window, kill it. If it does, but is > >> unacceptably slow, kill it. > >> > >> Not at all difficult. > >> > >> On Fri, Sep 20, 2013 at 8:08 AM, akonsu wrote: > >> > Hello, > >> > > >> > I have posted this question to stackoverflow > >> > > >> > (http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) > >> > and received one helpful response, but I am still not clear about the > >> > right > >> > way to do it. > >> > > >> > I also have seen this thread: > >> > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, > >> > which is > >> > somewhat similar but it still did not hep me. > >> > > >> > I would appreciate more help. Here is the question: > >> > > >> > > >> > > >> > I have a data source process that sends messages to worker processes. To > >> > keep memory consumption under control, I need to terminate the workers > >> > that > >> > are slow retrieving their messages from their mailboxes. > >> > > >> > I am new to Erlang, I would appreciate any pointers. If this is > >> > difficult to > >> > accomplish with Erlang messages, maybe I can use sockets? If so, are > >> > there > >> > examples? > >> > > >> > EDIT: > >> > > >> > I have a registered process that reads from the web and generates a lot > >> > of > >> > data. It sends these data to all the "subscribed" processes using Erlang > >> > messages. For each particular piece of data, it sends the same message > >> > to > >> > all subscribers. > >> > > >> > I also have a web server that streams the data that the registered > >> > process > >> > reads. So, when a http client connects, the web server creates a process > >> > and > >> > this process subscribes to the registered process and starts receiving > >> > its > >> > messages. > >> > > >> > The registered process uses monitors to monitor subscribers. The > >> > subscribers > >> > are controlled by the web server, and when a connection is closed, the > >> > process that were serving this connection, dies. > >> > > >> > There is no acknowledgement, that is, subscribers do not respond when a > >> > message is sent to them. Although I can program them this way, but I > >> > think > >> > it is too much traffic. > >> > > >> > Basically I want to close the connection if a http client is too slow. > >> > > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > >> > > >> > >> > >> > >> -- > >> > >> "Genius might be described as a supreme capacity for getting its > >> possessors > >> into trouble of all kinds." > >> -- Samuel Butler > > > > > > > > -- > > "Genius might be described as a supreme capacity for getting its possessors > into trouble of all kinds." > -- Samuel Butler > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From akonsu@REDACTED Fri Sep 20 21:17:59 2013 From: akonsu@REDACTED (akonsu) Date: Fri, 20 Sep 2013 15:17:59 -0400 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: <1261717861.54348.1379704025869.JavaMail.zimbra@erlang-solutions.com> References: <1261717861.54348.1379704025869.JavaMail.zimbra@erlang-solutions.com> Message-ID: (The solution with a stack is basically what Michael Truog proposed as well, although he proposed a queue). Is there a reference to the Fred Hebert's talk at the Erlang Factory? I am new to erlang, and I do not see how I can in a single process receive messages, put them into another stack/queue and process them at the same time... spawn yet another process for each incoming message? I also am not sure about sending timestamps with each message, because time is relative and different nodes have different clocks... besides it takes time for each message to get delivered, and I do not see how I can measure this time and disregard it. 2013/9/20 Robert Virding > If you are going to use timestamps the timer call you want to avoid > erlang:now() as its guarantees that each call generates a larger value than > the previous which requires synchronisation between all the schedulers in > Erlang VM. Somewhere I have a pathological example which goes slower the > more cores you use because of this. > > IIRC doing process_info(message-queue_len) also has it s problems as the > process does not keep a count of messages but counts them each time you > call. IIRC! > > An alternative solution is one presented at an Erlang Factory in San > Francisco by Fred Hebert used by the company he was working for. They > didn't want to kill the process but efficiently throw away messages that > had timed out. Their system did allowed them to process messages not in the > order they arrived and throw away messages that had timed out. Fred will > have to correct me here if I am wrong. > > The problem with having timestamps and receiving down the message queue > checking for timed out messages is that this takes time so you will lose > more messages because of this. Their solution was to push the messages on a > stack when they arrived then when you want to process a message you just > take the top one off the stack. If you get a timed out message then you > know that all those further down the stack are older so you can throw then > away without checking. IIRC even more they bounded the stack. > > You could augment this with a stack count so easily keep track if the > process is falling behind or not. > > You could use this to either kill the process or just throw away messages. > > Pretty nifty I think as it reverses how you normally expect to do this. > > Robert > > > > ----- Original Message ----- > > From: "Michael Loftis" > > To: "akonsu" > > Cc: erlang-questions@REDACTED > > Sent: Friday, 20 September, 2013 8:29:09 PM > > Subject: Re: [erlang-questions] Terminate process if it is slow at > receiving messages > > > > I never said go synchronous/use acks. Just send a completely seperate > > timing message occasionally. The mailbox queue is going to normally > > (well, depends a little on your code) be worked FIFO. So throwing a > > timing message at the process (possibly even from a completely > > different process) occasionally to check it's health. > > > > On Fri, Sep 20, 2013 at 9:38 AM, akonsu wrote: > > > thanks. but this would generate twice as much traffic because of the > > > acknowledgement responses. Maybe instead of erlang messages I can use > > > sockets to communicate? > > > > > > > > > 2013/9/20 Michael Loftis > > >> > > >> Pretty simple actually...send them a message, perhaps with a timestamp > > >> (os:timestamp()) - the expectation is that the process responds back. > > >> If it does not at all within your window, kill it. If it does, but is > > >> unacceptably slow, kill it. > > >> > > >> Not at all difficult. > > >> > > >> On Fri, Sep 20, 2013 at 8:08 AM, akonsu wrote: > > >> > Hello, > > >> > > > >> > I have posted this question to stackoverflow > > >> > > > >> > ( > http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages > ) > > >> > and received one helpful response, but I am still not clear about > the > > >> > right > > >> > way to do it. > > >> > > > >> > I also have seen this thread: > > >> > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, > > >> > which is > > >> > somewhat similar but it still did not hep me. > > >> > > > >> > I would appreciate more help. Here is the question: > > >> > > > >> > > > >> > > > >> > I have a data source process that sends messages to worker > processes. To > > >> > keep memory consumption under control, I need to terminate the > workers > > >> > that > > >> > are slow retrieving their messages from their mailboxes. > > >> > > > >> > I am new to Erlang, I would appreciate any pointers. If this is > > >> > difficult to > > >> > accomplish with Erlang messages, maybe I can use sockets? If so, are > > >> > there > > >> > examples? > > >> > > > >> > EDIT: > > >> > > > >> > I have a registered process that reads from the web and generates a > lot > > >> > of > > >> > data. It sends these data to all the "subscribed" processes using > Erlang > > >> > messages. For each particular piece of data, it sends the same > message > > >> > to > > >> > all subscribers. > > >> > > > >> > I also have a web server that streams the data that the registered > > >> > process > > >> > reads. So, when a http client connects, the web server creates a > process > > >> > and > > >> > this process subscribes to the registered process and starts > receiving > > >> > its > > >> > messages. > > >> > > > >> > The registered process uses monitors to monitor subscribers. The > > >> > subscribers > > >> > are controlled by the web server, and when a connection is closed, > the > > >> > process that were serving this connection, dies. > > >> > > > >> > There is no acknowledgement, that is, subscribers do not respond > when a > > >> > message is sent to them. Although I can program them this way, but I > > >> > think > > >> > it is too much traffic. > > >> > > > >> > Basically I want to close the connection if a http client is too > slow. > > >> > > > >> > _______________________________________________ > > >> > erlang-questions mailing list > > >> > erlang-questions@REDACTED > > >> > http://erlang.org/mailman/listinfo/erlang-questions > > >> > > > >> > > >> > > >> > > >> -- > > >> > > >> "Genius might be described as a supreme capacity for getting its > > >> possessors > > >> into trouble of all kinds." > > >> -- Samuel Butler > > > > > > > > > > > > > > -- > > > > "Genius might be described as a supreme capacity for getting its > possessors > > into trouble of all kinds." > > -- Samuel Butler > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Fri Sep 20 21:29:51 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 20 Sep 2013 15:29:51 -0400 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: <1261717861.54348.1379704025869.JavaMail.zimbra@erlang-solutions.com> Message-ID: <20130920192949.GI13826@ferdair.local> I transcribed the entire talk content: http://ferd.ca/rtb-where-erlang-blooms.html The library mentioned is dispcount: https://github.com/ferd/dispcount It allows to shed load instantly, and is meant to be used in low-latency cases. I've recently written PO Box: https://github.com/ferd/pobox This one implements bounded queues and stacks for Erlang processes, where a process middleman acts like a buffer from which you can filter data and whatnot, and do batching at the same time. It might be somewhat appropriate for what you need, although it won't do much to instantly know what processes are too slow. It'll only shed load. Regards, Fred. On 09/20, akonsu wrote: > (The solution with a stack is basically what Michael Truog proposed as > well, although he proposed a queue). Is there a reference to the Fred > Hebert's talk at the Erlang Factory? I am new to erlang, and I do not see > how I can in a single process receive messages, put them into another > stack/queue and process them at the same time... spawn yet another process > for each incoming message? > > I also am not sure about sending timestamps with each message, because time > is relative and different nodes have different clocks... besides it takes > time for each message to get delivered, and I do not see how I can measure > this time and disregard it. > > > 2013/9/20 Robert Virding > > > If you are going to use timestamps the timer call you want to avoid > > erlang:now() as its guarantees that each call generates a larger value than > > the previous which requires synchronisation between all the schedulers in > > Erlang VM. Somewhere I have a pathological example which goes slower the > > more cores you use because of this. > > > > IIRC doing process_info(message-queue_len) also has it s problems as the > > process does not keep a count of messages but counts them each time you > > call. IIRC! > > > > An alternative solution is one presented at an Erlang Factory in San > > Francisco by Fred Hebert used by the company he was working for. They > > didn't want to kill the process but efficiently throw away messages that > > had timed out. Their system did allowed them to process messages not in the > > order they arrived and throw away messages that had timed out. Fred will > > have to correct me here if I am wrong. > > > > The problem with having timestamps and receiving down the message queue > > checking for timed out messages is that this takes time so you will lose > > more messages because of this. Their solution was to push the messages on a > > stack when they arrived then when you want to process a message you just > > take the top one off the stack. If you get a timed out message then you > > know that all those further down the stack are older so you can throw then > > away without checking. IIRC even more they bounded the stack. > > > > You could augment this with a stack count so easily keep track if the > > process is falling behind or not. > > > > You could use this to either kill the process or just throw away messages. > > > > Pretty nifty I think as it reverses how you normally expect to do this. > > > > Robert > > > > > > > > ----- Original Message ----- > > > From: "Michael Loftis" > > > To: "akonsu" > > > Cc: erlang-questions@REDACTED > > > Sent: Friday, 20 September, 2013 8:29:09 PM > > > Subject: Re: [erlang-questions] Terminate process if it is slow at > > receiving messages > > > > > > I never said go synchronous/use acks. Just send a completely seperate > > > timing message occasionally. The mailbox queue is going to normally > > > (well, depends a little on your code) be worked FIFO. So throwing a > > > timing message at the process (possibly even from a completely > > > different process) occasionally to check it's health. > > > > > > On Fri, Sep 20, 2013 at 9:38 AM, akonsu wrote: > > > > thanks. but this would generate twice as much traffic because of the > > > > acknowledgement responses. Maybe instead of erlang messages I can use > > > > sockets to communicate? > > > > > > > > > > > > 2013/9/20 Michael Loftis > > > >> > > > >> Pretty simple actually...send them a message, perhaps with a timestamp > > > >> (os:timestamp()) - the expectation is that the process responds back. > > > >> If it does not at all within your window, kill it. If it does, but is > > > >> unacceptably slow, kill it. > > > >> > > > >> Not at all difficult. > > > >> > > > >> On Fri, Sep 20, 2013 at 8:08 AM, akonsu wrote: > > > >> > Hello, > > > >> > > > > >> > I have posted this question to stackoverflow > > > >> > > > > >> > ( > > http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages > > ) > > > >> > and received one helpful response, but I am still not clear about > > the > > > >> > right > > > >> > way to do it. > > > >> > > > > >> > I also have seen this thread: > > > >> > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, > > > >> > which is > > > >> > somewhat similar but it still did not hep me. > > > >> > > > > >> > I would appreciate more help. Here is the question: > > > >> > > > > >> > > > > >> > > > > >> > I have a data source process that sends messages to worker > > processes. To > > > >> > keep memory consumption under control, I need to terminate the > > workers > > > >> > that > > > >> > are slow retrieving their messages from their mailboxes. > > > >> > > > > >> > I am new to Erlang, I would appreciate any pointers. If this is > > > >> > difficult to > > > >> > accomplish with Erlang messages, maybe I can use sockets? If so, are > > > >> > there > > > >> > examples? > > > >> > > > > >> > EDIT: > > > >> > > > > >> > I have a registered process that reads from the web and generates a > > lot > > > >> > of > > > >> > data. It sends these data to all the "subscribed" processes using > > Erlang > > > >> > messages. For each particular piece of data, it sends the same > > message > > > >> > to > > > >> > all subscribers. > > > >> > > > > >> > I also have a web server that streams the data that the registered > > > >> > process > > > >> > reads. So, when a http client connects, the web server creates a > > process > > > >> > and > > > >> > this process subscribes to the registered process and starts > > receiving > > > >> > its > > > >> > messages. > > > >> > > > > >> > The registered process uses monitors to monitor subscribers. The > > > >> > subscribers > > > >> > are controlled by the web server, and when a connection is closed, > > the > > > >> > process that were serving this connection, dies. > > > >> > > > > >> > There is no acknowledgement, that is, subscribers do not respond > > when a > > > >> > message is sent to them. Although I can program them this way, but I > > > >> > think > > > >> > it is too much traffic. > > > >> > > > > >> > Basically I want to close the connection if a http client is too > > slow. > > > >> > > > > >> > _______________________________________________ > > > >> > erlang-questions mailing list > > > >> > erlang-questions@REDACTED > > > >> > http://erlang.org/mailman/listinfo/erlang-questions > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> > > > >> "Genius might be described as a supreme capacity for getting its > > > >> possessors > > > >> into trouble of all kinds." > > > >> -- Samuel Butler > > > > > > > > > > > > > > > > > > > > -- > > > > > > "Genius might be described as a supreme capacity for getting its > > possessors > > > into trouble of all kinds." > > > -- Samuel Butler > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Fri Sep 20 21:39:46 2013 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 20 Sep 2013 12:39:46 -0700 Subject: [erlang-questions] Terminate process if it is slow at receiving messages In-Reply-To: References: <1261717861.54348.1379704025869.JavaMail.zimbra@erlang-solutions.com> Message-ID: <523CA482.7040405@gmail.com> One way of tracking the time, which is used in CloudI, is to use erlang:send_after/3 with a timeout to return a TimerRef within the sending Erlang pid, and erlang:cancel_timer/1 upon receipt of a response (to return the timeout remaining), assuming you didn't receive the erlang:send_after/3 message (if you did, the request timed out). That makes the request synchronous, which is necessary for message flow control (i.e., to control the memory consumption). On 09/20/2013 12:17 PM, akonsu wrote: > (The solution with a stack is basically what Michael Truog proposed as well, although he proposed a queue). Is there a reference to the Fred Hebert's talk at the Erlang Factory? I am new to erlang, and I do not see how I can in a single process receive messages, put them into another stack/queue and process them at the same time... spawn yet another process for each incoming message? > > I also am not sure about sending timestamps with each message, because time is relative and different nodes have different clocks... besides it takes time for each message to get delivered, and I do not see how I can measure this time and disregard it. > > > 2013/9/20 Robert Virding > > > If you are going to use timestamps the timer call you want to avoid erlang:now() as its guarantees that each call generates a larger value than the previous which requires synchronisation between all the schedulers in Erlang VM. Somewhere I have a pathological example which goes slower the more cores you use because of this. > > IIRC doing process_info(message-queue_len) also has it s problems as the process does not keep a count of messages but counts them each time you call. IIRC! > > An alternative solution is one presented at an Erlang Factory in San Francisco by Fred Hebert used by the company he was working for. They didn't want to kill the process but efficiently throw away messages that had timed out. Their system did allowed them to process messages not in the order they arrived and throw away messages that had timed out. Fred will have to correct me here if I am wrong. > > The problem with having timestamps and receiving down the message queue checking for timed out messages is that this takes time so you will lose more messages because of this. Their solution was to push the messages on a stack when they arrived then when you want to process a message you just take the top one off the stack. If you get a timed out message then you know that all those further down the stack are older so you can throw then away without checking. IIRC even more they bounded the stack. > > You could augment this with a stack count so easily keep track if the process is falling behind or not. > > You could use this to either kill the process or just throw away messages. > > Pretty nifty I think as it reverses how you normally expect to do this. > > Robert > > > > ----- Original Message ----- > > From: "Michael Loftis" > > > To: "akonsu" > > > Cc: erlang-questions@REDACTED > > Sent: Friday, 20 September, 2013 8:29:09 PM > > Subject: Re: [erlang-questions] Terminate process if it is slow at receiving messages > > > > I never said go synchronous/use acks. Just send a completely seperate > > timing message occasionally. The mailbox queue is going to normally > > (well, depends a little on your code) be worked FIFO. So throwing a > > timing message at the process (possibly even from a completely > > different process) occasionally to check it's health. > > > > On Fri, Sep 20, 2013 at 9:38 AM, akonsu > wrote: > > > thanks. but this would generate twice as much traffic because of the > > > acknowledgement responses. Maybe instead of erlang messages I can use > > > sockets to communicate? > > > > > > > > > 2013/9/20 Michael Loftis > > > >> > > >> Pretty simple actually...send them a message, perhaps with a timestamp > > >> (os:timestamp()) - the expectation is that the process responds back. > > >> If it does not at all within your window, kill it. If it does, but is > > >> unacceptably slow, kill it. > > >> > > >> Not at all difficult. > > >> > > >> On Fri, Sep 20, 2013 at 8:08 AM, akonsu > wrote: > > >> > Hello, > > >> > > > >> > I have posted this question to stackoverflow > > >> > > > >> > (http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages) > > >> > and received one helpful response, but I am still not clear about the > > >> > right > > >> > way to do it. > > >> > > > >> > I also have seen this thread: > > >> > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html, > > >> > which is > > >> > somewhat similar but it still did not hep me. > > >> > > > >> > I would appreciate more help. Here is the question: > > >> > > > >> > > > >> > > > >> > I have a data source process that sends messages to worker processes. To > > >> > keep memory consumption under control, I need to terminate the workers > > >> > that > > >> > are slow retrieving their messages from their mailboxes. > > >> > > > >> > I am new to Erlang, I would appreciate any pointers. If this is > > >> > difficult to > > >> > accomplish with Erlang messages, maybe I can use sockets? If so, are > > >> > there > > >> > examples? > > >> > > > >> > EDIT: > > >> > > > >> > I have a registered process that reads from the web and generates a lot > > >> > of > > >> > data. It sends these data to all the "subscribed" processes using Erlang > > >> > messages. For each particular piece of data, it sends the same message > > >> > to > > >> > all subscribers. > > >> > > > >> > I also have a web server that streams the data that the registered > > >> > process > > >> > reads. So, when a http client connects, the web server creates a process > > >> > and > > >> > this process subscribes to the registered process and starts receiving > > >> > its > > >> > messages. > > >> > > > >> > The registered process uses monitors to monitor subscribers. The > > >> > subscribers > > >> > are controlled by the web server, and when a connection is closed, the > > >> > process that were serving this connection, dies. > > >> > > > >> > There is no acknowledgement, that is, subscribers do not respond when a > > >> > message is sent to them. Although I can program them this way, but I > > >> > think > > >> > it is too much traffic. > > >> > > > >> > Basically I want to close the connection if a http client is too slow. > > >> > > > >> > _______________________________________________ > > >> > erlang-questions mailing list > > >> > erlang-questions@REDACTED > > >> > http://erlang.org/mailman/listinfo/erlang-questions > > >> > > > >> > > >> > > >> > > >> -- > > >> > > >> "Genius might be described as a supreme capacity for getting its > > >> possessors > > >> into trouble of all kinds." > > >> -- Samuel Butler > > > > > > > > > > > > > > -- > > > > "Genius might be described as a supreme capacity for getting its possessors > > into trouble of all kinds." > > -- Samuel Butler > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw@REDACTED Sat Sep 21 06:40:13 2013 From: flw@REDACTED (Florian Waas) Date: Fri, 20 Sep 2013 21:40:13 -0700 Subject: [erlang-questions] ELDAP: peer verification fails due to mangled SSL options? Message-ID: Currently, eldap does not support peer verification (equivalent to ldap.conf's TLS_REQCERT). Turns out eldap:do_connect/3 always prefixes the caller's ssl options with {verify, 0} under the covers which renders a {verify, verify_peer} from the caller ineffective: https://github.com/erlang/otp/blob/maint/lib/eldap/src/eldap.erl#L392 As far as I can tell, there's no good/obvious reason for this -- and after removing this automatic prefix, it works as one would expect. Just a bug or anybody know of some rationale why verification is prevented this way? -fl. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt@REDACTED Sat Sep 21 11:17:59 2013 From: matt@REDACTED (Matt Lewandowsky) Date: Sat, 21 Sep 2013 02:17:59 -0700 Subject: [erlang-questions] Cross-compiling tests with quoted ENV vars In-Reply-To: References: , Message-ID: Thanks for the reply, Lukas. I have fixed the problem with quoted CFLAGS in my local tree (and a few other things which trying to build the tests exposed). Unfortunately, I'm now stuck in ts_erl_config.erl. It appears to use OsType extensively, which wouldn't be much of a problem so far except for the fact that my target system doesn't use -lpthread for POSIX threads. I can't just add a new case there because since because I'm cross-compiling from Linux for not-Linux, it will always match the {unix,_} case. I'm not finding any way to actually determine the cross-compilation OS type there. I expect this is more of not quite understanding just how the pieces of the cross compile support are supposed to mesh with the tests than anything else. I'll continue poking at it over the weekend, but if you (or anyone else) have any pointers which would speed this up, I'd be greatly appreciative. It feels like I must be almost done with this step of the process since it's trying to link. For what it's worth, I've already used Erlang on the target platform (including verifying that os:type() is what I expected it to be), so the tests really are just a method for me to validate that the result is correct. --Matt -- Matt Lewandowsky Big Geek Greenviolet matt@REDACTED http://www.greenviolet.net +1 415 578 5782 (US) +44 844 484 8254 (UK) ________________________________ > Date: Thu, 19 Sep 2013 09:23:00 +0200 > Subject: Re: [erlang-questions] Cross-compiling tests with quoted ENV vars > From: lukas@REDACTED > To: matt@REDACTED > CC: erlang-questions@REDACTED > > Hello Matt, > > As you have guessed the cross compile support for tests is indeed a > very lightly-used path and also quite new. > > Right now we run nightly cross-compile tests for x86_64-saf-linux-gnu > and powerpc-dso-linux-gnu (from Linux x86) and have not attempted to > get it to work on any other. > > As neither of these use quoted CFLAGS, that specific option has slipped > through and what you have found is indeed a bug. A patch that fixes > this would be very appreciated! > > Lukas > > > On Wed, Sep 18, 2013 at 1:33 AM, Matt Lewandowsky > > wrote: > Hello, > > I am attempting to cross-compile OTP for a new platform, with a new > $ERL_TOP/xcomp/erl-xcomp-*.conf file. Unfortunately, I've hit an interesting > issue and it appears the only other xcomp platform which would > encounter it is > VxWorks, which I do not have access to. > > In my xcomp conf file, I use quoted values for CFLAGS and other environment > variables. These get re-double-quoted by ts_install.erl as such: > > Env: [{"CFLAGS", > "\"-mcpu=cortex-a9 -Wno-psabi -fstack-protector-all > -D_FORTIFY_SOURCE=2 > -O2 -Wformat -Wformat-security -Werror=format-security\""}, > > As the VxWorks target also has quoted variables, how do the tests get > compiled > for that platform since the documented procedure in the "Testing the cross > compiled system" section of the User's Guide *should* fail, per the results > I'm seeing? Are tests not cross-compiled by anyone, ever, for that platform? > Is there some undocumented magic I'm missing? Is this a legitimate bug I > should fix while I trod down this obviously lightly-used path? > > Thanks, > > --Matt > > P.S. As there doesn't seem to be any naming convention for > configuration files > in the xcomp directory and not much to work from, are there recommendations > for the configuration file I'm working on? The host triplet is long, so would > "something appropriate" be a good enough choice? I expect a> 40 character > filename would be balked at. :) > > -- > Matt Lewandowsky > Big Geek > Greenviolet > matt@REDACTED http://www.greenviolet.net > +1 415 578 5782 (US) +44 844 484 8254 (UK) > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > From n.oxyde@REDACTED Sat Sep 21 16:56:22 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sat, 21 Sep 2013 16:56:22 +0200 Subject: [erlang-questions] Threaded VM with llvm-gcc Message-ID: <75E2E84D-4193-4437-B370-DD0CCA4850B3@gmail.com> Hello, I read in the installation guide [1] that it is recommended to use GCC 4.2 to build Erlang because CLang and LLVM cannot use jump tables. There is two problems with this: - This is not consistent with what the configure scripts actually do: CLang 3 and later use jump tables when compiling Erlang [2]. - There is no reason to not use LLVM 2.7 and later for beam_emu.c because it does compile jump tables efficiently [3]. Feel free to compile the following example code to LLVM-IR with llvm-gcc -s -emit-llvm or clang -s -emit-llvm: int main () { __label__ lbl1; __label__ lbl2; int x = magic(); static void *jtab[2]; jtab[0] = &&lbl1; jtab[1] = &&lbl2; goto *jtab[x]; lbl1: return 1; lbl2: return 2; ; return 0; } Unfortunately I have no idea how to fix this, as there is no way to get the LLVM version from any preprocessor macro [4]. Should we just remove the LLVM check from the test and let the snippet of code fail to compile [5]? Regards, [1] http://www.erlang.org/doc/installation_guide/INSTALL.html#Building-a-fast-Erlang-VM-on-Mac-OS-Lion [2] https://github.com/erlang/otp/blob/589a9ed126d205007e79c22053d1b156a383d99f/aclocal.m4#L217-218 [3] http://blog.llvm.org/2010/01/address-of-label-and-indirect-branches.html [4] http://stackoverflow.com/questions/1617877/how-to-detect-llvm-and-its-version-through-define-directives [5] https://github.com/erlang/otp/blob/589a9ed126d205007e79c22053d1b156a383d99f/aclocal.m4#L219-220 -- Anthony Ramine From shefys@REDACTED Sun Sep 22 02:08:43 2013 From: shefys@REDACTED (Yuri Shefer) Date: Sat, 21 Sep 2013 17:08:43 -0700 Subject: [erlang-questions] io_lib printable list Message-ID: Hi all, I have a silly question. Why io_lib:printable_list() accept characters only in the quotes? Is it needed even for the numbers? 8> io:getopts(). [{expand_fun,#Fun}, {echo,true}, {binary,false}, {encoding,latin1}] 9> io_lib:printable_list(1). false 10> io_lib:printable_list("1"). true -- Best regards, Yuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Sun Sep 22 10:15:00 2013 From: gleber.p@REDACTED (Gleb Peregud) Date: Sun, 22 Sep 2013 10:15:00 +0200 Subject: [erlang-questions] io_lib printable list In-Reply-To: References: Message-ID: Because "1" is actually a syntactic sugar for [$1] which is syntactic sugar for [49] On Sun, Sep 22, 2013 at 2:08 AM, Yuri Shefer wrote: > Hi all, > > I have a silly question. > > Why io_lib:printable_list() accept characters only in the quotes? Is it > needed even for the numbers? > > 8> io:getopts(). > [{expand_fun,#Fun}, > {echo,true}, > {binary,false}, > {encoding,latin1}] > 9> io_lib:printable_list(1). > false > 10> io_lib:printable_list("1"). > true > > > -- > Best regards, > Yuri > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From melvyn.ferrao@REDACTED Sun Sep 22 15:18:59 2013 From: melvyn.ferrao@REDACTED (Melvyn Ferrao) Date: Sun, 22 Sep 2013 18:48:59 +0530 Subject: [erlang-questions] Understanding the scalability of Erlang Message-ID: It is said that thousands of processes can be spawned to do the similar task concurrently and Erlang is good at handling it. If there is more work to be done, we can simply and safely add more worker processes and that makes it scalable. What I fail to understand is that if the work performed by each work is itself resource-intensive, how will Erlang be able to handle it? For instance, if entries are being made into a table by several sources and an Erlang application withing its hundreds of processes reads rows from the table and does something, this is obviously likely to cause resource burden. Every worker will try to pull a record from the table. If this is a bad example, consider a worker that has to perform a highly CPU-intensive computation in memory. Thousands of such workers running concurrently will overwork the CPU. Please rectify my understanding of the scalability in Erlang: Erlang processes get time slices of the CPU only if there is work available for them. OS processes on the other hand get time slices regardless of whether they are idle. The startup and shutdown time of Erlang processes is much lower than that of OS processes. Apart from the above two points is there something about Erlang that makes it scalable? Thanks, Melvyn -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergej.jurecko@REDACTED Sun Sep 22 15:43:21 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sun, 22 Sep 2013 15:43:21 +0200 Subject: [erlang-questions] Understanding the scalability of Erlang In-Reply-To: References: Message-ID: Erlang makes it easy to spread the workload across CPU cores on a single machine and spread the workload across many machines (distributed erlang over a network). Per machine you are of course limited to the CPU capabilities of that machine. Erlang does not break any laws of physics. You can do all of that in other languages, but it will require a lot more work. Sergej On Sep 22, 2013, at 3:18 PM, Melvyn Ferrao wrote: > It is said that thousands of processes can be spawned to do the similar task concurrently and Erlang is good at handling it. If there is more work to be done, we can simply and safely add more worker processes and that makes it scalable. > > What I fail to understand is that if the work performed by each work is itself resource-intensive, how will Erlang be able to handle it? For instance, if entries are being made into a table by several sources and an Erlang application withing its hundreds of processes reads rows from the table and does something, this is obviously likely to cause resource burden. Every worker will try to pull a record from the table. > If this is a bad example, consider a worker that has to perform a highly CPU-intensive computation in memory. Thousands of such workers running concurrently will overwork the CPU. > > Please rectify my understanding of the scalability in Erlang: > Erlang processes get time slices of the CPU only if there is work available for them. OS processes on the other hand get time slices regardless of whether they are idle. > The startup and shutdown time of Erlang processes is much lower than that of OS processes. > > Apart from the above two points is there something about Erlang that makes it scalable? > > Thanks, > Melvyn > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From freza@REDACTED Sun Sep 22 16:19:21 2013 From: freza@REDACTED (Jachym Holecek) Date: Sun, 22 Sep 2013 10:19:21 -0400 Subject: [erlang-questions] Understanding the scalability of Erlang In-Reply-To: References: Message-ID: <20130922141920.GA25462@circlewave.net> # Melvyn Ferrao 2013-09-22: > It is said that thousands of processes can be spawned to do the similar > task concurrently and Erlang is good at handling it. If there is more work > to be done, we can simply and safely add more worker processes and that > makes it scalable. The ability to spawn large number of very cheap processes and the fact of eliminating resource sharing between them nearly completely are good enablers for scalability (in addition to enabling other, parhaps more important, things). That said however, nothing in the world will make arbitrary application "scale" by magic. There is no excuse for proper design and implementation towards the required operational characteristics at application level. Let me also note this is a thorougly end-to-end affair: scalably designed application relying on "lame" protocol stack won't do wonders. So our unit of analysis here is a whole node, or a bunch of nodes, not isolated pieces here and there. This goes against the widespread religions of "layering" and "abstraction boundaries", which is why these are helpful tools but evil masters. (This is something I wanted to say for a while now, made it fit in this context slightly forcefully. :-) > What I fail to understand is that if the work performed by each work is > itself resource-intensive, how will Erlang be able to handle it? For > instance, if entries are being made into a table by several sources and an > Erlang application withing its hundreds of processes reads rows from the > table and does something, this is obviously likely to cause resource > burden. Every worker will try to pull a record from the table. A hard fact of nature -- you've introduced a shared resource in your architecture, now you have the possibility of it becoming a bottleneck. Not an outright necessity though, measurements will tell. > If this is a bad example, consider a worker that has to perform a highly > CPU-intensive computation in memory. Thousands of such workers running > concurrently will overwork the CPU. Nothing "overworks" a CPU, it's not going to melt or something. :-) The more of them you have the longer the run queues thus the higher execution latency per worker. You're right in suggesting there are limits to per-node execution capacity, that's something to consider and overload protection of various kinds would typically be a "must have" item. These should probably be configurable at runtime. > Please rectify my understanding of the scalability in Erlang: > Erlang processes get time slices of the CPU only if there is work available > for them. OS processes on the other hand get time slices regardless of > whether they are idle. No. They both get a timeslice if they're "ready for execution", that is "not blocking". Erlang process blocks if it sits in a receive clause. OS process blocks if it sits in a blocking syscall. > The startup and shutdown time of Erlang processes is much lower than that > of OS processes. Yes, and Erlang processes are cheaper/smaller than OS processes. There's also an important synergy between the programming language and underlying runtime system, something you don't have in the native case. > Apart from the above two points is there something about Erlang that makes > it scalable? I've simplified a lot but hopefully the above helps. Let's also note that there are problem domains where it's important to cheaply process large shared data structures in parallel. Those won't necessarily play well with Erlang. BR, -- Jachym From freza@REDACTED Sun Sep 22 16:25:23 2013 From: freza@REDACTED (Jachym Holecek) Date: Sun, 22 Sep 2013 10:25:23 -0400 Subject: [erlang-questions] Understanding the scalability of Erlang In-Reply-To: <20130922141920.GA25462@circlewave.net> References: <20130922141920.GA25462@circlewave.net> Message-ID: <20130922142523.GB25462@circlewave.net> # Jachym Holecek 2013-09-22: > The ability to spawn large number of very cheap processes and the fact > of eliminating resource sharing between them nearly completely are good > enablers for scalability (in addition to enabling other, parhaps more > important, things). That said however, nothing in the world will make > arbitrary application "scale" by magic. There is no excuse for proper > design and implementation towards the required operational characteristics > at application level. Correction: I meant "no excuse for lack of" there. BR, -- Jachym From max.lapshin@REDACTED Sun Sep 22 23:20:53 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 22 Sep 2013 14:20:53 -0700 (PDT) Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: This is a very good project. I've made a fork: https://github.com/flussonic/myproto and added following patches: 1) renamed mysql.peg to mysql_proto.peg because name of module mysql.erl will clash with existing client drivers 2) refactored packet decoder so that it will work not only on localhost, but also in the internet. {tcp,Socket,Bin} doesn't always contain whole packet =) 3) added some client-server tests and embedded nanomysql.erl for it. 4) added optional ranch worker. I don't want to use my_acceptor, so socket acceptor should be an optional thing 5) added packing of integers 6) embedded it into erlyvideo and now it can send live stats data from system to plain mysql client! Thank you again for this project. I think that mysql server implementation is a very good to export data from erlang to web applications like PHP or Rails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pillai.sharmila@REDACTED Sun Sep 22 23:24:23 2013 From: pillai.sharmila@REDACTED (Sharmila Pillai) Date: Sun, 22 Sep 2013 22:24:23 +0100 Subject: [erlang-questions] SCTP support Message-ID: Hi all, I am trying to get SCTP to work on Linux. > uname -a Linux linux 3.0.82-0.7-default #1 SMP Thu Jun 27 13:19:18 UTC 2013 (6efde93) x86_64 x86_64 x86_64 GNU/Linux lksctp-tools is installed. > rpm -qa | grep sctp lksctp-tools-1.0.9-1.22 SCTP is supported. > checksctp SCTP supported But when I try to use SCTP, I get eprotonosupport. > ~/software/erlang/R16B02/bin/erl Erlang R16B02 (erts-5.10.3) [source] [64-bit] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10.3 (abort with ^G) 1> 1> gen_sctp:open(). {error,eprotonosupport} Help please! /S -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Sun Sep 22 23:47:42 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Sun, 22 Sep 2013 17:47:42 -0400 Subject: [erlang-questions] SCTP support In-Reply-To: References: Message-ID: On Sun, Sep 22, 2013 at 5:24 PM, Sharmila Pillai wrote: > Hi all, > > I am trying to get SCTP to work on Linux. > > > uname -a > Linux linux 3.0.82-0.7-default #1 SMP Thu Jun 27 13:19:18 UTC 2013 > (6efde93) x86_64 x86_64 x86_64 GNU/Linux > > lksctp-tools is installed. > > > rpm -qa | grep sctp > lksctp-tools-1.0.9-1.22 > > SCTP is supported. > > > checksctp > SCTP supported > > But when I try to use SCTP, I get eprotonosupport. > > > ~/software/erlang/R16B02/bin/erl > Erlang R16B02 (erts-5.10.3) [source] [64-bit] [async-threads:10] [hipe] > [kernel-poll:false] > > Eshell V5.10.3 (abort with ^G) > 1> > 1> gen_sctp:open(). > {error,eprotonosupport} > > Help please! > After installing SCTP support on Linux, are you sure you reran configure for your Erlang/OTP build and then recompiled? On ubuntu 12.04 I have SCTP working just fine with Erlang R16B02, with these packages installed: $ dpkg --get-selections | grep sctp libsctp-dev install libsctp1 install lksctp-tools install Perhaps you're missing the dev package? --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From bombadil@REDACTED Mon Sep 23 01:46:40 2013 From: bombadil@REDACTED (Manuel A. Rubio "Bombadil") Date: Mon, 23 Sep 2013 01:46:40 +0200 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: Hi Max, thanks to you, your enhancements are great! :-) I just merged them. I'd like to do a benchmark, but by the moment, sysbench needs to have PREPARE implemented so, I've try it "by-hand" with a shell script: My results were: One process (in the for section): * MySQL real: 7-10 sec. * myproto: ~50 sec. 16 threads: * MySQL real: ~10 sec. * myproto: ~55 sec. 64 threads: * MySQL real: ~14 sec. * myproto: ~60 sec. I think I need to do better performance analysis in myproto to achieve better results :-P Thanks! Manuel Rubio. ------- #!/bin/bash DATA="" for i in $(seq 1 10000); do DATA="$DATA SELECT 1;" done echo "SQL Generated!" PIDS="" for i in $(seq 1 64); do echo $DATA | mysql -utest -ptest -h127.0.0.1 test &> /dev/null & PID=$! echo "PID $PID launched!" PIDS="$PIDS $!" done wait $PIDS ---------- From sean@REDACTED Mon Sep 23 08:00:01 2013 From: sean@REDACTED (Functional Jobs) Date: Mon, 23 Sep 2013 02:00:01 -0400 Subject: [erlang-questions] New Functional Programming Job Opportunities Message-ID: <523fd8fd380b5@functionaljobs.com> Here are some functional programming job opportunities that were posted recently: Developer - Erlang at Klarna http://functionaljobs.com/jobs/8648-developer-erlang-at-klarna Cheers, Sean Murphy FunctionalJobs.com From hellkvist@REDACTED Mon Sep 23 08:01:39 2013 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Mon, 23 Sep 2013 08:01:39 +0200 Subject: [erlang-questions] (no subject) Message-ID: Hi, Here is a question about why I sometimes get an etimedout error from gen_tcp:recv. I open a tcp socket on the server side using lines like: {ok, LSock} = gen_tcp:listen(Port, [binary, {packet, line}, {active, false}]), {ok, Sock} = gen_tcp:accept(LSock), do_recv(Sock). where do_recv is something like this: do_recv(Sock) -> case gen_tcp:recv(Sock, 0) of {ok, Data} -> do_recv(Sock); {error, Reason} -> ?DAWN_EVENT("tcp_bus got error and shut down: ~p", [Reason]), telnet_subscriber:unsubscribe({Sock, self()}) end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Mon Sep 23 08:10:21 2013 From: hellkvist@REDACTED (Stefan Hellkvist) Date: Mon, 23 Sep 2013 08:10:21 +0200 Subject: [erlang-questions] etimedout from gen_tcp:recv - or definition of infinity Message-ID: Sorry about this...pressed send by mistake (should never write the address before completing the message): I'll try again: The code went in the first message there (below), but here is my question: How can I get a etimedout message from gen_tcp:recv in this case? I get it if I leave the the connection idling for a couple of hours. The clients initially connect to the server, sends some messages, and then go idle and the server then get etimedout. According to the manual pages of gen_tcp all default values for timeout values are set to infinity which to me would imply that the connection can close and all that and experience other errors but I should at least never get a etimedout value unless the length of "infinity" is highly exaggerated. Does anyone have any idea why I can get the etimedout response from gen_tcp:recv in this case? Have I misunderstood the documentation? This happens on R15B01 on Ubuntu 12.04 64bit server. Kind regards, Stefan Hellkvist On Mon, Sep 23, 2013 at 8:01 AM, Stefan Hellkvist wrote: > Hi, > > Here is a question about why I sometimes get an etimedout error from > gen_tcp:recv. > > I open a tcp socket on the server side using lines like: > > {ok, LSock} = gen_tcp:listen(Port, [binary, {packet, line}, {active, > false}]), > {ok, Sock} = gen_tcp:accept(LSock), > do_recv(Sock). > > > where do_recv is something like this: > > > do_recv(Sock) -> > case gen_tcp:recv(Sock, 0) of > {ok, Data} -> > %handle data > do_recv(Sock); > > {error, Reason} -> > log("tcp_bus got error and shut down: ~p", [Reason]) > end. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Sep 23 08:58:19 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 23 Sep 2013 10:58:19 +0400 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: <973fa02284c84b5cc9ed7e5214a1654d@bosqueviejo.net> References: <973fa02284c84b5cc9ed7e5214a1654d@bosqueviejo.net> Message-ID: Yes, I also had problems with parsing 20 MB query =) And there are some more often used mysql queries that are not supported by this parser right now. In any case, it may be a very useful module, because it allows the same feature as Sphinx search engine does: access to data via unified and common interface. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yash_ganthe@REDACTED Mon Sep 23 08:58:34 2013 From: yash_ganthe@REDACTED (Yash Ganthe) Date: Mon, 23 Sep 2013 06:58:34 +0000 Subject: [erlang-questions] Re-using Mnesia disk files for a different node Message-ID: Hi, On a node named node1@REDACTED I created an Mnesia schema using: mnesia:create_schema([node()]). and created a table using: mnesia:create_table(tab1, [{disc_copies, [node()]}, {attributes, record_info(fields, tab1)}]). The mnesia directory was set to "mnesia_node1". The directory was created and the DB files were created inside it. Once the node was stopped, the DB files still exist in the directory. I hoped to re-use the DB as the mnesia DB for another node, node2. So I started the other node using: erl -name node2 -mnesia dir "mnesia_node1". On node2 when I start mnesia and do a tv:start(), I see the tab1 listed in the schema. However, it shows no records. I was expecting Mnesia to use the directory as the location of all DB files. Is there something I have missed? Thanks, Yash DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Sep 23 10:34:53 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 23 Sep 2013 10:34:53 +0200 Subject: [erlang-questions] etimedout from gen_tcp:recv - or definition of infinity In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 8:10 AM, Stefan Hellkvist wrote: > According to the manual pages of gen_tcp all default values for timeout > values are set to infinity which to me would imply that the connection can > close and all that and experience other errors but I should at least never > get a etimedout value unless the length of "infinity" is highly > exaggerated. While the Erlang VM will not result in a timeout, the underlying operating system can send the ETIMEDOUT error (see man 2 recv). So even if you specifically ask for a poll on the socket, there may still be an underlying error propagating to you. The most probable reason for this has to do with something low-level TCP connectivity. I would suggest running tcpdump on the connection and inspect the data (perhaps in Wireshark). -------------- next part -------------- An HTML attachment was scrubbed... URL: From xramtsov@REDACTED Mon Sep 23 11:21:48 2013 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Mon, 23 Sep 2013 19:21:48 +1000 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> Message-ID: <5240082C.1070502@gmail.com> On 23.09.2013 09:46, Manuel A. Rubio "Bombadil" wrote: > Hi Max, > > thanks to you, your enhancements are great! :-) I just merged them. Any response on my pull request? https://github.com/bosqueviejo/myproto/pull/2 -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From send2philip@REDACTED Mon Sep 23 12:12:59 2013 From: send2philip@REDACTED (Philip Clarke) Date: Mon, 23 Sep 2013 11:12:59 +0100 Subject: [erlang-questions] Re-using Mnesia disk files for a different node In-Reply-To: References: Message-ID: Hi Yash, (forwarding my reply to the list this time to see if there is any feedback on this advice) Unfortunately the mnesia schema lists which nodes are associated with which tables. In your case your new node (node2) will look at the schema and know that it does not have any tables. It would be able to have remote access to the tables if node1 was still running, but this does not suit your use case. If you want to move a mnesia database from one node to another, I have found that you need to follow there are three steps: 1) backup the current database with mnesia:backup 2) rewrite the schema table in your backup (to replace node1 with node2) using mnesia:traverse_backup 3) restore the backup to your new node with mnesia:restore For step 2, see the function change_node_name in the code example in http://www.erlang.org/doc/apps/mnesia/Mnesia_chap7.html#id78880 Regards Philip On Mon, Sep 23, 2013 at 7:58 AM, Yash Ganthe wrote: > Hi,**** > > ** ** > > On a node named node1@REDACTED I created an Mnesia schema using:**** > > mnesia:create_schema([node()]).**** > > and created a table using:**** > > mnesia:create_table(tab1, [{disc_copies, [node()]}, {attributes, > record_info(fields, tab1)}]).**** > > ** ** > > The mnesia directory was set to "mnesia_node1". The directory was created > and the DB files were created inside it.**** > > ** ** > > Once the node was stopped, the DB files still exist in the directory. I > hoped to re-use the DB as the mnesia DB for another node, node2.**** > > ** ** > > So I started the other node using:**** > > erl -name node2 -mnesia dir "mnesia_node1".**** > > ** ** > > On node2 when I start mnesia and do a tv:start(), I see the tab1 listed in > the schema. However, it shows no records. I was expecting Mnesia to use the > directory as the location of all DB files.**** > > ** ** > > Is there something I have missed?**** > > ** ** > > Thanks,**** > > Yash**** > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Ltd. It is intended > only for the use of the individual or entity to which it is addressed. If > you are not the intended recipient, you are not authorized to read, retain, > copy, print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Persistent Systems Ltd. does not accept any liability for > virus infected mails. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Sep 23 17:40:03 2013 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 23 Sep 2013 17:40:03 +0200 Subject: [erlang-questions] Cross-compiling tests with quoted ENV vars In-Reply-To: References: Message-ID: Hello, On Sat, Sep 21, 2013 at 11:17 AM, Matt Lewandowsky wrote: > > It appears to use OsType extensively, which wouldn't be much of a problem > so far except for the fact that my target system doesn't use -lpthread for > POSIX threads. I can't just add a new case there because since because I'm > cross-compiling from Linux for not-Linux, it will always match the {unix,_} > case. I'm not finding any way to actually determine the cross-compilation > OS type there. I expect this is more of not quite understanding just how > the pieces of the cross compile support are supposed to mesh with the tests > than anything else. > I think you should be able to specify an os by directly calling ts_install:install/2 instead of going through ts:install/1. As you noticed, all of ts_install and friends have only been "ported" to work with unix/win32, so you will have to add a few clauses for any new operating system. Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bombadil@REDACTED Mon Sep 23 18:40:33 2013 From: bombadil@REDACTED (Manuel A. Rubio "Bombadil") Date: Mon, 23 Sep 2013 18:40:33 +0200 Subject: [erlang-questions] [ANN] myproto 0.2.0 In-Reply-To: <5240082C.1070502@gmail.com> References: <25af660faf2b79d1f1ff1e48eb746329@bosqueviejo.net> <5240082C.1070502@gmail.com> Message-ID: Sorry, I didn't saw your pull request. It's merged now ;-) Thanks. Manuel Rubio. El 2013-09-23 11:21, Evgeniy Khramtsov escribi?: > On 23.09.2013 09:46, Manuel A. Rubio "Bombadil" wrote: >> Hi Max, >> >> thanks to you, your enhancements are great! :-) I just merged them. > > Any response on my pull request? > https://github.com/bosqueviejo/myproto/pull/2 From jonas.falkevik@REDACTED Tue Sep 24 00:33:36 2013 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Tue, 24 Sep 2013 00:33:36 +0200 Subject: [erlang-questions] (no subject) In-Reply-To: References: Message-ID: <965B9226-A2B5-4D65-A084-C5D914FC8092@mobilearts.com> Hi, it looks like your OS is returning the posix's error code etimedout. Mening that the TCP connection timed out. Most probably because you sent some data that could not be delivered within the your OS specified TCP retransmit / timer settings. Did you send some data over the TCP connection at any time? Jonas On Sep 23, 2013, at 8:01 , Stefan Hellkvist wrote: > Hi, > > Here is a question about why I sometimes get an etimedout error from gen_tcp:recv. > > I open a tcp socket on the server side using lines like: > > {ok, LSock} = gen_tcp:listen(Port, [binary, {packet, line}, {active, false}]), > {ok, Sock} = gen_tcp:accept(LSock), > do_recv(Sock). > > > where do_recv is something like this: > > > do_recv(Sock) -> > case gen_tcp:recv(Sock, 0) of > {ok, Data} -> > > do_recv(Sock); > > {error, Reason} -> > ?DAWN_EVENT("tcp_bus got error and shut down: ~p", [Reason]), > telnet_subscriber:unsubscribe({Sock, self()}) > end. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From steven.charles.davis@REDACTED Tue Sep 24 02:15:58 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Mon, 23 Sep 2013 19:15:58 -0500 Subject: [erlang-questions] eep-18 Message-ID: Since... JSON.parse("{\"-12.3\":\"test\"}"); ...is valid according to RFC-4627 ...is it possible to retrieve "test" from the key "-12.3" in javascript? (It's obviously possible in erlang) /s From akonsu@REDACTED Tue Sep 24 04:21:30 2013 From: akonsu@REDACTED (akonsu) Date: Mon, 23 Sep 2013 22:21:30 -0400 Subject: [erlang-questions] eep-18 In-Reply-To: References: Message-ID: Do you mean like this? var h = {"-12.3":"test"}; var v = h["-12.3"]; 2013/9/23 Steve Davis > Since... > > JSON.parse("{\"-12.3\":\"test\"}"); > > ...is valid according to RFC-4627 > > ...is it possible to retrieve "test" from the key "-12.3" in javascript? > > (It's obviously possible in erlang) > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hellkvist@REDACTED Tue Sep 24 06:43:01 2013 From: hellkvist@REDACTED (hellkvist@REDACTED) Date: Tue, 24 Sep 2013 06:43:01 +0200 Subject: [erlang-questions] (no subject) In-Reply-To: <965B9226-A2B5-4D65-A084-C5D914FC8092@mobilearts.com> References: <965B9226-A2B5-4D65-A084-C5D914FC8092@mobilearts.com> Message-ID: <63FEC698-DEB3-49DA-BEF6-5C992C9DA0D0@gmail.com> Yes, data was sent (and also received at the other end) some time before. Then the connection just idled for quite some time (hours) and then this happened. Stefan ___ main(i){putchar((1312900435>>((i-1)/2|i>5)*8)+!(1&i|i>4))^78&&main(++i);} > 24 sep 2013 kl. 00:33 skrev Jonas Falkevik : > > Hi, > it looks like your OS is returning the posix's error code etimedout. Mening that the TCP connection timed out. > Most probably because you sent some data that could not be delivered within the your OS specified TCP retransmit / timer settings. > > Did you send some data over the TCP connection at any time? > > Jonas > >> On Sep 23, 2013, at 8:01 , Stefan Hellkvist wrote: >> >> Hi, >> >> Here is a question about why I sometimes get an etimedout error from gen_tcp:recv. >> >> I open a tcp socket on the server side using lines like: >> >> {ok, LSock} = gen_tcp:listen(Port, [binary, {packet, line}, {active, false}]), >> {ok, Sock} = gen_tcp:accept(LSock), >> do_recv(Sock). >> >> >> where do_recv is something like this: >> >> >> do_recv(Sock) -> >> case gen_tcp:recv(Sock, 0) of >> {ok, Data} -> >> >> do_recv(Sock); >> >> {error, Reason} -> >> ?DAWN_EVENT("tcp_bus got error and shut down: ~p", [Reason]), >> telnet_subscriber:unsubscribe({Sock, self()}) >> end. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bgustavsson@REDACTED Tue Sep 24 07:37:37 2013 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Tue, 24 Sep 2013 07:37:37 +0200 Subject: [erlang-questions] Threaded VM with llvm-gcc In-Reply-To: <75E2E84D-4193-4437-B370-DD0CCA4850B3@gmail.com> References: <75E2E84D-4193-4437-B370-DD0CCA4850B3@gmail.com> Message-ID: On Sat, Sep 21, 2013 at 4:56 PM, Anthony Ramine wrote: > Hello, > > I read in the installation guide [1] that it is recommended to use GCC 4.2 > to build Erlang because CLang and LLVM cannot use jump tables. > > At least in the version of the installation guide you linked to, it says nothing about jump tables being the reason for the recommendation. It says that the performance will be worse if GCC is not used, but it does not say why. There is two problems with this: > > - This is not consistent with what the configure scripts actually do: > CLang 3 and later use jump tables when compiling Erlang [2]. > Yes, it is consistent since the installation guide only talks about performance, not jump tables. > - There is no reason to not use LLVM 2.7 and later for beam_emu.c because > it does compile jump tables efficiently [3]. > > It does compile jump tables efficiently, but it does a worse job of register allocation, especially for huge function such as process_main() in beam_emu.c. That said, code generation has improved in each new version of clang, making it less important to use a "real" GCC. > Unfortunately I have no idea how to fix this, as there is no way to get > the LLVM version from any preprocessor macro [4]. Should we just remove the > LLVM check from the test and let the snippet of code fail to compile [5]? > > What is the problem you want to fix? Have you observed any situation where jump tables where not used when they should be? The test is where to protect against an old broken version of llvm-gcc that did not handle jump tables correctly. It will also make sure that Erlang/OTP can be compiled on systems where no compiler that supports jump tables can be found. /Bj?rn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Tue Sep 24 08:47:02 2013 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 24 Sep 2013 01:47:02 -0500 Subject: [erlang-questions] RELEASE project D2.2 (WP2), DTrace, and timestamps Message-ID: <54069.1380005222@snookles.snookles.com> Howdy. I was reading this status report from the RELEASE project from earlier this year: http://release-project.softlab.ntua.gr/documents/D2.2.pdf I see this on page 16: "we added timestamps to all new and existing DTrace probes that we wanted to associate with some trace (or proling) message." As an example, on page 17: probe message__queued(char *receiver, uint32_t size, uint32_t queue_len, int token_label, int token_previous, int token_current, uint64_t ts) Parameters: [...] * ts: the timestamp (in microseconds) I'm a bit curious about the addition of these timestamp items to various probes. The D2.2.pdf paper doesn't say what the timestamps are, e.g., wallclock time-of-day (UNIX time_t/epoch-style, perhaps, but with microsecond resolution?) or relative to the start of the virtual machine, or some other basis. DTrace provides a reliable, strictly increasing timestamp value, whose D script variable name is "timestamp", which has nanosecond resolution. A related variable, "vtimestamp", provides the time (also in nanoseconds) that the thread has been on-CPU. The SystemTap framework also supplies a similar built-in functions, e.g., gettimeofday_ns() and gettimeofday_us(). It seems to me that adding the "ts" argument to the message-queued probe and others isn't Strictly Necessary(tm) for DTrace or SystemTap. I don't know enough about LTT:NG to know if they were added to make life with LTT easier? Thoughts? -Scott From jonas.falkevik@REDACTED Tue Sep 24 10:36:18 2013 From: jonas.falkevik@REDACTED (Jonas Falkevik) Date: Tue, 24 Sep 2013 10:36:18 +0200 Subject: [erlang-questions] (no subject) In-Reply-To: <63FEC698-DEB3-49DA-BEF6-5C992C9DA0D0@gmail.com> References: <965B9226-A2B5-4D65-A084-C5D914FC8092@mobilearts.com> <63FEC698-DEB3-49DA-BEF6-5C992C9DA0D0@gmail.com> Message-ID: I see that there are a different thread where Jesper answered as well. Sorry I missed that one. And there you mention that you are using Ubuntu, then you can check using netstat if you have any un-acked data (Send-Q)? Unless you have done that already. Or if you analyze the connection on both sides using wireshark or such tool, as Jesper mentioned, to see if there are any missing packets / acks or other things happening. Do you have some sort of NAT or state full firewall between the hosts? Do you enable keepalive on the TCP connection? Jonas On Sep 24, 2013, at 6:43 , hellkvist@REDACTED wrote: > Yes, data was sent (and also received at the other end) some time before. Then the connection just idled for quite some time (hours) and then this happened. > > Stefan > > ___ > main(i){putchar((1312900435>>((i-1)/2|i>5)*8)+!(1&i|i>4))^78&&main(++i);} > > > 24 sep 2013 kl. 00:33 skrev Jonas Falkevik : > >> Hi, >> it looks like your OS is returning the posix's error code etimedout. Mening that the TCP connection timed out. >> Most probably because you sent some data that could not be delivered within the your OS specified TCP retransmit / timer settings. >> >> Did you send some data over the TCP connection at any time? >> >> Jonas >> >> On Sep 23, 2013, at 8:01 , Stefan Hellkvist wrote: >> >>> Hi, >>> >>> Here is a question about why I sometimes get an etimedout error from gen_tcp:recv. >>> >>> I open a tcp socket on the server side using lines like: >>> >>> {ok, LSock} = gen_tcp:listen(Port, [binary, {packet, line}, {active, false}]), >>> {ok, Sock} = gen_tcp:accept(LSock), >>> do_recv(Sock). >>> >>> >>> where do_recv is something like this: >>> >>> >>> do_recv(Sock) -> >>> case gen_tcp:recv(Sock, 0) of >>> {ok, Data} -> >>> >>> do_recv(Sock); >>> >>> {error, Reason} -> >>> ?DAWN_EVENT("tcp_bus got error and shut down: ~p", [Reason]), >>> telnet_subscriber:unsubscribe({Sock, self()}) >>> end. >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Tue Sep 24 11:50:05 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Tue, 24 Sep 2013 11:50:05 +0200 Subject: [erlang-questions] Threaded VM with llvm-gcc In-Reply-To: References: <75E2E84D-4193-4437-B370-DD0CCA4850B3@gmail.com> Message-ID: <03DB64B5-A64A-42A1-843A-64D7875C068D@gmail.com> Hello Bj?rn, Replied inline. Regards, Le 24 sept. 2013 ? 07:37, Bj?rn Gustavsson a ?crit : > At least in the version of the installation guide you linked to, > it says nothing about jump tables being the reason for the > recommendation. It says that the performance will be worse > if GCC is not used, but it does not say why. Oh right. > It does compile jump tables efficiently, but it does a worse job > of register allocation, especially for huge function such as > process_main() in beam_emu.c. That said, code generation > has improved in each new version of clang, making it less > important to use a "real" GCC. Ok, then it is kinda confusing that the only EMU_CC test is whether &&label is supported or not. > What is the problem you want to fix? Have you observed > any situation where jump tables where not used when they > should be? No. > The test is where to protect against an old broken version > of llvm-gcc that did not handle jump tables correctly. > It will also make sure that Erlang/OTP can be compiled > on systems where no compiler that supports jump tables > can be found. Shouldn't the test be fixed for more recent versions? I don't like limiting stuff because of old and outdated versions. -- Anthony Ramine From pillai.sharmila@REDACTED Tue Sep 24 12:18:02 2013 From: pillai.sharmila@REDACTED (Sharmila Pillai) Date: Tue, 24 Sep 2013 11:18:02 +0100 Subject: [erlang-questions] SCTP support In-Reply-To: References: Message-ID: Thanks Steve. Yes, I was missing the dev package. I have now installed the dev package, reconfigured Erlang/OTP with --enable-sctp and it works. In case someone needs for suse linux , these are the packages: > rpm -qa | grep sctp lksctp-tools-devel-1.0.9-1.22 lksctp-tools-1.0.9-1.22 /Sharmila. On Sun, Sep 22, 2013 at 10:47 PM, Steve Vinoski wrote: > > > > On Sun, Sep 22, 2013 at 5:24 PM, Sharmila Pillai < > pillai.sharmila@REDACTED> wrote: > >> Hi all, >> >> I am trying to get SCTP to work on Linux. >> >> > uname -a >> Linux linux 3.0.82-0.7-default #1 SMP Thu Jun 27 13:19:18 UTC 2013 >> (6efde93) x86_64 x86_64 x86_64 GNU/Linux >> >> lksctp-tools is installed. >> >> > rpm -qa | grep sctp >> lksctp-tools-1.0.9-1.22 >> >> SCTP is supported. >> >> > checksctp >> SCTP supported >> >> But when I try to use SCTP, I get eprotonosupport. >> >> > ~/software/erlang/R16B02/bin/erl >> Erlang R16B02 (erts-5.10.3) [source] [64-bit] [async-threads:10] [hipe] >> [kernel-poll:false] >> >> Eshell V5.10.3 (abort with ^G) >> 1> >> 1> gen_sctp:open(). >> {error,eprotonosupport} >> >> Help please! >> > > After installing SCTP support on Linux, are you sure you reran configure > for your Erlang/OTP build and then recompiled? > > On ubuntu 12.04 I have SCTP working just fine with Erlang R16B02, with > these packages installed: > > $ dpkg --get-selections | grep sctp > libsctp-dev install > libsctp1 install > lksctp-tools install > > Perhaps you're missing the dev package? > > --steve > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Tue Sep 24 15:34:15 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Tue, 24 Sep 2013 08:34:15 -0500 Subject: [erlang-questions] eep-18 In-Reply-To: References: Message-ID: <3F114520-7D78-415C-BE5A-4CE7ECA21759@gmail.com> Ah yes. Thank you Akonsu! It seems I had not come across a profound inconsistency in JSON after all. /s On Sep 23, 2013, at 9:21 PM, akonsu wrote: > Do you mean like this? > > > var h = {"-12.3":"test"}; > var v = h["-12.3"]; > > > > 2013/9/23 Steve Davis > Since... > > JSON.parse("{\"-12.3\":\"test\"}"); > > ...is valid according to RFC-4627 > > ...is it possible to retrieve "test" from the key "-12.3" in javascript? > > (It's obviously possible in erlang) > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k4t3r1n4@REDACTED Tue Sep 24 16:14:00 2013 From: k4t3r1n4@REDACTED (Katerina Roukounaki) Date: Tue, 24 Sep 2013 17:14:00 +0300 Subject: [erlang-questions] RELEASE project D2.2 (WP2), DTrace, and timestamps In-Reply-To: <54069.1380005222@snookles.snookles.com> References: <54069.1380005222@snookles.snookles.com> Message-ID: Scott, The main purpose of the work, that the report you read refers to, was to enhance / modify the initial set of probes you had provided, so that someone (in our case that someone was Percept) that is currently using Erlang built-in mechanism, in order to profile an application, can use DTrace / SystemTap to collect the same information. That work had nothing to do with LTTng. Having that in mind, the reason we decided to add our own time-stamps to DTrace probes was because what we want to know is "when the event happened", rather than "when the probe was fired" (trace messages carry time-stamps that hold the same piece of information). The time-stamps are generated in a way similar to that, in which the values for the ts field of the normal trace messages are generated. Katerina On 24 September 2013 09:47, Scott Lystig Fritchie wrote: > Howdy. I was reading this status report from the RELEASE project from > earlier this year: > > http://release-project.softlab.ntua.gr/documents/D2.2.pdf > > I see this on page 16: > > "we added timestamps to all new and existing DTrace probes that we > wanted to associate with some trace (or proling) message." > > As an example, on page 17: > > probe message__queued(char *receiver, uint32_t size, uint32_t > queue_len, > int token_label, int token_previous, int token_current, > uint64_t ts) > > Parameters: > [...] > * ts: the timestamp (in microseconds) > > I'm a bit curious about the addition of these timestamp items to various > probes. The D2.2.pdf paper doesn't say what the timestamps are, e.g., > wallclock time-of-day (UNIX time_t/epoch-style, perhaps, but with > microsecond resolution?) or relative to the start of the virtual > machine, or some other basis. > > DTrace provides a reliable, strictly increasing timestamp value, whose D > script variable name is "timestamp", which has nanosecond resolution. A > related variable, "vtimestamp", provides the time (also in nanoseconds) > that the thread has been on-CPU. > > The SystemTap framework also supplies a similar built-in functions, > e.g., gettimeofday_ns() and gettimeofday_us(). > > It seems to me that adding the "ts" argument to the message-queued probe > and others isn't Strictly Necessary(tm) for DTrace or SystemTap. I > don't know enough about LTT:NG to know if they were added to make life > with LTT easier? Thoughts? > > -Scott > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasingleton.work@REDACTED Tue Sep 24 16:46:58 2013 From: jasingleton.work@REDACTED (jasingleton.work@REDACTED) Date: Tue, 24 Sep 2013 07:46:58 -0700 (PDT) Subject: [erlang-questions] Erlang n00b figuring out bins Message-ID: <21000597-8eb6-4c0d-82e9-6e1f76ea120c@googlegroups.com> I've done a lot of development, but it's been quite a while since I've done anything functional. Erlang is still brand-new for me, so be warned. I've got a (conceptual) string arriving in my sw as a binary, e.g. <<"stuff.junk.excess">> and trying to transform it into a list of strings, e.g. ["stuff", "junk", "excess"] The mechanism I'm using is giving me a list of lists of binaries, e.g. [ [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>], [<<"j">>, <<"u">> (etc) I ended up hacking it to just take each of the lists of binaries and run it through list_to_binary and then binary to list. The result is what I really wanted, which is list_to_binary(binary_to_list( [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>])) gives me back "stuff" That's all well and good, and functional, but has two problems: 1) I honestly don't get why it worked; I would expect those two operations to give me back something unchanged. 2) There's got to be a more graceful way to achieve this. Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From roe.adrian@REDACTED Tue Sep 24 21:46:32 2013 From: roe.adrian@REDACTED (Adrian Roe) Date: Tue, 24 Sep 2013 20:46:32 +0100 Subject: [erlang-questions] Fw: Erlang n00b figuring out bins In-Reply-To: References: <21000597-8eb6-4c0d-82e9-6e1f76ea120c@googlegroups.com> Message-ID: <178486F220764E608389A3078C966EAB@gmail.com> Initial message failed - retrying -- Adrian Roe Sent with Sparrow (http://www.sparrowmailapp.com/?sig) Forwarded message: > From: Adrian Roe > To: jasingleton.work@REDACTED > Cc: erlang-programming@REDACTED > Date: Tuesday, 24 September 2013 20:44:41 > Subject: Re: [erlang-questions] Erlang n00b figuring out bins > > The first thing I'd say is to ask why you want a list of strings (lists) - I initially went down the strings are lists route but am definitely in the strings as binaries camp now, as are an increasing number of Erlang projects. Definitely a question of style, but the style du jour appears to be binaries. Also the example you give doesn't run for me (as I'd expect) - binary_to_list( [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>]) is trying to convert a list of binaries rather than a binary which is what it expects > > Eshell V5.10.1 (abort with ^G) > 1> binary_to_list( [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>]). > ** exception error: bad argument > in function binary_to_list/1 > called as binary_to_list([<<"s">>,<<"t">>,<<"u">>,<<"f">>,<<"f">>]) > > > That aside there are two jobs going on there: > You need to split the input on "." > You want the output as a list of strings (= list of lists) > > You can go about that in (at least) two different ways > > Split the binary and convert the binary parts into strings > Convert the input to a list and split it > > Thankfully there are library calls that do just what you need. Lets look at splitting then converting - binary:split/2 looks like it does the job > > Eshell V5.10.1 (abort with ^G) > 1> X = <<"stuff.junk.excess">>. > <<"stuff.junk.excess">> > 2> binary:split(X, [<<".">>]). > [<<"stuff">>,<<"junk.excess">>] > > > Close, but not quite what we want as it has only split on the first occurrence. Thankfully there's binary:split/3 which gives us a little more control > > 3> binary:split(X, [<<".">>], [global]). > [<<"stuff">>,<<"junk">>,<<"excess">>] > > > Perfect - now we just need to convert the binary strings into lists. Maybe something like > > 4> [binary_to_list(Y) || Y <- binary:split(X, [<<".">>], [global])]. > ["stuff","junk","excess"] > > > Again with the other approach (convert then split), there's a function that does what we need - string:tokens/2. This expects a string (aka list) as input > > 5> string:tokens(binary_to_list(X), "."). > ["stuff","junk","excess"] > > > Not sure if any of that counts as "more graceful" but more widely you should definitely stick with it - Erlang is certainly worth the initial head-scratching. > > Adrian > -- > > Dr Adrian Roe > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > > > On Tuesday, 24 September 2013 at 15:46, jasingleton.work@REDACTED (mailto:jasingleton.work@REDACTED) wrote: > > > I've done a lot of development, but it's been quite a while since I've done anything > > functional. Erlang is still brand-new for me, so be warned. > > > > I've got a (conceptual) string arriving in my sw as a binary, e.g. <<"stuff.junk.excess">> > > and trying to transform it into a list of strings, e.g. ["stuff", "junk", "excess"] > > The mechanism I'm using is giving me a list of lists of binaries, e.g. > > [ [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>], [<<"j">>, <<"u">> (etc) > > > > I ended up hacking it to just take each of the lists of binaries and run it through > > list_to_binary and then binary to list. The result is what I really wanted, which is > > list_to_binary(binary_to_list( [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>])) > > gives me back > > "stuff" > > > > That's all well and good, and functional, but has two problems: > > 1) I honestly don't get why it worked; I would expect those two operations to > > give me back something unchanged. > > 2) There's got to be a more graceful way to achieve this. > > > > Thoughts? > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Tue Sep 24 21:57:10 2013 From: bob@REDACTED (Bob Ippolito) Date: Tue, 24 Sep 2013 12:57:10 -0700 Subject: [erlang-questions] Fw: Erlang n00b figuring out bins In-Reply-To: <178486F220764E608389A3078C966EAB@gmail.com> References: <21000597-8eb6-4c0d-82e9-6e1f76ea120c@googlegroups.com> <178486F220764E608389A3078C966EAB@gmail.com> Message-ID: I think the desire to convert the binary to lists is a bit questionable in the first place. Also, if it is text, binary_to_list/1 is probably the wrong way to do it, since it will be a list of bytes in some encoding and not unicode code points. unicode:characters_to_list/1 might be more appropriate, but without any context it's hard to give good suggestions. On Tue, Sep 24, 2013 at 12:46 PM, Adrian Roe wrote: > Initial message failed - retrying > > -- > Adrian Roe > Sent with Sparrow > > Forwarded message: > > *From:* Adrian Roe > *To:* jasingleton.work@REDACTED > *Cc:* erlang-programming@REDACTED > *Date:* Tuesday, 24 September 2013 20:44:41 > *Subject:* Re: [erlang-questions] Erlang n00b figuring out bins > > The first thing I'd say is to ask why you want a list of strings (lists) > - I initially went down the strings are lists route but am definitely in > the strings as binaries camp now, as are an increasing number of Erlang > projects. Definitely a question of style, but the style du jour appears to > be binaries. Also the example you give doesn't run for me (as I'd expect) - binary_to_list( > [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>]) is trying to convert a > list of binaries rather than a binary which is what it expects > > Eshell V5.10.1 (abort with ^G) > 1> binary_to_list( [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>]). > ** exception error: bad argument > in function binary_to_list/1 > called as binary_to_list([<<"s">>,<<"t">>,<<"u">>,<<"f">>,<<"f">>]) > > That aside there are two jobs going on there: > > - You need to split the input on "." > - You want the output as a list of strings (= list of lists) > > You can go about that in (at least) two different ways > > - Split the binary and convert the binary parts into strings > - Convert the input to a list and split it > > Thankfully there are library calls that do just what you need. Lets look > at splitting then converting - binary:split/2 looks like it does the job > > Eshell V5.10.1 (abort with ^G) > 1> X = <<"stuff.junk.excess">>. > <<"stuff.junk.excess">> > 2> binary:split(X, [<<".">>]). > [<<"stuff">>,<<"junk.excess">>] > > Close, but not quite what we want as it has only split on the first > occurrence. Thankfully there's binary:split/3 which gives us a little more > control > > 3> binary:split(X, [<<".">>], [global]). > [<<"stuff">>,<<"junk">>,<<"excess">>] > > Perfect - now we just need to convert the binary strings into lists. > Maybe something like > > 4> [binary_to_list(Y) || Y <- binary:split(X, [<<".">>], [global])]. > ["stuff","junk","excess"] > > Again with the other approach (convert then split), there's a function > that does what we need - string:tokens/2. This expects a string (aka list) > as input > > 5> string:tokens(binary_to_list(X), "."). > ["stuff","junk","excess"] > > Not sure if any of that counts as "more graceful" but more widely you > should definitely stick with it - Erlang is certainly worth the initial > head-scratching. > > Adrian > -- > Dr Adrian Roe > Sent with Sparrow > > On Tuesday, 24 September 2013 at 15:46, jasingleton.work@REDACTED wrote: > > I've done a lot of development, but it's been quite a while since I've > done anything > functional. Erlang is still brand-new for me, so be warned. > > I've got a (conceptual) string arriving in my sw as a binary, e.g. > <<"stuff.junk.excess">> > and trying to transform it into a list of strings, e.g. ["stuff", "junk", > "excess"] > The mechanism I'm using is giving me a list of lists of binaries, e.g. > [ [<<"s">>, <<"t">>, <<"u">>, <<"f">>, <<"f">>], [<<"j">>, <<"u">> (etc) > > I ended up hacking it to just take each of the lists of binaries and run > it through > list_to_binary and then binary to list. The result is what I really > wanted, which is > list_to_binary(binary_to_list( [<<"s">>, <<"t">>, <<"u">>, <<"f">>, > <<"f">>])) > gives me back > "stuff" > > That's all well and good, and functional, but has two problems: > 1) I honestly don't get why it worked; I would expect those two operations > to > give me back something unchanged. > 2) There's got to be a more graceful way to achieve this. > > Thoughts? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown@REDACTED Tue Sep 24 23:57:31 2013 From: ryankbrown@REDACTED (Ryan Brown) Date: Tue, 24 Sep 2013 15:57:31 -0600 Subject: [erlang-questions] Problematic SSL message publication Message-ID: We have a fairly high-volume (28m+ messages/day) application that works, essentially, as a pub-sub for many internal and external applications. We are experiencing an issue under load where and subscribers that are using ssl are experiencing a significant slowness in delivery. When we restart the server we begin watching the processes in our application and we can, very reliably and fairly quickly, see the ssl_manager_sup process message queue start increasing in size. As this happens, all messages for those subscribers with ssl endpoints begin to back-up and delivery slows to a near halt. It appears that ssl is a bit of a known issue within erlang. Or, at least, a tricky use-case to perfect. Are there any known issues that we can address in terms of configuration, best practices in setting-up ssl connections or anything else for that matter we can be doing to resolve this issue or more clearly identify the root cause? We're in active troubleshooting mode and doing what we can to dig deeper including dissecting the code to the kernel. We are running Erlang 15B02 (64bit). Best regards. Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy.gooden@REDACTED Wed Sep 25 02:23:18 2013 From: andy.gooden@REDACTED (Andy Gooden) Date: Tue, 24 Sep 2013 17:23:18 -0700 Subject: [erlang-questions] Gen_fsm : race condition with a next_state timeout of zero? Message-ID: Hi All, I have a gen_fsm, where upon completing the handling of a message in state_A, I want to immediately transition into state_B... so I have state_A evaluate to: {next_state, state_B, State, Timeout = 0}. In my code, this is the only way to transition into state_B, so I only have implemented: state_B ( timeout, State ) -> ... Is it possible, if other processes are sending my fsm messages, for me to get a message in my queue while executing state_A code, but then have to handle that message with state_B code? ie will the immediate transition to state_B via the Timeout = 0 trump any messages that landed in the queue? The reason I ask is that I see an error message that looks like state_B is having to handle a message other than 'timeout', even though the only way to get into state_B is from state_A with Timeout = 0. Thanks! Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Wed Sep 25 06:15:21 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Wed, 25 Sep 2013 07:15:21 +0300 Subject: [erlang-questions] Gen_fsm : race condition with a next_state timeout of zero? In-Reply-To: References: Message-ID: Hello, Yes, it is possible. Timeout message is stored to mailbox if there are other messages in mailbox they are evaluated before timeout. I am doing explicit calls to avoid such issues. stateA(...) -> %% do something stateB(myevent Best Regards, Dmitry >-|-|-*> > On 25.9.2013, at 3.23, Andy Gooden wrote: > > Hi All, > I have a gen_fsm, where upon completing the handling of a message in state_A, I want to immediately transition into state_B... so I have state_A evaluate to: {next_state, state_B, State, Timeout = 0}. > > In my code, this is the only way to transition into state_B, so I only have implemented: > state_B ( timeout, State ) -> ... > > Is it possible, if other processes are sending my fsm messages, for me to get a message in my queue while executing state_A code, but then have to handle that message with state_B code? > ie will the immediate transition to state_B via the Timeout = 0 trump any messages that landed in the queue? > > The reason I ask is that I see an error message that looks like state_B is having to handle a message other than 'timeout', even though the only way to get into state_B is from state_A with Timeout = 0. > > Thanks! > Andy > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ingela.andin@REDACTED Wed Sep 25 10:09:06 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 25 Sep 2013 10:09:06 +0200 Subject: [erlang-questions] Problematic SSL message publication In-Reply-To: References: Message-ID: Hi! 2013/9/24 Ryan Brown > We have a fairly high-volume (28m+ messages/day) application that works, > essentially, as a pub-sub for many internal and external applications. We > are experiencing an issue under load where and subscribers that are using > ssl are experiencing a significant slowness in delivery. When we restart > the server we begin watching the processes in our application and we can, > very reliably and fairly quickly, see the ssl_manager_sup process message > queue start increasing in size. As this happens, all messages for those > subscribers with ssl endpoints begin to back-up and delivery slows to a > near halt. > > What messages are there in the queue? It appears that ssl is a bit of a known issue within erlang. Or, at least, > a tricky use-case to perfect. Are there any known issues that we can > address in terms of configuration, best practices in setting-up ssl > connections or anything else for that matter we can be doing to resolve > this issue or more clearly identify the root cause? We're in active > troubleshooting mode and doing what we can to dig deeper including > dissecting the code to the kernel. > > We are running Erlang 15B02 (64bit). > > > So that would be ssl-5.1? Is it possible to reproduce the problem on the latest version ? Regards Ingela Erlang/OTP team - Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown@REDACTED Wed Sep 25 14:18:20 2013 From: ryankbrown@REDACTED (Ryan Brown) Date: Wed, 25 Sep 2013 06:18:20 -0600 Subject: [erlang-questions] Problematic SSL message publication In-Reply-To: References: Message-ID: Hi Ingela, Thank you for the response. I suppose I don't really know how to see what messages are in the ssl queue. What I am looking at is nregs and watching the "msgs" column growing at a steady pace. Is there a way to look at these messages? As for reproducing in the latest version of ssl. I was planning on upgrading a machine to 16 today and attempting to reproduce. This should get me the latest ssl version correct? Best regards, Ryan Brown On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin wrote: > > Hi! > > 2013/9/24 Ryan Brown > >> We have a fairly high-volume (28m+ messages/day) application that works, >> essentially, as a pub-sub for many internal and external applications. We >> are experiencing an issue under load where and subscribers that are using >> ssl are experiencing a significant slowness in delivery. When we restart >> the server we begin watching the processes in our application and we can, >> very reliably and fairly quickly, see the ssl_manager_sup process message >> queue start increasing in size. As this happens, all messages for those >> subscribers with ssl endpoints begin to back-up and delivery slows to a >> near halt. >> >> > What messages are there in the queue? > > > It appears that ssl is a bit of a known issue within erlang. Or, at least, >> a tricky use-case to perfect. Are there any known issues that we can >> address in terms of configuration, best practices in setting-up ssl >> connections or anything else for that matter we can be doing to resolve >> this issue or more clearly identify the root cause? We're in active >> troubleshooting mode and doing what we can to dig deeper including >> dissecting the code to the kernel. >> >> We are running Erlang 15B02 (64bit). >> >> >> > So that would be ssl-5.1? Is it possible to reproduce the problem on the > latest version ? > > Regards Ingela Erlang/OTP team - Ericsson AB > -- -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Wed Sep 25 14:58:27 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 25 Sep 2013 14:58:27 +0200 Subject: [erlang-questions] Problematic SSL message publication In-Reply-To: References: Message-ID: Hi! 2013/9/25 Ryan Brown > Hi Ingela, > > Thank you for the response. I suppose I don't really know how to see what > messages are in the ssl queue. What I am looking at is nregs and watching > the "msgs" column growing at a steady pace. Is there a way to look at these > messages? > > You can use the observer application to look at erlang processes. http://www.erlang.org/erldoc?q=observe&x=0&y=0 > As for reproducing in the latest version of ssl. I was planning on > upgrading a machine to 16 today and attempting to reproduce. This should > get me the latest ssl version correct? > > R16B02 (ssl-5.3.1) is the latest. There is a bug in ssl-5.1 (R15B02) version (fix in ssl5.2) that perhaps could be related to your problem. Regards Ingela Erlang/OTP Team - Ericsson AB On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin wrote: > > Hi! > > 2013/9/24 Ryan Brown > >> We have a fairly high-volume (28m+ messages/day) application that works, >> essentially, as a pub-sub for many internal and external applications. We >> are experiencing an issue under load where and subscribers that are using >> ssl are experiencing a significant slowness in delivery. When we restart >> the server we begin watching the processes in our application and we can, >> very reliably and fairly quickly, see the ssl_manager_sup process message >> queue start increasing in size. As this happens, all messages for those >> subscribers with ssl endpoints begin to back-up and delivery slows to a >> near halt. >> >> > What messages are there in the queue? > > > It appears that ssl is a bit of a known issue within erlang. Or, at >> least, a tricky use-case to perfect. Are there any known issues that we can >> address in terms of configuration, best practices in setting-up ssl >> connections or anything else for that matter we can be doing to resolve >> this issue or more clearly identify the root cause? We're in active >> troubleshooting mode and doing what we can to dig deeper including >> dissecting the code to the kernel. >> >> We are running Erlang 15B02 (64bit). >> >> >> > So that would be ssl-5.1? Is it possible to reproduce the problem on the > latest version ? > > Regards Ingela Erlang/OTP team - Ericsson AB > -- > -rb > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmeiklejohn@REDACTED Wed Sep 25 16:44:06 2013 From: cmeiklejohn@REDACTED (Christopher Meiklejohn) Date: Wed, 25 Sep 2013 10:44:06 -0400 Subject: [erlang-questions] Erlang Workshop '13 Party Message-ID: Hey everyone, We wanted to let everyone know that Basho is celebrating Erlang Workshop '13 at its offices in Cambridge, Massachusetts on Friday, September 27 at 8:00pm. Please RSVP if you're interested: https://guestlistapp.com/events/178131 This event is open to the public and Basho will provide food and beverage. We will be hosting Kostis Sagonas from Uppsala University who will be giving a talk on Concuerror, a testing tool that supports development of concurrent Erlang programs. -- Kostis Sagonas Uppsala University and National Technical University of Athens Concurrent programs are notoriously difficult to get completely error free. This, sadly, is as true in Erlang as it is in other languages. Specific process interleavings that occur only rarely can result in unexpected errors that manage to puzzle and surprise even expert programmers. Furthermore concurrency-related errors are often difficult to reproduce. This talk will show some examples of such errors. More importantly, it will introduce Concuerror, a systematic testing (aka stateless model checking) tool that supports the test-driven development of concurrent Erlang programs in order to detect early and eliminate the vast majority of concurrency-related errors (e.g. data races, unexpected crashes, deadlocks, etc.) that may occur in their execution. A salient feature of Concuerror is that it is completely automatic and works out of the box with existing (Eunit) test suites and in principle finds all concurrency related errors that are triggered by a specific test. In addition, the tool shows execution traces that lead to errors, it can deterministically replay them, and comes with a variety of options to speed up the exploration of possible interleavings. The talk will also include a live demo and, hopefully, lots of interaction with the audience. -- Christopher Meiklejohn Software Engineer Basho Technologies, Inc. From robert.virding@REDACTED Wed Sep 25 18:28:04 2013 From: robert.virding@REDACTED (Robert Virding) Date: Wed, 25 Sep 2013 18:28:04 +0200 (CEST) Subject: [erlang-questions] Understanding the scalability of Erlang In-Reply-To: <20130922142523.GB25462@circlewave.net> References: <20130922141920.GA25462@circlewave.net> <20130922142523.GB25462@circlewave.net> Message-ID: <1107562927.195067.1380126484548.JavaMail.zimbra@erlang-solutions.com> ----- Original Message ----- > From: "Jachym Holecek" > > # Jachym Holecek 2013-09-22: > > The ability to spawn large number of very cheap processes and the fact > > of eliminating resource sharing between them nearly completely are good > > enablers for scalability (in addition to enabling other, parhaps more > > important, things). That said however, nothing in the world will make > > arbitrary application "scale" by magic. There is no excuse for proper > > design and implementation towards the required operational characteristics > > at application level. > > Correction: I meant "no excuse for lack of" there. > > BR, > -- Jachym For a very good description of at least part of the problem read about Amdahl's Law, http://en.wikipedia.org/wiki/Amdahl%27s_law . It shows very clearly why you need proper design and implementation to get scalability. Robert From ryankbrown@REDACTED Wed Sep 25 19:07:59 2013 From: ryankbrown@REDACTED (Ryan Brown) Date: Wed, 25 Sep 2013 11:07:59 -0600 Subject: [erlang-questions] Problematic SSL message publication In-Reply-To: References: Message-ID: Excellent Ingela! By upgrading to the R!6B02 we have been able to eliminate the above stated issue with the Erlang message queues growing. However, we are still seeing VERY slow throughput to ssl endpoints. In our limited, scaled-back, test. We are seeing message throughput to an ssl endpoint at 12 msgs/sec and non-ssl of 300+ msgs/sec. Is there a connection limit for ssl connections related to Erlang directly? Thank you! Best regards, Ryan Brown On Wed, Sep 25, 2013 at 6:58 AM, Ingela Andin wrote: > Hi! > > > 2013/9/25 Ryan Brown > >> Hi Ingela, >> >> Thank you for the response. I suppose I don't really know how to see what >> messages are in the ssl queue. What I am looking at is nregs and watching >> the "msgs" column growing at a steady pace. Is there a way to look at these >> messages? >> >> > You can use the observer application to look at erlang processes. > > http://www.erlang.org/erldoc?q=observe&x=0&y=0 > > > >> As for reproducing in the latest version of ssl. I was planning on >> upgrading a machine to 16 today and attempting to reproduce. This should >> get me the latest ssl version correct? >> >> > R16B02 (ssl-5.3.1) is the latest. There is a bug in ssl-5.1 (R15B02) > version (fix in ssl5.2) that perhaps could be related to your problem. > > Regards Ingela Erlang/OTP Team - Ericsson AB > > > On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin wrote: > >> >> Hi! >> >> 2013/9/24 Ryan Brown >> >>> We have a fairly high-volume (28m+ messages/day) application that works, >>> essentially, as a pub-sub for many internal and external applications. We >>> are experiencing an issue under load where and subscribers that are using >>> ssl are experiencing a significant slowness in delivery. When we restart >>> the server we begin watching the processes in our application and we can, >>> very reliably and fairly quickly, see the ssl_manager_sup process message >>> queue start increasing in size. As this happens, all messages for those >>> subscribers with ssl endpoints begin to back-up and delivery slows to a >>> near halt. >>> >>> >> What messages are there in the queue? >> >> >> It appears that ssl is a bit of a known issue within erlang. Or, at >>> least, a tricky use-case to perfect. Are there any known issues that we can >>> address in terms of configuration, best practices in setting-up ssl >>> connections or anything else for that matter we can be doing to resolve >>> this issue or more clearly identify the root cause? We're in active >>> troubleshooting mode and doing what we can to dig deeper including >>> dissecting the code to the kernel. >>> >>> We are running Erlang 15B02 (64bit). >>> >>> >>> >> So that would be ssl-5.1? Is it possible to reproduce the problem on the >> latest version ? >> >> Regards Ingela Erlang/OTP team - Ericsson AB >> > > > > -- >> -rb >> > > -- -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christopher.Phillips@REDACTED Wed Sep 25 20:28:11 2013 From: Christopher.Phillips@REDACTED (Phillips, Christopher) Date: Wed, 25 Sep 2013 18:28:11 +0000 Subject: [erlang-questions] Distributed apps when application terminates Message-ID: I have a release built around a distributed application. If I spin two nodes up, things are configured properly such that if I attach and q() out of the node the application is actively running on, failover occurs, the application starts up on the other node. What I'm finding is that in the same situation, if I kill the top level supervisor (either by directly sending it an exit message, or having a child fail enough times to pass the max restart threshold), I _don't_ fail over. I do, however, receive a node down message on the other node. I'm wondering if this is intentional, a bug, or if I'm doing something wrong. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yogishb@REDACTED Wed Sep 25 20:57:16 2013 From: yogishb@REDACTED (Yogish Baliga) Date: Wed, 25 Sep 2013 11:57:16 -0700 Subject: [erlang-questions] Distributed apps when application terminates In-Reply-To: References: Message-ID: According to distributed app documentation: If the node where the application is running goes down, the application is restarted (after the specified timeout) at the first node, specified by the distributed configuration parameter, which is up and running. This is called a *failover*. In your case, your node did not go down but supervisor is stopped. I did a test in the past of the application fail over by disabling ethernet adapater on the master node. -- baliga On Wed, Sep 25, 2013 at 11:28 AM, Phillips, Christopher < Christopher.Phillips@REDACTED> wrote: > I have a release built around a distributed application. > > If I spin two nodes up, things are configured properly such that if I > attach and q() out of the node the application is actively running on, > failover occurs, the application starts up on the other node. > > What I'm finding is that in the same situation, if I kill the top level > supervisor (either by directly sending it an exit message, or having a > child fail enough times to pass the max restart threshold), I _don't_ fail > over. I do, however, receive a node down message on the other node. I'm > wondering if this is intentional, a bug, or if I'm doing something wrong. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christopher.Phillips@REDACTED Wed Sep 25 21:11:23 2013 From: Christopher.Phillips@REDACTED (Phillips, Christopher) Date: Wed, 25 Sep 2013 19:11:23 +0000 Subject: [erlang-questions] Distributed apps when application terminates In-Reply-To: Message-ID: The node appears to have gone down; if I'm attached to it I get dropped back to the shell. If I check for running Erlang processes (ps aux | grep beam) I see nothing. The other node received a 'nodedown' message. That's what's confusing me; if it was still up I'd understand, and in the past when just stopping the application manually I accepted it not failing over. This is a bit different. From: Yogish Baliga > Date: Wednesday, September 25, 2013 2:57 PM To: Chris Phillips > Cc: "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application terminates According to distributed app documentation: If the node where the application is running goes down, the application is restarted (after the specified timeout) at the first node, specified by the distributed configuration parameter, which is up and running. This is called a failover. In your case, your node did not go down but supervisor is stopped. I did a test in the past of the application fail over by disabling ethernet adapater on the master node. -- baliga On Wed, Sep 25, 2013 at 11:28 AM, Phillips, Christopher > wrote: I have a release built around a distributed application. If I spin two nodes up, things are configured properly such that if I attach and q() out of the node the application is actively running on, failover occurs, the application starts up on the other node. What I'm finding is that in the same situation, if I kill the top level supervisor (either by directly sending it an exit message, or having a child fail enough times to pass the max restart threshold), I _don't_ fail over. I do, however, receive a node down message on the other node. I'm wondering if this is intentional, a bug, or if I'm doing something wrong. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From yogishb@REDACTED Wed Sep 25 21:23:22 2013 From: yogishb@REDACTED (Yogish Baliga) Date: Wed, 25 Sep 2013 12:23:22 -0700 Subject: [erlang-questions] Distributed apps when application terminates In-Reply-To: References: Message-ID: May be dumb question, but worth asking. Is application started on the fail over node? -- baliga On Wed, Sep 25, 2013 at 12:11 PM, Phillips, Christopher < Christopher.Phillips@REDACTED> wrote: > The node appears to have gone down; if I'm attached to it I get > dropped back to the shell. If I check for running Erlang processes (ps aux > | grep beam) I see nothing. The other node received a 'nodedown' message. > That's what's confusing me; if it was still up I'd understand, and in the > past when just stopping the application manually I accepted it not failing > over. This is a bit different. > > From: Yogish Baliga > Date: Wednesday, September 25, 2013 2:57 PM > To: Chris Phillips > Cc: "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application > terminates > > According to distributed app documentation: > > If the node where the application is running goes down, the application is > restarted (after the specified timeout) at the first node, specified by the > distributed configuration parameter, which is up and running. This is > called a *failover*. > > In your case, your node did not go down but supervisor is stopped. I did > a test in the past of the application fail over by disabling ethernet > adapater on the master node. > > -- baliga > > > > On Wed, Sep 25, 2013 at 11:28 AM, Phillips, Christopher < > Christopher.Phillips@REDACTED> wrote: > >> I have a release built around a distributed application. >> >> If I spin two nodes up, things are configured properly such that if I >> attach and q() out of the node the application is actively running on, >> failover occurs, the application starts up on the other node. >> >> What I'm finding is that in the same situation, if I kill the top level >> supervisor (either by directly sending it an exit message, or having a >> child fail enough times to pass the max restart threshold), I _don't_ fail >> over. I do, however, receive a node down message on the other node. I'm >> wondering if this is intentional, a bug, or if I'm doing something wrong. >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christopher.Phillips@REDACTED Wed Sep 25 21:32:07 2013 From: Christopher.Phillips@REDACTED (Phillips, Christopher) Date: Wed, 25 Sep 2013 19:32:07 +0000 Subject: [erlang-questions] Distributed apps when application terminates In-Reply-To: Message-ID: Yes, started but not loaded; I.e., running application:start(app name) again leads to an error, already started, but application:which_applications() does not list it. Same setup as I have when failover works by explicitly killing the initial node. From: Yogish Baliga > Date: Wednesday, September 25, 2013 3:23 PM To: Chris Phillips > Cc: Yogish Baliga >, "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application terminates May be dumb question, but worth asking. Is application started on the fail over node? -- baliga On Wed, Sep 25, 2013 at 12:11 PM, Phillips, Christopher > wrote: The node appears to have gone down; if I'm attached to it I get dropped back to the shell. If I check for running Erlang processes (ps aux | grep beam) I see nothing. The other node received a 'nodedown' message. That's what's confusing me; if it was still up I'd understand, and in the past when just stopping the application manually I accepted it not failing over. This is a bit different. From: Yogish Baliga > Date: Wednesday, September 25, 2013 2:57 PM To: Chris Phillips > Cc: "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application terminates According to distributed app documentation: If the node where the application is running goes down, the application is restarted (after the specified timeout) at the first node, specified by the distributed configuration parameter, which is up and running. This is called a failover. In your case, your node did not go down but supervisor is stopped. I did a test in the past of the application fail over by disabling ethernet adapater on the master node. -- baliga On Wed, Sep 25, 2013 at 11:28 AM, Phillips, Christopher > wrote: I have a release built around a distributed application. If I spin two nodes up, things are configured properly such that if I attach and q() out of the node the application is actively running on, failover occurs, the application starts up on the other node. What I'm finding is that in the same situation, if I kill the top level supervisor (either by directly sending it an exit message, or having a child fail enough times to pass the max restart threshold), I _don't_ fail over. I do, however, receive a node down message on the other node. I'm wondering if this is intentional, a bug, or if I'm doing something wrong. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christopher.Phillips@REDACTED Wed Sep 25 22:06:09 2013 From: Christopher.Phillips@REDACTED (Phillips, Christopher) Date: Wed, 25 Sep 2013 20:06:09 +0000 Subject: [erlang-questions] Distributed apps when application terminates In-Reply-To: Message-ID: I think I figured it out. I tried it again from start; I believe I had started the app on the second node outside of the release (I.e., started the release binary with a clean target, and then manually started the app with application:start(app name) ), such that the kernel wasn't kicking in. I just repeated it from scratch, making sure I started the actual release target, and it appears to work fine. From: , Chris Phillips > Date: Wednesday, September 25, 2013 3:32 PM To: Yogish Baliga > Cc: "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application terminates Yes, started but not loaded; I.e., running application:start(app name) again leads to an error, already started, but application:which_applications() does not list it. Same setup as I have when failover works by explicitly killing the initial node. From: Yogish Baliga > Date: Wednesday, September 25, 2013 3:23 PM To: Chris Phillips > Cc: Yogish Baliga >, "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application terminates May be dumb question, but worth asking. Is application started on the fail over node? -- baliga On Wed, Sep 25, 2013 at 12:11 PM, Phillips, Christopher > wrote: The node appears to have gone down; if I'm attached to it I get dropped back to the shell. If I check for running Erlang processes (ps aux | grep beam) I see nothing. The other node received a 'nodedown' message. That's what's confusing me; if it was still up I'd understand, and in the past when just stopping the application manually I accepted it not failing over. This is a bit different. From: Yogish Baliga > Date: Wednesday, September 25, 2013 2:57 PM To: Chris Phillips > Cc: "erlang-questions@REDACTED" > Subject: Re: [erlang-questions] Distributed apps when application terminates According to distributed app documentation: If the node where the application is running goes down, the application is restarted (after the specified timeout) at the first node, specified by the distributed configuration parameter, which is up and running. This is called a failover. In your case, your node did not go down but supervisor is stopped. I did a test in the past of the application fail over by disabling ethernet adapater on the master node. -- baliga On Wed, Sep 25, 2013 at 11:28 AM, Phillips, Christopher > wrote: I have a release built around a distributed application. If I spin two nodes up, things are configured properly such that if I attach and q() out of the node the application is actively running on, failover occurs, the application starts up on the other node. What I'm finding is that in the same situation, if I kill the top level supervisor (either by directly sending it an exit message, or having a child fail enough times to pass the max restart threshold), I _don't_ fail over. I do, however, receive a node down message on the other node. I'm wondering if this is intentional, a bug, or if I'm doing something wrong. _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown@REDACTED Wed Sep 25 23:28:02 2013 From: ryankbrown@REDACTED (Ryan Brown) Date: Wed, 25 Sep 2013 15:28:02 -0600 Subject: [erlang-questions] Problematic SSL message publication In-Reply-To: References: Message-ID: Thank you Ingela. I will work on creating a test to compare to openssl. In the mean time, what we are currently doing is this. Our application sends requests to various hosts via ibrowse (http/s). The messages are basic HTTP form POSTs (application/x-www-form-urlencoded). Our test just sends these messages to our application which pops them on a queue in RabbitMQ then one-by-one pulls them off and attempt to deliver them. Fairly straightforward really. Best, Ryan On Wed, Sep 25, 2013 at 2:59 PM, Ingela Andin wrote: > Hi! > > On Wed, Sep 25, 2013 at 7:07 PM, Ryan Brown wrote: > >> Excellent Ingela! By upgrading to the R!6B02 we have been able to >> eliminate the above stated issue with the Erlang message queues growing. >> However, we are still seeing VERY slow throughput to ssl endpoints. In our >> limited, scaled-back, test. We are seeing message throughput to an ssl >> endpoint at 12 msgs/sec and non-ssl of 300+ msgs/sec. Is there a connection >> limit for ssl connections related to Erlang directly? >> >> Thank you! >> >> >> > Could you send me more details how you are performing this test. It would > be interesting if I could run the same kind of test as you do. And when you > say messages what more exactly do mean? Messages can be on many abstraction > levels (tcp messages, tls messages, http messages etc). Could you perform > the same kind of throughput test running for instance openssl? > > Regards Ingela Erlang/OTP team - Ericsson AB > > > > >> >> On Wed, Sep 25, 2013 at 6:58 AM, Ingela Andin wrote: >> >>> Hi! >>> >>> >>> 2013/9/25 Ryan Brown >>> >>>> Hi Ingela, >>>> >>>> Thank you for the response. I suppose I don't really know how to see >>>> what messages are in the ssl queue. What I am looking at is nregs and >>>> watching the "msgs" column growing at a steady pace. Is there a way to look >>>> at these messages? >>>> >>>> >>> You can use the observer application to look at erlang processes. >>> >>> http://www.erlang.org/erldoc?q=observe&x=0&y=0 >>> >>> >>> >>>> As for reproducing in the latest version of ssl. I was planning on >>>> upgrading a machine to 16 today and attempting to reproduce. This should >>>> get me the latest ssl version correct? >>>> >>>> >>> R16B02 (ssl-5.3.1) is the latest. There is a bug in ssl-5.1 (R15B02) >>> version (fix in ssl5.2) that perhaps could be related to your problem. >>> >>> Regards Ingela Erlang/OTP Team - Ericsson AB >>> >>> >>> On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin wrote: >>> >>>> >>>> Hi! >>>> >>>> 2013/9/24 Ryan Brown >>>> >>>>> We have a fairly high-volume (28m+ messages/day) application that >>>>> works, essentially, as a pub-sub for many internal and external >>>>> applications. We are experiencing an issue under load where and subscribers >>>>> that are using ssl are experiencing a significant slowness in delivery. >>>>> When we restart the server we begin watching the processes in our >>>>> application and we can, very reliably and fairly quickly, see the >>>>> ssl_manager_sup process message queue start increasing in size. As this >>>>> happens, all messages for those subscribers with ssl endpoints begin to >>>>> back-up and delivery slows to a near halt. >>>>> >>>>> >>>> What messages are there in the queue? >>>> >>>> >>>> It appears that ssl is a bit of a known issue within erlang. Or, at >>>>> least, a tricky use-case to perfect. Are there any known issues that we can >>>>> address in terms of configuration, best practices in setting-up ssl >>>>> connections or anything else for that matter we can be doing to resolve >>>>> this issue or more clearly identify the root cause? We're in active >>>>> troubleshooting mode and doing what we can to dig deeper including >>>>> dissecting the code to the kernel. >>>>> >>>>> We are running Erlang 15B02 (64bit). >>>>> >>>>> >>>>> >>>> So that would be ssl-5.1? Is it possible to reproduce the problem on >>>> the latest version ? >>>> >>>> Regards Ingela Erlang/OTP team - Ericsson AB >>>> >>> >>> >>> >>> -- >>>> -rb >>>> >>> >>> >> >> >> -- >> -rb >> > > -- -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From ludovic@REDACTED Thu Sep 26 00:20:49 2013 From: ludovic@REDACTED (Ludovic Demblans) Date: Thu, 26 Sep 2013 00:20:49 +0200 Subject: [erlang-questions] Desing of MVC models In-Reply-To: References: <5232C07B.7000306@ninenines.eu> <5FBD49CE-545A-4605-8850-87ADC8217E8C@gmail.com> Message-ID: Sorry i missed some replies, Thank you all for your great advice ! - Ludovic From bengt.kleberg@REDACTED Thu Sep 26 08:54:40 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 26 Sep 2013 08:54:40 +0200 Subject: [erlang-questions] beginner: ttb, add node to existing trace setup? Message-ID: <1380178480.4796.9.camel@sekic1152.release> Greetings, Is it possible to ad a node to an already existing trace setup? I did not find any explanation in the manual on how to setup a trace (ttb:tracer(node()), add ttb:p(all, [call]) and then start tracing on ttb:tp(module, func) and then add another node (newly created) to the same trace setup (module, func on call). Anybody that has done this? bengt From kenji.rikitake@REDACTED Thu Sep 26 08:55:09 2013 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Thu, 26 Sep 2013 15:55:09 +0900 Subject: [erlang-questions] Threaded VM with llvm-gcc In-Reply-To: <03DB64B5-A64A-42A1-843A-64D7875C068D@gmail.com> References: <75E2E84D-4193-4437-B370-DD0CCA4850B3@gmail.com> <03DB64B5-A64A-42A1-843A-64D7875C068D@gmail.com> Message-ID: <20130926065509.GA75077@k2r.org> I'd like to find out if any performance increase is obtained by the compile-time options for LLVM (clang) on FreeBSD 9-STABLE and later. (FreeBSD 10 will drop gcc and focus on clang as the default compiler) Kenji Rikitake ++> Anthony Ramine [2013-09-24 11:50:05 +0200]: > Le 24 sept. 2013 ? 07:37, Bj?rn Gustavsson a ?crit : > > The test is where to protect against an old broken version > > of llvm-gcc that did not handle jump tables correctly. > > It will also make sure that Erlang/OTP can be compiled > > on systems where no compiler that supports jump tables > > can be found. > > Shouldn't the test be fixed for more recent versions? I don't like limiting stuff because of old and outdated versions. From ignatovs@REDACTED Thu Sep 26 17:58:36 2013 From: ignatovs@REDACTED (Sergey Ignatov) Date: Thu, 26 Sep 2013 08:58:36 -0700 (PDT) Subject: [erlang-questions] [ANN] Erlang plugin for IntelliJ IDEA version 0.5 Message-ID: Hi all, Today I'm happy to announce the new version of Erlang plugin for IntelliJ-based IDEs. Some highlights of the latest version: - Rerun failed tests action - Enhanced completion for attributes and include/include_lib attributes - Introduce variable and function refactorings - Go to super action for callback implementations - Improved EUnit support: run single test and rerun failed tests action - Enhanced native compilation: parse_transform and behaviour detection - Debugger (experimental) - Enhanced formatter and color schema settings - Rebar-based compilation - Performance improvements - RubyMine and other "small IDEs" compatibility - Bugfixes - And many others, please check out release notes: https://github.com/ignatov/intellij-erlang#version-05 Many thanks to our new contributor: Sergey Savenko who joined our small team this summer. Plugin page in IntelliJ repository: http://plugins.jetbrains.com/plugin/?pluginId=7083 Source code: https://github.com/ignatov/intellij-erlang If you are interested in contribution to the plugin, feel free to use the issue tracker: https://github.com/ignatov/intellij-erlang/issues Stay tuned. Cheers, Sergey Ignatov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donpedrothird@REDACTED Thu Sep 26 23:46:48 2013 From: donpedrothird@REDACTED (Evgeny M) Date: Thu, 26 Sep 2013 14:46:48 -0700 (PDT) Subject: [erlang-questions] Re-using Mnesia disk files for a different node In-Reply-To: References: Message-ID: <337ba68c-7624-4fd0-9f73-ca975aba37a4@googlegroups.com> You do can convert live database, but there are some limitation, and you could lose some of your data. You will lose all data which is not yet transferred from log file. If you don't want to risk losing data, you may open DECISION_TAB.LOG and LATEST.LOG with disc_log module and convert them as well. They contain many different types of records though. This method is for disk-only-copy tables only. I think there could be problems with disk-copy tables, as ets dumps contain node information too. But maybe not. So You need to stop mnesia, and open file schema.DAT with dets:open_file(). It contains records of type {schema, SchemaName, ParamsList}. ParamsList is a proplist. Node name (node_name()) contains in {ram_copies,[node_name()]}, {disc_copies, [node_name()]}, {disc_only_copies, [node_name()]}, {version, {Version, {node_name(), TimeStamp}}} and {cookie, {TimeStamp, node_name()}}. {cookie, ..} and {version, ..} could contain an empty list as value as well. You need to iterate records in this table, change nodename() from old to current node name (or just change any to current node name if you have single-node database) and insert them back. Then close the dets file, delete two files - DECISION_TAB.LOG and LATEST.LOG, and start mnesia again. This could be done in auto mode on each restart, if you're brave one. Just stop mnesia, open scheme.DAT, check if all record have current node, and if some have different node name do the conversion. ???????????, 23 ???????? 2013 ?., 10:58:34 UTC+4 ???????????? Yash Ganthe ???????: > > Hi, > > > > On a node named node1@REDACTED I created an Mnesia schema using: > > mnesia:create_schema([node()]). > > and created a table using: > > mnesia:create_table(tab1, [{disc_copies, [node()]}, {attributes, > record_info(fields, tab1)}]). > > > > The mnesia directory was set to "mnesia_node1". The directory was created > and the DB files were created inside it. > > > > Once the node was stopped, the DB files still exist in the directory. I > hoped to re-use the DB as the mnesia DB for another node, node2. > > > > So I started the other node using: > > erl -name node2 -mnesia dir "mnesia_node1". > > > > On node2 when I start mnesia and do a tv:start(), I see the tab1 listed in > the schema. However, it shows no records. I was expecting Mnesia to use the > directory as the location of all DB files. > > > > Is there something I have missed? > > > > Thanks, > > Yash > > DISCLAIMER ========== This e-mail may contain privileged and confidential > information which is the property of Persistent Systems Ltd. It is intended > only for the use of the individual or entity to which it is addressed. If > you are not the intended recipient, you are not authorized to read, retain, > copy, print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Persistent Systems Ltd. does not accept any liability for > virus infected mails. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Fri Sep 27 18:12:34 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 27 Sep 2013 18:12:34 +0200 Subject: [erlang-questions] [ANN] iota - a tool for enforcing clean separation of responsibilities In-Reply-To: References: Message-ID: On Thu, Sep 19, 2013 at 7:56 PM, Joao Neves wrote: > Did you ever wish you could tell someone "just because I'm exporting > this it doesn't mean you should be using it" through your code? > > iota is a tool (and a library) that uses Erlang attributes to allow > you to specify application-level APIs implicitly by declaring what > exported functions in each module compose the overall API external > applications should use to interact with yours, with more checks > coming soon(TM). > > iota was mostly inspired by the Kappa tool used internally at Klarna > and mentioned by Cons T ?hs in his latest talk at the EUC in > Stockholm. Kappa is used to make some sense of the free-for-all > Erlang allows by having only a broad concept of exported and > non-exported functions and performs a number of tasks and techniques > that are in part an artifact of the development environment of > Klarna and are probably not appropriate for most other people. > > With iota, you have a more generic tool that allows you to keep some > sanity regarding the interactions between OTP applications while > allowing you to export the functions you need inside your own > application more freely. By using Erlang attributes we make these > checks entirely up to you, so if you feel you must ignore them or > don't want to use iota, no one is going to stop you! > > iota is hopefully also ready to be turned into rebar or EDTS plugins > for a more streamlined integration into your workflows, like as a > post-compilation hook. > > But enough with the chit-chat! You can get it (and more details on > how to use it) here: https://github.com/jpgneves/iota > > If you find it useful (or not), let me know! Any comments, questions > and pull requests are welcome! > > Many thanks to Samuel Rivas, H?kan Nilsson and Thomas J?rvstrand for > providing feedback and useful libraries to build this with, and to > Klarna Engineering in general for providing the necessary > inspiration and ideas for this initial release. Thanks for sharing, this looks useful. Have you considered adding support to erlc or Dialyzer similar to -callback? From peralta.alejandro@REDACTED Fri Sep 27 18:24:55 2013 From: peralta.alejandro@REDACTED (Ale) Date: Fri, 27 Sep 2013 13:24:55 -0300 Subject: [erlang-questions] [OT] Today's XKCD seems appropriate for the list. Message-ID: http://xkcd.com/1270/ Have a good weekend. :-) Regards. -- Ale. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tristan.sloughter@REDACTED Fri Sep 27 18:42:47 2013 From: tristan.sloughter@REDACTED (Tristan Sloughter) Date: Fri, 27 Sep 2013 09:42:47 -0700 Subject: [erlang-questions] Erlang/OTP installs broken Hipe app Message-ID: <1380300167.8589.27259237.4F1044A1@webmail.messagingengine.com> If you install R16*, I know this is the case for source with hipe disabled and the Erlang Solutions deb package which doesn't include hipe, it installs a hipe app: erlang/lib/hipe-3.10.2.1/ This "app" contains some beam files and hipe.app. BUT hipe.app lists modules that don't exist, like hipe.beam! If hipe is enabled when you compile/install the hipe app is not broken like this. Same with if you install the erlang-hipe package from Erlang Solutions along with the erlang package. Why is this the case? This breaks relx because it expects all apps it searches (even if it is not including that app in the release) to be a proper OTP app. I think it may be an issue with reltool as well, but "fixed" by "{app, hipe, [{incl_cond, exclude}]}," which is a horrific solution. -- Tristan Sloughter tristan.sloughter@REDACTED From mjtruog@REDACTED Fri Sep 27 19:01:46 2013 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 27 Sep 2013 10:01:46 -0700 Subject: [erlang-questions] Erlang/OTP installs broken Hipe app In-Reply-To: <1380300167.8589.27259237.4F1044A1@webmail.messagingengine.com> References: <1380300167.8589.27259237.4F1044A1@webmail.messagingengine.com> Message-ID: <5245B9FA.7060506@gmail.com> I know I have this problem when Erlang is compiled with: ./configure --enable-threads --enable-smp-support --enable-kernel-poll --disable-hipe I have a hipe Erlang application installed with hipe.beam mentioned within the hipe.app file, but the hipe.beam file does not exist. So, the first problem is that hipe is installed in a broken way when it is disabled. The second problem is that there is an attempt by reltool to include hipe when it is not a dependency for any Erlang applications involved in the release. An example reltool.config file that avoids this problem with "{app, hipe, [{incl_cond, exclude}]}," is https://github.com/okeuday/CloudI/blob/master/examples/hello_world2/reltool.config . So, you can explicitly prevent reltool from making the mistake of forcing hipe into your release for no reason, however, there should be no need to do so. I believe this has been a bug in the recent group of R16 releases, but I am not sure if it goes back further. On 09/27/2013 09:42 AM, Tristan Sloughter wrote: > If you install R16*, I know this is the case for source with hipe > disabled and the Erlang Solutions deb package which doesn't include > hipe, it installs a hipe app: erlang/lib/hipe-3.10.2.1/ > > This "app" contains some beam files and hipe.app. BUT hipe.app lists > modules that don't exist, like hipe.beam! > > If hipe is enabled when you compile/install the hipe app is not broken > like this. Same with if you install the erlang-hipe package from Erlang > Solutions along with the erlang package. > > Why is this the case? This breaks relx because it expects all apps it > searches (even if it is not including that app in the release) to be a > proper OTP app. I think it may be an issue with reltool as well, but > "fixed" by "{app, hipe, [{incl_cond, exclude}]}," which is a horrific > solution. > From maxim@REDACTED Fri Sep 27 19:06:50 2013 From: maxim@REDACTED (Maxim Sokhatsky) Date: Fri, 27 Sep 2013 20:06:50 +0300 Subject: [erlang-questions] Erlang/OTP installs broken Hipe app In-Reply-To: <1380300167.8589.27259237.4F1044A1@webmail.messagingengine.com> References: <1380300167.8589.27259237.4F1044A1@webmail.messagingengine.com> Message-ID: I'm confirming that --disable-hipe option produces and includes broken hipe application. It was so in R14, R15, R16. /maxim > From: tristan.sloughter@REDACTED > To: erlang-questions@REDACTED > Date: Fri, 27 Sep 2013 09:42:47 -0700 > Subject: [erlang-questions] Erlang/OTP installs broken Hipe app > > If you install R16*, I know this is the case for source with hipe > disabled and the Erlang Solutions deb package which doesn't include > hipe, it installs a hipe app: erlang/lib/hipe-3.10.2.1/ > > This "app" contains some beam files and hipe.app. BUT hipe.app lists > modules that don't exist, like hipe.beam! > > If hipe is enabled when you compile/install the hipe app is not broken > like this. Same with if you install the erlang-hipe package from Erlang > Solutions along with the erlang package. > > Why is this the case? This breaks relx because it expects all apps it > searches (even if it is not including that app in the release) to be a > proper OTP app. I think it may be an issue with reltool as well, but > "fixed" by "{app, hipe, [{incl_cond, exclude}]}," which is a horrific > solution. > > -- > Tristan Sloughter > tristan.sloughter@REDACTED > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ericbmerritt@REDACTED Fri Sep 27 19:17:32 2013 From: ericbmerritt@REDACTED (Eric Merritt) Date: Fri, 27 Sep 2013 10:17:32 -0700 Subject: [erlang-questions] Erlang/OTP installs broken Hipe app In-Reply-To: References: <1380300167.8589.27259237.4F1044A1@webmail.messagingengine.com> Message-ID: So we need to take a multi-pronged approach. I think 1) I guess we need to have a list of 'known broken apps' that we exclude by default, and warn about 2) Need to send a bug to the OTP team, or at least post something to the Erlang mailing list. They should not be breaking crap. Eric On Fri, Sep 27, 2013 at 10:06 AM, Maxim Sokhatsky wrote: > I'm confirming that --disable-hipe option produces and includes broken hipe application. > It was so in R14, R15, R16. > > /maxim > >> From: tristan.sloughter@REDACTED >> To: erlang-questions@REDACTED >> Date: Fri, 27 Sep 2013 09:42:47 -0700 >> Subject: [erlang-questions] Erlang/OTP installs broken Hipe app >> >> If you install R16*, I know this is the case for source with hipe >> disabled and the Erlang Solutions deb package which doesn't include >> hipe, it installs a hipe app: erlang/lib/hipe-3.10.2.1/ >> >> This "app" contains some beam files and hipe.app. BUT hipe.app lists >> modules that don't exist, like hipe.beam! >> >> If hipe is enabled when you compile/install the hipe app is not broken >> like this. Same with if you install the erlang-hipe package from Erlang >> Solutions along with the erlang package. >> >> Why is this the case? This breaks relx because it expects all apps it >> searches (even if it is not including that app in the release) to be a >> proper OTP app. I think it may be an issue with reltool as well, but >> "fixed" by "{app, hipe, [{incl_cond, exclude}]}," which is a horrific >> solution. >> >> -- >> Tristan Sloughter >> tristan.sloughter@REDACTED >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From sevenjp@REDACTED Fri Sep 27 21:05:17 2013 From: sevenjp@REDACTED (=?UTF-8?Q?Jo=C3=A3o_Neves?=) Date: Fri, 27 Sep 2013 21:05:17 +0200 Subject: [erlang-questions] [ANN] iota - a tool for enforcing clean separation of responsibilities In-Reply-To: References: Message-ID: I haven't given too much thought into it, but I did try to write iota in a way that it could be plugged in to as much as possible, so it's definitely not out of my mind. I might look into it soon. Thanks for your feedback and I hope it's useful! -- Jo?o Neves 2013/9/27 Tuncer Ayaz > On Thu, Sep 19, 2013 at 7:56 PM, Joao Neves wrote: > > Did you ever wish you could tell someone "just because I'm exporting > > this it doesn't mean you should be using it" through your code? > > > > iota is a tool (and a library) that uses Erlang attributes to allow > > you to specify application-level APIs implicitly by declaring what > > exported functions in each module compose the overall API external > > applications should use to interact with yours, with more checks > > coming soon(TM). > > > > iota was mostly inspired by the Kappa tool used internally at Klarna > > and mentioned by Cons T ?hs in his latest talk at the EUC in > > Stockholm. Kappa is used to make some sense of the free-for-all > > Erlang allows by having only a broad concept of exported and > > non-exported functions and performs a number of tasks and techniques > > that are in part an artifact of the development environment of > > Klarna and are probably not appropriate for most other people. > > > > With iota, you have a more generic tool that allows you to keep some > > sanity regarding the interactions between OTP applications while > > allowing you to export the functions you need inside your own > > application more freely. By using Erlang attributes we make these > > checks entirely up to you, so if you feel you must ignore them or > > don't want to use iota, no one is going to stop you! > > > > iota is hopefully also ready to be turned into rebar or EDTS plugins > > for a more streamlined integration into your workflows, like as a > > post-compilation hook. > > > > But enough with the chit-chat! You can get it (and more details on > > how to use it) here: https://github.com/jpgneves/iota > > > > If you find it useful (or not), let me know! Any comments, questions > > and pull requests are welcome! > > > > Many thanks to Samuel Rivas, H?kan Nilsson and Thomas J?rvstrand for > > providing feedback and useful libraries to build this with, and to > > Klarna Engineering in general for providing the necessary > > inspiration and ideas for this initial release. > > Thanks for sharing, this looks useful. > > Have you considered adding support to erlc or Dialyzer similar to > -callback? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Fri Sep 27 21:58:52 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 27 Sep 2013 21:58:52 +0200 Subject: [erlang-questions] [ANN] iota - a tool for enforcing clean separation of responsibilities In-Reply-To: References: Message-ID: On Fri, Sep 27, 2013 at 9:05 PM, Joao Neves wrote: > I haven't given too much thought into it, but I did try to write > iota in a way that it could be plugged in to as much as possible, so > it's definitely not out of my mind. > > I might look into it soon. Thanks for your feedback and I hope it's > useful! Alternatively, you could try to turn it into a predefined Xref query, but I don't know how feasible that is. > 2013/9/27 Tuncer Ayaz >> >> On Thu, Sep 19, 2013 at 7:56 PM, Joao Neves wrote: >> > Did you ever wish you could tell someone "just because I'm exporting >> > this it doesn't mean you should be using it" through your code? >> > >> > iota is a tool (and a library) that uses Erlang attributes to allow >> > you to specify application-level APIs implicitly by declaring what >> > exported functions in each module compose the overall API external >> > applications should use to interact with yours, with more checks >> > coming soon(TM). >> > >> > iota was mostly inspired by the Kappa tool used internally at Klarna >> > and mentioned by Cons T ?hs in his latest talk at the EUC in >> > Stockholm. Kappa is used to make some sense of the free-for-all >> > Erlang allows by having only a broad concept of exported and >> > non-exported functions and performs a number of tasks and techniques >> > that are in part an artifact of the development environment of >> > Klarna and are probably not appropriate for most other people. >> > >> > With iota, you have a more generic tool that allows you to keep some >> > sanity regarding the interactions between OTP applications while >> > allowing you to export the functions you need inside your own >> > application more freely. By using Erlang attributes we make these >> > checks entirely up to you, so if you feel you must ignore them or >> > don't want to use iota, no one is going to stop you! >> > >> > iota is hopefully also ready to be turned into rebar or EDTS plugins >> > for a more streamlined integration into your workflows, like as a >> > post-compilation hook. >> > >> > But enough with the chit-chat! You can get it (and more details on >> > how to use it) here: https://github.com/jpgneves/iota >> > >> > If you find it useful (or not), let me know! Any comments, questions >> > and pull requests are welcome! >> > >> > Many thanks to Samuel Rivas, H?kan Nilsson and Thomas J?rvstrand for >> > providing feedback and useful libraries to build this with, and to >> > Klarna Engineering in general for providing the necessary >> > inspiration and ideas for this initial release. >> >> Thanks for sharing, this looks useful. >> >> Have you considered adding support to erlc or Dialyzer similar to >> -callback? From mallen@REDACTED Fri Sep 27 23:28:04 2013 From: mallen@REDACTED (Mark Allen) Date: Fri, 27 Sep 2013 16:28:04 -0500 Subject: [erlang-questions] [ANN] Functional Programming User Group in Houston TX Message-ID: <079166BB-BC6E-44A5-AD01-4DC248C90CC8@alertlogic.com> Just a quick note to announce that we've started a functional programming user group in Houston TX. While our company primarily uses Erlang, we welcome users of any functional language. If you (or a friend) are in southeastern Texas, please consider joining our group. Our first meeting is in October. Details are available here: http://www.meetup.com/Houston-Functional-Programmers/ Thanks. Mark Allen Sr Software Developer Alert Logic E: mallen@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Sep 28 09:48:00 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 28 Sep 2013 09:48:00 +0200 Subject: [erlang-questions] any usb lib for Erlang Message-ID: Hi all, I am looking for a USB library for Erlang that would allow me to detect the insertion of usb devices like any wifi/bt dongles or hard disks. Is there ay more recent lib than this one ? https://github.com/tonyrog/libusb - benoit -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Sat Sep 28 12:53:09 2013 From: tony@REDACTED (Tony Rogvall) Date: Sat, 28 Sep 2013 12:53:09 +0200 Subject: [erlang-questions] any usb lib for Erlang In-Reply-To: References: Message-ID: <59FF895D-7219-47BF-A104-8BCF99B34154@rogvall.se> Hi! Sadly enough the libusb do not (did not at the time) support plug an play, this must be handled by some other type of application. I did solve it for serial usb devices by using fnotify (github.com/Feuerlabs/fnotify) code found in (github.com/tonyrog/uart/src/uart_devices.erl) Please hint to the list (or to me) if you find some nice plug and play api that is multi platform. (I have use for it in an other project github.com/tonyrog/hid ) Thanks /Tony On 28 sep 2013, at 09:48, Benoit Chesneau wrote: > Hi all, > > I am looking for a USB library for Erlang that would allow me to detect the insertion of usb devices like any wifi/bt dongles or hard disks. Is there ay more recent lib than this one ? > > https://github.com/tonyrog/libusb > > > - benoit > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Sat Sep 28 13:30:38 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Sat, 28 Sep 2013 13:30:38 +0200 Subject: [erlang-questions] any usb lib for Erlang In-Reply-To: <59FF895D-7219-47BF-A104-8BCF99B34154@rogvall.se> References: <59FF895D-7219-47BF-A104-8BCF99B34154@rogvall.se> Message-ID: On Sat, Sep 28, 2013 at 12:53 PM, Tony Rogvall wrote: > Hi! > > Sadly enough the libusb do not (did not at the time) support plug an play, > this must be handled by some > other type of application. I did solve it for serial usb devices by using > fnotify (github.com/Feuerlabs/fnotify) > code found in (github.com/tonyrog/uart/src/uart_devices.erl) > smart! Sounds like a good idea. > > Please hint to the list (or to me) if you find some nice plug and play api > that is multi platform. > (I have use for it in an other project github.com/tonyrog/hid ) > Yup still looking at it. there are probably some system libs that does it. Odd enough most low level libs in erlang are outdated and done for most 4-5 years ago.... > > Thanks > > /Tony > > On 28 sep 2013, at 09:48, Benoit Chesneau wrote: > > Hi all, > > I am looking for a USB library for Erlang that would allow me to detect > the insertion of usb devices like any wifi/bt dongles or hard disks. Is > there ay more recent lib than this one ? > > https://github.com/tonyrog/libusb > > > - benoit > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > "Installing applications can lead to corruption over time. Applications > gradually write over each other's libraries, partial upgrades occur, user > and system errors happen, and minute changes may be unnoticeable and > difficult to fix" > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Sun Sep 29 11:29:02 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Sun, 29 Sep 2013 10:29:02 +0100 Subject: [erlang-questions] Dangers of generating a large erlang module Message-ID: <5247F2DE.4080304@llaisdy.com> Dear All I am generating some erlang code from a csv file and using the function save_compile_and_load/2 below to load it into the running system. The input csv is essentially a series of rules, so the outpout erlang has roughly the same number of lines as the input has lines. save_compile_and_load(FilePath, Text) -> file:write_file(FilePath, Text), {ok, ModName} = compile:file(FilePath), % <-- {module, _M} = code:load_file(ModName), code:purge(ModName), ok. All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. Running on a Mac (oldish, probably with 32bit emulator), compiling (the line marked in the code below) causes an error and crashes the VM, with several repeats of this error message: beam.smp(1992,0xb022b000) malloc: *** mmap(size=913309696) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Running on linux with (presumably) a 64bit emulator, the compile works, but takes about 15 minutes. I'm not especially bothered that it takes a long time, but I am worried that this is something that could crash the whole system. I'm looking into splitting up the code into several smaller modules, but obviously I'd rather avoid crashing the VM. Questions: - is compiling large files on a 64bit emulator safe? What effects could there be on other processes? - can I wrap the compile:file call in something so that it will bail out if memory issues look likely (try/catch is no use)? With thanks and best wishes Ivan -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From n.oxyde@REDACTED Sun Sep 29 12:38:45 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sun, 29 Sep 2013 12:38:45 +0200 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: <5247F2DE.4080304@llaisdy.com> References: <5247F2DE.4080304@llaisdy.com> Message-ID: Hello Ivan, Out of curiosity, what does it look like? Pattern matching on literal values in Erlang is done with a binary search over the sorted list of patterns, I am not sure this would play well with your use case even if the compilation didn't bring the VM down. Regards, Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : > All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. -- Anthony Ramine From gustav.simonsson@REDACTED Sun Sep 29 12:49:35 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Sun, 29 Sep 2013 12:49:35 +0200 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: <5247F2DE.4080304@llaisdy.com> References: <5247F2DE.4080304@llaisdy.com> Message-ID: What kind of rules are contained in the CSV files? What information is defined in each rule? It's probably easier parsing the rules to some Erlang data type which can be given to a generic function. The internal format of the rules could be kept in e.g. ETS or in the state of a gen_server. Cheers, Gustav Simonsson On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin wrote: > Dear All > > I am generating some erlang code from a csv file and using the function > save_compile_and_load/2 below to load it into the running system. The > input csv is essentially a series of rules, so the outpout erlang has > roughly the same number of lines as the input has lines. > > save_compile_and_load(**FilePath, Text) -> > file:write_file(FilePath, Text), > {ok, ModName} = compile:file(FilePath), % <-- > {module, _M} = code:load_file(ModName), > code:purge(ModName), > ok. > > All goes well on small test files, but the files I want to use IRL are > relatively large --- around 120,000 lines. > > Running on a Mac (oldish, probably with 32bit emulator), compiling (the > line marked in the code below) causes an error and crashes the VM, with > several repeats of this error message: > > beam.smp(1992,0xb022b000) malloc: > *** mmap(size=913309696) failed (error code=12) > *** error: can't allocate region > *** set a breakpoint in malloc_error_break to debug > > Running on linux with (presumably) a 64bit emulator, the compile works, > but takes about 15 minutes. > > I'm not especially bothered that it takes a long time, but I am worried > that this is something that could crash the whole system. > > I'm looking into splitting up the code into several smaller modules, but > obviously I'd rather avoid crashing the VM. > > Questions: > - is compiling large files on a 64bit emulator safe? What effects could > there be on other processes? > - can I wrap the compile:file call in something so that it will bail out > if memory issues look likely (try/catch is no use)? > > With thanks and best wishes > > Ivan > > > -- > ==============================**============================== > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/**ivanuemlianin > > festina lente > ==============================**============================== > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Sun Sep 29 12:50:15 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Sun, 29 Sep 2013 11:50:15 +0100 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> Message-ID: <524805E7.8090001@llaisdy.com> Dear Anthony Thanks for your comment. Yes, that's exactly what the generated module is doing. The generated module has a single function with many clauses like this: f(<<"trigger", Rest/binary) -> ... This is why (as far as I can work out) the generated code has to be so big. I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. Best wishes Ivan On 29/09/2013 11:38, Anthony Ramine wrote: > Hello Ivan, > > Out of curiosity, what does it look like? > > Pattern matching on literal values in Erlang is done with a binary search over the sorted list of patterns, I am not sure this would play well with your use case even if the compilation didn't bring the VM down. > > Regards, > > Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : > >> All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From ivan@REDACTED Sun Sep 29 12:53:16 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Sun, 29 Sep 2013 11:53:16 +0100 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> Message-ID: <5248069C.1030402@llaisdy.com> Thanks for your comment. On 29/09/2013 11:49, Gustav Simonsson wrote: > What kind of rules are contained in the CSV files? What information is > defined in each rule? See reply to Anthony Ramine. > It's probably easier parsing the rules to some Erlang data type which > can be given to a generic function. > The internal format of the rules could be kept in e.g. ETS or in the > state of a gen_server. Agreed. However, these rules are going to be heavily consulted. Wouldn't gen_server and even ETS be more of a bottleneck danger? Best wishes Ivan > Cheers, > Gustav Simonsson > > > On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin > wrote: > > Dear All > > I am generating some erlang code from a csv file and using the > function save_compile_and_load/2 below to load it into the running > system. The input csv is essentially a series of rules, so the > outpout erlang has roughly the same number of lines as the input has > lines. > > save_compile_and_load(__FilePath, Text) -> > file:write_file(FilePath, Text), > {ok, ModName} = compile:file(FilePath), % <-- > {module, _M} = code:load_file(ModName), > code:purge(ModName), > ok. > > All goes well on small test files, but the files I want to use IRL > are relatively large --- around 120,000 lines. > > Running on a Mac (oldish, probably with 32bit emulator), compiling > (the line marked in the code below) causes an error and crashes the > VM, with several repeats of this error message: > > beam.smp(1992,0xb022b000) malloc: > *** mmap(size=913309696) failed (error code=12) > *** error: can't allocate region > *** set a breakpoint in malloc_error_break to debug > > Running on linux with (presumably) a 64bit emulator, the compile > works, but takes about 15 minutes. > > I'm not especially bothered that it takes a long time, but I am > worried that this is something that could crash the whole system. > > I'm looking into splitting up the code into several smaller modules, > but obviously I'd rather avoid crashing the VM. > > Questions: > - is compiling large files on a 64bit emulator safe? What effects > could there be on other processes? > - can I wrap the compile:file call in something so that it will bail > out if memory issues look likely (try/catch is no use)? > > With thanks and best wishes > > Ivan > > > -- > ==============================__============================== > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/__ivanuemlianin > > > festina lente > ==============================__============================== > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From sergej.jurecko@REDACTED Sun Sep 29 13:16:38 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sun, 29 Sep 2013 13:16:38 +0200 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: <5248069C.1030402@llaisdy.com> References: <5247F2DE.4080304@llaisdy.com> <5248069C.1030402@llaisdy.com> Message-ID: Try a public ets table with read_concurrency true. Sergej On Sep 29, 2013, at 12:53 PM, Ivan Uemlianin wrote: > Thanks for your comment. > > On 29/09/2013 11:49, Gustav Simonsson wrote: >> What kind of rules are contained in the CSV files? What information is >> defined in each rule? > > See reply to Anthony Ramine. > >> It's probably easier parsing the rules to some Erlang data type which >> can be given to a generic function. >> The internal format of the rules could be kept in e.g. ETS or in the >> state of a gen_server. > > Agreed. However, these rules are going to be heavily consulted. Wouldn't gen_server and even ETS be more of a bottleneck danger? > > Best wishes > > Ivan > > >> Cheers, >> Gustav Simonsson >> >> >> On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin > > wrote: >> >> Dear All >> >> I am generating some erlang code from a csv file and using the >> function save_compile_and_load/2 below to load it into the running >> system. The input csv is essentially a series of rules, so the >> outpout erlang has roughly the same number of lines as the input has >> lines. >> >> save_compile_and_load(__FilePath, Text) -> >> file:write_file(FilePath, Text), >> {ok, ModName} = compile:file(FilePath), % <-- >> {module, _M} = code:load_file(ModName), >> code:purge(ModName), >> ok. >> >> All goes well on small test files, but the files I want to use IRL >> are relatively large --- around 120,000 lines. >> >> Running on a Mac (oldish, probably with 32bit emulator), compiling >> (the line marked in the code below) causes an error and crashes the >> VM, with several repeats of this error message: >> >> beam.smp(1992,0xb022b000) malloc: >> *** mmap(size=913309696) failed (error code=12) >> *** error: can't allocate region >> *** set a breakpoint in malloc_error_break to debug >> >> Running on linux with (presumably) a 64bit emulator, the compile >> works, but takes about 15 minutes. >> >> I'm not especially bothered that it takes a long time, but I am >> worried that this is something that could crash the whole system. >> >> I'm looking into splitting up the code into several smaller modules, >> but obviously I'd rather avoid crashing the VM. >> >> Questions: >> - is compiling large files on a 64bit emulator safe? What effects >> could there be on other processes? >> - can I wrap the compile:file call in something so that it will bail >> out if memory issues look likely (try/catch is no use)? >> >> With thanks and best wishes >> >> Ivan >> >> >> -- >> ==============================__============================== >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> www.llaisdy.com >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/__ivanuemlianin >> >> >> festina lente >> ==============================__============================== >> _________________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/__listinfo/erlang-questions >> >> >> > > -- > ============================================================ > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/ivanuemlianin > > festina lente > ============================================================ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From valentin@REDACTED Sun Sep 29 13:19:21 2013 From: valentin@REDACTED (Valentin Micic) Date: Sun, 29 Sep 2013 13:19:21 +0200 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: <524805E7.8090001@llaisdy.com> References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: > > I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. > Your statement seems a bit counter-intuitive. I think you'd need to read data from somewhere in order to generate the code which incorporates such data. Then, you'd need to compile and load thus generated code. Finally, you'd need to execute this code in order to update the database. Needless to say, you'd need to repeat the whole cycle whenever data changes. What am I missing? Assuming that generated code executes faster (which we cannot know for certain), and thus not a bottleneck itself; wouldn't it be reasonable to predict that the whole process of code generation, etc. would be even bigger bottleneck? Kind regards V/ From ivan@REDACTED Sun Sep 29 13:23:44 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Sun, 29 Sep 2013 12:23:44 +0100 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: <52480DC0.9070908@llaisdy.com> The unstated assumption was that rule changes happen at predictable times and can be scheduled conveniently. > ... > Finally, you'd need to execute this code in order to update > the database. So far no database is involved, everything is in the generated code. Best wishes Ivan On 29/09/2013 12:19, Valentin Micic wrote: >> >> I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. >> > > Your statement seems a bit counter-intuitive. > > I think you'd need to read data from somewhere in order to generate the code which incorporates such data. > Then, you'd need to compile and load thus generated code. > Finally, you'd need to execute this code in order to update the database. > > Needless to say, you'd need to repeat the whole cycle whenever data changes. > > What am I missing? > Assuming that generated code executes faster (which we cannot know for certain), and thus not a bottleneck itself; wouldn't it be reasonable to predict that the whole process of code generation, etc. would be even bigger bottleneck? > > Kind regards > > V/ > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From ivan@REDACTED Sun Sep 29 13:27:02 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Sun, 29 Sep 2013 12:27:02 +0100 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <5248069C.1030402@llaisdy.com> Message-ID: <52480E86.7080001@llaisdy.com> Thanks, that sounds worth a go. Generating the code into smaller modules was starting to get hacky :P Best wishes Ivan On 29/09/2013 12:16, Sergej Jurecko wrote: > Try a public ets table with read_concurrency true. > > > Sergej > > On Sep 29, 2013, at 12:53 PM, Ivan Uemlianin wrote: > >> Thanks for your comment. >> >> On 29/09/2013 11:49, Gustav Simonsson wrote: >>> What kind of rules are contained in the CSV files? What information is >>> defined in each rule? >> >> See reply to Anthony Ramine. >> >>> It's probably easier parsing the rules to some Erlang data type which >>> can be given to a generic function. >>> The internal format of the rules could be kept in e.g. ETS or in the >>> state of a gen_server. >> >> Agreed. However, these rules are going to be heavily consulted. Wouldn't gen_server and even ETS be more of a bottleneck danger? >> >> Best wishes >> >> Ivan >> >> >>> Cheers, >>> Gustav Simonsson >>> >>> >>> On Sun, Sep 29, 2013 at 11:29 AM, Ivan Uemlianin >> > wrote: >>> >>> Dear All >>> >>> I am generating some erlang code from a csv file and using the >>> function save_compile_and_load/2 below to load it into the running >>> system. The input csv is essentially a series of rules, so the >>> outpout erlang has roughly the same number of lines as the input has >>> lines. >>> >>> save_compile_and_load(__FilePath, Text) -> >>> file:write_file(FilePath, Text), >>> {ok, ModName} = compile:file(FilePath), % <-- >>> {module, _M} = code:load_file(ModName), >>> code:purge(ModName), >>> ok. >>> >>> All goes well on small test files, but the files I want to use IRL >>> are relatively large --- around 120,000 lines. >>> >>> Running on a Mac (oldish, probably with 32bit emulator), compiling >>> (the line marked in the code below) causes an error and crashes the >>> VM, with several repeats of this error message: >>> >>> beam.smp(1992,0xb022b000) malloc: >>> *** mmap(size=913309696) failed (error code=12) >>> *** error: can't allocate region >>> *** set a breakpoint in malloc_error_break to debug >>> >>> Running on linux with (presumably) a 64bit emulator, the compile >>> works, but takes about 15 minutes. >>> >>> I'm not especially bothered that it takes a long time, but I am >>> worried that this is something that could crash the whole system. >>> >>> I'm looking into splitting up the code into several smaller modules, >>> but obviously I'd rather avoid crashing the VM. >>> >>> Questions: >>> - is compiling large files on a 64bit emulator safe? What effects >>> could there be on other processes? >>> - can I wrap the compile:file call in something so that it will bail >>> out if memory issues look likely (try/catch is no use)? >>> >>> With thanks and best wishes >>> >>> Ivan >>> >>> >>> -- >>> ==============================__============================== >>> Ivan A. Uemlianin PhD >>> Llaisdy >>> Speech Technology Research and Development >>> >>> ivan@REDACTED >>> www.llaisdy.com >>> llaisdy.wordpress.com >>> github.com/llaisdy >>> www.linkedin.com/in/__ivanuemlianin >>> >>> >>> festina lente >>> ==============================__============================== >>> _________________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/__listinfo/erlang-questions >>> >>> >>> >> >> -- >> ============================================================ >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> www.llaisdy.com >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/ivanuemlianin >> >> festina lente >> ============================================================ >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================ From eriksoe@REDACTED Sun Sep 29 18:48:49 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sun, 29 Sep 2013 18:48:49 +0200 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: <524805E7.8090001@llaisdy.com> References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: A thing which I discovered recently (in connection with mochiglobal) is that compiling code containing large binaries, or large amounts of binaries, is quite memory-intensive. As I recall it, the numbers were ~64 bytes of RAM per byte in a binary metal; twice as much if on a 64 bit emulator. Which means that if you want to compile modules containing (in sum) multimegabyte binaries, doing so from Erlang source or from full Erlang AST is a no-go. Iirc, it is feasible if starting from Core Erlang. /Erik Den 29/09/2013 12.50 skrev "Ivan Uemlianin" : > Dear Anthony > > Thanks for your comment. > > Yes, that's exactly what the generated module is doing. The generated > module has a single function with many clauses like this: > > f(<<"trigger", Rest/binary) -> ... > > This is why (as far as I can work out) the generated code has to be so big. > > I prefer the idea of generating and loading code to, say, updating a > database table, because it seems faster and less likely to lead to > bottlenecks. > > Best wishes > > Ivan > > > On 29/09/2013 11:38, Anthony Ramine wrote: > >> Hello Ivan, >> >> Out of curiosity, what does it look like? >> >> Pattern matching on literal values in Erlang is done with a binary search >> over the sorted list of patterns, I am not sure this would play well with >> your use case even if the compilation didn't bring the VM down. >> >> Regards, >> >> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >> >> All goes well on small test files, but the files I want to use IRL are >>> relatively large --- around 120,000 lines. >>> >> >> > -- > ==============================**============================== > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/**ivanuemlianin > > festina lente > ==============================**============================== > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Sun Sep 29 19:20:03 2013 From: ivan@REDACTED (Ivan uemlianin) Date: Sun, 29 Sep 2013 18:20:03 +0100 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: Thanks! I think I'll try and head in that direction. I've had a few goes at other methods (db lookup etc) and they're much slower than this "dynamic hardcoding"). I'll sniff around for Core Erlang tutorials. Best wishes Ivan -- festina lente On 29 Sep 2013, at 17:48, Erik S?e S?rensen wrote: > A thing which I discovered recently (in connection with mochiglobal) is that compiling code containing large binaries, or large amounts of binaries, is quite memory-intensive. As I recall it, the numbers were ~64 bytes of RAM per byte in a binary metal; twice as much if on a 64 bit emulator. > Which means that if you want to compile modules containing (in sum) multimegabyte binaries, doing so from Erlang source or from full Erlang AST is a no-go. Iirc, it is feasible if starting from Core Erlang. > /Erik > > Den 29/09/2013 12.50 skrev "Ivan Uemlianin" : >> Dear Anthony >> >> Thanks for your comment. >> >> Yes, that's exactly what the generated module is doing. The generated module has a single function with many clauses like this: >> >> f(<<"trigger", Rest/binary) -> ... >> >> This is why (as far as I can work out) the generated code has to be so big. >> >> I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. >> >> Best wishes >> >> Ivan >> >> >> On 29/09/2013 11:38, Anthony Ramine wrote: >>> Hello Ivan, >>> >>> Out of curiosity, what does it look like? >>> >>> Pattern matching on literal values in Erlang is done with a binary search over the sorted list of patterns, I am not sure this would play well with your use case even if the compilation didn't bring the VM down. >>> >>> Regards, >>> >>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>> >>>> All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. >> >> -- >> ============================================================ >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> www.llaisdy.com >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/ivanuemlianin >> >> festina lente >> ============================================================ >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sun Sep 29 19:36:02 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sun, 29 Sep 2013 19:36:02 +0200 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: Core Erlang is an intermediate representation in the Erlang compiler - but also (afaik) a fairly well-defined/public one and one that is stable. I don't think you'll find much in the vein of tutorials. Try getting erlc to output the intermediate format, though, for a small program similar to what you'll be using it for. Den 29/09/2013 19.20 skrev "Ivan uemlianin" : > Thanks! I think I'll try and head in that direction. I've had a few goes > at other methods (db lookup etc) and they're much slower than this "dynamic > hardcoding"). I'll sniff around for Core Erlang tutorials. > > Best wishes > > Ivan > > > -- > festina lente > > > On 29 Sep 2013, at 17:48, Erik S?e S?rensen wrote: > > A thing which I discovered recently (in connection with mochiglobal) is > that compiling code containing large binaries, or large amounts of > binaries, is quite memory-intensive. As I recall it, the numbers were ~64 > bytes of RAM per byte in a binary metal; twice as much if on a 64 bit > emulator. > Which means that if you want to compile modules containing (in sum) > multimegabyte binaries, doing so from Erlang source or from full Erlang AST > is a no-go. Iirc, it is feasible if starting from Core Erlang. > /Erik > Den 29/09/2013 12.50 skrev "Ivan Uemlianin" : > >> Dear Anthony >> >> Thanks for your comment. >> >> Yes, that's exactly what the generated module is doing. The generated >> module has a single function with many clauses like this: >> >> f(<<"trigger", Rest/binary) -> ... >> >> This is why (as far as I can work out) the generated code has to be so >> big. >> >> I prefer the idea of generating and loading code to, say, updating a >> database table, because it seems faster and less likely to lead to >> bottlenecks. >> >> Best wishes >> >> Ivan >> >> >> On 29/09/2013 11:38, Anthony Ramine wrote: >> >>> Hello Ivan, >>> >>> Out of curiosity, what does it look like? >>> >>> Pattern matching on literal values in Erlang is done with a binary >>> search over the sorted list of patterns, I am not sure this would play well >>> with your use case even if the compilation didn't bring the VM down. >>> >>> Regards, >>> >>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>> >>> All goes well on small test files, but the files I want to use IRL are >>>> relatively large --- around 120,000 lines. >>>> >>> >>> >> -- >> ==============================**============================== >> Ivan A. Uemlianin PhD >> Llaisdy >> Speech Technology Research and Development >> >> ivan@REDACTED >> www.llaisdy.com >> llaisdy.wordpress.com >> github.com/llaisdy >> www.linkedin.com/in/**ivanuemlianin >> >> festina lente >> ==============================**============================== >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan@REDACTED Sun Sep 29 20:24:45 2013 From: ivan@REDACTED (Ivan uemlianin) Date: Sun, 29 Sep 2013 19:24:45 +0100 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: That's what I've just done :D Core Erlang looks very verbose but quite regular & probably not difficult to generate. My questions now are: - are there any libraries "out there" for generating Core Erlang, or do we all roll our own? - how would one use compile:file or compile:forms with core erlang? I haven't been able to find any documentation (haven't read Richard Carlsson's Introduction paper yet). Many thanks Ivan -- festina lente On 29 Sep 2013, at 18:36, Erik S?e S?rensen wrote: > Core Erlang is an intermediate representation in the Erlang compiler - but also (afaik) a fairly well-defined/public one and one that is stable. > I don't think you'll find much in the vein of tutorials. Try getting erlc to output the intermediate format, though, for a small program similar to what you'll be using it for. > > Den 29/09/2013 19.20 skrev "Ivan uemlianin" : >> Thanks! I think I'll try and head in that direction. I've had a few goes at other methods (db lookup etc) and they're much slower than this "dynamic hardcoding"). I'll sniff around for Core Erlang tutorials. >> >> Best wishes >> >> Ivan >> >> >> -- >> festina lente >> >> >> On 29 Sep 2013, at 17:48, Erik S?e S?rensen wrote: >> >>> A thing which I discovered recently (in connection with mochiglobal) is that compiling code containing large binaries, or large amounts of binaries, is quite memory-intensive. As I recall it, the numbers were ~64 bytes of RAM per byte in a binary metal; twice as much if on a 64 bit emulator. >>> Which means that if you want to compile modules containing (in sum) multimegabyte binaries, doing so from Erlang source or from full Erlang AST is a no-go. Iirc, it is feasible if starting from Core Erlang. >>> /Erik >>> >>> Den 29/09/2013 12.50 skrev "Ivan Uemlianin" : >>>> Dear Anthony >>>> >>>> Thanks for your comment. >>>> >>>> Yes, that's exactly what the generated module is doing. The generated module has a single function with many clauses like this: >>>> >>>> f(<<"trigger", Rest/binary) -> ... >>>> >>>> This is why (as far as I can work out) the generated code has to be so big. >>>> >>>> I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. >>>> >>>> Best wishes >>>> >>>> Ivan >>>> >>>> >>>> On 29/09/2013 11:38, Anthony Ramine wrote: >>>>> Hello Ivan, >>>>> >>>>> Out of curiosity, what does it look like? >>>>> >>>>> Pattern matching on literal values in Erlang is done with a binary search over the sorted list of patterns, I am not sure this would play well with your use case even if the compilation didn't bring the VM down. >>>>> >>>>> Regards, >>>>> >>>>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>>>> >>>>>> All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. >>>> >>>> -- >>>> ============================================================ >>>> Ivan A. Uemlianin PhD >>>> Llaisdy >>>> Speech Technology Research and Development >>>> >>>> ivan@REDACTED >>>> www.llaisdy.com >>>> llaisdy.wordpress.com >>>> github.com/llaisdy >>>> www.linkedin.com/in/ivanuemlianin >>>> >>>> festina lente >>>> ============================================================ >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Sun Sep 29 20:50:22 2013 From: mjtruog@REDACTED (Michael Truog) Date: Sun, 29 Sep 2013 11:50:22 -0700 Subject: [erlang-questions] Dangers of generating a large erlang module In-Reply-To: <524805E7.8090001@llaisdy.com> References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: <5248766E.4030205@gmail.com> Depending on what all the pattern matching looked like, you could use strings (lists of integers) instead of binaries, with a trie (https://github.com/okeuday/trie) using patterns. So the function you mentioned would look like: trie:store("trigger*", ..., Trie) Later, you would use trie:find_match/2 to get a match, and trie:pattern_parse/2 to get any pattern matched values (in this case, whatever comes after "trigger"). You could use ct_expand from https://github.com/uwiger/parse_trans to expand it at compile time, but it may still consume too much memory. Just an idea, it still seems nice being able to have a static lookup in a module that goes through a regular compilation. On 09/29/2013 03:50 AM, Ivan Uemlianin wrote: > Dear Anthony > > Thanks for your comment. > > Yes, that's exactly what the generated module is doing. The generated module has a single function with many clauses like this: > > f(<<"trigger", Rest/binary) -> ... > > This is why (as far as I can work out) the generated code has to be so big. > > I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. > > Best wishes > > Ivan > > > On 29/09/2013 11:38, Anthony Ramine wrote: >> Hello Ivan, >> >> Out of curiosity, what does it look like? >> >> Pattern matching on literal values in Erlang is done with a binary search over the sorted list of patterns, I am not sure this would play well with your use case even if the compilation didn't bring the VM down. >> >> Regards, >> >> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >> >>> All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. >> > From eriksoe@REDACTED Sun Sep 29 20:56:16 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sun, 29 Sep 2013 20:56:16 +0200 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: I think the direct way would be to generate the internal form of CE, then feed it to the compiler; don't know exactly how easy they've made it :-) Although I hope it's not too complicated; if you/we figure it out, I might update mochiglobal to use it, given time. Den 29/09/2013 20.24 skrev "Ivan uemlianin" : > That's what I've just done :D Core Erlang looks very verbose but quite > regular & probably not difficult to generate. > > My questions now are: > - are there any libraries "out there" for generating Core Erlang, or do we > all roll our own? > - how would one use compile:file or compile:forms with core erlang? I > haven't been able to find any documentation (haven't read Richard > Carlsson's Introduction paper yet). > > Many thanks > > Ivan > > -- > festina lente > > > On 29 Sep 2013, at 18:36, Erik S?e S?rensen wrote: > > Core Erlang is an intermediate representation in the Erlang compiler - > but also (afaik) a fairly well-defined/public one and one that is stable. > I don't think you'll find much in the vein of tutorials. Try getting erlc > to output the intermediate format, though, for a small program similar to > what you'll be using it for. > Den 29/09/2013 19.20 skrev "Ivan uemlianin" : > >> Thanks! I think I'll try and head in that direction. I've had a few goes >> at other methods (db lookup etc) and they're much slower than this "dynamic >> hardcoding"). I'll sniff around for Core Erlang tutorials. >> >> Best wishes >> >> Ivan >> >> >> -- >> festina lente >> >> >> On 29 Sep 2013, at 17:48, Erik S?e S?rensen wrote: >> >> A thing which I discovered recently (in connection with mochiglobal) is >> that compiling code containing large binaries, or large amounts of >> binaries, is quite memory-intensive. As I recall it, the numbers were ~64 >> bytes of RAM per byte in a binary metal; twice as much if on a 64 bit >> emulator. >> Which means that if you want to compile modules containing (in sum) >> multimegabyte binaries, doing so from Erlang source or from full Erlang AST >> is a no-go. Iirc, it is feasible if starting from Core Erlang. >> /Erik >> Den 29/09/2013 12.50 skrev "Ivan Uemlianin" : >> >>> Dear Anthony >>> >>> Thanks for your comment. >>> >>> Yes, that's exactly what the generated module is doing. The generated >>> module has a single function with many clauses like this: >>> >>> f(<<"trigger", Rest/binary) -> ... >>> >>> This is why (as far as I can work out) the generated code has to be so >>> big. >>> >>> I prefer the idea of generating and loading code to, say, updating a >>> database table, because it seems faster and less likely to lead to >>> bottlenecks. >>> >>> Best wishes >>> >>> Ivan >>> >>> >>> On 29/09/2013 11:38, Anthony Ramine wrote: >>> >>>> Hello Ivan, >>>> >>>> Out of curiosity, what does it look like? >>>> >>>> Pattern matching on literal values in Erlang is done with a binary >>>> search over the sorted list of patterns, I am not sure this would play well >>>> with your use case even if the compilation didn't bring the VM down. >>>> >>>> Regards, >>>> >>>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>>> >>>> All goes well on small test files, but the files I want to use IRL are >>>>> relatively large --- around 120,000 lines. >>>>> >>>> >>>> >>> -- >>> ==============================**============================== >>> Ivan A. Uemlianin PhD >>> Llaisdy >>> Speech Technology Research and Development >>> >>> ivan@REDACTED >>> www.llaisdy.com >>> llaisdy.wordpress.com >>> github.com/llaisdy >>> www.linkedin.com/in/**ivanuemlianin >>> >>> festina lente >>> ==============================**============================== >>> ______________________________**_________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/**listinfo/erlang-questions >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Sun Sep 29 20:59:22 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Sun, 29 Sep 2013 20:59:22 +0200 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: <5248788A.5030707@ninenines.eu> On 09/29/2013 08:24 PM, Ivan uemlianin wrote: > That's what I've just done :D Core Erlang looks very verbose but quite > regular & probably not difficult to generate. > > My questions now are: > - are there any libraries "out there" for generating Core Erlang, or do > we all roll our own? Look at the cerl module. It's just a matter of generating the proper structure using those functions. > - how would one use compile:file or compile:forms with core erlang? I > haven't been able to find any documentation (haven't read Richard > Carlsson's Introduction paper yet). Do read it. But to answer that question, compile:forms with from_core option. > Many thanks > > Ivan > > -- > festina lente > > > On 29 Sep 2013, at 18:36, Erik S?e S?rensen > wrote: > >> Core Erlang is an intermediate representation in the Erlang compiler >> - but also (afaik) a fairly well-defined/public one and one that is >> stable. >> I don't think you'll find much in the vein of tutorials. Try getting >> erlc to output the intermediate format, though, for a small program >> similar to what you'll be using it for. >> >> Den 29/09/2013 19.20 skrev "Ivan uemlianin" > >: >> >> Thanks! I think I'll try and head in that direction. I've had a >> few goes at other methods (db lookup etc) and they're much slower >> than this "dynamic hardcoding"). I'll sniff around for Core Erlang >> tutorials. >> >> Best wishes >> >> Ivan >> >> >> -- >> festina lente >> >> >> On 29 Sep 2013, at 17:48, Erik S?e S?rensen > > wrote: >> >>> A thing which I discovered recently (in connection with >>> mochiglobal) is that compiling code containing large binaries, or >>> large amounts of binaries, is quite memory-intensive. As I >>> recall it, the numbers were ~64 bytes of RAM per byte in a binary >>> metal; twice as much if on a 64 bit emulator. >>> Which means that if you want to compile modules containing (in >>> sum) multimegabyte binaries, doing so from Erlang source or from >>> full Erlang AST is a no-go. Iirc, it is feasible if starting >>> from Core Erlang. >>> /Erik >>> >>> Den 29/09/2013 12.50 skrev "Ivan Uemlianin" >> >: >>> >>> Dear Anthony >>> >>> Thanks for your comment. >>> >>> Yes, that's exactly what the generated module is doing. The >>> generated module has a single function with many clauses like >>> this: >>> >>> f(<<"trigger", Rest/binary) -> ... >>> >>> This is why (as far as I can work out) the generated code has >>> to be so big. >>> >>> I prefer the idea of generating and loading code to, say, >>> updating a database table, because it seems faster and less >>> likely to lead to bottlenecks. >>> >>> Best wishes >>> >>> Ivan >>> >>> >>> On 29/09/2013 11:38, Anthony Ramine wrote: >>> >>> Hello Ivan, >>> >>> Out of curiosity, what does it look like? >>> >>> Pattern matching on literal values in Erlang is done with >>> a binary search over the sorted list of patterns, I am >>> not sure this would play well with your use case even if >>> the compilation didn't bring the VM down. >>> >>> Regards, >>> >>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>> >>> All goes well on small test files, but the files I >>> want to use IRL are relatively large --- around >>> 120,000 lines. >>> >>> >>> >>> -- >>> ==============================__============================== >>> Ivan A. Uemlianin PhD >>> Llaisdy >>> Speech Technology Research and Development >>> >>> ivan@REDACTED >>> www.llaisdy.com >>> llaisdy.wordpress.com >>> github.com/llaisdy >>> www.linkedin.com/in/__ivanuemlianin >>> >>> >>> festina lente >>> ==============================__============================== >>> _________________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/__listinfo/erlang-questions >>> >>> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From amindfv@REDACTED Sun Sep 29 23:01:35 2013 From: amindfv@REDACTED (Tom Murphy) Date: Sun, 29 Sep 2013 14:01:35 -0700 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: <5248788A.5030707@ninenines.eu> References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> <5248788A.5030707@ninenines.eu> Message-ID: There's also just: erlc +to_core foo.erl erlc +from_core foo.core Tom Murphy On Sun, Sep 29, 2013 at 11:59 AM, Lo?c Hoguin wrote: > On 09/29/2013 08:24 PM, Ivan uemlianin wrote: > >> That's what I've just done :D Core Erlang looks very verbose but quite >> regular & probably not difficult to generate. >> >> My questions now are: >> - are there any libraries "out there" for generating Core Erlang, or do >> we all roll our own? >> > > Look at the cerl module. It's just a matter of generating the proper > structure using those functions. > > > - how would one use compile:file or compile:forms with core erlang? I >> haven't been able to find any documentation (haven't read Richard >> Carlsson's Introduction paper yet). >> > > Do read it. But to answer that question, compile:forms with from_core > option. > > Many thanks >> >> Ivan >> >> -- >> festina lente >> >> >> On 29 Sep 2013, at 18:36, Erik S?e S?rensen > > wrote: >> >> Core Erlang is an intermediate representation in the Erlang compiler >>> - but also (afaik) a fairly well-defined/public one and one that is >>> stable. >>> I don't think you'll find much in the vein of tutorials. Try getting >>> erlc to output the intermediate format, though, for a small program >>> similar to what you'll be using it for. >>> >>> Den 29/09/2013 19.20 skrev "Ivan uemlianin" >> >: >>> >>> >>> Thanks! I think I'll try and head in that direction. I've had a >>> few goes at other methods (db lookup etc) and they're much slower >>> than this "dynamic hardcoding"). I'll sniff around for Core Erlang >>> tutorials. >>> >>> Best wishes >>> >>> Ivan >>> >>> >>> -- >>> festina lente >>> >>> >>> On 29 Sep 2013, at 17:48, Erik S?e S?rensen >> > wrote: >>> >>> A thing which I discovered recently (in connection with >>>> mochiglobal) is that compiling code containing large binaries, or >>>> large amounts of binaries, is quite memory-intensive. As I >>>> recall it, the numbers were ~64 bytes of RAM per byte in a binary >>>> metal; twice as much if on a 64 bit emulator. >>>> Which means that if you want to compile modules containing (in >>>> sum) multimegabyte binaries, doing so from Erlang source or from >>>> full Erlang AST is a no-go. Iirc, it is feasible if starting >>>> from Core Erlang. >>>> /Erik >>>> >>>> Den 29/09/2013 12.50 skrev "Ivan Uemlianin" >>> >: >>>> >>>> >>>> Dear Anthony >>>> >>>> Thanks for your comment. >>>> >>>> Yes, that's exactly what the generated module is doing. The >>>> generated module has a single function with many clauses like >>>> this: >>>> >>>> f(<<"trigger", Rest/binary) -> ... >>>> >>>> This is why (as far as I can work out) the generated code has >>>> to be so big. >>>> >>>> I prefer the idea of generating and loading code to, say, >>>> updating a database table, because it seems faster and less >>>> likely to lead to bottlenecks. >>>> >>>> Best wishes >>>> >>>> Ivan >>>> >>>> >>>> On 29/09/2013 11:38, Anthony Ramine wrote: >>>> >>>> Hello Ivan, >>>> >>>> Out of curiosity, what does it look like? >>>> >>>> Pattern matching on literal values in Erlang is done with >>>> a binary search over the sorted list of patterns, I am >>>> not sure this would play well with your use case even if >>>> the compilation didn't bring the VM down. >>>> >>>> Regards, >>>> >>>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>>> >>>> All goes well on small test files, but the files I >>>> want to use IRL are relatively large --- around >>>> 120,000 lines. >>>> >>>> >>>> >>>> -- >>>> ==============================**__============================* >>>> *== >>>> >>>> Ivan A. Uemlianin PhD >>>> Llaisdy >>>> Speech Technology Research and Development >>>> >>>> ivan@REDACTED >>>> www.llaisdy.com >>>> llaisdy.wordpress.com >>>> github.com/llaisdy >>>> www.linkedin.com/in/__**ivanuemlianin >>>> >>>> > >>>> >>>> festina lente >>>> ==============================**__============================* >>>> *== >>>> ______________________________**___________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>> erlang.org > >>>> http://erlang.org/mailman/__**listinfo/erlang-questions >>>> >>>> > >>>> >>>> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> >> > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Mon Sep 30 07:31:56 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Mon, 30 Sep 2013 07:31:56 +0200 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> <5248788A.5030707@ninenines.eu> Message-ID: Yes, indeed; considering that the goal is reduction of memory use, however, what we want is to avoid intermediate representations which explode in size on binaries. I don't know anything about the from_core parser, but the string-as-list representation of a binary literal is in itself a factor 32/64 blowup. :-/ (Btw: I was wrong about introductions to Core Erlang; threre is one. Writing on the phone has its fact-checking disadvantages...) Den 29/09/2013 23.01 skrev "Tom Murphy" : > There's also just: > > erlc +to_core foo.erl > erlc +from_core foo.core > > Tom Murphy > > > On Sun, Sep 29, 2013 at 11:59 AM, Lo?c Hoguin wrote: > >> On 09/29/2013 08:24 PM, Ivan uemlianin wrote: >> >>> That's what I've just done :D Core Erlang looks very verbose but quite >>> regular & probably not difficult to generate. >>> >>> My questions now are: >>> - are there any libraries "out there" for generating Core Erlang, or do >>> we all roll our own? >>> >> >> Look at the cerl module. It's just a matter of generating the proper >> structure using those functions. >> >> >> - how would one use compile:file or compile:forms with core erlang? I >>> haven't been able to find any documentation (haven't read Richard >>> Carlsson's Introduction paper yet). >>> >> >> Do read it. But to answer that question, compile:forms with from_core >> option. >> >> Many thanks >>> >>> Ivan >>> >>> -- >>> festina lente >>> >>> >>> On 29 Sep 2013, at 18:36, Erik S?e S?rensen >> > wrote: >>> >>> Core Erlang is an intermediate representation in the Erlang compiler >>>> - but also (afaik) a fairly well-defined/public one and one that is >>>> stable. >>>> I don't think you'll find much in the vein of tutorials. Try getting >>>> erlc to output the intermediate format, though, for a small program >>>> similar to what you'll be using it for. >>>> >>>> Den 29/09/2013 19.20 skrev "Ivan uemlianin" >>> >: >>>> >>>> >>>> Thanks! I think I'll try and head in that direction. I've had a >>>> few goes at other methods (db lookup etc) and they're much slower >>>> than this "dynamic hardcoding"). I'll sniff around for Core Erlang >>>> tutorials. >>>> >>>> Best wishes >>>> >>>> Ivan >>>> >>>> >>>> -- >>>> festina lente >>>> >>>> >>>> On 29 Sep 2013, at 17:48, Erik S?e S?rensen >>> > wrote: >>>> >>>> A thing which I discovered recently (in connection with >>>>> mochiglobal) is that compiling code containing large binaries, or >>>>> large amounts of binaries, is quite memory-intensive. As I >>>>> recall it, the numbers were ~64 bytes of RAM per byte in a binary >>>>> metal; twice as much if on a 64 bit emulator. >>>>> Which means that if you want to compile modules containing (in >>>>> sum) multimegabyte binaries, doing so from Erlang source or from >>>>> full Erlang AST is a no-go. Iirc, it is feasible if starting >>>>> from Core Erlang. >>>>> /Erik >>>>> >>>>> Den 29/09/2013 12.50 skrev "Ivan Uemlianin" >>>> >: >>>>> >>>>> >>>>> Dear Anthony >>>>> >>>>> Thanks for your comment. >>>>> >>>>> Yes, that's exactly what the generated module is doing. The >>>>> generated module has a single function with many clauses like >>>>> this: >>>>> >>>>> f(<<"trigger", Rest/binary) -> ... >>>>> >>>>> This is why (as far as I can work out) the generated code has >>>>> to be so big. >>>>> >>>>> I prefer the idea of generating and loading code to, say, >>>>> updating a database table, because it seems faster and less >>>>> likely to lead to bottlenecks. >>>>> >>>>> Best wishes >>>>> >>>>> Ivan >>>>> >>>>> >>>>> On 29/09/2013 11:38, Anthony Ramine wrote: >>>>> >>>>> Hello Ivan, >>>>> >>>>> Out of curiosity, what does it look like? >>>>> >>>>> Pattern matching on literal values in Erlang is done with >>>>> a binary search over the sorted list of patterns, I am >>>>> not sure this would play well with your use case even if >>>>> the compilation didn't bring the VM down. >>>>> >>>>> Regards, >>>>> >>>>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>>>> >>>>> All goes well on small test files, but the files I >>>>> want to use IRL are relatively large --- around >>>>> 120,000 lines. >>>>> >>>>> >>>>> >>>>> -- >>>>> ==============================**__============================ >>>>> **== >>>>> >>>>> Ivan A. Uemlianin PhD >>>>> Llaisdy >>>>> Speech Technology Research and Development >>>>> >>>>> ivan@REDACTED >>>>> www.llaisdy.com >>>>> llaisdy.wordpress.com >>>>> github.com/llaisdy >>>>> www.linkedin.com/in/__**ivanuemlianin >>>>> >>>>> > >>>>> >>>>> festina lente >>>>> ==============================**__============================ >>>>> **== >>>>> ______________________________**___________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>> erlang.org > >>>>> http://erlang.org/mailman/__**listinfo/erlang-questions >>>>> >>>>> > >>>>> >>>>> >>> >>> ______________________________**_________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/**listinfo/erlang-questions >>> >>> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Mon Sep 30 10:03:39 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 30 Sep 2013 10:03:39 +0200 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> Message-ID: I wrote a Core Erlang transform if you want to look at third-party Core Erlang code: https://github.com/nox/shippai Le 29 sept. 2013 ? 20:24, Ivan uemlianin a ?crit : > That's what I've just done :D Core Erlang looks very verbose but quite regular & probably not difficult to generate. > > My questions now are: > - are there any libraries "out there" for generating Core Erlang, or do we all roll our own? > - how would one use compile:file or compile:forms with core erlang? I haven't been able to find any documentation (haven't read Richard Carlsson's Introduction paper yet). > > Many thanks > > Ivan > > -- > festina lente > > > On 29 Sep 2013, at 18:36, Erik S?e S?rensen wrote: > >> Core Erlang is an intermediate representation in the Erlang compiler - but also (afaik) a fairly well-defined/public one and one that is stable. >> I don't think you'll find much in the vein of tutorials. Try getting erlc to output the intermediate format, though, for a small program similar to what you'll be using it for. >> >> Den 29/09/2013 19.20 skrev "Ivan uemlianin" : >> Thanks! I think I'll try and head in that direction. I've had a few goes at other methods (db lookup etc) and they're much slower than this "dynamic hardcoding"). I'll sniff around for Core Erlang tutorials. >> >> Best wishes >> >> Ivan >> >> >> -- >> festina lente >> >> >> On 29 Sep 2013, at 17:48, Erik S?e S?rensen wrote: >> >>> A thing which I discovered recently (in connection with mochiglobal) is that compiling code containing large binaries, or large amounts of binaries, is quite memory-intensive. As I recall it, the numbers were ~64 bytes of RAM per byte in a binary metal; twice as much if on a 64 bit emulator. >>> Which means that if you want to compile modules containing (in sum) multimegabyte binaries, doing so from Erlang source or from full Erlang AST is a no-go. Iirc, it is feasible if starting from Core Erlang. >>> /Erik >>> >>> Den 29/09/2013 12.50 skrev "Ivan Uemlianin" : >>> Dear Anthony >>> >>> Thanks for your comment. >>> >>> Yes, that's exactly what the generated module is doing. The generated module has a single function with many clauses like this: >>> >>> f(<<"trigger", Rest/binary) -> ... >>> >>> This is why (as far as I can work out) the generated code has to be so big. >>> >>> I prefer the idea of generating and loading code to, say, updating a database table, because it seems faster and less likely to lead to bottlenecks. >>> >>> Best wishes >>> >>> Ivan >>> >>> >>> On 29/09/2013 11:38, Anthony Ramine wrote: >>> Hello Ivan, >>> >>> Out of curiosity, what does it look like? >>> >>> Pattern matching on literal values in Erlang is done with a binary search over the sorted list of patterns, I am not sure this would play well with your use case even if the compilation didn't bring the VM down. >>> >>> Regards, >>> >>> Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : >>> >>> All goes well on small test files, but the files I want to use IRL are relatively large --- around 120,000 lines. >>> >>> >>> -- >>> ============================================================ >>> Ivan A. Uemlianin PhD >>> Llaisdy >>> Speech Technology Research and Development >>> >>> ivan@REDACTED >>> www.llaisdy.com >>> llaisdy.wordpress.com >>> github.com/llaisdy >>> www.linkedin.com/in/ivanuemlianin >>> >>> festina lente >>> ============================================================ >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions -- Anthony Ramine From n.oxyde@REDACTED Mon Sep 30 10:07:18 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 30 Sep 2013 10:07:18 +0200 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> <5248788A.5030707@ninenines.eu> Message-ID: <275AB917-1340-4CEC-AEEF-8158C212CD01@gmail.com> Hello Tom, The compiler `erlc` automatically pass from_core to the compile module if the file extension is .core. Regards, Le 29 sept. 2013 ? 23:01, Tom Murphy a ?crit : > There's also just: > > erlc +to_core foo.erl > erlc +from_core foo.core > > Tom Murphy > > > On Sun, Sep 29, 2013 at 11:59 AM, Lo?c Hoguin wrote: > On 09/29/2013 08:24 PM, Ivan uemlianin wrote: > That's what I've just done :D Core Erlang looks very verbose but quite > regular & probably not difficult to generate. > > My questions now are: > - are there any libraries "out there" for generating Core Erlang, or do > we all roll our own? > > Look at the cerl module. It's just a matter of generating the proper structure using those functions. > > > - how would one use compile:file or compile:forms with core erlang? I > haven't been able to find any documentation (haven't read Richard > Carlsson's Introduction paper yet). > > Do read it. But to answer that question, compile:forms with from_core option. > > Many thanks > > Ivan > > -- > festina lente > > > On 29 Sep 2013, at 18:36, Erik S?e S?rensen > wrote: > > Core Erlang is an intermediate representation in the Erlang compiler > - but also (afaik) a fairly well-defined/public one and one that is > stable. > I don't think you'll find much in the vein of tutorials. Try getting > erlc to output the intermediate format, though, for a small program > similar to what you'll be using it for. > > Den 29/09/2013 19.20 skrev "Ivan uemlianin" >: > > > Thanks! I think I'll try and head in that direction. I've had a > few goes at other methods (db lookup etc) and they're much slower > than this "dynamic hardcoding"). I'll sniff around for Core Erlang > tutorials. > > Best wishes > > Ivan > > > -- > festina lente > > > On 29 Sep 2013, at 17:48, Erik S?e S?rensen > wrote: > > A thing which I discovered recently (in connection with > mochiglobal) is that compiling code containing large binaries, or > large amounts of binaries, is quite memory-intensive. As I > recall it, the numbers were ~64 bytes of RAM per byte in a binary > metal; twice as much if on a 64 bit emulator. > Which means that if you want to compile modules containing (in > sum) multimegabyte binaries, doing so from Erlang source or from > full Erlang AST is a no-go. Iirc, it is feasible if starting > from Core Erlang. > /Erik > > Den 29/09/2013 12.50 skrev "Ivan Uemlianin" >: > > > Dear Anthony > > Thanks for your comment. > > Yes, that's exactly what the generated module is doing. The > generated module has a single function with many clauses like > this: > > f(<<"trigger", Rest/binary) -> ... > > This is why (as far as I can work out) the generated code has > to be so big. > > I prefer the idea of generating and loading code to, say, > updating a database table, because it seems faster and less > likely to lead to bottlenecks. > > Best wishes > > Ivan > > > On 29/09/2013 11:38, Anthony Ramine wrote: > > Hello Ivan, > > Out of curiosity, what does it look like? > > Pattern matching on literal values in Erlang is done with > a binary search over the sorted list of patterns, I am > not sure this would play well with your use case even if > the compilation didn't bring the VM down. > > Regards, > > Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : > > All goes well on small test files, but the files I > want to use IRL are relatively large --- around > 120,000 lines. > > > > -- > ==============================__============================== > > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/__ivanuemlianin > > > festina lente > ==============================__============================== > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Anthony Ramine From n.oxyde@REDACTED Mon Sep 30 10:08:59 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Mon, 30 Sep 2013 10:08:59 +0200 Subject: [erlang-questions] Generating Core Erlang -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> <5248788A.5030707@ninenines.eu> Message-ID: Hello Erik, Use +time to measure which passes are taking the most time. IIRC the slow pass with regard to modules with lot of binaries is either a Core Erlang optimization pass or the Core to Kernel v3_kernel one. In both cases, targeting Core directly won't fix the problem. Regards, Le 30 sept. 2013 ? 07:31, Erik S?e S?rensen a ?crit : > Yes, indeed; considering that the goal is reduction of memory use, however, what we want is to avoid intermediate representations which explode in size on binaries. I don't know anything about the from_core parser, but the string-as-list representation of a binary literal is in itself a factor 32/64 blowup. :-/ > (Btw: I was wrong about introductions to Core Erlang; threre is one. Writing on the phone has its fact-checking disadvantages...) > > Den 29/09/2013 23.01 skrev "Tom Murphy" : > There's also just: > > erlc +to_core foo.erl > erlc +from_core foo.core > > Tom Murphy > > > On Sun, Sep 29, 2013 at 11:59 AM, Lo?c Hoguin wrote: > On 09/29/2013 08:24 PM, Ivan uemlianin wrote: > That's what I've just done :D Core Erlang looks very verbose but quite > regular & probably not difficult to generate. > > My questions now are: > - are there any libraries "out there" for generating Core Erlang, or do > we all roll our own? > > Look at the cerl module. It's just a matter of generating the proper structure using those functions. > > > - how would one use compile:file or compile:forms with core erlang? I > haven't been able to find any documentation (haven't read Richard > Carlsson's Introduction paper yet). > > Do read it. But to answer that question, compile:forms with from_core option. > > Many thanks > > Ivan > > -- > festina lente > > > On 29 Sep 2013, at 18:36, Erik S?e S?rensen > wrote: > > Core Erlang is an intermediate representation in the Erlang compiler > - but also (afaik) a fairly well-defined/public one and one that is > stable. > I don't think you'll find much in the vein of tutorials. Try getting > erlc to output the intermediate format, though, for a small program > similar to what you'll be using it for. > > Den 29/09/2013 19.20 skrev "Ivan uemlianin" >: > > > Thanks! I think I'll try and head in that direction. I've had a > few goes at other methods (db lookup etc) and they're much slower > than this "dynamic hardcoding"). I'll sniff around for Core Erlang > tutorials. > > Best wishes > > Ivan > > > -- > festina lente > > > On 29 Sep 2013, at 17:48, Erik S?e S?rensen > wrote: > > A thing which I discovered recently (in connection with > mochiglobal) is that compiling code containing large binaries, or > large amounts of binaries, is quite memory-intensive. As I > recall it, the numbers were ~64 bytes of RAM per byte in a binary > metal; twice as much if on a 64 bit emulator. > Which means that if you want to compile modules containing (in > sum) multimegabyte binaries, doing so from Erlang source or from > full Erlang AST is a no-go. Iirc, it is feasible if starting > from Core Erlang. > /Erik > > Den 29/09/2013 12.50 skrev "Ivan Uemlianin" >: > > > Dear Anthony > > Thanks for your comment. > > Yes, that's exactly what the generated module is doing. The > generated module has a single function with many clauses like > this: > > f(<<"trigger", Rest/binary) -> ... > > This is why (as far as I can work out) the generated code has > to be so big. > > I prefer the idea of generating and loading code to, say, > updating a database table, because it seems faster and less > likely to lead to bottlenecks. > > Best wishes > > Ivan > > > On 29/09/2013 11:38, Anthony Ramine wrote: > > Hello Ivan, > > Out of curiosity, what does it look like? > > Pattern matching on literal values in Erlang is done with > a binary search over the sorted list of patterns, I am > not sure this would play well with your use case even if > the compilation didn't bring the VM down. > > Regards, > > Le 29 sept. 2013 ? 11:29, Ivan Uemlianin a ?crit : > > All goes well on small test files, but the files I > want to use IRL are relatively large --- around > 120,000 lines. > > > > -- > ==============================__============================== > > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > www.llaisdy.com > llaisdy.wordpress.com > github.com/llaisdy > www.linkedin.com/in/__ivanuemlianin > > > festina lente > ==============================__============================== > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Anthony Ramine From ivan@REDACTED Mon Sep 30 10:09:16 2013 From: ivan@REDACTED (Ivan Uemlianin) Date: Mon, 30 Sep 2013 09:09:16 +0100 Subject: [erlang-questions] SOLVED -- Re: Dangers of generating a large erlang module In-Reply-To: References: <5247F2DE.4080304@llaisdy.com> <524805E7.8090001@llaisdy.com> <5248788A.5030707@ninenines.eu> Message-ID: <524931AC.4050901@llaisdy.com> Dear All Thanks to everyone for all help and comments. A less glamorous solution than Core Erlang came to me last night while I was doing the ironing. As binaries seemed to be the problem I worked out a way to remove all binaries from the generated code, and use lists instead. Then I have a wrapper function in the non-generated (?) code which converts binaries to lists and lists to binaries as appropriate. Now the 120k+ ruleset compiles in a few minutes on a 32bit emulator, so it doesn't look like there's any danger of the running out of memory. Using the ruleset is still amazingly fast: less than ten *micro*seconds for the wrapper function to return. I'll still look into Core Erlang so I'm ready for next time. Very best wishes to all Ivan On 30/09/2013 06:31, Erik S?e S?rensen wrote: > Yes, indeed; considering that the goal is reduction of memory use, > however, what we want is to avoid intermediate representations which > explode in size on binaries. I don't know anything about the from_core > parser, but the string-as-list representation of a binary literal is in > itself a factor 32/64 blowup. :-/ > (Btw: I was wrong about introductions to Core Erlang; threre is one. > Writing on the phone has its fact-checking disadvantages...) > > Den 29/09/2013 23.01 skrev "Tom Murphy" >: > > There's also just: > > erlc +to_core foo.erl > erlc +from_core foo.core > > Tom Murphy > > > On Sun, Sep 29, 2013 at 11:59 AM, Lo?c Hoguin > wrote: > > On 09/29/2013 08:24 PM, Ivan uemlianin wrote: > > That's what I've just done :D Core Erlang looks very > verbose but quite > regular & probably not difficult to generate. > > My questions now are: > - are there any libraries "out there" for generating Core > Erlang, or do > we all roll our own? > > > Look at the cerl module. It's just a matter of generating the > proper structure using those functions. > > > - how would one use compile:file or compile:forms with core > erlang? I > haven't been able to find any documentation (haven't read > Richard > Carlsson's Introduction paper yet). > > > Do read it. But to answer that question, compile:forms with > from_core option. > > Many thanks > > Ivan > > -- > festina lente > > > On 29 Sep 2013, at 18:36, Erik S?e S?rensen > > >> wrote: > > Core Erlang is an intermediate representation in the > Erlang compiler > - but also (afaik) a fairly well-defined/public one and > one that is > stable. > I don't think you'll find much in the vein of tutorials. > Try getting > erlc to output the intermediate format, though, for a > small program > similar to what you'll be using it for. > > Den 29/09/2013 19.20 skrev "Ivan uemlianin" > > >>: > > > Thanks! I think I'll try and head in that > direction. I've had a > few goes at other methods (db lookup etc) and > they're much slower > than this "dynamic hardcoding"). I'll sniff around > for Core Erlang > tutorials. > > Best wishes > > Ivan > > > -- > festina lente > > > On 29 Sep 2013, at 17:48, Erik S?e S?rensen > > >> wrote: > > A thing which I discovered recently (in > connection with > mochiglobal) is that compiling code containing > large binaries, or > large amounts of binaries, is quite > memory-intensive. As I > recall it, the numbers were ~64 bytes of RAM > per byte in a binary > metal; twice as much if on a 64 bit emulator. > Which means that if you want to compile modules > containing (in > sum) multimegabyte binaries, doing so from > Erlang source or from > full Erlang AST is a no-go. Iirc, it is > feasible if starting > from Core Erlang. > /Erik > > Den 29/09/2013 12.50 skrev "Ivan Uemlianin" > > >>: > > > Dear Anthony > > Thanks for your comment. > > Yes, that's exactly what the generated > module is doing. The > generated module has a single function with > many clauses like > this: > > f(<<"trigger", Rest/binary) -> ... > > This is why (as far as I can work out) the > generated code has > to be so big. > > I prefer the idea of generating and loading > code to, say, > updating a database table, because it seems > faster and less > likely to lead to bottlenecks. > > Best wishes > > Ivan > > > On 29/09/2013 11:38, Anthony Ramine wrote: > > Hello Ivan, > > Out of curiosity, what does it look like? > > Pattern matching on literal values in > Erlang is done with > a binary search over the sorted list of > patterns, I am > not sure this would play well with your > use case even if > the compilation didn't bring the VM down. > > Regards, > > Le 29 sept. 2013 ? 11:29, Ivan > Uemlianin a ?crit : > > All goes well on small test files, > but the files I > want to use IRL are relatively > large --- around > 120,000 lines. > > > > -- > > ==============================____============================__== > > Ivan A. Uemlianin PhD > Llaisdy > Speech Technology Research and Development > > ivan@REDACTED > > > www.llaisdy.com > > llaisdy.wordpress.com > > github.com/llaisdy > > www.linkedin.com/in/____ivanuemlianin > > > > > festina lente > > ==============================____============================__== > > ___________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > > > > http://erlang.org/mailman/____listinfo/erlang-questions > > > > > > > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > _________________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/__listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- ============================================================ Ivan A. Uemlianin PhD Llaisdy Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com github.com/llaisdy www.linkedin.com/in/ivanuemlianin festina lente ============================================================