From jixiuf@REDACTED Fri Feb 1 03:09:22 2013 From: jixiuf@REDACTED (jixiuf) Date: Fri, 01 Feb 2013 10:09:22 +0800 (CST) Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: References: <510AE2EA.4060308@jkemp.net> Message-ID: <20130201.100922.280107508503578068.jixiuf@gmail.com> >> This is the part that was failing for me, actually - I get an error about >> reloading a beam file when I ran the make from within emacs. Works OK from a >> command line, but not from emacs. > > This is what I have in my emacs.d/init.el: > > (global-set-key (kbd "C-x RET") 'compile) > > So it's CTRL-C M and I get the compile prompt, which defaults to "make > -k", then enter. Of course I have a Makefile that contains ./rebar > compile in the default target. > > Emacs knows how to go to the source line on error with CTRL ` > > I personally would not want my compiler to auto-compile when I saved > the file, though I realize some IDEs not only do that, but try to > compile as you type! Everyone has a different thing though. > I write erlang-dired-mode.el for emacs https://raw.github.com/jixiuf/erlang-dired-mode/master/erlang-dired-mode.el there is a function called `erlang-auto-compile' maybe can help you if you use emacs. From yurinvv@REDACTED Fri Feb 1 04:16:37 2013 From: yurinvv@REDACTED (Slava Yurin) Date: Fri, 01 Feb 2013 10:16:37 +0700 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <510ADD5A.4060103@jkemp.net> References: <510ADD5A.4060103@jkemp.net> Message-ID: <335831359688597@web12f.yandex.ru> An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Feb 1 04:57:39 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 1 Feb 2013 06:57:39 +0300 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <335831359688597@web12f.yandex.ru> References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> Message-ID: Frankly speaking, it is one of the reasons, why I don't use releases at all: it is a horrible mess with file locations. I always configure Emakefile and run in erlang console make:all([load]). It is much faster and more predictable than all your reloaders and syncs. On Fri, Feb 1, 2013 at 7:16 AM, Slava Yurin wrote: > Hi John. > > Maybe https://github.com/tex/ssync can help you. > > 01.02.2013, 04:09, "John Kemp" : > > Hi, > > I have been writing an application using rebar to build the app. I run > the server and open a console. I then change the source code and > apparently I need to leave the console, 'make rel' and restart the app > and the console. > > I'm sure there's a better way to do what I'm doing. > > One project I found is https://github.com/rustyio/sync, but this did not > (yet) work for me according to the instructions. > > How can I have my rebar (or just Erlang) app automatically recompile > ('make rel') when one of the source files changes, or run a command from > within the console to do this? > > Cheers, > > JohnK > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samuelrivas@REDACTED Fri Feb 1 07:56:54 2013 From: samuelrivas@REDACTED (Samuel) Date: Fri, 1 Feb 2013 07:56:54 +0100 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> Message-ID: make:all([load]) was my usual way of working until not so long ago. The main problem is that projects usually come without Emakefiles, but I had this in my erlang default, which worked in most of the cases: make() -> make([]). make(Options) -> make:all( [load, debug_info | Options] ++ [{i, "../include"}, {outdir, "../ebin"}]). Lately I have been using EDTS (https://github.com/tjarvstrand/edts) which is truly awesome, but it breaks my usual workflow as it compiles every time I save, so make gently refuses to compile anything, thus not load the new code. But edts automatically loads all changes in the node shell (emacs wrapper around the erlang shell) which is in many ways better than the default erlang shell, so I now use that and rely on edts reloading all my changes automatically. It is working well so far, and is slightly faster than my previous edit-save-make:all workflow. Best -- Samuel From qoocku@REDACTED Fri Feb 1 09:25:52 2013 From: qoocku@REDACTED (=?UTF-8?B?RGFtaWFuIERvYnJvY3p5xYRza2k=?=) Date: Fri, 01 Feb 2013 09:25:52 +0100 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <510ADD5A.4060103@jkemp.net> References: <510ADD5A.4060103@jkemp.net> Message-ID: <510B7C10.3010802@gmail.com> A long time ago I used the attached bash script which utilizes "inotifywait" tool (I think it's Linux only feature) to wait for some files changed and then it does a command you've given. I used it for TDD ;);) Theoretically you may use "inotifywait" to compile if some sources changed and then to reload the modules if *.beam files changed (due to compilation) using some rpc:call to specified node. -- D. W dniu 31.01.2013 22:08, John Kemp pisze: > Hi, > > I have been writing an application using rebar to build the app. I run > the server and open a console. I then change the source code and > apparently I need to leave the console, 'make rel' and restart the app > and the console. > > I'm sure there's a better way to do what I'm doing. > > One project I found is https://github.com/rustyio/sync, but this did > not (yet) work for me according to the instructions. > > How can I have my rebar (or just Erlang) app automatically recompile > ('make rel') when one of the source files changes, or run a command > from within the console to do this? > > Cheers, > > JohnK > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- A non-text attachment was scrubbed... Name: fmon.sh Type: application/x-shellscript Size: 1877 bytes Desc: not available URL: From n.oxyde@REDACTED Fri Feb 1 13:05:37 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Fri, 1 Feb 2013 13:05:37 +0100 Subject: [erlang-questions] OTP R16A has been released In-Reply-To: <510AC5D4.7040500@ernovation.com> References: <510AC5D4.7040500@ernovation.com> Message-ID: <274E5960-EB96-43CB-99E0-7E0AD26E28F7@gmail.com> IMHO these two files shouldn't even be versioned. There are some stuff in the OTP build process that makes me cry, like how there are 4 identical aclocal.m4 files in 4 different directories when there could be only one either at the repository root or in erts/automake/ so that erts can be built standalone. -- Anthony Ramine Le 31 janv. 2013 ? 20:28, Erik Reitsma a ?crit : > Hi Kenneth, > > Could you update the config.sub and config.guess to include arm-linux-androideabi, unless there is a reason to keep the 2007 version in favor of the one from 2012-12-30? The fewer files I need to update to cross-compile for Android, the better! > > Regards, > *Erik. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kostis@REDACTED Fri Feb 1 13:35:32 2013 From: kostis@REDACTED (Kostis Sagonas) Date: Fri, 01 Feb 2013 13:35:32 +0100 Subject: [erlang-questions] OTP R16A has been released In-Reply-To: <274E5960-EB96-43CB-99E0-7E0AD26E28F7@gmail.com> References: <510AC5D4.7040500@ernovation.com> <274E5960-EB96-43CB-99E0-7E0AD26E28F7@gmail.com> Message-ID: <510BB694.8080206@cs.ntua.gr> On 02/01/2013 01:05 PM, Anthony Ramine wrote: > IMHO these two files shouldn't even be versioned. There are some stuff in > the OTP build process that makes me cry, like how there are 4 identical > aclocal.m4 files in 4 different directories when there could be only one > either at the repository root or in erts/automake/ so that erts can be > built standalone. You are very kind in your words... ;) There are actually 6 of them, not just 4, all identical! kostis@REDACTED:~/HiPE/otp$ wc `find . -name aclocal.m4` 1905 5931 61331 ./lib/megaco/aclocal.m4 1905 5931 61331 ./lib/wx/aclocal.m4 1905 5931 61331 ./lib/erl_interface/aclocal.m4 1905 5931 61331 ./lib/odbc/aclocal.m4 1905 5931 61331 ./aclocal.m4 1905 5931 61331 ./erts/aclocal.m4 For what's worth, I second your suggestions. Cheers, Kostis From define.null@REDACTED Fri Feb 1 13:47:14 2013 From: define.null@REDACTED (Defnull) Date: Fri, 1 Feb 2013 04:47:14 -0800 (PST) Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: References: <4E1D3FED.4070409@gmail.com> Message-ID: <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> This syntax is broken in R15B03, I failed to make edoc use it. On Wednesday, July 13, 2011 1:30:43 PM UTC+4, Lukas Larsson wrote: > > Hi, > > It is not supported via the traditional edoc syntax, however it is > supported by the new type documentation generation. If you write > > -type key() :: binary() | atom(). > %% This type is awesome. Really, it is great. > > The comment below the type will be included in the generated edoc. > > Lukas > > On Wed, Jul 13, 2011 at 8:49 AM, Richard Carlsson > > wrote: > >> On 2011-07-12 23:25, Daniel Goertzen wrote: >> >>> I am trying to document a "-type" close to the top of my modules like >>> this... >>> >>> %% @doc This type is awesome. Really, it is great. >>> -type key() :: binary() | atom(). >>> >>> ... but it collides with my module @doc entry above it ("multiple @doc >>> tag" error). >>> >>> Is there a right way to document types or is this not supported right >>> now? The generated documentation shows all my types nicely, but I'd >>> love to have a bit of explanation appear with them. >>> >> >> It's not supported right now, but it's a good idea. Thanks. >> >> /Richard >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-q...@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From define.null@REDACTED Fri Feb 1 14:08:45 2013 From: define.null@REDACTED (Defnull) Date: Fri, 1 Feb 2013 05:08:45 -0800 (PST) Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> Message-ID: <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com> My fault, type should be used in specs in order to appear in the documentation On Friday, February 1, 2013 4:47:14 PM UTC+4, Defnull wrote: > > This syntax is broken in R15B03, I failed to make edoc use it. > > On Wednesday, July 13, 2011 1:30:43 PM UTC+4, Lukas Larsson wrote: >> >> Hi, >> >> It is not supported via the traditional edoc syntax, however it is >> supported by the new type documentation generation. If you write >> >> -type key() :: binary() | atom(). >> %% This type is awesome. Really, it is great. >> >> The comment below the type will be included in the generated edoc. >> >> Lukas >> >> On Wed, Jul 13, 2011 at 8:49 AM, Richard Carlsson wrote: >> >>> On 2011-07-12 23:25, Daniel Goertzen wrote: >>> >>>> I am trying to document a "-type" close to the top of my modules like >>>> this... >>>> >>>> %% @doc This type is awesome. Really, it is great. >>>> -type key() :: binary() | atom(). >>>> >>>> ... but it collides with my module @doc entry above it ("multiple @doc >>>> tag" error). >>>> >>>> Is there a right way to document types or is this not supported right >>>> now? The generated documentation shows all my types nicely, but I'd >>>> love to have a bit of explanation appear with them. >>>> >>> >>> It's not supported right now, but it's a good idea. Thanks. >>> >>> /Richard >>> >>> ______________________________**_________________ >>> erlang-questions mailing list >>> erlang-q...@REDACTED >>> http://erlang.org/mailman/**listinfo/erlang-questions >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From arif@REDACTED Fri Feb 1 16:04:22 2013 From: arif@REDACTED (arif@REDACTED) Date: Fri, 1 Feb 2013 07:04:22 -0800 Subject: [erlang-questions] Websocket client for yaws server - problem with fragmentation? Message-ID: <58a4c2db448767ace0df65607fecad5e.squirrel@fulvetta.riseup.net> Hi, I have tried to write some client functions to test the yaws websocket server (https://github.com/arifishaq/erlws.git) I have discovered a couple of interesting things I didn't know about: - there is support for the ping message - there is support for fragmented messages On the latter, I have run into some problems and am curious to know what's wrong, if anything is. I am testing the server usign ct_run (see the suite in the test directory) In practice, everything works, except when I try to send fragmented messages. If there isn't a pause of at least 1 ms between the sending of each fragmented message, the tests fragmented_text_delivery and fragmented_binary_delivery fail. Could this be a problem with the common test framework? Or with yaws? Actually, the server does collect all the fragments and reports the composed message to the callback module. As I am a newbie to Erlang, I still don't know how to use trace or dbg to try to figure out what's going on. Thanks for any suggestions. Best regards Arif From serge@REDACTED Fri Feb 1 16:20:36 2013 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 01 Feb 2013 10:20:36 -0500 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com> References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com> Message-ID: <510BDD44.20007@aleynikov.org> I have a similar question. In the following test module the generated documentation for proto() type includes proper comment, but the options() doc doesn't include comment. It seems to be immaterial if "%% Protocol options." appear on the same line as the ending "]." of the options() type or the next line. Why is the edoc generator doesn't include comment for the options() type? -module(test). -export([t/2]). -type proto() :: tcp | ssl. %% Protocol type. -type options() :: [ {server, Server::string()} | {port, Port::integer()} ]. %% Protocol options. %% * Server - server to connect to (no MX lookup) %% * Relay - domain to do MX lookup of list of servers -spec t(Proto :: proto(), Options :: options()) -> ok. t(_Proto, _Options) -> ok. Segment of relevant generated docs:

options()

options() =
    [{server, Server :: string()} | {port, Port :: integer()}]

proto()

proto() = tcp | ssl

Protocol type.

> On Wednesday, July 13, 2011 1:30:43 PM UTC+4, Lukas Larsson wrote: > Hi, > > It is not supported via the traditional edoc syntax, however it > is supported by the new type documentation generation. If you write > > -type key() :: binary() | atom(). > %% This type is awesome. Really, it is great. > > The comment below the type will be included in the generated edoc. > > Lukas From john@REDACTED Fri Feb 1 16:40:25 2013 From: john@REDACTED (John Kemp) Date: Fri, 1 Feb 2013 10:40:25 -0500 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <335831359688597@web12f.yandex.ru> References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> Message-ID: <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> This does not seem to work on a Mac FWIW since there is no equivalent of the inotify tools on OS X, for similar reasons to Damian's suggestion. Otherwise (on Linux), it looks promising... Cheers, John On Jan 31, 2013, at 10:16 PM, Slava Yurin wrote: > Hi John. > > Maybe https://github.com/tex/ssync can help you. > > 01.02.2013, 04:09, "John Kemp" : > Hi, > > I have been writing an application using rebar to build the app. I run > the server and open a console. I then change the source code and > apparently I need to leave the console, 'make rel' and restart the app > and the console. > > I'm sure there's a better way to do what I'm doing. > > One project I found is https://github.com/rustyio/sync, but this did not > (yet) work for me according to the instructions. > > How can I have my rebar (or just Erlang) app automatically recompile > ('make rel') when one of the source files changes, or run a command from > within the console to do this? > > Cheers, > > JohnK > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From g@REDACTED Fri Feb 1 16:48:05 2013 From: g@REDACTED (Garrett Smith) Date: Fri, 1 Feb 2013 09:48:05 -0600 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> Message-ID: For dev purposes, I bet that a few millisecond delay in change detection + recompile is okay. inotify facilities is a bit much IMO for this application. The Mochiweb reloader polls for changes to the beam files themselves and there is a delay, but it's not a problem, unless you're not *really* human. On Fri, Feb 1, 2013 at 9:40 AM, John Kemp wrote: > This does not seem to work on a Mac FWIW since there is no equivalent of the inotify tools on OS X, for similar reasons to Damian's suggestion. Otherwise (on Linux), it looks promising... > > Cheers, > > John > > On Jan 31, 2013, at 10:16 PM, Slava Yurin wrote: > >> Hi John. >> >> Maybe https://github.com/tex/ssync can help you. >> >> 01.02.2013, 04:09, "John Kemp" : >> Hi, >> >> I have been writing an application using rebar to build the app. I run >> the server and open a console. I then change the source code and >> apparently I need to leave the console, 'make rel' and restart the app >> and the console. >> >> I'm sure there's a better way to do what I'm doing. >> >> One project I found is https://github.com/rustyio/sync, but this did not >> (yet) work for me according to the instructions. >> >> How can I have my rebar (or just Erlang) app automatically recompile >> ('make rel') when one of the source files changes, or run a command from >> within the console to do this? >> >> Cheers, >> >> JohnK >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From serge@REDACTED Fri Feb 1 16:53:05 2013 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 01 Feb 2013 10:53:05 -0500 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> Message-ID: <510BE4E1.3070608@aleynikov.org> Another alternative to autoloading of modified beam files the moment modification is made is to include the following user_default module [1] in the ERL_LIBS path, which will add a handful of useful commands to the shell including l() that will reload all modified beams. Serge [1] https://github.com/saleyn/util/blob/master/src/user_default.erl On 2/1/2013 10:40 AM, John Kemp wrote: > This does not seem to work on a Mac FWIW since there is no equivalent of the inotify tools on OS X, for similar reasons to Damian's suggestion. Otherwise (on Linux), it looks promising... > > Cheers, > > John > > On Jan 31, 2013, at 10:16 PM, Slava Yurin wrote: > >> Hi John. >> >> Maybe https://github.com/tex/ssync can help you. >> >> 01.02.2013, 04:09, "John Kemp" : >> Hi, >> >> I have been writing an application using rebar to build the app. I run >> the server and open a console. I then change the source code and >> apparently I need to leave the console, 'make rel' and restart the app >> and the console. >> >> I'm sure there's a better way to do what I'm doing. >> >> One project I found is https://github.com/rustyio/sync, but this did not >> (yet) work for me according to the instructions. >> >> How can I have my rebar (or just Erlang) app automatically recompile >> ('make rel') when one of the source files changes, or run a command from >> within the console to do this? >> >> Cheers, >> >> JohnK >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mononcqc@REDACTED Fri Feb 1 17:36:27 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Fri, 1 Feb 2013 11:36:27 -0500 Subject: [erlang-questions] OTP R16A has been released In-Reply-To: References: <20130130154013.GA10589@ferdmbp.local> Message-ID: <20130201163626.GA98072@ferdmbp.local> Hi, just to let you (and other readers) know, I've just updated the current erlang-history repository to work with R16A. (https://github.com/ferd/erlang-history) I've tested it locally on my machines and it still works fine here. Regards, Fred. On 01/30, Motiejus Jak?tys wrote: > On Wed, Jan 30, 2013 at 3:40 PM, Fred Hebert wrote: > > Also: > > > > OTP-10739 Add search to Erlang shell's history. Thanks to Fred > > Herbert. > > History between sessions is not implemented (I get readline-style > ctrl-r in same session fine which is super cool)? > > /home/motiejus$ erl -config hist.config -name labas > Erlang R16A (erts-5.10) [source] [smp:2:2] [async-threads:10] [hipe] > [kernel-poll:false] > > Eshell V5.10 (abort with ^G) > (labas@REDACTED)1> application:get_all_env(kernel). > [{hist_file,"erlang-history"}, > {included_applications,[]}, > {error_logger,tty}, > {hist_size,120}, > {hist_drop,["q().","init:stop()."]}] > (labas@REDACTED)2> q(). > ok > (labas@REDACTED)3> % > /home/motiejus$ ls -lA | grep erlang > -r-------- 1 motiejus motiejus 20 Rgp 23 23:00 .erlang.cookie > > > -- > Motiejus Jak?tys From wallentin.dahlberg@REDACTED Fri Feb 1 18:43:24 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Fri, 1 Feb 2013 18:43:24 +0100 Subject: [erlang-questions] OTP R16A has been released In-Reply-To: <510BB694.8080206@cs.ntua.gr> References: <510AC5D4.7040500@ernovation.com> <274E5960-EB96-43CB-99E0-7E0AD26E28F7@gmail.com> <510BB694.8080206@cs.ntua.gr> Message-ID: "stand alone" apps with there own configure. These used to be links, but cannot be links anymore. We didn't want their macros to deviate. I utterly hate this too. 2013/2/1 Kostis Sagonas > On 02/01/2013 01:05 PM, Anthony Ramine wrote: > >> IMHO these two files shouldn't even be versioned. There are some stuff in >> the OTP build process that makes me cry, like how there are 4 identical >> aclocal.m4 files in 4 different directories when there could be only one >> either at the repository root or in erts/automake/ so that erts can be >> built standalone. >> > > You are very kind in your words... ;) > > There are actually 6 of them, not just 4, all identical! > > kostis@REDACTED:~/HiPE/**otp$ wc `find . -name aclocal.m4` > 1905 5931 61331 ./lib/megaco/aclocal.m4 > 1905 5931 61331 ./lib/wx/aclocal.m4 > 1905 5931 61331 ./lib/erl_interface/aclocal.m4 > 1905 5931 61331 ./lib/odbc/aclocal.m4 > 1905 5931 61331 ./aclocal.m4 > 1905 5931 61331 ./erts/aclocal.m4 > > For what's worth, I second your suggestions. > > Cheers, > Kostis > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Fri Feb 1 19:21:15 2013 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 01 Feb 2013 12:21:15 -0600 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: Message of "Fri, 01 Feb 2013 10:53:05 EST." <510BE4E1.3070608@aleynikov.org> Message-ID: <47389.1359742875@snookles.snookles.com> Serge Aleynikov wrote: sa> [1] https://github.com/saleyn/util/blob/master/src/user_default.erl Howdy, Serge. I've been using the l() function from that user_default.erl module for years. It's tremendously handy. There are times where I want to reload several modules after recompiling but load them sometime at my choosing rather than immediately after the compiler is done. l() does that. -Scott From dch@REDACTED Fri Feb 1 19:39:25 2013 From: dch@REDACTED (Dave Cottlehuber) Date: Fri, 1 Feb 2013 19:39:25 +0100 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> Message-ID: On 1 February 2013 16:40, John Kemp wrote: > This does not seem to work on a Mac FWIW since there is no equivalent of the inotify tools on OS X, for similar reasons to Damian's suggestion. Otherwise (on Linux), it looks promising... I've used rustyio/sync successfully on Windows and Mac, with notifu and growl respectively. More info in the README.md at https://github.com/rustyio/sync The key things I found: - run sync *before* you start editing your erlang files - if the beam doesn't change, nothing will get reloaded A+ Dave From jeremy@REDACTED Fri Feb 1 19:54:38 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Fri, 1 Feb 2013 10:54:38 -0800 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> Message-ID: +1 for rustyio/sync. Haven't had any problems. @John it looks like you are running a released version compiled with rebar? Sync or anything else won't work with that approach. Load the beam files directly with erl -pa path/to/ebin/dirs deps/*/ebin On Fri, Feb 1, 2013 at 10:39 AM, Dave Cottlehuber wrote: > On 1 February 2013 16:40, John Kemp wrote: > > This does not seem to work on a Mac FWIW since there is no equivalent of > the inotify tools on OS X, for similar reasons to Damian's suggestion. > Otherwise (on Linux), it looks promising... > > I've used rustyio/sync successfully on Windows and Mac, with notifu > and growl respectively. More info in the README.md at > https://github.com/rustyio/sync > > The key things I found: > > - run sync *before* you start editing your erlang files > - if the beam doesn't change, nothing will get reloaded > > A+ > Dave > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Fri Feb 1 20:09:37 2013 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 01 Feb 2013 14:09:37 -0500 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <47389.1359742875@snookles.snookles.com> References: <47389.1359742875@snookles.snookles.com> Message-ID: <510C12F1.8040903@aleynikov.org> Hi Scott, Nice to know I'm not the only one finding it useful! ;-) Other than for auto-compilation/loading of erlydtl templates, I also find that having control over the moment of code loading is more advantageous, as the function shows what modules were loaded, so you know what version of code is running. Concerning the user_default, I believe I added a couple more useful functions there (i.e tc/2, tc/4) after I published this module a while back. Enjoy! ;-) Serge On 2/1/2013 1:21 PM, Scott Lystig Fritchie wrote: > Serge Aleynikov wrote: > > sa> [1] https://github.com/saleyn/util/blob/master/src/user_default.erl > > Howdy, Serge. I've been using the l() function from that > user_default.erl module for years. It's tremendously handy. There are > times where I want to reload several modules after recompiling but load > them sometime at my choosing rather than immediately after the compiler > is done. l() does that. > > -Scott > From diego.llarrull@REDACTED Fri Feb 1 21:01:10 2013 From: diego.llarrull@REDACTED (Diego Llarrull) Date: Fri, 01 Feb 2013 17:01:10 -0300 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <510C12F1.8040903@aleynikov.org> References: <47389.1359742875@snookles.snookles.com> <510C12F1.8040903@aleynikov.org> Message-ID: <510C1F06.3080304@tecso.coop> Hello everyone, I'm working on a riak_core based project, that is, one which uses rebar + GNU make and where the src files are located in different folders than the beam files, which are inside inside zipped (.ez) files. sync + libnotify will do the trick for unzipped files, but some extra trickery was needed to load the new .beam files a) compile the modified sources b) (create and) run a script that generates the .zip files containing the .beams and deploys those files in their destination directory c) run the following code: reload() -> Modules = [ X || {X,L} <- code:all_loaded(), is_list(L),string:str(L, ?PATH_TO_CODE_DIR) =/= 0], lists:map(fun(X) -> c:l(X) end, Modules). Hope this helps anyone, right now I'm trying out ssync to see if it is able to handle zipped files. Best regards, Diego On 01/02/13 16:09, Serge Aleynikov wrote: > Hi Scott, > > Nice to know I'm not the only one finding it useful! ;-) > > Other than for auto-compilation/loading of erlydtl templates, I also > find that having control over the moment of code loading is more > advantageous, as the function shows what modules were loaded, so you > know what version of code is running. > > Concerning the user_default, I believe I added a couple more useful > functions there (i.e tc/2, tc/4) after I published this module a while > back. Enjoy! ;-) > > Serge > > On 2/1/2013 1:21 PM, Scott Lystig Fritchie wrote: >> Serge Aleynikov wrote: >> >> sa> [1] https://github.com/saleyn/util/blob/master/src/user_default.erl >> >> Howdy, Serge. I've been using the l() function from that >> user_default.erl module for years. It's tremendously handy. There are >> times where I want to reload several modules after recompiling but load >> them sometime at my choosing rather than immediately after the compiler >> is done. l() does that. >> >> -Scott >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From nyarly@REDACTED Fri Feb 1 21:41:05 2013 From: nyarly@REDACTED (Judson Lester) Date: Fri, 1 Feb 2013 12:41:05 -0800 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <510C1F06.3080304@tecso.coop> References: <47389.1359742875@snookles.snookles.com> <510C12F1.8040903@aleynikov.org> <510C1F06.3080304@tecso.coop> Message-ID: On a related note, has anyone written Common Test hooks to compile/reload the CUT modules before running tests? When I get back to Erlang, it's the top of my list, but I really wouldn't complain if it's been done already. Judson On Fri, Feb 1, 2013 at 12:01 PM, Diego Llarrull wrote: > Hello everyone, > > I'm working on a riak_core based project, that is, one which uses rebar + > GNU make and where the src files are located in different folders than the > beam files, which are inside inside zipped (.ez) files. > > sync + libnotify will do the trick for unzipped files, but some extra > trickery was needed to load the new .beam files > > a) compile the modified sources > b) (create and) run a script that generates the .zip files containing the > .beams and deploys those files in their destination directory > c) run the following code: > > reload() -> > Modules = [ X || {X,L} <- code:all_loaded(), is_list(L),string:str(L, > ?PATH_TO_CODE_DIR) =/= 0], > lists:map(fun(X) -> c:l(X) end, Modules). > > Hope this helps anyone, right now I'm trying out ssync to see if it is > able to handle zipped files. > > Best regards, > > Diego > > > On 01/02/13 16:09, Serge Aleynikov wrote: > >> Hi Scott, >> >> Nice to know I'm not the only one finding it useful! ;-) >> >> Other than for auto-compilation/loading of erlydtl templates, I also >> find that having control over the moment of code loading is more >> advantageous, as the function shows what modules were loaded, so you >> know what version of code is running. >> >> Concerning the user_default, I believe I added a couple more useful >> functions there (i.e tc/2, tc/4) after I published this module a while >> back. Enjoy! ;-) >> >> Serge >> >> On 2/1/2013 1:21 PM, Scott Lystig Fritchie wrote: >> >>> Serge Aleynikov wrote: >>> >>> sa> [1] https://github.com/saleyn/**util/blob/master/src/user_** >>> default.erl >>> >>> Howdy, Serge. I've been using the l() function from that >>> user_default.erl module for years. It's tremendously handy. There are >>> times where I want to reload several modules after recompiling but load >>> them sometime at my choosing rather than immediately after the compiler >>> is done. l() does that. >>> >>> -Scott >>> >>> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gumm@REDACTED Fri Feb 1 21:46:28 2013 From: gumm@REDACTED (Jesse Gumm) Date: Fri, 1 Feb 2013 14:46:28 -0600 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> Message-ID: I too run Rusty's sync. There are a few situations where sync breaks down a bit: 1) Includes - sync does not detect changes to include files, and so if you were to change an include with records, for example, you could end up with modules that have the old version of the record loaded, and other modules with the new version of the record loaded. 2) parse_transform - From what I understand, sync has some issue with parse_transforms. This is what, as far as I understand, inspired the development of tex/ssync - which works similarly to sync, but watches all files (even rebar.config, from what it seems) and relies on rebar to recompile everything, then reloads the beams. I haven't played with ssync at all, but I've been meaning to, it seems a worthwhile effort to offload the compiling to rebar and just watch for file changes to trigger the recompile. -Jesse On Fri, Feb 1, 2013 at 12:54 PM, Jeremy Ong wrote: > +1 for rustyio/sync. > > Haven't had any problems. @John it looks like you are running a released > version compiled with rebar? Sync or anything else won't work with that > approach. Load the beam files directly with erl -pa path/to/ebin/dirs > deps/*/ebin > > > On Fri, Feb 1, 2013 at 10:39 AM, Dave Cottlehuber wrote: > >> On 1 February 2013 16:40, John Kemp wrote: >> > This does not seem to work on a Mac FWIW since there is no equivalent >> of the inotify tools on OS X, for similar reasons to Damian's suggestion. >> Otherwise (on Linux), it looks promising... >> >> I've used rustyio/sync successfully on Windows and Mac, with notifu >> and growl respectively. More info in the README.md at >> https://github.com/rustyio/sync >> >> The key things I found: >> >> - run sync *before* you start editing your erlang files >> - if the beam doesn't change, nothing will get reloaded >> >> A+ >> Dave >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Fri Feb 1 22:28:43 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Fri, 01 Feb 2013 22:28:43 +0100 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> <8B5F28A9-F0FE-469A-B0E1-11CA26A9242E@jkemp.net> Message-ID: <510C338B.6090408@gmail.com> On 2013-02-01 16:48, Garrett Smith wrote: > For dev purposes, I bet that a few millisecond delay in change > detection + recompile is okay. inotify facilities is a bit much IMO > for this application. > > The Mochiweb reloader polls for changes to the beam files themselves > and there is a delay, but it's not a problem, unless you're not > *really* human. There's also this: https://github.com/richcarl/file_monitor I also had a couple of rather experimental modules called code_monitor and autoload - I used to keep them under my EUnit development repo, but I recently removed them and made a new repo for file_monitor. The other files can be dug out from the history (https://github.com/richcarl/eunit/commits/master) if you want to play with them. /Richard From steven.charles.davis@REDACTED Sat Feb 2 01:47:48 2013 From: steven.charles.davis@REDACTED (Steve Davis) Date: Fri, 1 Feb 2013 16:47:48 -0800 (PST) Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: References: <510ADD5A.4060103@jkemp.net> <335831359688597@web12f.yandex.ru> Message-ID: <3fec85bd-6f31-4c07-ac4f-b0008e9765f0@googlegroups.com> +1 On Thursday, January 31, 2013 9:57:39 PM UTC-6, Max Lapshin wrote: > > Frankly speaking, it is one of the reasons, why I don't use releases at > all: it is a horrible mess with file locations. > > I always configure Emakefile and run in erlang console make:all([load]). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Sat Feb 2 09:39:15 2013 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Sat, 02 Feb 2013 11:39:15 +0300 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <510ADD5A.4060103@jkemp.net> References: <510ADD5A.4060103@jkemp.net> Message-ID: <510CD0B3.9010307@gmail.com> We use modified version of sync https://github.com/ten0s/sync that supports releases, detects new files, and monitors changes in include files (at least it should, don't use this feature often) as far as the next requirements are met: 1. Make sync somehow available to your release. Include it into your release or put it in the erlang libs. 2. Build the project with debug_info. 3. Remove {debug_info, strip} from reltool.config 4. Add {excl_archive_filters, [".*"]} to reltool.config 5. Start sync:start() before making any changes. 6. Disable any growl notifications (optional) We also edit files in emacs with flymake mode enabled using custom erlang syntax checker for *.erl, *.hrl, *.config files https://github.com/ten0s/syntaxerl In general our workflow is: 1. Build the project with rebar $ make 2. Start it in the console mode $ make console 3. Start sync > sync:start(). 4. Now start edit files. Whenever flymake says that the file is OK pretty soon sync will reload it. This workflow works fine until you need to make big changes like adding a child to a supervisor or make a new application. In this case go to point 1 and start over. Hope this helps. On 02/01/2013 12:08 AM, John Kemp wrote: > Hi, > > I have been writing an application using rebar to build the app. I run > the server and open a console. I then change the source code and > apparently I need to leave the console, 'make rel' and restart the app > and the console. > > I'm sure there's a better way to do what I'm doing. > > One project I found is https://github.com/rustyio/sync, but this did > not (yet) work for me according to the instructions. > > How can I have my rebar (or just Erlang) app automatically recompile > ('make rel') when one of the source files changes, or run a command > from within the console to do this? > > Cheers, > > JohnK > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Dmitry Klionsky From sergej.jurecko@REDACTED Sat Feb 2 09:58:16 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sat, 2 Feb 2013 09:58:16 +0100 Subject: [erlang-questions] "Automatic" recompile app from a console In-Reply-To: <510CD0B3.9010307@gmail.com> References: <510ADD5A.4060103@jkemp.net> <510CD0B3.9010307@gmail.com> Message-ID: <4FD6F5AC-7EF5-4C93-8833-5416319FC363@gmail.com> We actually were not aware of sync and developed our own similar solution. It simply works by monitoring ~/ebin, ~/etc, ~/src,~/include,~/apps/../ebin,~/apps/.../src,~/apps/.../dtl,~/deps/../ebin We have a lot of django template files and we put them in dtl folders. They get compiled with erlydtl. If a .hrl file gets changed, all source files get compiled for that project as well. For instance hrlname.hrl, will cause all sources that start with hrlname_ to get compiled. Same with base django template files. if file name is projectname_base.dtl, every .dtl file that starts with projectname_ will get compiled. We use one large repository for all projects and because there are quite a few apps at the moment, we have simple C/C++ programs that use platform specific APIs for file notifications (FSEvents for osx, inotify for linux). They are pretty trivial one file sources that get compiled automatically on erlang startup if they are not compiled yet. Then they are run with open_port and communicate via stdout. If anyone wants to make use of those C/C++ file notification programs we can gladly open source them. Sergej On Feb 2, 2013, at 9:39 AM, Dmitry Klionsky wrote: > We use modified version of sync https://github.com/ten0s/sync that supports releases, detects new files, and monitors changes in include files (at least it should, don't use this feature often) as far as the next requirements are met: > 1. Make sync somehow available to your release. Include it into your release or put it in the erlang libs. > 2. Build the project with debug_info. > 3. Remove {debug_info, strip} from reltool.config > 4. Add {excl_archive_filters, [".*"]} to reltool.config > 5. Start sync:start() before making any changes. > 6. Disable any growl notifications (optional) > > We also edit files in emacs with flymake mode enabled using custom erlang syntax checker for *.erl, *.hrl, *.config files https://github.com/ten0s/syntaxerl > > In general our workflow is: > 1. Build the project with rebar > $ make > 2. Start it in the console mode > $ make console > 3. Start sync > > sync:start(). > 4. Now start edit files. Whenever flymake says that the file is OK pretty soon sync will reload it. This workflow works fine until you need to make big changes like adding a child to a supervisor or make a new application. In this case go to point 1 and start over. > > Hope this helps. > > > On 02/01/2013 12:08 AM, John Kemp wrote: >> Hi, >> >> I have been writing an application using rebar to build the app. I run the server and open a console. I then change the source code and apparently I need to leave the console, 'make rel' and restart the app and the console. >> >> I'm sure there's a better way to do what I'm doing. >> >> One project I found is https://github.com/rustyio/sync, but this did not (yet) work for me according to the instructions. >> >> How can I have my rebar (or just Erlang) app automatically recompile ('make rel') when one of the source files changes, or run a command from within the console to do this? >> >> Cheers, >> >> JohnK >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > -- > Best regards, > Dmitry Klionsky > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From msegalis@REDACTED Sat Feb 2 15:29:54 2013 From: msegalis@REDACTED (Morgan Segalis) Date: Sat, 2 Feb 2013 15:29:54 +0100 Subject: [erlang-questions] multiple process handling one udp listen socket ? Message-ID: Hi everyone, I'm scratching my head with a particular problem... I would like to create my own udp relay server. Right now my udp relay server works fine, but it is only using one process... [CODE] start_server() -> spawn(?MODULE, sl, [80]). sl(Port) -> {ok, Socket} = gen_udp:open(Port, [binary]), inet:setopts(Socket, [{active, true}]), listen(Socket). listen(Socket) -> receive {udp, _ , Host, Port, <<0>>} -> io:fwrite("Host: ~p~n", [Host]), io:fwrite("DistantPort: ~p~n", [Port]), io:fwrite("server received:~p~n",[Socket]), io:fwrite("LocalPort: ~p~n", [inet:port(Socket)]), Message = lists:flatten(io_lib:format("~p\0", [Port])), Res = gen_udp:send(Socket, Host, Port, Message), io:fwrite("Sent back: ~p // ~p~n", [Port, Res]), listen(Socket); {udp, _, _, _, <<1:8, Ip1:1/big-unsigned-integer-unit:8, Ip2:1/big-unsigned-integer-unit:8, Ip3:1/big-unsigned-integer-unit:8, Ip4:1/big-unsigned-integer-unit:8, Port1:1/big-unsigned-integer-unit:8, Port2:1/big-unsigned-integer-unit:8, Bin/binary>>} -> Ip = {Ip1, Ip2, Ip3, Ip4}, Port = Port1 * 256 + Port2, gen_udp:send(Socket, Ip, Port, Bin), listen(Socket); Any -> listen(Socket) end. [/CODE] I'm worrying that if multiple (hundred, thousand ?) clients are using it, the process will be overwhelmed... Since right now I'm only listening one port (right now 80), I do not seems to see how to make multiple process to handle the same socket. Since udp is quite different from tcp since it does not need to connect/accept connections. I would like to know if there is a way to get multiple process to handle the listen socket Even if it means that each process is handling only one client (specific host ?)... The only thing is, that the listen port should be the same for all client... Thank you for your help, Morgan. From sergej.jurecko@REDACTED Sat Feb 2 15:44:13 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sat, 2 Feb 2013 15:44:13 +0100 Subject: [erlang-questions] multiple process handling one udp listen socket ? In-Reply-To: References: Message-ID: You can't listen on multiple processes. Do you really need all communication through a single port? This is generally not how protocols based on UDP work. If you really have to do it, you have to increase receive buffer for it. inet:setopts(Sock,[{recbuf,SomethingBig}]). Do not do anything other than receive bytes on that socket/process. Open a new process and socket for processing and reply. {active,true} is not a good idea on that process. False or once are safer options. Sergej On Feb 2, 2013, at 3:29 PM, Morgan Segalis wrote: > Hi everyone, > > I'm scratching my head with a particular problem... > I would like to create my own udp relay server. > > Right now my udp relay server works fine, but it is only using one process... > > [CODE] > > start_server() -> > spawn(?MODULE, sl, [80]). > > sl(Port) -> > {ok, Socket} = gen_udp:open(Port, [binary]), > inet:setopts(Socket, [{active, true}]), > listen(Socket). > > listen(Socket) -> > receive > {udp, _ , Host, Port, <<0>>} -> > io:fwrite("Host: ~p~n", [Host]), > io:fwrite("DistantPort: ~p~n", [Port]), > io:fwrite("server received:~p~n",[Socket]), > io:fwrite("LocalPort: ~p~n", [inet:port(Socket)]), > Message = lists:flatten(io_lib:format("~p\0", [Port])), > Res = gen_udp:send(Socket, Host, Port, Message), > io:fwrite("Sent back: ~p // ~p~n", [Port, Res]), > listen(Socket); > {udp, _, _, _, <<1:8, > Ip1:1/big-unsigned-integer-unit:8, > Ip2:1/big-unsigned-integer-unit:8, > Ip3:1/big-unsigned-integer-unit:8, > Ip4:1/big-unsigned-integer-unit:8, > Port1:1/big-unsigned-integer-unit:8, > Port2:1/big-unsigned-integer-unit:8, > Bin/binary>>} -> > Ip = {Ip1, Ip2, Ip3, Ip4}, > Port = Port1 * 256 + Port2, > gen_udp:send(Socket, Ip, Port, Bin), > listen(Socket); > Any -> > listen(Socket) > end. > > [/CODE] > > I'm worrying that if multiple (hundred, thousand ?) clients are using it, the process will be overwhelmed... > Since right now I'm only listening one port (right now 80), I do not seems to see how to make multiple process to handle the same socket. Since udp is quite different from tcp since it does not need to connect/accept connections. > > I would like to know if there is a way to get multiple process to handle the listen socket Even if it means that each process is handling only one client (specific host ?)... The only thing is, that the listen port should be the same for all client... > > Thank you for your help, > > Morgan. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From msegalis@REDACTED Sat Feb 2 15:56:34 2013 From: msegalis@REDACTED (Morgan Segalis) Date: Sat, 2 Feb 2013 15:56:34 +0100 Subject: [erlang-questions] multiple process handling one udp listen socket ? In-Reply-To: References: Message-ID: <9E85F546-7A6D-4069-8388-6F69C63B5197@gmail.com> Hi Sergej, Thank you for your answer... Actually, I would like to listen to port 80 for firewall blocking issue, at least I'm sure that port 80 is not blocked. Since the issue is for bypassing firewall, unfortunately, creating another socket to communicate with the client won't work (The socket has to be initiated from the client to pass)... What I'm really worrying about is to handle bandwidth as fast as it can goes (1GB/s) without the thread (process) to be overwhelmed by it. What if the udp open {active, false} is set ? could I open multiple process with the same socket and wait for data on this process with gen_recv ? Le 2 f?vr. 2013 ? 15:44, Sergej Jurecko a ?crit : > You can't listen on multiple processes. Do you really need all communication through a single port? This is generally not how protocols based on UDP work. > If you really have to do it, you have to increase receive buffer for it. inet:setopts(Sock,[{recbuf,SomethingBig}]). > Do not do anything other than receive bytes on that socket/process. Open a new process and socket for processing and reply. > {active,true} is not a good idea on that process. False or once are safer options. > > Sergej > > On Feb 2, 2013, at 3:29 PM, Morgan Segalis wrote: > >> Hi everyone, >> >> I'm scratching my head with a particular problem... >> I would like to create my own udp relay server. >> >> Right now my udp relay server works fine, but it is only using one process... >> >> [CODE] >> >> start_server() -> >> spawn(?MODULE, sl, [80]). >> >> sl(Port) -> >> {ok, Socket} = gen_udp:open(Port, [binary]), >> inet:setopts(Socket, [{active, true}]), >> listen(Socket). >> >> listen(Socket) -> >> receive >> {udp, _ , Host, Port, <<0>>} -> >> io:fwrite("Host: ~p~n", [Host]), >> io:fwrite("DistantPort: ~p~n", [Port]), >> io:fwrite("server received:~p~n",[Socket]), >> io:fwrite("LocalPort: ~p~n", [inet:port(Socket)]), >> Message = lists:flatten(io_lib:format("~p\0", [Port])), >> Res = gen_udp:send(Socket, Host, Port, Message), >> io:fwrite("Sent back: ~p // ~p~n", [Port, Res]), >> listen(Socket); >> {udp, _, _, _, <<1:8, >> Ip1:1/big-unsigned-integer-unit:8, >> Ip2:1/big-unsigned-integer-unit:8, >> Ip3:1/big-unsigned-integer-unit:8, >> Ip4:1/big-unsigned-integer-unit:8, >> Port1:1/big-unsigned-integer-unit:8, >> Port2:1/big-unsigned-integer-unit:8, >> Bin/binary>>} -> >> Ip = {Ip1, Ip2, Ip3, Ip4}, >> Port = Port1 * 256 + Port2, >> gen_udp:send(Socket, Ip, Port, Bin), >> listen(Socket); >> Any -> >> listen(Socket) >> end. >> >> [/CODE] >> >> I'm worrying that if multiple (hundred, thousand ?) clients are using it, the process will be overwhelmed... >> Since right now I'm only listening one port (right now 80), I do not seems to see how to make multiple process to handle the same socket. Since udp is quite different from tcp since it does not need to connect/accept connections. >> >> I would like to know if there is a way to get multiple process to handle the listen socket Even if it means that each process is handling only one client (specific host ?)... The only thing is, that the listen port should be the same for all client... >> >> Thank you for your help, >> >> Morgan. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > From sergej.jurecko@REDACTED Sat Feb 2 16:14:41 2013 From: sergej.jurecko@REDACTED (Sergej Jurecko) Date: Sat, 2 Feb 2013 16:14:41 +0100 Subject: [erlang-questions] multiple process handling one udp listen socket ? In-Reply-To: <9E85F546-7A6D-4069-8388-6F69C63B5197@gmail.com> References: <9E85F546-7A6D-4069-8388-6F69C63B5197@gmail.com> Message-ID: <95DC69D0-C1E8-4BCA-A6E7-C67AEE2BAFF9@gmail.com> Google "udp punch" if you can make it work with that. {active,false} and multiple processes calling recv I very much doubt will work. Try it :) You will definitely not be able to handle 1gbit on 1 socket process. Not even close. If there really is no other option, I would write the program in C, which would then send data to erlang over tcp. The C program would listen, then send the data to erlang over a pool of TCP connections. Sergej On Feb 2, 2013, at 3:56 PM, Morgan Segalis wrote: > Hi Sergej, > > Thank you for your answer... > Actually, I would like to listen to port 80 for firewall blocking issue, at least I'm sure that port 80 is not blocked. > > Since the issue is for bypassing firewall, unfortunately, creating another socket to communicate with the client won't work (The socket has to be initiated from the client to pass)... > > What I'm really worrying about is to handle bandwidth as fast as it can goes (1GB/s) without the thread (process) to be overwhelmed by it. > > What if the udp open {active, false} is set ? could I open multiple process with the same socket and wait for data on this process with gen_recv ? > > Le 2 f?vr. 2013 ? 15:44, Sergej Jurecko a ?crit : > >> You can't listen on multiple processes. Do you really need all communication through a single port? This is generally not how protocols based on UDP work. >> If you really have to do it, you have to increase receive buffer for it. inet:setopts(Sock,[{recbuf,SomethingBig}]). >> Do not do anything other than receive bytes on that socket/process. Open a new process and socket for processing and reply. >> {active,true} is not a good idea on that process. False or once are safer options. >> >> Sergej >> >> On Feb 2, 2013, at 3:29 PM, Morgan Segalis wrote: >> >>> Hi everyone, >>> >>> I'm scratching my head with a particular problem... >>> I would like to create my own udp relay server. >>> >>> Right now my udp relay server works fine, but it is only using one process... >>> >>> [CODE] >>> >>> start_server() -> >>> spawn(?MODULE, sl, [80]). >>> >>> sl(Port) -> >>> {ok, Socket} = gen_udp:open(Port, [binary]), >>> inet:setopts(Socket, [{active, true}]), >>> listen(Socket). >>> >>> listen(Socket) -> >>> receive >>> {udp, _ , Host, Port, <<0>>} -> >>> io:fwrite("Host: ~p~n", [Host]), >>> io:fwrite("DistantPort: ~p~n", [Port]), >>> io:fwrite("server received:~p~n",[Socket]), >>> io:fwrite("LocalPort: ~p~n", [inet:port(Socket)]), >>> Message = lists:flatten(io_lib:format("~p\0", [Port])), >>> Res = gen_udp:send(Socket, Host, Port, Message), >>> io:fwrite("Sent back: ~p // ~p~n", [Port, Res]), >>> listen(Socket); >>> {udp, _, _, _, <<1:8, >>> Ip1:1/big-unsigned-integer-unit:8, >>> Ip2:1/big-unsigned-integer-unit:8, >>> Ip3:1/big-unsigned-integer-unit:8, >>> Ip4:1/big-unsigned-integer-unit:8, >>> Port1:1/big-unsigned-integer-unit:8, >>> Port2:1/big-unsigned-integer-unit:8, >>> Bin/binary>>} -> >>> Ip = {Ip1, Ip2, Ip3, Ip4}, >>> Port = Port1 * 256 + Port2, >>> gen_udp:send(Socket, Ip, Port, Bin), >>> listen(Socket); >>> Any -> >>> listen(Socket) >>> end. >>> >>> [/CODE] >>> >>> I'm worrying that if multiple (hundred, thousand ?) clients are using it, the process will be overwhelmed... >>> Since right now I'm only listening one port (right now 80), I do not seems to see how to make multiple process to handle the same socket. Since udp is quite different from tcp since it does not need to connect/accept connections. >>> >>> I would like to know if there is a way to get multiple process to handle the listen socket Even if it means that each process is handling only one client (specific host ?)... The only thing is, that the listen port should be the same for all client... >>> >>> Thank you for your help, >>> >>> Morgan. >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> > From sigxcpu@REDACTED Sat Feb 2 21:52:04 2013 From: sigxcpu@REDACTED (Alexandru Pirvulescu) Date: Sat, 2 Feb 2013 22:52:04 +0200 Subject: [erlang-questions] Memory growing Message-ID: Hello, If somebody could help me with an issue, I would be grateful. I'm pulling my hair off trying to figure out why an erlang app keeps growing in memory usage. The erlang:memory() after 3 days shows this: [{total,325732292}, {processes,54520269}, {processes_used,54492115}, {system,271212023}, {atom,932793}, {atom_used,912937}, {binary,2348080}, {code,8288354}, {ets,487392}, {maximum,398189416}] The interesting part is "system" section. All other data is fine (processes goes to few MBs during the night as all the started procs are either hibernated - for long running or they die because of inactivity). There are two issues here: - "system" keeps growing (around 800MB/week) - OS (Linux) shows a significantly different amount of memory usage (almost 2x, 610MB resident in this case) The only case where I've managed to make "system" grow is by calling erlang:monitor() multiple times. It grows with ~13K/call. Erlang release used is R15B03. The questions are: - is there any hidden function to inspect all the active process monitors? (as the single place where I use erlang:monitor looks fine. The spawned kid does its job and dies, so the ref should be garbage collected). - is there any way to inspect "system" section and find out what it contains? Looking at memory chunks in process map got me nowhere Thank you, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sat Feb 2 22:01:18 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sat, 2 Feb 2013 23:01:18 +0200 Subject: [erlang-questions] Memory growing In-Reply-To: References: Message-ID: <76091FD4-255F-48A2-96CF-52432F667CF9@gmail.com> Hello, This is extremely hard to answer your question not knowing the application context, what it does, etc. Just keep in mode that memory report is given in words with is either 4 or 8 byte depends on system config. This is a reason why linux show diff value. I would start with entop utility and check top processes with high men and high reductions. https://github.com/mazenharake/entop Please share those views! Best Regards, Dmitry On Feb 2, 2013, at 10:52 PM, Alexandru Pirvulescu wrote: > Hello, > > If somebody could help me with an issue, I would be grateful. I'm pulling my hair off trying to figure out why an erlang app keeps growing in memory usage. The erlang:memory() after 3 days shows this: > > [{total,325732292}, > {processes,54520269}, > {processes_used,54492115}, > {system,271212023}, > {atom,932793}, > {atom_used,912937}, > {binary,2348080}, > {code,8288354}, > {ets,487392}, > {maximum,398189416}] > > The interesting part is "system" section. All other data is fine (processes goes to few MBs during the night as all the started procs are either hibernated - for long running or they die because of inactivity). > > There are two issues here: > - "system" keeps growing (around 800MB/week) > - OS (Linux) shows a significantly different amount of memory usage (almost 2x, 610MB resident in this case) > > The only case where I've managed to make "system" grow is by calling erlang:monitor() multiple times. It grows with ~13K/call. > > Erlang release used is R15B03. > > The questions are: > - is there any hidden function to inspect all the active process monitors? (as the single place where I use erlang:monitor looks fine. The spawned kid does its job and dies, so the ref should be garbage collected). > - is there any way to inspect "system" section and find out what it contains? Looking at memory chunks in process map got me nowhere > > Thank you, > Alex > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From sigxcpu@REDACTED Sat Feb 2 22:11:21 2013 From: sigxcpu@REDACTED (Alex P) Date: Sat, 2 Feb 2013 13:11:21 -0800 (PST) Subject: [erlang-questions] Memory growing In-Reply-To: <76091FD4-255F-48A2-96CF-52432F667CF9@gmail.com> References: <76091FD4-255F-48A2-96CF-52432F667CF9@gmail.com> Message-ID: <5abf0f0e-b710-4eb3-99cf-4bef23b6153d@googlegroups.com> Hi Dmitry, Here is the output of the entop (as you can see, the processes are not the issue here, "system" section is): Node: 'agstats@REDACTED' (Connected) (R15B03/5.9.3.1) unix (linux 3.2.0) CPU:16 SMP +A:0 Time: local time 13:09:10, up for 003:12:35:31, 0ms latency, Processes: total 334 (RQ 0) at 52918 RpI using 49898.4k (49993.9k allocated) Memory: Sys 264699.0k, Atom 892.0k/910.9k, Bin 1826.3k, Code 8097.1k, Ets 474.5k Interval 1000ms, Sorting on "Reductions" (Descending), Retrieved in 12ms Pid Registered Name Reductions MQueue HSize SSize HTot <0.326.0> - 25014751303 0 10946 9 39603 <0.301.0> agstats_amqp_consume 1281607832 0 2584 9 3571 <0.315.0> agstats_dispatcher 472013246 0 17711 9 18321 <0.310.0> - 210373059 0 987 9 1364 <0.293.0> process_killer 197156469 0 1597 9 3194 0.7437.1563> - 144466013 0 75025 9 150050 <0.305.0> - 122348400 0 1597 9 2207 <0.316.0> agstats_snapshotter 101550524 0 2584 9 2961 <0.299.0> agstats_db_mongo 100126516 0 10946 9 39603 <0.331.0> - 78832553 0 610 6 1220 <0.325.0> - 75253366 0 610 9 987 <0.330.0> - 71854078 0 987 9 1597 <0.312.0> - 68821776 0 2584 7 3571 <0.313.0> - 62769588 0 233 9 1220 <0.322.0> - 43514944 0 987 9 1364 .12614.1563> - 24678619 0 46368 9 92736 <0.82.0> timer_server 22905189 0 2584 9 2961 <0.327.0> - 22070516 0 610 9 1220 On Saturday, February 2, 2013 11:01:18 PM UTC+2, Dmitry Kolesnikov wrote: > > Hello, > > This is extremely hard to answer your question not knowing the application > context, what it does, etc. > Just keep in mode that memory report is given in words with is either 4 or > 8 byte depends on system config. This is a reason why linux show diff > value. > > I would start with entop utility and check top processes with high men and > high reductions. > https://github.com/mazenharake/entop > > Please share those views! > > Best Regards, Dmitry > > On Feb 2, 2013, at 10:52 PM, Alexandru Pirvulescu > > wrote: > > Hello, > > If somebody could help me with an issue, I would be grateful. I'm pulling > my hair off trying to figure out why an erlang app keeps growing in memory > usage. The erlang:memory() after 3 days shows this: > > [{total,325732292}, > {processes,54520269}, > {processes_used,54492115}, > * {system,271212023}, > * {atom,932793}, > {atom_used,912937}, > {binary,2348080}, > {code,8288354}, > {ets,487392}, > {maximum,398189416}] > > The interesting part is "system" section. All other data is fine > (processes goes to few MBs during the night as all the started procs are > either hibernated - for long running or they die because of inactivity). > > There are two issues here: > - "system" keeps growing (around 800MB/week) > - OS (Linux) shows a significantly different amount of memory usage > (almost 2x, 610MB resident in this case) > > The only case where I've managed to make "system" grow is by calling > erlang:monitor() multiple times. It grows with ~13K/call. > > Erlang release used is R15B03. > > The questions are: > - is there any hidden function to inspect all the active process monitors? > (as the single place where I use erlang:monitor looks fine. The spawned kid > does its job and dies, so the ref should be garbage collected). > - is there any way to inspect "system" section and find out what it > contains? Looking at memory chunks in process map got me nowhere > > Thank you, > Alex > > > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablo.vb80@REDACTED Sat Feb 2 22:29:20 2013 From: pablo.vb80@REDACTED (Pablo Vieytes) Date: Sat, 2 Feb 2013 22:29:20 +0100 Subject: [erlang-questions] =?windows-1252?q?about_your_answer_to_the_post?= =?windows-1252?q?_=93_Erlang_Jobs=3F=94?= Message-ID: Dear Mr Molinaro, I hope you find this message well. It is about your answer to the post ?[erlang-questions] Erlang Jobs?? in the erlang mailing list. > Anthony Molinaro > Fri Mar 23 19:03:32 CET 2012 > > OpenX is always looking for Erlang engineers. We have offices in > Los Angeles, New York, and London. The office in London is small > and doesn't actually do development, but we might be able to swing > a development position or two there. If there is anyone interested > please forward me a resume. > -Anthony It is a little bit old but maybe you are still interested in receiving resumes. I have been working on job positions programmer, developer and software engineer since 2007 in. Lately I have specialized in functional and concurrent programming (Erlang/OTP). The Erlang project I have been involved, is not in production yet. So I decided to get an "Erlang Certificate" to prove my Erlang knowledge (Certificate number: ESL20130118/01). I also have extensive work experience as software developer with other technologies. I have been working with a wide range of programming languages. That includes, from assembler and low-level C for microcontrollers and embedded system to OOP languages such C++ or Python between others. I have also work with web technologies as Django, jQuery, CSS, XHTML? I also enjoy contributing to opensoftware projects in my free time. Since I started to work with Erlang, I realized I would like to further my career in the designing and developing concurrent fault tolerant systems with high availability. I believe that your company could be a perfect place to achieve it. If you find well you can check my attached CV. In closing I thank you for your time and I hope this message doesn't bother you. Best regards. Pablo Vieytes . -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Pablo Vieytes.pdf Type: application/pdf Size: 89863 bytes Desc: not available URL: From pablo.vb80@REDACTED Sun Feb 3 08:57:54 2013 From: pablo.vb80@REDACTED (Pablo Vieytes) Date: Sun, 3 Feb 2013 08:57:54 +0100 Subject: [erlang-questions] =?windows-1252?q?about_your_answer_to_the_post?= =?windows-1252?q?_=93_Erlang_Jobs=3F=94?= In-Reply-To: References: Message-ID: Hello every body, I really concerned about what I did. I made a BIG mistake. I wanted to send a personal message but I sent it to the whole list. I know the erlang list is not the place for this kind of messages. I hope you can forgive me. I am really sorry, this will never happen agian. please accept my apologies. 2013/2/3 Ulf Angermann > Hi, > > do you live in Essen (NRW)? If so, perhabs we can meet, because i live in > M?lheim an der Ruhr. > > Best Regards, > Ulf > > > 2013/2/2 Pablo Vieytes > >> Dear Mr Molinaro, >> >> I hope you find this message well. It is about your answer to the post >> ?[erlang-questions] Erlang Jobs?? in the erlang mailing list. >> >> > Anthony Molinaro >> > Fri Mar 23 19:03:32 CET 2012 >> > >> > OpenX is always looking for Erlang engineers. We have offices in >> > Los Angeles, New York, and London. The office in London is small >> > and doesn't actually do development, but we might be able to swing >> > a development position or two there. If there is anyone interested >> > please forward me a resume. >> >> > -Anthony >> >> It is a little bit old but maybe you are still interested in receiving >> resumes. >> >> I have been working on job positions programmer, developer and software >> engineer since 2007 in. Lately I have specialized in functional and >> concurrent programming (Erlang/OTP). The Erlang project I have been >> involved, is not in production yet. So I decided to get an "Erlang >> Certificate" to prove my Erlang knowledge (Certificate number: >> ESL20130118/01). >> >> I also have extensive work experience as software developer with other >> technologies. I have been working with a wide range of programming >> languages. That includes, from assembler and low-level C for >> microcontrollers and embedded system to OOP languages such C++ or Python >> between others. I have also work with web technologies as Django, jQuery, >> CSS, XHTML? I also enjoy contributing to opensoftware projects in my free >> time. >> >> Since I started to work with Erlang, I realized I would like to further >> my career in the designing and developing concurrent fault tolerant systems >> with high availability. I believe that your company could be a perfect >> place to achieve it. >> >> If you find well you can check my attached CV. >> >> In closing I thank you for your time and I hope this message doesn't >> bother you. >> >> Best regards. >> >> Pablo Vieytes . >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anotherworldofworld@REDACTED Sun Feb 3 09:24:40 2013 From: anotherworldofworld@REDACTED (Alex toyer) Date: Sun, 3 Feb 2013 14:24:40 +0600 Subject: [erlang-questions] [ANN] Ybot-0.3 chat robot released Message-ID: Hello Erlangers, Ybot-0.3 released. Ybot - chat robot Github Hubot inspired written with erlang/otp. There are many bug fixes and features in new release. There are http supporting added and irc/xmpp over ssl, gtalk supporting and new plugins, command history and many many more. Ybot-0.3 chnagelog: * New wat plugin * Irc ssl supporting --> #16 * New irc configuration options: {use_ssl, true | false} * New irc configuration options: {port, PortNumber :: integer()} * New check-site plugin added * hacker_news plugin improved. Added two modes. * Join to channels with key to irc transport added. * ruby.rb plugin added. Eval simple ruby expression. * ip.py plugin added. Return Ybot external ip address. * stackoverflow search plugin added. * Dynamic loading plugins. * Added command history. * commands_history configuration parameter added. * historycommandlimit_count configuration parameter added. * Added http transport. * Added PASS paraemetr to IRC. * Irc private messages support added --> # 24 * XMPP-muc private message supporting added. * XMPP single user chat supporting added. * XMPP ssl support added. * GTALK support added. * New option xmpp port added. * New option xmpp use_ssl added. * Transport options validating added. * Procfile added for Heroku deploying support. * Added experemental message parser to irc handler --> #23 Pull requests are welcome! -------------- next part -------------- An HTML attachment was scrubbed... URL: From alessandro.sivieri@REDACTED Sun Feb 3 17:55:33 2013 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Sun, 3 Feb 2013 17:55:33 +0100 Subject: [erlang-questions] Function specs Message-ID: Hi all, I was wondering if there is a way to get the function specs (-spec tag) from source files (I guess they are not translated into something in the compiled files). I would like to create an automatic generator of Web forms for an application where Erlang modules are installed and their functions executed (I'm not boring you with the application details), and it would be nice to get the function parameter types "automatically" from the source, provided that developers write the correct signature in the tag. Maybe a parse transform module could be useful? -- Sivieri Alessandro alessandro.sivieri@REDACTED http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sun Feb 3 18:23:02 2013 From: erlang@REDACTED (Joe Armstrong) Date: Sun, 3 Feb 2013 18:23:02 +0100 Subject: [erlang-questions] Function specs In-Reply-To: References: Message-ID: You need to parse the source to get this information - example ... -module(spec). -compile(export_all). test() -> F = "/Users/joe/nobackup/installed/otp_src_R15B/lib/stdlib/src/lists.erl", {ok, L} = epp:parse_file(F, "", ""), [dump(I) || I <- L]. dump({attribute,_,spec,S} = Form) -> Str = erl_pp:form(Form), io:format("~s~n",[Str]); dump(_) -> void. When I ran this I got -spec append(List1, List2) -> List3 when is_subtype(List1, [T]), is_subtype(List2, [T]), is_subtype(List3, [T]), is_subtype(T, term()). ... etc. epp:parse parses the source. erl_pp pretty prints the forms in the parse tree. If you take a look at the forms returned by erl_pp you should get the idea ... Cheers /Joe On Sun, Feb 3, 2013 at 5:55 PM, Alessandro Sivieri < alessandro.sivieri@REDACTED> wrote: > Hi all, > > I was wondering if there is a way to get the function specs (-spec tag) > from source files (I guess they are not translated into something in the > compiled files). > I would like to create an automatic generator of Web forms for an > application where Erlang modules are installed and their functions executed > (I'm not boring you with the application details), and it would be nice to > get the function parameter types "automatically" from the source, provided > that developers write the correct signature in the tag. > Maybe a parse transform module could be useful? > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://sivieri.wordpress.com/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alessandro.sivieri@REDACTED Sun Feb 3 19:27:43 2013 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Sun, 3 Feb 2013 19:27:43 +0100 Subject: [erlang-questions] Function specs In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 6:23 PM, Joe Armstrong wrote: > epp:parse parses the source. erl_pp pretty prints the forms in the parse > tree. > > If you take a look at the forms returned by erl_pp you should get the idea > ... > > Thanks, that's exactly what I was looking for! -- Sivieri Alessandro alessandro.sivieri@REDACTED http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From janos.n.hary@REDACTED Sun Feb 3 23:36:35 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Sun, 3 Feb 2013 23:36:35 +0100 Subject: [erlang-questions] Lager trace_file filter Message-ID: <002701ce025e$ecadfd50$c609f7f0$@gmail.com> Hi All, Is there a way to specify 'orelse' relation between lager:trace_file filters? For example I'd like to allow trace messages coming only from module mod1 and mod2: lager:trace_file("log/trace.log", [{module, mod1}, {module, mod2}]) but it doesn't work. Thanks, Janos -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew@REDACTED Mon Feb 4 06:41:53 2013 From: andrew@REDACTED (Andrew Thompson) Date: Mon, 4 Feb 2013 00:41:53 -0500 Subject: [erlang-questions] Lager trace_file filter In-Reply-To: <002701ce025e$ecadfd50$c609f7f0$@gmail.com> References: <002701ce025e$ecadfd50$c609f7f0$@gmail.com> Message-ID: <20130204054153.GB869@hijacked.us> On Sun, Feb 03, 2013 at 11:36:35PM +0100, Janos Hary wrote: > Is there a way to specify 'orelse' relation between lager:trace_file > filters? > > > > For example I'd like to allow trace messages coming only from module mod1 > and mod2: > > lager:trace_file("log/trace.log", [{module, mod1}, {module, mod2}]) > > but it doesn't work. Unfortunately, there is not. To be fast, trace filters need to be able to 'fail fast' and discard messages they can not possibly match. Thus, I decided that 'orelse' expressions would, at least initially, not be allowed. I'm open to suggestions on making 'orelse' expressions cheap enough to test that we can still fail fast when matching messages. Also, this doesn't implement or, but DeadZen has added some comparator support to lager tracing, which I hope to be able to integrate to master soon: https://github.com/DeadZen/lager/tree/dz-lager-stream Andrew From jeremy@REDACTED Mon Feb 4 07:32:43 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sun, 3 Feb 2013 22:32:43 -0800 Subject: [erlang-questions] Observer as a web application? Message-ID: I am interested in an application like observer that is served through a browser as opposed to a desktop gui application. I've found observer very useful for monitoring cluster health, ensuring there are no netsplits before performing an operation, etc. The only problem is, it's not very convenient to run observer on production (ssh, set up a node, X11 forwarding or ssh tunnel, etc). It would be ideal I think, if observer was a web application that used websocket or something to send data to the browser for easier monitoring on a configurable port with configurable authentication. Two questions: 1. Does something like this already exist? 2. If not, is there any interest in building it? Cheers, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Mon Feb 4 08:04:22 2013 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Mon, 04 Feb 2013 10:04:22 +0300 Subject: [erlang-questions] Observer as a web application? In-Reply-To: References: Message-ID: <510F5D76.40803@gmail.com> Maybe something like this: http://www.metabrew.com/article/bigwig-erlang-webtool-spawnfest On 02/04/2013 09:32 AM, Jeremy Ong wrote: > I am interested in an application like observer that is served through > a browser as opposed to a desktop gui application. > > I've found observer very useful for monitoring cluster health, > ensuring there are no netsplits before performing an operation, etc. > The only problem is, it's not very convenient to run observer on > production (ssh, set up a node, X11 forwarding or ssh tunnel, etc). > > It would be ideal I think, if observer was a web application that used > websocket or something to send data to the browser for easier > monitoring on a configurable port with configurable authentication. > > Two questions: > > 1. Does something like this already exist? > 2. If not, is there any interest in building it? > > Cheers, > Jeremy > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Dmitry Klionsky -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Mon Feb 4 08:12:12 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sun, 3 Feb 2013 23:12:12 -0800 Subject: [erlang-questions] Observer as a web application? In-Reply-To: <510F5D76.40803@gmail.com> References: <510F5D76.40803@gmail.com> Message-ID: Bigwig is certainly in the spirit of things but it leverages Appmon so will not work for newer Erlang versions (appmon has been superseded by observer; at least, that's what the documentation says). On Sun, Feb 3, 2013 at 11:04 PM, Dmitry Klionsky wrote: > Maybe something like this: > http://www.metabrew.com/article/bigwig-erlang-webtool-spawnfest > > > On 02/04/2013 09:32 AM, Jeremy Ong wrote: > > I am interested in an application like observer that is served through a > browser as opposed to a desktop gui application. > > I've found observer very useful for monitoring cluster health, ensuring > there are no netsplits before performing an operation, etc. The only > problem is, it's not very convenient to run observer on production (ssh, > set up a node, X11 forwarding or ssh tunnel, etc). > > It would be ideal I think, if observer was a web application that used > websocket or something to send data to the browser for easier monitoring on > a configurable port with configurable authentication. > > Two questions: > > 1. Does something like this already exist? > 2. If not, is there any interest in building it? > > Cheers, > Jeremy > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Best regards, > Dmitry Klionsky > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Mon Feb 4 09:01:10 2013 From: dangud@REDACTED (Dan Gudmundsson) Date: Mon, 4 Feb 2013 09:01:10 +0100 Subject: [erlang-questions] Observer as a web application? In-Reply-To: References: <510F5D76.40803@gmail.com> Message-ID: I believe it should still work, I use the appmon backend for observer too, and moved (in R16) that backend module to runtime_tools where it belongs. /Dan On Mon, Feb 4, 2013 at 8:12 AM, Jeremy Ong wrote: > Bigwig is certainly in the spirit of things but it leverages Appmon so will > not work for newer Erlang versions (appmon has been superseded by observer; > at least, that's what the documentation says). > > > On Sun, Feb 3, 2013 at 11:04 PM, Dmitry Klionsky > wrote: >> >> Maybe something like this: >> http://www.metabrew.com/article/bigwig-erlang-webtool-spawnfest >> >> >> On 02/04/2013 09:32 AM, Jeremy Ong wrote: >> >> I am interested in an application like observer that is served through a >> browser as opposed to a desktop gui application. >> >> I've found observer very useful for monitoring cluster health, ensuring >> there are no netsplits before performing an operation, etc. The only problem >> is, it's not very convenient to run observer on production (ssh, set up a >> node, X11 forwarding or ssh tunnel, etc). >> >> It would be ideal I think, if observer was a web application that used >> websocket or something to send data to the browser for easier monitoring on >> a configurable port with configurable authentication. >> >> Two questions: >> >> 1. Does something like this already exist? >> 2. If not, is there any interest in building it? >> >> Cheers, >> Jeremy >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> >> -- >> Best regards, >> Dmitry Klionsky > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From jeremy@REDACTED Mon Feb 4 09:08:47 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Mon, 4 Feb 2013 00:08:47 -0800 Subject: [erlang-questions] Observer as a web application? In-Reply-To: References: <510F5D76.40803@gmail.com> Message-ID: Ah I see. I had my doubts because it didn't seem like the project has been in active development (the required OTP version is R14) but I'll give it a shot. If I wanted to extract process tree data on my own to parse and display, would you recommend I use process_info in appmon? There isn't documentation for it but the implementation is pretty transparent. On Mon, Feb 4, 2013 at 12:01 AM, Dan Gudmundsson wrote: > I believe it should still work, I use the appmon backend for observer too, > and moved (in R16) that backend module to runtime_tools where it belongs. > > /Dan > > On Mon, Feb 4, 2013 at 8:12 AM, Jeremy Ong wrote: > > Bigwig is certainly in the spirit of things but it leverages Appmon so > will > > not work for newer Erlang versions (appmon has been superseded by > observer; > > at least, that's what the documentation says). > > > > > > On Sun, Feb 3, 2013 at 11:04 PM, Dmitry Klionsky > > wrote: > >> > >> Maybe something like this: > >> http://www.metabrew.com/article/bigwig-erlang-webtool-spawnfest > >> > >> > >> On 02/04/2013 09:32 AM, Jeremy Ong wrote: > >> > >> I am interested in an application like observer that is served through a > >> browser as opposed to a desktop gui application. > >> > >> I've found observer very useful for monitoring cluster health, ensuring > >> there are no netsplits before performing an operation, etc. The only > problem > >> is, it's not very convenient to run observer on production (ssh, set up > a > >> node, X11 forwarding or ssh tunnel, etc). > >> > >> It would be ideal I think, if observer was a web application that used > >> websocket or something to send data to the browser for easier > monitoring on > >> a configurable port with configurable authentication. > >> > >> Two questions: > >> > >> 1. Does something like this already exist? > >> 2. If not, is there any interest in building it? > >> > >> Cheers, > >> Jeremy > >> > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > >> > >> > >> -- > >> Best regards, > >> Dmitry Klionsky > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andra.dinu@REDACTED Mon Feb 4 13:13:05 2013 From: andra.dinu@REDACTED (Andra Dinu) Date: Mon, 4 Feb 2013 12:13:05 +0000 (GMT) Subject: [erlang-questions] Complete programme for Erlang Factory SF Bay 2013 In-Reply-To: <156219662.3351856.1359979740611.JavaMail.root@erlang-solutions.com> Message-ID: <291522492.3351923.1359979984983.JavaMail.root@erlang-solutions.com> Hi, The complete list of speakers for Erlang Factory SF Bay Area 2013 is up, and it?s the best up to date. On 21-22 March 50 speakers from companies such as WhatsApp, Nokia, Ericsson, Rakuten, HP, Microsoft, OpsCode, ESPN, Linden Lab, OpenX, Infoblox, Erlang Solutions and Concurix will showcase tools, libraries and experiences in building massively concurrent distributed systems. Full programme here: http://www.erlang-factory.com/conference/SFBay2013/speakers The keynote speakers are Microsoft language expert Erik Meijer and Mnesia creator Claes Wikstr?m. Other speakers include Erlang co-inventor Robert Virding, ?Learn you some Erlang? author Fred Hebert, Cowboy creator Loic Hoguin, Chicago Boss creator Evan Miller, Feuerlabs co-founder Ulf Wiger and Elixir Author Jose Valim. Talks will cover areas such as Big Data, Scalability and Multi-core, and use cases in Gaming, Social Media, Messaging, Web-development, Mobile, Telecoms and Databases. They will evaluate tools such as RabbitMQ, MongooseIM, Ejabberd and compare NoSQL databases such as Couch DB, Riak and LeoFS. The Conference will take place at the Marines? Memorial Club and Hotel in Union Square. A limited number of rooms is available at a secure rate, so remember to book as soon as possible: http://www.erlang-factory.com/conference/SFBay2013/venue . See you at the Erlang Factory SF Bay! -- Andra Dinu Marketing executive ERLANG SOLUTIONS LTD New Loom House 101 Back Church Lane London, E1 1LU United Kingdom Tel +44(0)2076550344 Mob +44(0)7983484387 www.erlang-solutions.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico.kruber@REDACTED Mon Feb 4 17:34:32 2013 From: nico.kruber@REDACTED (Nico Kruber) Date: Mon, 04 Feb 2013 17:34:32 +0100 Subject: [erlang-questions] edoc broken in current master / R16A! In-Reply-To: <3788796.M1GmZlaJoO@csr-pc40.zib.de> References: <4159396.DFkII5edO4@csr-pc40.zib.de> <3788796.M1GmZlaJoO@csr-pc40.zib.de> Message-ID: <5789034.tP9lXoejRZ@csr-pc40.zib.de> On Wednesday 30 Jan 2013 15:35:41 you wrote: > On Wednesday 30 Jan 2013 15:34:26 you wrote: > > we regularly check that Scalaris builds on master and since the checks of > > 11 Jan 2013 6am CET, creating the documentation with edoc fails. > > We don't use packages, although we do put files into sub-folders - anyway, > > I guess a function_clause should not be returned to the user at any point > > (see below) > > > > svn co http://scalaris.googlecode.com/svn/trunk/ scalaris > > cd scalaris > > make doc > > ------------------------------ > > /home/buildbot/apps/erlang-dev/bin/erl -noshell -run edoc_run application > > "'scalaris'" \ > > > > '"."' '[{def, {vsn, "0.5.0+svn"}}, no_packages, {preprocess, true}, > > > > {includes, ["include", "contrib/yaws/include", > > "contrib/log4erl/include"]}, > > {sort_functions, false}, {pretty_printer, erl_pp}, {macros, > > [{tid_not_builtin, true}, {with_export_type_support, true}, > > {have_cthooks_support, true}, {have_callback_support, true}]}]' > > edoc: edoc terminated abnormally: {function_clause, > > > > [{edoc_lib,find_sources_3, > > > > [["random_bias.erl",[...]|...], > > > > "./src/rrepair",rrepair|...], > > > > [{file,...},{...}]}, > > > > {edoc_lib,find_sources_2,5,[{...}|...]}, > > {edoc_lib,'-find_sources_3/6-lc$^0/1-0-', > > 5, > > > > [...]}, > > > > {edoc_lib,find_sources_2,5,...}, > > {edoc_lib,find_sources_1,...}, > > {edoc_lib,...}, > > {...}|...]}. > > > > make: *** [.make-doc] Error 1 > > ------------------------------ > > maybe this commit is related > https://github.com/erlang/otp/commit/beb660aa59a8d9cd7f65b8b91dc5dd9bc74015e > 0 > > Nico I just tested with R16A and the problem is the same From solomon.wzs@REDACTED Tue Feb 5 03:42:16 2013 From: solomon.wzs@REDACTED (Solomon) Date: Tue, 5 Feb 2013 10:42:16 +0800 Subject: [erlang-questions] How to see which processes used ets table? Message-ID: I checked system info with erlang:memory/1 and found the total amount of memory allocated for ets tables was about 5GB. Then I sum the memory size return by ets:i/0 and it was about 600MB. So which processes used ets table and did not return by ets:i/0? -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Tue Feb 5 05:05:55 2013 From: comptekki@REDACTED (Wes James) Date: Mon, 4 Feb 2013 21:05:55 -0700 Subject: [erlang-questions] stunnel and erlang Message-ID: Has anyone used stunnel to send messages between erlang nodes? I have asked in the past about secure communication between erlang nodes and there doesn't seem to be a straight forward way to do this. Would stunnel work? I have successfully set up stunnel to do pop3 on port 110 to gmails ssl port, but it seems that erlang has more than one port it needs to deal with (epmd and others). Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From romain.lenglet@REDACTED Tue Feb 5 05:36:18 2013 From: romain.lenglet@REDACTED (Romain Lenglet) Date: Mon, 4 Feb 2013 20:36:18 -0800 Subject: [erlang-questions] stunnel and erlang In-Reply-To: References: Message-ID: <920A3E5E-E4C8-49EB-97F2-E9A390E45E34@berabera.info> Hi Wes, Have you tried using the "-proto_dist inet_tls" option? http://www.erlang.org/doc/apps/ssl/ssl_distribution.html -- Romain Lenglet On Feb 4, 2013, at 8:05 PM, Wes James wrote: > Has anyone used stunnel to send messages between erlang nodes? I have asked in the past about secure communication between erlang nodes and there doesn't seem to be a straight forward way to do this. Would stunnel work? I have successfully set up stunnel to do pop3 on port 110 to gmails ssl port, but it seems that erlang has more than one port it needs to deal with (epmd and others). > > Thanks, > > Wes > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From comptekki@REDACTED Tue Feb 5 06:07:06 2013 From: comptekki@REDACTED (Wes James) Date: Mon, 4 Feb 2013 22:07:06 -0700 Subject: [erlang-questions] stunnel and erlang In-Reply-To: <920A3E5E-E4C8-49EB-97F2-E9A390E45E34@berabera.info> References: <920A3E5E-E4C8-49EB-97F2-E9A390E45E34@berabera.info> Message-ID: No. Nobody mentioned it last time. I'll look into it. Thanks! Wes On Feb 4, 2013 9:36 PM, "Romain Lenglet" wrote: > Hi Wes, > Have you tried using the "-proto_dist inet_tls" option? > http://www.erlang.org/doc/apps/ssl/ssl_distribution.html > -- > Romain Lenglet > > > On Feb 4, 2013, at 8:05 PM, Wes James wrote: > > > Has anyone used stunnel to send messages between erlang nodes? I have > asked in the past about secure communication between erlang nodes and there > doesn't seem to be a straight forward way to do this. Would stunnel work? > I have successfully set up stunnel to do pop3 on port 110 to gmails ssl > port, but it seems that erlang has more than one port it needs to deal with > (epmd and others). > > > > Thanks, > > > > Wes > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Tue Feb 5 07:57:35 2013 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 04 Feb 2013 22:57:35 -0800 Subject: [erlang-questions] [ANN] CloudI 1.1.1 Released! Message-ID: <5110AD5F.9060601@gmail.com> Download 1.1.1 from http://sourceforge.net/projects/cloudi/files/latest/download What is CloudI? CloudI (http://www.cloudi.org/) is meant to be a "universal integrator" using an Erlang core, with its actor model, to pursue efficiency, fault-tolerance and scalability. The CloudI API provides a minimal interface to communicate among actors that are called services or jobs (they are services, ran as if they are jobs, in parallel), so programming language agnostic, database agnostic, and messaging bus agnostic integration can occur. CloudI currently integrates with the programming languages Erlang, C/C++, Java, Python, and Ruby, the databases PostgreSQL, MySQL, couchdb, memcached, and tokyotyrant, the messaging bus ZeroMQ and the internal CloudI messaging. HTTP is also supported for easy integration with cowboy and misultin. If anyone is willing to get involved, don't hesitate to contact me or start looking at the code. What is CloudI? summary: (short answer) "An application server that efficiently integrates with many languages, many databases, and many messaging buses in a way that is both scalable and fault-tolerant." (shorter answer) "A rock-solid transaction processing system for flexible software development." (shortest answer) "A Cloud at the lowest level." About the release: If you have had problems installing CloudI in the past, you should no longer have issues. I have been working through build and install issues on Ubuntu and OSX, but no issues currently remain. If you run into any problems, please don't hesitate to report them (here https://github.com/okeuday/CloudI/issues/new ). The latest news is below: * use patterns instead of prefixes for ACL strings * add the following destination refresh methods: immediate_furthest, lazy_furthest, immediate_local, lazy_local, immediate_remote, and lazy_remote * build system fixes and documentation fixes for installation * fix CloudI API uuids to be completely standard v1 * fix external service queueing during initialization Please mention any problems, issues, or ideas! Thanks, Michael From gustav.simonsson@REDACTED Tue Feb 5 08:51:16 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Tue, 5 Feb 2013 08:51:16 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: Note that erlang:memory/1 gives memory in bytes, whereas ets:i/0 gives it in words. On a 64-bit system 625M words would be 5GB, so the numbers you get might be correct. Try summing the memory from ets:i/0 up exact and compare it to erlang:memory(ets). Cheers, Gustav Simonsson On Tue, Feb 5, 2013 at 3:42 AM, Solomon wrote: > I checked system info with erlang:memory/1 and found the total amount of > memory allocated for ets tables was about 5GB. > > Then I sum the memory size return by ets:i/0 and it was about 600MB. > > So which processes used ets table and did not return by ets:i/0? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattevans123@REDACTED Tue Feb 5 15:59:36 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Tue, 5 Feb 2013 09:59:36 -0500 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: , Message-ID: Hi, I don't wish to thread-jack, but I am anyway ;-) This pertains to a feature I would like to see in Erlang. The ability to set an optional "memory limit" when a process and ETS table is created (and maybe a global optional per-process limit when the VM is started). I've seen a few cases where, due to software bugs, a process size grows and grows; unfortunately as things stand today the result is your entire VM crashing - hopefully leaving you with a crash_dump. Having such a limit could cause the process to terminate (producing a OOM crash report in erlang.log) and the crashing process could be handled with supervisor rules. Even better you can envisage setting the limits artificially low during testing to catch these types of bugs early on. Cheers Matt Date: Tue, 5 Feb 2013 08:51:16 +0100 From: gustav.simonsson@REDACTED To: solomon.wzs@REDACTED CC: erlang-questions@REDACTED Subject: Re: [erlang-questions] How to see which processes used ets table? Note that erlang:memory/1 gives memory in bytes, whereas ets:i/0 gives it in words. On a 64-bit system 625M words would be 5GB, so the numbers you get might be correct. Try summing the memory from ets:i/0 up exact and compare it to erlang:memory(ets). Cheers, Gustav Simonsson On Tue, Feb 5, 2013 at 3:42 AM, Solomon wrote: I checked system info with erlang:memory/1 and found the total amount of memory allocated for ets tables was about 5GB. Then I sum the memory size return by ets:i/0 and it was about 600MB. So which processes used ets table and did not return by ets:i/0? _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From msegalis@REDACTED Tue Feb 5 16:06:49 2013 From: msegalis@REDACTED (Morgan Segalis) Date: Tue, 5 Feb 2013 16:06:49 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: <58A4CD5B-F157-434B-8E63-63CAFFF9B4F0@gmail.com> I have crafted a little function for process monitoring purpose, it might help you, if you're memory issue is process related... Hope it will help [CODE] info() -> get_info(erlang:processes(), dict:new()). get_info([], Dict) -> io:fwrite("Dict: ~p~n", [Dict]); get_info([P|L], Dict) -> case erlang:process_info(P, dictionary) of {dictionary, Info} -> Initial_call = get_dict('$initial_call', Info), Ancestor = get_dict('$ancestors', Info), case dict:find({Initial_call, Ancestor}, Dict) of {ok, Nb} -> IncrNb = Nb + 1; error -> IncrNb = 1 end, Dict2 = dict:store({Initial_call, Ancestor}, IncrNb, Dict), get_info(L, Dict2); undefined -> case dict:find(undefined, Dict) of {ok, Nb} -> IncrNb = Nb + 1; error -> IncrNb = 1 end, Dict2 = dict:store(undefined, IncrNb, Dict), get_info(L, Dict2) end. [/CODE] Le 5 f?vr. 2013 ? 03:42, Solomon a ?crit : > I checked system info with erlang:memory/1 and found the total amount of memory allocated for ets tables was about 5GB. > > Then I sum the memory size return by ets:i/0 and it was about 600MB. > > So which processes used ets table and did not return by ets:i/0? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From derekbrown121@REDACTED Tue Feb 5 18:10:25 2013 From: derekbrown121@REDACTED (Derek Brown) Date: Tue, 5 Feb 2013 12:10:25 -0500 Subject: [erlang-questions] Evening School of Erlang Message-ID: I'm curious about the classes described at https://www.erlang-solutions.com/services/evening-school-erlang. One, any impressions, and two, any plans for the New York City version, as alluded to at that page. Thanks, Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From derekbrown121@REDACTED Tue Feb 5 18:14:54 2013 From: derekbrown121@REDACTED (Derek Brown) Date: Tue, 5 Feb 2013 12:14:54 -0500 Subject: [erlang-questions] Evening School of Erlang (with proper link) Message-ID: I'm curious about the classes described at https://www.erlang-solutions.com/services/evening-school-erlang . One, any impressions, and two, any plans for the New York City version, as alluded to at that page. Thanks, Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Tue Feb 5 18:33:53 2013 From: raould@REDACTED (Raoul Duke) Date: Tue, 5 Feb 2013 09:33:53 -0800 Subject: [erlang-questions] Evening School of Erlang In-Reply-To: References: Message-ID: and 3 please do them again in San Jose so i can attend the next time! soon! :-) From emofine@REDACTED Tue Feb 5 20:45:50 2013 From: emofine@REDACTED (Edwin Fine) Date: Tue, 05 Feb 2013 14:45:50 -0500 Subject: [erlang-questions] OTP R16A EPMD IPv6 support In-Reply-To: References: Message-ID: <5111616E.6030002@gmail.com> Although it does not seem to be in the release notes, it appears that R16 does support epmd using IPv6 addresses, but it is not enabled by default. To enable it, it seems that you need to export CPPFLAGS=-DEPMD6 before running configure, assuming that the building system provides inet_pton. After building R16A, to assign some IPv6 and IPv4-on-IPv6 addresses, I got this to work: $ epmd -daemon -address ::FFFF:0.0.0.0,fded:1486:2e44::1:2:1 $ netstat -tnl|grep 4369 tcp6 0 0 ::1:4369 :::* LISTEN tcp6 0 0 fded:1486:2e44::1::4369 :::* LISTEN tcp6 0 0 0.0.0.0:4369 :::* LISTEN It does not look as if the Erlang Solutions build of R16A has the IPv6 epmd support enabled on the Ubuntu 11.04 64-bit package I installed. My questions are: 1. Did I understand this correctly? 2. Why is EPMD IPv6 support not enabled by default using autoconfig? 3. Is the EPMD IPv6 support intended to be used in this release, or is it disabled for some reason? Ed From bryan@REDACTED Tue Feb 5 21:14:28 2013 From: bryan@REDACTED (Bryan Hughes) Date: Tue, 05 Feb 2013 12:14:28 -0800 Subject: [erlang-questions] Erlang + Facebook Message-ID: <51116824.9010108@go-factory.net> Though some of you might find this interesting if you havent already seen it... http://techcrunch.com/2013/02/04/opscode-is-facebooks-answer-for-scaling-its-carl-sagan-size-deployments/ -- Bryan Hughes *Go Factory* http://www.go-factory.net /"Art is never finished, only abandoned. - Leonardo da Vinci"/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From amindfv@REDACTED Tue Feb 5 23:03:40 2013 From: amindfv@REDACTED (amindfv@REDACTED) Date: Tue, 5 Feb 2013 17:03:40 -0500 Subject: [erlang-questions] Evening School of Erlang In-Reply-To: References: Message-ID: <5AF71FDC-BA16-4055-832A-6B852AB4E714@gmail.com> San Jose hasn't happened yet - I believe you can still get tickets. Tom Sent from my iPhone On Feb 5, 2013, at 12:33 PM, Raoul Duke wrote: > and 3 please do them again in San Jose so i can attend the next time! soon! :-) > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From raould@REDACTED Tue Feb 5 23:04:16 2013 From: raould@REDACTED (Raoul Duke) Date: Tue, 5 Feb 2013 14:04:16 -0800 Subject: [erlang-questions] Evening School of Erlang In-Reply-To: <5AF71FDC-BA16-4055-832A-6B852AB4E714@gmail.com> References: <5AF71FDC-BA16-4055-832A-6B852AB4E714@gmail.com> Message-ID: > San Jose hasn't happened yet - I believe you can still get tickets. i won't be around for it this feb :-( From francesco@REDACTED Tue Feb 5 23:54:26 2013 From: francesco@REDACTED (Francesco Cesarini) Date: Tue, 05 Feb 2013 22:54:26 +0000 Subject: [erlang-questions] Evening School of Erlang In-Reply-To: References: <5AF71FDC-BA16-4055-832A-6B852AB4E714@gmail.com> Message-ID: <51118DA2.1090103@erlang-solutions.com> If we get enough attendees, there will be a second round in about 4 - 6 months. It all depends on demand and local companies recruiting for Erlang developers willing to sponsor and how many of the attendees find Erlang jobs. San Mateo/Palo Alto might also be an option for you. San Francisco classes finished last week and a new round of dates will soon be announced. In regards to New York, we are looking at the latter half of this year. Derek - if you are able to help us secure a venue, it would be great! Other locations currently include Chicago, DC, Boston, Pasadena and Santa Monica. Francesco On 05/02/2013 22:04, Raoul Duke wrote: >> San Jose hasn't happened yet - I believe you can still get tickets. > i won't be around for it this feb :-( > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Erlang Solutions Ltd. http://www.erlang-solutions.com From robert.virding@REDACTED Wed Feb 6 01:05:19 2013 From: robert.virding@REDACTED (Robert Virding) Date: Wed, 6 Feb 2013 00:05:19 +0000 (GMT) Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: <58A4CD5B-F157-434B-8E63-63CAFFF9B4F0@gmail.com> Message-ID: <51805448.3506909.1360109119321.JavaMail.root@erlang-solutions.com> You have to be a little careful when doing process_info/1/2 on unknown processes. For example if you are worried about the size of message queues then doing process_info(Pid, messages) is not a good idea as it will copy the whole message queue. As in this case using dictionary as it will also copy the dictionary. Process_info() can be very useful but it is one of those BIFs with which you have to exercise care. As is processes() if you have many processes. Robert > From: "Morgan Segalis" > > I have crafted a little function for process monitoring purpose, it > might help you, if you're memory issue is process related... > > Hope it will help > > [CODE] > info() -> > get_info(erlang:processes(), dict:new()). > > get_info([], Dict) -> > io:fwrite("Dict: ~p~n", [Dict]); > get_info([P|L], Dict) -> > case erlang:process_info(P, dictionary) of > {dictionary, Info} -> > Initial_call = get_dict('$initial_call', Info), > Ancestor = get_dict('$ancestors', Info), > case dict:find({Initial_call, Ancestor}, Dict) of > {ok, Nb} -> > IncrNb = Nb + 1; > error -> > IncrNb = 1 > end, > Dict2 = dict:store({Initial_call, Ancestor}, IncrNb, Dict), > get_info(L, Dict2); > undefined -> > case dict:find(undefined, Dict) of > {ok, Nb} -> > IncrNb = Nb + 1; > error -> > IncrNb = 1 > end, > Dict2 = dict:store(undefined, IncrNb, Dict), > get_info(L, Dict2) > end. > [/CODE] > > Le 5 f?vr. 2013 ? 03:42, Solomon a ?crit : > > > I checked system info with erlang:memory/1 and found the total > > amount of memory allocated for ets tables was about 5GB. > > > > Then I sum the memory size return by ets:i/0 and it was about > > 600MB. > > > > So which processes used ets table and did not return by ets:i/0? > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From msegalis@REDACTED Wed Feb 6 01:18:56 2013 From: msegalis@REDACTED (Morgan Segalis) Date: Wed, 6 Feb 2013 01:18:56 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: <51805448.3506909.1360109119321.JavaMail.root@erlang-solutions.com> References: <51805448.3506909.1360109119321.JavaMail.root@erlang-solutions.com> Message-ID: <0980D531-BC13-4E04-A3DB-3016065EF2EE@gmail.com> Yeah of course, not to use on a production server, I forgot to mention it. Le 6 f?vr. 2013 ? 01:05, Robert Virding a ?crit : > You have to be a little careful when doing process_info/1/2 on unknown processes. For example if you are worried about the size of message queues then doing process_info(Pid, messages) is not a good idea as it will copy the whole message queue. As in this case using dictionary as it will also copy the dictionary. Process_info() can be very useful but it is one of those BIFs with which you have to exercise care. As is processes() if you have many processes. > > Robert > >> From: "Morgan Segalis" >> >> I have crafted a little function for process monitoring purpose, it >> might help you, if you're memory issue is process related... >> >> Hope it will help >> >> [CODE] >> info() -> >> get_info(erlang:processes(), dict:new()). >> >> get_info([], Dict) -> >> io:fwrite("Dict: ~p~n", [Dict]); >> get_info([P|L], Dict) -> >> case erlang:process_info(P, dictionary) of >> {dictionary, Info} -> >> Initial_call = get_dict('$initial_call', Info), >> Ancestor = get_dict('$ancestors', Info), >> case dict:find({Initial_call, Ancestor}, Dict) of >> {ok, Nb} -> >> IncrNb = Nb + 1; >> error -> >> IncrNb = 1 >> end, >> Dict2 = dict:store({Initial_call, Ancestor}, IncrNb, Dict), >> get_info(L, Dict2); >> undefined -> >> case dict:find(undefined, Dict) of >> {ok, Nb} -> >> IncrNb = Nb + 1; >> error -> >> IncrNb = 1 >> end, >> Dict2 = dict:store(undefined, IncrNb, Dict), >> get_info(L, Dict2) >> end. >> [/CODE] >> >> Le 5 f?vr. 2013 ? 03:42, Solomon a ?crit : >> >>> I checked system info with erlang:memory/1 and found the total >>> amount of memory allocated for ets tables was about 5GB. >>> >>> Then I sum the memory size return by ets:i/0 and it was about >>> 600MB. >>> >>> So which processes used ets table and did not return by ets:i/0? >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> From rapsey@REDACTED Wed Feb 6 07:55:37 2013 From: rapsey@REDACTED (Rapsey) Date: Wed, 6 Feb 2013 07:55:37 +0100 Subject: [erlang-questions] [ANN] CloudI 1.1.1 Released! In-Reply-To: <5110AD5F.9060601@gmail.com> References: <5110AD5F.9060601@gmail.com> Message-ID: In my humble opinion you really need some sort of quickstart or walk through guide on how to use it. Many people like to learn by examples and there is nothing of this sort there. I read the faq, the github readme, this email and I still have no idea how to use it or have any degree of certainty what exactly it is for. Sergej On Tue, Feb 5, 2013 at 7:57 AM, Michael Truog wrote: > Download 1.1.1 from > http://sourceforge.net/projects/cloudi/files/latest/download > > What is CloudI? > CloudI (http://www.cloudi.org/) is meant to be a "universal integrator" > using an Erlang core, with its actor model, to pursue efficiency, > fault-tolerance and scalability. The CloudI API provides a minimal > interface to communicate among actors that are called services or jobs > (they are services, ran as if they are jobs, in parallel), so programming > language agnostic, database agnostic, and messaging bus agnostic > integration can occur. CloudI currently integrates with the programming > languages Erlang, C/C++, Java, Python, and Ruby, the databases PostgreSQL, > MySQL, couchdb, memcached, and tokyotyrant, the messaging bus ZeroMQ and > the internal CloudI messaging. HTTP is also supported for easy integration > with cowboy and misultin. If anyone is willing to get involved, don't > hesitate to contact me or start looking at the code. > > What is CloudI? summary: > (short answer) "An application server that efficiently integrates with > many languages, many databases, and many messaging buses in a way that is > both scalable and fault-tolerant." > (shorter answer) "A rock-solid transaction processing system for flexible > software development." > (shortest answer) "A Cloud at the lowest level." > > About the release: > If you have had problems installing CloudI in the past, you should no > longer have issues. I have been working through build and install issues > on Ubuntu and OSX, but no issues currently remain. If you run into any > problems, please don't hesitate to report them (here > https://github.com/okeuday/CloudI/issues/new ). > > The latest news is below: > * use patterns instead of prefixes for ACL strings > * add the following destination refresh methods: > immediate_furthest, lazy_furthest, > immediate_local, lazy_local, > immediate_remote, and lazy_remote > * build system fixes and documentation fixes for installation > * fix CloudI API uuids to be completely standard v1 > * fix external service queueing during initialization > > Please mention any problems, issues, or ideas! > > Thanks, > Michael > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Wed Feb 6 08:05:14 2013 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 05 Feb 2013 23:05:14 -0800 Subject: [erlang-questions] [ANN] CloudI 1.1.1 Released! In-Reply-To: References: <5110AD5F.9060601@gmail.com> Message-ID: <511200AA.5020600@gmail.com> I know there isn't a guickstart or user tutorial yet, and I have been putting that off. The closest thing that currently exists is the tests, which do serve as examples (https://github.com/okeuday/CloudI/tree/master/src/tests), but I understand there should be a user tutorial that provides an explanation and context for creating a service, so that people don't get driven away by too much time spent. I will do that for the next release to make sure that gets done before I add more features/services (you are not the first person to complain about this). Thanks, Michael On 02/05/2013 10:55 PM, Rapsey wrote: > In my humble opinion you really need some sort of quickstart or walk through guide on how to use it. Many people like to learn by examples and there is nothing of this sort there. I read the faq, the github readme, this email and I still have no idea how to use it or have any degree of certainty what exactly it is for. > > > Sergej > > On Tue, Feb 5, 2013 at 7:57 AM, Michael Truog > wrote: > > Download 1.1.1 from http://sourceforge.net/projects/cloudi/files/latest/download > > What is CloudI? > CloudI (http://www.cloudi.org/) is meant to be a "universal integrator" using an Erlang core, with its actor model, to pursue efficiency, fault-tolerance and scalability. The CloudI API provides a minimal interface to communicate among actors that are called services or jobs (they are services, ran as if they are jobs, in parallel), so programming language agnostic, database agnostic, and messaging bus agnostic integration can occur. CloudI currently integrates with the programming languages Erlang, C/C++, Java, Python, and Ruby, the databases PostgreSQL, MySQL, couchdb, memcached, and tokyotyrant, the messaging bus ZeroMQ and the internal CloudI messaging. HTTP is also supported for easy integration with cowboy and misultin. If anyone is willing to get involved, don't hesitate to contact me or start looking at the code. > > What is CloudI? summary: > (short answer) "An application server that efficiently integrates with many languages, many databases, and many messaging buses in a way that is both scalable and fault-tolerant." > (shorter answer) "A rock-solid transaction processing system for flexible software development." > (shortest answer) "A Cloud at the lowest level." > > About the release: > If you have had problems installing CloudI in the past, you should no longer have issues. I have been working through build and install issues on Ubuntu and OSX, but no issues currently remain. If you run into any problems, please don't hesitate to report them (here https://github.com/okeuday/CloudI/issues/new ). > > The latest news is below: > * use patterns instead of prefixes for ACL strings > * add the following destination refresh methods: > immediate_furthest, lazy_furthest, > immediate_local, lazy_local, > immediate_remote, and lazy_remote > * build system fixes and documentation fixes for installation > * fix CloudI API uuids to be completely standard v1 > * fix external service queueing during initialization > > Please mention any problems, issues, or ideas! > > Thanks, > Michael > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fithis2001@REDACTED Wed Feb 6 15:28:10 2013 From: fithis2001@REDACTED (Vasileios Anagnostopoulos) Date: Wed, 6 Feb 2013 16:28:10 +0200 Subject: [erlang-questions] Slimmed down version Message-ID: Hi, Erlang/OTP is a big package to download and build. Is there any option to provide a slimmed down version? Only the bare minimum? I think that OTP must become more modular and rely on an external package system. Regards Vasileios -- Dr. Vasileios Anagnostopoulos (MSc,PhD) Researcher/Developer ICCS/NTUA 9 Heroon Polytechneiou Str., Zografou 15773 Athens,Greece T (+30) 2107723404 M (+30) 6936935388 E vanag@REDACTED>www.ntua.gr -------------- next part -------------- An HTML attachment was scrubbed... URL: From dronnikov@REDACTED Wed Feb 6 15:30:44 2013 From: dronnikov@REDACTED (Vladimir Dronnikov) Date: Wed, 6 Feb 2013 18:30:44 +0400 Subject: [erlang-questions] Slimmed down version In-Reply-To: References: Message-ID: Hi! Of particular need imho is to make wx* optional, as it pulls X and supporting libraries in. --Vladimir On Wed, Feb 6, 2013 at 6:28 PM, Vasileios Anagnostopoulos < fithis2001@REDACTED> wrote: > > Hi, > > Erlang/OTP is a big package to download and build. Is there any option to > provide a slimmed down version? Only the bare minimum? > I think that OTP must become more modular and rely on an external package > system. > > > Regards > Vasileios > -- > > Dr. Vasileios Anagnostopoulos (MSc,PhD) > Researcher/Developer > ICCS/NTUA > 9 Heroon Polytechneiou Str., Zografou > 15773 Athens,Greece > T (+30) 2107723404 > M (+30) 6936935388 > E vanag@REDACTED>www.ntua.gr > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Feb 6 16:07:30 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Wed, 6 Feb 2013 15:07:30 +0000 Subject: [erlang-questions] Slimmed down version In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 2:28 PM, Vasileios Anagnostopoulos wrote: > > Hi, > > Erlang/OTP is a big package to download and build. Is there any option to > provide a slimmed down version? Only the bare minimum? > I think that OTP must become more modular and rely on an external package > system. There is a mechanism for that in debian/ubuntu: motiejus@REDACTED:~$ apt-get install -s erlang-base NOTE: This is only a simulation! Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: erlang-crypto erlang-syntax-tools libsctp1 lksctp-tools Suggested packages: erlang-tools erlang erlang-manpages erlang-doc The following NEW packages will be installed: erlang-base erlang-crypto erlang-syntax-tools libsctp1 lksctp-tools 0 upgraded, 5 newly installed, 0 to remove and 10 not upgraded. Inst erlang-base (1:14.b.4-dfsg-1ubuntu1 Ubuntu:12.04/precise [amd64]) Inst erlang-crypto (1:14.b.4-dfsg-1ubuntu1 Ubuntu:12.04/precise [amd64]) Inst erlang-syntax-tools (1:14.b.4-dfsg-1ubuntu1 Ubuntu:12.04/precise [amd64]) Inst libsctp1 (1.0.11+dfsg-2 Ubuntu:12.04/precise [amd64]) Inst lksctp-tools (1.0.11+dfsg-2 Ubuntu:12.04/precise [amd64]) Conf erlang-base (1:14.b.4-dfsg-1ubuntu1 Ubuntu:12.04/precise [amd64]) Conf erlang-crypto (1:14.b.4-dfsg-1ubuntu1 Ubuntu:12.04/precise [amd64]) Conf erlang-syntax-tools (1:14.b.4-dfsg-1ubuntu1 Ubuntu:12.04/precise [amd64]) Conf libsctp1 (1.0.11+dfsg-2 Ubuntu:12.04/precise [amd64]) Conf lksctp-tools (1.0.11+dfsg-2 Ubuntu:12.04/precise [amd64]) which looks quite minimal to me. -- Motiejus Jak?tys From magnus@REDACTED Wed Feb 6 18:18:06 2013 From: magnus@REDACTED (Magnus Henoch) Date: Wed, 06 Feb 2013 17:18:06 +0000 Subject: [erlang-questions] ssl cb_info vs custom socket library Message-ID: Hi all, I'm trying to use the ssl application together with a custom socket library, using the cb_info option to pass in the module name. However, at some places the ssl application assumes that the socket is a "real" socket, and calls inet:setopts on it - though ssl_connection:workaround_transport_delivery_problems specifically checks for this and avoids inet:setopts if the transport module is not gen_tcp. I realise that cb_info is an undocumented option, so I'd appreciate any opinions before spending more time on this. Are there any other reasons why using a different transport callback would not work? Would a patch fixing such problems be welcome? Regards, Magnus From derekbrown121@REDACTED Wed Feb 6 21:27:16 2013 From: derekbrown121@REDACTED (Derek Brown) Date: Wed, 6 Feb 2013 15:27:16 -0500 Subject: [erlang-questions] Evening School of Erlang Message-ID: Francesco- I'm looking into New York locations. I have the feeling that you'll have a lot of attendance interest in NYC. > Date: Tue, 05 Feb 2013 22:54:26 +0000 > From: Francesco Cesarini > To: erlang-questions@REDACTED, Monika Jarzyna > > Subject: Re: [erlang-questions] Evening School of Erlang > Message-ID: <51118DA2.1090103@REDACTED> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > If we get enough attendees, there will be a second round in about 4 - > 6 > months. It all depends on demand and local companies recruiting for > Erlang developers willing to sponsor and how many of the attendees > find > Erlang jobs. San Mateo/Palo Alto might also be an option for you. San > Francisco classes finished last week and a new round of dates will > soon > be announced. > > In regards to New York, we are looking at the latter half of this > year. > Derek - if you are able to help us secure a venue, it would be great! > Other locations currently include Chicago, DC, Boston, Pasadena and > Santa Monica. > > Francesco > > On 05/02/2013 22:04, Raoul Duke wrote: > >> San Jose hasn't happened yet - I believe you can still get tickets. > > i won't be around for it this feb :-( > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -- > Erlang Solutions Ltd. > http://www.erlang-solutions.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Wed Feb 6 23:04:44 2013 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 7 Feb 2013 11:04:44 +1300 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: , Message-ID: On 6/02/2013, at 3:59 AM, Matthew Evans wrote: > This pertains to a feature I would like to see in Erlang. > The ability to set an optional "memory limit" ... That persuaded me to write a new EEP, which I've just sent off. In the future, instead of saying "a feature I would like to see", you'll be able to say "why isn't EEP XX implemented yet?" (:-) From mattevans123@REDACTED Wed Feb 6 23:13:21 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Wed, 6 Feb 2013 17:13:21 -0500 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: , , Message-ID: Awesome ... Let's hope someone from Ericsson likes the idea. I recall a long time ago Joe Armstrong talking about something similar. ---------------------------------------- > Subject: Re: [erlang-questions] How to see which processes used ets table? > From: ok@REDACTED > Date: Thu, 7 Feb 2013 11:04:44 +1300 > CC: gustav.simonsson@REDACTED; solomon.wzs@REDACTED; erlang-questions@REDACTED > To: mattevans123@REDACTED > > > On 6/02/2013, at 3:59 AM, Matthew Evans wrote: > > This pertains to a feature I would like to see in Erlang. > > The ability to set an optional "memory limit" ... > > That persuaded me to write a new EEP, which I've just > sent off. > In the future, instead of saying "a feature I would like > to see", you'll be able to say "why isn't EEP XX > implemented yet?" (:-) > > From wallentin.dahlberg@REDACTED Thu Feb 7 00:09:36 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Thu, 7 Feb 2013 00:09:36 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: 2013/2/5 Matthew Evans > Hi, > > I don't wish to thread-jack, but I am anyway ;-) > > This pertains to a feature I would like to see in Erlang. The ability to > set an optional "memory limit" when a process and ETS table is created (and > maybe a global optional per-process limit when the VM is started). I've > seen a few cases where, due to software bugs, a process size grows and > grows; unfortunately as things stand today the result is your entire VM > crashing - hopefully leaving you with a crash_dump. > > Having such a limit could cause the process to terminate (producing a OOM > crash report in erlang.log) and the crashing process could be handled with > supervisor rules. Even better you can envisage setting the limits > artificially low during testing to catch these types of bugs early on. > Not this again. This idea is not an uninteresting one. A safeguard against bad flow control in application design. We, well I, implemented a "limits" prototype where I could set max heap sizes on processes. The process would be killed with an exception limit error if it ever reached its max heap size. It worked fine in most cases but if I remember correctly we couldn't guard against binary creation fast enough (without killing performance). I'm pretty sure we could remedy those problems though, but by then my work queue was reprioritized. We ultimately decided against it, citing bad application design as exhibit A and root cause to runaway memory consumption; thus not strictly needed. =) I think the biggest advantages of limits is like you say, catching problems early in design. However, I think you can simulate some of this already by using a server process with: erlang:system_monitor(MonitorPid, [{large_heap, HeapSize}]), ... receive {monitor, GcPid, large_heap, _} -> purge_with_fire(GcPid); ... That might suffice for developing purposes. If this should be considered again something more extensive then just "limit heaps on processes and ets" should be considered I think. I would like a subscriber system on erts where a could subscribe to different part of the system and handle notifications as I see fit. The trouble here would be "how fast do I need to handle this?". Do a message suffice or do I need to wrap the notification in a fun like a callback? Perhaps it has som validity to do something simple instead. If a good enough proposal is put forward it will be considered. // Bj?rn-Egil > > Cheers > > Matt > > ------------------------------ > Date: Tue, 5 Feb 2013 08:51:16 +0100 > From: gustav.simonsson@REDACTED > To: solomon.wzs@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] How to see which processes used ets table? > > > Note that erlang:memory/1 gives memory in bytes, whereas ets:i/0 gives it > in words. On a 64-bit system 625M words would be 5GB, so the numbers you > get might be correct. Try summing the memory from ets:i/0 up exact and > compare it to erlang:memory(ets). > > Cheers, > Gustav Simonsson > > > On Tue, Feb 5, 2013 at 3:42 AM, Solomon wrote: > > I checked system info with erlang:memory/1 and found the total amount of > memory allocated for ets tables was about 5GB. > > Then I sum the memory size return by ets:i/0 and it was about 600MB. > > So which processes used ets table and did not return by ets:i/0? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ erlang-questions mailing > list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Thu Feb 7 00:17:00 2013 From: comptekki@REDACTED (Wes James) Date: Wed, 6 Feb 2013 16:17:00 -0700 Subject: [erlang-questions] erlang messaging over ssl - compiling .rel file during make Message-ID: I have erlang working with ssl now per Romain Lenglet's suggestion. Is there a way to compile the .rel file with the "make" process via erlang cli (I didn't find a cli util) or do I need to make an escript that does it. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Feb 7 02:31:30 2013 From: ok@REDACTED (ok@REDACTED) Date: Thu, 7 Feb 2013 14:31:30 +1300 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: "Bj?rn-Egil Dahlberg" wrote: > We, well I, implemented a "limits" prototype where I could set max heap > sizes on processes. The process would be killed with an exception limit > error if it ever reached its max heap size. [snip] > > We ultimately decided against it, citing bad application design as exhibit > A and root cause to runaway memory consumption; thus not strictly needed. > =) That's not why it isn't needed, that's why it IS needed. > I think the biggest advantages of limits is like you say, catching > problems early in design. I'm writing this reply through a pain-in-the-posterior WebMail system. Why? Because my normal machine is unavailable. Why? Because (a) my files were arbitrarily moved off my machine onto NFS some years ago, *despite* my explicit request that my files never be moved again without consulting me, (b) Mac OS X 10.6.8, which has served me well with no problems, apparently got one update too many (said to be to address Java problems) last week and demanded to be shut down; (c) when it came back, NFS connection was lost. The technical support people say that the simplest fix is to destroy my environment and give me 10.7 instead. (I'm using 10.7 now, as a matter of fact, on a loaner box; Mail does not recognise my old mailbox, my Dock set-up has been trashed, and so it goes.) The problem here is that a well designed and tested system that has worked flawlessly for years can stop working as a result of some apparently unrelated upgrade. > However, I think you can simulate some of this already by using a server > process with: > > erlang:system_monitor(MonitorPid, [{large_heap, HeapSize}]), > ... > receive > {monitor, GcPid, large_heap, _} -> purge_with_fire(GcPid); > ... > What's wrong with this approach? (1) It checks whether *ANY* process gets a heap exceeding the specified size. But some processes might legitimately need a large heap, while others might be out of control if they grow to more than a few kilobytes. (2) The limit that is checked is "the sum of the sizes of all memory blocks allocated for all heap generations". But that is something that is entirely out of the control of an Erlang programmer. The thing a programmer (or a process) can be held responsible for (within reason) is the number of words of memory actually *live*. (3) *ANY* process can change this; it is not limited to the process itself (good for protection against accidents) or its creator (good for protection against malice). (4) Judging from the description of system_monitor/0, there can only be ONE system monitor for ALL processes. If there is a monitor watching for distribution problems (busy_dist_port) and an unrelated piece of code installs a monitor for large heaps, the distribution problem monitor is wiped away. (5) As noted in the documentation, the monitoring process is not exempt from monitoring, so you can easily get into a recursive loop where memory overflow in the monitoring processes cause messages to be sent to it which cause ... I could go on. These are not the only flaws I see with this interface. Let's just say it's on my list of interfaces to avoid while I still have my sanity. The only alternative I've been able to think of with the existing interfaces is to create a watchdog process that checks process_info(Target, memory) periodically, which is doable, and which can check for message queue growth as well. From essen@REDACTED Thu Feb 7 02:50:14 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Thu, 07 Feb 2013 02:50:14 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: <51130856.80304@ninenines.eu> On 02/07/2013 02:31 AM, ok@REDACTED wrote: > The only alternative I've been able to think of with the > existing interfaces is to create a watchdog process that > checks process_info(Target, memory) periodically, which is > doable, and which can check for message queue growth as well. This is probably the least insane approach. A maximum limit on the memory used by a process could certainly prevent processes who start using too much because of bugs, but it might also have the inverse effect of killing legitimate processes. The chances of this happening increase with the number of updates the code receives, as you will most likely forget to update it. There's only one sane approach to make sure a system works as intended for any size of inputs over any length of time: testing. And more than that, property based testing. If your system crashes because of memory issues and you didn't catch it while testing, you might consider writing some properties and feeding your application bigger chunks of data over longer periods of time. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From wallentin.dahlberg@REDACTED Thu Feb 7 03:14:30 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Thu, 7 Feb 2013 03:14:30 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: 2013/2/7 > "Bj?rn-Egil Dahlberg" wrote: > > > We, well I, implemented a "limits" prototype where I could set max heap > > sizes on processes. The process would be killed with an exception limit > > error if it ever reached its max heap size. [snip] > > > > We ultimately decided against it, citing bad application design as > exhibit > > A and root cause to runaway memory consumption; thus not strictly needed. > > =) > > That's not why it isn't needed, that's why it IS needed. > The "=)" refers to the somewhat unconvincingly conclusion. Ironi is not well suited for internet medium. It's not without truth though. The flow control problem is solved via good design, but I absolutely agree that limits are a handy tool for developers and can certainly help developers find those issues. Developers wants as much rope they can get from a system. > I think the biggest advantages of limits is like you say, catching > > problems early in design. > > > The problem here is that a well designed and tested system that > has worked flawlessly for years can stop working as a result of > some apparently unrelated upgrade. > Well, normally one tests an upgraded system before deployment but I agree with your point. > > However, I think you can simulate some of this already by using a server > > process with: > > > > erlang:system_monitor(MonitorPid, [{large_heap, HeapSize}]), > > ... > > receive > > {monitor, GcPid, large_heap, _} -> purge_with_fire(GcPid); > > ... > > > > What's wrong with this approach? > *Bracing for impact* > (1) It checks whether *ANY* process gets a heap exceeding the > specified size. But some processes might legitimately > need a large heap, while others might be out of control if > they grow to more than a few kilobytes. > > (2) The limit that is checked is "the sum of the sizes of all > memory blocks allocated for all heap generations". But that > is something that is entirely out of the control of an > Erlang programmer. The thing a programmer (or a process) > can be held responsible for (within reason) is the number > of words of memory actually *live*. > > (3) *ANY* process can change this; it is not limited to the > process itself (good for protection against accidents) or > its creator (good for protection against malice). > > (4) Judging from the description of system_monitor/0, there > can only be ONE system monitor for ALL processes. If > there is a monitor watching for distribution problems > (busy_dist_port) and an unrelated piece of code installs > a monitor for large heaps, the distribution problem monitor > is wiped away. > > (5) As noted in the documentation, the monitoring process is not > exempt from monitoring, so you can easily get into a > recursive loop where memory overflow in the monitoring > processes cause messages to be sent to it which cause ... > This is great! You have identified several problems that can be fed into a limits proposal and how it should be solved. Please note that system_monitor suggestion in this mail was intended as a "fall-forward" (swedish idiom?). The development case .. but I think you understood that. I could go on. These are not the only flaws I see with this > interface. Let's just say it's on my list of interfaces to > avoid while I still have my sanity. > Haha. =) I will not be as judgmental in my opinion of that interface but it has some shortcomings, that's for sure. > The only alternative I've been able to think of with the > existing interfaces is to create a watchdog process that > checks process_info(Target, memory) periodically, which is > doable, and which can check for message queue growth as well. > I wouldn't recommend such an approach either. Especially if you want it in production. You will kill any sort of performance if you do this. Also, if you implement something within erts that tries to check the *complete* msg queue (both inner and outer) you will also kill any sort of application performance. I think the best approach is the garbage collector still. You have complete access to the process and can make judgment calls there. Also, processes is just one type of entity for limits. You have to consider *all* aspects of the runtime system if you wish to make a limits proposal that can stop OOM problems. I'm still not convinced that this should be prioritized over something else. Runaway memory is, most of the time, bad flow control. Wouldn't it be better to solve that instead? // Bj?rn-Egil -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuulos@REDACTED Thu Feb 7 08:11:18 2013 From: tuulos@REDACTED (Ville Tuulos) Date: Wed, 6 Feb 2013 23:11:18 -0800 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 6:14 PM, Bj?rn-Egil Dahlberg wrote: > I'm still not convinced that this should be prioritized over something else. > Runaway memory is, most of the time, bad flow control. Wouldn't it be better > to solve that instead? Here's my data point in support for the limits: Bad flow control is exactly the reason why the process limits would be useful for Disco: Disco communicates with an external worker process over stdin/stdout using standard ports. As far as I know, open_port doesn't include any mechanism to limit a rogue external process from flooding the message queue of the Erlang process with tons of messages in the matter of seconds, which can cause the whole VM to run out of memory. This could be fixed either by improving the flow control in open_port or by limiting the number of messages and/or memory that can be used by the controlling process. Instead of trying to fix all possible horror scenarios with flow control, the per-process limits seems like a more robust solution. It is pretty much the same reason why OS-level containers such as OpenVZ and LXC are so extremely useful, even if you don't need to worry about security. Ville From ok@REDACTED Thu Feb 7 09:39:53 2013 From: ok@REDACTED (ok@REDACTED) Date: Thu, 7 Feb 2013 21:39:53 +1300 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: <51e1f553bf774403f3b88e79a84a50fc.squirrel@chasm.otago.ac.nz> "Bj?rn-Egil Dahlberg" wrote: >> The problem here is that a well designed and tested system that >> has worked flawlessly for years can stop working as a result of >> some apparently unrelated upgrade. >> > > Well, normally one tests an upgraded system before deployment but I agree > with your point. One would have surmised that Apple had thoroughly tested their Java-related patches, but allegedly that's what killed NFS on my desktop machine. One would think that a payroll system would be thoroughly tested before abruptly moving 110,000 teachers over to it, but apparently four separate consultants told the New Zealand government to go ahead with deployment despite Talent2 telling them there were 147 known serious bugs. There's all that "normally" stuff I tell me students to do, and then there's what happens in the "real" world. >> The only alternative I've been able to think of with the >> existing interfaces is to create a watchdog process that >> checks process_info(Target, memory) periodically, which is >> doable, and which can check for message queue growth as well. >> > > I wouldn't recommend such an approach either. Especially if you want it in > production. You will kill any sort of performance if you do this. Also, if > you implement something within erts that tries to check the *complete* msg > queue (both inner and outer) you will also kill any sort of application > performance. This caught my eye. What are "inner" and "outer" message queues? One aspect of Erlang implementation I've never understood is how message queues actually work. > > I think the best approach is the garbage collector still. Well, the EEP I posted today suggests checking memory limits at garbage collection time. > Also, processes is just one type of entity for limits. True. I've received the impression that they are the most commonly reported kind. You have to consider > *all* aspects of the runtime system if you wish to make a limits proposal > that can stop OOM problems. That sounds reasonable, but I'm not sure that waiting until we can guarantee to stop *all* OOM problems is better than reducing the incidence of *some* OOM problems sooner. > > I'm still not convinced that this should be prioritized over something > else. Runaway memory is, most of the time, bad flow control. Wouldn't it > be better to solve that instead? Perhaps I spend too much time reading the Haskell mailing list, where runaway memory is commoner than in Erlang and has many and subtle causes. Perhaps I've spent too much time hacking Smalltalk, where at least in my programs memory gets chewed up by data structures I didn't even know were being allocated, let alone retained. It would be *wonderful* to have clearly written advice and supporting libraries to get flow control right. It may well be *better* to do that first. But it won't stop OOM problems. From attila.r.nohl@REDACTED Thu Feb 7 11:52:10 2013 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Thu, 7 Feb 2013 11:52:10 +0100 Subject: [erlang-questions] How to see which processes used ets table? In-Reply-To: References: Message-ID: 2013/2/7 Bj?rn-Egil Dahlberg : [...] > Not this again. > > This idea is not an uninteresting one. A safeguard against bad flow control > in application design. I try to recall the last few out of memory situations I saw: - the system makes a CORBA call, but the remote site goes down without any warning at an inappropriate time (there were no non-ACKd TCP segments out there, then we didn't even get an ICMP error or RST answer) and by the time the OS noticed that the connection went down, the queue of a gen_server got too big. The solution was not not block the gen_server when the CORBA call blocks - this is a kind of flow control issue - there was a programming error, some developer failed to delete the old entries from the ets database when those entries should have been deleted, then ets grew too big - there was a programming error, the client didn't expect the "." and the ".." directory in the filelisting, so the client happily changed into these directories over and over until the path grew too big I think killing the too big process could have saved the whole VM and the application in general, but of course, the crash dump was very useful in all cases. From robert.virding@REDACTED Thu Feb 7 11:57:43 2013 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 7 Feb 2013 10:57:43 +0000 (GMT) Subject: [erlang-questions] Load path for erlang:load_nif In-Reply-To: <492930095.3559990.1360234540774.JavaMail.root@erlang-solutions.com> Message-ID: <1184325436.3560036.1360234663655.JavaMail.root@erlang-solutions.com> Any suggestions for how to NOT have to give a hardwired name for a NIF file to erlang:load_nif/1? Preferably specifying at run-time, but compile-time is ok. Robert From fithis2001@REDACTED Thu Feb 7 12:46:58 2013 From: fithis2001@REDACTED (Vasileios Anagnostopoulos) Date: Thu, 7 Feb 2013 13:46:58 +0200 Subject: [erlang-questions] Porters's guide? Message-ID: Hi, the Erlang site has good information for building it on windows. However it is not helpful for someone wishing to port Erlang to haiku/eCommstation/Genode and so on. Is there any plan for a porters' guide? Regards Vasileios -- Dr. Vasileios Anagnostopoulos (MSc,PhD) Researcher/Developer ICCS/NTUA 9 Heroon Polytechneiou Str., Zografou 15773 Athens,Greece T (+30) 2107723404 M (+30) 6936935388 E vanag@REDACTED>www.ntua.gr -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.simonsson@REDACTED Thu Feb 7 13:21:40 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Thu, 7 Feb 2013 13:21:40 +0100 Subject: [erlang-questions] Slimmed down version In-Reply-To: References: Message-ID: Hi, I updated https://github.com/Gustav-Simonsson/Stripped-Erlang after testing It with R15B03-1. You can use either the script or the steps in it to create a slimmed down Erlang release. Cheers, Gustav Simonsson On Wed, Feb 6, 2013 at 3:28 PM, Vasileios Anagnostopoulos < fithis2001@REDACTED> wrote: > > Hi, > > Erlang/OTP is a big package to download and build. Is there any option to > provide a slimmed down version? Only the bare minimum? > I think that OTP must become more modular and rely on an external package > system. > > > Regards > Vasileios > -- > > Dr. Vasileios Anagnostopoulos (MSc,PhD) > Researcher/Developer > ICCS/NTUA > 9 Heroon Polytechneiou Str., Zografou > 15773 Athens,Greece > T (+30) 2107723404 > M (+30) 6936935388 > E vanag@REDACTED>www.ntua.gr > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingela.andin@REDACTED Thu Feb 7 14:23:23 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 7 Feb 2013 14:23:23 +0100 Subject: [erlang-questions] ssl cb_info vs custom socket library In-Reply-To: References: Message-ID: Hi! The cb_info option is to not lock the SSL implementation to use TCP as transport protocol. An other reliable transport protocol implemented with same API could be used. At the moment we do not have any other transport protocol. The transport workaround is for the TCP protocol specifically, so the check is to avoid the whole workaround. Your problem however is that the assumption is that it will be an inet socket. And inet:setops will be called directly on the socket. So to enable what you want to do probably there should be an callback module for inet too. We would rather not wrap gen_tcp to allow it to be the same module! Regards Ingela Erlang/OTP team - Ericsson AB 2013/2/6, Magnus Henoch : > Hi all, > > I'm trying to use the ssl application together with a custom socket > library, using the cb_info option to pass in the module name. However, > at some places the ssl application assumes that the socket is a "real" > socket, and calls inet:setopts on it - though > ssl_connection:workaround_transport_delivery_problems specifically > checks for this and avoids inet:setopts if the transport module is not > gen_tcp. > > I realise that cb_info is an undocumented option, so I'd appreciate any > opinions before spending more time on this. Are there any other reasons > why using a different transport callback would not work? Would a patch > fixing such problems be welcome? > > Regards, > Magnus > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From calleja.justin@REDACTED Thu Feb 7 14:55:12 2013 From: calleja.justin@REDACTED (Justin Calleja) Date: Thu, 7 Feb 2013 14:55:12 +0100 Subject: [erlang-questions] How to get syntax tree out of abstract forms? Message-ID: Hi, Let's say I have this module: -module(hello). -export([send_hello/1, receive_hello/0]). send_hello(To) -> To ! {hello, self()}. receive_hello() -> receive {hello, From} -> {ok, From}; Msg -> {unknown_msg, Msg} end. which gives the following abstract forms (lets bind to Forms for ease of reference): [{attribute,1,file,{"../src/eaop_explore/hello.erl",1}}, {attribute,1,module,hello}, {attribute,2,export,[{send_hello,1},{receive_hello,0}]}, {function,4,send_hello,1, [{clause,4, [{var,4,'To'}], [], [{op,5,'!', {var,5,'To'}, {tuple,5,[{atom,5,hello},{call,5,{atom,5,self},[]}]}}]}]}, {function,7,receive_hello,0, [{clause,7,[],[], [{'receive',8, [{clause,9, [{tuple,9,[{atom,9,hello},{var,9,'From'}]}], [], [{tuple,10,[{atom,10,ok},{var,10,'From'}]}]}, {clause,11, [{var,11,'Msg'}], [], [{tuple,12,[{atom,12,unknown_msg},{var,12,'Msg'}]}]}]}]}]}, {eof,13}] I would like to use the postorder/2 function in the comments of erl_syntax: postorder(F, Tree) -> F(case erl_syntax:subtrees(Tree) of [] -> Tree; List -> erl_syntax:update_tree(Tree, [[postorder(F, Subtree) || Subtree <- Group] || Group <- List]) end). to traverse a syntax tree and do some kind of manipulation. The problems is that, although the documentation (or how I understood the doc anyway) claims that abstract forms are a subset of syntax trees, I cannot do the following: postorder(getAtomFun(), Forms) where: getAtomFun() -> fun(Node) -> case erl_syntax:type(Node) of atom -> case erl_syntax:atom_name(Node) of "module" -> Node; "export" -> Node; _ -> erl_syntax:atom("a_" ++ erl_syntax:atom_name(Node)) end; _ -> Node end end. However, if Forms is constructed manually using the erl_syntax module, I can do that. e.g. taking a small part of Forms and constructing using erl_syntax (namely, only the module and export attributes), I can do the following: get_tree() -> erl_syntax:form_list([ erl_syntax:attribute(erl_syntax:atom(module), [erl_syntax:atom(hello)]), erl_syntax:attribute(erl_syntax:atom(export), [ erl_syntax:list([ erl_syntax:arity_qualifier(erl_syntax:atom(send_hello), erl_syntax:integer(1)), erl_syntax:arity_qualifier(erl_syntax:atom(receive_hello), erl_syntax:integer(0)) ]) ]) ]). 20> postorder(getAtomFun(), get_tree()) ... 21> erl_syntax:revert_forms(v(20)). [{attribute,0,module,a_hello}, {attribute,0,export,[{a_send_hello,1},{a_receive_hello,0}]}] The problem is that I can't seem to be able to do such an operation unless I can somehow get the forms into syntax trees. Is there a function I'm missing in erl_syntax which transforms from forms to syntax trees? (revert_forms/1 exists to do the transformation in the other way). Regards, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Thu Feb 7 15:03:56 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 07 Feb 2013 15:03:56 +0100 Subject: [erlang-questions] How to get syntax tree out of abstract forms? In-Reply-To: References: Message-ID: <5113B44C.6070301@gmail.com> On 02/07/2013 02:55 PM, Justin Calleja wrote: > I would like to use the postorder/2 function in the comments of erl_syntax: > > postorder(F, Tree) -> > F(case erl_syntax:subtrees(Tree) of > [] -> Tree; > List -> erl_syntax:update_tree(Tree, > [[postorder(F, Subtree) > || Subtree <- Group] > || Group <- List]) > end). > > to traverse a syntax tree and do some kind of manipulation. > > The problems is that, although the documentation (or how I understood > the doc anyway) claims that abstract forms are a subset of syntax trees, Yes, an abstract form is also an AST that can be used with e.g. erl_syntax:subtrees(). However, a *list* of abstract forms is not a tree. What you can do is to give your list to erl_syntax:form_list(Fs) to get a single AST that represents all the forms, and then pass that tree to your traversal function. /Richard From calleja.justin@REDACTED Thu Feb 7 15:30:38 2013 From: calleja.justin@REDACTED (Justin Calleja) Date: Thu, 7 Feb 2013 15:30:38 +0100 Subject: [erlang-questions] How to get syntax tree out of abstract forms? In-Reply-To: <5113B44C.6070301@gmail.com> References: <5113B44C.6070301@gmail.com> Message-ID: Hi Richard, Thanks for the reply :) I had already tried erl_syntax:form_list/1 on the Forms. The output is: 21> erl_syntax:form_list(Forms). {tree,form_list, {attr,0,[],none}, [{attribute,1,file,{"../src/eaop_explore/hello.erl",1}}, {attribute,1,module,hello}, {attribute,2,export,[{send_hello,1},{receive_hello,0}]}, {function,4,send_hello,1, [{clause,4, [{var,4,'To'}], [], [{op,5,'!', {var,5,'To'}, {tuple,5,[{atom,5,hello},{call,5,{atom,5,self},[]}]}}]}]}, {function,7,receive_hello,0, [{clause,7,[],[], [{'receive',8, [{clause,9, [{tuple,9,[{atom,9,hello},{var,9,'From'}]}], [], [{tuple,10,[{atom,10,ok},{var,10,'From'}]}]}, {clause,11, [{var,11,'Msg'}], [], [{tuple,12, [{atom,12,unknown_msg},{var,12,'Msg'}]}]}]}]}]}, {eof,13}]} But, when I did: 24> module_attr:postorder(module_attr:getAtomFun(), v(21)). {tree,form_list, {attr,0,[],none}, [{tree,attribute, {attr,1,[],none}, {attribute, {tree,atom,{attr,0,[],none},a_file}, [{tree,string, {attr,1,[],none}, "../src/eaop_explore/hello.erl"}, {tree,integer,{attr,1,[],none},1}]}}, {tree,attribute, {attr,1,[],none}, {attribute, {tree,atom,{attr,1,[],none},module}, [{tree,atom,{attr,0,[],none},a_hello}]}}, {tree,attribute, {attr,2,[],none}, {attribute, {tree,atom,{attr,2,[],none},export}, [{tree,list, {attr,2,[],none}, {list, [{tree,arity_qualifier, {attr,2,[],none}, {arity_qualifier,{tree,...},{...}}}, {tree,arity_qualifier, {attr,2,[],...}, {arity_qualifier,{...},...}}], none}}]}}, {tree,function, {attr,4,[],none}, {function, {tree,atom,{attr,0,[],none},a_send_hello}, [{tree,clause, {attr,4,[],none}, {clause, [{var,4,'To'}], none, [{tree,infix_expr,{attr,...},{...}}]}}]}}, {tree,function, {attr,7,[],none}, {function, {tree,atom,{attr,0,[],none},a_receive_hello}, [{tree,clause, {attr,7,[],none}, {clause,[],none,[{tree,receive_expr,{...},...}]}}]}}, {eof,13}]} and then: 25> erl_syntax:revert_forms(v(24)). ** exception error: {error, {tree,attribute, {attr,1,[],none}, {attribute, {atom,0,a_file}, [{string,1,"../src/eaop_explore/hello.erl"}, {integer,1,1}]}}} in function erl_syntax:revert_forms/1 (erl_syntax.erl, line 6648) lol, taking a look at it now I can see what I did... changed the 'file' attribute to 'a_file'... it works if you remove the following (or at least change 'a_file' to 'file') {tree,attribute, {attr,1,[],none}, {attribute, {tree,atom,{attr,0,[],none},a_file}, [{tree,string,{attr,1,[],none},"../src/eaop_explore/hello.erl"}, {tree,integer,{attr,1,[],none},1}]}}, How stupid of me ^^; Already came across the same problem with the 'export' and 'module' attributes just 30 mins or so ago. Thanks for the help!! Regards, Justin On 7 February 2013 15:03, Richard Carlsson wrote: > On 02/07/2013 02:55 PM, Justin Calleja wrote: > >> I would like to use the postorder/2 function in the comments of >> erl_syntax: >> >> postorder(F, Tree) -> >> F(case erl_syntax:subtrees(Tree) of >> [] -> Tree; >> List -> erl_syntax:update_tree(Tree, >> [[postorder(F, Subtree) >> || Subtree <- Group] >> || Group <- List]) >> end). >> >> to traverse a syntax tree and do some kind of manipulation. >> >> The problems is that, although the documentation (or how I understood >> the doc anyway) claims that abstract forms are a subset of syntax trees, >> > > Yes, an abstract form is also an AST that can be used with e.g. > erl_syntax:subtrees(). However, a *list* of abstract forms is not a tree. > What you can do is to give your list to erl_syntax:form_list(Fs) to get a > single AST that represents all the forms, and then pass that tree to your > traversal function. > > /Richard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egil@REDACTED Thu Feb 7 16:27:49 2013 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Thu, 7 Feb 2013 16:27:49 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> Message-ID: <5113C7F5.9050500@erlang.org> I dug out what I wrote a year ago .. eep-draft: https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md Reference implementation: https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 Remember, this is a prototype and a reference implementation. There is a couple of issues not addressed or at least open-ended. * Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option. * ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing. * In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well. * Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even more queues for scalability reasons this will cost even more. * What about other memory users? Drivers? NIFs? I do believe in increments in development as long it is path to the envisioned goal. And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes. // Bj?rn-Egil On 2013-02-06 23:03, Richard O'Keefe wrote: > Just today, I saw Matthew Evans' > > This pertains to a feature I would like to see > in Erlang. The ability to set an optional > "memory limit" when a process and ETS table is > created (and maybe a global optional per-process > limit when the VM is started). I've seen a few > cases where, due to software bugs, a process size > grows and grows; unfortunately as things stand > today the result is your entire VM crashing - > hopefully leaving you with a crash_dump. > > Having such a limit could cause the process to > terminate (producing a OOM crash report in > erlang.log) and the crashing process could be > handled with supervisor rules. Even better you > can envisage setting the limits artificially low > during testing to catch these types of bugs early on. > > in my mailbox. I have seen too many such e-mail messages. > Here's a specific proposal. It's time _something_ was done > about this kind of problem. I don't expect that my EEP is > the best way to deal with it, but at least there's going to > be something for people to point to. > > > > _______________________________________________ > eeps mailing list > eeps@REDACTED > http://erlang.org/mailman/listinfo/eeps -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattevans123@REDACTED Thu Feb 7 17:04:23 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Thu, 7 Feb 2013 11:04:23 -0500 Subject: [erlang-questions] erlang:delete_element/2 missing Message-ID: Hi, It's in the Erlang documentation. Has this been removed (tried on R15B01 and R15B03)? Erlang R15B01 (erts-5.9.1) [source] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.1 ?(abort with ^G) 1> erlang:delete_element(2, {one, two, three}). ** exception error: undefined function erlang:delete_element/2 2>? From sigxcpu@REDACTED Thu Feb 7 17:08:58 2013 From: sigxcpu@REDACTED (Alex P) Date: Thu, 7 Feb 2013 08:08:58 -0800 (PST) Subject: [erlang-questions] Memory growing In-Reply-To: References: Message-ID: <74db6641-f140-45f5-8509-c3c6979f0c57@googlegroups.com> I've tested NIFs from lists module and they seem to be fine. So, coming back to square one, how can I debug what is allocated in the "system" section of erlang:memory/0 ? Alex On Saturday, February 2, 2013 10:52:04 PM UTC+2, Alex P wrote: > > Hello, > > If somebody could help me with an issue, I would be grateful. I'm pulling > my hair off trying to figure out why an erlang app keeps growing in memory > usage. The erlang:memory() after 3 days shows this: > > [{total,325732292}, > {processes,54520269}, > {processes_used,54492115}, > * {system,271212023}, > * {atom,932793}, > {atom_used,912937}, > {binary,2348080}, > {code,8288354}, > {ets,487392}, > {maximum,398189416}] > > The interesting part is "system" section. All other data is fine > (processes goes to few MBs during the night as all the started procs are > either hibernated - for long running or they die because of inactivity). > > There are two issues here: > - "system" keeps growing (around 800MB/week) > - OS (Linux) shows a significantly different amount of memory usage > (almost 2x, 610MB resident in this case) > > The only case where I've managed to make "system" grow is by calling > erlang:monitor() multiple times. It grows with ~13K/call. > > Erlang release used is R15B03. > > The questions are: > - is there any hidden function to inspect all the active process monitors? > (as the single place where I use erlang:monitor looks fine. The spawned kid > does its job and dies, so the ref should be garbage collected). > - is there any way to inspect "system" section and find out what it > contains? Looking at memory chunks in process map got me nowhere > > Thank you, > Alex > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Thu Feb 7 17:14:15 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Thu, 7 Feb 2013 11:14:15 -0500 Subject: [erlang-questions] [erlang-bugs] erlang:delete_element/2 missing In-Reply-To: References: Message-ID: On Thu, Feb 7, 2013 at 11:04 AM, Matthew Evans wrote: > > Hi, > > It's in the Erlang documentation. Has this been removed (tried on R15B01 > and R15B03)? > > > > Erlang R15B01 (erts-5.9.1) [source] [smp:8:8] [async-threads:0] [hipe] > [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> erlang:delete_element(2, {one, two, three}). > ** exception error: undefined function erlang:delete_element/2 > 2> That's a new bif for R16: Erlang R16A (erts-5.10) [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false] Eshell V5.10 (abort with ^G) 1> erlang:delete_element(2, {one, two, three}). {one,three} 2> --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From egil@REDACTED Thu Feb 7 17:14:54 2013 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Thu, 7 Feb 2013 17:14:54 +0100 Subject: [erlang-questions] erlang:delete_element/2 missing In-Reply-To: References: Message-ID: <5113D2FE.8070507@erlang.org> It's in R16A merge branchMerge branch 'egil/r16/insert_delete_element/OTP-10643' 8f4e7139ba015e765c5c523c578aaad4a8580f51 // Bj?rn-Egil On 2013-02-07 17:04, Matthew Evans wrote: > Hi, > > It's in the Erlang documentation. Has this been removed (tried on R15B01 and R15B03)? > > > > Erlang R15B01 (erts-5.9.1) [source] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> erlang:delete_element(2, {one, two, three}). > ** exception error: undefined function erlang:delete_element/2 > 2> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mattevans123@REDACTED Thu Feb 7 17:21:45 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Thu, 7 Feb 2013 11:21:45 -0500 Subject: [erlang-questions] Memory growing In-Reply-To: <74db6641-f140-45f5-8509-c3c6979f0c57@googlegroups.com> References: , <74db6641-f140-45f5-8509-c3c6979f0c57@googlegroups.com> Message-ID: Can you kill the VM with a SIGUSR1 and look at the resulting crash dump? Start an Erlang shell, start crashdump viewer and poke around: crashdump_viewer:start(). ________________________________ > Date: Thu, 7 Feb 2013 08:08:58 -0800 > From: sigxcpu@REDACTED > To: erlang-programming@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Memory growing > > I've tested NIFs from lists module and they seem to be fine. > > So, coming back to square one, how can I debug what is allocated in the > "system" section of erlang:memory/0 ? > > Alex > > On Saturday, February 2, 2013 10:52:04 PM UTC+2, Alex P wrote: > Hello, > > If somebody could help me with an issue, I would be grateful. I'm > pulling my hair off trying to figure out why an erlang app keeps > growing in memory usage. The erlang:memory() after 3 days shows this: > > [{total,325732292}, > {processes,54520269}, > {processes_used,54492115}, > {system,271212023}, > {atom,932793}, > {atom_used,912937}, > {binary,2348080}, > {code,8288354}, > {ets,487392}, > {maximum,398189416}] > > The interesting part is "system" section. All other data is fine > (processes goes to few MBs during the night as all the started procs > are either hibernated - for long running or they die because of > inactivity). > > There are two issues here: > - "system" keeps growing (around 800MB/week) > - OS (Linux) shows a significantly different amount of memory usage > (almost 2x, 610MB resident in this case) > > The only case where I've managed to make "system" grow is by calling > erlang:monitor() multiple times. It grows with ~13K/call. > > Erlang release used is R15B03. > > The questions are: > - is there any hidden function to inspect all the active process > monitors? (as the single place where I use erlang:monitor looks fine. > The spawned kid does its job and dies, so the ref should be garbage > collected). > - is there any way to inspect "system" section and find out what it > contains? Looking at memory chunks in process map got me nowhere > > Thank you, > Alex > > > > _______________________________________________ erlang-questions > mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mattevans123@REDACTED Thu Feb 7 17:29:15 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Thu, 7 Feb 2013 11:29:15 -0500 Subject: [erlang-questions] [erlang-bugs] erlang:delete_element/2 missing In-Reply-To: References: , Message-ID: Ugh...thanks Steve, I'm not sure I want to move to R16A yet.? I guess I'll do something like this. 4> Tupple = {one,two,three}. 5> lists:foldl(fun(C,Acc) when C/=2 -> erlang:append_element(Acc,erlang:element(C,Tupple)); (_,Acc) -> Acc end,{},[1,2,3]). {one,three} ________________________________ > Date: Thu, 7 Feb 2013 11:14:15 -0500 > Subject: Re: [erlang-bugs] erlang:delete_element/2 missing > From: vinoski@REDACTED > To: mattevans123@REDACTED > CC: erlang-questions@REDACTED; erlang-bugs@REDACTED > > > > On Thu, Feb 7, 2013 at 11:04 AM, Matthew Evans > > wrote: > > Hi, > > It's in the Erlang documentation. Has this been removed (tried on > R15B01 and R15B03)? > > > > Erlang R15B01 (erts-5.9.1) [source] [smp:8:8] [async-threads:0] [hipe] > [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> erlang:delete_element(2, {one, two, three}). > ** exception error: undefined function erlang:delete_element/2 > 2> > > That's a new bif for R16: > > Erlang R16A (erts-5.10) [source] [64-bit] [smp:8:8] [async-threads:10] > [kernel-poll:false] > > Eshell V5.10 (abort with ^G) > 1> erlang:delete_element(2, {one, two, three}). > {one,three} > 2> > > --steve From mattevans123@REDACTED Thu Feb 7 18:51:23 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Thu, 7 Feb 2013 12:51:23 -0500 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <5113C7F5.9050500@erlang.org> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz>, <5113C7F5.9050500@erlang.org> Message-ID: ________________________________ > Date: Thu, 7 Feb 2013 16:27:49 +0100 > From: egil@REDACTED > To: eeps@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue > for Erlang > > I dug out what I wrote a year ago .. > > eep-draft: > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md > > Reference implementation: > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 > Remember, this is a prototype and a reference implementation. > > There is a couple of issues not addressed or at least open-ended. > > * Should processes be able to set limits on other processes? I think > not though my draft argues for it. It introduces unnecessary restraints > on erts and hinders performance. 'save_calls' is such an option.? I agree, other processes should not be able to do this. It kind of goes against Erlang's ideas of process isolation. > * ets - if your table increases beyond some limit. Who should we > punish? The inserter? The owner? What would be the rationale? We cannot > just punish the inserter, the ets table is still there taking a lot of > memory and no other process could insert into the table. They would be > killed as well. Remove the owner and hence the table (and potential > heir)? What kind of problems would arise then? Limits should be tied > into a supervision strategy and restart the whole thing. I think just the owner is good enough. It would be nice if the inserter could do it, but I imagine that's a non-trivial feature to implement.? > > * Message queues. In the current implementation of message queues we > have two queues. An inner one which is locked by the receiver process > while executing and an outer one which other processes will use and > thus not compete for a message queue lock with the executing process. > When the inner queue is depleted the receiver process will lock the > outer queue and move the entire thing to the inner one. Rinse and > repeat. The only guarantee we have to ensure with our implementation > is: signal order between two processes. So, in the future we might have > several queues to improve performance. If you introduce monitoring of > the total number messages in the abstracted queue (all the queues) this > will most probable kill any sort of scalability. For instance a sender > would not be allowed to check the inner queue for this reason. Would a > "fast" counter check in the inner queue be allowed? Perhaps if it is > fast enough, but any sort of bookkeeping costs performance. If we > introduce even more queues for scalability reasons this will cost even > more. I wasn't aware that the workings of the message queues was that complex internally. You are correct, that this monitoring must not be at the expense of performance, maybe the "check" could be made when the scheduler decides to schedule and execute a process. Or maybe it can check every X reductions? This means a process may not terminate at exactly the user-defined threshold, but I think most people can live with that. > I do believe in increments in development as long it is path to the > envisioned goal. > And to reiterate, i'm not convinced that limits on just processes is > the way to go. I think a complete monitoring system should be > envisioned, not just for processes. Personally I think a complete monitoring system would be great, but I think process limits are also invaluable.? I point you to Joe Armstrong's thesis "Making reliable distributed systems in the presence of software errors". Certainly memory leaks are a perfectly valid software error, one that as things stand today will crash your VM. Cheers Matt From tuulos@REDACTED Thu Feb 7 20:12:14 2013 From: tuulos@REDACTED (Ville Tuulos) Date: Thu, 7 Feb 2013 11:12:14 -0800 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <5113C7F5.9050500@erlang.org> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> Message-ID: On Thu, Feb 7, 2013 at 7:27 AM, Bj?rn-Egil Dahlberg wrote: > I dug out what I wrote a year ago .. > > eep-draft: > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md > > Reference implementation: > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 > Remember, this is a prototype and a reference implementation. > > There is a couple of issues not addressed or at least open-ended. Looks great! I truly hope this will get accepted rather sooner than later, at least as an experimental feature. Does the proposal cover refc binaries as well? Almost all interesting use cases that I can imagine for limits involve binaries. Here's one test case: 1. Create a web server in Erlang e.g. with Mochiweb, Cowboy, Inets. 2. Create a request handler that expects to receive Zip files, which it extracts with zip:extract(Request, [memory]). 3. Create a zip bomb [1]: dd if=/dev/zero bs=1M count=8000 | zip req.zip - 4. POST the small req.zip to the web server. 5. See the VM go down in flames. Obviously the per-process limits would elegantly solve this problem if they covered binaries as well. A far less elegant solution would be to handle the unsafe decompression with an external process and open_port (inefficient) or implement a sophisticated alternative to the zip module which handles the limits by itself (inefficient, annoying). I understand that limiting the message queue / ets / NIFs can be trickier. Just covering the basic max-heap with binaries would be a good starting point. Ville [1] http://en.wikipedia.org/wiki/Zip_bomb > * Should processes be able to set limits on other processes? I think not > though my draft argues for it. It introduces unnecessary restraints on erts > and hinders performance. 'save_calls' is such an option. > > * ets - if your table increases beyond some limit. Who should we punish? The > inserter? The owner? What would be the rationale? We cannot just punish the > inserter, the ets table is still there taking a lot of memory and no other > process could insert into the table. They would be killed as well. Remove > the owner and hence the table (and potential heir)? What kind of problems > would arise then? Limits should be tied into a supervision strategy and > restart the whole thing. > > * In my draft and reference implementation I use soft limits. Once a process > reaches its limit it will be marked for termination by an exit signal. The > trouble here is there is no real guarantee for how long this will take. A > process can continue appending a binary for a short while and ending the > beam with OOM still. (If I remember it correctly you have to schedule out to > terminate a process in SMP thus you need to bump all reduction. But, not all > things handle return values from the garbage collector, most notably within > the append_binary instruction). There may be other issues as well. > > * Message queues. In the current implementation of message queues we have > two queues. An inner one which is locked by the receiver process while > executing and an outer one which other processes will use and thus not > compete for a message queue lock with the executing process. When the inner > queue is depleted the receiver process will lock the outer queue and move > the entire thing to the inner one. Rinse and repeat. The only guarantee we > have to ensure with our implementation is: signal order between two > processes. So, in the future we might have several queues to improve > performance. If you introduce monitoring of the total number messages in the > abstracted queue (all the queues) this will most probable kill any sort of > scalability. For instance a sender would not be allowed to check the inner > queue for this reason. Would a "fast" counter check in the inner queue be > allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs > performance. If we introduce even more queues for scalability reasons this > will cost even more. > > * What about other memory users? Drivers? NIFs? > > I do believe in increments in development as long it is path to the > envisioned goal. > And to reiterate, i'm not convinced that limits on just processes is the way > to go. I think a complete monitoring system should be envisioned, not just > for processes. > > // Bj?rn-Egil > > On 2013-02-06 23:03, Richard O'Keefe wrote: > > Just today, I saw Matthew Evans' > > This pertains to a feature I would like to see > in Erlang. The ability to set an optional > "memory limit" when a process and ETS table is > created (and maybe a global optional per-process > limit when the VM is started). I've seen a few > cases where, due to software bugs, a process size > grows and grows; unfortunately as things stand > today the result is your entire VM crashing - > hopefully leaving you with a crash_dump. > > Having such a limit could cause the process to > terminate (producing a OOM crash report in > erlang.log) and the crashing process could be > handled with supervisor rules. Even better you > can envisage setting the limits artificially low > during testing to catch these types of bugs early on. > > in my mailbox. I have seen too many such e-mail messages. > Here's a specific proposal. It's time _something_ was done > about this kind of problem. I don't expect that my EEP is > the best way to deal with it, but at least there's going to > be something for people to point to. > > > > _______________________________________________ > eeps mailing list > eeps@REDACTED > http://erlang.org/mailman/listinfo/eeps > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From ok@REDACTED Thu Feb 7 21:51:39 2013 From: ok@REDACTED (ok@REDACTED) Date: Fri, 8 Feb 2013 09:51:39 +1300 Subject: [erlang-questions] [erlang-bugs] erlang:delete_element/2 missing In-Reply-To: References: , Message-ID: <7c406f7a30d007780378aa6f39c17541.squirrel@chasm.otago.ac.nz> > I guess I'll do something like this. > > 4> Tupple = {one,two,three}. > 5> lists:foldl(fun(C,Acc) when C/=2 -> > erlang:append_element(Acc,erlang:element(C,Tupple)); (_,Acc) -> Acc > end,{},[1,2,3]). > {one,three} Yike. Hello O(N**2)! delete_element(Index, Tuple) when is_tuple(Tuple), is_integer(Index), tuple_size(Tuple) >= Index, Index >= 1 -> L = tuple_to_list(Tuple), {F,[_|B]} = lists:split(Index - 1, L), list_to_tuple(F ++ B). At least that's O(N). Even a BIF can't beat that (although it can improve the constant factor). From ok@REDACTED Thu Feb 7 22:01:12 2013 From: ok@REDACTED (ok@REDACTED) Date: Fri, 8 Feb 2013 10:01:12 +1300 Subject: [erlang-questions] Porters's guide? In-Reply-To: References: Message-ID: <2527f286467d2419f1e5fc602528648d.squirrel@chasm.otago.ac.nz> > Hi, > > the Erlang site has good information for building it on windows. However > it > is not helpful for someone wishing to port Erlang to > haiku/eCommstation/Genode and so on. Is there any plan for a porters' > guide? +1. A colleague of mine here is building a machine out of boards with 4 ARM chips each (so of course they are 4ARMs) and is writing a minimalist operating system for them. I'd really like to have Erlang on that machine, but don't know how much is _needed_ from the OS. From paul.james.barry@REDACTED Thu Feb 7 22:21:00 2013 From: paul.james.barry@REDACTED (Paul Barry) Date: Thu, 7 Feb 2013 21:21:00 +0000 Subject: [erlang-questions] Porters's guide? In-Reply-To: <2527f286467d2419f1e5fc602528648d.squirrel@chasm.otago.ac.nz> References: <2527f286467d2419f1e5fc602528648d.squirrel@chasm.otago.ac.nz> Message-ID: Just a thought...but, I'd imagine (at the very least) you'd need a C compiler on your minimalist OS. Have you considered looking at Minix? http://www.minix3.org/ Paul. On 7 February 2013 21:01, wrote: > > Hi, > > > > the Erlang site has good information for building it on windows. However > > it > > is not helpful for someone wishing to port Erlang to > > haiku/eCommstation/Genode and so on. Is there any plan for a porters' > > guide? > > +1. A colleague of mine here is building a machine out of boards > with 4 ARM chips each (so of course they are 4ARMs) and is writing > a minimalist operating system for them. I'd really like to have > Erlang on that machine, but don't know how much is _needed_ from the > OS. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Feb 7 22:26:34 2013 From: ok@REDACTED (ok@REDACTED) Date: Fri, 8 Feb 2013 10:26:34 +1300 Subject: [erlang-questions] Porters's guide? In-Reply-To: References: <2527f286467d2419f1e5fc602528648d.squirrel@chasm.otago.ac.nz> Message-ID: > Just a thought...but, I'd imagine (at the very least) you'd need a C > compiler on your minimalist OS. We have a compiler that generates code that can be loaded into the machines; it's a compiler _for_ the OS but not _on_ the OS. I am well aware of Minix. However, there is very little that Minix does that we actually want. All we need is threads (not processes) and communication with other nodes and the host (USB at the moment) and block-level raw SSD access. Think VERTEX not UNIX. From ok@REDACTED Thu Feb 7 22:32:49 2013 From: ok@REDACTED (ok@REDACTED) Date: Fri, 8 Feb 2013 10:32:49 +1300 Subject: [erlang-questions] Porters's guide? In-Reply-To: References: <2527f286467d2419f1e5fc602528648d.squirrel@chasm.otago.ac.nz> Message-ID: <2202e1ca2f808816ecc7ec7aad3fc15b.squirrel@chasm.otago.ac.nz> > Just a thought...but, I'd imagine (at the very least) you'd need a C > compiler on your minimalist OS. Have you considered looking at Minix? > http://www.minix3.org/ PS: I specifically said that the machine my colleague is building is a cluster of *ARM* boards. Minix3 is 32-bit x86 only. From paul.james.barry@REDACTED Thu Feb 7 22:43:02 2013 From: paul.james.barry@REDACTED (Paul Barry) Date: Thu, 7 Feb 2013 21:43:02 +0000 Subject: [erlang-questions] Porters's guide? In-Reply-To: <2202e1ca2f808816ecc7ec7aad3fc15b.squirrel@chasm.otago.ac.nz> References: <2527f286467d2419f1e5fc602528648d.squirrel@chasm.otago.ac.nz> <2202e1ca2f808816ecc7ec7aad3fc15b.squirrel@chasm.otago.ac.nz> Message-ID: Sorry... I missed that (and jumped in too quickly with a response). Apologies. On 7 February 2013 21:32, wrote: > > Just a thought...but, I'd imagine (at the very least) you'd need a C > > compiler on your minimalist OS. Have you considered looking at Minix? > > http://www.minix3.org/ > > PS: I specifically said that the machine my colleague is building > is a cluster of *ARM* boards. Minix3 is 32-bit x86 only. > > > -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Thu Feb 7 23:24:36 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Thu, 7 Feb 2013 23:24:36 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> Message-ID: 2013/2/7 Ville Tuulos > On Thu, Feb 7, 2013 at 7:27 AM, Bj?rn-Egil Dahlberg > wrote: > > I dug out what I wrote a year ago .. > > > > eep-draft: > > > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md > > > > Reference implementation: > > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 > > Remember, this is a prototype and a reference implementation. > > > > There is a couple of issues not addressed or at least open-ended. > > Looks great! I truly hope this will get accepted rather sooner than > later, at least as an experimental feature. > Well as I said previously, I still have some issues with this approach. Envision the goal and then see if this is a good first step. Meaning: if we want to include other resource limits that is not process oriented how would we then design the interface then? A resource limits interface perhaps? > > Does the proposal cover refc binaries as well? Almost all interesting > use cases that I can imagine for limits involve binaries. > My implementation doesn't cover it. It covers process memory blocks, i.e. everything except refc-binaries. (well their headers are covered: procbins) Binaries are special beasts and needs special care. The easiest way to implement refc binary limits would be to use the virtual binary heap concept already present in the gc. By using vheaps we would count all the memory referenced by procbins in the process heap. Now, here it gets interesting. Several procbins may refer to the same refc binary blob and that memory would be counted several times. You would have to take special accounting care if you want refc uniqueness which would make it kind of expensive. You can't mark the binary as counted since other processes might be gc:ed and they might also reference it so you would have to make some other accommodations. Other than that you can just use the same idea as heap_size limits. The checks are similar and the api could be too: erlang:process_flag(limits, [{bin_vheap_size, Size}]). // Bj?rn-Egil > Here's one test case: > > 1. Create a web server in Erlang e.g. with Mochiweb, Cowboy, Inets. > 2. Create a request handler that expects to receive Zip files, which > it extracts with zip:extract(Request, [memory]). > 3. Create a zip bomb [1]: dd if=/dev/zero bs=1M count=8000 | zip req.zip - > 4. POST the small req.zip to the web server. > 5. See the VM go down in flames. > > Obviously the per-process limits would elegantly solve this problem if > they covered binaries as well. A far less elegant solution would be to > handle the unsafe decompression with an external process and open_port > (inefficient) or implement a sophisticated alternative to the zip > module which handles the limits by itself (inefficient, annoying). > > I understand that limiting the message queue / ets / NIFs can be > trickier. Just covering the basic max-heap with binaries would be a > good starting point. > > Ville > > [1] http://en.wikipedia.org/wiki/Zip_bomb > > > * Should processes be able to set limits on other processes? I think not > > though my draft argues for it. It introduces unnecessary restraints on > erts > > and hinders performance. 'save_calls' is such an option. > > > > * ets - if your table increases beyond some limit. Who should we punish? > The > > inserter? The owner? What would be the rationale? We cannot just punish > the > > inserter, the ets table is still there taking a lot of memory and no > other > > process could insert into the table. They would be killed as well. Remove > > the owner and hence the table (and potential heir)? What kind of problems > > would arise then? Limits should be tied into a supervision strategy and > > restart the whole thing. > > > > * In my draft and reference implementation I use soft limits. Once a > process > > reaches its limit it will be marked for termination by an exit signal. > The > > trouble here is there is no real guarantee for how long this will take. A > > process can continue appending a binary for a short while and ending the > > beam with OOM still. (If I remember it correctly you have to schedule > out to > > terminate a process in SMP thus you need to bump all reduction. But, not > all > > things handle return values from the garbage collector, most notably > within > > the append_binary instruction). There may be other issues as well. > > > > * Message queues. In the current implementation of message queues we have > > two queues. An inner one which is locked by the receiver process while > > executing and an outer one which other processes will use and thus not > > compete for a message queue lock with the executing process. When the > inner > > queue is depleted the receiver process will lock the outer queue and move > > the entire thing to the inner one. Rinse and repeat. The only guarantee > we > > have to ensure with our implementation is: signal order between two > > processes. So, in the future we might have several queues to improve > > performance. If you introduce monitoring of the total number messages in > the > > abstracted queue (all the queues) this will most probable kill any sort > of > > scalability. For instance a sender would not be allowed to check the > inner > > queue for this reason. Would a "fast" counter check in the inner queue be > > allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs > > performance. If we introduce even more queues for scalability reasons > this > > will cost even more. > > > > * What about other memory users? Drivers? NIFs? > > > > I do believe in increments in development as long it is path to the > > envisioned goal. > > And to reiterate, i'm not convinced that limits on just processes is the > way > > to go. I think a complete monitoring system should be envisioned, not > just > > for processes. > > > > // Bj?rn-Egil > > > > On 2013-02-06 23:03, Richard O'Keefe wrote: > > > > Just today, I saw Matthew Evans' > > > > This pertains to a feature I would like to see > > in Erlang. The ability to set an optional > > "memory limit" when a process and ETS table is > > created (and maybe a global optional per-process > > limit when the VM is started). I've seen a few > > cases where, due to software bugs, a process size > > grows and grows; unfortunately as things stand > > today the result is your entire VM crashing - > > hopefully leaving you with a crash_dump. > > > > Having such a limit could cause the process to > > terminate (producing a OOM crash report in > > erlang.log) and the crashing process could be > > handled with supervisor rules. Even better you > > can envisage setting the limits artificially low > > during testing to catch these types of bugs early on. > > > > in my mailbox. I have seen too many such e-mail messages. > > Here's a specific proposal. It's time _something_ was done > > about this kind of problem. I don't expect that my EEP is > > the best way to deal with it, but at least there's going to > > be something for people to point to. > > > > > > > > _______________________________________________ > > eeps mailing list > > eeps@REDACTED > > http://erlang.org/mailman/listinfo/eeps > > > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Fri Feb 8 00:25:48 2013 From: tony@REDACTED (Tony Rogvall) Date: Fri, 8 Feb 2013 00:25:48 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <5113C7F5.9050500@erlang.org> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> Message-ID: <2F13D417-AA59-4C2D-842C-313D693997CA@rogvall.se> For what it is worth. I located my old implementation (three years old :-) https://github.com/tonyrog/otp/tree/limits/ A very brief description (I presented some early stuff for OTP around then) I do not think that max_message_queue_len is implemented, not really defined. Too many options here. /Tony Resource Limits in Erlang ========================= # Why? - Ability to detect and kill runaway processes. - Detect and kill zombies. - Basis of safe mode framework. - Excellent to use in debugging. # How? - Limits are checked at context switch and garbage collection time. - Relatively light weight. - Create limits by using spawn_opt. - Limits are inherited by spawned processes. - If a limit is reached a signal 'system_limit' is raised. ## max_memory Limit the amount of memory a process may use. Account for all memory a process is using at any given time. This includes heap, stack, tables, links and messages. The memory is shared among all processes spawned by the process that where limited. spawn\_opt can be used to give away some of that memory to child processes. ## max\_time Controls how many milliseconds a process may use in "wall clock" time. Created (sub-)processes will only be able to run for the remaining time. ## max\_cpu Controls how many milliseconds a process may run in cpu time. The cpu time is consumed by the process and all it?s spawned (sub-)processes. ## max\_reductions A lighter version of max\_cpu. One reductions does approximately corresponds to a function call. ## max\_processes Limit number of running (sub-)processes that may be running at any given time. ## max\_ports Limit number of open ports that can be open at any given time. ## max\_tables Limit number of ets tables that may be open at any given time. ## max\_message\_queue\_len Limit the size of the message queue. Who dies when the limit is reached? Either sender or receiver? Maybe add a dangerous block option? # process\_info process\_info is used to read the current limits and the "remaining" quota. process_info(Pid, max\_cpu) is used to read the number of milliseconds set for execution while process_info(Pid, remaining\_cpu) return how many cpu milliseconds that remain to execute. The items include: max\_process, max\_ports, max\_tables, max\_memory, max\_reductions, max\_message\_queue\_len, max\_cpu, max\_time, remaining\_process, remaining\_ports, remaining_tables, remaining\_memory, remaining\_reductions, remaining\_message\_queue\_len, remaining\_cpu, remaining\_time On 7 feb 2013, at 16:27, Bj?rn-Egil Dahlberg wrote: > I dug out what I wrote a year ago .. > > eep-draft: > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md > > Reference implementation: > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 > Remember, this is a prototype and a reference implementation. > > There is a couple of issues not addressed or at least open-ended. > > * Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option. > > * ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing. > > * In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well. > > * Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even more queues for scalability reasons this will cost even more. > > * What about other memory users? Drivers? NIFs? > > I do believe in increments in development as long it is path to the envisioned goal. > And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes. > > // Bj?rn-Egil > > On 2013-02-06 23:03, Richard O'Keefe wrote: >> Just today, I saw Matthew Evans' >> >> This pertains to a feature I would like to see >> in Erlang. The ability to set an optional >> "memory limit" when a process and ETS table is >> created (and maybe a global optional per-process >> limit when the VM is started). I've seen a few >> cases where, due to software bugs, a process size >> grows and grows; unfortunately as things stand >> today the result is your entire VM crashing - >> hopefully leaving you with a crash_dump. >> >> Having such a limit could cause the process to >> terminate (producing a OOM crash report in >> erlang.log) and the crashing process could be >> handled with supervisor rules. Even better you >> can envisage setting the limits artificially low >> during testing to catch these types of bugs early on. >> >> in my mailbox. I have seen too many such e-mail messages. >> Here's a specific proposal. It's time _something_ was done >> about this kind of problem. I don't expect that my EEP is >> the best way to deal with it, but at least there's going to >> be something for people to point to. >> >> >> >> _______________________________________________ >> eeps mailing list >> eeps@REDACTED >> http://erlang.org/mailman/listinfo/eeps > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Fri Feb 8 01:02:37 2013 From: comptekki@REDACTED (Wes James) Date: Thu, 7 Feb 2013 17:02:37 -0700 Subject: [erlang-questions] run_erl and erl -boot Message-ID: I tried to get run_erl and erl -boot working (-boot offering ssl communication between nodes). I have a script that the run_erl runs fine with my pre -boot setup, but when trying to use -boot I can ping the node, but the receive loop app doesn't seem to work in this configuration. Is this supposed to work? What I done that works is use the yaws startup script (which works fine with my app), but when I add the -boot parameters (and all the cert paths, etc), the communication isn't working. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From wallentin.dahlberg@REDACTED Fri Feb 8 02:23:36 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Fri, 8 Feb 2013 02:23:36 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <2F13D417-AA59-4C2D-842C-313D693997CA@rogvall.se> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <2F13D417-AA59-4C2D-842C-313D693997CA@rogvall.se> Message-ID: 2013/2/8 Tony Rogvall > For what it is worth. > I located my old implementation (three years old :-) > ^^ > > https://github.com/tonyrog/otp/tree/limits/ > > A very brief description (I presented some early stuff for OTP around then) > I do not think that max_message_queue_len is implemented, > not really defined. Too many options here. > I forgot about that one =) I wonder if I was present at your presentation. I recall something about safe-erlang with quotas, is that the same thing? Give process groups allotment of time (reductions) and memory .. well any resource really. That is a bit better then "Don't OOM!" Hmm .. inheritance .. neat, but I don't know. Do you have any docs about your thoughts about it? // Bj?rn-Egil > > /Tony > > Resource Limits in Erlang > ========================= > > # Why? > > - Ability to detect and kill runaway processes. > - Detect and kill zombies. > - Basis of safe mode framework. > - Excellent to use in debugging. > > # How? > > - Limits are checked at context switch and garbage collection time. > - Relatively light weight. > - Create limits by using spawn_opt. > - Limits are inherited by spawned processes. > - If a limit is reached a signal 'system_limit' is raised. > > ## max_memory > Limit the amount of memory a process may use. Account for all memory a > process is using at any given time. This includes heap, stack, tables, > links and messages. The memory is shared among all processes spawned by > the > process that where limited. spawn\_opt can be used to give > away some of that memory to child processes. > > ## max\_time > Controls how many milliseconds a process may use in "wall clock" time. > Created (sub-)processes will only be able to run for the remaining time. > > ## max\_cpu > Controls how many milliseconds a process may run in cpu time. > The cpu time is consumed by the process and all it?s spawned > (sub-)processes. > > ## max\_reductions > A lighter version of max\_cpu. One reductions does approximately > corresponds > to a function call. > > ## max\_processes > Limit number of running (sub-)processes that may be running at any given > time. > > ## max\_ports > Limit number of open ports that can be open at any given time. > > ## max\_tables > Limit number of ets tables that may be open at any given time. > > ## max\_message\_queue\_len > Limit the size of the message queue. Who dies when the limit is reached? > Either sender or receiver? Maybe add a dangerous block option? > > # process\_info > > process\_info is used to read the current limits and the > "remaining" quota. > process_info(Pid, max\_cpu) is used to read the number of > milliseconds set for execution while process_info(Pid, remaining\_cpu) > return how many cpu milliseconds that remain to execute. > The items include: max\_process, max\_ports, max\_tables, max\_memory, > max\_reductions, max\_message\_queue\_len, max\_cpu, max\_time, > remaining\_process, remaining\_ports, remaining_tables, remaining\_memory, > remaining\_reductions, remaining\_message\_queue\_len, > remaining\_cpu, remaining\_time > > > On 7 feb 2013, at 16:27, Bj?rn-Egil Dahlberg wrote: > > I dug out what I wrote a year ago .. > > eep-draft: > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md > > Reference implementation: > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 > Remember, this is a prototype and a reference implementation. > > There is a couple of issues not addressed or at least open-ended. > > * Should processes be able to set limits on other processes? I think not > though my draft argues for it. It introduces unnecessary restraints on erts > and hinders performance. 'save_calls' is such an option. > > * ets - if your table increases beyond some limit. Who should we punish? > The inserter? The owner? What would be the rationale? We cannot just punish > the inserter, the ets table is still there taking a lot of memory and no > other process could insert into the table. They would be killed as well. > Remove the owner and hence the table (and potential heir)? What kind of > problems would arise then? Limits should be tied into a supervision > strategy and restart the whole thing. > > * In my draft and reference implementation I use soft limits. Once a > process reaches its limit it will be marked for termination by an exit > signal. The trouble here is there is no real guarantee for how long this > will take. A process can continue appending a binary for a short while and > ending the beam with OOM still. (If I remember it correctly you have to > schedule out to terminate a process in SMP thus you need to bump all > reduction. But, not all things handle return values from the garbage > collector, most notably within the append_binary instruction). There may be > other issues as well. > > * Message queues. In the current implementation of message queues we have > two queues. An inner one which is locked by the receiver process while > executing and an outer one which other processes will use and thus not > compete for a message queue lock with the executing process. When the inner > queue is depleted the receiver process will lock the outer queue and move > the entire thing to the inner one. Rinse and repeat. The only guarantee we > have to ensure with our implementation is: signal order between two > processes. So, in the future we might have several queues to improve > performance. If you introduce monitoring of the total number messages in > the abstracted queue (all the queues) this will most probable kill any sort > of scalability. For instance a sender would not be allowed to check the > inner queue for this reason. Would a "fast" counter check in the inner > queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping > costs performance. If we introduce even more queues for scalability reasons > this will cost even more. > > * What about other memory users? Drivers? NIFs? > > I do believe in increments in development as long it is path to the > envisioned goal. > And to reiterate, i'm not convinced that limits on just processes is the > way to go. I think a complete monitoring system should be envisioned, not > just for processes. > > // Bj?rn-Egil > > On 2013-02-06 23:03, Richard O'Keefe wrote: > > Just today, I saw Matthew Evans' > > This pertains to a feature I would like to see > in Erlang. The ability to set an optional > "memory limit" when a process and ETS table is > created (and maybe a global optional per-process > limit when the VM is started). I've seen a few > cases where, due to software bugs, a process size > grows and grows; unfortunately as things stand > today the result is your entire VM crashing - > hopefully leaving you with a crash_dump. > > Having such a limit could cause the process to > terminate (producing a OOM crash report in > erlang.log) and the crashing process could be > handled with supervisor rules. Even better you > can envisage setting the limits artificially low > during testing to catch these types of bugs early on. > > in my mailbox. I have seen too many such e-mail messages. > Here's a specific proposal. It's time _something_ was done > about this kind of problem. I don't expect that my EEP is > the best way to deal with it, but at least there's going to > be something for people to point to. > > > > > _______________________________________________ > eeps mailing listeeps@REDACTED://erlang.org/mailman/listinfo/eeps > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > "Installing applications can lead to corruption over time. Applications > gradually write over each other's libraries, partial upgrades occur, user > and system errors happen, and minute changes may be unnoticeable and > difficult to fix" > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuulos@REDACTED Fri Feb 8 02:28:06 2013 From: tuulos@REDACTED (Ville Tuulos) Date: Thu, 7 Feb 2013 17:28:06 -0800 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> Message-ID: On Thu, Feb 7, 2013 at 2:24 PM, Bj?rn-Egil Dahlberg wrote: > 2013/2/7 Ville Tuulos >> >> On Thu, Feb 7, 2013 at 7:27 AM, Bj?rn-Egil Dahlberg >> wrote: >> > I dug out what I wrote a year ago .. >> > >> > eep-draft: >> > >> > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md >> > >> > Reference implementation: >> > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 >> > Remember, this is a prototype and a reference implementation. >> > >> > There is a couple of issues not addressed or at least open-ended. >> >> Looks great! I truly hope this will get accepted rather sooner than >> later, at least as an experimental feature. > > > Well as I said previously, I still have some issues with this approach. > Envision the goal and then see if this is a good first step. > Meaning: if we want to include other resource limits that is not process > oriented how would we then design the interface then? A resource limits > interface perhaps? The general case is a can of worms. It took 5-6 years for Linux to get process containers [1] in a decent shape. I hope that a consensus could be reached that a relatively simple max-heap limit would be useful enough as such for now, without having to worry about all the other cases (ref. Tony's email). The interface could be pretty much as in EEP 0042. If ever someone had courage to propose a sane, generic resource limit interface, maybe after the max-heap had been battle-hardened over several years, it shouldn't be too difficult to unify / deprecate the old options. >> Does the proposal cover refc binaries as well? Almost all interesting >> use cases that I can imagine for limits involve binaries. > > > My implementation doesn't cover it. It covers process memory blocks, i.e. > everything except refc-binaries. (well their headers are covered: procbins) > > Binaries are special beasts and needs special care. The easiest way to > implement refc binary limits would be to use the virtual binary heap concept > already present in the gc. > > By using vheaps we would count all the memory referenced by procbins in the > process heap. Now, here it gets interesting. Several procbins may refer to > the same refc binary blob and that memory would be counted several times. > You would have to take special accounting care if you want refc uniqueness > which would make it kind of expensive. You can't mark the binary as counted > since other processes might be gc:ed and they might also reference it so you > would have to make some other accommodations. > > Other than that you can just use the same idea as heap_size limits. The > checks are similar and the api could be too: erlang:process_flag(limits, > [{bin_vheap_size, Size}]). If I understood correctly, bin_vheap_size sounds exactly what I would need. I think the semantics should be strictly related to _creation_ of binaries, not sharing. The idea is to block a rogue process from creating tons of binaries, which is hard / impossible to do now. On the other hand, it is easy for a process to discard large binaries from its inbox, if it so wishes. The limits could either ignore binaries whose reference count > 1 or count them only against the limit of the process that originally created it, if possible. I guess I would need to read the source to understand the issue with double-counting :) One optimization could be that if the pessimistic assumption that every procbin refers to a separate refc blob results to a number that is less than the limit, we know that there is nothing to worry about, which should be the typical case. Ville [1] http://lwn.net/Articles/236038/ > // Bj?rn-Egil > >> >> Here's one test case: >> >> 1. Create a web server in Erlang e.g. with Mochiweb, Cowboy, Inets. >> 2. Create a request handler that expects to receive Zip files, which >> it extracts with zip:extract(Request, [memory]). >> 3. Create a zip bomb [1]: dd if=/dev/zero bs=1M count=8000 | zip req.zip - >> 4. POST the small req.zip to the web server. >> 5. See the VM go down in flames. >> >> Obviously the per-process limits would elegantly solve this problem if >> they covered binaries as well. A far less elegant solution would be to >> handle the unsafe decompression with an external process and open_port >> (inefficient) or implement a sophisticated alternative to the zip >> module which handles the limits by itself (inefficient, annoying). >> >> I understand that limiting the message queue / ets / NIFs can be >> trickier. Just covering the basic max-heap with binaries would be a >> good starting point. >> >> Ville >> >> [1] http://en.wikipedia.org/wiki/Zip_bomb >> >> > * Should processes be able to set limits on other processes? I think not >> > though my draft argues for it. It introduces unnecessary restraints on >> > erts >> > and hinders performance. 'save_calls' is such an option. >> > >> > * ets - if your table increases beyond some limit. Who should we punish? >> > The >> > inserter? The owner? What would be the rationale? We cannot just punish >> > the >> > inserter, the ets table is still there taking a lot of memory and no >> > other >> > process could insert into the table. They would be killed as well. >> > Remove >> > the owner and hence the table (and potential heir)? What kind of >> > problems >> > would arise then? Limits should be tied into a supervision strategy and >> > restart the whole thing. >> > >> > * In my draft and reference implementation I use soft limits. Once a >> > process >> > reaches its limit it will be marked for termination by an exit signal. >> > The >> > trouble here is there is no real guarantee for how long this will take. >> > A >> > process can continue appending a binary for a short while and ending the >> > beam with OOM still. (If I remember it correctly you have to schedule >> > out to >> > terminate a process in SMP thus you need to bump all reduction. But, not >> > all >> > things handle return values from the garbage collector, most notably >> > within >> > the append_binary instruction). There may be other issues as well. >> > >> > * Message queues. In the current implementation of message queues we >> > have >> > two queues. An inner one which is locked by the receiver process while >> > executing and an outer one which other processes will use and thus not >> > compete for a message queue lock with the executing process. When the >> > inner >> > queue is depleted the receiver process will lock the outer queue and >> > move >> > the entire thing to the inner one. Rinse and repeat. The only guarantee >> > we >> > have to ensure with our implementation is: signal order between two >> > processes. So, in the future we might have several queues to improve >> > performance. If you introduce monitoring of the total number messages in >> > the >> > abstracted queue (all the queues) this will most probable kill any sort >> > of >> > scalability. For instance a sender would not be allowed to check the >> > inner >> > queue for this reason. Would a "fast" counter check in the inner queue >> > be >> > allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs >> > performance. If we introduce even more queues for scalability reasons >> > this >> > will cost even more. >> > >> > * What about other memory users? Drivers? NIFs? >> > >> > I do believe in increments in development as long it is path to the >> > envisioned goal. >> > And to reiterate, i'm not convinced that limits on just processes is the >> > way >> > to go. I think a complete monitoring system should be envisioned, not >> > just >> > for processes. >> > >> > // Bj?rn-Egil >> > >> > On 2013-02-06 23:03, Richard O'Keefe wrote: >> > >> > Just today, I saw Matthew Evans' >> > >> > This pertains to a feature I would like to see >> > in Erlang. The ability to set an optional >> > "memory limit" when a process and ETS table is >> > created (and maybe a global optional per-process >> > limit when the VM is started). I've seen a few >> > cases where, due to software bugs, a process size >> > grows and grows; unfortunately as things stand >> > today the result is your entire VM crashing - >> > hopefully leaving you with a crash_dump. >> > >> > Having such a limit could cause the process to >> > terminate (producing a OOM crash report in >> > erlang.log) and the crashing process could be >> > handled with supervisor rules. Even better you >> > can envisage setting the limits artificially low >> > during testing to catch these types of bugs early on. >> > >> > in my mailbox. I have seen too many such e-mail messages. >> > Here's a specific proposal. It's time _something_ was done >> > about this kind of problem. I don't expect that my EEP is >> > the best way to deal with it, but at least there's going to >> > be something for people to point to. >> > >> > >> > >> > _______________________________________________ >> > eeps mailing list >> > eeps@REDACTED >> > http://erlang.org/mailman/listinfo/eeps >> > >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > From mjtruog@REDACTED Fri Feb 8 03:01:31 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 07 Feb 2013 18:01:31 -0800 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <5113C7F5.9050500@erlang.org> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> Message-ID: <51145C7B.2030806@gmail.com> An idea which seems like it should be included in this EEP, is simply a function to determine how much real memory is consumed by an erlang variable. I understand we have erlang:external_size which provides the size based on the external term format. I also understand that the memory pools make erlang consume more than the size that would be returned. There is some documentation on memory consumption of erlang data here: http://www.erlang.org/doc/efficiency_guide/advanced.html#id68921 . I attempted to use an older version of this documentation to write my own function, but ran into problems (created output that seemed incorrect) and taking into account HiPE data is a separate part that would generate different results. So, these erlang limits are great additions, but it seems like we would need a function for understanding where the limits are broken, for debugging and development (and possibly dynamic runtime changes). It would be interesting if the function was allowed in guard tests, which might help for embedded development. Thanks, Michael On 02/07/2013 07:27 AM, Bj?rn-Egil Dahlberg wrote: > I dug out what I wrote a year ago .. > > eep-draft: > https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md > > Reference implementation: > https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 > Remember, this is a prototype and a reference implementation. > > There is a couple of issues not addressed or at least open-ended. > > * Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option. > > * ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing. > > * In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well. > > * Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even > more queues for scalability reasons this will cost even more. > > * What about other memory users? Drivers? NIFs? > > I do believe in increments in development as long it is path to the envisioned goal. > And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes. > > // Bj?rn-Egil > > On 2013-02-06 23:03, Richard O'Keefe wrote: >> Just today, I saw Matthew Evans' >> >> This pertains to a feature I would like to see >> in Erlang. The ability to set an optional >> "memory limit" when a process and ETS table is >> created (and maybe a global optional per-process >> limit when the VM is started). I've seen a few >> cases where, due to software bugs, a process size >> grows and grows; unfortunately as things stand >> today the result is your entire VM crashing - >> hopefully leaving you with a crash_dump. >> >> Having such a limit could cause the process to >> terminate (producing a OOM crash report in >> erlang.log) and the crashing process could be >> handled with supervisor rules. Even better you >> can envisage setting the limits artificially low >> during testing to catch these types of bugs early on. >> >> in my mailbox. I have seen too many such e-mail messages. >> Here's a specific proposal. It's time _something_ was done >> about this kind of problem. I don't expect that my EEP is >> the best way to deal with it, but at least there's going to >> be something for people to point to. >> >> >> >> _______________________________________________ >> eeps mailing list >> eeps@REDACTED >> http://erlang.org/mailman/listinfo/eeps > > > > _______________________________________________ > eeps mailing list > eeps@REDACTED > http://erlang.org/mailman/listinfo/eeps -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Feb 8 03:39:55 2013 From: ok@REDACTED (ok@REDACTED) Date: Fri, 8 Feb 2013 15:39:55 +1300 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <2F13D417-AA59-4C2D-842C-313D693997CA@rogvall.se> Message-ID: > > Hmm .. inheritance .. neat, but I don't know. It's not clear how much sense inheritance makes. There have been concurrent systems in which a process is born with some share of system resources and can portion its share out amongst its children, so that a bound on memory or CPU time or whatever really _is_ a bound on the total resource consumption on and on behalf of that activity. Then you run into issues like having a dead child return unused resources to its parent's account. This protects against malice. If inheritance is just inheritance, then a limited process can consume unbounded amounts of memory by creating lots of children. This protects against accident, sort of, but has no protection against malice. From ok@REDACTED Fri Feb 8 04:04:38 2013 From: ok@REDACTED (ok@REDACTED) Date: Fri, 8 Feb 2013 16:04:38 +1300 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <51145C7B.2030806@gmail.com> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> Message-ID: <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> > An idea which seems like it should be included in this EEP, is simply a > function to determine how much real memory is consumed by an erlang > variable. Given a language with immutable data structures and free sharing, I am having a hard time trying to think what this might mean. Consider as one of the *easy* cases L0 = [], L1 = [L0|L0], ... L99 = [L98|L98] The amount of memory it really uses is just 99 cons cells, presumably 198 words. The amount that will be calculated by a recursive sizer is about 2^99 (approximately). There are at least the following notions: - size in external representation - amount copied when sent to a local PID - amount found by a recursive walker - amount used taking internal cycles into account - fair share considering all references in the current PID - fair share including references to binaries from other PIDs. It seems to me that there is one thing that _could_ be done but would be a fair bit of work, and that's allocation profiling. Arrange to generate different BEAM code which records each (amount of memory, source location where allocation happens) so that you can run a test case and see which expressions are responsible for allocating how much memory. From wallentin.dahlberg@REDACTED Fri Feb 8 04:27:54 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Fri, 8 Feb 2013 04:27:54 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> Message-ID: 2013/2/8 > > An idea which seems like it should be included in this EEP, is simply a > > function to determine how much real memory is consumed by an erlang > > variable. > > Given a language with immutable data structures and free sharing, > I am having a hard time trying to think what this might mean. > Consider as one of the *easy* cases > > L0 = [], > L1 = [L0|L0], > ... > L99 = [L98|L98] > > The amount of memory it really uses is just 99 cons cells, > presumably 198 words. The amount that will be calculated > by a recursive sizer is about 2^99 (approximately). > Btw, If its just the matter of memory then there exists erts_debug:size/1 and erts_debug:flat_size/1 (calculates number of words): Eshell V5.9 (abort with ^G) 1> A = {1,2,3}. {1,2,3} 2> B = [A|A]. [{1,2,3}|{1,2,3}] 3> erts_debug:size(B). 6 4> erts_debug:flat_size(B). 10 > > There are at least the following notions: > - size in external representation > - amount copied when sent to a local PID > - amount found by a recursive walker > - amount used taking internal cycles into account > - fair share considering all references in the current PID > - fair share including references to binaries from other PIDs. > > It seems to me that there is one thing that _could_ be done > but would be a fair bit of work, and that's allocation > profiling. Arrange to generate different BEAM code which > records each (amount of memory, source location where > allocation happens) so that you can run a test case and see > which expressions are responsible for allocating how much > memory. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Feb 8 04:36:29 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 07 Feb 2013 19:36:29 -0800 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> Message-ID: <511472BD.8090200@gmail.com> On 02/07/2013 07:27 PM, Bj?rn-Egil Dahlberg wrote: > > > 2013/2/8 > > > > An idea which seems like it should be included in this EEP, is simply a > > function to determine how much real memory is consumed by an erlang > > variable. > > Given a language with immutable data structures and free sharing, > I am having a hard time trying to think what this might mean. > Consider as one of the *easy* cases > > L0 = [], > L1 = [L0|L0], > ... > L99 = [L98|L98] > > The amount of memory it really uses is just 99 cons cells, > presumably 198 words. The amount that will be calculated > by a recursive sizer is about 2^99 (approximately). > > > Btw, If its just the matter of memory then there exists erts_debug:size/1 and erts_debug:flat_size/1 (calculates number of words): > > Eshell V5.9 (abort with ^G) > 1> A = {1,2,3}. > {1,2,3} > 2> B = [A|A]. > [{1,2,3}|{1,2,3}] > 3> erts_debug:size(B). > 6 > 4> erts_debug:flat_size(B). > 10 > That is what I was looking for. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Feb 8 04:47:13 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 07 Feb 2013 19:47:13 -0800 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> Message-ID: <51147541.7060303@gmail.com> On 02/07/2013 07:04 PM, ok@REDACTED wrote: >> An idea which seems like it should be included in this EEP, is simply a >> function to determine how much real memory is consumed by an erlang >> variable. > Given a language with immutable data structures and free sharing, > I am having a hard time trying to think what this might mean. > Consider as one of the *easy* cases > > L0 = [], > L1 = [L0|L0], > ... > L99 = [L98|L98] > > The amount of memory it really uses is just 99 cons cells, > presumably 198 words. The amount that will be calculated > by a recursive sizer is about 2^99 (approximately). > > There are at least the following notions: > - size in external representation > - amount copied when sent to a local PID > - amount found by a recursive walker > - amount used taking internal cycles into account > - fair share considering all references in the current PID > - fair share including references to binaries from other PIDs. > > It seems to me that there is one thing that _could_ be done > but would be a fair bit of work, and that's allocation > profiling. Arrange to generate different BEAM code which > records each (amount of memory, source location where > allocation happens) so that you can run a test case and see > which expressions are responsible for allocating how much > memory. The erts_debug:size/1 that Bj?rn-Egil mentioned is really what I want, but we would need it to be documented to have this low-level view of memory consumption be official (it would also be cool if it could be mentioned in the memory portion of the efficiency guide). I know the view of the allocators with profiling can be done with instrument and erts_alloc_config, which provides a high-level view of memory usage by processes and allocators. So, that part seems fine, though it is pretty complex. Having a function like erts_debug:size/1 can help the developer avoid all the allocation complexity, and just focus on potential memory problems when errors occur, which seems like the next step after this EEP would be implemented. From ccfoley@REDACTED Fri Feb 8 09:59:28 2013 From: ccfoley@REDACTED (Christopher Foley) Date: Fri, 08 Feb 2013 08:59:28 +0000 Subject: [erlang-questions] Writing a DSL in ERlang Message-ID: <5114BE70.4060308@gmail.com> Hi, I'm am interested in creating an external DSL with Erlang and what are the steps involved. So very basically I want to convert some English like language into running erlang code. My goal is to build up a full erlang source file relative to the dsl file I have as input. I don't necessarily need to print out the source file to the file system, holding it in memory is sufficient and allowing its execution. Sample DSL syntax; {createhandler, "TestHandler", {handles, [ {x, {action, printToScreen}}, {y, {action, writeToStorage}} ]} } So from above I want to create a gen_event handler with a handle_event function which pattern matches on 'x' and 'y' and performs the actions defined. I have used this to just show what I would like to do, what I'm interested in is the steps and approaches to doing this. The steps I see are as follows: 1. Parse the dsl text file, create different tokens. I presume that I have to write my own parser as its dependent on the structure of the dsl language itself? 2. Match these tokens to chunks of erlang code, is this correct? What is best approach here; create a 'fun' for each chunk of code. How would I then simply put these funs together to represent the overall module? 3. Once I have the code combined into one entity(representing a module), then I can presumably use the compile module. I am only coming back to writing erlang (ten years since I coded in erlang), so apologies if questions are fairly basic. Any info appreciated. Chris. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gustav.simonsson@REDACTED Fri Feb 8 10:54:40 2013 From: gustav.simonsson@REDACTED (Gustav Simonsson) Date: Fri, 8 Feb 2013 10:54:40 +0100 Subject: [erlang-questions] Writing a DSL in ERlang In-Reply-To: <5114BE70.4060308@gmail.com> References: <5114BE70.4060308@gmail.com> Message-ID: Hi, Some links to OTP stdlib modules which you might find useful: http://www.erlang.org/doc/man/erl_parse.html http://www.erlang.org/doc/man/erl_scan.html http://www.erlang.org/doc/man/erl_eval.html http://erlang.org/doc/man/erl_syntax.html Cheers, Gustav On Fri, Feb 8, 2013 at 9:59 AM, Christopher Foley wrote: > Hi, > > I'm am interested in creating an external DSL with Erlang and what are the > steps involved. So very basically I want to convert some English like > language into running erlang code. My goal is to build up a full erlang > source file relative to the dsl file I have as input. I don't necessarily > need to print out the source file to the file system, holding it in memory > is sufficient and allowing its execution. > > Sample DSL syntax; {createhandler, "TestHandler", {handles, [ {x, {action, > printToScreen}}, {y, {action, writeToStorage}} ]} } > > So from above I want to create a gen_event handler with a handle_event > function which pattern matches on 'x' and 'y' and performs the actions > defined. I have used this to just show what I would like to do, what I'm > interested in is the steps and approaches to doing this. > > The steps I see are as follows: > > 1. Parse the dsl text file, create different tokens. I presume that I > have to write my own parser as its dependent on the structure of the dsl > language itself? > 2. Match these tokens to chunks of erlang code, is this correct? What > is best approach here; create a 'fun' for each chunk of code. How would I > then simply put these funs together to represent the overall module? > 3. Once I have the code combined into one entity(representing a > module), then I can presumably use the compile module. > > I am only coming back to writing erlang (ten years since I coded in > erlang), so apologies if questions are fairly basic. Any info appreciated. > > Chris. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hq@REDACTED Fri Feb 8 12:13:59 2013 From: hq@REDACTED (Adam Rutkowski) Date: Fri, 8 Feb 2013 12:13:59 +0100 Subject: [erlang-questions] Writing a DSL in ERlang In-Reply-To: <5114BE70.4060308@gmail.com> References: <5114BE70.4060308@gmail.com> Message-ID: On 8 Feb 2013, at 09:59, Christopher Foley wrote: > Hi, > > I'm am interested in creating an external DSL with Erlang and what are the steps involved. So very basically I want to convert some English like language into running erlang code. My goal is to build up a full erlang source file relative to the dsl file I have as input. I don't necessarily need to print out the source file to the file system, holding it in memory is sufficient and allowing its execution. > > Sample DSL syntax; {createhandler, "TestHandler", {handles, [ {x, {action, printToScreen}}, {y, {action, writeToStorage}} ]} } > > So from above I want to create a gen_event handler with a handle_event function which pattern matches on 'x' and 'y' and performs the actions defined. I have used this to just show what I would like to do, what I'm interested in is the steps and approaches to doing this. > > The steps I see are as follows: > ? Parse the dsl text file, create different tokens. I presume that I have to write my own parser as its dependent on the structure of the dsl language itself? > ? Match these tokens to chunks of erlang code, is this correct? What is best approach here; create a 'fun' for each chunk of code. How would I then simply put these funs together to represent the overall module? > ? Once I have the code combined into one entity(representing a module), then I can presumably use the compile module. > I am only coming back to writing erlang (ten years since I coded in erlang), so apologies if questions are fairly basic. Any info appreciated. > Chris. Take a look at smerl - https://github.com/yariv/erlyweb/blob/master/src/smerl/smerl.erl -- Adam From carlsson.richard@REDACTED Fri Feb 8 14:18:53 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Fri, 08 Feb 2013 14:18:53 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <51147541.7060303@gmail.com> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> <51147541.7060303@gmail.com> Message-ID: <5114FB3D.9020301@gmail.com> On 02/08/2013 04:47 AM, Michael Truog wrote: > The erts_debug:size/1 that Bj?rn-Egil mentioned is really what I want, but we would need it to be documented to have this low-level view of memory consumption be official (it would also be cool if it could be mentioned in the memory portion of the efficiency guide). Also, it's currently very slow on large structures, being implemented in Erlang (using gb_trees and debugging BIFs like erts_debug:same/2). Do not use in production if the structures may be huge. The erts_debug:flat_size/1 function is fast and implemented in C, but doesn't give you the true size since it ignores sharing. If Kostis Sagonas' work on sharing-preserving copying operations was included in OTP, we could have an efficient version of erts_debug:size/1. /Richard From egil@REDACTED Fri Feb 8 14:29:01 2013 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Fri, 8 Feb 2013 14:29:01 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <5114FB3D.9020301@gmail.com> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> <51147541.7060303@gmail.com> <5114FB3D.9020301@gmail.com> Message-ID: <5114FD9D.3080304@erlang.org> On 2013-02-08 14:18, Richard Carlsson wrote: > On 02/08/2013 04:47 AM, Michael Truog wrote: >> The erts_debug:size/1 that Bj?rn-Egil mentioned is really what I >> want, but we would need it to be documented to have this low-level >> view of memory consumption be official (it would also be cool if it >> could be mentioned in the memory portion of the efficiency guide). > > Also, it's currently very slow on large structures, being implemented > in Erlang (using gb_trees and debugging BIFs like erts_debug:same/2). > Do not use in production if the structures may be huge. Good point! I would say don't use it in production *period*. As the name suggests it's used for debugging/development purposes. > > The erts_debug:flat_size/1 function is fast and implemented in C, but > doesn't give you the true size since it ignores sharing. If Kostis > Sagonas' work on sharing-preserving copying operations was included in > OTP, we could have an efficient version of erts_debug:size/1. I haven't seen the final implementation yet. I wonder if it's complete? Preserved sharing in copy would be really nice as long as it is efficient. // Bj?rn-Egil From icfp.publicity@REDACTED Fri Feb 8 19:59:51 2013 From: icfp.publicity@REDACTED (David Van Horn) Date: Fri, 8 Feb 2013 13:59:51 -0500 Subject: [erlang-questions] ICFP 2013: Call for papers Message-ID: ===================================================================== 18th ACM SIGPLAN International Conference on Functional Programming ICFP 2013 Boston, MA, USA, 25-27 September 2013 http://www.icfpconference.org/icfp2013 ===================================================================== Important Dates ~~~~~~~~~~~~~~~ Submissions due: Thursday, 28 March 2013 23:59 UTC-11 (Pago Pago, American Samoa, time) Author response: Wednesday, 22 May 0:00 UTC-11 Friday, 24 May 2013 23:59 UTC-11 Notification: Friday, 7 June 2013 Final copy due: Friday, 5 July 2013 Scope ~~~~~ ICFP 2013 seeks original papers on the art and science of functional programming. Submissions are invited on all topics from principles to practice, from foundations to features, and from abstraction to application. The scope includes all languages that encourage functional programming, including both purely applicative and imperative languages, as well as languages with objects, concurrency, or parallelism. Topics of interest include (but are not limited to): * Language Design: concurrency and distribution; modules; components and composition; metaprogramming; interoperability; type systems; relations to imperative, object-oriented, or logic programming * Implementation: abstract machines; virtual machines; interpretation; compilation; compile-time and run-time optimization; memory management; multi-threading; exploiting parallel hardware; interfaces to foreign functions, services, components, or low-level machine resources * Software-Development Techniques: algorithms and data structures; design patterns; specification; verification; validation; proof assistants; debugging; testing; tracing; profiling * Foundations: formal semantics; lambda calculus; rewriting; type theory; monads; continuations; control; state; effects; program verification; dependent types * Analysis and Transformation: control-flow; data-flow; abstract interpretation; partial evaluation; program calculation * Applications and Domain-Specific Languages: symbolic computing; formal-methods tools; artificial intelligence; systems programming; distributed-systems and web programming; hardware design; databases; XML processing; scientific and numerical computing; graphical user interfaces; multimedia programming; scripting; system administration; security * Education: teaching introductory programming; parallel programming; mathematical proof; algebra * Functional Pearls: elegant, instructive, and fun essays on functional programming * Experience Reports: short papers that provide evidence that functional programming really works or describe obstacles that have kept it from working If you are concerned about the appropriateness of some topic, do not hesitate to contact the program chair. Abbreviated instructions for authors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * By Thursday, 28 March 2013, 23:59 UTC-11 (American Samoa time), submit a full paper of at most 12 pages (6 pages for an Experience Report), including bibliography and figures. The deadlines will be strictly enforced and papers exceeding the page limits will be summarily rejected. * Authors have the option to attach supplementary material to a submission, on the understanding that reviewers may choose not to look at it. * Each submission must adhere to SIGPLAN's republication policy, as explained on the web at http://www.acm.org/sigplan/republicationpolicy.htm * Authors of resubmitted (but previously rejected) papers have the option to attach an annotated copy of the reviews of their previous submission(s), explaining how they have addressed these previous reviews in the present submission. If a reviewer identifies him/herself as a reviewer of this previous submission and wishes to see how his/her comments have been addressed, the program chair will communicate to this reviewer the annotated copy of his/her previous review. Otherwise, no reviewer will read the annotated copies of the previous reviews. Overall, a submission will be evaluated according to its relevance, correctness, significance, originality, and clarity. It should explain its contributions in both general and technical terms, clearly identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. The technical content should be accessible to a broad audience. Functional Pearls and Experience Reports are separate categories of papers that need not report original research results and must be marked as such at the time of submission. Detailed guidelines on both categories are on the conference web site. Proceedings will be published by ACM Press. Authors of accepted submissions are expected to transfer the copyright to the ACM. Presentations will be videotaped and released online if the presenter consents. Formatting: Submissions must be in PDF format printable in black and white on US Letter sized paper and interpretable by Ghostscript. Papers must adhere to the standard ACM conference format: two columns, nine-point font on a ten-point baseline, with columns 20pc (3.33in) wide and 54pc (9in) tall, with a column gutter of 2pc (0.33in). A suitable document template for LaTeX is available:http://www.acm.org/sigs/sigplan/authorInformation.htm Submission: Submissions will be accepted on the web athttps://www.easychair.org/conferences/?conf=icfp2013 . Improved versions of a paper may be submitted at any point before the submission deadline using the same web interface. Author response: Authors will have a 72-hour period, starting at 0:00 UTC-11 on Wednesday, 22 May 2013, to read reviews and respond to them. Special Journal Issue: There will be a special issue of the Journal of Functional Programming with papers from ICFP 2013. The program committee will invite the authors of select accepted papers to submit a journal version to this issue. General Chair: Greg Morrisett, Harvard University Program Chair: Tarmo Uustalu, Institute of Cybernetics, Tallinn Program Committee: Thorsten Altenkirch, University of Nottingham Olaf Chitil, University of Kent Silvia Ghilezan, University of Novi Sad Michael Hanus, Christian-Albrechts-Universit?t zu Kiel Fritz Henglein, University of Copenhagen Mauro Jaskelioff, Universidad Nacional de Rosario Alan Jeffrey, Alcatel-Lucent Bell Labs Shin-ya Katsumata, Kyoto University Shriram Krishnamurthi, Brown University John Launchbury, Galois Ryan Newton, Indiana University Sungwoo Park, Pohang University of Science and Technology Sam Staton, University of Cambridge Nikhil Swamy, Microsoft Research, Redmond, WA Dimitrios Vytiniotis, Microsoft Research, Cambridge -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilya.khlopotov@REDACTED Fri Feb 8 20:05:37 2013 From: ilya.khlopotov@REDACTED (ILYA Khlopotov) Date: Fri, 8 Feb 2013 11:05:37 -0800 Subject: [erlang-questions] Writing a DSL in ERlang (ILYA Khlopotov) Message-ID: You might want to consider elixir for this simple task. Since it compiles into beam you can use generated modules from erlang. If you really want erlang code you could extract generated erlang from compiled beam (you need debug_info for that though). I know that it's not elixir mailing list so I'm apologize for providing a solution in this language. However it might save a lot of effort compared to writing DSL in erlang. Your dsl code might look like (for complete code see https://gist.github.com/khia/4741045): defmacro action(match, [do: body]) do quote do def handle_info(unquote(match), state) do unquote(body) {:ok, state} end end end defmacro createhandler(name, [do: body]) do quote do defmodule unquote(name) do unquote(body) end end end Having that you can write your definition files using your own macro (action and createhandler) as: createhandler TestHandler do action {:x, x}, do: IO.puts "x: #{inspect x}" action {:y, y, seed} do # example of calling erlang modules :random.seed(seed) value = :random.uniform(y) IO.puts "random.uniform(${y}) -> #{value} " end end If your actions can be a single expression you can go further and define macro for ::: (or something else) and then you might be able turn it into (WARNING pseudo code) createhandler TestHandler do {:x, x} ::: IO.puts("x: #{inspect x}") {:y, y} ::: IO.puts("random.uniform(${y}) -> #{value} ") end Best regards, ILYA > I'm am interested in creating an external DSL with Erlang and what are > the steps involved. So very basically I want to convert some English > like language into running erlang code. My goal is to build up a full > erlang source file relative to the dsl file I have as input. I don't > necessarily need to print out the source file to the file system, > holding it in memory is sufficient and allowing its execution. > Sample DSL syntax; {createhandler, "TestHandler", {handles, [ {x, > {action, printToScreen}}, {y, {action, writeToStorage}} ]} } > So from above I want to create a gen_event handler with a handle_event > function which pattern matches on 'x' and 'y' and performs the actions > defined. I have used this to just show what I would like to do, what I'm > interested in is the steps and approaches to doing this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Fri Feb 8 23:01:42 2013 From: comptekki@REDACTED (Wes James) Date: Fri, 8 Feb 2013 15:01:42 -0700 Subject: [erlang-questions] two questions -- -boot in erlsrv and .erlang.cookies on windows Message-ID: I have -boot working on linux. Now I'm trying to get it to work with erlsrv for windows. I put the -boot in the args part of erlsrv, but it doesn't seem to take. Any ideas how to get this to work? Also, I've tried several different folders for .cookie.erlang on windows and I can't get it to take. Even putting it in my c:\Users\username and run the erlang app, it doesn't seem to be picked up. If I run with -setcookie, it works fine. I run "set" on windows in cmd.exe and see the HOME var is set. Also, if I can get this to work, where would the .erlang.cookie go when run as a service. I just ran erl.exe and it did not create an .erlang.cookie in my c:\Users\username folder (after I moved one that I had manually put there). Is it supposed to create one on windows? Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Fri Feb 8 23:09:07 2013 From: steve@REDACTED (Steve Strong) Date: Fri, 8 Feb 2013 23:09:07 +0100 Subject: [erlang-questions] binary:compile_pattern - I'm very confused! Message-ID: Hi, I've got a binary split that I'm doing fairly regularly, so I wanted to use a compiled pattern to make it more efficient. I don't have any state floating around (it's very much a utility function), so my plan was to compile the pattern in an erl shell and just paste the resultant term directly into my source code (with an appropriate comment!). However, things are not as they seem: Erlang R15B02 (erts-5.9.2) [source] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.2 (abort with ^G) 1> C = binary:compile_pattern(<<0,0,1,0>>). {bm,<<>>} 2> X = <<2,3,4,0,0,1,0,5,6,7>>. <<2,3,4,0,0,1,0,5,6,7>> 3> binary:split(X, C). [<<2,3,4>>,<<5,6,7>>] 4> binary:split(X, {bm,<<>>}). ** exception error: bad argument in function binary:split/3 (binary.erl, line 84) 5> D = binary_to_term(term_to_binary(C)). {bm,<<>>} 6> C = D. {bm,<<>>} 7> D =:= C. true 8> binary:split(X, D). ** exception error: bad argument in function binary:split/3 (binary.erl, line 84) 9> The above should be fairly self-explanatory - I'm very confused and either being particularly dim tonight, or there is some dark magic at work :) Can anyone shed any light? Cheers, Steve -- Steve Strong Sent with Sparrow (http://www.sparrowmailapp.com/?sig) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Feb 8 23:36:02 2013 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 8 Feb 2013 23:36:02 +0100 Subject: [erlang-questions] binary:compile_pattern - I'm very confused! In-Reply-To: References: Message-ID: There is dark magic at work. What you got there is what is referred to as a "magic" binary. It represents some internal state within the erlang emulator and is used by binary:compile_pattern and nifs. You will have to store it someplace else (like you state) and fetch it from there when you are doing the split. On Fri, Feb 8, 2013 at 11:09 PM, Steve Strong wrote: > Hi, > > I've got a binary split that I'm doing fairly regularly, so I wanted to > use a compiled pattern to make it more efficient. I don't have any state > floating around (it's very much a utility function), so my plan was to > compile the pattern in an erl shell and just paste the resultant term > directly into my source code (with an appropriate comment!). However, > things are not as they seem: > > Erlang R15B02 (erts-5.9.2) [source] [smp:8:8] [async-threads:0] [hipe] > [kernel-poll:false] > > Eshell V5.9.2 (abort with ^G) > 1> C = binary:compile_pattern(<<0,0,1,0>>). > {bm,<<>>} > 2> X = <<2,3,4,0,0,1,0,5,6,7>>. > <<2,3,4,0,0,1,0,5,6,7>> > 3> binary:split(X, C). > [<<2,3,4>>,<<5,6,7>>] > 4> binary:split(X, {bm,<<>>}). > ** exception error: bad argument > in function binary:split/3 (binary.erl, line 84) > 5> D = binary_to_term(term_to_binary(C)). > {bm,<<>>} > 6> C = D. > {bm,<<>>} > 7> D =:= C. > true > 8> binary:split(X, D). > ** exception error: bad argument > in function binary:split/3 (binary.erl, line 84) > 9> > > The above should be fairly self-explanatory - I'm very confused and either > being particularly dim tonight, or there is some dark magic at work :) Can > anyone shed any light? > > Cheers, > > Steve > > -- > Steve Strong > Sent with Sparrow > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Fri Feb 8 23:36:17 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Fri, 8 Feb 2013 17:36:17 -0500 Subject: [erlang-questions] binary:compile_pattern - I'm very confused! In-Reply-To: References: Message-ID: On Fri, Feb 8, 2013 at 5:09 PM, Steve Strong wrote: > Hi, > > I've got a binary split that I'm doing fairly regularly, so I wanted to > use a compiled pattern to make it more efficient. I don't have any state > floating around (it's very much a utility function), so my plan was to > compile the pattern in an erl shell and just paste the resultant term > directly into my source code (with an appropriate comment!). However, > things are not as they seem: > > Erlang R15B02 (erts-5.9.2) [source] [smp:8:8] [async-threads:0] [hipe] > [kernel-poll:false] > > Eshell V5.9.2 (abort with ^G) > 1> C = binary:compile_pattern(<<0,0,1,0>>). > {bm,<<>>} > On the Erlang binary module man page ( http://www.erlang.org/doc/man/binary.html) it describes the return value of compile_pattern as follows: "Opaque data-type representing a compiled search-pattern. Guaranteed to be a tuple() to allow programs to distinguish it from non precompiled search patterns." The binary:compile_pattern function is a BIF that creates a "magic binary", much like the resources one can create in a NIF, and these appear as empty binaries in Erlang code as you can see in your shell output, but they're really opaque data types. Because of this, your approach won't work because the subsequent functions you're calling think you're passing an actual empty binary instead of the required opaque type. --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvliwanag@REDACTED Sat Feb 9 13:41:27 2013 From: jvliwanag@REDACTED (Jan Vincent Liwanag) Date: Sat, 9 Feb 2013 20:41:27 +0800 Subject: [erlang-questions] inline specs Message-ID: <5DBDFFAD-D436-4E8F-A124-5E793F539BFF@gmail.com> Is there any way wherein instead of typing: -spec add(X::number(), Y::number()) -> number(). add(X, Y) -> X + Y. I can just have it inline and do: add(X::number(), Y::number()) :: number() -> X + Y. Jan Vincent Liwanag jvliwanag@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Sat Feb 9 13:59:58 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Sat, 9 Feb 2013 13:59:58 +0100 Subject: [erlang-questions] inline specs In-Reply-To: <5DBDFFAD-D436-4E8F-A124-5E793F539BFF@gmail.com> References: <5DBDFFAD-D436-4E8F-A124-5E793F539BFF@gmail.com> Message-ID: <84F6CE9C-5AF2-4958-9E85-9A42706C562B@gmail.com> No. -- Anthony Ramine Le 9 f?vr. 2013 ? 13:41, Jan Vincent Liwanag a ?crit : > Is there any way wherein instead of typing: > > -spec add(X::number(), Y::number()) -> number(). > add(X, Y) -> > X + Y. > > I can just have it inline and do: > > add(X::number(), Y::number()) :: number() -> > X + Y. > > Jan Vincent Liwanag > jvliwanag@REDACTED From sigxcpu@REDACTED Sat Feb 9 14:09:50 2013 From: sigxcpu@REDACTED (Alex P) Date: Sat, 9 Feb 2013 05:09:50 -0800 (PST) Subject: [erlang-questions] Memory growing In-Reply-To: References: ,> <74db6641-f140-45f5-8509-c3c6979f0c57@googlegroups.com> Message-ID: <2324a65b-71f3-4548-a5a2-cf71092614fa@googlegroups.com> It seems that I've found the issue. I have a "process_killer" gen_server where processes can subscribe for periodic GC or kill. Its subscribe functions are called on each message received by some processes to postpone the GC/kill (something like re-arm). This process performs an erlang:monitor if not already monitored to catch a dead process and remove it from watch list. If I comment our the re-subscription line on each handled message, "system" area seems to behave normally. That means it is a bug in my process_killer that does leak monitor refs (remember you can call erlang:monitor multiple times and each call creates a reference). I was lead to this idea because I've tested a simple module which was calling erlang:monitor in a loop and I have seen ~13 bytes "system" area grow on each call. The workers themselves were OK because they would die anyway taking their monitors along with them. There is one long running (starts with the app, stops with the app) process that dispatches all the messages to the workers that was calling GC re-arm on each received message, so we're talking about tens of thousands of monitors spawned per hour and never released. I'm writing this answer here for future reference. TL;DR; make sure you are not leaking monitor refs on a long running process. On Thursday, February 7, 2013 6:21:45 PM UTC+2, Matthew Evans wrote: > > > Can you kill the VM with a SIGUSR1 and look at the resulting crash dump? > > Start an Erlang shell, start crashdump viewer and poke around: > > crashdump_viewer:start(). > > ________________________________ > > Date: Thu, 7 Feb 2013 08:08:58 -0800 > > From: sig...@REDACTED > > To: erlang-pr...@REDACTED > > CC: erlang-q...@REDACTED > > Subject: Re: [erlang-questions] Memory growing > > > > I've tested NIFs from lists module and they seem to be fine. > > > > So, coming back to square one, how can I debug what is allocated in the > > "system" section of erlang:memory/0 ? > > > > Alex > > > > On Saturday, February 2, 2013 10:52:04 PM UTC+2, Alex P wrote: > > Hello, > > > > If somebody could help me with an issue, I would be grateful. I'm > > pulling my hair off trying to figure out why an erlang app keeps > > growing in memory usage. The erlang:memory() after 3 days shows this: > > > > [{total,325732292}, > > {processes,54520269}, > > {processes_used,54492115}, > > {system,271212023}, > > {atom,932793}, > > {atom_used,912937}, > > {binary,2348080}, > > {code,8288354}, > > {ets,487392}, > > {maximum,398189416}] > > > > The interesting part is "system" section. All other data is fine > > (processes goes to few MBs during the night as all the started procs > > are either hibernated - for long running or they die because of > > inactivity). > > > > There are two issues here: > > - "system" keeps growing (around 800MB/week) > > - OS (Linux) shows a significantly different amount of memory usage > > (almost 2x, 610MB resident in this case) > > > > The only case where I've managed to make "system" grow is by calling > > erlang:monitor() multiple times. It grows with ~13K/call. > > > > Erlang release used is R15B03. > > > > The questions are: > > - is there any hidden function to inspect all the active process > > monitors? (as the single place where I use erlang:monitor looks fine. > > The spawned kid does its job and dies, so the ref should be garbage > > collected). > > - is there any way to inspect "system" section and find out what it > > contains? Looking at memory chunks in process map got me nowhere > > > > Thank you, > > Alex > > > > > > > > _______________________________________________ erlang-questions > > mailing list erlang-q...@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From msegalis@REDACTED Sat Feb 9 15:08:17 2013 From: msegalis@REDACTED (Morgan Segalis) Date: Sat, 9 Feb 2013 15:08:17 +0100 Subject: [erlang-questions] gen_tcp issue when socket interrupted by peer Message-ID: <6899E3B5-1942-4603-9FD0-2C043B01EEE5@gmail.com> Hi everyone, I'm facing an issue when the client wrongly interrupt the connection to the server... Actually, the client is a phone... When the phone goes from Wifi to (3G or No Connection) the tcp connection opened while it was on Wifi is still opened on the server. The weird thing is, If I send data to the "closed" socket, gen_tcp:send returns always ok, although it will never been received... I though that TCP was all about stability and message receiving (unlike UDP). Is that a normal behavior ? Or I am doing something wrong ? I'm using gen_server behaviour, and my socket options are : -define(TCP_OPTIONS, [list, {packet, 0}, {active, once}, {reuseaddr, true}, {mode, list}, {backlog, 128}, {send_timeout, 1}, {keepalive, true}]). of course I handle all tcp events like tcp_closed and tcp_error Please help ! Thank you. From essen@REDACTED Sat Feb 9 20:30:37 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Sat, 09 Feb 2013 20:30:37 +0100 Subject: [erlang-questions] gen_tcp issue when socket interrupted by peer In-Reply-To: <6899E3B5-1942-4603-9FD0-2C043B01EEE5@gmail.com> References: <6899E3B5-1942-4603-9FD0-2C043B01EEE5@gmail.com> Message-ID: <5116A3DD.1070109@ninenines.eu> On 02/09/2013 03:08 PM, Morgan Segalis wrote: > Hi everyone, > > I'm facing an issue when the client wrongly interrupt the connection to the server... > > Actually, the client is a phone... When the phone goes from Wifi to (3G or No Connection) the tcp connection opened while it was on Wifi is still opened on the server. > > The weird thing is, If I send data to the "closed" socket, gen_tcp:send returns always ok, although it will never been received... > I though that TCP was all about stability and message receiving (unlike UDP). TCP isn't perfect there, there can be cases where the socket is closed on one side and the other doesn't know. This can be solved by a timeout. I wasn't expecting sends to still succeed though. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From msegalis@REDACTED Sat Feb 9 21:10:14 2013 From: msegalis@REDACTED (Morgan Segalis) Date: Sat, 9 Feb 2013 21:10:14 +0100 Subject: [erlang-questions] gen_tcp issue when socket interrupted by peer In-Reply-To: <5116A3DD.1070109@ninenines.eu> References: <6899E3B5-1942-4603-9FD0-2C043B01EEE5@gmail.com> <5116A3DD.1070109@ninenines.eu> Message-ID: <34609ED4-3676-4E96-96F8-F365CA118871@gmail.com> Yeah, I was surprised too... After some research, I have found this article that assume what I though, it is not an Erlang specific issue, but TCP... http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html ... So I'll do my app-level ACK. Le 9 f?vr. 2013 ? 20:30, Lo?c Hoguin a ?crit : > On 02/09/2013 03:08 PM, Morgan Segalis wrote: >> Hi everyone, >> >> I'm facing an issue when the client wrongly interrupt the connection to the server... >> >> Actually, the client is a phone... When the phone goes from Wifi to (3G or No Connection) the tcp connection opened while it was on Wifi is still opened on the server. >> >> The weird thing is, If I send data to the "closed" socket, gen_tcp:send returns always ok, although it will never been received... >> I though that TCP was all about stability and message receiving (unlike UDP). > > TCP isn't perfect there, there can be cases where the socket is closed on one side and the other doesn't know. This can be solved by a timeout. I wasn't expecting sends to still succeed though. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu From jeremy@REDACTED Sat Feb 9 22:37:42 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sat, 9 Feb 2013 13:37:42 -0800 Subject: [erlang-questions] Dynamically access record fields Message-ID: Suppose I have a record that looks like -record(foo, {bucket1, bucket2}). I may want to pass in an argument specifying what bucket to perform the operation over. For example, Bucket = bucket1, then later, Use Foo#foo.Bucket for some operations and also modify Bucket with something like Foo#foo{Bucket = bar} Erlang doesn't seem to support this, and I don't see why not? Is there a better way to go about this? Real World Example, If the record stores data about two people playing a game against one another, and one of them disconnects, I want to perform some operations on the disconnected player's data, and notify the other player that the disconnect occurred. Not having this sort of functionality results in some code duplication and messy code (at least, if using records). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Sat Feb 9 22:40:29 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sat, 9 Feb 2013 13:40:29 -0800 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: Hmm, upon reading more, it seems like all record functionality is implemented in the compiler pass. It seems like the ability to address particular variable named locations in tuples would be desirable. I guess I'll have to use proplists or a dict for now, although it feels like overkill for what I'm doing. On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: > Suppose I have a record that looks like > > -record(foo, {bucket1, bucket2}). > > I may want to pass in an argument specifying what bucket to perform the > operation over. > > For example, > > Bucket = bucket1, > > then later, > > Use Foo#foo.Bucket for some operations and also modify Bucket with > something like > > Foo#foo{Bucket = bar} > > Erlang doesn't seem to support this, and I don't see why not? Is there a > better way to go about this? > > Real World Example, > > If the record stores data about two people playing a game against one > another, and one of them disconnects, I want to perform some operations on > the disconnected player's data, and notify the other player that the > disconnect occurred. Not having this sort of functionality results in some > code duplication and messy code (at least, if using records). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Sat Feb 9 23:25:18 2013 From: ulf@REDACTED (Ulf Wiger) Date: Sat, 9 Feb 2013 23:25:18 +0100 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: <735B6C55-14E6-49EB-A0C2-4DFFD4815473@feuerlabs.com> Take a look at exprecs.erl in https://github.com/uwiger/parse_trans https://github.com/uwiger/parse_trans/blob/master/doc/exprecs.md BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://www.feuerlabs.com 9 feb 2013 kl. 22:40 skrev Jeremy Ong : > Hmm, upon reading more, it seems like all record functionality is implemented in the compiler pass. > > It seems like the ability to address particular variable named locations in tuples would be desirable. I guess I'll have to use proplists or a dict for now, although it feels like overkill for what I'm doing. > > > On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: >> Suppose I have a record that looks like >> >> -record(foo, {bucket1, bucket2}). >> >> I may want to pass in an argument specifying what bucket to perform the operation over. >> >> For example, >> >> Bucket = bucket1, >> >> then later, >> >> Use Foo#foo.Bucket for some operations and also modify Bucket with something like >> >> Foo#foo{Bucket = bar} >> >> Erlang doesn't seem to support this, and I don't see why not? Is there a better way to go about this? >> >> Real World Example, >> >> If the record stores data about two people playing a game against one another, and one of them disconnects, I want to perform some operations on the disconnected player's data, and notify the other player that the disconnect occurred. Not having this sort of functionality results in some code duplication and messy code (at least, if using records). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sat Feb 9 23:33:26 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 10 Feb 2013 00:33:26 +0200 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: Hello, I am not precisely sure what you mean by saying "Erlang doesn't seem to support this ?" Record is a syntax sugar for tuples, all operations applicable for tuples are also valid for records. Keep in-mind that tuple elements are addressable. -module(rec). -export([test/0]). -record(foo, {a, b, c}). test() -> V1 = #foo{a="A", b="B", c="C"}, io:format("rec ~p~n", [V1]), io:format("rec got ~p~n", [element(#foo.a, V1)]), V2 = setelement(#foo.a, V1, "A+"), io:format("rec ~p~n", [V2]), io:format("rec got ~p~n", [element(#foo.b, V2)]), V3 = setelement(#foo.b, V2, "B+"), io:format("-> ~p~n", [V3]). - Dmitry On Feb 9, 2013, at 11:40 PM, Jeremy Ong wrote: > Hmm, upon reading more, it seems like all record functionality is implemented in the compiler pass. > > It seems like the ability to address particular variable named locations in tuples would be desirable. I guess I'll have to use proplists or a dict for now, although it feels like overkill for what I'm doing. > > > On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: > Suppose I have a record that looks like > > -record(foo, {bucket1, bucket2}). > > I may want to pass in an argument specifying what bucket to perform the operation over. > > For example, > > Bucket = bucket1, > > then later, > > Use Foo#foo.Bucket for some operations and also modify Bucket with something like > > Foo#foo{Bucket = bar} > > Erlang doesn't seem to support this, and I don't see why not? Is there a better way to go about this? > > Real World Example, > > If the record stores data about two people playing a game against one another, and one of them disconnects, I want to perform some operations on the disconnected player's data, and notify the other player that the disconnect occurred. Not having this sort of functionality results in some code duplication and messy code (at least, if using records). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Sat Feb 9 23:48:37 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sat, 9 Feb 2013 14:48:37 -0800 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: On Sat, Feb 9, 2013 at 2:33 PM, Dmitry Kolesnikov wrote: > Hello, > > I am not precisely sure what you mean by saying "Erlang doesn't seem to > support this ?" > Record is a syntax sugar for tuples, all operations applicable for tuples > are also valid for records. > Keep in-mind that tuple elements are addressable. > > > -module(rec). > > -export([test/0]). > > -record(foo, {a, b, c}). > > test() -> > V1 = #foo{a="A", b="B", c="C"}, > io:format("rec ~p~n", [V1]), > io:format("rec got ~p~n", [element(#foo.a, V1)]), > V2 = setelement(#foo.a, V1, "A+"), > > io:format("rec ~p~n", [V2]), > io:format("rec got ~p~n", [element(#foo.b, V2)]), > V3 = setelement(#foo.b, V2, "B+"), > > io:format("-> ~p~n", [V3]). > > > What I was saying was that you can't do something like Field = a, V2 = setelement(#foo.Field, V1, "A+") > > - Dmitry > > > On Feb 9, 2013, at 11:40 PM, Jeremy Ong wrote: > > Hmm, upon reading more, it seems like all record functionality is > implemented in the compiler pass. > > It seems like the ability to address particular variable named locations > in tuples would be desirable. I guess I'll have to use proplists or a dict > for now, although it feels like overkill for what I'm doing. > > > On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: > >> Suppose I have a record that looks like >> >> -record(foo, {bucket1, bucket2}). >> >> I may want to pass in an argument specifying what bucket to perform the >> operation over. >> >> For example, >> >> Bucket = bucket1, >> >> then later, >> >> Use Foo#foo.Bucket for some operations and also modify Bucket with >> something like >> >> Foo#foo{Bucket = bar} >> >> Erlang doesn't seem to support this, and I don't see why not? Is there a >> better way to go about this? >> >> Real World Example, >> >> If the record stores data about two people playing a game against one >> another, and one of them disconnects, I want to perform some operations on >> the disconnected player's data, and notify the other player that the >> disconnect occurred. Not having this sort of functionality results in some >> code duplication and messy code (at least, if using records). >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Sat Feb 9 23:48:58 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sat, 9 Feb 2013 14:48:58 -0800 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <735B6C55-14E6-49EB-A0C2-4DFFD4815473@feuerlabs.com> References: <735B6C55-14E6-49EB-A0C2-4DFFD4815473@feuerlabs.com> Message-ID: On Sat, Feb 9, 2013 at 2:25 PM, Ulf Wiger wrote: > Take a look at exprecs.erl in https://github.com/uwiger/parse_trans > > https://github.com/uwiger/parse_trans/blob/master/doc/exprecs.md > > Thanks I'll take a look. > BR, > Ulf W > > Ulf Wiger, Feuerlabs, Inc. > http://www.feuerlabs.com > > 9 feb 2013 kl. 22:40 skrev Jeremy Ong : > > Hmm, upon reading more, it seems like all record functionality is > implemented in the compiler pass. > > It seems like the ability to address particular variable named locations > in tuples would be desirable. I guess I'll have to use proplists or a dict > for now, although it feels like overkill for what I'm doing. > > > On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: > >> Suppose I have a record that looks like >> >> -record(foo, {bucket1, bucket2}). >> >> I may want to pass in an argument specifying what bucket to perform the >> operation over. >> >> For example, >> >> Bucket = bucket1, >> >> then later, >> >> Use Foo#foo.Bucket for some operations and also modify Bucket with >> something like >> >> Foo#foo{Bucket = bar} >> >> Erlang doesn't seem to support this, and I don't see why not? Is there a >> better way to go about this? >> >> Real World Example, >> >> If the record stores data about two people playing a game against one >> another, and one of them disconnects, I want to perform some operations on >> the disconnected player's data, and notify the other player that the >> disconnect occurred. Not having this sort of functionality results in some >> code duplication and messy code (at least, if using records). >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sat Feb 9 23:57:16 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 10 Feb 2013 00:57:16 +0200 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: <2265D3EF-DC0E-41A9-AEAC-09D113ED0EEC@gmail.com> Hello, > What I was saying was that you can't do something like > > Field = a, > V2 = setelement(#foo.Field, V1, "A+") but you can do Field = #foo.a, V2 = setelement(Field, V1, "A+") and do not see any difference, isn't it? - Dmitry On Feb 10, 2013, at 12:48 AM, Jeremy Ong wrote: > > > > On Sat, Feb 9, 2013 at 2:33 PM, Dmitry Kolesnikov wrote: > Hello, > > I am not precisely sure what you mean by saying "Erlang doesn't seem to support this ?" > Record is a syntax sugar for tuples, all operations applicable for tuples are also valid for records. > Keep in-mind that tuple elements are addressable. > > > -module(rec). > > -export([test/0]). > > -record(foo, {a, b, c}). > > test() -> > V1 = #foo{a="A", b="B", c="C"}, > io:format("rec ~p~n", [V1]), > io:format("rec got ~p~n", [element(#foo.a, V1)]), > V2 = setelement(#foo.a, V1, "A+"), > > io:format("rec ~p~n", [V2]), > io:format("rec got ~p~n", [element(#foo.b, V2)]), > V3 = setelement(#foo.b, V2, "B+"), > > io:format("-> ~p~n", [V3]). > > > > What I was saying was that you can't do something like > > Field = a, > V2 = setelement(#foo.Field, V1, "A+") > > > - Dmitry > > > On Feb 9, 2013, at 11:40 PM, Jeremy Ong wrote: > >> Hmm, upon reading more, it seems like all record functionality is implemented in the compiler pass. >> >> It seems like the ability to address particular variable named locations in tuples would be desirable. I guess I'll have to use proplists or a dict for now, although it feels like overkill for what I'm doing. >> >> >> On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: >> Suppose I have a record that looks like >> >> -record(foo, {bucket1, bucket2}). >> >> I may want to pass in an argument specifying what bucket to perform the operation over. >> >> For example, >> >> Bucket = bucket1, >> >> then later, >> >> Use Foo#foo.Bucket for some operations and also modify Bucket with something like >> >> Foo#foo{Bucket = bar} >> >> Erlang doesn't seem to support this, and I don't see why not? Is there a better way to go about this? >> >> Real World Example, >> >> If the record stores data about two people playing a game against one another, and one of them disconnects, I want to perform some operations on the disconnected player's data, and notify the other player that the disconnect occurred. Not having this sort of functionality results in some code duplication and messy code (at least, if using records). >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Sun Feb 10 00:05:09 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sat, 9 Feb 2013 15:05:09 -0800 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <2265D3EF-DC0E-41A9-AEAC-09D113ED0EEC@gmail.com> References: <2265D3EF-DC0E-41A9-AEAC-09D113ED0EEC@gmail.com> Message-ID: Interesting, I wasn't reading your code carefully. I didn't realize that #record.field actually produced an index. This seems to work then. I think it be cool to have some syntactic sugar for it nonetheless, since we can just expand the input (with variables) into expressions using setelement and element. On Sat, Feb 9, 2013 at 2:57 PM, Dmitry Kolesnikov wrote: > Hello, > > What I was saying was that you can't do something like > > Field = a, > V2 = setelement(#foo.Field, V1, "A+") > > > but you can do > > Field = #foo.a, > V2 = setelement(Field, V1, "A+") > > and do not see any difference, isn't it? > > - Dmitry > > > On Feb 10, 2013, at 12:48 AM, Jeremy Ong wrote: > > > > > On Sat, Feb 9, 2013 at 2:33 PM, Dmitry Kolesnikov wrote: > >> Hello, >> >> I am not precisely sure what you mean by saying "Erlang doesn't seem to >> support this ?" >> Record is a syntax sugar for tuples, all operations applicable for tuples >> are also valid for records. >> Keep in-mind that tuple elements are addressable. >> >> >> -module(rec). >> >> -export([test/0]). >> >> -record(foo, {a, b, c}). >> >> test() -> >> V1 = #foo{a="A", b="B", c="C"}, >> io:format("rec ~p~n", [V1]), >> io:format("rec got ~p~n", [element(#foo.a, V1)]), >> V2 = setelement(#foo.a, V1, "A+"), >> >> io:format("rec ~p~n", [V2]), >> io:format("rec got ~p~n", [element(#foo.b, V2)]), >> V3 = setelement(#foo.b, V2, "B+"), >> >> io:format("-> ~p~n", [V3]). >> >> >> > What I was saying was that you can't do something like > > Field = a, > V2 = setelement(#foo.Field, V1, "A+") > > >> >> - Dmitry >> >> >> On Feb 9, 2013, at 11:40 PM, Jeremy Ong wrote: >> >> Hmm, upon reading more, it seems like all record functionality is >> implemented in the compiler pass. >> >> It seems like the ability to address particular variable named locations >> in tuples would be desirable. I guess I'll have to use proplists or a dict >> for now, although it feels like overkill for what I'm doing. >> >> >> On Sat, Feb 9, 2013 at 1:37 PM, Jeremy Ong wrote: >> >>> Suppose I have a record that looks like >>> >>> -record(foo, {bucket1, bucket2}). >>> >>> I may want to pass in an argument specifying what bucket to perform the >>> operation over. >>> >>> For example, >>> >>> Bucket = bucket1, >>> >>> then later, >>> >>> Use Foo#foo.Bucket for some operations and also modify Bucket with >>> something like >>> >>> Foo#foo{Bucket = bar} >>> >>> Erlang doesn't seem to support this, and I don't see why not? Is there a >>> better way to go about this? >>> >>> Real World Example, >>> >>> If the record stores data about two people playing a game against one >>> another, and one of them disconnects, I want to perform some operations on >>> the disconnected player's data, and notify the other player that the >>> disconnect occurred. Not having this sort of functionality results in some >>> code duplication and messy code (at least, if using records). >>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierrefenoll@REDACTED Sun Feb 10 02:02:12 2013 From: pierrefenoll@REDACTED (Pierre Fenoll) Date: Sun, 10 Feb 2013 02:02:12 +0100 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <2265D3EF-DC0E-41A9-AEAC-09D113ED0EEC@gmail.com> References: <2265D3EF-DC0E-41A9-AEAC-09D113ED0EEC@gmail.com> Message-ID: On 9 February 2013 23:57, Dmitry Kolesnikov wrote: > Hello, > > What I was saying was that you can't do something like > > Field = a, > V2 = setelement(#foo.Field, V1, "A+") > > > but you can do > > Field = #foo.a, > V2 = setelement(Field, V1, "A+") > > and do not see any difference, isn't it? > > - Dmitry Well yes there is. What I think Jeremy wants is to dynamically concatenate record names, which is as you know impossible due to record names being resolve at compile-time. For instance, you can do this get_foo_a() -> element( binary_to_term(list_to_binary( binary_to_list(term_to_binary(#foo.a)))), M). However, you cannot do that: get_foo(Field=a) -> element( binary_to_term(list_to_binary( binary_to_list(term_to_binary(#foo.)) ++ atom_to_list(Field))), M). * 1: syntax error before: ')' (#foo. throws the syntax error, so does #foo). Isn't it a drawback? Is it even good Erlang practice? I don't want to troll. Is there an EEP about setting record names dynamically? Regards, -- Pierre Fenoll From max.lapshin@REDACTED Sun Feb 10 16:49:32 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 10 Feb 2013 18:49:32 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition Message-ID: Is it possible to specify cowboy routes with method (GET/POST) as a routing condition? -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Sun Feb 10 19:46:07 2013 From: robert.virding@REDACTED (Robert Virding) Date: Sun, 10 Feb 2013 18:46:07 +0000 (GMT) Subject: [erlang-questions] Writing a DSL in ERlang (ILYA Khlopotov) In-Reply-To: Message-ID: <1605645557.10937999.1360521967478.JavaMail.root@erlang-solutions.com> Or use one of the lisp front-ends for erlang, LFE or Joxa, to get ultimate macro facilities. Robert ----- Original Message ----- > From: "ILYA Khlopotov" > To: erlang-questions@REDACTED > Sent: Friday, 8 February, 2013 8:05:37 PM > Subject: Re: [erlang-questions] Writing a DSL in ERlang (ILYA > Khlopotov) > You might want to consider elixir for this simple task. Since it > compiles into beam you can use generated modules from erlang. > If you really want erlang code you could extract generated erlang > from compiled beam (you need debug_info for that though). > I know that it's not elixir mailing list so I'm apologize for > providing a solution in this language. However it might save a lot > of effort compared to writing DSL in erlang. > Your dsl code might look like (for complete code see > https://gist.github.com/khia/4741045 ): > defmacro action(match, [do: body]) do > quote do > def handle_info(unquote(match), state) do > unquote(body) > {:ok, state} > end > end > end > defmacro createhandler(name, [do: body]) do > quote do > defmodule unquote(name) do > unquote(body) > end > end > end > Having that you can write your definition files using your own macro > (action and createhandler) as: > createhandler TestHandler do > action {:x, x}, do: IO.puts "x: #{inspect x}" > action {:y, y, seed} do > # example of calling erlang modules > :random.seed(seed) > value = :random.uniform(y) > IO.puts "random.uniform(${y}) -> #{value} " > end > end > If your actions can be a single expression you can go further and > define macro for ::: (or something else) and then you might be able > turn it into (WARNING pseudo code) > createhandler TestHandler do > {:x, x} ::: IO.puts("x: #{inspect x}") > {:y, y} ::: IO.puts("random.uniform(${y}) -> #{value} ") > end > Best regards, > ILYA > > I'm am interested in creating an external DSL with Erlang and what > > are > > the steps involved. So very basically I want to convert some > > English > > like language into running erlang code. My goal is to build up a > > full > > erlang source file relative to the dsl file I have as input. I > > don't > > necessarily need to print out the source file to the file system, > > holding it in memory is sufficient and allowing its execution. > > Sample DSL syntax; {createhandler, "TestHandler", {handles, [ {x, > > {action, printToScreen}}, {y, {action, writeToStorage}} ]} } > > So from above I want to create a gen_event handler with a > > handle_event > > function which pattern matches on 'x' and 'y' and performs the > > actions > > defined. I have used this to just show what I would like to do, > > what I'm > > interested in is the steps and approaches to doing this. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Feb 11 00:06:01 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Sun, 10 Feb 2013 18:06:01 -0500 Subject: [erlang-questions] Cowboy routes with Method as a condition Message-ID: Cowboy routes to resources, on which operations are applied depending on method/headers/body, so no. cowboy_rest is what you most likely want. Max Lapshin wrote: >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Mon Feb 11 00:53:55 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sun, 10 Feb 2013 15:53:55 -0800 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) Message-ID: https://github.com/jeremyong/websocket_client For those of you who have adopted websocket technology in your servers, I have created a basic client for use in benchmarking and creating system/integration tests for erlang code. It can speak using either the gen_tcp or ssl transport and currently handles text and binary messages. I modeled the behaviour itself after essen's cowboy websocket_handler so the usage should be very similar (I also used cowboy's code as reference in a few places, I hope you don't mind Loic). It is still missing a lot of base functionality, but I am releasing it because I have already found it useful to benchmark my own application's performance. So feel free to use it! And if you have a spare moment, feel free to contribute code/tests. The root level readme has a bunch of todos :) Cheers, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Feb 11 01:01:16 2013 From: ok@REDACTED (ok@REDACTED) Date: Mon, 11 Feb 2013 13:01:16 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: <1ca008748c8eb8b5845f1b238cd59ccf.squirrel@chasm.otago.ac.nz> > Hmm, upon reading more, it seems like all record functionality is > implemented in the compiler pass. That is what the manual says. > > It seems like the ability to address particular variable named locations > in > tuples would be desirable. Why? Suppose I want to tell a function "I want you to work on the 'david' field of an 'icke' record." If the -record(icke, {...,david...}) declaration is in scope, I can pass in #icke.david . If the -record declaration is NOT in scope, I have no right to expect there to *be* a 'david' field or, if there is one, for it to mean anything I might imagine it to mean. For that matter, if the -record declaration is in scope, I can pass a *function* fun (#icke{david = X}) -> X end or fun (Icke, New_X) -> Icke#icke{david = New_X} end to say what I want. From ok@REDACTED Mon Feb 11 01:07:58 2013 From: ok@REDACTED (ok@REDACTED) Date: Mon, 11 Feb 2013 13:07:58 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: <0480c78ac94ffea9272f66f1f757c34b.squirrel@chasm.otago.ac.nz> > On Sat, Feb 9, 2013 at 2:33 PM, Dmitry Kolesnikov > What I was saying was that you can't do something like > > Field = a, > V2 = setelement(#foo.Field, V1, "A+") Yes. Like it says on the box, "records". You can't do this in Pascal, Modula, Modula2, Oberon, Ada, C, ... Records are not dictionaries and do not store field names. That's pretty much the _point_ of records, actually. But you *CAN* do Index = #foo.a, V1 = setelement(Index, V1, "A+") It's all spelled out in the manual. In this case, section 9.3, "Accessing Record Fields", second paragraph. From jeremy@REDACTED Mon Feb 11 01:10:29 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Sun, 10 Feb 2013 16:10:29 -0800 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <0480c78ac94ffea9272f66f1f757c34b.squirrel@chasm.otago.ac.nz> References: <0480c78ac94ffea9272f66f1f757c34b.squirrel@chasm.otago.ac.nz> Message-ID: On Sun, Feb 10, 2013 at 4:07 PM, wrote: > > On Sat, Feb 9, 2013 at 2:33 PM, Dmitry Kolesnikov > > > What I was saying was that you can't do something like > > > > Field = a, > > V2 = setelement(#foo.Field, V1, "A+") > > Yes. Like it says on the box, "records". > You can't do this in Pascal, Modula, Modula2, Oberon, Ada, C, ... > Records are not dictionaries and do not store field names. > That's pretty much the _point_ of records, actually. > > But you *CAN* do > > Index = #foo.a, > V1 = setelement(Index, V1, "A+") > > It's all spelled out in the manual. > In this case, section 9.3, "Accessing Record Fields", > second paragraph. > > > And I'm doing exactly this. I'm arguing pure semantics, not *capability*. I just don't see the harm in allowing the former, but I'm completely prepared to ignore this issue entirely for now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Mon Feb 11 01:26:56 2013 From: ok@REDACTED (ok@REDACTED) Date: Mon, 11 Feb 2013 13:26:56 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: <2265D3EF-DC0E-41A9-AEAC-09D113ED0EEC@gmail.com> Message-ID: <8f195b2923a3218bcae7367d7a227909.squirrel@chasm.otago.ac.nz> > On 9 February 2013 23:57, Dmitry Kolesnikov > wrote: > Well yes there is. > What I think Jeremy wants is to dynamically concatenate record names, No, that's what he's been asking about, but there is some useful work he is trying to do, and he needs a way to do that which is, if possible, good performance and good engineering. So far, we have - write accessor functions that provide precisely the access needed (fast, safe, maintainable) - use dictionaries (slow, dangerous, difficult to maintain) - pass funs (medium fast, safe, maintainable) - pass indices (fast, dangerous, medium maintainable) amongst other possibilities. > Isn't it a drawback? Isn't the inability to compute field names a drawback? No. Is it even good Erlang practice? Would computing field names be good practice if doable? No. > > I don't want to troll. Is there an EEP about setting record names > dynamically? The EEPS site can be read by anybody. Go over to www.erlang.org/eeps/ and look. I don't see one, and I can promise you that there will never be one from my keyboard. I _have_ proposed frames, which _would_ allow dynamic field names: frame_has(Frame, Key) -> true | false frame_value(Frame, Key) -> term() | an error exit Hmm. I forgot to include setframe_value(Frame, Key, New_Value); the effect is achievable via list_to_frame([{Key,New_Value}|frame_to_list(Frame)]) but it could be implemented rather faster. It's worth bearing in mind that updating records is expensive. Each update builds a whole new record. By looking at the BEAM code generated for bar(Old) -> Mid = Old#foo{bar=1}, ugh(Old), New = Mid#foo{ugh=2}, ugh(New), New. you will see that the compiler isn't very smart: Mid has only one use, so the definitions of Mid and New could be fused into a single update, but they are not. (The test that Old is a foo cannot move, but the allocation and rebinding steps could have moved.) To do anything that would make it easier to update a single field would be to encourage inefficient programming. From ok@REDACTED Mon Feb 11 02:17:18 2013 From: ok@REDACTED (ok@REDACTED) Date: Mon, 11 Feb 2013 14:17:18 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: <0480c78ac94ffea9272f66f1f757c34b.squirrel@chasm.otago.ac.nz> Message-ID: > I just don't see the harm in allowing the former, but I'm completely > prepared to ignore this issue entirely for now. > The harm in dynamic field names includes but is not limited to: - slowing programs down - making code less readable for human beings - reducing the effectiveness of type checking - inadvertently providing wider indirect access to fields than intended compared with hand-written access functions From essen@REDACTED Mon Feb 11 08:37:41 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 11 Feb 2013 08:37:41 +0100 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) Message-ID: Sounds cool! Jeremy Ong wrote: >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Feb 11 08:41:57 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 11 Feb 2013 10:41:57 +0300 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) In-Reply-To: References: Message-ID: Hi. This works for me excelently: https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/websocket_client.erl Perhaps it will help you -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Feb 11 08:50:02 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 11 Feb 2013 08:50:02 +0100 Subject: [erlang-questions] Dynamically access record fields Message-ID: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> - Micro optimization. Most applications wouldn't care for it either - That's a developer's choice - BS, you have the same issue with Var = #rec.field, no difference there - See previous point ok@REDACTED wrote: > >> I just don't see the harm in allowing the former, but I'm completely >> prepared to ignore this issue entirely for now. >> >The harm in dynamic field names includes but is not limited to: > - slowing programs down > - making code less readable for human beings > - reducing the effectiveness of type checking > - inadvertently providing wider indirect access to fields > than intended compared with hand-written access functions > > >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Mon Feb 11 09:01:55 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 11 Feb 2013 11:01:55 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: Message-ID: cowboy_rest is incredibly hard to use because there is even no full list of callbacks. Webmachine diagram is good, but it lacks name of callbacks. I understand that it is hard to document it, but I can't help you with it because I don't understand this code in depth. I want to put Amazon S3 API in routes and it looks like rest is a bad idea. I've ended with catch-all route and manual routing in init/3 handler -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Mon Feb 11 09:05:58 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Mon, 11 Feb 2013 00:05:58 -0800 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) In-Reply-To: References: Message-ID: Thanks Max I'll take a look. Unfortunately, I need ssl so I'll just keep hacking away at the one I just released for now. Additional points (that you may not care about for your purposes, but matter to me): It looks like you aren't doing any challenge verification. There doesn't seem to be any length validation on non-text/binary frames. Also, you aren't doing client to server masking. On Sun, Feb 10, 2013 at 11:41 PM, Max Lapshin wrote: > Hi. > > This works for me excelently: > > > https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/websocket_client.erl > > Perhaps it will help you > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Feb 11 09:07:25 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 11 Feb 2013 11:07:25 +0300 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) In-Reply-To: References: Message-ID: Yes, I skip all these things. Hope that you will be more diligent than me =) On Mon, Feb 11, 2013 at 12:05 PM, Jeremy Ong wrote: > Thanks Max I'll take a look. Unfortunately, I need ssl so I'll just keep > hacking away at the one I just released for now. > > Additional points (that you may not care about for your purposes, but > matter to me): > > It looks like you aren't doing any challenge verification. There doesn't > seem to be any length validation on non-text/binary frames. Also, you > aren't doing client to server masking. > > > On Sun, Feb 10, 2013 at 11:41 PM, Max Lapshin wrote: > >> Hi. >> >> This works for me excelently: >> >> >> https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/websocket_client.erl >> >> Perhaps it will help you >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Mon Feb 11 09:08:30 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Mon, 11 Feb 2013 00:08:30 -0800 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) In-Reply-To: References: Message-ID: Heh, well, I have ssl, challenge verification, and client to server masking. I'll try to get around to the other opcodes too, one step at a time. :) On Mon, Feb 11, 2013 at 12:07 AM, Max Lapshin wrote: > Yes, I skip all these things. > > Hope that you will be more diligent than me =) > > > On Mon, Feb 11, 2013 at 12:05 PM, Jeremy Ong wrote: > >> Thanks Max I'll take a look. Unfortunately, I need ssl so I'll just keep >> hacking away at the one I just released for now. >> >> Additional points (that you may not care about for your purposes, but >> matter to me): >> >> It looks like you aren't doing any challenge verification. There doesn't >> seem to be any length validation on non-text/binary frames. Also, you >> aren't doing client to server masking. >> >> >> On Sun, Feb 10, 2013 at 11:41 PM, Max Lapshin wrote: >> >>> Hi. >>> >>> This works for me excelently: >>> >>> >>> https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/websocket_client.erl >>> >>> Perhaps it will help you >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Feb 11 09:15:26 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 11 Feb 2013 09:15:26 +0100 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) Message-ID: Note that Cowboy rejects unmasked clients now. Max Lapshin wrote: >Yes, I skip all these things. > >Hope that you will be more diligent than me =) > > > >On Mon, Feb 11, 2013 at 12:05 PM, Jeremy Ong wrote: > >Thanks Max I'll take a look. Unfortunately, I need ssl so I'll just keep hacking away at the one I just released for now. > > >Additional points (that you may not care about for your purposes, but matter to me): > > >It looks like you aren't doing any challenge verification. There doesn't seem to be any length validation on non-text/binary frames. Also, you aren't doing client to server masking. > > > >On Sun, Feb 10, 2013 at 11:41 PM, Max Lapshin wrote: > >Hi. > >This works for me excelently:? > >https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/websocket_client.erl > >Perhaps it will help you > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apauley@REDACTED Mon Feb 11 13:31:30 2013 From: apauley@REDACTED (Andreas Pauley) Date: Mon, 11 Feb 2013 14:31:30 +0200 Subject: [erlang-questions] Distributed OTP Apps: Failover and Takeover Message-ID: Hi everyone, I've made a demo app of mine distributed to test failover and takeover, after reading the "Distributed OTP Applications" chapter in Learn you some Erlang. The failover and takeover works great if I kill the running beam (eg. with kill -9). However, I tried sending kill signals to the Pids of both my application behavior and the top supervisor that gets started by the application. This crashes the VM, but failover does not happen. Is this unsupported, or should I do something to enable failover in this scenario? I've done a more complete writeup with code and output here: https://github.com/apauley/dark-overlord#when-processes-die-a-guide-to-the-afterlife But in a nutshell, I crash the VM with the commands below, and then automatic failover to my second node does not happen: $ ./rel/overlord/bin/overlord console Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] [dtrace] 14:03:20.581 overlord@REDACTED [overlord_app] <0.56.0> || Starting app: normal 14:03:20.582 overlord@REDACTED [hypnosponge_sup] <0.57.0> || Hello from the hypnosponge supervisor (overlord@REDACTED)1> Sup = pid(0, 57, 0). <0.57.0> (overlord@REDACTED)2> exit(Sup, kill). =ERROR REPORT==== 11-Feb-2013::14:04:46 === ** Generic server minion_supersup terminating ** Last message in was {'EXIT',<0.57.0>,killed} ** When Server state == {state, {local,minion_supersup}, simple_one_for_one, [{child,undefined,minion_makeshift_sup, {minion_makeshift_sup,start_link,[]}, temporary,5000,worker, [minion_makeshift_sup]}], undefined,1,3,[],minion_supersup,[]} ** Reason for termination == ** killed true (overlord@REDACTED)3> =INFO REPORT==== 11-Feb-2013::14:04:46 === application: overlord exited: killed type: permanent (overlord@REDACTED)3> {"Kernel pid terminated",application_controller,"{application_terminated,overlord,killed}"} Crash dump was written to: erl_crash.dump Kernel pid terminated (application_controller) ({application_terminated,overlord,killed}) -- http://pauley.org.za/ http://twitter.com/apauley http://www.meetup.com/lambda-luminaries/ From janos.n.hary@REDACTED Mon Feb 11 13:46:52 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Mon, 11 Feb 2013 13:46:52 +0100 Subject: [erlang-questions] two questions -- -boot in erlsrv and .erlang.cookies on windows In-Reply-To: References: Message-ID: <004801ce0855$de6115f0$9b2341d0$@gmail.com> On Windows you can place the .erlang and .erlang.cookie files in ERL_TOP\usr. It's not a per user location but safer then HOME. (By default a service runs as System user and you can try setting HOME in the 'System variables' section of the Environment Variables window. Alternatively you can specify a different user for the service, but that user needs 'Log on as service' permission.) For your other question I have no experience but according to the documentation you should try invoking start_erl from erl_srv instead of erl.exe. Janos From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Wes James Sent: Friday, February 08, 2013 11:02 PM To: erlang-questions@REDACTED Subject: [erlang-questions] two questions -- -boot in erlsrv and .erlang.cookies on windows I have -boot working on linux. Now I'm trying to get it to work with erlsrv for windows. I put the -boot in the args part of erlsrv, but it doesn't seem to take. Any ideas how to get this to work? Also, I've tried several different folders for .cookie.erlang on windows and I can't get it to take. Even putting it in my c:\Users\username and run the erlang app, it doesn't seem to be picked up. If I run with -setcookie, it works fine. I run "set" on windows in cmd.exe and see the HOME var is set. Also, if I can get this to work, where would the .erlang.cookie go when run as a service. I just ran erl.exe and it did not create an .erlang.cookie in my c:\Users\username folder (after I moved one that I had manually put there). Is it supposed to create one on windows? Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From Maxim.Minin@REDACTED Mon Feb 11 14:02:41 2013 From: Maxim.Minin@REDACTED (Minin Maxim) Date: Mon, 11 Feb 2013 14:02:41 +0100 Subject: [erlang-questions] question about eldap and ssh Message-ID: Hi i try to build a release of an application with dependencies of some standard libs and fail because not all necessary applications are have been packaged. I use eldap and ssh in my application. Eldap itself needs the lib ans1 and ssh - public_key. But both have not in their app-files the ans1 and public_key as dependency. As a result this two applications (ans1 and public_key) are not in my release. Is it a bug, that the dependencies are missing in the app-file of eldap and ssh or am I doing something wrong? thanks Maxim Minin -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Mon Feb 11 15:58:20 2013 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 11 Feb 2013 15:58:20 +0100 Subject: [erlang-questions] Following a send_timeout inet:close() no longer closes the socket. In-Reply-To: References: Message-ID: Hello, If you do not use {send_timeout_close, true} on the socket, the stack will try to send the data until it succeeds or the remote end closes the connection. This means that the close will wait for the send buffer to clear before closing the connection. The same is true when using port_close. The inet driver logic will detect that there is data on the way out and wait for that data to be sent before closing. So to get the behaviour you want you should set send_timeout_close to true, setting this to true is also the recommended in the setting in the documentation. I also noticed that in your testcase you did not close the listen socket after the test was done which caused the testcase to leave the listen socket open in between runs. Lukas On Thu, Jan 31, 2013 at 12:45 AM, Paul Cager wrote: > I am seeing a problem with erts-5.9.3.1 regarding TCP and send_timeout. > Please see the example program, below, but to summarise it: > > * A server accepts an incoming connection with options {send_timeout, > 250}. > * The server continually writes data until it gets an {error, timeout}. > The server then calls inet:close(Sock). > * However the socket isn't actually closed - netstat reports that it is > still established (and stays that way for at least an hour, probably for > ever). > * I've enabled debug tracing in drivers/common/inet_drv.c (+ added a > couple of debugs to show the return value of inet_subscribe). I've also > added tracing to show when desc_close is being called in that module. > * Normally calling inet:close() results in tcp_inet_stop and desc_close > being called. When inet:close() is called *after* an {error, timeout} > response "inet_ctl(606): INET_REQ_SUBSCRIBE" is called, but not > tcp_inet_stop (or desc_close), which is consistent with the socket > remaining ESTABLISHED. > * Using port_close rather than inet:close shows the same behaviour. > > That does not sound like the correct behaviour. > > Here is the debug output for a "normal" inet:close(): > > tcp_inet_ctl(606): 24 > inet_ctl(606): INET_REQ_SUBSCRIBE > inet_ctl(606): Calling inet_subscribe > inet_subscribe returned 6 > tcp_inet_stop(606) {s=10 > deq(606): queue empty > release_buffer: 1460 > tcp_inet_stop(606) } > desc_close(606): 10 > > Here is the debug output when receiving a {error, timeout} and calling > inet:close(): > > tcp_sendv(606): s=10, about to send 0,8192 bytes > tcp_sendv(606): s=10, only sent 512/2 of 8192/2 bytes/items > tcp_sendv(606): s=10, Send failed, queuing > sock_select(606): flags=02, onoff=1, event_mask=03 > tcp_inet_commandv(606) } > tcp_inet_commanv(606) {s=10 > tcp_sendv(606): s=10, sender forced busy > tcp_inet_commandv(606) } > tcp_inet_timeout(606) {s=10 > inet_reply_error_am 35 523 > tcp_inet_timeout(606) } > ** timeout - closing > ** CLOSING port_info: {connected,<0.2.0>} > tcp_inet_ctl(606): 24 > inet_ctl(606): INET_REQ_SUBSCRIBE > inet_ctl(606): Calling inet_subscribe > inet_subscribe returned 6 > tcp_inet_ctl(606): 11 > inet_ctl(606): GETSTAT > > Do you have any ideas why this happens? > > > TEST PROGRAM > ============= > > -module(test_tcp_close). > > -compile(export_all). > > main() -> > {ok, LSock} = gen_tcp:listen(1616, [binary, {packet,0}, {active, false}, > {reuseaddr, true}]), > > spawn(?MODULE, connect_and_idle, [1616]), > > {ok, Sock} = gen_tcp:accept(LSock), > > io:format("** Accepted a connection~n", []), > > ok = inet:setopts(Sock, [ > %% {send_timeout, 250}, {send_timeout_close, true}, > {send_timeout, 250}, > {active, once}, {nodelay, true}, {sndbuf, 131072}, {keepalive, true} > ]), > > fill_write(Sock). > > fill_write(Sock) -> > %% Make sure the output buffer becomes full so we trigger send_timeout. > Size = 8 * 1024 * 8, %% 8 KB (k *bytes*). > case gen_tcp:send(Sock, <<0:Size>>) of > {error, timeout} -> > io:format("** timeout - closing~n", []), > io:format("** CLOSING port_info: ~p~n", [catch erlang:port_info(Sock, > connected)]), > inet:close(Sock), > %% Wait at least INET_CLOSE_TIMEOUT and a bit > timer:sleep(10000), > io:format("** CLOSED port_info: ~p~n", [catch erlang:port_info(Sock, > connected)]); > {error, X} -> > io:format("** Unexpected send error: ~p~n", [X]), > io:format("** port_info: ~p~n", [catch erlang:port_info(Sock, connected)]); > ok -> > %% Uncomment to test an early close > %% io:format("** Closing Sock early~n", []), > %% inet:close(Sock), > %% io:format("** Closed~n", []), > timer:sleep(1000), > fill_write(Sock) > end. > > connect_and_idle(Port) -> > {ok, _Sock} = gen_tcp:connect("localhost", Port, [{active,false}, > {mode,binary}, {packet, 0}]), > timer:sleep(infinity). > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From define.null@REDACTED Mon Feb 11 16:02:42 2013 From: define.null@REDACTED (Vasily Demidenok) Date: Mon, 11 Feb 2013 19:02:42 +0400 Subject: [erlang-questions] Gproc, nb_wait and global name registration. Message-ID: In my application nb_wait function is used frequently and it turned out a dissapointment that gproc does not support globally registered names. While digging in the code I found out that such function as where works with global and local registered names in the common manner and to my mind the function nb_wait could work with it similiarly to local names. Are there any limitations or dificulties which i missed here, or it is just not implemented because of the other reasons? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinoski@REDACTED Mon Feb 11 16:09:30 2013 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 11 Feb 2013 10:09:30 -0500 Subject: [erlang-questions] Following a send_timeout inet:close() no longer closes the socket. In-Reply-To: References: Message-ID: Hi Paul, To add to what Lukas explained, using your test program on both OS X and Linux, lsof initially shows both the client socket and the server socket open while sending is occurring: beam.smp 23986 steve 9u IPv4 969680 0t0 TCP localhost:48399->localhost:1616 (ESTABLISHED) beam.smp 23986 steve 10u IPv4 969681 0t0 TCP localhost:1616->localhost:48399 (ESTABLISHED) If you enable {send_timeout_close,true}, once the sender hits its timeout, the server socket gets closed just as it should, and only the client socket remains open: beam.smp 23986 steve 9u IPv4 969680 0t0 TCP localhost:48399->localhost:1616 (ESTABLISHED) Your client isn't closed because it's passive and there's unreceived data in the socket. If you extend your client to recv the data that was sent, it gets {error, closed} immediately once all the data the server sent is consumed, and at that point lsof shows the client socket is also closed. As Lukas hinted, once you hand data to TCP it's going to try to keep sending, as that's what it's designed to do. Perhaps you can instead explain what you're trying to achieve, and maybe we can steer you in the right direction? --steve On Mon, Feb 11, 2013 at 9:58 AM, Lukas Larsson wrote: > Hello, > > If you do not use {send_timeout_close, true} on the socket, the stack will > try to send the data until it succeeds or the remote end closes the > connection. This means that the close will wait for the send buffer to > clear before closing the connection. > > The same is true when using port_close. The inet driver logic will detect > that there is data on the way out and wait for that data to be sent > before closing. > > So to get the behaviour you want you should set send_timeout_close to > true, setting this to true is also the recommended in the setting in the > documentation. > > I also noticed that in your testcase you did not close the listen socket > after the test was done which caused the testcase to leave the listen > socket open in between runs. > > Lukas > > > On Thu, Jan 31, 2013 at 12:45 AM, Paul Cager wrote: > >> I am seeing a problem with erts-5.9.3.1 regarding TCP and send_timeout. >> Please see the example program, below, but to summarise it: >> >> * A server accepts an incoming connection with options {send_timeout, >> 250}. >> * The server continually writes data until it gets an {error, >> timeout}. The server then calls inet:close(Sock). >> * However the socket isn't actually closed - netstat reports that it >> is still established (and stays that way for at least an hour, probably for >> ever). >> * I've enabled debug tracing in drivers/common/inet_drv.c (+ added a >> couple of debugs to show the return value of inet_subscribe). I've also >> added tracing to show when desc_close is being called in that module. >> * Normally calling inet:close() results in tcp_inet_stop and >> desc_close being called. When inet:close() is called *after* an {error, >> timeout} response "inet_ctl(606): INET_REQ_SUBSCRIBE" is called, but not >> tcp_inet_stop (or desc_close), which is consistent with the socket >> remaining ESTABLISHED. >> * Using port_close rather than inet:close shows the same behaviour. >> >> That does not sound like the correct behaviour. >> >> Here is the debug output for a "normal" inet:close(): >> >> tcp_inet_ctl(606): 24 >> inet_ctl(606): INET_REQ_SUBSCRIBE >> inet_ctl(606): Calling inet_subscribe >> inet_subscribe returned 6 >> tcp_inet_stop(606) {s=10 >> deq(606): queue empty >> release_buffer: 1460 >> tcp_inet_stop(606) } >> desc_close(606): 10 >> >> Here is the debug output when receiving a {error, timeout} and calling >> inet:close(): >> >> tcp_sendv(606): s=10, about to send 0,8192 bytes >> tcp_sendv(606): s=10, only sent 512/2 of 8192/2 bytes/items >> tcp_sendv(606): s=10, Send failed, queuing >> sock_select(606): flags=02, onoff=1, event_mask=03 >> tcp_inet_commandv(606) } >> tcp_inet_commanv(606) {s=10 >> tcp_sendv(606): s=10, sender forced busy >> tcp_inet_commandv(606) } >> tcp_inet_timeout(606) {s=10 >> inet_reply_error_am 35 523 >> tcp_inet_timeout(606) } >> ** timeout - closing >> ** CLOSING port_info: {connected,<0.2.0>} >> tcp_inet_ctl(606): 24 >> inet_ctl(606): INET_REQ_SUBSCRIBE >> inet_ctl(606): Calling inet_subscribe >> inet_subscribe returned 6 >> tcp_inet_ctl(606): 11 >> inet_ctl(606): GETSTAT >> >> Do you have any ideas why this happens? >> >> >> TEST PROGRAM >> ============= >> >> -module(test_tcp_close). >> >> -compile(export_all). >> >> main() -> >> {ok, LSock} = gen_tcp:listen(1616, [binary, {packet,0}, {active, false}, >> {reuseaddr, true}]), >> >> spawn(?MODULE, connect_and_idle, [1616]), >> >> {ok, Sock} = gen_tcp:accept(LSock), >> >> io:format("** Accepted a connection~n", []), >> >> ok = inet:setopts(Sock, [ >> %% {send_timeout, 250}, {send_timeout_close, true}, >> {send_timeout, 250}, >> {active, once}, {nodelay, true}, {sndbuf, 131072}, {keepalive, true} >> ]), >> >> fill_write(Sock). >> >> fill_write(Sock) -> >> %% Make sure the output buffer becomes full so we trigger send_timeout. >> Size = 8 * 1024 * 8, %% 8 KB (k *bytes*). >> case gen_tcp:send(Sock, <<0:Size>>) of >> {error, timeout} -> >> io:format("** timeout - closing~n", []), >> io:format("** CLOSING port_info: ~p~n", [catch erlang:port_info(Sock, >> connected)]), >> inet:close(Sock), >> %% Wait at least INET_CLOSE_TIMEOUT and a bit >> timer:sleep(10000), >> io:format("** CLOSED port_info: ~p~n", [catch erlang:port_info(Sock, >> connected)]); >> {error, X} -> >> io:format("** Unexpected send error: ~p~n", [X]), >> io:format("** port_info: ~p~n", [catch erlang:port_info(Sock, >> connected)]); >> ok -> >> %% Uncomment to test an early close >> %% io:format("** Closing Sock early~n", []), >> %% inet:close(Sock), >> %% io:format("** Closed~n", []), >> timer:sleep(1000), >> fill_write(Sock) >> end. >> >> connect_and_idle(Port) -> >> {ok, _Sock} = gen_tcp:connect("localhost", Port, [{active,false}, >> {mode,binary}, {packet, 0}]), >> timer:sleep(infinity). >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.pennebaker@REDACTED Mon Feb 11 16:09:31 2013 From: andrew.pennebaker@REDACTED (Andrew Pennebaker) Date: Mon, 11 Feb 2013 10:09:31 -0500 Subject: [erlang-questions] Best actor libraries in other languages? Message-ID: Of course Erlang has a killer actors library as a core component of itself. Sadly, Erlang is not always an option (e.g., customer requires a JVM language). Which actor libraries in other languages come close to Erlang's in terms of: * Syntax conciseness (spawn, !, receive) * Polymorphism (receiving different kinds of messages) * Fault-tolerance (GC per actor, actor crash doesn't crash the system) * Live updates (different code versions up at the same time, hot code updates) Does Scala and/or Akka offer anything like these features? What about Ruby's Celluloid? What are the best actor libraries in Common Lisp and Scheme? The Haskell community seems to prefer STM over actors. How does the Erlang community feel about STM? -- Cheers, Andrew Pennebaker www.yellosoft.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.pennebaker@REDACTED Mon Feb 11 16:13:15 2013 From: andrew.pennebaker@REDACTED (Andrew Pennebaker) Date: Mon, 11 Feb 2013 10:13:15 -0500 Subject: [erlang-questions] GADTs / polymorphic data types? Message-ID: Does Erlang have anything like Haskell GADTs? How are structured data types declared in Erlang? -- Cheers, Andrew Pennebaker www.yellosoft.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From comptekki@REDACTED Mon Feb 11 16:27:42 2013 From: comptekki@REDACTED (Wes James) Date: Mon, 11 Feb 2013 08:27:42 -0700 Subject: [erlang-questions] two questions -- -boot in erlsrv and .erlang.cookies on windows In-Reply-To: <004801ce0855$de6115f0$9b2341d0$@gmail.com> References: <004801ce0855$de6115f0$9b2341d0$@gmail.com> Message-ID: Thanks Janos. I'll take a look. Wes On Mon, Feb 11, 2013 at 5:46 AM, Janos Hary wrote: > On Windows you can place the .erlang and .erlang.cookie files in > ERL_TOP\usr. It?s not a per user location but safer then HOME. (By default > a service runs as System user and you can try setting HOME in the ?System > variables? section of the Environment Variables window. Alternatively you > can specify a different user for the service, but that user needs ?Log on > as service? permission.)**** > > ** ** > > For your other question I have no experience but according to the > documentation you should try invoking start_erl from erl_srv instead of > erl.exe.**** > > ** ** > > Janos**** > > ** ** > > *From:* erlang-questions-bounces@REDACTED [mailto: > erlang-questions-bounces@REDACTED] *On Behalf Of *Wes James > *Sent:* Friday, February 08, 2013 11:02 PM > *To:* erlang-questions@REDACTED > *Subject:* [erlang-questions] two questions -- -boot in erlsrv and > .erlang.cookies on windows**** > > ** ** > > I have -boot working on linux. Now I'm trying to get it to work with > erlsrv for windows. I put the -boot in the args part of erlsrv, but it > doesn't seem to take. Any ideas how to get this to work?**** > > Also, I've tried several different folders for .cookie.erlang on windows > and I can't get it to take. Even putting it in my c:\Users\username and > run the erlang app, it doesn't seem to be picked up. If I run with > -setcookie, it works fine. I run "set" on windows in cmd.exe and see the > HOME var is set. Also, if I can get this to work, where would the > .erlang.cookie go when run as a service. I just ran erl.exe and it did not > create an .erlang.cookie in my c:\Users\username folder (after I moved one > that I had manually put there). Is it supposed to create one on windows?* > *** > > Thanks, > > Wes**** > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Feb 11 16:44:53 2013 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 11 Feb 2013 16:44:53 +0100 Subject: [erlang-questions] Gproc, nb_wait and global name registration. In-Reply-To: References: Message-ID: There is a test case for nb_wait in the distributed case. What is it that doesn't work for you? BR, Ulf W (I note that there is an issue with global aggregated counters. I will try to take a look at that later.) On 11 Feb 2013, at 16:02, Vasily Demidenok wrote: > In my application nb_wait function is used frequently and it turned out a dissapointment that gproc does not support globally registered names. While digging in the code I found out that such function as where works with global and local registered names in the common manner and to my mind the function nb_wait could work with it similiarly to local names. Are there any limitations or dificulties which i missed here, or it is just not implemented because of the other reasons? _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From define.null@REDACTED Mon Feb 11 17:00:48 2013 From: define.null@REDACTED (Defnull) Date: Mon, 11 Feb 2013 08:00:48 -0800 (PST) Subject: [erlang-questions] Gproc, nb_wait and global name registration. In-Reply-To: References: Message-ID: <60273a6c-1929-4453-9a93-93ea94890a05@googlegroups.com> Sorry, my fault. Missed the call to gen_leader in the code. =( On Monday, February 11, 2013 7:44:53 PM UTC+4, Ulf Wiger wrote: > > > There is a test case for nb_wait in the distributed case. What is it that > doesn't work for you? > > BR, > Ulf W > > (I note that there is an issue with global aggregated counters. I will try > to take a look at that later.) > > On 11 Feb 2013, at 16:02, Vasily Demidenok wrote: > > In my application nb_wait function is used frequently and it turned out a > dissapointment that gproc does not support globally registered names. While > digging in the code I found out that such function as where works with > global and local registered names in the common manner and to my mind the > function nb_wait could work with it similiarly to local names. Are there > any limitations or dificulties which i missed here, or it is just not > implemented because of the other reasons? > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Mon Feb 11 17:03:46 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Mon, 11 Feb 2013 16:03:46 +0000 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: On 11 Feb 2013, at 15:09, Andrew Pennebaker wrote: > > The Haskell community seems to prefer STM over actors. How does the Erlang community feel about STM? > Not all the time - see http://haskell-distributed.github.com. Here we've basically copied Erlang's approach, though of course we use STM a lot internally to implement it. ;) Cheers, Tim From mattevans123@REDACTED Mon Feb 11 18:20:11 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Mon, 11 Feb 2013 12:20:11 -0500 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: I took a quick look at Kilim for Java: http://www.malhar.net/sriram/kilim/kilim_ecoop08.pdf It is also used by Erjang. However, I ended up with something so close to Erlang that we convinced our boss to use Erlang instead. Matt Date: Mon, 11 Feb 2013 10:09:31 -0500 From: andrew.pennebaker@REDACTED To: erlang-questions@REDACTED Subject: [erlang-questions] Best actor libraries in other languages? Of course Erlang has a killer actors library as a core component of itself. Sadly, Erlang is not always an option (e.g., customer requires a JVM language). Which actor libraries in other languages come close to Erlang's in terms of: * Syntax conciseness (spawn, !, receive)* Polymorphism (receiving different kinds of messages)* Fault-tolerance (GC per actor, actor crash doesn't crash the system) * Live updates (different code versions up at the same time, hot code updates) Does Scala and/or Akka offer anything like these features? What about Ruby's Celluloid? What are the best actor libraries in Common Lisp and Scheme? The Haskell community seems to prefer STM over actors. How does the Erlang community feel about STM? -- Cheers, Andrew Pennebakerwww.yellosoft.us _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From watson.timothy@REDACTED Mon Feb 11 18:23:51 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Mon, 11 Feb 2013 17:23:51 +0000 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: On 11 Feb 2013, at 17:20, Matthew Evans wrote: > I took a quick look at Kilim for Java: > > http://www.malhar.net/sriram/kilim/kilim_ecoop08.pdf > > It is also used by Erjang. However, I ended up with something so close to Erlang that we convinced our boss to use Erlang instead. > He he, yes - if you want Erlang style concurrency/distribution then choosing Erlang is a very sensible course of action! :) It's nice these other libraries are cropping up though - imitation is the sincerest form of flattery after all. From maruthavanan_s@REDACTED Mon Feb 11 18:29:11 2013 From: maruthavanan_s@REDACTED (Maruthavanan Subbarayan) Date: Mon, 11 Feb 2013 12:29:11 -0500 Subject: [erlang-questions] Alternatives to Erlang Message-ID: Hi, I am on the way to creating an application which would create atoms dynamically. These atoms are created because I have user registering into my system and each user has set of 10 mnesia tables to be created. Since atoms has a limitation I am afraid that if 100K users register into my system I may run of atoms for creating tables for more users. Creating an ETS table with name will give me table identifier with integer. Does a similar thing exists for mnesia too? How can I organize such a system? Thanks,Marutha -------------- next part -------------- An HTML attachment was scrubbed... URL: From maruthavanan_s@REDACTED Mon Feb 11 18:31:34 2013 From: maruthavanan_s@REDACTED (Maruthavanan Subbarayan) Date: Mon, 11 Feb 2013 12:31:34 -0500 Subject: [erlang-questions] Alternatives to Erlang In-Reply-To: References: Message-ID: Sorry for the typo.. The subject was supposed to be "Alternatives to Atoms/ mnesia table names" /RM From: maruthavanan_s@REDACTED To: erlang-questions@REDACTED Date: Mon, 11 Feb 2013 12:29:11 -0500 Subject: [erlang-questions] Alternatives to Erlang Hi, I am on the way to creating an application which would create atoms dynamically. These atoms are created because I have user registering into my system and each user has set of 10 mnesia tables to be created. Since atoms has a limitation I am afraid that if 100K users register into my system I may run of atoms for creating tables for more users. Creating an ETS table with name will give me table identifier with integer. Does a similar thing exists for mnesia too? How can I organize such a system? Thanks,Marutha _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Feb 11 18:50:45 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 11 Feb 2013 12:50:45 -0500 Subject: [erlang-questions] Alternatives to Erlang In-Reply-To: References: Message-ID: <20130211175045.GA43364@ferdmbp.local> I would first of all be more worried of having literally a million database tables if 100k users register than using as many atoms. What drives your design in a way that forces you to have so many database tables in the first place? Regards, Fred. On 02/11, Maruthavanan Subbarayan wrote: > Hi, > I am on the way to creating an application which would create atoms dynamically. > These atoms are created because I have user registering into my system and each user has set of 10 mnesia tables to be created. Since atoms has a limitation I am afraid that if 100K users register into my system I may run of atoms for creating tables for more users. > Creating an ETS table with name will give me table identifier with integer. Does a similar thing exists for mnesia too? > How can I organize such a system? > Thanks,Marutha > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mattevans123@REDACTED Mon Feb 11 19:06:19 2013 From: mattevans123@REDACTED (Matthew Evans) Date: Mon, 11 Feb 2013 13:06:19 -0500 Subject: [erlang-questions] Alternatives to Erlang In-Reply-To: <20130211175045.GA43364@ferdmbp.local> References: , <20130211175045.GA43364@ferdmbp.local> Message-ID: I would agree with this comment. Maybe instead you should think about a process for each user, and each process having a gb_tree in the state record to save data. If you want persistance you "could" use dets instead.... Matt > Date: Mon, 11 Feb 2013 12:50:45 -0500 > From: mononcqc@REDACTED > To: maruthavanan_s@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Alternatives to Erlang > > I would first of all be more worried of having literally a million > database tables if 100k users register than using as many atoms. > > What drives your design in a way that forces you to have so many > database tables in the first place? > > Regards, > Fred. > > On 02/11, Maruthavanan Subbarayan wrote: > > Hi, > > I am on the way to creating an application which would create atoms dynamically. > > These atoms are created because I have user registering into my system and each user has set of 10 mnesia tables to be created. Since atoms has a limitation I am afraid that if 100K users register into my system I may run of atoms for creating tables for more users. > > Creating an ETS table with name will give me table identifier with integer. Does a similar thing exists for mnesia too? > > How can I organize such a system? > > Thanks,Marutha > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Mon Feb 11 19:23:40 2013 From: raould@REDACTED (Raoul Duke) Date: Mon, 11 Feb 2013 10:23:40 -0800 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: https://github.com/laforge49/JActor From raould@REDACTED Mon Feb 11 19:24:34 2013 From: raould@REDACTED (Raoul Duke) Date: Mon, 11 Feb 2013 10:24:34 -0800 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: > The Haskell community seems to prefer STM over actors. How does the Erlang > community feel about STM? also, check the Clojure community. check Cliff Click from Azure. From jeremy@REDACTED Mon Feb 11 19:29:28 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Mon, 11 Feb 2013 10:29:28 -0800 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: Scala and Java have http://akka.io/ On Mon, Feb 11, 2013 at 10:24 AM, Raoul Duke wrote: > > The Haskell community seems to prefer STM over actors. How does the > Erlang > > community feel about STM? > > also, > > check the Clojure community. > > check Cliff Click from Azure. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Mon Feb 11 19:39:20 2013 From: steve@REDACTED (Steve Strong) Date: Mon, 11 Feb 2013 19:39:20 +0100 Subject: [erlang-questions] binary:compile_pattern - I'm very confused! In-Reply-To: References: Message-ID: <59EF02BC598045FF8818EAD11D6DBE61@srstrong.com> All makes sense, although for know I'll continue do describe it as evilness ;) Cheers, Steve -- Steve Strong Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Friday, 8 February 2013 at 23:36, Steve Vinoski wrote: > > > On Fri, Feb 8, 2013 at 5:09 PM, Steve Strong wrote: > > Hi, > > > > I've got a binary split that I'm doing fairly regularly, so I wanted to use a compiled pattern to make it more efficient. I don't have any state floating around (it's very much a utility function), so my plan was to compile the pattern in an erl shell and just paste the resultant term directly into my source code (with an appropriate comment!). However, things are not as they seem: > > > > Erlang R15B02 (erts-5.9.2) [source] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] > > > > Eshell V5.9.2 (abort with ^G) > > 1> C = binary:compile_pattern(<<0,0,1,0>>). > > {bm,<<>>} > > > > > > > On the Erlang binary module man page (http://www.erlang.org/doc/man/binary.html) it describes the return value of compile_pattern as follows: > > "Opaque data-type representing a compiled search-pattern. Guaranteed to be a tuple() to allow programs to distinguish it from non precompiled search patterns." > > The binary:compile_pattern function is a BIF that creates a "magic binary", much like the resources one can create in a NIF, and these appear as empty binaries in Erlang code as you can see in your shell output, but they're really opaque data types. Because of this, your approach won't work because the subsequent functions you're calling think you're passing an actual empty binary instead of the required opaque type. > > --steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Feb 11 20:55:03 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 11 Feb 2013 22:55:03 +0300 Subject: [erlang-questions] Alternatives to Erlang In-Reply-To: References: <20130211175045.GA43364@ferdmbp.local> Message-ID: If you have 10 tables per user, than you are using not mnesia as database, but erlang:register as your database. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostis@REDACTED Mon Feb 11 22:47:49 2013 From: kostis@REDACTED (Kostis Sagonas) Date: Mon, 11 Feb 2013 22:47:49 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <5114FD9D.3080304@erlang.org> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> <51147541.7060303@gmail.com> <5114FB3D.9020301@gmail.com> <5114FD9D.3080304@erlang.org> Message-ID: <51196705.2080802@cs.ntua.gr> On 02/08/2013 02:29 PM, Bj?rn-Egil Dahlberg wrote: > >> >> The erts_debug:flat_size/1 function is fast and implemented in C, but >> doesn't give you the true size since it ignores sharing. If Kostis >> Sagonas' work on sharing-preserving copying operations was included in >> OTP, we could have an efficient version of erts_debug:size/1. > > I haven't seen the final implementation yet. I wonder if it's complete? > Preserved sharing in copy would be really nice as long as it is efficient. First of all, I would like to correct Richard a bit. Although I am very heavily involved in the implementation of an Erlang VM that preserves term sharing, most of the work so far has been performed by my very good colleague Nikos Papaspyrou. A paper we wrote about this work has been published in the 2012 ACM Erlang Workshop and can be accessed at: http://dx.doi.org/10.1145/2364489.2364493 or more conveniently at: http://user.it.uu.se/~kostis/Papers/erlang12_sharing.pdf Although its contents are quite technical, I strongly recommend reading at least its introduction and Section 2 which explains all you wanted to know about term sharing in Erlang/OTP. Now, regarding the actual implementation it is also publicly available and can be found at: https://github.com/nickie/otp It's based on vanilla R16A but adds a --enable-sharing-preserving configure flag that can be optionally enabled to give a VM that preserves term sharing in intra-node process spawns and in message passing. As far as we know it works at well as the default system and its overhead is extremely small in all things we have tried. (Of course, it also gives quite big/unbounded? speedups whenever a lot of shared terms are copied between processes.) We have discussed with the OTP team the possibility of the system being included in R16B, so that the community has a chance to try and evaluate it, but no decision has been reached so far. The only remaining issue we are aware of has to do with code unloading: since terms in the constant pool of a module are not copied, as the vanilla system does, the code loader has to either preserve these terms or copy them on demand when a module is purged/reloaded. The OTP team has promised to look into this at some point but this work has not been prioritized yet. But the whole system should be usable without problems for all applications where unloading of modules with constants does not take place. Note that the implementation is restricted to intra-node term sharing: i.e. it does yet not support term sharing when terms are sent to different nodes. This is something we are currently working on and will most likely require the design of a new extern term format. But this is an orthogonal issue. As a relatively minor point we have not tested this on Windows; though we suspect that there are only minor issues to possibly address there. Somebody with a Windows development environment should do that (hint, hint...). Kostis From comptekki@REDACTED Mon Feb 11 22:59:23 2013 From: comptekki@REDACTED (Wes James) Date: Mon, 11 Feb 2013 14:59:23 -0700 Subject: [erlang-questions] erlsrv -d console Message-ID: When I run erlsrv -d console on win8 test machine, (should be same on win7 if you have that) I'm not getting a console. If I change it to new, I do get a debug file in -w folder. Can anyone verify this on win7/8 to see if a console opens with -d console. My command line is: erlsrv add "srvc-name" -name node@REDACTED -d console -w C:/somepath -args "-s module" I'm ultimately trying to figure out why when erlsrv is run, a .erlang.cookie file is not being created in c:/somepath or is not being read from c:/somepath if I put a custom one there or even in ERL_TOP/usr. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Feb 12 00:24:59 2013 From: ok@REDACTED (ok@REDACTED) Date: Tue, 12 Feb 2013 12:24:59 +1300 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: <79caa2913be419e6f7c99efc7a4594a3.squirrel@chasm.otago.ac.nz> > The Haskell community seems to prefer STM over actors. How does the Erlang > community feel about STM? I'm not the Erlang community, but I hope I'm part of it. There is a very good thing about Transactional Memory and two rather bad things. + The very good thing is that it composes. - One bad thing is that when contention is high it can thrash. - Another is that it encourages you to keep thinking imperatively. There is a very special thing about TM in Haskell, which is that it encourages you to keep the number of mutable variables small. Quoting "Real World Haskell": Even in a concurrent program, we strive to keep as much of our code as possible purely functional. This makes our code easier both to reason about and to test. It also gives the underlying STM engine less work to do, since the data involved is not transactional. This makes TM programming in Haskell very different from TM programming in Java. Actors fit distribution rather better than TM does, and Erlang comes from a distributed programming context, while Haskell has historically been more concerned with single machine programs. From ok@REDACTED Tue Feb 12 00:27:23 2013 From: ok@REDACTED (ok@REDACTED) Date: Tue, 12 Feb 2013 12:27:23 +1300 Subject: [erlang-questions] Alternatives to Erlang In-Reply-To: References: Message-ID: "Maruthavanan Subbarayan" wrote: > These atoms are created because I have user registering into my system and > each user has set of 10 mnesia tables to be created. Since atoms has a > limitation I am afraid that if 100K users register into my system I may > run of atoms for creating tables for more users. It seems a little odd to have even one table for each user. Would you care to explain your design in a little more detail? Suppose you were planning to use CouchDB or Riak as your persistent data store. What would your design look like then? From mjtruog@REDACTED Tue Feb 12 00:38:49 2013 From: mjtruog@REDACTED (Michael Truog) Date: Mon, 11 Feb 2013 15:38:49 -0800 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: <51198109.2040803@gmail.com> Kilim is a bit old and supposedly jetlang is preferred (http://code.google.com/p/jetlang/)... it is used in akka for its actors implementation which is better than the normal Scala actors. However, I agree it is better to just use Erlang. You can try to have fault-tolerant Java source code, but the JVM has a shared heap, so there is no principle of isolation at the core. So, if you try hard enough, you can always put a square peg in a round hole, but why bother? On 02/11/2013 09:20 AM, Matthew Evans wrote: > I took a quick look at Kilim for Java: > > http://www.malhar.net/sriram/kilim/kilim_ecoop08.pdf > > It is also used by Erjang. However, I ended up with something so close to Erlang that we convinced our boss to use Erlang instead. > > Matt > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > Date: Mon, 11 Feb 2013 10:09:31 -0500 > From: andrew.pennebaker@REDACTED > To: erlang-questions@REDACTED > Subject: [erlang-questions] Best actor libraries in other languages? > > Of course Erlang has a killer actors library as a core component of itself. Sadly, Erlang is not always an option (e.g., customer requires a JVM language). Which actor libraries in other languages come close to Erlang's in terms of: > > * Syntax conciseness (spawn, !, receive) > * Polymorphism (receiving different kinds of messages) > * Fault-tolerance (GC per actor, actor crash doesn't crash the system) > * Live updates (different code versions up at the same time, hot code updates) > > Does Scala and/or Akka offer anything like these features? What about Ruby's Celluloid? What are the best actor libraries in Common Lisp and Scheme? > > The Haskell community seems to prefer STM over actors. How does the Erlang community feel about STM? > > -- > Cheers, > > Andrew Pennebaker > www.yellosoft.us > > _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Feb 12 02:06:56 2013 From: ok@REDACTED (ok@REDACTED) Date: Tue, 12 Feb 2013 14:06:56 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> References: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> Message-ID: <1c6adb6f76e1a669ecc99054a1f043e9.squirrel@chasm.otago.ac.nz> "Lo?c Hoguin" wrote > - Micro optimization. Most applications wouldn't care for it either Applications cannot care or not care for anything. If people are willing to pay the price, fine; my problem was that I saw no evidence that any proponent realised that there _was_ a price "I just don't see the harm". > - That's a developer's choice Making code less readable may be a developer's _choice_, but if anyone else might be required to read the code, it isn't a developer's _right_. > - BS, you have the same issue with Var = #rec.field, no difference there Reducing the effectiveness of type checking? Yes, we do have the same issue with Var = #rec.field, BUT I don't say that using that is a good idea either. > - See previous point Inadvertently providing wider access? No, this one is quite different from #rec.field. You can only use #rec.field within the compile-time scope of a -record declaration for 'rec'. If #rec.Field were allowed, Field could be created at a point where the record declaration was _not_ visible. #rec.field has to be inside the module; use of it is not inadvertent; and you can _find_ it trivially with a text editor, so you can very easily check which fields might be accessed that way. As far as I can see, that's not true of #rec.Field, where Field comes from some unknown arbitrary module. The key point is that we just plain DON'T NEED #rec.Field; anything we could do with it can be done without it more clearly using existing mechanisms, with better control over access and all sorts of other programmer-liked goodness. Erlang needs dynamic field access like oysters need shooting-sticks. From essen@REDACTED Tue Feb 12 03:04:31 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 12 Feb 2013 03:04:31 +0100 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <1c6adb6f76e1a669ecc99054a1f043e9.squirrel@chasm.otago.ac.nz> References: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> <1c6adb6f76e1a669ecc99054a1f043e9.squirrel@chasm.otago.ac.nz> Message-ID: <5119A32F.6090108@ninenines.eu> On 02/12/2013 02:06 AM, ok@REDACTED wrote: > "Lo?c Hoguin" wrote > >> - Micro optimization. Most applications wouldn't care for it either > Applications cannot care or not care for anything. > If people are willing to pay the price, fine; > my problem was that I saw no evidence that any proponent > realised that there _was_ a price "I just don't see the harm". Pedantic point. If execution speed isn't an application requirement, then there's no need to even concern yourself about it, because everything else is more important. You'll also most likely improve the execution speed of your system by improving inter-process communication and application architecture rather than having to execute one less instruction here and there. >> - That's a developer's choice > Making code less readable may be a developer's _choice_, > but if anyone else might be required to read the code, > it isn't a developer's _right_. You seem to think code readability isn't subjective. >> - BS, you have the same issue with Var = #rec.field, no difference there > Reducing the effectiveness of type checking? > Yes, we do have the same issue with Var = #rec.field, > BUT I don't say that using that is a good idea either. Sometimes there's no good solution and you still have to get things done. However, as far as "bad" solutions go, #rec.field is worse than 'field', because you'll get a meaningless integer when you output the value when debugging, instead of what could have been an atom. >> - See previous point > Inadvertently providing wider access? No, this one is > quite different from #rec.field. You can only use > #rec.field within the compile-time scope of a -record > declaration for 'rec'. If #rec.Field were allowed, > Field could be created at a point where the record > declaration was _not_ visible. #rec.field has to be > inside the module; use of it is not inadvertent; and > you can _find_ it trivially with a text editor, so > you can very easily check which fields might be accessed > that way. As far as I can see, that's not true of > #rec.Field, where Field comes from some unknown arbitrary > module. Well, #rec.Field is not possible to begin with, and wouldn't make sense for records. But it would definitely be good for another data type, as I'll explain. > The key point is that we just plain DON'T NEED #rec.Field; > anything we could do with it can be done without it more > clearly using existing mechanisms, with better control over > access and all sorts of other programmer-liked goodness. There's nothing clear in having to write: g(bindings, #http_req{bindings=Ret}) -> Ret; g(body_state, #http_req{body_state=Ret}) -> Ret; g(buffer, #http_req{buffer=Ret}) -> Ret; g(connection, #http_req{connection=Ret}) -> Ret; g(cookies, #http_req{cookies=Ret}) -> Ret; g(fragment, #http_req{fragment=Ret}) -> Ret; g(headers, #http_req{headers=Ret}) -> Ret; g(host, #http_req{host=Ret}) -> Ret; g(host_info, #http_req{host_info=Ret}) -> Ret; g(meta, #http_req{meta=Ret}) -> Ret; g(method, #http_req{method=Ret}) -> Ret; g(multipart, #http_req{multipart=Ret}) -> Ret; g(onresponse, #http_req{onresponse=Ret}) -> Ret; g(p_headers, #http_req{p_headers=Ret}) -> Ret; g(path, #http_req{path=Ret}) -> Ret; g(path_info, #http_req{path_info=Ret}) -> Ret; g(peer, #http_req{peer=Ret}) -> Ret; g(pid, #http_req{pid=Ret}) -> Ret; g(port, #http_req{port=Ret}) -> Ret; g(qs, #http_req{qs=Ret}) -> Ret; g(qs_vals, #http_req{qs_vals=Ret}) -> Ret; g(resp_body, #http_req{resp_body=Ret}) -> Ret; g(resp_headers, #http_req{resp_headers=Ret}) -> Ret; g(resp_state, #http_req{resp_state=Ret}) -> Ret; g(socket, #http_req{socket=Ret}) -> Ret; g(transport, #http_req{transport=Ret}) -> Ret; g(version, #http_req{version=Ret}) -> Ret. Instead of g(Field, Req) -> Req#http_req.Field. Better control over access? You can still do it in the guard if you need to. I'm sure you'll tell me this should be generated? That's neither clear nor practical. And the generated code would likely not give good control over access either anyway. > Erlang needs dynamic field access like oysters need shooting-sticks. For records I agree, for almost anything else (dict | sets | proplists | maps | ...) I don't. Especially more so in Erlang. You're being very affirmative over what people do or don't need, but you seem to forget the one thing that people need the most: getting things done. Perhaps a solution is slightly more prone to some kinds of bugs, however making it harder to do something is as likely to create bugs. Erlang is wonderful in that you can easily contain bugs to the local process or group of processes, so I don't really see why we wouldn't want to have more productive solutions. (And yes, productivity is also subjective.) -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From dcartt@REDACTED Tue Feb 12 03:35:15 2013 From: dcartt@REDACTED (David Cartt) Date: Mon, 11 Feb 2013 18:35:15 -0800 Subject: [erlang-questions] Dynamically access record fields Message-ID: You could use record_info(fields, Record) to get an ordered list of the record fields and use that to find the tuple index of the requested field. Although the fields are returned in order, the documentation does not specify that as far as I've seen, only that it returns a list of the fields. A nasty bug would result if in a future release it stopped returning the fields in order, or even better, if for some records or in some situations it didn't return the fields in order. Hard coding your own field index table would get around this potential problem. Here's an example: -record(r, {r1="1", r2="2", r3="3"}). t(F, NewV) -> R = #r{}, OldV = get_value(F, R), R1 = set_value(F, R, NewV), ?debugFmt("R=~p, R1=~p, Old=~p",[R, R1, OldV]). get_value(F, R) -> element(field_index(F), R). set_value(F, R, V) -> setelement(field_index(F), R, V). field_index(F) -> index(F, record_info(fields, r), 2). index(M, [M|_], I) -> I; index(M, [_|T], I) -> index(M, T, I+1). Calling: t(r1, new) prints out: R={r,"1","2","3"}, R1={r,new,"2","3"}, Old="1" -------------- next part -------------- An HTML attachment was scrubbed... URL: From vatamane@REDACTED Tue Feb 12 04:04:55 2013 From: vatamane@REDACTED (Nick Vatamaniuc) Date: Mon, 11 Feb 2013 22:04:55 -0500 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: <5119B157.4080204@gmail.com> There is python-actors https://bitbucket.org/fzzzy/python-actors Features: * Receive pattern matching based on data shape. For example, [1,2,3] will match [int] and {"a":"b"} would match {"a":str} * Sending messages via "pid | msg" syntax * tcp and udp ports, with 'passive', 'active' and 'once' modes. * Linked processes * Inter-process messages via a rest service (using wsgi) It tries to copy Erlang, but it is far from it. Use it for experiments, but consider learning Erlang if you can. -Nick On 02/11/2013 10:09 AM, Andrew Pennebaker wrote: > Of course Erlang has a killer actors library as a core component of > itself. Sadly, Erlang is not always an option (e.g., customer requires a > JVM language). Which actor libraries in other languages come close to > Erlang's in terms of: > > * Syntax conciseness (spawn, !, receive) > * Polymorphism (receiving different kinds of messages) > * Fault-tolerance (GC per actor, actor crash doesn't crash the system) > * Live updates (different code versions up at the same time, hot code > updates) > > Does Scala and/or Akka offer anything like these features? What about > Ruby's Celluloid? What are the best actor libraries in Common Lisp and > Scheme? > > The Haskell community seems to prefer STM over actors. How does the > Erlang community feel about STM? > > -- > Cheers, > > Andrew Pennebaker > www.yellosoft.us > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From janos.n.hary@REDACTED Tue Feb 12 08:34:13 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Tue, 12 Feb 2013 08:34:13 +0100 Subject: [erlang-questions] erlsrv -d console In-Reply-To: References: Message-ID: <000f01ce08f3$5b7284f0$12578ed0$@gmail.com> Wes, I tested your command line on Windows 7 Ultimate. When the service starts a window named 'Interactive Services Detection' pops up. It has a 'View the messages' button. Pressing that changes to an empty desktop where I can interact with an erl.exe shell, also having a similar window on that desktop to switch back to my normal environment. Next I changed the service Log on user from Local System to my account, but then neither saw the pop up nor the shell window. After running the service I found an .erlang.cookie file created in my C:\Windows folder containing the value displayed by erlang:get_cookie/0 in the service's shell. But I found no way to use a cookie file placed elsewhere. Janos From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Wes James Sent: Monday, February 11, 2013 10:59 PM To: erlang-questions@REDACTED Subject: [erlang-questions] erlsrv -d console When I run erlsrv -d console on win8 test machine, (should be same on win7 if you have that) I'm not getting a console. If I change it to new, I do get a debug file in -w folder. Can anyone verify this on win7/8 to see if a console opens with -d console. My command line is: erlsrv add "srvc-name" -name node@REDACTED -d console -w C:/somepath -args "-s module" I'm ultimately trying to figure out why when erlsrv is run, a .erlang.cookie file is not being created in c:/somepath or is not being read from c:/somepath if I put a custom one there or even in ERL_TOP/usr. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Feb 12 10:45:06 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 12 Feb 2013 10:45:06 +0100 Subject: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang In-Reply-To: <51196705.2080802@cs.ntua.gr> References: <85819AB4-AD04-401F-B814-88157B71BE07@cs.otago.ac.nz> <5113C7F5.9050500@erlang.org> <51145C7B.2030806@gmail.com> <7f3a113d335f690d5d826ea0f5b33f59.squirrel@chasm.otago.ac.nz> <51147541.7060303@gmail.com> <5114FB3D.9020301@gmail.com> <5114FD9D.3080304@erlang.org> <51196705.2080802@cs.ntua.gr> Message-ID: <511A0F22.3010006@gmail.com> On 2013-02-11 22:47, Kostis Sagonas wrote: > The only remaining issue we are aware of has to do with code unloading: > since terms in the constant pool of a module are not copied, as the > vanilla system does, the code loader has to either preserve these terms > or copy them on demand when a module is purged/reloaded. The OTP team > has promised to look into this at some point but this work has not been > prioritized yet. But the whole system should be usable without problems > for all applications where unloading of modules with constants does not > take place. While trying to figure out the much harder problem of garbage collecting globally shared constant pool data, couldn't you modify your copying routine to just make an old-style non-sharing-preserving copy of the subterms that belong to the constant pool? Since you already check whether a pointer points to the local heap (the common case), it should be cheap to add some extra check to the case of pointers outside the heap. Possibly, you don't even have to do any extra checking, just default to the old-style copying routines for all off-heap subterms. This doesn't preserve sharing within constant terms in modules when sent as messages, but at least would enable sharing-preserving copying for all dynamic data. Or am I missing something? /Richard From evrenweb@REDACTED Tue Feb 12 10:50:18 2013 From: evrenweb@REDACTED (Evren Bayraktar) Date: Tue, 12 Feb 2013 11:50:18 +0200 Subject: [erlang-questions] MongoDB EnsureIndex Message-ID: Hi everyone, Is there a any one who uses emongo or another library to use mongodb in erlang ? I couldn't use ensureindex property with emongo. I don't know whether it is possible or not :) I saw ensure_index function in this file ( https://github.com/JacobVorreuter/emongo/blob/master/src/emongo.erl ) then used it but it always gets ok. Actually, I want to use Mongo's Geospatial property with erlang. Thats why I guess, I need to use ensureIndex. If there is any other method to use geospatial search in Erlang I can use it. Best Regards ! Evren -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve@REDACTED Tue Feb 12 10:53:20 2013 From: steve@REDACTED (Steve Strong) Date: Tue, 12 Feb 2013 10:53:20 +0100 Subject: [erlang-questions] MongoDB EnsureIndex In-Reply-To: References: Message-ID: <510C6568056842459431E6A1F33D3685@srstrong.com> Hi, We use https://github.com/mongodb/mongodb-erlang for erlang - seems to work well enough, and certainly supports creating indexes etc. cheers, Steve -- Steve Strong Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, 12 February 2013 at 10:50, Evren Bayraktar wrote: > Hi everyone, > > Is there a any one who uses emongo or another library to use mongodb in erlang ? I couldn't use ensureindex property with emongo. I don't know whether it is possible or not :) > > I saw ensure_index function in this file ( https://github.com/JacobVorreuter/emongo/blob/master/src/emongo.erl ) then used it but it always gets ok. > > Actually, I want to use Mongo's Geospatial property with erlang. Thats why I guess, I need to use ensureIndex. If there is any other method to use geospatial search in Erlang I can use it. > > Best Regards ! > Evren > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED (mailto:erlang-questions@REDACTED) > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dm.klionsky@REDACTED Tue Feb 12 11:23:17 2013 From: dm.klionsky@REDACTED (Dmitry Klionsky) Date: Tue, 12 Feb 2013 13:23:17 +0300 Subject: [erlang-questions] MongoDB EnsureIndex In-Reply-To: <510C6568056842459431E6A1F33D3685@srstrong.com> References: <510C6568056842459431E6A1F33D3685@srstrong.com> Message-ID: <511A1815.5030508@gmail.com> +1 On 02/12/2013 12:53 PM, Steve Strong wrote: > Hi, > > We use https://github.com/mongodb/mongodb-erlang for erlang - seems to > work well enough, and certainly supports creating indexes etc. > > cheers, > > Steve > > -- > Steve Strong > Sent with Sparrow > > On Tuesday, 12 February 2013 at 10:50, Evren Bayraktar wrote: > >> Hi everyone, >> >> Is there a any one who uses emongo or another library to use mongodb >> in erlang ? I couldn't use ensureindex property with emongo. I don't >> know whether it is possible or not :) >> >> I saw ensure_index function in this file ( >> https://github.com/JacobVorreuter/emongo/blob/master/src/emongo.erl ) >> then used it but it always gets ok. >> >> Actually, I want to use Mongo's Geospatial property with erlang. >> Thats why I guess, I need to use ensureIndex. If there is any other >> method to use geospatial search in Erlang I can use it. >> >> Best Regards ! >> Evren >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Best regards, Dmitry Klionsky -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Tue Feb 12 13:00:09 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Tue, 12 Feb 2013 04:00:09 -0800 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) In-Reply-To: References: Message-ID: I've updated this library to include ping/pong/close frames now (this is the last time I spam the listserv about this) so it is fairly usable. I also fixed a bug with the handshake validation because I didn't read the crypto documentation carefully. The example ( https://github.com/jeremyong/websocket_client/blob/master/examples/sample_ws_handler.erl) should be simple enough for anybody to get started if they're interested. On Mon, Feb 11, 2013 at 12:15 AM, Lo?c Hoguin wrote: > Note that Cowboy rejects unmasked clients now. > > > Max Lapshin wrote: > > Yes, I skip all these things. > > Hope that you will be more diligent than me =) > > > On Mon, Feb 11, 2013 at 12:05 PM, Jeremy Ong wrote: > >> Thanks Max I'll take a look. Unfortunately, I need ssl so I'll just keep >> hacking away at the one I just released for now. >> >> Additional points (that you may not care about for your purposes, but >> matter to me): >> >> It looks like you aren't doing any challenge verification. There doesn't >> seem to be any length validation on non-text/binary frames. Also, you >> aren't doing client to server masking. >> >> >> On Sun, Feb 10, 2013 at 11:41 PM, Max Lapshin wrote: >> >>> Hi. >>> >>> This works for me excelently: >>> >>> >>> https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/websocket_client.erl >>> >>> Perhaps it will help you >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Feb 12 13:08:22 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 12 Feb 2013 15:08:22 +0300 Subject: [erlang-questions] I've started to implement a 6455 compliant erlang websocket client (help/users appreciated) In-Reply-To: References: Message-ID: Great, thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From be.dmitry@REDACTED Tue Feb 12 13:43:07 2013 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Tue, 12 Feb 2013 16:43:07 +0400 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <5119A32F.6090108@ninenines.eu> References: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> <1c6adb6f76e1a669ecc99054a1f043e9.squirrel@chasm.otago.ac.nz> <5119A32F.6090108@ninenines.eu> Message-ID: > g(bindings, #http_req{bindings=Ret}) -> Ret; > g(body_state, #http_req{body_state=Ret}) -> Ret; > g(buffer, #http_req{buffer=Ret}) -> Ret; > g(connection, #http_req{connection=Ret}) -> Ret; > g(cookies, #http_req{cookies=Ret}) -> Ret; > g(fragment, #http_req{fragment=Ret}) -> Ret; > g(headers, #http_req{headers=Ret}) -> Ret; > g(host, #http_req{host=Ret}) -> Ret; > g(host_info, #http_req{host_info=Ret}) -> Ret; > g(meta, #http_req{meta=Ret}) -> Ret; > g(method, #http_req{method=Ret}) -> Ret; > g(multipart, #http_req{multipart=Ret}) -> Ret; > g(onresponse, #http_req{onresponse=Ret}) -> Ret; > g(p_headers, #http_req{p_headers=Ret}) -> Ret; > g(path, #http_req{path=Ret}) -> Ret; > g(path_info, #http_req{path_info=Ret}) -> Ret; > g(peer, #http_req{peer=Ret}) -> Ret; > g(pid, #http_req{pid=Ret}) -> Ret; > g(port, #http_req{port=Ret}) -> Ret; > g(qs, #http_req{qs=Ret}) -> Ret; > g(qs_vals, #http_req{qs_vals=Ret}) -> Ret; > g(resp_body, #http_req{resp_body=Ret}) -> Ret; > g(resp_headers, #http_req{resp_headers=Ret}) -> Ret; > g(resp_state, #http_req{resp_state=Ret}) -> Ret; > g(socket, #http_req{socket=Ret}) -> Ret; > g(transport, #http_req{transport=Ret}) -> Ret; > g(version, #http_req{version=Ret}) -> Ret. It is possible to write: -record(record1, {a, b, c}). -record(record2, {c, b, a}). g(Field, #record1{} = Record) -> g(Field, Record, record_info(fields, record1), 2); g(Field, #record2{} = Record) -> g(Field, Record, record_info(fields, record2), 2). g(Field, Record, [Field | _], I) -> element(I, Record); g(Field, Record, [_ | Fields], I) -> g(Field, Record, Fields, I + 1); g(_, _, _, _) -> error(badarg). -- Dmitry Belyaev On 12.02.2013, at 6:04, Lo?c Hoguin wrote: > On 02/12/2013 02:06 AM, ok@REDACTED wrote: >> "Lo?c Hoguin" wrote >> >>> - Micro optimization. Most applications wouldn't care for it either >> Applications cannot care or not care for anything. >> If people are willing to pay the price, fine; >> my problem was that I saw no evidence that any proponent >> realised that there _was_ a price "I just don't see the harm". > > Pedantic point. If execution speed isn't an application requirement, then there's no need to even concern yourself about it, because everything else is more important. > > You'll also most likely improve the execution speed of your system by improving inter-process communication and application architecture rather than having to execute one less instruction here and there. > >>> - That's a developer's choice >> Making code less readable may be a developer's _choice_, >> but if anyone else might be required to read the code, >> it isn't a developer's _right_. > > You seem to think code readability isn't subjective. > >>> - BS, you have the same issue with Var = #rec.field, no difference there >> Reducing the effectiveness of type checking? >> Yes, we do have the same issue with Var = #rec.field, >> BUT I don't say that using that is a good idea either. > > Sometimes there's no good solution and you still have to get things done. However, as far as "bad" solutions go, #rec.field is worse than 'field', because you'll get a meaningless integer when you output the value when debugging, instead of what could have been an atom. > >>> - See previous point >> Inadvertently providing wider access? No, this one is >> quite different from #rec.field. You can only use >> #rec.field within the compile-time scope of a -record >> declaration for 'rec'. If #rec.Field were allowed, >> Field could be created at a point where the record >> declaration was _not_ visible. #rec.field has to be >> inside the module; use of it is not inadvertent; and >> you can _find_ it trivially with a text editor, so >> you can very easily check which fields might be accessed >> that way. As far as I can see, that's not true of >> #rec.Field, where Field comes from some unknown arbitrary >> module. > > Well, #rec.Field is not possible to begin with, and wouldn't make sense for records. But it would definitely be good for another data type, as I'll explain. > >> The key point is that we just plain DON'T NEED #rec.Field; >> anything we could do with it can be done without it more >> clearly using existing mechanisms, with better control over >> access and all sorts of other programmer-liked goodness. > > There's nothing clear in having to write: > > g(bindings, #http_req{bindings=Ret}) -> Ret; > g(body_state, #http_req{body_state=Ret}) -> Ret; > g(buffer, #http_req{buffer=Ret}) -> Ret; > g(connection, #http_req{connection=Ret}) -> Ret; > g(cookies, #http_req{cookies=Ret}) -> Ret; > g(fragment, #http_req{fragment=Ret}) -> Ret; > g(headers, #http_req{headers=Ret}) -> Ret; > g(host, #http_req{host=Ret}) -> Ret; > g(host_info, #http_req{host_info=Ret}) -> Ret; > g(meta, #http_req{meta=Ret}) -> Ret; > g(method, #http_req{method=Ret}) -> Ret; > g(multipart, #http_req{multipart=Ret}) -> Ret; > g(onresponse, #http_req{onresponse=Ret}) -> Ret; > g(p_headers, #http_req{p_headers=Ret}) -> Ret; > g(path, #http_req{path=Ret}) -> Ret; > g(path_info, #http_req{path_info=Ret}) -> Ret; > g(peer, #http_req{peer=Ret}) -> Ret; > g(pid, #http_req{pid=Ret}) -> Ret; > g(port, #http_req{port=Ret}) -> Ret; > g(qs, #http_req{qs=Ret}) -> Ret; > g(qs_vals, #http_req{qs_vals=Ret}) -> Ret; > g(resp_body, #http_req{resp_body=Ret}) -> Ret; > g(resp_headers, #http_req{resp_headers=Ret}) -> Ret; > g(resp_state, #http_req{resp_state=Ret}) -> Ret; > g(socket, #http_req{socket=Ret}) -> Ret; > g(transport, #http_req{transport=Ret}) -> Ret; > g(version, #http_req{version=Ret}) -> Ret. > > Instead of > > g(Field, Req) -> Req#http_req.Field. > > Better control over access? You can still do it in the guard if you need to. > > I'm sure you'll tell me this should be generated? That's neither clear nor practical. And the generated code would likely not give good control over access either anyway. > >> Erlang needs dynamic field access like oysters need shooting-sticks. > > For records I agree, for almost anything else (dict | sets | proplists | maps | ...) I don't. Especially more so in Erlang. > > You're being very affirmative over what people do or don't need, but you seem to forget the one thing that people need the most: getting things done. Perhaps a solution is slightly more prone to some kinds of bugs, however making it harder to do something is as likely to create bugs. Erlang is wonderful in that you can easily contain bugs to the local process or group of processes, so I don't really see why we wouldn't want to have more productive solutions. (And yes, productivity is also subjective.) > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Tue Feb 12 14:01:58 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 12 Feb 2013 14:01:58 +0100 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> <1c6adb6f76e1a669ecc99054a1f043e9.squirrel@chasm.otago.ac.nz> <5119A32F.6090108@ninenines.eu> Message-ID: <511A3D46.6080301@ninenines.eu> On 02/12/2013 01:43 PM, Dmitry Belyaev wrote: >> g(bindings, #http_req{bindings=Ret}) -> Ret; >> g(body_state, #http_req{body_state=Ret}) -> Ret; >> g(buffer, #http_req{buffer=Ret}) -> Ret; >> g(connection, #http_req{connection=Ret}) -> Ret; >> g(cookies, #http_req{cookies=Ret}) -> Ret; >> g(fragment, #http_req{fragment=Ret}) -> Ret; >> g(headers, #http_req{headers=Ret}) -> Ret; >> g(host, #http_req{host=Ret}) -> Ret; >> g(host_info, #http_req{host_info=Ret}) -> Ret; >> g(meta, #http_req{meta=Ret}) -> Ret; >> g(method, #http_req{method=Ret}) -> Ret; >> g(multipart, #http_req{multipart=Ret}) -> Ret; >> g(onresponse, #http_req{onresponse=Ret}) -> Ret; >> g(p_headers, #http_req{p_headers=Ret}) -> Ret; >> g(path, #http_req{path=Ret}) -> Ret; >> g(path_info, #http_req{path_info=Ret}) -> Ret; >> g(peer, #http_req{peer=Ret}) -> Ret; >> g(pid, #http_req{pid=Ret}) -> Ret; >> g(port, #http_req{port=Ret}) -> Ret; >> g(qs, #http_req{qs=Ret}) -> Ret; >> g(qs_vals, #http_req{qs_vals=Ret}) -> Ret; >> g(resp_body, #http_req{resp_body=Ret}) -> Ret; >> g(resp_headers, #http_req{resp_headers=Ret}) -> Ret; >> g(resp_state, #http_req{resp_state=Ret}) -> Ret; >> g(socket, #http_req{socket=Ret}) -> Ret; >> g(transport, #http_req{transport=Ret}) -> Ret; >> g(version, #http_req{version=Ret}) -> Ret. > > It is possible to write: > > -record(record1, {a, b, c}). > -record(record2, {c, b, a}). > > g(Field, #record1{} = Record) -> > g(Field, Record, record_info(fields, record1), 2); > g(Field, #record2{} = Record) -> > g(Field, Record, record_info(fields, record2), 2). > > g(Field, Record, [Field | _], I) -> > element(I, Record); > g(Field, Record, [_ | Fields], I) -> > g(Field, Record, Fields, I + 1); > g(_, _, _, _) -> > error(badarg). Yes, you can write this, but copy pasting clauses was much faster to write and a lot less prone to bugs. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From ok@REDACTED Tue Feb 12 14:06:23 2013 From: ok@REDACTED (ok@REDACTED) Date: Wed, 13 Feb 2013 02:06:23 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: References: Message-ID: <3bf6ebe0fc4d01686aa2c2b32ef5c67d.squirrel@chasm.otago.ac.nz> I still say that Erlang needs dynamic record fields the way oysters need shooting sticks, but here's how it could be done. Step 1. "To each module using records, a pseudo function is added during compilation to obtain information about records: record_info(fields, Record) -> [Field] record_info(size, Record) -> Size " Extend this with record_info({index,Field}, Record) -> Index The term "pseudo function" (which ought to be "pseudo-function" or "pseudofunction") refers to the way the compiler reports an error if you call record_info/2 with non-literal arguments. Step 2. Instead of reporting an error, have the compiler report a warning, and actually generate an executable module-local definition of the record_info/2 function; the existing behaviour then just becomes the compiler special-casing a known function, which is no more surprising than compiling 1+1 as 2. Step 3. There is no step 3. If you want there to be a step 3, you can write -compile({inline, [field/3,field/3]}). field(Tuple, Record, Field) when is_tuple(Tuple), tuple_size(Tuple) == record_info(size, Record), element(1,Tuple) == Record -> I = record_info({index,Field}, Record), element(I, Tuple). field(Tuple, Record, Field, New_Value) when is_tuple(Tuple), tuple_size(Tuple) == record_info(size, Record), element(1,Tuple) == Record -> I = record_info({index,Field}, Record), setelement(I, Tuple, New_Value). and stuff them in a file which you -include whenever you have a real need for this feature. (Which is when oysters have a real need for shooting-sticks.) The thing that makes this approach almost attractive is that it very slightly extends an existing interface and then just removes a highly unusual restriction from it. (Which other Erlang functions must have literal arguments?) From freeakk@REDACTED Tue Feb 12 15:05:22 2013 From: freeakk@REDACTED (Michael Uvarov) Date: Tue, 12 Feb 2013 17:05:22 +0300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <3bf6ebe0fc4d01686aa2c2b32ef5c67d.squirrel@chasm.otago.ac.nz> References: <3bf6ebe0fc4d01686aa2c2b32ef5c67d.squirrel@chasm.otago.ac.nz> Message-ID: Here is a proof of concept of this idea. https://github.com/mad-cocktail/vodka It is not ready for general use. From bchesneau@REDACTED Tue Feb 12 16:28:17 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 12 Feb 2013 16:28:17 +0100 Subject: [erlang-questions] how to debug inet_res? Message-ID: Hi all, I am using a custom version of Erlang on iOS and for some reasons I ignore yet the DNS resolution is failing in some networks. Since you can't launch ports on iOS due to the sandboxing, Erlang is using the module inet_res to do the resolution. Which works well on some network but return {error, nxdomain} in an office at Singapore. Testing the same domain name on desktop with native or inet_res return the IP address correctly. For now I have written a small nif to do the gethostbyname call but I'm still curious to know why it fails on IOS. not sure which kind of test I should do. Any idea? - benoit From robert.virding@REDACTED Tue Feb 12 16:36:17 2013 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 12 Feb 2013 15:36:17 +0000 (GMT) Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <3bf6ebe0fc4d01686aa2c2b32ef5c67d.squirrel@chasm.otago.ac.nz> Message-ID: <1657317903.15432106.1360683377150.JavaMail.root@erlang-solutions.com> Won't this whole issue become a moot point once we get structs/frames/... ? The we can both have static, checked records and dynamic frames. I personally be glad to see both. Robert > From: ok@REDACTED > > I still say that Erlang needs dynamic record fields > the way oysters need shooting sticks, but here's > how it could be done. > > Step 1. > "To each module using records, a pseudo function > is added during compilation to obtain information > about records: > > record_info(fields, Record) -> [Field] > record_info(size, Record) -> Size > " > Extend this with > > record_info({index,Field}, Record) -> Index > > The term "pseudo function" (which ought to be > "pseudo-function" or "pseudofunction") refers > to the way the compiler reports an error if you > call record_info/2 with non-literal arguments. > > Step 2. > Instead of reporting an error, have the compiler > report a warning, and actually generate an > executable module-local definition of the > record_info/2 function; the existing behaviour > then just becomes the compiler special-casing a > known function, which is no more surprising than > compiling 1+1 as 2. > > Step 3. > There is no step 3. If you want there to be a > step 3, you can write > > -compile({inline, [field/3,field/3]}). > > field(Tuple, Record, Field) > when is_tuple(Tuple), > tuple_size(Tuple) == record_info(size, Record), > element(1,Tuple) == Record > -> I = record_info({index,Field}, Record), > element(I, Tuple). > > field(Tuple, Record, Field, New_Value) > when is_tuple(Tuple), > tuple_size(Tuple) == record_info(size, Record), > element(1,Tuple) == Record > -> I = record_info({index,Field}, Record), > setelement(I, Tuple, New_Value). > > and stuff them in a file which you -include whenever > you have a real need for this feature. (Which is > when oysters have a real need for shooting-sticks.) > > The thing that makes this approach almost attractive is > that it very slightly extends an existing interface and > then just removes a highly unusual restriction from it. > (Which other Erlang functions must have literal arguments?) > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From bgustavsson@REDACTED Tue Feb 12 17:17:14 2013 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Tue, 12 Feb 2013 17:17:14 +0100 Subject: [erlang-questions] edoc broken in current master / R16A! In-Reply-To: <5789034.tP9lXoejRZ@csr-pc40.zib.de> References: <4159396.DFkII5edO4@csr-pc40.zib.de> <3788796.M1GmZlaJoO@csr-pc40.zib.de> <5789034.tP9lXoejRZ@csr-pc40.zib.de> Message-ID: The following patch seems to fix the problem: git fetch git://github.com/bjorng/otp.git bjorn/remove-packages/OTP-10348 https://github.com/bjorng/otp/commit/e16a70c146b320e80332768bf471750753591987 Can you or someone familiar with scalaris verify that doc build now works? (E.g. that *all* documentation is build.) On Mon, Feb 4, 2013 at 5:34 PM, Nico Kruber wrote: > On Wednesday 30 Jan 2013 15:35:41 you wrote: > > On Wednesday 30 Jan 2013 15:34:26 you wrote: > > > we regularly check that Scalaris builds on master and since the checks > of > > > 11 Jan 2013 6am CET, creating the documentation with edoc fails. > > > We don't use packages, although we do put files into sub-folders - > anyway, > > > I guess a function_clause should not be returned to the user at any > point > > > (see below) > > > > > > svn co http://scalaris.googlecode.com/svn/trunk/ scalaris > > > cd scalaris > > > make doc > > > ------------------------------ > > > /home/buildbot/apps/erlang-dev/bin/erl -noshell -run edoc_run > application > > > "'scalaris'" \ > > > > > > '"."' '[{def, {vsn, "0.5.0+svn"}}, no_packages, {preprocess, true}, > > > > > > {includes, ["include", "contrib/yaws/include", > > > "contrib/log4erl/include"]}, > > > {sort_functions, false}, {pretty_printer, erl_pp}, {macros, > > > [{tid_not_builtin, true}, {with_export_type_support, true}, > > > {have_cthooks_support, true}, {have_callback_support, true}]}]' > > > edoc: edoc terminated abnormally: {function_clause, > > > > > > [{edoc_lib,find_sources_3, > > > > > > [["random_bias.erl",[...]|...], > > > > > > "./src/rrepair",rrepair|...], > > > > > > [{file,...},{...}]}, > > > > > > > {edoc_lib,find_sources_2,5,[{...}|...]}, > > > > {edoc_lib,'-find_sources_3/6-lc$^0/1-0-', > > > 5, > > > > > > [...]}, > > > > > > {edoc_lib,find_sources_2,5,...}, > > > {edoc_lib,find_sources_1,...}, > > > {edoc_lib,...}, > > > {...}|...]}. > > > > > > make: *** [.make-doc] Error 1 > > > ------------------------------ > > > > maybe this commit is related > > > https://github.com/erlang/otp/commit/beb660aa59a8d9cd7f65b8b91dc5dd9bc74015e > > 0 > > > > Nico > > I just tested with R16A and the problem is the same > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From raimo+erlang-questions@REDACTED Tue Feb 12 17:17:46 2013 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 12 Feb 2013 17:17:46 +0100 Subject: [erlang-questions] how to debug inet_res? In-Reply-To: References: Message-ID: <20130212161746.GA4812@erix.ericsson.se> On Tue, Feb 12, 2013 at 04:28:17PM +0100, Benoit Chesneau wrote: > Hi all, > > I am using a custom version of Erlang on iOS and for some reasons I > ignore yet the DNS resolution is failing in some networks. Since you > can't launch ports on iOS due to the sandboxing, Erlang is using the > module inet_res to do the resolution. Which works well on some network > but return {error, nxdomain} in an office at Singapore. > > Testing the same domain name on desktop with native or inet_res return > the IP address correctly. For now I have written a small nif to do the > gethostbyname call but I'm still curious to know why it fails on IOS. > not sure which kind of test I should do. Any idea? inet_res is the DNS resolver client written in Erlang. It does not call gethostbyname. The module inet_gethost_native uses an external port program to call gethostbyname. For inet_res, first try this: rr(inet_res). inet_res:resolve("google.com", in, a). If that does not work; I first suggest you call inet_db:get_rc() to get a summary of what settings are used (on both machines) and compare. Then you can look at the internal ETS table ets:i(inet_db) and compare them on both machines. To use a specific nameserver: inet_res:resolve("google.com", in, a, [{nameservers,[{{8,8,8,8},53}]}]). That looks up Googles address at Googles public nameserver 8.8.8.8. It does not work for me since I am behind a company firewall that does not allow outgoing DNS queries. I have to use a local nameserver. That local nameserver should be found in the ets table inet_db which should show in the inet_db:get_rc() output. If so e.g inet_res:gethostbyname("google.com") will produce a hostent record. Note. There is some magic in play indicated by an automatic default entry {resolv_conf,"/etc/resolv.conf"} in the inet_db:get_rc() output. That causes inet_res to update inet_db with content from /etc/resolv.conf before doing a DNS query. So do an inet_res call first; then if it does not work check the internal tables. Compare the tables before and after the first call. You can also add an argument 'verbose' to the argument 4 option list of inet_res:resolve/4 that will show which servers are tried and what they reply / timeout. > > - benoit > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From ali.sabil@REDACTED Tue Feb 12 17:53:56 2013 From: ali.sabil@REDACTED (Ali Sabil) Date: Tue, 12 Feb 2013 17:53:56 +0100 Subject: [erlang-questions] MongoDB EnsureIndex In-Reply-To: <511A1815.5030508@gmail.com> References: <510C6568056842459431E6A1F33D3685@srstrong.com> <511A1815.5030508@gmail.com> Message-ID: I would recommend the soundrop fork of the mongodb driver: https://github.com/soundrop/mongodb-erlang the details can be found here: https://github.com/mongodb/mongodb-erlang/pull/10 Best, Ali On Tue, Feb 12, 2013 at 11:23 AM, Dmitry Klionsky wrote: > +1 > > > On 02/12/2013 12:53 PM, Steve Strong wrote: > > Hi, > > We use https://github.com/mongodb/mongodb-erlang for erlang - seems to > work well enough, and certainly supports creating indexes etc. > > cheers, > > Steve > > -- > Steve Strong > Sent with Sparrow > > On Tuesday, 12 February 2013 at 10:50, Evren Bayraktar wrote: > > Hi everyone, > > Is there a any one who uses emongo or another library to use mongodb in > erlang ? I couldn't use ensureindex property with emongo. I don't know > whether it is possible or not :) > > I saw ensure_index function in this file ( > https://github.com/JacobVorreuter/emongo/blob/master/src/emongo.erl ) > then used it but it always gets ok. > > Actually, I want to use Mongo's Geospatial property with erlang. Thats > why I guess, I need to use ensureIndex. If there is any other method to use > geospatial search in Erlang I can use it. > > Best Regards ! > Evren > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions > > > > -- > Best regards, > Dmitry Klionsky > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bgustavsson@REDACTED Tue Feb 12 18:33:57 2013 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Tue, 12 Feb 2013 18:33:57 +0100 Subject: [erlang-questions] edoc broken in current master / R16A! In-Reply-To: References: <4159396.DFkII5edO4@csr-pc40.zib.de> <3788796.M1GmZlaJoO@csr-pc40.zib.de> <5789034.tP9lXoejRZ@csr-pc40.zib.de> Message-ID: I amended the commit so that the filename:join/2 is not called if there is no "package" (sub directory): https://github.com/bjorng/otp/commit/78b38983172b131ab6d995f95fc27d8bbf4efcb6 Same branch name: git fetch git://github.com/bjorng/otp.git bjorn/remove-packages/OTP-10348 On Tue, Feb 12, 2013 at 5:17 PM, Bj?rn Gustavsson wrote: > The following patch seems to fix the problem: > > git fetch git://github.com/bjorng/otp.git bjorn/remove-packages/OTP-10348 > > > https://github.com/bjorng/otp/commit/e16a70c146b320e80332768bf471750753591987 > > Can you or someone familiar with scalaris verify that doc > build now works? (E.g. that *all* documentation is build.) > > > On Mon, Feb 4, 2013 at 5:34 PM, Nico Kruber wrote: > >> On Wednesday 30 Jan 2013 15:35:41 you wrote: >> > On Wednesday 30 Jan 2013 15:34:26 you wrote: >> > > we regularly check that Scalaris builds on master and since the >> checks of >> > > 11 Jan 2013 6am CET, creating the documentation with edoc fails. >> > > We don't use packages, although we do put files into sub-folders - >> anyway, >> > > I guess a function_clause should not be returned to the user at any >> point >> > > (see below) >> > > >> > > svn co http://scalaris.googlecode.com/svn/trunk/ scalaris >> > > cd scalaris >> > > make doc >> > > ------------------------------ >> > > /home/buildbot/apps/erlang-dev/bin/erl -noshell -run edoc_run >> application >> > > "'scalaris'" \ >> > > >> > > '"."' '[{def, {vsn, "0.5.0+svn"}}, no_packages, {preprocess, >> true}, >> > > >> > > {includes, ["include", "contrib/yaws/include", >> > > "contrib/log4erl/include"]}, >> > > {sort_functions, false}, {pretty_printer, erl_pp}, {macros, >> > > [{tid_not_builtin, true}, {with_export_type_support, true}, >> > > {have_cthooks_support, true}, {have_callback_support, true}]}]' >> > > edoc: edoc terminated abnormally: {function_clause, >> > > >> > > [{edoc_lib,find_sources_3, >> > > >> > > [["random_bias.erl",[...]|...], >> > > >> > > "./src/rrepair",rrepair|...], >> > > >> > > [{file,...},{...}]}, >> > > >> > > >> {edoc_lib,find_sources_2,5,[{...}|...]}, >> > > >> {edoc_lib,'-find_sources_3/6-lc$^0/1-0-', >> > > 5, >> > > >> > > [...]}, >> > > >> > > {edoc_lib,find_sources_2,5,...}, >> > > {edoc_lib,find_sources_1,...}, >> > > {edoc_lib,...}, >> > > {...}|...]}. >> > > >> > > make: *** [.make-doc] Error 1 >> > > ------------------------------ >> > >> > maybe this commit is related >> > >> https://github.com/erlang/otp/commit/beb660aa59a8d9cd7f65b8b91dc5dd9bc74015e >> > 0 >> > >> > Nico >> >> I just tested with R16A and the problem is the same >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Tue Feb 12 18:36:12 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 12 Feb 2013 18:36:12 +0100 Subject: [erlang-questions] [ANN] Cowboy 0.8.0 Message-ID: <511A7D8C.5050107@ninenines.eu> Hello there! Cowboy 0.8 has been released. Cowboy is a small, fast and modular HTTP, REST and Websocket server. https://github.com/extend/cowboy/ The number of contributors who helped make this release considerably increased. Cowboy is available thanks to the code contributions from 50 users, double from the last release where 25 contributed. The number of users has also greatly increased. Cowboy is being used in ad bidding, set-top boxes, live TV events, content streaming services, and many more exciting areas. This new version has many highlights. You can take a look at the changelog for detailed information about the many changes. https://github.com/extend/cowboy/blob/master/CHANGELOG.md Cowboy scalability has been greatly improved in this version. This has been observed many times in production, including in the AdGear Tracker project (http://ferd.ca/rtb-where-erlang-blooms.html) where updated nodes were able to handle 2 times more requests compared to older nodes. This improvement cannot be observed in "hello world" types of benchmarks. An article will soon be published to explain the reasons for this. Cowboy now features a brand new user guide. It is still a work in progress, so please open a ticket on Github if something is missing or incorrect. http://ninenines.eu/docs/en/cowboy/HEAD/guide/introduction Remaining work before 1.0 include REST improvements and SPDY support. The rest of the API should now be very close to stable. I am looking for a good writer who would like to co-author a Cowboy book. The book will be accessible to people who don't know Erlang but will also contain everything there is to know about Cowboy, making it suitable for both beginners and experts. Contact me if you are interested. I now take donations in addition to commercial support options, to allow individual users to help the project stay alive and kicking. http://ninenines.eu/support Hope you enjoy it. As always, please send me as much feedback as possible to help me improve things even more, preferrably through Github tickets if it's related to code or documentation. Thanks for reading. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From jeremy@REDACTED Tue Feb 12 18:37:18 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Tue, 12 Feb 2013 09:37:18 -0800 Subject: [erlang-questions] [99s-extend] [ANN] Cowboy 0.8.0 In-Reply-To: <511A7D8C.5050107@ninenines.eu> References: <511A7D8C.5050107@ninenines.eu> Message-ID: Congrats! On Tue, Feb 12, 2013 at 9:36 AM, Lo?c Hoguin wrote: > Hello there! > > Cowboy 0.8 has been released. Cowboy is a small, fast and modular HTTP, > REST and Websocket server. > > https://github.com/extend/**cowboy/ > > The number of contributors who helped make this release considerably > increased. Cowboy is available thanks to the code contributions from 50 > users, double from the last release where 25 contributed. > > The number of users has also greatly increased. Cowboy is being used in ad > bidding, set-top boxes, live TV events, content streaming services, and > many more exciting areas. > > This new version has many highlights. You can take a look at the changelog > for detailed information about the many changes. > > https://github.com/extend/**cowboy/blob/master/CHANGELOG.**md > > Cowboy scalability has been greatly improved in this version. This has > been observed many times in production, including in the AdGear Tracker > project (http://ferd.ca/rtb-where-**erlang-blooms.html) > where updated nodes were able to handle 2 times more requests compared to > older nodes. This improvement cannot be observed in "hello world" types of > benchmarks. An article will soon be published to explain the reasons for > this. > > Cowboy now features a brand new user guide. It is still a work in > progress, so please open a ticket on Github if something is missing or > incorrect. > > http://ninenines.eu/docs/en/**cowboy/HEAD/guide/introduction > > Remaining work before 1.0 include REST improvements and SPDY support. The > rest of the API should now be very close to stable. > > I am looking for a good writer who would like to co-author a Cowboy book. > The book will be accessible to people who don't know Erlang but will also > contain everything there is to know about Cowboy, making it suitable for > both beginners and experts. Contact me if you are interested. > > I now take donations in addition to commercial support options, to allow > individual users to help the project stay alive and kicking. > > http://ninenines.eu/support > > Hope you enjoy it. As always, please send me as much feedback as possible > to help me improve things even more, preferrably through Github tickets if > it's related to code or documentation. > > Thanks for reading. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > ______________________________**_________________ > Extend mailing list > Extend@REDACTED > http://lists.ninenines.eu:81/**listinfo/extend > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Tue Feb 12 18:44:28 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 12 Feb 2013 20:44:28 +0300 Subject: [erlang-questions] [99s-extend] [ANN] Cowboy 0.8.0 In-Reply-To: References: <511A7D8C.5050107@ninenines.eu> Message-ID: Great, Loic. As I've told already, it would be great to listen to your experience about issues that you meet on high loads: smooth scaling, predictionable behaviour of server, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre@REDACTED Tue Feb 12 18:47:12 2013 From: andre@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Tue, 12 Feb 2013 18:47:12 +0100 Subject: [erlang-questions] [99s-extend] [ANN] Cowboy 0.8.0 In-Reply-To: References: <511A7D8C.5050107@ninenines.eu> Message-ID: That's all great news! Thanks for Cowboy, Loic! ;) On 12 February 2013 18:44, Max Lapshin wrote: > Great, Loic. > > As I've told already, it would be great to listen to your experience about > issues that you meet on high loads: smooth scaling, predictionable behaviour > of server, etc. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From bchesneau@REDACTED Tue Feb 12 18:49:57 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Tue, 12 Feb 2013 18:49:57 +0100 Subject: [erlang-questions] how to debug inet_res? In-Reply-To: <20130212161746.GA4812@erix.ericsson.se> References: <20130212161746.GA4812@erix.ericsson.se> Message-ID: On Tue, Feb 12, 2013 at 5:17 PM, Raimo Niskanen wrote: > On Tue, Feb 12, 2013 at 04:28:17PM +0100, Benoit Chesneau wrote: >> Hi all, >> >> I am using a custom version of Erlang on iOS and for some reasons I >> ignore yet the DNS resolution is failing in some networks. Since you >> can't launch ports on iOS due to the sandboxing, Erlang is using the >> module inet_res to do the resolution. Which works well on some network >> but return {error, nxdomain} in an office at Singapore. >> >> Testing the same domain name on desktop with native or inet_res return >> the IP address correctly. For now I have written a small nif to do the >> gethostbyname call but I'm still curious to know why it fails on IOS. >> not sure which kind of test I should do. Any idea? > > inet_res is the DNS resolver client written in Erlang. It does not call > gethostbyname. The module inet_gethost_native uses an external port > program to call gethostbyname. > > For inet_res, first try this: > rr(inet_res). > inet_res:resolve("google.com", in, a). > > If that does not work; I first suggest you call inet_db:get_rc() to get a > summary of what settings are used (on both machines) and compare. I did that this is why i'm speaking about inet_res. On ios: {file, dns} where the desktop has native. To compare both machine I did a test using `inet_res:gethostbyname/1` and `inet_hosts:gethostbyname/1` on the same network (wifi). in each case inet_hosts weren't working (desktop is on the mac), strangely it returns {error, nxdomain} on the desktop and {error, timeout} on ios. inet_res resolution is working on the desktop and not on iOS. > > Then you can look at the internal ETS table ets:i(inet_db) and compare them > on both machines. > > To use a specific nameserver: > inet_res:resolve("google.com", in, a, [{nameservers,[{{8,8,8,8},53}]}]). > > That looks up Googles address at Googles public nameserver 8.8.8.8. > It does not work for me since I am behind a company firewall that does not > allow outgoing DNS queries. I have to use a local nameserver. That local > nameserver should be found in the ets table inet_db which should show in > the inet_db:get_rc() output. If so e.g inet_res:gethostbyname("google.com") > will produce a hostent record. > > Note. There is some magic in play indicated by an automatic default entry > {resolv_conf,"/etc/resolv.conf"} in the inet_db:get_rc() output. That causes > inet_res to update inet_db with content from /etc/resolv.conf before doing > a DNS query. > > So do an inet_res call first; then if it does not work check the internal > tables. Compare the tables before and after the first call. Thanks will try that. > > You can also add an argument 'verbose' to the argument 4 option list of > inet_res:resolve/4 that will show which servers are tried and what they > reply / timeout. > I didn't know about the verbose option, will try it as well. I will update the thread with my results here. - beno?t From ok@REDACTED Tue Feb 12 21:40:41 2013 From: ok@REDACTED (ok@REDACTED) Date: Wed, 13 Feb 2013 09:40:41 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <1657317903.15432106.1360683377150.JavaMail.root@erlang-solutions.com> References: <1657317903.15432106.1360683377150.JavaMail.root@erlang-solutions.com> Message-ID: <51f1565cca44b76d2537c46e011b6305.squirrel@chasm.otago.ac.nz> > Won't this whole issue become a moot point once we get structs/frames/... Yes. I pointed that out in my first message in this thread. (I hope to get my desktop machine back today, 12 days after -- or so I am told -- a patch to Java screwed up NFS. Then I'll be able to use Mail again instead of this foul webmail system.) From comptekki@REDACTED Tue Feb 12 22:11:12 2013 From: comptekki@REDACTED (Wes James) Date: Tue, 12 Feb 2013 14:11:12 -0700 Subject: [erlang-questions] how to get ssl communication between nodes - windows erlsrv script Message-ID: I was able to get ssl communication between nodes on linux. Here is a windows .cmd script for installing erlsrv with ssl communication: @rem @rem sample erlang install service script @rem - add then start ecom erlang service @rem - if it already exists, stop then remove first before adding and starting @rem @set erl_service=my_service @erlsrv list %erl_service% | @findstr /i "%erl_service%" @echo. @if errorlevel 1 goto install @rem echo Stopping and removing erl service... @erlsrv remove "%erl_service%" :install @set module=-s my_module @set onfail=-onfail restart @set node=-name node_name @set root=-w "C:/path" @set srvc_name=-i my_service @set boot=-boot c:/path/start_ssl_ecom -proto_dist inet_tls -ssl_dist_opt server_certfile c:/path/ssl/cert.pem -ssl_dist_opt server_keyfile c:/path/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true @set cookie=-setcookie my_cookie @set args=-args "%boot% %cookie% %module%" @echo. @rem echo Adding erl service... @erlsrv add "%erl_service%" %node% %root% %onfail% %args% %srvc_name% @echo. @rem echo Starting erl service... @erlsrv start %erl_service% @echo. @pause Now if I can figure out why it's not picking up the .erlang.cookie from -w path, I'll be able to remove the -setcookie option. I've tried putting the .erlang.cookie in a lot of different places to see if it gets read - no dice yet. I have been able to get .erlang.cookie created in my windows profile c:/users/username when I run erl -name, but not yet with erlsrv. I'm excited to get ssl working. I have wanted it working for https://github.com/comptekki/esysman so I can send commands encrypted and not in the clear. Thanks to all that have contributed to my questions about this! wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Tue Feb 12 22:33:58 2013 From: ok@REDACTED (ok@REDACTED) Date: Wed, 13 Feb 2013 10:33:58 +1300 Subject: [erlang-questions] Dynamically access record fields In-Reply-To: <5119A32F.6090108@ninenines.eu> References: <2avy72dnt7rfiiy9nejj9ktl.1360569002562@email.android.com> <1c6adb6f76e1a669ecc99054a1f043e9.squirrel@chasm.otago.ac.nz> <5119A32F.6090108@ninenines.eu> Message-ID: <3fd0171489267ff3c1ab38ad0e1d0dcb.squirrel@chasm.otago.ac.nz> The central fact here is that we have NO evidence that anyone anywhere has ever been prevented from "getting things done" by the lack of this "feature". The last time I checked, over half a dozen different ways to do what the original poster wanted had been suggested. Here's one of the ones I suggested, spelled out in full. (No prizes awarded for guessing which of Reynolds' papers inspired it.) -define(GET(F,X), (hd(F))(X)). -define(UPD(F,X,V), (tl(F))(X, V)). -define(REF(R,F), [ fun (ZX) -> ZX#R.F end | fun (ZX, ZV) -> ZX#R{F = ZV} end]). ?REF(record_name, field_name) returns a pair of functions, a "get" function and an "update" function. ?GET(field_pair, record_value) applies the get function in a field pair to a record value. ?UPD(field_pair, record_value, new_value_for_field) applies the update function in a field pair to a record value and a new value for the field. So we have - a single value - that can be passed around, sent in a message, &c - that identifies a field - and can be used to dynamically select - or update that field. AND - by searching for occurrences of ?REF, you can discover which fields may be dynamically referenced. As for readability, just because something is assessed differently by different people, that does not mean that there are no comparisons can be made. There are indeed variations in taste about things like well_separated or baStudlyCaps or runtogethernames, about indentation and use of white space, about a bunch of things. But _within_ any single style, requiring someone to perform more mental work to discover information they need makes it less readable. At one extreme, you have COBOL, where interesting facts are lost in a vast forest of verbiage. At the other extreme, you have J, where you have to puzzle over every single character: avg=: +/ % # Haskell can be very close to the J end of the spectrum; Haskell written by masters can require Vetinari-level puzzle skills to decode. Erlang is somewhere in the middle, where the code isn't so bulky that the facts you need are needles in a haystack, and not so terse that you don't recognise them when you see them. From comptekki@REDACTED Wed Feb 13 00:27:01 2013 From: comptekki@REDACTED (Wes James) Date: Tue, 12 Feb 2013 16:27:01 -0700 Subject: [erlang-questions] how to get ssl communication between nodes - windows erlsrv script In-Reply-To: References: Message-ID: I got it to work by putting .erlang.cookie in c:\windows. In my testing I had put a file there previously, but it wouldn't be read if the file wasn't set to readonly. Thanks, wes On Tue, Feb 12, 2013 at 2:17 PM, Dave Cottlehuber wrote: > Last time I looked at the source workdir wasn't for changing where the > cookie got read from. > > But you should be able to load the cookie & start the network > programatically, something like this, starting up in non-distributed > mode: > > net_kernel:start([]). %% uses longnames by default > net_kernel:start([, shortnames]). %% if you prefer shortname. > erlang:set_cookie(node(), yumyumyum). %% yes it's an atom. > > This has the added bonus of being able to stash your cookie in an > encrypted form either on disk or in the beam. > > Let me know if this works out, I haven't tried it but use similar > things for monitoring. > > And I'm one of those closet Erlang Windows users too. > > A+ > Dave > > On 12 February 2013 22:11, Wes James wrote: > > I was able to get ssl communication between nodes on linux. Here is a > > windows .cmd script for installing erlsrv with ssl communication: > > > > @rem > > @rem sample erlang install service script > > @rem - add then start ecom erlang service > > @rem - if it already exists, stop then remove first before adding and > > starting > > @rem > > > > @set erl_service=my_service > > @erlsrv list %erl_service% | @findstr /i "%erl_service%" > > @echo. > > @if errorlevel 1 goto install > > @rem echo Stopping and removing erl service... > > @erlsrv remove "%erl_service%" > > > > :install > > @set module=-s my_module > > @set onfail=-onfail restart > > @set node=-name node_name > > @set root=-w "C:/path" > > @set srvc_name=-i my_service > > @set boot=-boot c:/path/start_ssl_ecom -proto_dist inet_tls -ssl_dist_opt > > server_certfile c:/path/ssl/cert.pem -ssl_dist_opt server_keyfile > > c:/path/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true > > client_secure_renegotiate true > > @set cookie=-setcookie my_cookie > > @set args=-args "%boot% %cookie% %module%" > > > > @echo. > > @rem echo Adding erl service... > > @erlsrv add "%erl_service%" %node% %root% %onfail% %args% %srvc_name% > > @echo. > > @rem echo Starting erl service... > > @erlsrv start %erl_service% > > @echo. > > @pause > > > > Now if I can figure out why it's not picking up the .erlang.cookie from > -w > > path, I'll be able to remove the -setcookie option. I've tried putting > the > > .erlang.cookie in a lot of different places to see if it gets read - no > dice > > yet. I have been able to get .erlang.cookie created in my windows > profile > > c:/users/username when I run erl -name, but not yet with erlsrv. > > > > I'm excited to get ssl working. I have wanted it working for > > https://github.com/comptekki/esysman so I can send commands encrypted > and > > not in the clear. > > > > Thanks to all that have contributed to my questions about this! > > > > wes > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby@REDACTED Wed Feb 13 01:42:49 2013 From: toby@REDACTED (Toby Thain) Date: Tue, 12 Feb 2013 19:42:49 -0500 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: Message-ID: <511AE189.4000604@telegraphics.com.au> On 11/02/13 10:09 AM, Andrew Pennebaker wrote: > Of course Erlang has a killer actors library as a core component of > itself. Sadly, Erlang is not always an option (e.g., customer requires a > JVM language). Which actor libraries in other languages come close to > Erlang's in terms of: > > * Syntax conciseness (spawn, !, receive) > * Polymorphism (receiving different kinds of messages) > * Fault-tolerance (GC per actor, actor crash doesn't crash the system) Akka and Scala specifically offer all the above. > * Live updates (different code versions up at the same time, hot code > updates) This can be done on the JVM, as I understand it (JRebel etc) but I can't speak from experience. Certainly it's not as designed-in and natural as Erlang. --Toby > > Does Scala and/or Akka offer anything like these features? What about > Ruby's Celluloid? What are the best actor libraries in Common Lisp and > Scheme? > > The Haskell community seems to prefer STM over actors. How does the > Erlang community feel about STM? > > -- > Cheers, > > Andrew Pennebaker > www.yellosoft.us > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From eriksoe@REDACTED Wed Feb 13 02:14:56 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Wed, 13 Feb 2013 02:14:56 +0100 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: <511AE189.4000604@telegraphics.com.au> References: <511AE189.4000604@telegraphics.com.au> Message-ID: 2013/2/13 Toby Thain > On 11/02/13 10:09 AM, Andrew Pennebaker wrote: > >> Of course Erlang has a killer actors library as a core component of >> itself. Sadly, Erlang is not always an option (e.g., customer requires a >> JVM language). > > Oh, but Erlang *is* a JVM language these days :-) Albeit not maturely enough for production use, last I checked. > Which actor libraries in other languages come close to >> Erlang's in terms of: >> >> * Syntax conciseness (spawn, !, receive) >> > Missing from the list: selective receive. :-) > * Polymorphism (receiving different kinds of messages) >> * Fault-tolerance (GC per actor, actor crash doesn't crash the system) >> > > Akka and Scala specifically offer all the above. Probably not the "GC per actor" part... What other systems *do* provide that, though? (I suspect Dart does, but it is lacking in other respects.) * Live updates (different code versions up at the same time, hot code > updates) > This can be done on the JVM, as I understand it (JRebel etc) but I can't > speak from experience. Certainly it's not as designed-in and natural as > Erlang. > Interesting; I didn't know about JRebel and LiveRebel. > --Toby > > > >> Does Scala and/or Akka offer anything like these features? What about >> Ruby's Celluloid? What are the best actor libraries in Common Lisp and >> Scheme? >> >> The Haskell community seems to prefer STM over actors. How does the >> Erlang community feel about STM? >> >> -- >> Cheers, >> >> Andrew Pennebaker >> www.yellosoft.us >> >> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Wed Feb 13 02:40:28 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Wed, 13 Feb 2013 02:40:28 +0100 Subject: [erlang-questions] GADTs / polymorphic data types? In-Reply-To: References: Message-ID: Erlang is a dynamically typed language, so you don't really declare data types. You can just use the data representation you choose. That is: you may declare types using "-type" and "-spec" - but the compiler and runtime as such don't care about such declarations; they are for the separate type checker "dialyzer". The form GADTs usually take is tuples-with-fixed-atoms-as-first-element (or just atoms for variants that don't contain data). Haskell GADTs like data Either a b = Left a | Right b data Maybe a = Some a | None would be described in Erlang as -type either(A,B) :: {left,A} | {right,B}. -type maybe(A) :: {some,A} | none. and used in function specs as -spec swap_left_and_right/1 :: (either(A,B)) -> either(B,A). -spec retract_left/1 :: (either(A,B)) -> maybe(A). See also, for instance, http://learnyousomeerlang.com/types-or-lack-thereof and http://learnyousomeerlang.com/dialyzer 2013/2/11 Andrew Pennebaker > Does Erlang have anything like Haskell GADTs? How are structured data > types declared in Erlang? > > -- > Cheers, > > Andrew Pennebaker > www.yellosoft.us > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jinni.park@REDACTED Wed Feb 13 05:27:55 2013 From: jinni.park@REDACTED (Park, Sungjin) Date: Wed, 13 Feb 2013 13:27:55 +0900 Subject: [erlang-questions] [99s-extend] [ANN] Cowboy 0.8.0 In-Reply-To: References: <511A7D8C.5050107@ninenines.eu> Message-ID: Congratulations Loic. Community bless you. On Wed, Feb 13, 2013 at 2:47 AM, Andr? Graf wrote: > That's all great news! > Thanks for Cowboy, Loic! ;) > > On 12 February 2013 18:44, Max Lapshin wrote: > > Great, Loic. > > > > As I've told already, it would be great to listen to your experience > about > > issues that you meet on high loads: smooth scaling, predictionable > behaviour > > of server, etc. > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Park, Sungjin ------------------------------------------------------------------------------------------------------------------- Peculiar travel suggestions are dancing lessons from god. -- The Books of Bokonon ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From chandu.gokul138@REDACTED Wed Feb 13 08:03:34 2013 From: chandu.gokul138@REDACTED (Gokul Evuri) Date: Wed, 13 Feb 2013 08:03:34 +0100 Subject: [erlang-questions] Messing with heart. Port and NIF, which one is better? Message-ID: Hello, I am beginner-level erlang user and i got two questions here, is there a way to save my processes if the VM crashes? And the second question, Is there any good argument to use NIF instead of creating a connected process for a port. P.S: I know if there is a bug in NIF the VM might crash unlike a port-process, which can be restarted when it crashes. Thank You, -- *Gokul Reddy Evuri,* -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Wed Feb 13 08:41:56 2013 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 13 Feb 2013 08:41:56 +0100 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: <511AE189.4000604@telegraphics.com.au> References: <511AE189.4000604@telegraphics.com.au> Message-ID: On Wed, Feb 13, 2013 at 1:42 AM, Toby Thain wrote: > On 11/02/13 10:09 AM, Andrew Pennebaker wrote: > >> Of course Erlang has a killer actors library as a core component of >> itself. Sadly, Erlang is not always an option (e.g., customer requires a >> JVM language). Which actor libraries in other languages come close to >> Erlang's in terms of: >> >> * Syntax conciseness (spawn, !, receive) >> * Polymorphism (receiving different kinds of messages) >> * Fault-tolerance (GC per actor, actor crash doesn't crash the system) >> > > Akka and Scala specifically offer all the above. Plus the actor model does not address problems of fault propagation. The statement "X has Erlang like semantics" usually means "X has Erlang like semantics except for error handling" - handling errors out-of-band is a big win. Isolation is the key property that must be guaranteed for actors. If <> can crash <> code then all bets are off. /Joe > > > * Live updates (different code versions up at the same time, hot code >> updates) >> > > This can be done on the JVM, as I understand it (JRebel etc) but I can't > speak from experience. Certainly it's not as designed-in and natural as > Erlang. > > --Toby > > > >> Does Scala and/or Akka offer anything like these features? What about >> Ruby's Celluloid? What are the best actor libraries in Common Lisp and >> Scheme? >> >> The Haskell community seems to prefer STM over actors. How does the >> Erlang community feel about STM? >> >> -- >> Cheers, >> >> Andrew Pennebaker >> www.yellosoft.us >> >> >> >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yilmazhuseyin@REDACTED Wed Feb 13 09:05:20 2013 From: yilmazhuseyin@REDACTED (Huseyin Yilmaz) Date: Wed, 13 Feb 2013 10:05:20 +0200 Subject: [erlang-questions] testing strategy question. Message-ID: Hi, I am relatively new to erlang and I am trying to write a project to grasp the language and otp basics. In my application, I am using e-units for testing. here is some sample code get_room_get_code_test() -> chat:stop(), Code = 1, ?assertEqual(ok, chat:start()), {ok, Room} = start_link(Code), ?assertEqual({ok, 1}, get_code(Room)), ?assertEqual({ok, Room}, get_room(1)), ?assertEqual(ok, chat:stop()). all of my codes have this structure. 1) Stop chat server, if it is running (if previous test failed before stopping running server) 2) Start chat server. 3) do tests. 4) stop server. Writing this every time is look really bad. (even if would make start_test_env, stop_test_env methods, I would still need to write a structure to make sure test environment is properly destroyed for every failed test.) So I decided to write it with common test. init_per_suite(Config) -> ok = chat:start(), error_logger:info_msg("Chat application stopped~n"), Config. end_per_suite(_Config) -> ok = chat:stop(), error_logger:info_msg("Chat application stopped~n"), ok. get_room_get_code_test_case() -> []. get_room_get_code_test_case(_Config) -> Code = 1, {ok, Room} = c_room:start_link(Code), {ok, Code} = c_room:get_code(Room), {ok, Room} = c_room:get_room(1). With common tests my tests seems a lot better. But if I have an error to console. instead of it goes to common test report. I was wondering what people are using in those situations. What I want to do is to run the test cases every time I save my files to see if anything is broken. So I do not really want to run the test cases and go to my browser to see the results every time I save a file. Another thing is ?assert macros. Those macros give a nice output in eunits but common_tests does not have them. So it seems to me like we do not really interested in common_test output. we only want to see if it tests. So this means they are only for to run before going production? Any comment on this would be appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olav@REDACTED Wed Feb 13 09:39:37 2013 From: olav@REDACTED (Olav Frengstad) Date: Wed, 13 Feb 2013 09:39:37 +0100 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: Message-ID: I use a small wrapper over cowboy_http_hander that let's me bind a method to a function call. method_handlers(Req, State) -> [{<<"GET">>, read}, {<<"POST">>, update}, Req, State]. read(Req, State) -> {Req, State}. update(Req, State) -> {Req, State}. The easiest making handle/2 call handle_method(Method, Req, State) or similar. If you want something pluggable you can look at https://github.com/lafka/tavern Olav 2013/2/11 Max Lapshin : > cowboy_rest is incredibly hard to use because there is even no full list of > callbacks. Webmachine diagram is good, but it lacks name of callbacks. > > I understand that it is hard to document it, but I can't help you with it > because I don't understand this code in depth. > > I want to put Amazon S3 API in routes and it looks like rest is a bad idea. > I've ended with catch-all route and manual routing in init/3 handler > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Med Vennlig Hilsen Olav Frengstad Systemutvikler // FWT +47 920 42 090 From max.lapshin@REDACTED Wed Feb 13 09:58:57 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 13 Feb 2013 11:58:57 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: Message-ID: It is great, but I think that it would be a good idea to add some more data to Constraints. Loic, if you think that it is good idea, I can add patch so that there will be 'method' in Constraints for better routing. M? -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Feb 13 10:00:04 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 13 Feb 2013 10:00:04 +0100 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: Message-ID: <511B5614.2000602@ninenines.eu> On 02/13/2013 09:58 AM, Max Lapshin wrote: > It is great, but I think that it would be a good idea to add some more > data to Constraints. > > Loic, if you think that it is good idea, I can add patch so that there > will be 'method' in Constraints for better routing. M? Constraints act on bindings. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From max.lapshin@REDACTED Wed Feb 13 10:05:39 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 13 Feb 2013 12:05:39 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: <511B5614.2000602@ninenines.eu> References: <511B5614.2000602@ninenines.eu> Message-ID: I understand it. But look at rails routing: you can add constraints not only on bindings, but also on methods. It is very convenient, because it allows CRUD routing from routes and it makes code smaller due to removing protection of GET method. If I add route: "/items/:item/destroy", item_handler, [destroy] than I need to check if this method is POST, because it is a very bad idea to make any destructive action on GET. If I add: "items/:item/destroy", [{method,<<"POST">>}], item_handler, [destroy] I don't need to write useless code. Why are you against it? Let's use not 'method', but <<"method">> or any other name that will not clash with bindings. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Wed Feb 13 10:08:58 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Wed, 13 Feb 2013 10:08:58 +0100 Subject: [erlang-questions] testing strategy question. In-Reply-To: References: Message-ID: Eunit has a similar thing. They call it "fixtures": http://www.erlang.org/doc/apps/eunit/chapter.html#Fixtures Den 13/02/2013 09.05 skrev "Huseyin Yilmaz" : > Hi, I am relatively new to erlang and I am trying to write a project to > grasp the language and otp basics. In my application, I am using e-units > for testing. here is some sample code > > > get_room_get_code_test() -> > chat:stop(), > Code = 1, > ?assertEqual(ok, chat:start()), > {ok, Room} = start_link(Code), > ?assertEqual({ok, 1}, get_code(Room)), > ?assertEqual({ok, Room}, get_room(1)), > ?assertEqual(ok, chat:stop()). > > > all of my codes have this structure. > > 1) Stop chat server, if it is running (if previous test failed before > stopping running server) > 2) Start chat server. > 3) do tests. > 4) stop server. > > > Writing this every time is look really bad. (even if would make > start_test_env, stop_test_env methods, I would still need to write a > structure to make sure test environment is properly destroyed for every > failed test.) > > So I decided to write it with common test. > > init_per_suite(Config) -> > ok = chat:start(), > error_logger:info_msg("Chat application stopped~n"), > Config. > > end_per_suite(_Config) -> > ok = chat:stop(), > error_logger:info_msg("Chat application stopped~n"), > ok. > > get_room_get_code_test_case() -> > []. > > get_room_get_code_test_case(_Config) -> > Code = 1, > {ok, Room} = c_room:start_link(Code), > {ok, Code} = c_room:get_code(Room), > {ok, Room} = c_room:get_room(1). > > > > With common tests my tests seems a lot better. But if I have an error to > console. instead of it goes to common test report. > > I was wondering what people are using in those situations. > > What I want to do is to run the test cases every time I save my files to > see if anything is broken. So I do not really want to run the test cases > and go to my browser to see the results every time I save a file. > > Another thing is ?assert macros. Those macros give a nice output in > eunits but common_tests does not have them. So it seems to me like we do > not really interested in common_test output. we only want to see if it > tests. So this means they are only for to run before going production? > > Any comment on this would be appreciated. > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ola.Backstrom@REDACTED Wed Feb 13 10:23:18 2013 From: Ola.Backstrom@REDACTED (=?iso-8859-1?Q?Ola__B=E4ckstr=F6m?=) Date: Wed, 13 Feb 2013 09:23:18 +0000 Subject: [erlang-questions] testing strategy question. In-Reply-To: References: Message-ID: It is possible to combine and use the assert macros from eunit inside Commont test code. Just bring all in: -module(some_SUITE). -compile(export_all). -include_lib("common_test/include/ct.hrl"). -include_lib("eunit/include/eunit.hrl"). (+ all standard CT suite testcase setup stuff) And then in a test function should_work(_Config) -> ?assertEqual(ok, some:work()). (you might get a warning like .../ct/some_SUITE.erl:0: missing specification for function test/0, but that is just eunit complaining, but Common Test will execute it nicely) /Ola From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Huseyin Yilmaz Sent: den 13 februari 2013 09:05 To: erlang-questions@REDACTED Subject: [erlang-questions] testing strategy question. Hi, I am relatively new to erlang and I am trying to write a project to grasp the language and otp basics. In my application, I am using e-units for testing. here is some sample code get_room_get_code_test() -> chat:stop(), Code = 1, ?assertEqual(ok, chat:start()), {ok, Room} = start_link(Code), ?assertEqual({ok, 1}, get_code(Room)), ?assertEqual({ok, Room}, get_room(1)), ?assertEqual(ok, chat:stop()). all of my codes have this structure. 1) Stop chat server, if it is running (if previous test failed before stopping running server) 2) Start chat server. 3) do tests. 4) stop server. Writing this every time is look really bad. (even if would make start_test_env, stop_test_env methods, I would still need to write a structure to make sure test environment is properly destroyed for every failed test.) So I decided to write it with common test. init_per_suite(Config) -> ok = chat:start(), error_logger:info_msg("Chat application stopped~n"), Config. end_per_suite(_Config) -> ok = chat:stop(), error_logger:info_msg("Chat application stopped~n"), ok. get_room_get_code_test_case() -> []. get_room_get_code_test_case(_Config) -> Code = 1, {ok, Room} = c_room:start_link(Code), {ok, Code} = c_room:get_code(Room), {ok, Room} = c_room:get_room(1). With common tests my tests seems a lot better. But if I have an error to console. instead of it goes to common test report. I was wondering what people are using in those situations. What I want to do is to run the test cases every time I save my files to see if anything is broken. So I do not really want to run the test cases and go to my browser to see the results every time I save a file. Another thing is ?assert macros. Those macros give a nice output in eunits but common_tests does not have them. So it seems to me like we do not really interested in common_test output. we only want to see if it tests. So this means they are only for to run before going production? Any comment on this would be appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Feb 13 10:27:48 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 13 Feb 2013 10:27:48 +0100 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> Message-ID: <511B5C94.1020606@ninenines.eu> Rails is a framework, Cowboy is trying hard not to be. HTTP has a long section about mapping URIs to resources on which you perform actions. Actions are identified by methods, but aren't used to find the resource to be used. Cowboy simply follows HTTP. Adding ways to use methods to map to handlers or functions directly into Cowboy would just break this concept of resources that Cowboy is trying hard to protect. It shouldn't be difficult to write a middleware that allows you to do this efficiently if that's what you want, or fork the current router to add the functionality, but that's something that won't land in Cowboy master. On 02/13/2013 10:05 AM, Max Lapshin wrote: > I understand it. > > But look at rails routing: you can add constraints not only on bindings, > but also on methods. It is very convenient, because it allows CRUD > routing from routes and it makes code smaller due to removing protection > of GET method. > > If I add route: > > "/items/:item/destroy", item_handler, [destroy] > > than I need to check if this method is POST, because it is a very bad > idea to make any destructive action on GET. > If I add: > > "items/:item/destroy", [{method,<<"POST">>}], item_handler, [destroy] > > I don't need to write useless code. > > Why are you against it? > Let's use not 'method', but <<"method">> or any other name that will > not clash with bindings. > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From max.lapshin@REDACTED Wed Feb 13 11:18:18 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 13 Feb 2013 13:18:18 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: <511B5C94.1020606@ninenines.eu> References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> Message-ID: Ok, I hear you, but I don't understand you. It is a very strange decision for me, because I usually prefer to select design choices that make programming convenient for programmers. Middleware is a very bad design choice, because it requires duplicating code between configuration of router and middleware and it will end as throwing away cowboy_router at all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Feb 13 11:29:10 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 13 Feb 2013 11:29:10 +0100 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> Message-ID: <511B6AF6.3030002@ninenines.eu> On 02/13/2013 11:18 AM, Max Lapshin wrote: > Ok, I hear you, but I don't understand you. > > It is a very strange decision for me, because I usually prefer to select > design choices that make programming convenient for programmers. That's an odd statement, I'm not sure why you would think that's not what I'm doing. What I'm not doing, however, is make easy things easier and hard things harder, like Rails does. > Middleware is a very bad design choice, because it requires duplicating > code between configuration of router and middleware and it will end as > throwing away cowboy_router at all. Router is middleware configuration so that's probably okay. What I don't really understand is why you need this feature at all considering you can already have a clause per method easily (which is fairly similar to a function per method). All you have to do is: handle(Req, State) -> handle_method(cowboy_req:get(method, Req), Req, State). So I'm not sure what you are expecting. If this doesn't work out then there's plenty other solutions, but I don't know which one is best for your case. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From max.lapshin@REDACTED Wed Feb 13 11:29:54 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 13 Feb 2013 13:29:54 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> Message-ID: I can give you an example: URL /users/15 GET /users/15 should be routed to function that shows user DELETE /users/15 should be route to function that destroys user Currently you are forcing user to spread routing among different places: "/users/:user_id", user_handler, [] and there, in init I need to check if this method is GET or DELETE or anything else but with method constraints, it will work as following: "/users/:user_id", [{method,<<"GET">>}], user_handler, [show] "/users/:user_id", [{method,<<"DELETE">>}], user_handler, [destroy] I don't see any HTTP pure theory violation in it. It looks rather convenient and more clear than checking which method was passed by user. -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Feb 13 11:34:31 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 13 Feb 2013 13:34:31 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: <511B6AF6.3030002@ninenines.eu> References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> Message-ID: handle(Req, State) -> handle_method(cowboy_req:get(method, Req), Req, State). will require handle_method(<<"GET">>, Req, State) handle_method(<<"DELETE">>, Req, State) handle_method(Other, Req, State) here we copy from module to module default reply on "method not supported" or 404, whatever we choose. with method constraints code can look so: init(_, Req, [Action]) -> {ok, Req, Action}. handle(Req, show) -> .. handle(Req, destroy) -> .. handle(Req, list) -> .. . And no "default handler" because it is not required. Only good requests can pass to this handler. -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Feb 13 11:39:00 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 13 Feb 2013 11:39:00 +0100 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> Message-ID: <511B6D44.900@ninenines.eu> On 02/13/2013 11:34 AM, Max Lapshin wrote: > handle(Req, State) -> > handle_method(cowboy_req:get(method, Req), Req, State). will require > > > handle_method(<<"GET">>, Req, State) > handle_method(<<"DELETE">>, Req, State) > handle_method(Other, Req, State) > here we copy from module to module default reply on "method not > supported" or 404, whatever we choose. > > > with method constraints code can look so: > > > init(_, Req, [Action]) -> > {ok, Req, Action}. > > > handle(Req, show) -> > .. > > handle(Req, destroy) -> > .. > > handle(Req, list) -> > .. > . > > And no "default handler" because it is not required. Only good requests > can pass to this handler. Then in your init function do something like your_lib:route_method(Req) which returns {ok, Req, Action} if it's good or {shutdown, ...} if not? It'll do exactly what you just said. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From max.lapshin@REDACTED Wed Feb 13 11:42:34 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 13 Feb 2013 13:42:34 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: <511B6D44.900@ninenines.eu> References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> <511B6D44.900@ninenines.eu> Message-ID: Loic, my idea about routing is that router is a piece of software that translates input http request into presentation, convenient for handler and selects proper handler for this with proper arguments. I don't understand your logic: you allow to run regexp to validate if constraint is a digit: "/users/:user_id", [{user_id,fun() -> .. end}], but you are strongly against validating method in constraints. You tell me to spread logic between your router and my router. You tell me that I need to use two routers that will somehow interfere with each other. What is the idea? I really don't understand. On Wed, Feb 13, 2013 at 2:39 PM, Lo?c Hoguin wrote: > On 02/13/2013 11:34 AM, Max Lapshin wrote: > >> handle(Req, State) -> >> handle_method(cowboy_req:get(**method, Req), Req, State). will >> require >> >> >> handle_method(<<"GET">>, Req, State) >> handle_method(<<"DELETE">>, Req, State) >> handle_method(Other, Req, State) >> here we copy from module to module default reply on "method not >> supported" or 404, whatever we choose. >> >> >> with method constraints code can look so: >> >> >> init(_, Req, [Action]) -> >> {ok, Req, Action}. >> >> >> handle(Req, show) -> >> .. >> >> handle(Req, destroy) -> >> .. >> >> handle(Req, list) -> >> .. >> . >> >> And no "default handler" because it is not required. Only good requests >> can pass to this handler. >> > > Then in your init function do something like your_lib:route_method(Req) > which returns {ok, Req, Action} if it's good or {shutdown, ...} if not? > It'll do exactly what you just said. > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Feb 13 12:02:05 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 13 Feb 2013 12:02:05 +0100 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> <511B6D44.900@ninenines.eu> Message-ID: <511B72AD.4020102@ninenines.eu> The HTTP RFC says this on page 6: It builds on the discipline of reference provided by the Uniform Resource Identifier (URI) [3], as a location (URL) [4] or name (URN) [20], for indicating the resource to which a method is to be applied. It then defines resources as follow on page 9: A network data object or service that can be identified by a URI, as defined in section 3.2. Resources may be available in multiple representations (e.g. multiple languages, data formats, size, and resolutions) or vary in other ways. It then describes how to locate resources in many of the following sections, starting with section 3.2 page 18. Cowboy implements this. It maps URIs to resources. It makes absolutely no assumptions as to how your resources operate. Anything that is part of the URI can and will be used to locate resources. But if it's not part of the URI, then it's not something used to find resources as defined by the HTTP standard, which makes it out of the scope of the project. It still provides you with many ways to plug yourself into Cowboy to achieve what you want, which include: * Middleware * Protocol upgrade (what REST handlers are doing) * init/3 based method dispatching * handle/2 based method dispatching And I'm sure there's more. On 02/13/2013 11:42 AM, Max Lapshin wrote: > Loic, my idea about routing is that router is a piece of software that > translates input http request into presentation, convenient for handler > and selects proper handler for this with proper arguments. > > I don't understand your logic: you allow to run regexp to validate if > constraint is a digit: "/users/:user_id", [{user_id,fun() -> .. end}], > but you are strongly against validating method in constraints. > > You tell me to spread logic between your router and my router. You tell > me that I need to use two routers that will somehow interfere with each > other. What is the idea? I really don't understand. > > > On Wed, Feb 13, 2013 at 2:39 PM, Lo?c Hoguin > wrote: > > On 02/13/2013 11:34 AM, Max Lapshin wrote: > > handle(Req, State) -> > handle_method(cowboy_req:get(__method, Req), Req, State). > will require > > > handle_method(<<"GET">>, Req, State) > handle_method(<<"DELETE">>, Req, State) > handle_method(Other, Req, State) > here we copy from module to module default reply on "method not > supported" or 404, whatever we choose. > > > with method constraints code can look so: > > > init(_, Req, [Action]) -> > {ok, Req, Action}. > > > handle(Req, show) -> > .. > > handle(Req, destroy) -> > .. > > handle(Req, list) -> > .. > . > > And no "default handler" because it is not required. Only good > requests > can pass to this handler. > > > Then in your init function do something like > your_lib:route_method(Req) which returns {ok, Req, Action} if it's > good or {shutdown, ...} if not? It'll do exactly what you just said. > > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From gumm@REDACTED Wed Feb 13 14:46:07 2013 From: gumm@REDACTED (Jesse Gumm) Date: Wed, 13 Feb 2013 07:46:07 -0600 Subject: [erlang-questions] [ANN] Cowboy 0.8.0 In-Reply-To: <511A7D8C.5050107@ninenines.eu> References: <511A7D8C.5050107@ninenines.eu> Message-ID: Great news! Congrats! On Feb 12, 2013 11:36 AM, "Lo?c Hoguin" wrote: > Hello there! > > Cowboy 0.8 has been released. Cowboy is a small, fast and modular HTTP, > REST and Websocket server. > > https://github.com/extend/**cowboy/ > > The number of contributors who helped make this release considerably > increased. Cowboy is available thanks to the code contributions from 50 > users, double from the last release where 25 contributed. > > The number of users has also greatly increased. Cowboy is being used in ad > bidding, set-top boxes, live TV events, content streaming services, and > many more exciting areas. > > This new version has many highlights. You can take a look at the changelog > for detailed information about the many changes. > > https://github.com/extend/**cowboy/blob/master/CHANGELOG.**md > > Cowboy scalability has been greatly improved in this version. This has > been observed many times in production, including in the AdGear Tracker > project (http://ferd.ca/rtb-where-**erlang-blooms.html) > where updated nodes were able to handle 2 times more requests compared to > older nodes. This improvement cannot be observed in "hello world" types of > benchmarks. An article will soon be published to explain the reasons for > this. > > Cowboy now features a brand new user guide. It is still a work in > progress, so please open a ticket on Github if something is missing or > incorrect. > > http://ninenines.eu/docs/en/**cowboy/HEAD/guide/introduction > > Remaining work before 1.0 include REST improvements and SPDY support. The > rest of the API should now be very close to stable. > > I am looking for a good writer who would like to co-author a Cowboy book. > The book will be accessible to people who don't know Erlang but will also > contain everything there is to know about Cowboy, making it suitable for > both beginners and experts. Contact me if you are interested. > > I now take donations in addition to commercial support options, to allow > individual users to help the project stay alive and kicking. > > http://ninenines.eu/support > > Hope you enjoy it. As always, please send me as much feedback as possible > to help me improve things even more, preferrably through Github tickets if > it's related to code or documentation. > > Thanks for reading. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico.kruber@REDACTED Wed Feb 13 15:34:33 2013 From: nico.kruber@REDACTED (Nico Kruber) Date: Wed, 13 Feb 2013 15:34:33 +0100 Subject: [erlang-questions] edoc broken in current master / R16A! In-Reply-To: References: <4159396.DFkII5edO4@csr-pc40.zib.de> Message-ID: <19753141.cxkctjQEvl@csr-pc40.zib.de> I just tested it with your branch and it works and builds all docs successfully (including the modules in sub-directories) (I found another issue while checking but will report it separately) Thank you Nico On Tuesday 12 Feb 2013 18:33:57 Bj?rn Gustavsson wrote: > I amended the commit so that the filename:join/2 is not called > if there is no "package" (sub directory): > > https://github.com/bjorng/otp/commit/78b38983172b131ab6d995f95fc27d8bbf4efcb > 6 > > Same branch name: > > git fetch git://github.com/bjorng/otp.git bjorn/remove-packages/OTP-10348 > > On Tue, Feb 12, 2013 at 5:17 PM, Bj?rn Gustavsson wrote: > > The following patch seems to fix the problem: > > > > git fetch git://github.com/bjorng/otp.git bjorn/remove-packages/OTP-10348 > > > > > > https://github.com/bjorng/otp/commit/e16a70c146b320e80332768bf471750753591 > > 987 > > > > Can you or someone familiar with scalaris verify that doc > > build now works? (E.g. that *all* documentation is build.) > > > > On Mon, Feb 4, 2013 at 5:34 PM, Nico Kruber wrote: > >> On Wednesday 30 Jan 2013 15:35:41 you wrote: > >> > On Wednesday 30 Jan 2013 15:34:26 you wrote: > >> > > we regularly check that Scalaris builds on master and since the > >> > >> checks of > >> > >> > > 11 Jan 2013 6am CET, creating the documentation with edoc fails. > >> > > We don't use packages, although we do put files into sub-folders - > >> > >> anyway, > >> > >> > > I guess a function_clause should not be returned to the user at any > >> > >> point > >> > >> > > (see below) > >> > > > >> > > svn co http://scalaris.googlecode.com/svn/trunk/ scalaris > >> > > cd scalaris > >> > > make doc > >> > > ------------------------------ > >> > > /home/buildbot/apps/erlang-dev/bin/erl -noshell -run edoc_run > >> > >> application > >> > >> > > "'scalaris'" \ > >> > > > >> > > '"."' '[{def, {vsn, "0.5.0+svn"}}, no_packages, {preprocess, > >> > >> true}, > >> > >> > > {includes, ["include", "contrib/yaws/include", > >> > > "contrib/log4erl/include"]}, > >> > > {sort_functions, false}, {pretty_printer, erl_pp}, {macros, > >> > > [{tid_not_builtin, true}, {with_export_type_support, true}, > >> > > {have_cthooks_support, true}, {have_callback_support, true}]}]' > >> > > edoc: edoc terminated abnormally: {function_clause, > >> > > > >> > > [{edoc_lib,find_sources_3, > >> > > > >> > > [["random_bias.erl",[...]|...], > >> > > > >> > > "./src/rrepair",rrepair|...], > >> > > > >> > > [{file,...},{...}]}, > >> > >> {edoc_lib,find_sources_2,5,[{...}|...]}, > >> > >> {edoc_lib,'-find_sources_3/6-lc$^0/1-0-', > >> > >> > > 5, > >> > > > >> > > [...]}, > >> > > > >> > > {edoc_lib,find_sources_2,5,...}, > >> > > {edoc_lib,find_sources_1,...}, > >> > > {edoc_lib,...}, > >> > > {...}|...]}. > >> > > > >> > > make: *** [.make-doc] Error 1 > >> > > ------------------------------ > >> > > >> > maybe this commit is related > >> > >> https://github.com/erlang/otp/commit/beb660aa59a8d9cd7f65b8b91dc5dd9bc740 > >> 15e>> > >> > 0 > >> > > >> > Nico > >> > >> I just tested with R16A and the problem is the same > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > > > > -- > > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From dronnikov@REDACTED Wed Feb 13 15:44:05 2013 From: dronnikov@REDACTED (Vladimir Dronnikov) Date: Wed, 13 Feb 2013 18:44:05 +0400 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: <511B72AD.4020102@ninenines.eu> References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> <511B6D44.900@ninenines.eu> <511B72AD.4020102@ninenines.eu> Message-ID: Max, one could insert a middleware before cowboy_router which mangle URI to be Method/URI and then in the router rules treat Method chunk as binding -- hence, method constraints. --Vladimir On Wed, Feb 13, 2013 at 3:02 PM, Lo?c Hoguin wrote: > The HTTP RFC says this on page 6: > > It builds on the discipline of reference > provided by the Uniform Resource Identifier (URI) [3], as a location > (URL) [4] or name (URN) [20], for indicating the resource to which a > method is to be applied. > > It then defines resources as follow on page 9: > > A network data object or service that can be identified by a URI, > as defined in section 3.2. Resources may be available in multiple > representations (e.g. multiple languages, data formats, size, and > resolutions) or vary in other ways. > > It then describes how to locate resources in many of the following > sections, starting with section 3.2 page 18. > > Cowboy implements this. It maps URIs to resources. It makes absolutely no > assumptions as to how your resources operate. Anything that is part of the > URI can and will be used to locate resources. But if it's not part of the > URI, then it's not something used to find resources as defined by the HTTP > standard, which makes it out of the scope of the project. > > It still provides you with many ways to plug yourself into Cowboy to > achieve what you want, which include: > > * Middleware > * Protocol upgrade (what REST handlers are doing) > * init/3 based method dispatching > * handle/2 based method dispatching > > And I'm sure there's more. > > > On 02/13/2013 11:42 AM, Max Lapshin wrote: > >> Loic, my idea about routing is that router is a piece of software that >> translates input http request into presentation, convenient for handler >> and selects proper handler for this with proper arguments. >> >> I don't understand your logic: you allow to run regexp to validate if >> constraint is a digit: "/users/:user_id", [{user_id,fun() -> .. end}], >> but you are strongly against validating method in constraints. >> >> You tell me to spread logic between your router and my router. You tell >> me that I need to use two routers that will somehow interfere with each >> other. What is the idea? I really don't understand. >> >> >> On Wed, Feb 13, 2013 at 2:39 PM, Lo?c Hoguin > > wrote: >> >> On 02/13/2013 11:34 AM, Max Lapshin wrote: >> >> handle(Req, State) -> >> handle_method(cowboy_req:get(_**_method, Req), Req, State). >> >> will require >> >> >> handle_method(<<"GET">>, Req, State) >> handle_method(<<"DELETE">>, Req, State) >> handle_method(Other, Req, State) >> here we copy from module to module default reply on "method >> not >> supported" or 404, whatever we choose. >> >> >> with method constraints code can look so: >> >> >> init(_, Req, [Action]) -> >> {ok, Req, Action}. >> >> >> handle(Req, show) -> >> .. >> >> handle(Req, destroy) -> >> .. >> >> handle(Req, list) -> >> .. >> . >> >> And no "default handler" because it is not required. Only good >> requests >> can pass to this handler. >> >> >> Then in your init function do something like >> your_lib:route_method(Req) which returns {ok, Req, Action} if it's >> good or {shutdown, ...} if not? It'll do exactly what you just said. >> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> >> >> > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Wed Feb 13 16:01:15 2013 From: g@REDACTED (Garrett Smith) Date: Wed, 13 Feb 2013 09:01:15 -0600 Subject: [erlang-questions] Messing with heart. Port and NIF, which one is better? In-Reply-To: References: Message-ID: Hi Gokul, On Wed, Feb 13, 2013 at 1:03 AM, Gokul Evuri wrote: > Hello, > I am beginner-level erlang user and i got two questions here, > is there a way to save my processes if the VM crashes? Welcome! If you use gen_server (or the simplified equivalent e2_service -- see http://e2project.org) you have a clear life cycle for your process: - init is used to define the initial state for the process when it starts - handle_xxx can be used to modify the state in response to various messages If you want to "persist your processes", you're talking about a) persisting the process state as it changes and b) providing a way to load the last known state at process start. E.g. init(InitArgs) -> {ok, load_state_from_disk(InitArgs)} handle_msg(Msg, _From, State) -> NewState = do_something_with_msg(Msg, State), save_state_to_disk(NewState), {reply, ok, NewState}. > And the second question, > Is there any good argument to use NIF instead of creating a connected > process for a port. The NIF interface is appropriate for defining simple functions in C. There are lots of 3rd party libraries where NIFs are used to plugin in long running, multi-threaded facilities, but this seems misguided to me. If there's even a small chance that your C program will crash, use a C port. Don't assume that the overhead of serializing messages over stdio is going to ruin your application performance. The safety of an external port is profoundly valuable. To rant slightly, it's surprisingly common to see people readily accept the "speed" trade off (use a NIF) over "safety" (use an external C port). This is an unfortunate trend. I use various NIF based libraries in production and routinely deal with Erlang crashes as a result (I plan to rewrite the more troubling ones as C ports). So what goes very fast (or so the thinking goes) suddenly goes to *zero* when the VM crashes -- and stays at zero until everything is restarted and initialized. This can be devastating in the very case where speed/throughput is most important -- very high load. Those are cases where a slightly slower interface can be paid off by a *much* faster recovery on error. The benefit goes up if you can distribute your work across multiple C ports -- the death of any one will only impact 1/N of your system. As an additional point, an external C port lets you write your C code using the Very Good Pattern of crash-early. I.e. if you don't like a particular state in your C process (e.g. assertion failure) call exit() and die! If you're forced to be defensive because you're afraid of killing the entire VM, your going *backward* to the days of bug-hiding and mysterious behavior. Why bother with Erlang in the first place? All that said, if your extension in C is a simple side effect free function or is otherwise "safe" -- a NIF will give you a simpler path to implement it and avoids the overhead of a system process. Garrett From raimo+erlang-questions@REDACTED Wed Feb 13 16:09:11 2013 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 13 Feb 2013 16:09:11 +0100 Subject: [erlang-questions] how to debug inet_res? In-Reply-To: References: <20130212161746.GA4812@erix.ericsson.se> Message-ID: <20130213150911.GA1360@erix.ericsson.se> On Tue, Feb 12, 2013 at 06:49:57PM +0100, Benoit Chesneau wrote: > On Tue, Feb 12, 2013 at 5:17 PM, Raimo Niskanen > wrote: > > On Tue, Feb 12, 2013 at 04:28:17PM +0100, Benoit Chesneau wrote: > >> Hi all, > >> > >> I am using a custom version of Erlang on iOS and for some reasons I > >> ignore yet the DNS resolution is failing in some networks. Since you > >> can't launch ports on iOS due to the sandboxing, Erlang is using the > >> module inet_res to do the resolution. Which works well on some network > >> but return {error, nxdomain} in an office at Singapore. > >> > >> Testing the same domain name on desktop with native or inet_res return > >> the IP address correctly. For now I have written a small nif to do the > >> gethostbyname call but I'm still curious to know why it fails on IOS. > >> not sure which kind of test I should do. Any idea? > > > > inet_res is the DNS resolver client written in Erlang. It does not call > > gethostbyname. The module inet_gethost_native uses an external port > > program to call gethostbyname. > > > > For inet_res, first try this: > > rr(inet_res). > > inet_res:resolve("google.com", in, a). > > > > If that does not work; I first suggest you call inet_db:get_rc() to get a > > summary of what settings are used (on both machines) and compare. > > I did that this is why i'm speaking about inet_res. > > On ios: {file, dns} where the desktop has native. To compare both > machine I did a test using `inet_res:gethostbyname/1` and > `inet_hosts:gethostbyname/1` on the same network (wifi). Did that mean there is an rc entry {lookup,[file,dns]} on iOS? > > in each case inet_hosts weren't working (desktop is on the mac), > strangely it returns {error, nxdomain} on the desktop and {error, > timeout} on ios. inet_res resolution is working on the desktop and not > on iOS. The module inet_hosts is the backend for the lookup method 'file' that looks into an internal cache of what was found when reading /etc/hosts. You can only expect to find 'localhost' and such there. The lookup method that calls gethostbyname in the OS is called 'native' and uses the backend module inet_gethost_native. It also uses an external port program inet_gethost that may pose problems for those who port to e.g iOS... Maybe that is why you have {lookup,[file,dns]} instead of the default {lookup,[native]}. The native resolver is the one that is aimed to be the most correct since it uses OS native calls hence the OS name resolution configuration. It may be so that the default behaviour for inet_res of monitoring /etc/resolv.conf does not work on iOS since there maybe is no such file; who knows how the resolver is configured on iOS? inet_res has to figure out a nameserver - try using {nameservers,[{{127,0,0,1},53}]} since there might be a DNS caching server running on the loopback interface. > > > > > Then you can look at the internal ETS table ets:i(inet_db) and compare them > > on both machines. > > > > To use a specific nameserver: > > inet_res:resolve("google.com", in, a, [{nameservers,[{{8,8,8,8},53}]}]). > > > > That looks up Googles address at Googles public nameserver 8.8.8.8. > > It does not work for me since I am behind a company firewall that does not > > allow outgoing DNS queries. I have to use a local nameserver. That local > > nameserver should be found in the ets table inet_db which should show in > > the inet_db:get_rc() output. If so e.g inet_res:gethostbyname("google.com") > > will produce a hostent record. > > > > Note. There is some magic in play indicated by an automatic default entry > > {resolv_conf,"/etc/resolv.conf"} in the inet_db:get_rc() output. That causes > > inet_res to update inet_db with content from /etc/resolv.conf before doing > > a DNS query. > > > > So do an inet_res call first; then if it does not work check the internal > > tables. Compare the tables before and after the first call. > > > Thanks will try that. > > > > You can also add an argument 'verbose' to the argument 4 option list of > > inet_res:resolve/4 that will show which servers are tried and what they > > reply / timeout. > > > > I didn't know about the verbose option, will try it as well. I will > update the thread with my results here. > > - beno?t > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From heinz@REDACTED Wed Feb 13 16:40:03 2013 From: heinz@REDACTED (Heinz Nikolaus Gies) Date: Wed, 13 Feb 2013 16:40:03 +0100 Subject: [erlang-questions] [ANN] Cowboy 0.8.0 In-Reply-To: <511A7D8C.5050107@ninenines.eu> References: <511A7D8C.5050107@ninenines.eu> Message-ID: <2C49D79F-9B9C-4888-A238-677E2A12811A@licenser.net> Awesome mate! On Feb 12, 2013, at 18:36, Lo?c Hoguin wrote: > Hello there! > > Cowboy 0.8 has been released. Cowboy is a small, fast and modular HTTP, REST and Websocket server. > > https://github.com/extend/cowboy/ > > The number of contributors who helped make this release considerably increased. Cowboy is available thanks to the code contributions from 50 users, double from the last release where 25 contributed. > > The number of users has also greatly increased. Cowboy is being used in ad bidding, set-top boxes, live TV events, content streaming services, and many more exciting areas. > > This new version has many highlights. You can take a look at the changelog for detailed information about the many changes. > > https://github.com/extend/cowboy/blob/master/CHANGELOG.md > > Cowboy scalability has been greatly improved in this version. This has been observed many times in production, including in the AdGear Tracker project (http://ferd.ca/rtb-where-erlang-blooms.html) where updated nodes were able to handle 2 times more requests compared to older nodes. This improvement cannot be observed in "hello world" types of benchmarks. An article will soon be published to explain the reasons for this. > > Cowboy now features a brand new user guide. It is still a work in progress, so please open a ticket on Github if something is missing or incorrect. > > http://ninenines.eu/docs/en/cowboy/HEAD/guide/introduction > > Remaining work before 1.0 include REST improvements and SPDY support. The rest of the API should now be very close to stable. > > I am looking for a good writer who would like to co-author a Cowboy book. The book will be accessible to people who don't know Erlang but will also contain everything there is to know about Cowboy, making it suitable for both beginners and experts. Contact me if you are interested. > > I now take donations in addition to commercial support options, to allow individual users to help the project stay alive and kicking. > > http://ninenines.eu/support > > Hope you enjoy it. As always, please send me as much feedback as possible to help me improve things even more, preferrably through Github tickets if it's related to code or documentation. > > Thanks for reading. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ingela.andin@REDACTED Wed Feb 13 16:53:42 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 13 Feb 2013 16:53:42 +0100 Subject: [erlang-questions] question about eldap and ssh In-Reply-To: References: Message-ID: Hello! What version are you using? Latest ssh specifies public_key as dependency application and I do not think that eldap should be rutime dependent on ASN1 (atleast not latest version). Regards Ingela Erlang/OTP team - Ericsson AB 2013/2/11, Minin Maxim : > Hi > > i try to build a release of an application with dependencies of some > standard libs and fail because not all necessary applications are have been > packaged. > I use eldap and ssh in my application. Eldap itself needs the lib ans1 and > ssh - public_key. But both have not in their app-files the ans1 and > public_key as dependency. As a result this two applications (ans1 and > public_key) are not in my release. > Is it a bug, that the dependencies are missing in the app-file of eldap and > ssh or am I doing something wrong? > > thanks > > Maxim Minin > From jesper.louis.andersen@REDACTED Wed Feb 13 17:58:03 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 13 Feb 2013 17:58:03 +0100 Subject: [erlang-questions] Alternatives to Erlang In-Reply-To: References: Message-ID: <16A73587-6B77-4ECA-84C3-DB28AC7864B6@erlang-solutions.com> On Feb 11, 2013, at 6:29 PM, Maruthavanan Subbarayan wrote: > Hi, > > I am on the way to creating an application which would create atoms dynamically. > > These atoms are created because I have user registering into my system and each user has set of 10 mnesia tables to be created. Since atoms has a limitation I am afraid that if 100K users register into my system I may run of atoms for creating tables for more users. > > Creating an ETS table with name will give me table identifier with integer. Does a similar thing exists for mnesia too? > You will run out of ETS tables long before you run out of atoms with that solution. You should probably have 10 mnesia tables in all, and then use a key where the user is part. That is, instead of storing `Key` as your key, you store `{User, Key}`. This means you would need 10 tables in all. And the performance would probably be better than keeping a large number of small tables. As for the atom limit, let users be stored as (small) binaries or integers. Then you avoid that problem entirely. If you need to register a process under a user id, then use the `gproc` application by Ulf Wiger to maintain the lookup table for you. Look at the {via, ?} registration in gen_server's for some hint. Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From watson.timothy@REDACTED Wed Feb 13 19:15:55 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Wed, 13 Feb 2013 18:15:55 +0000 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: <511AE189.4000604@telegraphics.com.au> Message-ID: <35E26B16-1746-4F75-ADA3-672887FC81D2@gmail.com> On 13 Feb 2013, at 07:41, Joe Armstrong wrote: > On Wed, Feb 13, 2013 at 1:42 AM, Toby Thain wrote: > On 11/02/13 10:09 AM, Andrew Pennebaker wrote: > Of course Erlang has a killer actors library as a core component of > itself. Sadly, Erlang is not always an option (e.g., customer requires a > JVM language). Which actor libraries in other languages come close to > Erlang's in terms of: > > * Syntax conciseness (spawn, !, receive) > * Polymorphism (receiving different kinds of messages) > * Fault-tolerance (GC per actor, actor crash doesn't crash the system) > > Akka and Scala specifically offer all the above. > As does Cloud Haskell, as well as selective receive and matching/condition-based receive clauses, timeouts, etc. I suspect Akka/Scala do as well, but I haven't looked. > Plus the actor model does not address problems of fault propagation. > Indeed! It is exactly why Erlang shines so much - it's the Erlang model that Cloud Haskell explicitly follows, not the Actor model. In fact, our general motto is "do what Erlang does". > The statement "X has Erlang like semantics" usually means "X has Erlang like semantics > except for error handling" - handling errors out-of-band is a big win. > > Isolation is the key property that must be guaranteed for actors. If <> can crash > <> code then all bets are off. > That is why Cloud Haskell was built to follow the "Unified Semantics for Future Erlang" paper (Svensson et al.) - send is async, process failures are completely isolated and only propagated via links (or signalled via monitors), trapping of exit signals is structured/scoped, etc. It is basically a homage to Erlang, rather than 'we can do actors/distribution too....' It is, of course, quite incomplete as it stands and there are various decisions that have been deferred until later. Not to mention the fact that Haskell has a very strict type system, so programming Cloud Haskell is a very different experience to programming Erlang - and there's no hot-upgrades or anything like that. The formal semantics are available at http://haskell-distributed.github.com/static/semantics.pdf - any feedback about the implementation or conceptual approach will be gratefully received!!! :) Of course, I spend my days programming Erlang, so hopefully (at night!) I'm bringing an Erlang-ish mindset to the Cloud Haskell project. One of these days I'm going to get around to writing a network-transport backend that Cloud Haskell nodes can use to speak the Erlang distribution protocol: Then we'll have H-Nodes as well as C-nodes/jinterface/pyinterface/etc. Someone has already done a bit of that work, but it's quite old and bit-rotten. Cheers, Tim > /Joe > > > > > > > > * Live updates (different code versions up at the same time, hot code > updates) > > This can be done on the JVM, as I understand it (JRebel etc) but I can't speak from experience. Certainly it's not as designed-in and natural as Erlang. > > --Toby > > > > Does Scala and/or Akka offer anything like these features? What about > Ruby's Celluloid? What are the best actor libraries in Common Lisp and > Scheme? > > The Haskell community seems to prefer STM over actors. How does the > Erlang community feel about STM? > > -- > Cheers, > > Andrew Pennebaker > www.yellosoft.us > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From toby@REDACTED Thu Feb 14 03:35:11 2013 From: toby@REDACTED (Toby Thain) Date: Wed, 13 Feb 2013 21:35:11 -0500 Subject: [erlang-questions] Best actor libraries in other languages? In-Reply-To: References: <511AE189.4000604@telegraphics.com.au> Message-ID: <511C4D5F.1020900@telegraphics.com.au> On 12/02/13 8:14 PM, Erik S?e S?rensen wrote: > 2013/2/13 Toby Thain > > > On 11/02/13 10:09 AM, Andrew Pennebaker wrote: > > Of course Erlang has a killer actors library as a core component of > itself. Sadly, Erlang is not always an option (e.g., customer > requires a > JVM language). > > Oh, but Erlang *is* a JVM language these days :-) Albeit not maturely > enough for production use, last I checked. > > Which actor libraries in other languages come close to > Erlang's in terms of: > > * Syntax conciseness (spawn, !, receive) > > Missing from the list: selective receive. :-) > > * Polymorphism (receiving different kinds of messages) > * Fault-tolerance (GC per actor, actor crash doesn't crash the > system) > > > Akka and Scala specifically offer all the above. > > Probably not the "GC per actor" part... Ah yes, good point. Thanks. --Toby > What other systems *do* provide that, though? > (I suspect Dart does, but it is lacking in other respects.) > > > * Live updates (different code versions up at the same time, hot code > updates) > > > This can be done on the JVM, as I understand it (JRebel etc) but I > can't speak from experience. Certainly it's not as designed-in and > natural as Erlang. > > > Interesting; I didn't know about JRebel and LiveRebel. > > > --Toby > > > > Does Scala and/or Akka offer anything like these features? What > about > Ruby's Celluloid? What are the best actor libraries in Common > Lisp and > Scheme? > > The Haskell community seems to prefer STM over actors. How does the > Erlang community feel about STM? > > -- > Cheers, > > Andrew Pennebaker > www.yellosoft.us > > > > ______________________________ _________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/ listinfo/erlang-questions > > > > ______________________________ _________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/ listinfo/erlang-questions > > > From jeremy@REDACTED Thu Feb 14 06:28:33 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Wed, 13 Feb 2013 21:28:33 -0800 Subject: [erlang-questions] erlang:system_info(port_limit) broken? Message-ID: This info argument is defined in www.erlang.org/doc/man/erlang.html#system_info_port_limit Steps to reproduce: 1. Invoke an erlang shell (erl) 2. erlang:system_info(port_limit). Returns ** exception error: bad argument in function erlang:system_info/1 called as erlang:system_info(port_limit) Any ideas? Cheers, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Thu Feb 14 06:29:53 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Wed, 13 Feb 2013 21:29:53 -0800 Subject: [erlang-questions] erlang:system_info(port_limit) broken? In-Reply-To: References: Message-ID: Sorry, neglected to mention that this is using R15B03-1. I've verified that this seems to not be working on amd64 hardware and intel i5 hardware in both linux and osx. On Wed, Feb 13, 2013 at 9:28 PM, Jeremy Ong wrote: > This info argument is defined in > www.erlang.org/doc/man/erlang.html#system_info_port_limit > > Steps to reproduce: > > 1. Invoke an erlang shell (erl) > 2. erlang:system_info(port_limit). > > Returns > > ** exception error: bad argument > in function erlang:system_info/1 > called as erlang:system_info(port_limit) > > Any ideas? > > Cheers, > Jeremy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Thu Feb 14 06:32:07 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Wed, 13 Feb 2013 21:32:07 -0800 Subject: [erlang-questions] Erlang's configure script Message-ID: I noticed today that when attempting to build erlang from source, the build will succeed with HIPE enabled even if M4 is not installed. The docs explicitly mention that this is a requirement so shouldn't not having M4 cause an error of sorts or at least give a notice at the end (like it does with wxwidgets or xsltproc) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Maxim.Minin@REDACTED Thu Feb 14 07:56:36 2013 From: Maxim.Minin@REDACTED (Minin Maxim) Date: Thu, 14 Feb 2013 07:56:36 +0100 Subject: [erlang-questions] question about eldap and ssh Message-ID: Hello Ingela, thank You for your answer. I use R15B03. You are right - in R16A ssh specifies public_key as dependency application. But eldap is still dependent on ASN1. If I grep, I get following result: grep -n asn src/eldap.erl 202:optional([]) -> asn1_NOVALUE; 376: TcpOpts = [{packet, asn1}, {active,false}], 683: {ok,Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message), 702: case asn1rt:decode('ELDAPv3', 'LDAPMessage', Packet) of The module asn1rt is a part of asn1 lib. Are you planning to remove the dependency from ASN1 kompletely? Thanks Maxim Minin Von: Ingela Andin [mailto:ingela.andin@REDACTED] Gesendet: Mittwoch, 13. Februar 2013 16:54 An: Minin Maxim Cc: erlang-questions@REDACTED Betreff: Re: [erlang-questions] question about eldap and ssh Hello! What version are you using? Latest ssh specifies public_key as dependency application and I do not think that eldap should be rutime dependent on ASN1 (atleast not latest version). Regards Ingela Erlang/OTP team - Ericsson AB 2013/2/11, Minin Maxim : > Hi > > i try to build a release of an application with dependencies of some > standard libs and fail because not all necessary applications are have > been packaged. > I use eldap and ssh in my application. Eldap itself needs the lib ans1 > and ssh - public_key. But both have not in their app-files the ans1 > and public_key as dependency. As a result this two applications (ans1 > and > public_key) are not in my release. > Is it a bug, that the dependencies are missing in the app-file of > eldap and ssh or am I doing something wrong? > > thanks > > Maxim Minin > From jinni.park@REDACTED Thu Feb 14 09:10:20 2013 From: jinni.park@REDACTED (Park, Sungjin) Date: Thu, 14 Feb 2013 17:10:20 +0900 Subject: [erlang-questions] About ssl connection speed. Message-ID: Hello guys, I'm experiencing that erlang server's ssl listener is significantly slower - by a factor of hundreds - than plain tcp listener. To give you some numbers, my server accepts less than 100 ssl clients per second while it accepts more than 10k tcp clients per second with the same service logic behind save for the transport types. I tested the same scenario with a server written in c for comparison and got numbers about 1k/s in ssl and 10k in tcp. I want to make sure if this is inevitable as long as I'm using erlang or not. -- Park, Sungjin ------------------------------------------------------------------------------------------------------------------- Peculiar travel suggestions are dancing lessons from god. -- The Books of Bokonon ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander_koch_log@REDACTED Thu Feb 14 08:02:42 2013 From: alexander_koch_log@REDACTED (alexander_koch_log) Date: Thu, 14 Feb 2013 08:02:42 +0100 Subject: [erlang-questions] Erlang I/O and NIF Message-ID: <511C8C12.8060106@lavabit.com> Hi, Came across this changelog https://github.com/basho/bitcask/pull/76/commits I know there have been multiple discussion on this mailing list regarding pure erlang I/O implementation vs NIF based implementation. I also understand that using NIF would block the erlang VM, hence it is not an easy choice to decide which approach to leverage in an application where lots of disk I/O should occur. This gives the impression that Erlang may not be at all the right choice for any application doing lots of I/O, in your opinion, does this statement hold true ? Most of these posts date from 2010 and I was curious whether this was still the case for Erlang as of today (as stated in the commit). Are they any work in the pipeline to improve disk I/O ? Is it safe to assume that "reasonable" performance is achievable when doing binary reads with erlang based I/O? Thanks, Alex From tuncer.ayaz@REDACTED Thu Feb 14 08:22:34 2013 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Thu, 14 Feb 2013 08:22:34 +0100 Subject: [erlang-questions] R16B enif_consume_timeslice Message-ID: R16B gained a new NIF API function that allows your NIFs to better co-operate with the vm. Documentation: https://github.com/erlang/otp/commit/ef3566f48e08#L0R586 Example: https://github.com/erlang/otp/commit/ef3566f48e08#L5R1459 Thanks Sverker! This could be quite useful and ease the wait for native processes :). From raimo+erlang-questions@REDACTED Thu Feb 14 10:46:46 2013 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 14 Feb 2013 10:46:46 +0100 Subject: [erlang-questions] List policy tweak In-Reply-To: <20130212174024.BF478E6726@smtp.hushmail.com> References: <20130212174024.BF478E6726@smtp.hushmail.com> Message-ID: <20130214094646.GA22350@erix.ericsson.se> I have decided to make all new subscribers moderated for their first post to block future single post totally off topic subscriptions. -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From kenneth.lundin@REDACTED Thu Feb 14 11:25:34 2013 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Thu, 14 Feb 2013 11:25:34 +0100 Subject: [erlang-questions] question about eldap and ssh In-Reply-To: References: Message-ID: Just a clarification. The dependency information given in the .app file is only intended to give info about the start order dependency. I.e. that another application needs to be started before this application is started. Some applications are just libraries of code and does not have any processes that need to be started. This is the case for public_key and asn1. So you cannot rely on the info in the .app files when it comes to run-time dependencys (you don't get the complete picture). For getting the complete picture you can use reltool and or xref. This is how it is today. It might be good to include a more complete dependency info in the .app files but we are not there yet. /Kenneth Erlang/OTP, Ericsson On Thu, Feb 14, 2013 at 7:56 AM, Minin Maxim wrote: > Hello Ingela, > > thank You for your answer. > I use R15B03. > You are right - in R16A ssh specifies public_key as dependency application. > But eldap is still dependent on ASN1. If I grep, I get following result: > > grep -n asn src/eldap.erl > > 202:optional([]) -> asn1_NOVALUE; > 376: TcpOpts = [{packet, asn1}, {active,false}], > 683: {ok,Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message), > 702: case asn1rt:decode('ELDAPv3', 'LDAPMessage', Packet) of > > The module asn1rt is a part of asn1 lib. Are you planning to remove the > dependency from ASN1 kompletely? > > Thanks > > Maxim Minin > > Von: Ingela Andin [mailto:ingela.andin@REDACTED] > Gesendet: Mittwoch, 13. Februar 2013 16:54 > An: Minin Maxim > Cc: erlang-questions@REDACTED > Betreff: Re: [erlang-questions] question about eldap and ssh > > Hello! > > What version are you using? Latest ssh specifies public_key as > dependency application and I do not think that eldap should be rutime > dependent on ASN1 (atleast not latest version). > > Regards Ingela Erlang/OTP team - Ericsson AB > > > 2013/2/11, Minin Maxim : > > Hi > > > > i try to build a release of an application with dependencies of some > > standard libs and fail because not all necessary applications are have > > been packaged. > > I use eldap and ssh in my application. Eldap itself needs the lib ans1 > > and ssh - public_key. But both have not in their app-files the ans1 > > and public_key as dependency. As a result this two applications (ans1 > > and > > public_key) are not in my release. > > Is it a bug, that the dependencies are missing in the app-file of > > eldap and ssh or am I doing something wrong? > > > > thanks > > > > Maxim Minin > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Thu Feb 14 11:30:50 2013 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Thu, 14 Feb 2013 04:30:50 -0600 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: Message of "Wed, 13 Feb 2013 09:01:15 CST." Message-ID: <90893.1360837850@snookles.snookles.com> I'm starting a new'ish thread to mention a bit of experience that Basho has had with NIFs in Riak. Garrett Smith wrote: >> And the second question, Is there any good argument to use NIF >> instead of creating a connected process for a port. gs> The NIF interface is appropriate for defining simple functions in C. gs> There are lots of 3rd party libraries where NIFs are used to plugin gs> in long running, multi-threaded facilities, but this seems misguided gs> to me. "Simple functions in C" is a tricky matter ... and it has gotten tricker with the Erlang/OTP releases R15 and R16. In R14 and earlier, it wasn't necessarily a horrible thing if you had C code (or C++ or Fortran or ...) that executed in NIF context for half a second or more. If your NIF was executing for that long, you knew that you were interfering with the Erlang scheduler Pthread that was executing your NIF's C/C++/Fortran/whatever code. That can cause some weird delays in executing other Erlang processes, but for some apps, that's OK. However, with R15, the internal guts of the Erlang process scheduler Pthreads has changed. Now, if you have a NIF that executes for even a few milliseconds, the scheduler algorithm can get confused. Instead of blocking an Erlang scheduler Pthread, you both block that Pthread *and* you might cause some other scheduler Pthreads to decide incorrectly to go to sleep (because there aren't enough runnable Erlang processes to bother trying to schedule). Your 8/16/24 CPU core box can find itself down to only 3 or 2 active Erlang scheduler Pthreads when there really is more than 2-3 cores of work waiting. So, suddenly your "simple functions in C" are now "simple functions in C that must finish execution in about 1 millisecond or less". If your C code might take longer than that, then you must use some kind of thread pool to transfer the long-running work away from the Erlang scheduler Pthread. Not simple at all, alas. -Scott From rickard@REDACTED Thu Feb 14 11:35:38 2013 From: rickard@REDACTED (Rickard Green) Date: Thu, 14 Feb 2013 11:35:38 +0100 Subject: [erlang-questions] erlang:system_info(port_limit) broken? In-Reply-To: References: Message-ID: The 'port_limit' argument was introduced in erts-5.10/OTP-R16A (as well as 'port_count'). That is, the badarg is expected in R15. Regards, Rickard Green, Erlang/OTP, Erlang/OTP On Feb 14, 2013, at 6:29 AM, Jeremy Ong wrote: > Sorry, neglected to mention that this is using R15B03-1. I've verified that this seems to not be working on amd64 hardware and intel i5 hardware in both linux and osx. > > > On Wed, Feb 13, 2013 at 9:28 PM, Jeremy Ong wrote: > This info argument is defined in www.erlang.org/doc/man/erlang.html#system_info_port_limit > > Steps to reproduce: > > 1. Invoke an erlang shell (erl) > 2. erlang:system_info(port_limit). > > Returns > > ** exception error: bad argument > in function erlang:system_info/1 > called as erlang:system_info(port_limit) > > Any ideas? > > Cheers, > Jeremy > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Thu Feb 14 11:52:17 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 14 Feb 2013 02:52:17 -0800 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: <90893.1360837850@snookles.snookles.com> References: <90893.1360837850@snookles.snookles.com> Message-ID: <511CC1E1.5010109@gmail.com> On 02/14/2013 02:30 AM, Scott Lystig Fritchie wrote: > I'm starting a new'ish thread to mention a bit of experience that Basho > has had with NIFs in Riak. > > Garrett Smith wrote: > >>> And the second question, Is there any good argument to use NIF >>> instead of creating a connected process for a port. > gs> The NIF interface is appropriate for defining simple functions in C. > gs> There are lots of 3rd party libraries where NIFs are used to plugin > gs> in long running, multi-threaded facilities, but this seems misguided > gs> to me. > > "Simple functions in C" is a tricky matter ... and it has gotten tricker > with the Erlang/OTP releases R15 and R16. > > In R14 and earlier, it wasn't necessarily a horrible thing if you had C > code (or C++ or Fortran or ...) that executed in NIF context for half a > second or more. If your NIF was executing for that long, you knew that > you were interfering with the Erlang scheduler Pthread that was > executing your NIF's C/C++/Fortran/whatever code. That can cause some > weird delays in executing other Erlang processes, but for some apps, > that's OK. > > However, with R15, the internal guts of the Erlang process scheduler > Pthreads has changed. Now, if you have a NIF that executes for even a > few milliseconds, the scheduler algorithm can get confused. Instead of > blocking an Erlang scheduler Pthread, you both block that Pthread *and* > you might cause some other scheduler Pthreads to decide incorrectly to > go to sleep (because there aren't enough runnable Erlang processes to > bother trying to schedule). Your 8/16/24 CPU core box can find itself > down to only 3 or 2 active Erlang scheduler Pthreads when there really > is more than 2-3 cores of work waiting. > > So, suddenly your "simple functions in C" are now "simple functions in C > that must finish execution in about 1 millisecond or less". If your C > code might take longer than that, then you must use some kind of thread > pool to transfer the long-running work away from the Erlang scheduler > Pthread. Not simple at all, alas. > > -Scott > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions These problems are what NIF native processes will solve, right? The only other alternative would be to use the async thread pool within a port driver, which may not help the schedulers and is obsoleted by native processes (not to mention the job queue per thread situation which can block on long jobs). From watson.timothy@REDACTED Thu Feb 14 12:41:24 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 14 Feb 2013 11:41:24 +0000 Subject: [erlang-questions] List policy tweak In-Reply-To: <20130214094646.GA22350@erix.ericsson.se> References: <20130212174024.BF478E6726@smtp.hushmail.com> <20130214094646.GA22350@erix.ericsson.se> Message-ID: On 14 Feb 2013, at 09:46, Raimo Niskanen wrote: > I have decided to make all new subscribers moderated for their first post > to block future single post totally off topic subscriptions. > -- This is good news. I don't suppose we can filter out that 'worldcomp' nonsense whilst we're at it can we!? > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rickard@REDACTED Thu Feb 14 12:52:36 2013 From: rickard@REDACTED (Rickard Green) Date: Thu, 14 Feb 2013 12:52:36 +0100 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: <511CC1E1.5010109@gmail.com> References: <90893.1360837850@snookles.snookles.com> <511CC1E1.5010109@gmail.com> Message-ID: <689F6548-625C-4447-9B9C-80195FE91E54@erlang.org> On Feb 14, 2013, at 11:52 AM, Michael Truog wrote: > On 02/14/2013 02:30 AM, Scott Lystig Fritchie wrote: >> I'm starting a new'ish thread to mention a bit of experience that Basho >> has had with NIFs in Riak. >> >> Garrett Smith wrote: >> >>>> And the second question, Is there any good argument to use NIF >>>> instead of creating a connected process for a port. >> gs> The NIF interface is appropriate for defining simple functions in C. >> gs> There are lots of 3rd party libraries where NIFs are used to plugin >> gs> in long running, multi-threaded facilities, but this seems misguided >> gs> to me. >> >> "Simple functions in C" is a tricky matter ... and it has gotten tricker >> with the Erlang/OTP releases R15 and R16. >> >> In R14 and earlier, it wasn't necessarily a horrible thing if you had C >> code (or C++ or Fortran or ...) that executed in NIF context for half a >> second or more. If your NIF was executing for that long, you knew that >> you were interfering with the Erlang scheduler Pthread that was >> executing your NIF's C/C++/Fortran/whatever code. That can cause some >> weird delays in executing other Erlang processes, but for some apps, >> that's OK. >> >> However, with R15, the internal guts of the Erlang process scheduler >> Pthreads has changed. Now, if you have a NIF that executes for even a >> few milliseconds, the scheduler algorithm can get confused. Instead of >> blocking an Erlang scheduler Pthread, you both block that Pthread *and* >> you might cause some other scheduler Pthreads to decide incorrectly to >> go to sleep (because there aren't enough runnable Erlang processes to >> bother trying to schedule). Your 8/16/24 CPU core box can find itself >> down to only 3 or 2 active Erlang scheduler Pthreads when there really >> is more than 2-3 cores of work waiting. >> >> So, suddenly your "simple functions in C" are now "simple functions in C >> that must finish execution in about 1 millisecond or less". If your C >> code might take longer than that, then you must use some kind of thread >> pool to transfer the long-running work away from the Erlang scheduler >> Pthread. Not simple at all, alas. >> Native code (drivers and NIFs) have always been expected to execute for very short periods of time. The major difference is that it is more clearly documented today. The number of problems you might run into if you run native code that do not behave well has increased though. This is, however, not new to R15. This has been the case since R11, due to optimizations of the smp runtime system. One such optimization was multiple run-queues that was introduced in R13. Regarding scheduling, R12 to R13 is where the major difference is. In some cases we could try to fix problems caused by native code that do not behave well. This would however very often cause a performance penalty that always have to be paid, and it would also prevent us from implementing a lot of optimizations. In my opinion this would just be plain wrong. The VM has never been intended for scheduling of arbitrary native code. A NIF or a driver is supposed to be aware of the VM and help it, not break it. >> -Scott >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > These problems are what NIF native processes will solve, right? No, but "dirty schedulers" were supposed to ease implementation of things like this. Note that already today you got all the primitives you need, as for example threads for NIFs and drivers. > The only other alternative would be to use the async thread pool within a port driver, which may not help the schedulers and is obsoleted by native processes (not to mention the job queue per thread situation which can block on long jobs). Regards, Rickard Green, Erlang/OTP, Ericsson AB From max.lapshin@REDACTED Thu Feb 14 13:05:33 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 14 Feb 2013 15:05:33 +0300 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: <511CC1E1.5010109@gmail.com> References: <90893.1360837850@snookles.snookles.com> <511CC1E1.5010109@gmail.com> Message-ID: Can enif_consume_timeslice help to mark nif-blocked pthread still active? -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Feb 14 14:03:39 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Thu, 14 Feb 2013 14:03:39 +0100 Subject: [erlang-questions] question about eldap and ssh In-Reply-To: References: Message-ID: <511CE0AB.4090706@ninenines.eu> Reltool and xref do not always get the complete picture, especially when you start putting module names into variables and only using them this way. It would be great if it could be more explicit. I know the dependency information in the .app file is currently for the start order dependency, but perhaps this meaning could be changed? Today starting a library application works, e.g. application:start(asn1), so it doesn't seem like it'd be too problematic to just put everything there? On 02/14/2013 11:25 AM, Kenneth Lundin wrote: > Just a clarification. > The dependency information given in the .app file is only intended to > give info about the start order dependency. I.e. that another > application needs to be started before this application is started. > Some applications are just libraries of code and does not have any > processes that need to be started. This is the case for public_key and asn1. > So you cannot rely on the info in the .app files when it comes to > run-time dependencys (you don't get the complete picture). > For getting the complete picture you can use reltool and or xref. > This is how it is today. It might be good to include a more complete > dependency info in the .app files but we are not there yet. > /Kenneth Erlang/OTP, Ericsson > > On Thu, Feb 14, 2013 at 7:56 AM, Minin Maxim > wrote: > > Hello Ingela, > > thank You for your answer. > I use R15B03. > You are right - in R16A ssh specifies public_key as dependency > application. > But eldap is still dependent on ASN1. If I grep, I get following result: > > grep -n asn src/eldap.erl > > 202:optional([]) -> asn1_NOVALUE; > 376: TcpOpts = [{packet, asn1}, {active,false}], > 683: {ok,Bytes} = asn1rt:encode('ELDAPv3', 'LDAPMessage', Message), > 702: case asn1rt:decode('ELDAPv3', 'LDAPMessage', Packet) of > > The module asn1rt is a part of asn1 lib. Are you planning to remove > the dependency from ASN1 kompletely? > > Thanks > > Maxim Minin > > Von: Ingela Andin [mailto:ingela.andin@REDACTED > ] > Gesendet: Mittwoch, 13. Februar 2013 16:54 > An: Minin Maxim > Cc: erlang-questions@REDACTED > Betreff: Re: [erlang-questions] question about eldap and ssh > > Hello! > > What version are you using? Latest ssh specifies public_key as > dependency application and I do not think that eldap should be > rutime dependent on ASN1 (atleast not latest version). > > Regards Ingela Erlang/OTP team - Ericsson AB > > > 2013/2/11, Minin Maxim >: > > Hi > > > > i try to build a release of an application with dependencies of some > > standard libs and fail because not all necessary applications are > have > > been packaged. > > I use eldap and ssh in my application. Eldap itself needs the lib > ans1 > > and ssh - public_key. But both have not in their app-files the ans1 > > and public_key as dependency. As a result this two applications > (ans1 > > and > > public_key) are not in my release. > > Is it a bug, that the dependencies are missing in the app-file of > > eldap and ssh or am I doing something wrong? > > > > thanks > > > > Maxim Minin > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From dronnikov@REDACTED Thu Feb 14 15:13:29 2013 From: dronnikov@REDACTED (Vladimir Dronnikov) Date: Thu, 14 Feb 2013 18:13:29 +0400 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> <511B6D44.900@ninenines.eu> <511B72AD.4020102@ninenines.eu> Message-ID: May be this could help: https://github.com/dvv/stable#cowboy_mrouter On Wed, Feb 13, 2013 at 6:44 PM, Vladimir Dronnikov wrote: > Max, one could insert a middleware before cowboy_router which mangle URI > to be Method/URI and then in the router rules treat Method chunk as binding > -- hence, method constraints. > > --Vladimir > > > On Wed, Feb 13, 2013 at 3:02 PM, Lo?c Hoguin wrote: > >> The HTTP RFC says this on page 6: >> >> It builds on the discipline of reference >> provided by the Uniform Resource Identifier (URI) [3], as a location >> (URL) [4] or name (URN) [20], for indicating the resource to which a >> method is to be applied. >> >> It then defines resources as follow on page 9: >> >> A network data object or service that can be identified by a URI, >> as defined in section 3.2. Resources may be available in multiple >> representations (e.g. multiple languages, data formats, size, and >> resolutions) or vary in other ways. >> >> It then describes how to locate resources in many of the following >> sections, starting with section 3.2 page 18. >> >> Cowboy implements this. It maps URIs to resources. It makes absolutely no >> assumptions as to how your resources operate. Anything that is part of the >> URI can and will be used to locate resources. But if it's not part of the >> URI, then it's not something used to find resources as defined by the HTTP >> standard, which makes it out of the scope of the project. >> >> It still provides you with many ways to plug yourself into Cowboy to >> achieve what you want, which include: >> >> * Middleware >> * Protocol upgrade (what REST handlers are doing) >> * init/3 based method dispatching >> * handle/2 based method dispatching >> >> And I'm sure there's more. >> >> >> On 02/13/2013 11:42 AM, Max Lapshin wrote: >> >>> Loic, my idea about routing is that router is a piece of software that >>> translates input http request into presentation, convenient for handler >>> and selects proper handler for this with proper arguments. >>> >>> I don't understand your logic: you allow to run regexp to validate if >>> constraint is a digit: "/users/:user_id", [{user_id,fun() -> .. end}], >>> but you are strongly against validating method in constraints. >>> >>> You tell me to spread logic between your router and my router. You tell >>> me that I need to use two routers that will somehow interfere with each >>> other. What is the idea? I really don't understand. >>> >>> >>> On Wed, Feb 13, 2013 at 2:39 PM, Lo?c Hoguin >> > wrote: >>> >>> On 02/13/2013 11:34 AM, Max Lapshin wrote: >>> >>> handle(Req, State) -> >>> handle_method(cowboy_req:get(_**_method, Req), Req, >>> State). >>> >>> will require >>> >>> >>> handle_method(<<"GET">>, Req, State) >>> handle_method(<<"DELETE">>, Req, State) >>> handle_method(Other, Req, State) >>> here we copy from module to module default reply on "method >>> not >>> supported" or 404, whatever we choose. >>> >>> >>> with method constraints code can look so: >>> >>> >>> init(_, Req, [Action]) -> >>> {ok, Req, Action}. >>> >>> >>> handle(Req, show) -> >>> .. >>> >>> handle(Req, destroy) -> >>> .. >>> >>> handle(Req, list) -> >>> .. >>> . >>> >>> And no "default handler" because it is not required. Only good >>> requests >>> can pass to this handler. >>> >>> >>> Then in your init function do something like >>> your_lib:route_method(Req) which returns {ok, Req, Action} if it's >>> good or {shutdown, ...} if not? It'll do exactly what you just said. >>> >>> >>> -- >>> Lo?c Hoguin >>> Erlang Cowboy >>> Nine Nines >>> http://ninenines.eu >>> >>> >>> >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> ______________________________**_________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/**listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Thu Feb 14 15:18:51 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 14 Feb 2013 17:18:51 +0300 Subject: [erlang-questions] Cowboy routes with Method as a condition In-Reply-To: References: <511B5614.2000602@ninenines.eu> <511B5C94.1020606@ninenines.eu> <511B6AF6.3030002@ninenines.eu> <511B6D44.900@ninenines.eu> <511B72AD.4020102@ninenines.eu> Message-ID: Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Thu Feb 14 15:46:34 2013 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Thu, 14 Feb 2013 15:46:34 +0100 Subject: [erlang-questions] Slow line oriented IO Message-ID: Hello, I know it was been already discussed here in list and it is also recurring topic for at least five years. But anyway I have been bitten by it again and also found pretty pathological case. I have 30MB text file and it has a few near to 1MB lines there. (I can provide file with same line lengths if somebody interested.) What I have been observed is that reading this file using raw file:read_line/1 takes 51s! For comparison I have tried some different approaches and what I got (line_read_test:read_std/1 is using file:read_line/1): 1> timer:tc(line_read_test,read_std,["test.txt"]). {51028105,2408} 2> timer:tc(line_read_test,read,["test.txt"]). {226220,2408} 3> timer:tc(line_read_test,read_port,["test.txt"]). {139388,2408} $time perl -nE'$i++}{say $i' test.txt 2408 real 0m0.053s user 0m0.044s sys 0m0.008s $ time wc -l test.txt 2408 test.txt real 0m0.013s user 0m0.004s sys 0m0.008s $ time ./a.out test.txt 2408 real 0m0.020s user 0m0.012s sys 0m0.008s It means erlang should be at least 225 times faster (line_read_test:read/1 which has flow control). Erlang can be 350 times faster (line_read_test:read_port/1 without flow control). Another high level language (perl) is almost thousand times faster. Special C program is almost four thousands times faster and old good glibc is two and half thousands times faster. Come on guys it is not even fun when simple (and wrong) erlang wrapper around standard module is more than two order of magnitude faster. >From mine experience when there is something two orders of magnitude slower it tells me there is something damn wrong. I have looked into efile_drv.c and ti is unfortunately far beyond mine C skill but if simple buffering and binary:match/2 can outperform it 225 times there has to be something rotten in this code. I have also experimented with read_ahead option in file:open and changing to less value makes thing worse. Just to make grasp how bad it is, in same time I'm able sort 150 million 64bit values (1.2GB of data) three times (one CPU core same HW). It is not in flow control, mine simple wrapper does flow control too. It can't make current code less intrusive, if it consumes 100% CPU for 51s instead of 226ms then it will definitely affect whole server. It is not in concurrent access, mine code allows concurrent access too. Admitting there is something broken is first step to fixing it. I hope I helped at least in this way. With best regards Hynek Vychodil -------------- next part -------------- A non-text attachment was scrubbed... Name: line_read_test.erl Type: application/octet-stream Size: 3764 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: line_read_test.c Type: text/x-csrc Size: 318 bytes Desc: not available URL: From raimo+erlang-questions@REDACTED Thu Feb 14 16:07:32 2013 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 14 Feb 2013 16:07:32 +0100 Subject: [erlang-questions] List policy tweak In-Reply-To: References: <20130212174024.BF478E6726@smtp.hushmail.com> <20130214094646.GA22350@erix.ericsson.se> Message-ID: <20130214150732.GB5477@erix.ericsson.se> On Thu, Feb 14, 2013 at 11:41:24AM +0000, Tim Watson wrote: > On 14 Feb 2013, at 09:46, Raimo Niskanen wrote: > > I have decided to make all new subscribers moderated for their first post > > to block future single post totally off topic subscriptions. > > -- > > This is good news. I don't suppose we can filter out that 'worldcomp' nonsense whilst we're at it can we!? That is the trigger and immediate purpose. The archives has been cleansed. > > > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From puzza007@REDACTED Thu Feb 14 16:16:01 2013 From: puzza007@REDACTED (Paul Oliver) Date: Thu, 14 Feb 2013 10:16:01 -0500 Subject: [erlang-questions] About ssl connection speed. In-Reply-To: References: Message-ID: Hi there, Check this thread http://erlang.org/pipermail/erlang-questions/2012-April/065811.html Cheers, Paul. On Thu, Feb 14, 2013 at 3:10 AM, Park, Sungjin wrote: > Hello guys, > > I'm experiencing that erlang server's ssl listener is significantly slower > - by a factor of hundreds - than plain tcp listener. To give you some > numbers, my server accepts less than 100 ssl clients per second while it > accepts more than 10k tcp clients per second with the same service logic > behind save for the transport types. I tested the same scenario with a > server written in c for comparison and got numbers about 1k/s in ssl and > 10k in tcp. > > I want to make sure if this is inevitable as long as I'm using erlang or > not. > > > -- > Park, Sungjin > > ------------------------------------------------------------------------------------------------------------------- > Peculiar travel suggestions are dancing lessons from god. > -- The Books of Bokonon > > ------------------------------------------------------------------------------------------------------------------- > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Thu Feb 14 16:25:30 2013 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 14 Feb 2013 16:25:30 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: Hello! If you start erlang with an async thread (erl +A 1) you will see radically different behaviour. For me the time it takes to use read_line drops from 58s to 2s. When doing file i/o you should always have some async threads to help you do the work, this is why in R16B we changed the default from 0 to 10 async threads. Lukas On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil wrote: > Hello, > I know it was been already discussed here in list and it is also recurring > topic for at least five years. But anyway I have been bitten by it again > and > also found pretty pathological case. I have 30MB text file and it has a few > near to 1MB lines there. (I can provide file with same line lengths if > somebody interested.) What I have been observed is that reading this file > using raw file:read_line/1 takes 51s! For comparison I have tried some > different approaches and what I got (line_read_test:read_std/1 is using > file:read_line/1): > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > {51028105,2408} > 2> timer:tc(line_read_test,read,["test.txt"]). > {226220,2408} > 3> timer:tc(line_read_test,read_port,["test.txt"]). > {139388,2408} > > $time perl -nE'$i++}{say $i' test.txt > 2408 > > real 0m0.053s > user 0m0.044s > sys 0m0.008s > > $ time wc -l test.txt > 2408 test.txt > > real 0m0.013s > user 0m0.004s > sys 0m0.008s > > $ time ./a.out test.txt > 2408 > > real 0m0.020s > user 0m0.012s > sys 0m0.008s > > It means erlang should be at least 225 times faster (line_read_test:read/1 > which has flow control). Erlang can be 350 times faster > (line_read_test:read_port/1 without flow control). Another high level > language (perl) is almost thousand times faster. Special C program is > almost > four thousands times faster and old good glibc is two and half thousands > times faster. Come on guys it is not even fun when simple (and wrong) > erlang > wrapper around standard module is more than two order of magnitude faster. > From mine experience when there is something two orders of magnitude slower > it tells me there is something damn wrong. I have looked into efile_drv.c > and > ti is unfortunately far beyond mine C skill but if simple buffering and > binary:match/2 can outperform it 225 times there has to be something rotten > in this code. > > I have also experimented with read_ahead option in file:open and changing > to > less value makes thing worse. > > Just to make grasp how bad it is, in same time I'm able sort 150 million > 64bit values (1.2GB of data) three times (one CPU core same HW). It is not > in > flow control, mine simple wrapper does flow control too. It can't make > current code less intrusive, if it consumes 100% CPU for 51s instead of > 226ms > then it will definitely affect whole server. It is not in concurrent > access, > mine code allows concurrent access too. Admitting there is something broken > is first step to fixing it. I hope I helped at least in this way. > > With best regards > Hynek Vychodil > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Thu Feb 14 17:16:45 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 14 Feb 2013 17:16:45 +0100 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: References: <90893.1360837850@snookles.snookles.com> <511CC1E1.5010109@gmail.com> Message-ID: <511D0DED.8@erix.ericsson.se> Max Lapshin wrote: > Can enif_consume_timeslice help to mark nif-blocked pthread still active? > Creating custom threads is one way to do lengthy native work while being nice to the VM. Another way is to divide your work into smaller pieces and do repeated calls from Erlang to your NIF until the work is done. This is where enif_consume_timeslice (new in R16B) can be useful. You tell the runtime system an estimation of how much of the scheduling timeslice (about 1ms) you have consumed and enif_consume_timeslice will return back if your timeslice is exhausted or not. Note that this is still co-operative scheduling. enif_consume_timeslice does NOT do any preemptive scheduling itself, you need to voluntarily return from your NIF in order to yield the scheduler thread to do other work. For drivers, erl_drv_consume_timeslice will also be available in R16B. /Sverker, Erlang/OTP, Ericsson From lukas@REDACTED Thu Feb 14 17:25:44 2013 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 14 Feb 2013 17:25:44 +0100 Subject: [erlang-questions] OTP R16A has been released In-Reply-To: <510AC5D4.7040500@ernovation.com> References: <510AC5D4.7040500@ernovation.com> Message-ID: Hello! I've updated config.sub and config.guess to the latest version. They should be visible on github tomorrow and will be part of the R16B release. Do you have to change many things within the vm to get it to run on android? If the changes are not too big and are #ifdef'ed nicely it would be neat to have them in the official release. Lukas On Thu, Jan 31, 2013 at 8:28 PM, Erik Reitsma wrote: > Hi Kenneth, > > Could you update the config.sub and config.guess to include > arm-linux-androideabi, unless there is a reason to keep the 2007 version in > favor of the one from 2012-12-30? The fewer files I need to update to > cross-compile for Android, the better! > > Regards, > *Erik. > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy@REDACTED Thu Feb 14 17:31:22 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Thu, 14 Feb 2013 08:31:22 -0800 Subject: [erlang-questions] erlang:system_info(port_limit) broken? In-Reply-To: References: Message-ID: Ah ok my apologies then. On Thu, Feb 14, 2013 at 2:35 AM, Rickard Green wrote: > The 'port_limit' argument was introduced in erts-5.10/OTP-R16A (as well > as 'port_count'). That is, the badarg is expected in R15. > > Regards, > Rickard Green, Erlang/OTP, Erlang/OTP > > On Feb 14, 2013, at 6:29 AM, Jeremy Ong wrote: > > > Sorry, neglected to mention that this is using R15B03-1. I've verified > that this seems to not be working on amd64 hardware and intel i5 hardware > in both linux and osx. > > > > > > On Wed, Feb 13, 2013 at 9:28 PM, Jeremy Ong > wrote: > > This info argument is defined in > www.erlang.org/doc/man/erlang.html#system_info_port_limit > > > > Steps to reproduce: > > > > 1. Invoke an erlang shell (erl) > > 2. erlang:system_info(port_limit). > > > > Returns > > > > ** exception error: bad argument > > in function erlang:system_info/1 > > called as erlang:system_info(port_limit) > > > > Any ideas? > > > > Cheers, > > Jeremy > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g@REDACTED Thu Feb 14 18:21:05 2013 From: g@REDACTED (Garrett Smith) Date: Thu, 14 Feb 2013 11:21:05 -0600 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: <90893.1360837850@snookles.snookles.com> References: <90893.1360837850@snookles.snookles.com> Message-ID: On Thu, Feb 14, 2013 at 4:30 AM, Scott Lystig Fritchie wrote: > I'm starting a new'ish thread to mention a bit of experience that Basho > has had with NIFs in Riak. > > Garrett Smith wrote: > >>> And the second question, Is there any good argument to use NIF >>> instead of creating a connected process for a port. > > gs> The NIF interface is appropriate for defining simple functions in C. > gs> There are lots of 3rd party libraries where NIFs are used to plugin > gs> in long running, multi-threaded facilities, but this seems misguided > gs> to me. > > "Simple functions in C" is a tricky matter ... and it has gotten tricker > with the Erlang/OTP releases R15 and R16. > > In R14 and earlier, it wasn't necessarily a horrible thing if you had C > code (or C++ or Fortran or ...) that executed in NIF context for half a > second or more. If your NIF was executing for that long, you knew that > you were interfering with the Erlang scheduler Pthread that was > executing your NIF's C/C++/Fortran/whatever code. That can cause some > weird delays in executing other Erlang processes, but for some apps, > that's OK. > > However, with R15, the internal guts of the Erlang process scheduler > Pthreads has changed. Now, if you have a NIF that executes for even a > few milliseconds, the scheduler algorithm can get confused. Instead of > blocking an Erlang scheduler Pthread, you both block that Pthread *and* > you might cause some other scheduler Pthreads to decide incorrectly to > go to sleep (because there aren't enough runnable Erlang processes to > bother trying to schedule). Your 8/16/24 CPU core box can find itself > down to only 3 or 2 active Erlang scheduler Pthreads when there really > is more than 2-3 cores of work waiting. > > So, suddenly your "simple functions in C" are now "simple functions in C > that must finish execution in about 1 millisecond or less". If your C > code might take longer than that, then you must use some kind of thread > pool to transfer the long-running work away from the Erlang scheduler > Pthread. Not simple at all, alas. Thanks for highlighting this Scott. Sean Cribbs went into some of these details last night at the Chicago Riak meetup. I imagine this has serious implications for the 0MQ bindings, which are NIF implemented. I'm currently running everything under R14, so am apparently insulated, but this overall sounds quite bad. Have you seen this behavior in port drivers? Garrett From max.lapshin@REDACTED Thu Feb 14 19:08:00 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 14 Feb 2013 21:08:00 +0300 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: References: <90893.1360837850@snookles.snookles.com> Message-ID: How do you write receivers in nif? -------------- next part -------------- An HTML attachment was scrubbed... URL: From emmiller@REDACTED Thu Feb 14 20:03:56 2013 From: emmiller@REDACTED (Evan Miller) Date: Thu, 14 Feb 2013 13:03:56 -0600 Subject: [erlang-questions] Did Erlang's grammar change in R16A? Message-ID: As I understood it, dots are legal in unquoted atom names. This appears not to be the case in R16A. Is this intentional? Erlang R15B03 (erts-5.9.3.1) [source] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.3.1 (abort with ^G) 1> a.b. 'a.b' 2> Erlang R16A (erts-5.10) [source] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] Eshell V5.10 (abort with ^G) 1> a.b. * 1: syntax error before: '.' 1> Kai Janson noticed this because the "mongodb" package fails to compile under R16A due to a syntax error on an atom with a dot in it. https://github.com/mongodb/mongodb-erlang I couldn't find any mention of changing the grammar in the README. Is this a bug? Evan From carlsson.richard@REDACTED Thu Feb 14 20:08:06 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Thu, 14 Feb 2013 20:08:06 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: Message-ID: <511D3616.2070303@gmail.com> On 2013-02-14 20:03, Evan Miller wrote: > As I understood it, dots are legal in unquoted atom names. This > appears not to be the case in R16A. Is this intentional? > > > Erlang R15B03 (erts-5.9.3.1) [source] [smp:8:8] [async-threads:0] > [hipe] [kernel-poll:false] > > Eshell V5.9.3.1 (abort with ^G) > 1> a.b. > 'a.b' > 2> > > > Erlang R16A (erts-5.10) [source] [smp:8:8] [async-threads:10] [hipe] > [kernel-poll:false] > > Eshell V5.10 (abort with ^G) > 1> a.b. > * 1: syntax error before: '.' > 1> > > > Kai Janson noticed this because the "mongodb" package fails to compile > under R16A due to a syntax error on an atom with a dot in it. > > https://github.com/mongodb/mongodb-erlang > > I couldn't find any mention of changing the grammar in the README. Is > this a bug? The automatic concatenation of atoms separated by dots was originally introduced with the experimental "packages" system, and was probably lost again when this feature was dropped from the codebase. /Richard From janos.n.hary@REDACTED Thu Feb 14 20:11:20 2013 From: janos.n.hary@REDACTED (Janos Hary) Date: Thu, 14 Feb 2013 20:11:20 +0100 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: References: <90893.1360837850@snookles.snookles.com> Message-ID: <007e01ce0ae7$12eabd60$38c03820$@gmail.com> I'm rewriting a large C++ project in Erlang. Some third party libraries have to remain in C/C++. When I chose the new implementation language for the project a major factor was the effort needed to interface it with legacy code. I evaluated many other languages and have to say Erlang is the winner is this field (as well as in many others). The libraries I'm using do REALY long running tasks (robotic control ~30sec, optical media burning ~30min). When I used them in a C++ GUI program I had to put them on background threads. I also needed to solve messaging between background and GUI thread to present progress messages. Interfacing a C++ code with another C++ code is just as tedious, if not more, then writing a NIF driver. I don't expect more support from my programming environment than Erlang provides. Some of our (programmers) tasks have certain complexity, and they cannot be solved always with the simplest tools. That's why we are well paid professionals :) Crashing the VM from a NIF also concerns me, and I think to run this functionality in a different node. So I would keep the simplicity of the NIF code and use Erlang's built in distributed features to run them in a separate VM. Janos -----Original Message----- From: erlang-questions-bounces@REDACTED [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Garrett Smith Sent: Thursday, February 14, 2013 6:21 PM To: Scott Lystig Fritchie Cc: Gokul Evuri; erlang-questions@REDACTED Subject: Re: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? On Thu, Feb 14, 2013 at 4:30 AM, Scott Lystig Fritchie wrote: > I'm starting a new'ish thread to mention a bit of experience that Basho > has had with NIFs in Riak. > > Garrett Smith wrote: > >>> And the second question, Is there any good argument to use NIF >>> instead of creating a connected process for a port. > > gs> The NIF interface is appropriate for defining simple functions in C. > gs> There are lots of 3rd party libraries where NIFs are used to plugin > gs> in long running, multi-threaded facilities, but this seems misguided > gs> to me. > > "Simple functions in C" is a tricky matter ... and it has gotten tricker > with the Erlang/OTP releases R15 and R16. > > In R14 and earlier, it wasn't necessarily a horrible thing if you had C > code (or C++ or Fortran or ...) that executed in NIF context for half a > second or more. If your NIF was executing for that long, you knew that > you were interfering with the Erlang scheduler Pthread that was > executing your NIF's C/C++/Fortran/whatever code. That can cause some > weird delays in executing other Erlang processes, but for some apps, > that's OK. > > However, with R15, the internal guts of the Erlang process scheduler > Pthreads has changed. Now, if you have a NIF that executes for even a > few milliseconds, the scheduler algorithm can get confused. Instead of > blocking an Erlang scheduler Pthread, you both block that Pthread *and* > you might cause some other scheduler Pthreads to decide incorrectly to > go to sleep (because there aren't enough runnable Erlang processes to > bother trying to schedule). Your 8/16/24 CPU core box can find itself > down to only 3 or 2 active Erlang scheduler Pthreads when there really > is more than 2-3 cores of work waiting. > > So, suddenly your "simple functions in C" are now "simple functions in C > that must finish execution in about 1 millisecond or less". If your C > code might take longer than that, then you must use some kind of thread > pool to transfer the long-running work away from the Erlang scheduler > Pthread. Not simple at all, alas. Thanks for highlighting this Scott. Sean Cribbs went into some of these details last night at the Chicago Riak meetup. I imagine this has serious implications for the 0MQ bindings, which are NIF implemented. I'm currently running everything under R14, so am apparently insulated, but this overall sounds quite bad. Have you seen this behavior in port drivers? Garrett _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions From mjtruog@REDACTED Thu Feb 14 20:28:36 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 14 Feb 2013 11:28:36 -0800 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: References: <90893.1360837850@snookles.snookles.com> Message-ID: <511D3AE4.2000700@gmail.com> On 02/14/2013 09:21 AM, Garrett Smith wrote: > On Thu, Feb 14, 2013 at 4:30 AM, Scott Lystig Fritchie > wrote: >> I'm starting a new'ish thread to mention a bit of experience that Basho >> has had with NIFs in Riak. >> >> Garrett Smith wrote: >> >>>> And the second question, Is there any good argument to use NIF >>>> instead of creating a connected process for a port. >> gs> The NIF interface is appropriate for defining simple functions in C. >> gs> There are lots of 3rd party libraries where NIFs are used to plugin >> gs> in long running, multi-threaded facilities, but this seems misguided >> gs> to me. >> >> "Simple functions in C" is a tricky matter ... and it has gotten tricker >> with the Erlang/OTP releases R15 and R16. >> >> In R14 and earlier, it wasn't necessarily a horrible thing if you had C >> code (or C++ or Fortran or ...) that executed in NIF context for half a >> second or more. If your NIF was executing for that long, you knew that >> you were interfering with the Erlang scheduler Pthread that was >> executing your NIF's C/C++/Fortran/whatever code. That can cause some >> weird delays in executing other Erlang processes, but for some apps, >> that's OK. >> >> However, with R15, the internal guts of the Erlang process scheduler >> Pthreads has changed. Now, if you have a NIF that executes for even a >> few milliseconds, the scheduler algorithm can get confused. Instead of >> blocking an Erlang scheduler Pthread, you both block that Pthread *and* >> you might cause some other scheduler Pthreads to decide incorrectly to >> go to sleep (because there aren't enough runnable Erlang processes to >> bother trying to schedule). Your 8/16/24 CPU core box can find itself >> down to only 3 or 2 active Erlang scheduler Pthreads when there really >> is more than 2-3 cores of work waiting. >> >> So, suddenly your "simple functions in C" are now "simple functions in C >> that must finish execution in about 1 millisecond or less". If your C >> code might take longer than that, then you must use some kind of thread >> pool to transfer the long-running work away from the Erlang scheduler >> Pthread. Not simple at all, alas. > Thanks for highlighting this Scott. > > Sean Cribbs went into some of these details last night at the Chicago > Riak meetup. > > I imagine this has serious implications for the 0MQ bindings, which > are NIF implemented. I'm currently running everything under R14, so am > apparently insulated, but this overall sounds quite bad. > > Have you seen this behavior in port drivers? > > Garrett > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > The erlzmq2 NIF uses a separate thread for the receive and enif_send is used to provide the incoming data, with locks inbetween. So, I don't see why the impact would be serious, just since a separate thread is used, while the NIF functions do not block in the C code. From mjtruog@REDACTED Thu Feb 14 20:39:02 2013 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 14 Feb 2013 11:39:02 -0800 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: <689F6548-625C-4447-9B9C-80195FE91E54@erlang.org> References: <90893.1360837850@snookles.snookles.com> <511CC1E1.5010109@gmail.com> <689F6548-625C-4447-9B9C-80195FE91E54@erlang.org> Message-ID: <511D3D56.1040908@gmail.com> On 02/14/2013 03:52 AM, Rickard Green wrote: > On Feb 14, 2013, at 11:52 AM, Michael Truog wrote: > >> On 02/14/2013 02:30 AM, Scott Lystig Fritchie wrote: >>> I'm starting a new'ish thread to mention a bit of experience that Basho >>> has had with NIFs in Riak. >>> >>> Garrett Smith wrote: >>> >>>>> And the second question, Is there any good argument to use NIF >>>>> instead of creating a connected process for a port. >>> gs> The NIF interface is appropriate for defining simple functions in C. >>> gs> There are lots of 3rd party libraries where NIFs are used to plugin >>> gs> in long running, multi-threaded facilities, but this seems misguided >>> gs> to me. >>> >>> "Simple functions in C" is a tricky matter ... and it has gotten tricker >>> with the Erlang/OTP releases R15 and R16. >>> >>> In R14 and earlier, it wasn't necessarily a horrible thing if you had C >>> code (or C++ or Fortran or ...) that executed in NIF context for half a >>> second or more. If your NIF was executing for that long, you knew that >>> you were interfering with the Erlang scheduler Pthread that was >>> executing your NIF's C/C++/Fortran/whatever code. That can cause some >>> weird delays in executing other Erlang processes, but for some apps, >>> that's OK. >>> >>> However, with R15, the internal guts of the Erlang process scheduler >>> Pthreads has changed. Now, if you have a NIF that executes for even a >>> few milliseconds, the scheduler algorithm can get confused. Instead of >>> blocking an Erlang scheduler Pthread, you both block that Pthread *and* >>> you might cause some other scheduler Pthreads to decide incorrectly to >>> go to sleep (because there aren't enough runnable Erlang processes to >>> bother trying to schedule). Your 8/16/24 CPU core box can find itself >>> down to only 3 or 2 active Erlang scheduler Pthreads when there really >>> is more than 2-3 cores of work waiting. >>> >>> So, suddenly your "simple functions in C" are now "simple functions in C >>> that must finish execution in about 1 millisecond or less". If your C >>> code might take longer than that, then you must use some kind of thread >>> pool to transfer the long-running work away from the Erlang scheduler >>> Pthread. Not simple at all, alas. >>> > Native code (drivers and NIFs) have always been expected to execute for very short periods of time. The major difference is that it is more clearly documented today. > > The number of problems you might run into if you run native code that do not behave well has increased though. This is, however, not new to R15. This has been the case since R11, due to optimizations of the smp runtime system. One such optimization was multiple run-queues that was introduced in R13. Regarding scheduling, R12 to R13 is where the major difference is. > > In some cases we could try to fix problems caused by native code that do not behave well. This would however very often cause a performance penalty that always have to be paid, and it would also prevent us from implementing a lot of optimizations. In my opinion this would just be plain wrong. The VM has never been intended for scheduling of arbitrary native code. A NIF or a driver is supposed to be aware of the VM and help it, not break it. > >>> -Scott >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> These problems are what NIF native processes will solve, right? > No, but "dirty schedulers" were supposed to ease implementation of things like this. Note that already today you got all the primitives you need, as for example threads for NIFs and drivers. > >> The only other alternative would be to use the async thread pool within a port driver, which may not help the schedulers and is obsoleted by native processes (not to mention the job queue per thread situation which can block on long jobs). > > Regards, > Rickard Green, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > I understand we have the thread primitives already for NIFs and drivers. It just bothers me that when you create your own thread pool, you put the burden on the Operating System kernel scheduler, causing CPU contention. It seems like the Erlang VM would have more insight as to how to schedule the NIFs, even if they are misbehaving, as long as reductions are bumped properly, or execution time is used as a way of extrapolating a reduction count that makes sense to the VM. One way of simplifying this, might be to have a yield function which is called frequently for blocking NIF functions, such that the yield function handles reporting a reduction count which impacts scheduling. You know more about the problems than I do, I am just voicing concern with having to depend on the kernel scheduler. Thanks, Michael From ingham.k@REDACTED Thu Feb 14 21:49:18 2013 From: ingham.k@REDACTED (Igor Karymov) Date: Fri, 15 Feb 2013 00:49:18 +0400 Subject: [erlang-questions] [ANN] erlang riak connection pool by Unison Technologies Message-ID: https://github.com/unisontech/uriak_pool Main features Graceful handling of riak nodes starts and shutdowns simultaneous work with different riak clusters parse_transform generation for interfaces from the client code simple configuring using erlang configuration files reconfiguration in runtime tested in production at Unison Technologies -------------- next part -------------- An HTML attachment was scrubbed... URL: From emmiller@REDACTED Thu Feb 14 22:38:13 2013 From: emmiller@REDACTED (Evan Miller) Date: Thu, 14 Feb 2013 15:38:13 -0600 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: <511D3616.2070303@gmail.com> References: <511D3616.2070303@gmail.com> Message-ID: On Thu, Feb 14, 2013 at 1:08 PM, Richard Carlsson wrote: > On 2013-02-14 20:03, Evan Miller wrote: >> >> As I understood it, dots are legal in unquoted atom names. This >> appears not to be the case in R16A. Is this intentional? >> >> >> Erlang R15B03 (erts-5.9.3.1) [source] [smp:8:8] [async-threads:0] >> [hipe] [kernel-poll:false] >> >> Eshell V5.9.3.1 (abort with ^G) >> 1> a.b. >> 'a.b' >> 2> >> >> >> Erlang R16A (erts-5.10) [source] [smp:8:8] [async-threads:10] [hipe] >> [kernel-poll:false] >> >> Eshell V5.10 (abort with ^G) >> 1> a.b. >> * 1: syntax error before: '.' >> 1> >> >> >> Kai Janson noticed this because the "mongodb" package fails to compile >> under R16A due to a syntax error on an atom with a dot in it. >> >> https://github.com/mongodb/mongodb-erlang >> >> I couldn't find any mention of changing the grammar in the README. Is >> this a bug? > > > The automatic concatenation of atoms separated by dots was originally > introduced with the experimental "packages" system, and was probably lost > again when this feature was dropped from the codebase. > > /Richard > Ah, I see. This is alluded to in the Bible, Chapter 2, Section 8, Footnote 7: "You might find that a period (.) can also be use in atoms--this is an unsupported extension to Erlang." From erlang@REDACTED Thu Feb 14 22:40:41 2013 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 14 Feb 2013 22:40:41 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: I wrote a little program to test this -module(test1). -compile(export_all). test1(N) -> timer:tc(?MODULE, make, [N]). test2() -> timer:tc(?MODULE, read, []). make(N) -> Line = lists:seq(30,59) ++ [$\n], file:write_file("big.tmp", lists:duplicate(N,Line)). read() -> {ok,B} = file:read_file("big.tmp"), L = split(B, [], []), length(L). split(<<10,B/binary>>, L1, L2) -> split(B, [], [list_to_binary(lists:reverse(L1))|L2]); split(<>, L1, L2) -> split(T, [H|L1], L2); split(<<>>, L1, L2) -> lists:reverse([list_to_binary(lists:reverse(L1))|L2]). The timing I got were that Erlang was 9 times slower than perl (or wc) which is more or less what I expected. If I wanted to speed this up I'd write a NIF to split the binary at the first newline character. I actually always use file:read_file(F) for everything - since getting the entire file in at one go always seems a good idea and I have small files (compared to my RAM) - I'd use file:pread for files that are too big for memory and do random access read. Reading the entire file seems a good idea for files less than 100MB since I have 4GB of memory. The OS seems to do a better job of caching entire files than I could ever do so I don't worry about re-reading them ... I have no idea why you see a factor of 250 - is this a memory problem. How much memory have you got? Does your program scale linearly with the file size - or does something go suddenly wrong as you increase the size of the file? Cheers /Joe On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil wrote: > Hello, > I know it was been already discussed here in list and it is also recurring > topic for at least five years. But anyway I have been bitten by it again > and > also found pretty pathological case. I have 30MB text file and it has a few > near to 1MB lines there. (I can provide file with same line lengths if > somebody interested.) What I have been observed is that reading this file > using raw file:read_line/1 takes 51s! For comparison I have tried some > different approaches and what I got (line_read_test:read_std/1 is using > file:read_line/1): > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > {51028105,2408} > 2> timer:tc(line_read_test,read,["test.txt"]). > {226220,2408} > 3> timer:tc(line_read_test,read_port,["test.txt"]). > {139388,2408} > > $time perl -nE'$i++}{say $i' test.txt > 2408 > > real 0m0.053s > user 0m0.044s > sys 0m0.008s > > $ time wc -l test.txt > 2408 test.txt > > real 0m0.013s > user 0m0.004s > sys 0m0.008s > > $ time ./a.out test.txt > 2408 > > real 0m0.020s > user 0m0.012s > sys 0m0.008s > > It means erlang should be at least 225 times faster (line_read_test:read/1 > which has flow control). Erlang can be 350 times faster > (line_read_test:read_port/1 without flow control). Another high level > language (perl) is almost thousand times faster. Special C program is > almost > four thousands times faster and old good glibc is two and half thousands > times faster. Come on guys it is not even fun when simple (and wrong) > erlang > wrapper around standard module is more than two order of magnitude faster. > From mine experience when there is something two orders of magnitude slower > it tells me there is something damn wrong. I have looked into efile_drv.c > and > ti is unfortunately far beyond mine C skill but if simple buffering and > binary:match/2 can outperform it 225 times there has to be something rotten > in this code. > > I have also experimented with read_ahead option in file:open and changing > to > less value makes thing worse. > > Just to make grasp how bad it is, in same time I'm able sort 150 million > 64bit values (1.2GB of data) three times (one CPU core same HW). It is not > in > flow control, mine simple wrapper does flow control too. It can't make > current code less intrusive, if it consumes 100% CPU for 51s instead of > 226ms > then it will definitely affect whole server. It is not in concurrent > access, > mine code allows concurrent access too. Admitting there is something broken > is first step to fixing it. I hope I helped at least in this way. > > With best regards > Hynek Vychodil > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Feb 14 23:12:59 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 15 Feb 2013 11:12:59 +1300 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> Message-ID: I had always assumed that dots, like at-signs, were allowed in Erlang atoms so that one could write an Erlang node name like erl_node@REDACTED as an atom. It was never clearly stated anywhere what the rules were, and the way that .x = x was particularly odd. From wallentin.dahlberg@REDACTED Thu Feb 14 23:18:44 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Thu, 14 Feb 2013 23:18:44 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> Message-ID: 2013/2/14 Evan Miller > On Thu, Feb 14, 2013 at 1:08 PM, Richard Carlsson > wrote: > > On 2013-02-14 20:03, Evan Miller wrote: > >> > >> As I understood it, dots are legal in unquoted atom names. This > >> appears not to be the case in R16A. Is this intentional? > >> > >> > >> Erlang R15B03 (erts-5.9.3.1) [source] [smp:8:8] [async-threads:0] > >> [hipe] [kernel-poll:false] > >> > >> Eshell V5.9.3.1 (abort with ^G) > >> 1> a.b. > >> 'a.b' > >> 2> > Btw, note that a.b actually becomes quoted in output. > >> > >> > >> Erlang R16A (erts-5.10) [source] [smp:8:8] [async-threads:10] [hipe] > >> [kernel-poll:false] > >> > >> Eshell V5.10 (abort with ^G) > >> 1> a.b. > >> * 1: syntax error before: '.' > >> 1> > >> > >> > >> Kai Janson noticed this because the "mongodb" package fails to compile > >> under R16A due to a syntax error on an atom with a dot in it. > >> > >> https://github.com/mongodb/mongodb-erlang > >> > >> I couldn't find any mention of changing the grammar in the README. Is > >> this a bug? > > > > > > The automatic concatenation of atoms separated by dots was originally > > introduced with the experimental "packages" system, and was probably lost > > again when this feature was dropped from the codebase. > > > > /Richard > > > > > Ah, I see. This is alluded to in the Bible, Chapter 2, Section 8, Footnote > 7: > > "You might find that a period (.) can also be use in atoms--this is an > unsupported extension to Erlang." > You don't miss an opportunity to be entertaining. =) It's an unfortunate consequence of removing packages. Dots were part of package paths and I guess it didn't need quoting out of convenience. Also I think it was a concatenated on a completely different level in the compiler. The mistake wasn't to remove unquoted dots but to never fix the *bug* with quoting in the first place. Anyhow, it's gone now I hope it will never see the light of day again. On a side note, I also have an issue with: * Creating atoms in runtime. It should only have been allowed in code and never by list_to_atom/1 or binary_to_atom/1,2 (binary_to_term would still be a thing though) * Using atoms as filenames and node-names, i.e. using atoms as strings. Atoms are atoms, NOT strings. If using atoms as strings were an performance issue, strings (lists) should have been fixed to remedy this. 2 cents poorer, // Bj?rn-Egil _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Feb 14 23:23:59 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Thu, 14 Feb 2013 23:23:59 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> Message-ID: <511D63FF.3010409@ninenines.eu> On 02/14/2013 11:18 PM, Bj?rn-Egil Dahlberg wrote: > * Creating atoms in runtime. It should only have been allowed in code > and never by list_to_atom/1 or binary_to_atom/1,2 (binary_to_term would > still be a thing though) Anthony Ramine has a "split the atoms" implementation of ROK's EEP in progress which allows to dynamically create garbage-collected atoms, fixing all issues related to creating them at runtime. You might want to take a look or help getting this in quicker: https://github.com/nox/otp/tree/eep20 -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From n.oxyde@REDACTED Thu Feb 14 23:41:07 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Thu, 14 Feb 2013 23:41:07 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: <511D63FF.3010409@ninenines.eu> References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: <11CF49DC-B02E-423E-8503-DEA3BC0B7317@gmail.com> It needs a lot of refactoring now that UTF-8 atoms are out, but I'll probably do it that when I have some time. Regards, -- Anthony Ramine Le 14 f?vr. 2013 ? 23:23, Lo?c Hoguin a ?crit : > On 02/14/2013 11:18 PM, Bj?rn-Egil Dahlberg wrote: >> * Creating atoms in runtime. It should only have been allowed in code >> and never by list_to_atom/1 or binary_to_atom/1,2 (binary_to_term would >> still be a thing though) > > Anthony Ramine has a "split the atoms" implementation of ROK's EEP in progress which allows to dynamically create garbage-collected atoms, fixing all issues related to creating them at runtime. You might want to take a look or help getting this in quicker: > > https://github.com/nox/otp/tree/eep20 > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From wallentin.dahlberg@REDACTED Thu Feb 14 23:42:49 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Thu, 14 Feb 2013 23:42:49 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: <511D63FF.3010409@ninenines.eu> References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: 2013/2/14 Lo?c Hoguin > On 02/14/2013 11:18 PM, Bj?rn-Egil Dahlberg wrote: > >> * Creating atoms in runtime. It should only have been allowed in code >> and never by list_to_atom/1 or binary_to_atom/1,2 (binary_to_term would >> still be a thing though) >> > > Anthony Ramine has a "split the atoms" implementation of ROK's EEP in > progress which allows to dynamically create garbage-collected atoms, fixing > all issues related to creating them at runtime. You might want to take a > look or help getting this in quicker: > My point was actually not to have atoms as strings (and prolog filenames). If I'm reading the EEP correctly: a local atom would actually require a larger heap space than an equivalent heap binary .. which is a feat all by itself. I gather that could be remedied though. Haven't looked at the code. I agree that atom gc is needed but it shouldn't be an excuse for using dynamic atoms instead of binaries. Besides, if locals atoms are larger than binaries why would you use atoms. (I might be wrong about the size though .. didn't look that hard). // Bj?rn-Egil > > https://github.com/nox/otp/**tree/eep20 > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Thu Feb 14 23:44:50 2013 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Thu, 14 Feb 2013 23:44:50 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: <511D68E2.7010807@ninenines.eu> On 02/14/2013 11:42 PM, Bj?rn-Egil Dahlberg wrote: > > > 2013/2/14 Lo?c Hoguin > > > On 02/14/2013 11:18 PM, Bj?rn-Egil Dahlberg wrote: > > * Creating atoms in runtime. It should only have been allowed in > code > and never by list_to_atom/1 or binary_to_atom/1,2 > (binary_to_term would > still be a thing though) > > > Anthony Ramine has a "split the atoms" implementation of ROK's EEP > in progress which allows to dynamically create garbage-collected > atoms, fixing all issues related to creating them at runtime. You > might want to take a look or help getting this in quicker: > > > My point was actually not to have atoms as strings (and prolog filenames). > > If I'm reading the EEP correctly: a local atom would actually require a > larger heap space than an equivalent heap binary .. which is a feat all > by itself. > > I gather that could be remedied though. Haven't looked at the code. > > I agree that atom gc is needed but it shouldn't be an excuse for using > dynamic atoms instead of binaries. Besides, if locals atoms are larger > than binaries why would you use atoms. (I might be wrong about the size > though .. didn't look that hard). Some things still only take atoms, like register/2. That's just one example out of many. :) -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From wallentin.dahlberg@REDACTED Thu Feb 14 23:49:19 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Thu, 14 Feb 2013 23:49:19 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: <511D68E2.7010807@ninenines.eu> References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> <511D68E2.7010807@ninenines.eu> Message-ID: 2013/2/14 Lo?c Hoguin > On 02/14/2013 11:42 PM, Bj?rn-Egil Dahlberg wrote: > >> >> >> 2013/2/14 Lo?c Hoguin > >> >> >> On 02/14/2013 11:18 PM, Bj?rn-Egil Dahlberg wrote: >> >> * Creating atoms in runtime. It should only have been allowed in >> code >> and never by list_to_atom/1 or binary_to_atom/1,2 >> (binary_to_term would >> still be a thing though) >> >> >> Anthony Ramine has a "split the atoms" implementation of ROK's EEP >> in progress which allows to dynamically create garbage-collected >> atoms, fixing all issues related to creating them at runtime. You >> might want to take a look or help getting this in quicker: >> >> >> My point was actually not to have atoms as strings (and prolog filenames). >> >> If I'm reading the EEP correctly: a local atom would actually require a >> larger heap space than an equivalent heap binary .. which is a feat all >> by itself. >> >> I gather that could be remedied though. Haven't looked at the code. >> >> I agree that atom gc is needed but it shouldn't be an excuse for using >> dynamic atoms instead of binaries. Besides, if locals atoms are larger >> than binaries why would you use atoms. (I might be wrong about the size >> though .. didn't look that hard). >> > > Some things still only take atoms, like register/2. That's just one > example out of many. :) Yeah .. kinda stuck with dynamically created atoms now. But using atoms as strings .. thats just a poor life choice. // Bj?rn-Egil .. needs .. sleep .. now > > > -- > Lo?c Hoguin > > Erlang Cowboy > Nine Nines > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.oxyde@REDACTED Fri Feb 15 00:05:25 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Fri, 15 Feb 2013 00:05:25 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: Hi Bj?rn, While a local atom may take more space than a heap binary, the amortized cost of comparing a local atom to another local atom with a different name is O(1) because their hash aren't the same. Furthermore, two different local atoms with the same name will become the very same term on the heap when gc happens after a comparison between the two of them. These two things won't ever happen with heap binaries. On an unrelated subject, I hope some at least some of the OTP team do look hard at EEPs. Regards, -- Anthony Ramine Le 14 f?vr. 2013 ? 23:42, Bj?rn-Egil Dahlberg a ?crit : > (I might be wrong about the size though .. didn't look that hard). From n.oxyde@REDACTED Fri Feb 15 00:06:33 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Fri, 15 Feb 2013 00:06:33 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: <511D68E2.7010807@ninenines.eu> References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> <511D68E2.7010807@ninenines.eu> Message-ID: Though the EEP doesn't cover these cases. If a local atom is needed by the global funs table or the process table, it will enter the global atom table. -- Anthony Ramine Le 14 f?vr. 2013 ? 23:44, Lo?c Hoguin a ?crit : > Some things still only take atoms, like register/2. That's just one example out of many. :) From ok@REDACTED Fri Feb 15 00:14:42 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 15 Feb 2013 12:14:42 +1300 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: <4FE32BA0-31DC-4476-B14E-FD41185138B5@cs.otago.ac.nz> On 15/02/2013, at 11:42 AM, Bj?rn-Egil Dahlberg wrote: > If I'm reading [EEP 20] correctly: a local atom would actually require a larger heap space than an equivalent heap binary .. which is a feat all by itself. A 'local atom' uses the same space *per reference* as existing atoms do. On the heap of the process that created it, it holds the bytes of the name plus a 3-word header (32-bit machine design). The minimum imaginable overhead would be a 1-word header, and that would sacrifice some of the efficiency that the 3-word header allows. Now, I'm looking at an elderly (R12B-5) set of sources (on an elderly machine), and I see that - a ProcBin requires 6 words on the process's heap - a Binary requires a 3-word header plus the bytes, or on a 32-bit machine, a 4-word header plus the bytes. I believe a Binary is the "equivalent heap binary" you are talking about. So no, a local atom would NOT require larger heap space than a binary. Why did you think it would? > I agree that atom gc is needed but it shouldn't be an excuse for using dynamic atoms instead of binaries. On the one hand, it isn't. The EEP is very explicit that the goal is _safety_ rather than _convenience_. However, haven't we had enough complaints about the confusion between lists of integers and strings of text? Don't you think that a clean distinction between a sequence of bits in some unknown encoding and a chunk of text in a known encoding is useful? > Besides, if locals atoms are larger than binaries why would you use atoms. (I might be wrong about the size though .. didn't look that hard). Oh, if people are bigger than houses, we should all use Perl. From rickard@REDACTED Fri Feb 15 00:14:48 2013 From: rickard@REDACTED (Rickard Green) Date: Fri, 15 Feb 2013 00:14:48 +0100 Subject: [erlang-questions] NIF appropriateness, was: Re: Messing with heart. Port and NIF, which one is better? In-Reply-To: <511D3D56.1040908@gmail.com> References: <90893.1360837850@snookles.snookles.com> <511CC1E1.5010109@gmail.com> <689F6548-625C-4447-9B9C-80195FE91E54@erlang.org> <511D3D56.1040908@gmail.com> Message-ID: <4F0E7B44-16FE-431E-B663-47A98D47EFFA@erlang.org> On Feb 14, 2013, at 8:39 PM, Michael Truog wrote: > On 02/14/2013 03:52 AM, Rickard Green wrote: >> On Feb 14, 2013, at 11:52 AM, Michael Truog wrote: >> >>> On 02/14/2013 02:30 AM, Scott Lystig Fritchie wrote: >>>> I'm starting a new'ish thread to mention a bit of experience that Basho >>>> has had with NIFs in Riak. >>>> >>>> Garrett Smith wrote: >>>> >>>>>> And the second question, Is there any good argument to use NIF >>>>>> instead of creating a connected process for a port. >>>> gs> The NIF interface is appropriate for defining simple functions in C. >>>> gs> There are lots of 3rd party libraries where NIFs are used to plugin >>>> gs> in long running, multi-threaded facilities, but this seems misguided >>>> gs> to me. >>>> >>>> "Simple functions in C" is a tricky matter ... and it has gotten tricker >>>> with the Erlang/OTP releases R15 and R16. >>>> >>>> In R14 and earlier, it wasn't necessarily a horrible thing if you had C >>>> code (or C++ or Fortran or ...) that executed in NIF context for half a >>>> second or more. If your NIF was executing for that long, you knew that >>>> you were interfering with the Erlang scheduler Pthread that was >>>> executing your NIF's C/C++/Fortran/whatever code. That can cause some >>>> weird delays in executing other Erlang processes, but for some apps, >>>> that's OK. >>>> >>>> However, with R15, the internal guts of the Erlang process scheduler >>>> Pthreads has changed. Now, if you have a NIF that executes for even a >>>> few milliseconds, the scheduler algorithm can get confused. Instead of >>>> blocking an Erlang scheduler Pthread, you both block that Pthread *and* >>>> you might cause some other scheduler Pthreads to decide incorrectly to >>>> go to sleep (because there aren't enough runnable Erlang processes to >>>> bother trying to schedule). Your 8/16/24 CPU core box can find itself >>>> down to only 3 or 2 active Erlang scheduler Pthreads when there really >>>> is more than 2-3 cores of work waiting. >>>> >>>> So, suddenly your "simple functions in C" are now "simple functions in C >>>> that must finish execution in about 1 millisecond or less". If your C >>>> code might take longer than that, then you must use some kind of thread >>>> pool to transfer the long-running work away from the Erlang scheduler >>>> Pthread. Not simple at all, alas. >>>> >> Native code (drivers and NIFs) have always been expected to execute for very short periods of time. The major difference is that it is more clearly documented today. >> >> The number of problems you might run into if you run native code that do not behave well has increased though. This is, however, not new to R15. This has been the case since R11, due to optimizations of the smp runtime system. One such optimization was multiple run-queues that was introduced in R13. Regarding scheduling, R12 to R13 is where the major difference is. >> >> In some cases we could try to fix problems caused by native code that do not behave well. This would however very often cause a performance penalty that always have to be paid, and it would also prevent us from implementing a lot of optimizations. In my opinion this would just be plain wrong. The VM has never been intended for scheduling of arbitrary native code. A NIF or a driver is supposed to be aware of the VM and help it, not break it. >> >>>> -Scott >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>> These problems are what NIF native processes will solve, right? >> No, but "dirty schedulers" were supposed to ease implementation of things like this. Note that already today you got all the primitives you need, as for example threads for NIFs and drivers. >> >>> The only other alternative would be to use the async thread pool within a port driver, which may not help the schedulers and is obsoleted by native processes (not to mention the job queue per thread situation which can block on long jobs). >> >> Regards, >> Rickard Green, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > I understand we have the thread primitives already for NIFs and drivers. It just bothers me that when you create your own thread pool, you put the burden on the Operating System kernel scheduler, causing CPU contention. It seems like the Erlang VM would have more insight as to how to schedule the NIFs, even if they are misbehaving, as long as reductions are bumped properly, or execution time is used as a way of extrapolating a reduction count that makes sense to the VM. One way of simplifying this, might be to have a yield function which is called frequently for blocking NIF functions, such that the yield function handles reporting a reduction count which impacts scheduling. You know more about the problems than I do, I am just voicing concern with having to depend on the kernel scheduler. > > Thanks, > Michael I don't see any real benefits of this approach. When you really need threading an enif_yield() function would not do you any good. That is, when linking against third-party libraries doing lengthy work (that you aren't able to modify), or when doing blocking I/O. Threading should preferably only be used as a last resort. If you can divide your lengthy work into multiple chunks and do multiple calls this is always preferred and much better. Implementing an enif_yield() that works on all platforms is also not as easy as it may seem. It can of course be implemented, but this is not a prioritized feature. Regards, Rickard Green, Erlang/OTP, Ericsson AB From wallentin.dahlberg@REDACTED Fri Feb 15 01:02:31 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Fri, 15 Feb 2013 01:02:31 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: 2013/2/15 Anthony Ramine > Hi Bj?rn, > > While a local atom may take more space than a heap binary, the > amortized cost of comparing a local atom to another local atom > with a different name is O(1) because their hash aren't the > same. > Ok, how does this perform against linear scan Eterm data array? The tradeoff being one word memory saved on the heap also? Keep in mind that most atoms are pretty short. > Furthermore, two different local atoms with the same name will > become the very same term on the heap when gc happens after > a comparison between the two of them. > You are forcing me to look at the EEP again. .. first thought you were messing with the arity thing meaning .. perhaps i should sleep. putting more stuff in the header .. seems good .. wait a minute .. are you proposing an immediate with a payload? so it's really a thing instead? and if it is a thing should you really be messing with the header meaning? Seems like it's a one pass gc strategy in unifying the atoms .. ok. I need to take a look at the proposal when all my neurons are firing ok. I can't possibly be reading that EEP correctly. These two things won't ever happen with heap binaries. > > On an unrelated subject, I hope some at least some of the OTP > team do look hard at EEPs. > Sure but my sleepy brain does not remember it all =) I gather your worried that your implementing something that will not be accepted. I can't answer that. I can say that a reference implementation for an EEP goes a long way though. I think there are many ways to skin this particular cat and I think it should debated what the most effective atom gc could be. > > Regards, > > -- > Anthony Ramine > > Le 14 f?vr. 2013 ? 23:42, Bj?rn-Egil Dahlberg a ?crit : > > > (I might be wrong about the size though .. didn't look that hard). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vychodil.hynek@REDACTED Fri Feb 15 01:28:24 2013 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Fri, 15 Feb 2013 01:28:24 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: I think it is misunderstanding. There is not problem with erlang io for normal files as you tested. Problem is with little bit unusual files with few long lines as I attached. It causes some pathological behavior in prim_file:read_line/1. It perfectly works for example for another 3GB file with 16 million lines which are all evenly long about 198B. It can process it in nice two minutes which is pretty (29MB/s and 148klines/s) but for another hundred times smaller file with seven thousand less lines it takes half of this time which is terribly wrong (0.5MB/s and 46lines/s). So I'm not pointing that erlang is bad for processing files generally but there is some nasty bug in prim_file:read_line/1 function. On Thu, Feb 14, 2013 at 10:40 PM, Joe Armstrong wrote: > I wrote a little program to test this > > -module(test1). > > -compile(export_all). > > test1(N) -> > timer:tc(?MODULE, make, [N]). > > test2() -> > timer:tc(?MODULE, read, []). > > > make(N) -> > Line = lists:seq(30,59) ++ [$\n], > file:write_file("big.tmp", > lists:duplicate(N,Line)). > > read() -> > {ok,B} = file:read_file("big.tmp"), > L = split(B, [], []), > length(L). > > split(<<10,B/binary>>, L1, L2) -> > split(B, [], [list_to_binary(lists:reverse(L1))|L2]); > split(<>, L1, L2) -> > split(T, [H|L1], L2); > split(<<>>, L1, L2) -> > lists:reverse([list_to_binary(lists:reverse(L1))|L2]). > > The timing I got were that Erlang was 9 times slower than perl (or wc) > which is more or less what I expected. If I wanted to speed this up > I'd write a NIF to split the binary at the first newline character. > > I actually always use file:read_file(F) for everything - since getting the > entire file in at one go always seems a good idea and I have small files > (compared to my RAM) - I'd use file:pread for files that are too big for > memory and do random access read. Reading the entire file seems > a good idea for files less than 100MB since I have 4GB of memory. > > The OS seems to do a better job of caching entire files than I could ever > do so I don't worry about re-reading them ... > > I have no idea why you see a factor of 250 - is this a memory problem. > How much memory have you got? Does your program scale linearly with > the file size - or does something go suddenly wrong as you increase the > size of the file? > > Cheers > > /Joe > > > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil > wrote: >> >> Hello, >> I know it was been already discussed here in list and it is also recurring >> topic for at least five years. But anyway I have been bitten by it again >> and >> also found pretty pathological case. I have 30MB text file and it has a >> few >> near to 1MB lines there. (I can provide file with same line lengths if >> somebody interested.) What I have been observed is that reading this file >> using raw file:read_line/1 takes 51s! For comparison I have tried some >> different approaches and what I got (line_read_test:read_std/1 is using >> file:read_line/1): >> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). >> {51028105,2408} >> 2> timer:tc(line_read_test,read,["test.txt"]). >> {226220,2408} >> 3> timer:tc(line_read_test,read_port,["test.txt"]). >> {139388,2408} >> >> $time perl -nE'$i++}{say $i' test.txt >> 2408 >> >> real 0m0.053s >> user 0m0.044s >> sys 0m0.008s >> >> $ time wc -l test.txt >> 2408 test.txt >> >> real 0m0.013s >> user 0m0.004s >> sys 0m0.008s >> >> $ time ./a.out test.txt >> 2408 >> >> real 0m0.020s >> user 0m0.012s >> sys 0m0.008s >> >> It means erlang should be at least 225 times faster (line_read_test:read/1 >> which has flow control). Erlang can be 350 times faster >> (line_read_test:read_port/1 without flow control). Another high level >> language (perl) is almost thousand times faster. Special C program is >> almost >> four thousands times faster and old good glibc is two and half thousands >> times faster. Come on guys it is not even fun when simple (and wrong) >> erlang >> wrapper around standard module is more than two order of magnitude faster. >> From mine experience when there is something two orders of magnitude >> slower >> it tells me there is something damn wrong. I have looked into efile_drv.c >> and >> ti is unfortunately far beyond mine C skill but if simple buffering and >> binary:match/2 can outperform it 225 times there has to be something >> rotten >> in this code. >> >> I have also experimented with read_ahead option in file:open and changing >> to >> less value makes thing worse. >> >> Just to make grasp how bad it is, in same time I'm able sort 150 million >> 64bit values (1.2GB of data) three times (one CPU core same HW). It is not >> in >> flow control, mine simple wrapper does flow control too. It can't make >> current code less intrusive, if it consumes 100% CPU for 51s instead of >> 226ms >> then it will definitely affect whole server. It is not in concurrent >> access, >> mine code allows concurrent access too. Admitting there is something >> broken >> is first step to fixing it. I hope I helped at least in this way. >> >> With best regards >> Hynek Vychodil >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: test.txt.bz2 Type: application/x-bzip2 Size: 1043 bytes Desc: not available URL: From vychodil.hynek@REDACTED Fri Feb 15 01:29:35 2013 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Fri, 15 Feb 2013 01:29:35 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: Using async thread drops time down to 4.6s which is still 20 times slower than mine simple wrapper. On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson wrote: > Hello! > > If you start erlang with an async thread (erl +A 1) you will see radically > different behaviour. For me the time it takes to use read_line drops from > 58s to 2s. When doing file i/o you should always have some async threads to > help you do the work, this is why in R16B we changed the default from 0 to > 10 async threads. > > Lukas > > > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil > wrote: >> >> Hello, >> I know it was been already discussed here in list and it is also recurring >> topic for at least five years. But anyway I have been bitten by it again >> and >> also found pretty pathological case. I have 30MB text file and it has a >> few >> near to 1MB lines there. (I can provide file with same line lengths if >> somebody interested.) What I have been observed is that reading this file >> using raw file:read_line/1 takes 51s! For comparison I have tried some >> different approaches and what I got (line_read_test:read_std/1 is using >> file:read_line/1): >> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). >> {51028105,2408} >> 2> timer:tc(line_read_test,read,["test.txt"]). >> {226220,2408} >> 3> timer:tc(line_read_test,read_port,["test.txt"]). >> {139388,2408} >> >> $time perl -nE'$i++}{say $i' test.txt >> 2408 >> >> real 0m0.053s >> user 0m0.044s >> sys 0m0.008s >> >> $ time wc -l test.txt >> 2408 test.txt >> >> real 0m0.013s >> user 0m0.004s >> sys 0m0.008s >> >> $ time ./a.out test.txt >> 2408 >> >> real 0m0.020s >> user 0m0.012s >> sys 0m0.008s >> >> It means erlang should be at least 225 times faster (line_read_test:read/1 >> which has flow control). Erlang can be 350 times faster >> (line_read_test:read_port/1 without flow control). Another high level >> language (perl) is almost thousand times faster. Special C program is >> almost >> four thousands times faster and old good glibc is two and half thousands >> times faster. Come on guys it is not even fun when simple (and wrong) >> erlang >> wrapper around standard module is more than two order of magnitude faster. >> From mine experience when there is something two orders of magnitude >> slower >> it tells me there is something damn wrong. I have looked into efile_drv.c >> and >> ti is unfortunately far beyond mine C skill but if simple buffering and >> binary:match/2 can outperform it 225 times there has to be something >> rotten >> in this code. >> >> I have also experimented with read_ahead option in file:open and changing >> to >> less value makes thing worse. >> >> Just to make grasp how bad it is, in same time I'm able sort 150 million >> 64bit values (1.2GB of data) three times (one CPU core same HW). It is not >> in >> flow control, mine simple wrapper does flow control too. It can't make >> current code less intrusive, if it consumes 100% CPU for 51s instead of >> 226ms >> then it will definitely affect whole server. It is not in concurrent >> access, >> mine code allows concurrent access too. Admitting there is something >> broken >> is first step to fixing it. I hope I helped at least in this way. >> >> With best regards >> Hynek Vychodil >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > From wallentin.dahlberg@REDACTED Fri Feb 15 01:33:29 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Fri, 15 Feb 2013 01:33:29 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: <4FE32BA0-31DC-4476-B14E-FD41185138B5@cs.otago.ac.nz> References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> <4FE32BA0-31DC-4476-B14E-FD41185138B5@cs.otago.ac.nz> Message-ID: 2013/2/15 Richard A. O'Keefe > > On 15/02/2013, at 11:42 AM, Bj?rn-Egil Dahlberg wrote: > > If I'm reading [EEP 20] correctly: a local atom would actually require a > larger heap space than an equivalent heap binary .. which is a feat all by > itself. > > A 'local atom' uses the same space *per reference* as existing atoms do. > On the heap of the process that created it, > it holds the bytes of the name plus a 3-word header (32-bit machine > design). > The minimum imaginable overhead would be a 1-word header, and that > would sacrifice some of the efficiency that the 3-word header allows. > > Now, I'm looking at an elderly (R12B-5) set of sources (on an elderly > machine), > and I see that > - a ProcBin requires 6 words on the process's heap > - a Binary requires a 3-word header plus the bytes, > or on a 32-bit machine, a 4-word header plus the bytes. > I believe a Binary is the "equivalent heap binary" you are talking about. > A heap binary has a header + size + data, minimum three words (Thats two words and data by your counting). This will only take two words (one word + data) in an upcoming release after R16B, sub-binaries and proc-bins will also loose one word of memory. along with some other stuff i won't go into > > So no, a local atom would NOT require larger heap space than a binary. > Why did you think it would? > +----------+ | size+tag | boxed object header; see below +----------+ | hashcode | a 32-bit hash code +----------+ | equivrep | points to Union/Find representative +----------+ | bytes of | | name ... | +----------+ This made me count 1, 2, 3, 4 boxes, i.e. minimum 4 words. 4 > 3 .. I wanted to answer your rhetorical(?) question just so we could launch this discussion on a complete tangent to outer space. So boxed pointers and unification will fix this memory eventually .. goody, goody. Geared towards many references one object usecase .. i get it. > > I agree that atom gc is needed but it shouldn't be an excuse for using > dynamic atoms instead of binaries. > > On the one hand, it isn't. The EEP is very explicit that the goal is > _safety_ rather than _convenience_. > Good point. Totally agree that safety > convenience. > > However, haven't we had enough complaints about the confusion between > lists of integers > and strings of text? Don't you think that a clean distinction between a > sequence of bits in some > unknown encoding and a chunk of text in a known encoding is useful? > Is that the argument for atoms as strings or the argument against it? Yet another way to represent a string? Granted, it's already there but I won't sway from my beliefs. =) > > > Besides, if locals atoms are larger than binaries why would you use > atoms. (I might be wrong about the size though .. didn't look that hard). > > Oh, if people are bigger than houses, we should all use Perl. > I always suspected we could just build extremely tall ivory towers. Though it would defy physics, it's completely alright since hubris trumps physics. The endgame is the same. A nice syntax, good apis, great performance and a lovely experience. Perl is probably the least lovely I know of. So a side note, last I looked at a roadmap with "when will we implement at atom gc?" I think a saw R20-ish. So it is naturally we haven't been thinking about it. Up til now I've been perfectly fine with saying "don't use dynamic atoms. It's bad". Actually I'm perfectly fine by still saying don't create atoms dynamically .. bad! I don't want one solution. I want many solutions. Choose the best one and then implement it. I want to say plenty more but .. // Bj?rn-Egil .. seriously need sleep -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Feb 15 01:48:13 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 15 Feb 2013 13:48:13 +1300 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: On 15/02/2013, at 1:02 PM, Bj?rn-Egil Dahlberg wrote: > > Ok, how does this perform against linear scan Eterm data array? Eh? You see to be comparing apples with bulldozers here. > The tradeoff being one word memory saved on the heap also? Keep in mind that most atoms are pretty short. With atoms typed by hand, that is the case. With atoms generated by programs (like the function names the Erlang compiler generates for list comprehensions), there is no tired-finger constraint to stop long atoms being used. > > .. first thought you were messing with the arity thing meaning .. perhaps i should sleep. putting more stuff in the header .. seems good EEP 20 was written with no knowledge of Erlang's low level implementation details. The background for it is a WAM-like architecture, with a 2-bit tag 00 immediate 01 box-of-bits 10 pointer to [_|_] (which has no header) 11 pointer to box-of-tagged-words and boxes have an "arity" field (the size of a tuple, the length of a binary) that includes a few "supertag" bits that say what kind of box it is. The "arity" field used in EEP 20 holds the bits that say "I am a local atom" and a length, encoded to make *both* "number of bytes" and "number of Unicode characters" constant-time operations. The equivrep field is what enables atoms that have been found to be equal to be chained together; if a 3-word header is too big (despite being the same size as or smaller than a binary's header), that word could be sacrificed. > .. wait a minute .. are you proposing an immediate with a payload? Eh? If you call a binary "an immediate with a payload", yes. Otherwise no. > > I think there are many ways to skin this particular cat and I think it should debated what the most effective atom gc could be. Yeah, well, EEP 20 was created on 25-Aug-2008. We've had 1635 days since then to debate atom gc, and *now* you want to start? The key thing about EEP 20 is that IT IS NOT ORIGINAL. The basic idea there is proven concurrent programming technology. We *know* this works. From n.oxyde@REDACTED Fri Feb 15 02:04:09 2013 From: n.oxyde@REDACTED (Anthony Ramine) Date: Fri, 15 Feb 2013 02:04:09 +0100 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: My (unfinished) implementation dates back from when Erlang didn't have UTF-8 atoms and I didn't think they would be coming that fast. So I didn't have to mess with the arity field and just used this structure: typedef struct local_atom_ { Eterm header; Eterm equivrep; Uint32 hash; Uint32 len; Eterm name[1]; // by the way can we use C99/C11 variable-length arrays in OTP? } LocalAtom; I didn't use the same structure as global atoms because they have a member specific to their hash table structure. When I make it handle UTF-8 atoms, I'll just split Uint32 len into two Uint16 bytes_len and Uint16 char_len; 16 bits ought to be enough, right? https://github.com/nox/otp/blob/bf3334c/erts/emulator/beam/erl_term.h#L561-567 That's an overhead of 3 words on 64-bit, 2 words on 64-bit with halfword emulator and 4 words on 32-bit. Should we worry about 4 words when safety is concerned? Should we worry about 4 words when the OTP XML parser cannot be used in production with user input because it uses atoms for XML names? We shouldn't. -- Anthony Ramine Le 15 f?vr. 2013 ? 01:48, Richard A. O'Keefe a ?crit : >> .. first thought you were messing with the arity thing meaning .. perhaps i should sleep. putting more stuff in the header .. seems good > > EEP 20 was written with no knowledge of Erlang's low level implementation details. > The background for it is a WAM-like architecture, with a 2-bit tag > 00 immediate > 01 box-of-bits > 10 pointer to [_|_] (which has no header) > 11 pointer to box-of-tagged-words > and boxes have an "arity" field (the size of a tuple, the length of a binary) > that includes a few "supertag" bits that say what kind of box it is. > The "arity" field used in EEP 20 holds the bits that say "I am a local atom" > and a length, encoded to make *both* "number of bytes" and "number of > Unicode characters" constant-time operations. > > The equivrep field is what enables atoms that have been found to be equal > to be chained together; if a 3-word header is too big (despite being the same > size as or smaller than a binary's header), that word could be sacrificed. From ok@REDACTED Fri Feb 15 02:05:30 2013 From: ok@REDACTED (Richard A. O'Keefe) Date: Fri, 15 Feb 2013 14:05:30 +1300 Subject: [erlang-questions] Did Erlang's grammar change in R16A? In-Reply-To: References: <511D3616.2070303@gmail.com> <511D63FF.3010409@ninenines.eu> Message-ID: <9911BD53-D756-4778-AE7C-6DEE0CC1AE4E@cs.otago.ac.nz> On 15/02/2013, at 1:48 PM, Richard A. O'Keefe wrote: > > EEP 20 was written with no knowledge of Erlang's low level implementation details. Correction: s/no knowledge/little knowledge and less understanding/ From rustompmody@REDACTED Fri Feb 15 06:36:41 2013 From: rustompmody@REDACTED (Rustom Mody) Date: Fri, 15 Feb 2013 11:06:41 +0530 Subject: [erlang-questions] Can Erlang come under attack? Message-ID: Please see http://pyfound.blogspot.in/2013/02/python-trademark-at-risk-in-europe-we.html I cannot say that I understand whats really going on The one thing that I get is that there was some minor neglect a decade or more ago. Maybe there are things that Erlang needs to beware of? -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony.houston@REDACTED Fri Feb 15 01:45:06 2013 From: anthony.houston@REDACTED (Anthony Houston) Date: Fri, 15 Feb 2013 10:45:06 +1000 Subject: [erlang-questions] Mnesia question Message-ID: I have downloaded and installed Erlang on my iMac. Is Mnesia database part of the base Erlang download or do I need the OTP development stuff. When I load Mnesia I do not get any messages of any sort so it is a bit confusing. Thanks. From cobus.carstens@REDACTED Fri Feb 15 07:20:27 2013 From: cobus.carstens@REDACTED (Cobus Carstens) Date: Thu, 14 Feb 2013 22:20:27 -0800 (PST) Subject: [erlang-questions] supervisor name spec Message-ID: Hi all The spec for supervisor:start_link/3 requires the SupName argument to be: -type sup_name() :: {'local', Name :: atom()} | {'global', Name :: atom()}. supervisor:start_link/3 calls gen_server:start_link/4, which does not have a spec, but according to the source comments: "Name ::= {local, atom()} | {global, atom()} | {via, atom(), term()}" gen_server:start_link/4 calls gen:start/6, with a spec which requires the Name argument to be: -type emgr_name() :: {'local', atom()} | {'global', term()} | {via, atom(), term()}. The question: Why does the SupName argument of supervisor:start_link/3 function have the more restrictive {'global', Name :: atom()} spec as opposed to {'global', term()}, which is supported by gen:start/6 ? P.S I used the R16A source code as reference, but it applies to earlier releases too. Regards, C -------------- next part -------------- An HTML attachment was scrubbed... URL: From bengt.kleberg@REDACTED Fri Feb 15 09:31:51 2013 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 15 Feb 2013 09:31:51 +0100 Subject: [erlang-questions] Mnesia question In-Reply-To: References: Message-ID: <1360917111.4758.15.camel@sekic1152.rnd.ki.sw.ericsson.se> Greetings, >From where have you downloaded Erlang? What did it say that you where downloading? You say that you load Mnesia. Is that from the Erlang shell? Given the Erlang shell, can you do: m(mnesia). and see if it works? bengt On Fri, 2013-02-15 at 10:45 +1000, Anthony Houston wrote: > I have downloaded and installed Erlang on my iMac. Is Mnesia database part of the base Erlang download or do I need the OTP development stuff. > > When I load Mnesia I do not get any messages of any sort so it is a bit confusing. > > Thanks. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From paul.james.barry@REDACTED Fri Feb 15 10:20:30 2013 From: paul.james.barry@REDACTED (Paul Barry) Date: Fri, 15 Feb 2013 09:20:30 +0000 Subject: [erlang-questions] Can Erlang come under attack? In-Reply-To: References: Message-ID: I'm not a legal expert, but the Python world is experiencing this issue because they didn't acquire "python.co.uk" way back when... and now the domain troll that has it has decided to be a bit of a [insert appropriate insult here]. In the Erlang world, I'd imagine Ericsson own the copyright to the name (and they have an interest in protecting it), and - anyways - we have Erlang Solutions based in Europe and they are doing a pretty good job of looking after the interests of the community and Erlang's good name, also. Paul. On 15 February 2013 05:36, Rustom Mody wrote: > Please see > > http://pyfound.blogspot.in/2013/02/python-trademark-at-risk-in-europe-we.html > > I cannot say that I understand whats really going on > The one thing that I get is that there was some minor neglect a decade or > more ago. > Maybe there are things that Erlang needs to beware of? > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.james.barry@REDACTED Fri Feb 15 10:29:08 2013 From: paul.james.barry@REDACTED (Paul Barry) Date: Fri, 15 Feb 2013 09:29:08 +0000 Subject: [erlang-questions] O'Reilly eBook Deal until 17th Feb Message-ID: Folks. O'Reilly Media are offering 50% off their Erlang eMaterials for the next 48 hours (or so). Details are here: http://shop.oreilly.com/category/deals/erlang-dotd.do?code=DEAL&imm_mid=0a1776&cmp=em-code-books-videos-deal-day-erlang-direct Spread the word... perhaps we can entice some new converts? Paul. -- Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc@REDACTED Fri Feb 15 11:00:29 2013 From: marc@REDACTED (Marc Worrell) Date: Fri, 15 Feb 2013 11:00:29 +0100 Subject: [erlang-questions] Zotonic presentation at Cross Functional Amsterdam Meetup Message-ID: <4B666701-81D4-452F-AA25-4DDA54CE1555@worrell.nl> Hi, Last january we had another great Cross Functional Amsterdam meet up at TTY, organized by Ward Bekker. I had the honor to give a presentation about Zotonic. Zotonic is the Erlang CMS and framework. The presentation focused primarily on some performance considerations and tradeoffs in the design of Zotonic. You can now read the sheets on Scribd: http://www.scribd.com/doc/125608814/Zotonic-Make-it-Faster Are you in Amsterdam on March 28th? Then come to the next meet up: http://www.meetup.com/funadam/ Kind Regards, Marc Worrell From erlang@REDACTED Fri Feb 15 11:49:05 2013 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 15 Feb 2013 11:49:05 +0100 Subject: [erlang-questions] Can Erlang come under attack? In-Reply-To: References: Message-ID: On Fri, Feb 15, 2013 at 10:20 AM, Paul Barry wrote: > I'm not a legal expert, but the Python world is experiencing this issue > because they didn't acquire "python.co.uk" way back when... and now the > domain troll that has it has decided to be a bit of a [insert appropriate > insult here]. > > In the Erlang world, I'd imagine Ericsson own the copyright to the name > (and they have an interest in protecting it), and - anyways - we have > Erlang Solutions based in Europe and they are doing a pretty good job of > looking after the interests of the community and Erlang's good name, also. > Correct. Ericsson did take out copyright for the name, or possibly even a trademark. This was done is a number of different countries (a dozen or so, there were no global ways to do this) - I can't remember the details nor for how long these agreements last. Cheers /Joe > > Paul. > > > On 15 February 2013 05:36, Rustom Mody wrote: > >> Please see >> >> http://pyfound.blogspot.in/2013/02/python-trademark-at-risk-in-europe-we.html >> >> I cannot say that I understand whats really going on >> The one thing that I get is that there was some minor neglect a decade or >> more ago. >> Maybe there are things that Erlang needs to beware of? >> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > > > -- > Paul Barry, w: http://paulbarry.itcarlow.ie - e: paul.barry@REDACTED > Lecturer, Computer Networking: Institute of Technology, Carlow, Ireland. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardprideauxevans@REDACTED Fri Feb 15 13:31:03 2013 From: richardprideauxevans@REDACTED (Richard Evans) Date: Fri, 15 Feb 2013 12:31:03 +0000 Subject: [erlang-questions] Our new game is powered by erlang. Thanks for building such a great environment! Message-ID: We just released our new game for the iPad on the app store: www.versu.com It is a text-based AI simulation of Jane Austen literature. The server is written in erlang (largely by James Mayfield and myself) and the simulator in c. The erlang tools we used (OTP, cowboy, riak) were excellent. Also, big thanks to this message board for helping out a couple of times when I got stuck. thanks for creating such a great language and environment! Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Fri Feb 15 13:43:29 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Fri, 15 Feb 2013 13:43:29 +0100 Subject: [erlang-questions] Our new game is powered by erlang. Thanks for building such a great environment! In-Reply-To: References: Message-ID: <511E2D71.1020408@ninenines.eu> On 02/15/2013 01:31 PM, Richard Evans wrote: > We just released our new game for the iPad on the app store: > www.versu.com > > It is a text-based AI simulation of Jane Austen literature. > > The server is written in erlang (largely by James Mayfield and myself) > and the simulator in c. The erlang tools we used (OTP, cowboy, riak) > were excellent. Also, big thanks to this message board for helping out a > couple of times when I got stuck. > > thanks for creating such a great language and environment! Awesome! -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From lukas@REDACTED Fri Feb 15 16:42:48 2013 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 15 Feb 2013 16:42:48 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: aha, you needed to have very long lines in the file to provoke the behaviour. The file I was experimenting on had the same line length all the time. I think I've found the culprit. Aparently the read_ahead option was ignored after the first buffer had been consumed. Try the patch below and see if it works for you: >From 0884480691e047801cef117b19151a7d34c9e438 Mon Sep 17 00:00:00 2001 From: Lukas Larsson Date: Fri, 15 Feb 2013 16:37:52 +0100 Subject: [PATCH] invoke_read_line now remembers read_ahead settings --- erts/emulator/drivers/common/efile_drv.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/erts/emulator/drivers/common/efile_drv.c b/erts/emulator/drivers/common/efile_drv.c index 2279fec..d81f116 100644 --- a/erts/emulator/drivers/common/efile_drv.c +++ b/erts/emulator/drivers/common/efile_drv.c @@ -1160,6 +1160,8 @@ static void invoke_read_line(void *data) /* Need more place */ ErlDrvSizeT need = (d->c.read_line.read_size >= DEFAULT_LINEBUF_SIZE) ? d->c.read_line.read_size + DEFAULT_LINEBUF_SIZE : DEFAULT_LINEBUF_SIZE; + if (need < (d->c.read_line.binp)->orig_size) + need = (d->c.read_line.binp)->orig_size; ErlDrvBinary *newbin = driver_alloc_binary(need); if (newbin == NULL) { d->result_ok = 0; -- 1.7.10.4 On Fri, Feb 15, 2013 at 1:29 AM, Hynek Vychodil wrote: > Using async thread drops time down to 4.6s which is still 20 times > slower than mine simple wrapper. > > On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson wrote: > > Hello! > > > > If you start erlang with an async thread (erl +A 1) you will see > radically > > different behaviour. For me the time it takes to use read_line drops from > > 58s to 2s. When doing file i/o you should always have some async threads > to > > help you do the work, this is why in R16B we changed the default from 0 > to > > 10 async threads. > > > > Lukas > > > > > > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil < > vychodil.hynek@REDACTED> > > wrote: > >> > >> Hello, > >> I know it was been already discussed here in list and it is also > recurring > >> topic for at least five years. But anyway I have been bitten by it again > >> and > >> also found pretty pathological case. I have 30MB text file and it has a > >> few > >> near to 1MB lines there. (I can provide file with same line lengths if > >> somebody interested.) What I have been observed is that reading this > file > >> using raw file:read_line/1 takes 51s! For comparison I have tried some > >> different approaches and what I got (line_read_test:read_std/1 is using > >> file:read_line/1): > >> > >> 1> timer:tc(line_read_test,read_std,["test.txt"]). > >> {51028105,2408} > >> 2> timer:tc(line_read_test,read,["test.txt"]). > >> {226220,2408} > >> 3> timer:tc(line_read_test,read_port,["test.txt"]). > >> {139388,2408} > >> > >> $time perl -nE'$i++}{say $i' test.txt > >> 2408 > >> > >> real 0m0.053s > >> user 0m0.044s > >> sys 0m0.008s > >> > >> $ time wc -l test.txt > >> 2408 test.txt > >> > >> real 0m0.013s > >> user 0m0.004s > >> sys 0m0.008s > >> > >> $ time ./a.out test.txt > >> 2408 > >> > >> real 0m0.020s > >> user 0m0.012s > >> sys 0m0.008s > >> > >> It means erlang should be at least 225 times faster > (line_read_test:read/1 > >> which has flow control). Erlang can be 350 times faster > >> (line_read_test:read_port/1 without flow control). Another high level > >> language (perl) is almost thousand times faster. Special C program is > >> almost > >> four thousands times faster and old good glibc is two and half thousands > >> times faster. Come on guys it is not even fun when simple (and wrong) > >> erlang > >> wrapper around standard module is more than two order of magnitude > faster. > >> From mine experience when there is something two orders of magnitude > >> slower > >> it tells me there is something damn wrong. I have looked into > efile_drv.c > >> and > >> ti is unfortunately far beyond mine C skill but if simple buffering and > >> binary:match/2 can outperform it 225 times there has to be something > >> rotten > >> in this code. > >> > >> I have also experimented with read_ahead option in file:open and > changing > >> to > >> less value makes thing worse. > >> > >> Just to make grasp how bad it is, in same time I'm able sort 150 million > >> 64bit values (1.2GB of data) three times (one CPU core same HW). It is > not > >> in > >> flow control, mine simple wrapper does flow control too. It can't make > >> current code less intrusive, if it consumes 100% CPU for 51s instead of > >> 226ms > >> then it will definitely affect whole server. It is not in concurrent > >> access, > >> mine code allows concurrent access too. Admitting there is something > >> broken > >> is first step to fixing it. I hope I helped at least in this way. > >> > >> With best regards > >> Hynek Vychodil > >> > >> _______________________________________________ > >> erlang-questions mailing list > >> erlang-questions@REDACTED > >> http://erlang.org/mailman/listinfo/erlang-questions > >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sasa555@REDACTED Fri Feb 15 16:47:25 2013 From: sasa555@REDACTED (sasa) Date: Fri, 15 Feb 2013 16:47:25 +0100 Subject: [erlang-questions] messages manipulatio Message-ID: Hello, A while ago I encountered the following situation: I had the gen_server base process P which would receive messages, and handle them by sending some data over the network. The messages were coming faster than they were being sent. I established the reason for this was the "randomness" of my network conditions. I also established that sending more messages at once was almost as fast as sending one message, i.e. the network push time wasn't highly dependent on the message size. To tackle this in a generic way, I devised an approach which has served me well in multiple places. I was repeatedly googling whether some similar solution exists, but I couldn't find it. Now, I'm not sure if I have reinvented a wheel, or the approach is not optimal, so I'm asking if you are aware of similar approaches, and are there any faults in this one? The approach I took is following: I split the server in two processes: the "facade" and the worker. The facade acceptes requests from clients, and stores them internally. While the worker is doing its work, new messages are stored in the facade. When the worker is available, it will take all accumulated messages from the facade and process them. These are the steps: 1. The facade receives messages, stores data in its list, and notifies the worker (without sending actual data), that some work is ready. 2. Upon receiving the notification, the worker first flushes its message queue by doing repeated receive ... after 0 as long as there are messages in the queue. 3. Then the worker pulls all messages from the facade. This is a gen_server:call to the facade which will return all messages, and at the same time remove them from its state. 4. Finally, the worker processes all messages. I found this approach useful because the delay on the worker adapts to the incoming message rate. If the worker can handle messages at the incoming rate, everything works without delay. If messages can't be handled at the incoming rate, the worker's delay will increase to accomodate the higher load. In other words, the worker will try to compensate the load by bulk processing messages. Obviously, this is useful only when process_time(N messages) < N * process_time(1 message). Another benefit I found is that I can vary the implementation of the facade i.e. I can store messages using different algorithms. In the first implementation, I stored messages in a list. In one variation, I used hash which allowed me to eliminate duplicate messages. Another variant was truncation of the list, which allowed me to discard old messages if the queue was getting too large. As I said, this has served me well in the production for more than a year, and I have finally found the time to make a generic library out of it. Before putting it public, I'd like to check if there are similar solutions, or alternative approaches? Thanks, and best regards, Sasa -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Fri Feb 15 16:58:49 2013 From: lukas@REDACTED (Lukas Larsson) Date: Fri, 15 Feb 2013 16:58:49 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: Oh, and here are the benchmarks for me now: R15B03: 1> timer:tc(line_read_test,read_std,["test.txt"]). {21307874,14448} R16B with patch: 1> timer:tc(line_read_test,read_std,["test.txt"]). {91885,14448} It's not every day you find a two line change which increases speed by a factor of 232! Lukas On Fri, Feb 15, 2013 at 4:42 PM, Lukas Larsson wrote: > aha, you needed to have very long lines in the file to provoke the > behaviour. The file I was experimenting on had the same line length all the > time. > > I think I've found the culprit. Aparently the read_ahead option was > ignored after the first buffer had been consumed. Try the patch below and > see if it works for you: > > From 0884480691e047801cef117b19151a7d34c9e438 Mon Sep 17 00:00:00 2001 > From: Lukas Larsson > Date: Fri, 15 Feb 2013 16:37:52 +0100 > Subject: [PATCH] invoke_read_line now remembers read_ahead settings > > --- > erts/emulator/drivers/common/efile_drv.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/erts/emulator/drivers/common/efile_drv.c > b/erts/emulator/drivers/common/efile_drv.c > index 2279fec..d81f116 100644 > --- a/erts/emulator/drivers/common/efile_drv.c > +++ b/erts/emulator/drivers/common/efile_drv.c > @@ -1160,6 +1160,8 @@ static void invoke_read_line(void *data) > /* Need more place */ > ErlDrvSizeT need = (d->c.read_line.read_size >= > DEFAULT_LINEBUF_SIZE) ? > d->c.read_line.read_size + DEFAULT_LINEBUF_SIZE : > DEFAULT_LINEBUF_SIZE; > + if (need < (d->c.read_line.binp)->orig_size) > + need = (d->c.read_line.binp)->orig_size; > ErlDrvBinary *newbin = driver_alloc_binary(need); > if (newbin == NULL) { > d->result_ok = 0; > -- > 1.7.10.4 > > > > > On Fri, Feb 15, 2013 at 1:29 AM, Hynek Vychodil wrote: > >> Using async thread drops time down to 4.6s which is still 20 times >> slower than mine simple wrapper. >> >> On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson wrote: >> > Hello! >> > >> > If you start erlang with an async thread (erl +A 1) you will see >> radically >> > different behaviour. For me the time it takes to use read_line drops >> from >> > 58s to 2s. When doing file i/o you should always have some async >> threads to >> > help you do the work, this is why in R16B we changed the default from 0 >> to >> > 10 async threads. >> > >> > Lukas >> > >> > >> > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil < >> vychodil.hynek@REDACTED> >> > wrote: >> >> >> >> Hello, >> >> I know it was been already discussed here in list and it is also >> recurring >> >> topic for at least five years. But anyway I have been bitten by it >> again >> >> and >> >> also found pretty pathological case. I have 30MB text file and it has a >> >> few >> >> near to 1MB lines there. (I can provide file with same line lengths if >> >> somebody interested.) What I have been observed is that reading this >> file >> >> using raw file:read_line/1 takes 51s! For comparison I have tried some >> >> different approaches and what I got (line_read_test:read_std/1 is using >> >> file:read_line/1): >> >> >> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). >> >> {51028105,2408} >> >> 2> timer:tc(line_read_test,read,["test.txt"]). >> >> {226220,2408} >> >> 3> timer:tc(line_read_test,read_port,["test.txt"]). >> >> {139388,2408} >> >> >> >> $time perl -nE'$i++}{say $i' test.txt >> >> 2408 >> >> >> >> real 0m0.053s >> >> user 0m0.044s >> >> sys 0m0.008s >> >> >> >> $ time wc -l test.txt >> >> 2408 test.txt >> >> >> >> real 0m0.013s >> >> user 0m0.004s >> >> sys 0m0.008s >> >> >> >> $ time ./a.out test.txt >> >> 2408 >> >> >> >> real 0m0.020s >> >> user 0m0.012s >> >> sys 0m0.008s >> >> >> >> It means erlang should be at least 225 times faster >> (line_read_test:read/1 >> >> which has flow control). Erlang can be 350 times faster >> >> (line_read_test:read_port/1 without flow control). Another high level >> >> language (perl) is almost thousand times faster. Special C program is >> >> almost >> >> four thousands times faster and old good glibc is two and half >> thousands >> >> times faster. Come on guys it is not even fun when simple (and wrong) >> >> erlang >> >> wrapper around standard module is more than two order of magnitude >> faster. >> >> From mine experience when there is something two orders of magnitude >> >> slower >> >> it tells me there is something damn wrong. I have looked into >> efile_drv.c >> >> and >> >> ti is unfortunately far beyond mine C skill but if simple buffering and >> >> binary:match/2 can outperform it 225 times there has to be something >> >> rotten >> >> in this code. >> >> >> >> I have also experimented with read_ahead option in file:open and >> changing >> >> to >> >> less value makes thing worse. >> >> >> >> Just to make grasp how bad it is, in same time I'm able sort 150 >> million >> >> 64bit values (1.2GB of data) three times (one CPU core same HW). It is >> not >> >> in >> >> flow control, mine simple wrapper does flow control too. It can't make >> >> current code less intrusive, if it consumes 100% CPU for 51s instead of >> >> 226ms >> >> then it will definitely affect whole server. It is not in concurrent >> >> access, >> >> mine code allows concurrent access too. Admitting there is something >> >> broken >> >> is first step to fixing it. I hope I helped at least in this way. >> >> >> >> With best regards >> >> Hynek Vychodil >> >> >> >> _______________________________________________ >> >> erlang-questions mailing list >> >> erlang-questions@REDACTED >> >> http://erlang.org/mailman/listinfo/erlang-questions >> >> >> > >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Fri Feb 15 17:35:04 2013 From: robert.virding@REDACTED (Robert Virding) Date: Fri, 15 Feb 2013 16:35:04 +0000 (GMT) Subject: [erlang-questions] R16B enif_consume_timeslice In-Reply-To: Message-ID: <1682477952.23449612.1360946104492.JavaMail.root@erlang-solutions.com> I like this patch. Hopefully it will make people realise that NIFs can adversely affect the real-time performance of their systems. This is a help for them. Robert ----- Original Message ----- > From: "Tuncer Ayaz" > To: erlang-questions@REDACTED > Cc: "Sverker Eriksson" > Sent: Thursday, 14 February, 2013 8:22:34 AM > Subject: [erlang-questions] R16B enif_consume_timeslice > > R16B gained a new NIF API function that allows your NIFs to better > co-operate with the vm. > > Documentation: > https://github.com/erlang/otp/commit/ef3566f48e08#L0R586 > Example: > https://github.com/erlang/otp/commit/ef3566f48e08#L5R1459 > > Thanks Sverker! This could be quite useful and ease the wait for > native processes :). > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From vychodil.hynek@REDACTED Sat Feb 16 00:57:39 2013 From: vychodil.hynek@REDACTED (Hynek Vychodil) Date: Sat, 16 Feb 2013 00:57:39 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: It works like charm. I have patched R15B03 ad with 1MB read ahead it is unbelievable fast 20> timer:tc(line_read_test,read_std,["test.txt"]). {30732,2408} It is even four times faster than port version 24> timer:tc(line_read_test,read_port,["test.txt"]). {135676,2408} It is also faster without async threads. Measurements above was with +A 10. Without async threads: 2> timer:tc(line_read_test,read_std,["test.txt"]). {19322,2408} I was expected time around 100ms but current behavior is far beyond expectation. I hope this patch will find way into upcoming R16. It seems to speed up line oriented io for "normal" files two times also. Thanks a lot Hynek Vychodil On Fri, Feb 15, 2013 at 4:58 PM, Lukas Larsson wrote: > Oh, and here are the benchmarks for me now: > > R15B03: > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > {21307874,14448} > > R16B with patch: > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > {91885,14448} > > It's not every day you find a two line change which increases speed by a > factor of 232! > > Lukas > > > On Fri, Feb 15, 2013 at 4:42 PM, Lukas Larsson > wrote: >> >> aha, you needed to have very long lines in the file to provoke the >> behaviour. The file I was experimenting on had the same line length all the >> time. >> >> I think I've found the culprit. Aparently the read_ahead option was >> ignored after the first buffer had been consumed. Try the patch below and >> see if it works for you: >> >> From 0884480691e047801cef117b19151a7d34c9e438 Mon Sep 17 00:00:00 2001 >> From: Lukas Larsson >> Date: Fri, 15 Feb 2013 16:37:52 +0100 >> Subject: [PATCH] invoke_read_line now remembers read_ahead settings >> >> --- >> erts/emulator/drivers/common/efile_drv.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/erts/emulator/drivers/common/efile_drv.c >> b/erts/emulator/drivers/common/efile_drv.c >> index 2279fec..d81f116 100644 >> --- a/erts/emulator/drivers/common/efile_drv.c >> +++ b/erts/emulator/drivers/common/efile_drv.c >> @@ -1160,6 +1160,8 @@ static void invoke_read_line(void *data) >> /* Need more place */ >> ErlDrvSizeT need = (d->c.read_line.read_size >= >> DEFAULT_LINEBUF_SIZE) ? >> d->c.read_line.read_size + DEFAULT_LINEBUF_SIZE : >> DEFAULT_LINEBUF_SIZE; >> + if (need < (d->c.read_line.binp)->orig_size) >> + need = (d->c.read_line.binp)->orig_size; >> ErlDrvBinary *newbin = driver_alloc_binary(need); >> if (newbin == NULL) { >> d->result_ok = 0; >> -- >> 1.7.10.4 >> >> >> >> >> On Fri, Feb 15, 2013 at 1:29 AM, Hynek Vychodil >> wrote: >>> >>> Using async thread drops time down to 4.6s which is still 20 times >>> slower than mine simple wrapper. >>> >>> On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson wrote: >>> > Hello! >>> > >>> > If you start erlang with an async thread (erl +A 1) you will see >>> > radically >>> > different behaviour. For me the time it takes to use read_line drops >>> > from >>> > 58s to 2s. When doing file i/o you should always have some async >>> > threads to >>> > help you do the work, this is why in R16B we changed the default from 0 >>> > to >>> > 10 async threads. >>> > >>> > Lukas >>> > >>> > >>> > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil >>> > >>> > wrote: >>> >> >>> >> Hello, >>> >> I know it was been already discussed here in list and it is also >>> >> recurring >>> >> topic for at least five years. But anyway I have been bitten by it >>> >> again >>> >> and >>> >> also found pretty pathological case. I have 30MB text file and it has >>> >> a >>> >> few >>> >> near to 1MB lines there. (I can provide file with same line lengths if >>> >> somebody interested.) What I have been observed is that reading this >>> >> file >>> >> using raw file:read_line/1 takes 51s! For comparison I have tried some >>> >> different approaches and what I got (line_read_test:read_std/1 is >>> >> using >>> >> file:read_line/1): >>> >> >>> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). >>> >> {51028105,2408} >>> >> 2> timer:tc(line_read_test,read,["test.txt"]). >>> >> {226220,2408} >>> >> 3> timer:tc(line_read_test,read_port,["test.txt"]). >>> >> {139388,2408} >>> >> >>> >> $time perl -nE'$i++}{say $i' test.txt >>> >> 2408 >>> >> >>> >> real 0m0.053s >>> >> user 0m0.044s >>> >> sys 0m0.008s >>> >> >>> >> $ time wc -l test.txt >>> >> 2408 test.txt >>> >> >>> >> real 0m0.013s >>> >> user 0m0.004s >>> >> sys 0m0.008s >>> >> >>> >> $ time ./a.out test.txt >>> >> 2408 >>> >> >>> >> real 0m0.020s >>> >> user 0m0.012s >>> >> sys 0m0.008s >>> >> >>> >> It means erlang should be at least 225 times faster >>> >> (line_read_test:read/1 >>> >> which has flow control). Erlang can be 350 times faster >>> >> (line_read_test:read_port/1 without flow control). Another high level >>> >> language (perl) is almost thousand times faster. Special C program is >>> >> almost >>> >> four thousands times faster and old good glibc is two and half >>> >> thousands >>> >> times faster. Come on guys it is not even fun when simple (and wrong) >>> >> erlang >>> >> wrapper around standard module is more than two order of magnitude >>> >> faster. >>> >> From mine experience when there is something two orders of magnitude >>> >> slower >>> >> it tells me there is something damn wrong. I have looked into >>> >> efile_drv.c >>> >> and >>> >> ti is unfortunately far beyond mine C skill but if simple buffering >>> >> and >>> >> binary:match/2 can outperform it 225 times there has to be something >>> >> rotten >>> >> in this code. >>> >> >>> >> I have also experimented with read_ahead option in file:open and >>> >> changing >>> >> to >>> >> less value makes thing worse. >>> >> >>> >> Just to make grasp how bad it is, in same time I'm able sort 150 >>> >> million >>> >> 64bit values (1.2GB of data) three times (one CPU core same HW). It is >>> >> not >>> >> in >>> >> flow control, mine simple wrapper does flow control too. It can't make >>> >> current code less intrusive, if it consumes 100% CPU for 51s instead >>> >> of >>> >> 226ms >>> >> then it will definitely affect whole server. It is not in concurrent >>> >> access, >>> >> mine code allows concurrent access too. Admitting there is something >>> >> broken >>> >> is first step to fixing it. I hope I helped at least in this way. >>> >> >>> >> With best regards >>> >> Hynek Vychodil >>> >> >>> >> _______________________________________________ >>> >> erlang-questions mailing list >>> >> erlang-questions@REDACTED >>> >> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >>> > >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> > From jeremy@REDACTED Sat Feb 16 02:40:00 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Fri, 15 Feb 2013 17:40:00 -0800 Subject: [erlang-questions] Spoof source ip address with ssl Message-ID: I noticed that gen_tcp supplies the {ip, IPAddr} option to allow the client to specify the source ip address. Does a similar option exist for ssl? I'm running into an issue with running out of ephemeral ports in load testing my application from a single machine. Thanks, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sat Feb 16 05:58:26 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sat, 16 Feb 2013 05:58:26 +0100 Subject: [erlang-questions] messages manipulatio In-Reply-To: References: Message-ID: I must confess that (at least at this late hour) your Step 2 confuses me. Why not just 1+3+4, or indeed *just* step 2? Regards, /Erik Den 15/02/2013 16.47 skrev "sasa" : > Hello, > > A while ago I encountered the following situation: > > I had the gen_server base process P which would receive messages, and > handle them by sending some data over the network. The messages were coming > faster than they were being sent. I established the reason for this was the > "randomness" of my network conditions. I also established that sending more > messages at once was almost as fast as sending one message, i.e. the > network push time wasn't highly dependent on the message size. > > To tackle this in a generic way, I devised an approach which has served me > well in multiple places. I was repeatedly googling whether some similar > solution exists, but I couldn't find it. Now, I'm not sure if I have > reinvented a wheel, or the approach is not optimal, so I'm asking if you > are aware of similar approaches, and are there any faults in this one? > > The approach I took is following: > > I split the server in two processes: the "facade" and the worker. The > facade acceptes requests from clients, and stores them internally. While > the worker is doing its work, new messages are stored in the facade. When > the worker is available, it will take all accumulated messages from the > facade and process them. > > These are the steps: > 1. The facade receives messages, stores data in its list, and notifies the > worker (without sending actual data), that some work is ready. > 2. Upon receiving the notification, the worker first flushes its message > queue by doing repeated receive ... after 0 as long as there are messages > in the queue. > 3. Then the worker pulls all messages from the facade. This is a > gen_server:call to the facade which will return all messages, and at the > same time remove them from its state. > 4. Finally, the worker processes all messages. > > > I found this approach useful because the delay on the worker adapts to the > incoming message rate. > If the worker can handle messages at the incoming rate, everything works > without delay. > If messages can't be handled at the incoming rate, the worker's delay will > increase to accomodate the higher load. In other words, the worker will try > to compensate the load by bulk processing messages. Obviously, this is > useful only when process_time(N messages) < N * process_time(1 message). > > Another benefit I found is that I can vary the implementation of the > facade i.e. I can store messages using different algorithms. In the first > implementation, I stored messages in a list. In one variation, I used hash > which allowed me to eliminate duplicate messages. Another variant was > truncation of the list, which allowed me to discard old messages if the > queue was getting too large. > > As I said, this has served me well in the production for more than a year, > and I have finally found the time to make a generic library out of it. > Before putting it public, I'd like to check if there are similar solutions, > or alternative approaches? > > Thanks, and best regards, > Sasa > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yashgt@REDACTED Sat Feb 16 11:40:44 2013 From: yashgt@REDACTED (yashgt@REDACTED) Date: Sat, 16 Feb 2013 10:40:44 +0000 Subject: [erlang-questions] How to detect connection requested by cnode? Message-ID: <511f6251.04a53a0a.6513.6de5@mx.google.com> An HTML attachment was scrubbed... URL: From sg2342@REDACTED Sat Feb 16 12:29:33 2013 From: sg2342@REDACTED (Stefan Grundmann) Date: Sat, 16 Feb 2013 11:29:33 +0000 Subject: [erlang-questions] Spoof source ip address with ssl In-Reply-To: References: Message-ID: <20130216112933.GA1403@bath> On Fri, Feb 15, 2013 at 05:40:00PM -0800, Jeremy Ong wrote: > I noticed that gen_tcp supplies the {ip, IPAddr} option to allow the client > to specify the source ip address. > > Does a similar option exist for ssl? I'm running into an issue with running > out of ephemeral ports in load testing my application from a single machine. yes. it is the same option. and in fact since gen_tcp is the (default) transport module used when calling ssl:connect/3,4 ... it is the same code. sg From sasa555@REDACTED Sat Feb 16 13:49:01 2013 From: sasa555@REDACTED (sasa) Date: Sat, 16 Feb 2013 13:49:01 +0100 Subject: [erlang-questions] messages manipulatio In-Reply-To: References: Message-ID: The step 2 is the consequence of the naive notification of the worker from the facade. Every time a facade receives a request it notifies the worker there is something. Imagine 1000 requests arrive while the worker is working. When the worker performs its current job, it will take the next message, and pull all data from the facade. Then it will do 999 of "dummy" processing, because all the work was already done in the processing of the 1st message. That is why, in my implementation, the worker flushes all incoming messages, then pulls all data from the facade. Admittedly, this could be avoided by introducing a more elaborate communication between the worker and the facade. In my mechanism, the facade is not aware whether the worker is doing some processing or not, so it blindly sends it a message each time it receives a processing request. On Sat, Feb 16, 2013 at 5:58 AM, Erik S?e S?rensen wrote: > I must confess that (at least at this late hour) your Step 2 confuses me. > Why not just 1+3+4, or indeed *just* step 2? > > Regards, > /Erik > Den 15/02/2013 16.47 skrev "sasa" : > >> Hello, >> >> A while ago I encountered the following situation: >> >> I had the gen_server base process P which would receive messages, and >> handle them by sending some data over the network. The messages were coming >> faster than they were being sent. I established the reason for this was the >> "randomness" of my network conditions. I also established that sending more >> messages at once was almost as fast as sending one message, i.e. the >> network push time wasn't highly dependent on the message size. >> >> To tackle this in a generic way, I devised an approach which has served >> me well in multiple places. I was repeatedly googling whether some similar >> solution exists, but I couldn't find it. Now, I'm not sure if I have >> reinvented a wheel, or the approach is not optimal, so I'm asking if you >> are aware of similar approaches, and are there any faults in this one? >> >> The approach I took is following: >> >> I split the server in two processes: the "facade" and the worker. The >> facade acceptes requests from clients, and stores them internally. While >> the worker is doing its work, new messages are stored in the facade. When >> the worker is available, it will take all accumulated messages from the >> facade and process them. >> >> These are the steps: >> 1. The facade receives messages, stores data in its list, and notifies >> the worker (without sending actual data), that some work is ready. >> 2. Upon receiving the notification, the worker first flushes its message >> queue by doing repeated receive ... after 0 as long as there are messages >> in the queue. >> 3. Then the worker pulls all messages from the facade. This is a >> gen_server:call to the facade which will return all messages, and at the >> same time remove them from its state. >> 4. Finally, the worker processes all messages. >> >> >> I found this approach useful because the delay on the worker adapts to >> the incoming message rate. >> If the worker can handle messages at the incoming rate, everything works >> without delay. >> If messages can't be handled at the incoming rate, the worker's delay >> will increase to accomodate the higher load. In other words, the worker >> will try to compensate the load by bulk processing messages. Obviously, >> this is useful only when process_time(N messages) < N * process_time(1 >> message). >> >> Another benefit I found is that I can vary the implementation of the >> facade i.e. I can store messages using different algorithms. In the first >> implementation, I stored messages in a list. In one variation, I used hash >> which allowed me to eliminate duplicate messages. Another variant was >> truncation of the list, which allowed me to discard old messages if the >> queue was getting too large. >> >> As I said, this has served me well in the production for more than a >> year, and I have finally found the time to make a generic library out of >> it. Before putting it public, I'd like to check if there are similar >> solutions, or alternative approaches? >> >> Thanks, and best regards, >> Sasa >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sasa555@REDACTED Sat Feb 16 13:49:46 2013 From: sasa555@REDACTED (sasa) Date: Sat, 16 Feb 2013 13:49:46 +0100 Subject: [erlang-questions] messages manipulatio In-Reply-To: <5AEB405C-E580-4D8F-B907-EE014AA581D9@dieswaytoofast.com> References: <5AEB405C-E580-4D8F-B907-EE014AA581D9@dieswaytoofast.com> Message-ID: Thanks! I'll take a look. On Sat, Feb 16, 2013 at 7:14 AM, Mahesh Paolini-Subramanya wrote: > Its pretty much a classic Active Queue Management issue, > i.e., its not the quantity of the traffic, but the "burst-y" nature of the > traffic that can confound you. > > There has been any amount of work done on this in the Networking field > (just google any combination of RED, CoDel, AQM, and Van Jacobsen), but its > only recently that some of the lessons have started making their way into > the software/"cloud" field. (I gave a preso' about this at TechMesh last > year > ?) > > For an erlang-baed implementation, take a look at Jesper Anderson's * > safety* valve. > It may not be distributed, but its certainly rigorously tested? > > Cheers > * > Mahesh Paolini-Subramanya > * > That Tall Bald Indian Guy... > Google+ | Blog > | Twitter | LinkedIn > > On Feb 15, 2013, at 9:17 PM, sasa wrote: > > Hello, > > A while ago I encountered the following situation: > > I had the gen_server base process P which would receive messages, and > handle them by sending some data over the network. The messages were coming > faster than they were being sent. I established the reason for this was the > "randomness" of my network conditions. I also established that sending more > messages at once was almost as fast as sending one message, i.e. the > network push time wasn't highly dependent on the message size. > > To tackle this in a generic way, I devised an approach which has served me > well in multiple places. I was repeatedly googling whether some similar > solution exists, but I couldn't find it. Now, I'm not sure if I have > reinvented a wheel, or the approach is not optimal, so I'm asking if you > are aware of similar approaches, and are there any faults in this one? > > The approach I took is following: > > I split the server in two processes: the "facade" and the worker. The > facade acceptes requests from clients, and stores them internally. While > the worker is doing its work, new messages are stored in the facade. When > the worker is available, it will take all accumulated messages from the > facade and process them. > > These are the steps: > 1. The facade receives messages, stores data in its list, and notifies the > worker (without sending actual data), that some work is ready. > 2. Upon receiving the notification, the worker first flushes its message > queue by doing repeated receive ... after 0 as long as there are messages > in the queue. > 3. Then the worker pulls all messages from the facade. This is a > gen_server:call to the facade which will return all messages, and at the > same time remove them from its state. > 4. Finally, the worker processes all messages. > > > I found this approach useful because the delay on the worker adapts to the > incoming message rate. > If the worker can handle messages at the incoming rate, everything works > without delay. > If messages can't be handled at the incoming rate, the worker's delay will > increase to accomodate the higher load. In other words, the worker will try > to compensate the load by bulk processing messages. Obviously, this is > useful only when process_time(N messages) < N * process_time(1 message). > > Another benefit I found is that I can vary the implementation of the > facade i.e. I can store messages using different algorithms. In the first > implementation, I stored messages in a list. In one variation, I used hash > which allowed me to eliminate duplicate messages. Another variant was > truncation of the list, which allowed me to discard old messages if the > queue was getting too large. > > As I said, this has served me well in the production for more than a year, > and I have finally found the time to make a generic library out of it. > Before putting it public, I'd like to check if there are similar solutions, > or alternative approaches? > > Thanks, and best regards, > Sasa > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sasa555@REDACTED Sat Feb 16 17:01:48 2013 From: sasa555@REDACTED (sasa) Date: Sat, 16 Feb 2013 17:01:48 +0100 Subject: [erlang-questions] messages manipulatio In-Reply-To: References: Message-ID: Having thought more, I realized the original approach can be simplified. This would be the modified version: 1. The facade keeps track whether the worker is available or it is working. 2. Upon receiving a message, the facade joins it with the messages already in the queue. Then, if the worker is available, it immediately sends data to it, and clears its own queue. 3. When the worker receives the notification, it performs the work, then it notifies the facade that it is available. 4. When the facade receives the information that the worker is available, if there is new data available, it will immediately send it to the worker as explained in 2. Thank you for your question, it had made me realized there is place for simplification. On Sat, Feb 16, 2013 at 5:58 AM, Erik S?e S?rensen wrote: > I must confess that (at least at this late hour) your Step 2 confuses me. > Why not just 1+3+4, or indeed *just* step 2? > > Regards, > /Erik > Den 15/02/2013 16.47 skrev "sasa" : > >> Hello, >> >> A while ago I encountered the following situation: >> >> I had the gen_server base process P which would receive messages, and >> handle them by sending some data over the network. The messages were coming >> faster than they were being sent. I established the reason for this was the >> "randomness" of my network conditions. I also established that sending more >> messages at once was almost as fast as sending one message, i.e. the >> network push time wasn't highly dependent on the message size. >> >> To tackle this in a generic way, I devised an approach which has served >> me well in multiple places. I was repeatedly googling whether some similar >> solution exists, but I couldn't find it. Now, I'm not sure if I have >> reinvented a wheel, or the approach is not optimal, so I'm asking if you >> are aware of similar approaches, and are there any faults in this one? >> >> The approach I took is following: >> >> I split the server in two processes: the "facade" and the worker. The >> facade acceptes requests from clients, and stores them internally. While >> the worker is doing its work, new messages are stored in the facade. When >> the worker is available, it will take all accumulated messages from the >> facade and process them. >> >> These are the steps: >> 1. The facade receives messages, stores data in its list, and notifies >> the worker (without sending actual data), that some work is ready. >> 2. Upon receiving the notification, the worker first flushes its message >> queue by doing repeated receive ... after 0 as long as there are messages >> in the queue. >> 3. Then the worker pulls all messages from the facade. This is a >> gen_server:call to the facade which will return all messages, and at the >> same time remove them from its state. >> 4. Finally, the worker processes all messages. >> >> >> I found this approach useful because the delay on the worker adapts to >> the incoming message rate. >> If the worker can handle messages at the incoming rate, everything works >> without delay. >> If messages can't be handled at the incoming rate, the worker's delay >> will increase to accomodate the higher load. In other words, the worker >> will try to compensate the load by bulk processing messages. Obviously, >> this is useful only when process_time(N messages) < N * process_time(1 >> message). >> >> Another benefit I found is that I can vary the implementation of the >> facade i.e. I can store messages using different algorithms. In the first >> implementation, I stored messages in a list. In one variation, I used hash >> which allowed me to eliminate duplicate messages. Another variant was >> truncation of the list, which allowed me to discard old messages if the >> queue was getting too large. >> >> As I said, this has served me well in the production for more than a >> year, and I have finally found the time to make a generic library out of >> it. Before putting it public, I'd like to check if there are similar >> solutions, or alternative approaches? >> >> Thanks, and best regards, >> Sasa >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhssler@REDACTED Sat Feb 16 20:33:04 2013 From: mhssler@REDACTED (=?ISO-8859-1?Q?Martin_H=E4ssler?=) Date: Sat, 16 Feb 2013 20:33:04 +0100 Subject: [erlang-questions] How to detect connection requested by cnode? Message-ID: On Sat, Feb 16, 2013 at 11:40 AM, wrote: > When a remote C-node connects to an Erlang node using ei_connect or > erl_connect, I would like the erlang node to do something, like print the > name of the c-node that requested the connection. Is this what you want? ok = net_kernel:monitor_nodes(true, [{node_type, all}]), receive {nodeup, Node, _} -> io:format("Node up: ~p~n", [Node]) end. From erlang@REDACTED Sat Feb 16 21:11:39 2013 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 16 Feb 2013 21:11:39 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: On Fri, Feb 15, 2013 at 4:58 PM, Lukas Larsson wrote: > Oh, and here are the benchmarks for me now: > > R15B03: > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > {21307874,14448} > > R16B with patch: > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > {91885,14448} > > It's not every day you find a two line change which increases speed by a > factor of 232! > > Lukas > Great - well patched /Joe > > > On Fri, Feb 15, 2013 at 4:42 PM, Lukas Larsson > wrote: > >> aha, you needed to have very long lines in the file to provoke the >> behaviour. The file I was experimenting on had the same line length all the >> time. >> >> I think I've found the culprit. Aparently the read_ahead option was >> ignored after the first buffer had been consumed. Try the patch below and >> see if it works for you: >> >> From 0884480691e047801cef117b19151a7d34c9e438 Mon Sep 17 00:00:00 2001 >> From: Lukas Larsson >> Date: Fri, 15 Feb 2013 16:37:52 +0100 >> Subject: [PATCH] invoke_read_line now remembers read_ahead settings >> >> --- >> erts/emulator/drivers/common/efile_drv.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/erts/emulator/drivers/common/efile_drv.c >> b/erts/emulator/drivers/common/efile_drv.c >> index 2279fec..d81f116 100644 >> --- a/erts/emulator/drivers/common/efile_drv.c >> +++ b/erts/emulator/drivers/common/efile_drv.c >> @@ -1160,6 +1160,8 @@ static void invoke_read_line(void *data) >> /* Need more place */ >> ErlDrvSizeT need = (d->c.read_line.read_size >= >> DEFAULT_LINEBUF_SIZE) ? >> d->c.read_line.read_size + DEFAULT_LINEBUF_SIZE : >> DEFAULT_LINEBUF_SIZE; >> + if (need < (d->c.read_line.binp)->orig_size) >> + need = (d->c.read_line.binp)->orig_size; >> ErlDrvBinary *newbin = driver_alloc_binary(need); >> if (newbin == NULL) { >> d->result_ok = 0; >> -- >> 1.7.10.4 >> >> >> >> >> On Fri, Feb 15, 2013 at 1:29 AM, Hynek Vychodil > > wrote: >> >>> Using async thread drops time down to 4.6s which is still 20 times >>> slower than mine simple wrapper. >>> >>> On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson wrote: >>> > Hello! >>> > >>> > If you start erlang with an async thread (erl +A 1) you will see >>> radically >>> > different behaviour. For me the time it takes to use read_line drops >>> from >>> > 58s to 2s. When doing file i/o you should always have some async >>> threads to >>> > help you do the work, this is why in R16B we changed the default from >>> 0 to >>> > 10 async threads. >>> > >>> > Lukas >>> > >>> > >>> > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil < >>> vychodil.hynek@REDACTED> >>> > wrote: >>> >> >>> >> Hello, >>> >> I know it was been already discussed here in list and it is also >>> recurring >>> >> topic for at least five years. But anyway I have been bitten by it >>> again >>> >> and >>> >> also found pretty pathological case. I have 30MB text file and it has >>> a >>> >> few >>> >> near to 1MB lines there. (I can provide file with same line lengths if >>> >> somebody interested.) What I have been observed is that reading this >>> file >>> >> using raw file:read_line/1 takes 51s! For comparison I have tried some >>> >> different approaches and what I got (line_read_test:read_std/1 is >>> using >>> >> file:read_line/1): >>> >> >>> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). >>> >> {51028105,2408} >>> >> 2> timer:tc(line_read_test,read,["test.txt"]). >>> >> {226220,2408} >>> >> 3> timer:tc(line_read_test,read_port,["test.txt"]). >>> >> {139388,2408} >>> >> >>> >> $time perl -nE'$i++}{say $i' test.txt >>> >> 2408 >>> >> >>> >> real 0m0.053s >>> >> user 0m0.044s >>> >> sys 0m0.008s >>> >> >>> >> $ time wc -l test.txt >>> >> 2408 test.txt >>> >> >>> >> real 0m0.013s >>> >> user 0m0.004s >>> >> sys 0m0.008s >>> >> >>> >> $ time ./a.out test.txt >>> >> 2408 >>> >> >>> >> real 0m0.020s >>> >> user 0m0.012s >>> >> sys 0m0.008s >>> >> >>> >> It means erlang should be at least 225 times faster >>> (line_read_test:read/1 >>> >> which has flow control). Erlang can be 350 times faster >>> >> (line_read_test:read_port/1 without flow control). Another high level >>> >> language (perl) is almost thousand times faster. Special C program is >>> >> almost >>> >> four thousands times faster and old good glibc is two and half >>> thousands >>> >> times faster. Come on guys it is not even fun when simple (and wrong) >>> >> erlang >>> >> wrapper around standard module is more than two order of magnitude >>> faster. >>> >> From mine experience when there is something two orders of magnitude >>> >> slower >>> >> it tells me there is something damn wrong. I have looked into >>> efile_drv.c >>> >> and >>> >> ti is unfortunately far beyond mine C skill but if simple buffering >>> and >>> >> binary:match/2 can outperform it 225 times there has to be something >>> >> rotten >>> >> in this code. >>> >> >>> >> I have also experimented with read_ahead option in file:open and >>> changing >>> >> to >>> >> less value makes thing worse. >>> >> >>> >> Just to make grasp how bad it is, in same time I'm able sort 150 >>> million >>> >> 64bit values (1.2GB of data) three times (one CPU core same HW). It >>> is not >>> >> in >>> >> flow control, mine simple wrapper does flow control too. It can't make >>> >> current code less intrusive, if it consumes 100% CPU for 51s instead >>> of >>> >> 226ms >>> >> then it will definitely affect whole server. It is not in concurrent >>> >> access, >>> >> mine code allows concurrent access too. Admitting there is something >>> >> broken >>> >> is first step to fixing it. I hope I helped at least in this way. >>> >> >>> >> With best regards >>> >> Hynek Vychodil >>> >> >>> >> _______________________________________________ >>> >> erlang-questions mailing list >>> >> erlang-questions@REDACTED >>> >> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >>> > >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sat Feb 16 21:14:36 2013 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 16 Feb 2013 21:14:36 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 12:57 AM, Hynek Vychodil wrote: > It works like charm. I have patched R15B03 ad with 1MB read ahead it > is unbelievable fast > > 20> timer:tc(line_read_test,read_std,["test.txt"]). > {30732,2408} > > It is even four times faster than port version > > 24> timer:tc(line_read_test,read_port,["test.txt"]). > {135676,2408} > > It is also faster without async threads. Measurements above was with > +A 10. Without async threads: > > 2> timer:tc(line_read_test,read_std,["test.txt"]). > {19322,2408} > > I was expected time around 100ms but current behavior is far beyond > expectation. I hope this patch will find way into upcoming R16. It > seems to speed up line oriented io for "normal" files two times also. > > Thanks a lot > Hynek Vychodil > And thanks to you - if we don't know it's broken we can't fix it - so all observations of weird behaviour are worthy of note. Cheers /Joe > > On Fri, Feb 15, 2013 at 4:58 PM, Lukas Larsson > wrote: > > Oh, and here are the benchmarks for me now: > > > > R15B03: > > > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > > {21307874,14448} > > > > R16B with patch: > > > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > > {91885,14448} > > > > It's not every day you find a two line change which increases speed by a > > factor of 232! > > > > Lukas > > > > > > On Fri, Feb 15, 2013 at 4:42 PM, Lukas Larsson < > lukas@REDACTED> > > wrote: > >> > >> aha, you needed to have very long lines in the file to provoke the > >> behaviour. The file I was experimenting on had the same line length all > the > >> time. > >> > >> I think I've found the culprit. Aparently the read_ahead option was > >> ignored after the first buffer had been consumed. Try the patch below > and > >> see if it works for you: > >> > >> From 0884480691e047801cef117b19151a7d34c9e438 Mon Sep 17 00:00:00 2001 > >> From: Lukas Larsson > >> Date: Fri, 15 Feb 2013 16:37:52 +0100 > >> Subject: [PATCH] invoke_read_line now remembers read_ahead settings > >> > >> --- > >> erts/emulator/drivers/common/efile_drv.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/erts/emulator/drivers/common/efile_drv.c > >> b/erts/emulator/drivers/common/efile_drv.c > >> index 2279fec..d81f116 100644 > >> --- a/erts/emulator/drivers/common/efile_drv.c > >> +++ b/erts/emulator/drivers/common/efile_drv.c > >> @@ -1160,6 +1160,8 @@ static void invoke_read_line(void *data) > >> /* Need more place */ > >> ErlDrvSizeT need = (d->c.read_line.read_size >= > >> DEFAULT_LINEBUF_SIZE) ? > >> d->c.read_line.read_size + DEFAULT_LINEBUF_SIZE : > >> DEFAULT_LINEBUF_SIZE; > >> + if (need < (d->c.read_line.binp)->orig_size) > >> + need = (d->c.read_line.binp)->orig_size; > >> ErlDrvBinary *newbin = driver_alloc_binary(need); > >> if (newbin == NULL) { > >> d->result_ok = 0; > >> -- > >> 1.7.10.4 > >> > >> > >> > >> > >> On Fri, Feb 15, 2013 at 1:29 AM, Hynek Vychodil < > vychodil.hynek@REDACTED> > >> wrote: > >>> > >>> Using async thread drops time down to 4.6s which is still 20 times > >>> slower than mine simple wrapper. > >>> > >>> On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson > wrote: > >>> > Hello! > >>> > > >>> > If you start erlang with an async thread (erl +A 1) you will see > >>> > radically > >>> > different behaviour. For me the time it takes to use read_line drops > >>> > from > >>> > 58s to 2s. When doing file i/o you should always have some async > >>> > threads to > >>> > help you do the work, this is why in R16B we changed the default > from 0 > >>> > to > >>> > 10 async threads. > >>> > > >>> > Lukas > >>> > > >>> > > >>> > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil > >>> > > >>> > wrote: > >>> >> > >>> >> Hello, > >>> >> I know it was been already discussed here in list and it is also > >>> >> recurring > >>> >> topic for at least five years. But anyway I have been bitten by it > >>> >> again > >>> >> and > >>> >> also found pretty pathological case. I have 30MB text file and it > has > >>> >> a > >>> >> few > >>> >> near to 1MB lines there. (I can provide file with same line lengths > if > >>> >> somebody interested.) What I have been observed is that reading this > >>> >> file > >>> >> using raw file:read_line/1 takes 51s! For comparison I have tried > some > >>> >> different approaches and what I got (line_read_test:read_std/1 is > >>> >> using > >>> >> file:read_line/1): > >>> >> > >>> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). > >>> >> {51028105,2408} > >>> >> 2> timer:tc(line_read_test,read,["test.txt"]). > >>> >> {226220,2408} > >>> >> 3> timer:tc(line_read_test,read_port,["test.txt"]). > >>> >> {139388,2408} > >>> >> > >>> >> $time perl -nE'$i++}{say $i' test.txt > >>> >> 2408 > >>> >> > >>> >> real 0m0.053s > >>> >> user 0m0.044s > >>> >> sys 0m0.008s > >>> >> > >>> >> $ time wc -l test.txt > >>> >> 2408 test.txt > >>> >> > >>> >> real 0m0.013s > >>> >> user 0m0.004s > >>> >> sys 0m0.008s > >>> >> > >>> >> $ time ./a.out test.txt > >>> >> 2408 > >>> >> > >>> >> real 0m0.020s > >>> >> user 0m0.012s > >>> >> sys 0m0.008s > >>> >> > >>> >> It means erlang should be at least 225 times faster > >>> >> (line_read_test:read/1 > >>> >> which has flow control). Erlang can be 350 times faster > >>> >> (line_read_test:read_port/1 without flow control). Another high > level > >>> >> language (perl) is almost thousand times faster. Special C program > is > >>> >> almost > >>> >> four thousands times faster and old good glibc is two and half > >>> >> thousands > >>> >> times faster. Come on guys it is not even fun when simple (and > wrong) > >>> >> erlang > >>> >> wrapper around standard module is more than two order of magnitude > >>> >> faster. > >>> >> From mine experience when there is something two orders of magnitude > >>> >> slower > >>> >> it tells me there is something damn wrong. I have looked into > >>> >> efile_drv.c > >>> >> and > >>> >> ti is unfortunately far beyond mine C skill but if simple buffering > >>> >> and > >>> >> binary:match/2 can outperform it 225 times there has to be something > >>> >> rotten > >>> >> in this code. > >>> >> > >>> >> I have also experimented with read_ahead option in file:open and > >>> >> changing > >>> >> to > >>> >> less value makes thing worse. > >>> >> > >>> >> Just to make grasp how bad it is, in same time I'm able sort 150 > >>> >> million > >>> >> 64bit values (1.2GB of data) three times (one CPU core same HW). It > is > >>> >> not > >>> >> in > >>> >> flow control, mine simple wrapper does flow control too. It can't > make > >>> >> current code less intrusive, if it consumes 100% CPU for 51s instead > >>> >> of > >>> >> 226ms > >>> >> then it will definitely affect whole server. It is not in concurrent > >>> >> access, > >>> >> mine code allows concurrent access too. Admitting there is something > >>> >> broken > >>> >> is first step to fixing it. I hope I helped at least in this way. > >>> >> > >>> >> With best regards > >>> >> Hynek Vychodil > >>> >> > >>> >> _______________________________________________ > >>> >> erlang-questions mailing list > >>> >> erlang-questions@REDACTED > >>> >> http://erlang.org/mailman/listinfo/erlang-questions > >>> >> > >>> > > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> > >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drew.varner@REDACTED Sun Feb 17 14:09:29 2013 From: drew.varner@REDACTED (Drew Varner) Date: Sun, 17 Feb 2013 17:09:29 +0400 Subject: [erlang-questions] Update to public_key in anticipation of CRL support Message-ID: <75FE6332-DAC8-4B19-AECB-E5FAAB288E57@redops.org> It looks like CRL support will enter Erlang in the future [1]. CRLs sometimes use the signature algorithm ID 1.3.14.3.2.29 for SHA instead of 1.2.840.113549.1.1.5 [2][3]. Can we update pubkey_cert.erl to understand this OID [4], even though CRL functionality isn't in place yet? It might be useful in other situations also. Thanks, Drew [1] http://erlang.org/pipermail/erlang-questions/2011-August/060639.html [2] http://javadoc.iaik.tugraz.at/iaik_jce/current/iaik/asn1/structures/AlgorithmID.html [3] https://bugzilla.mozilla.org/show_bug.cgi?id=405966 [4] https://github.com/erlang/otp/blob/maint/lib/public_key/src/pubkey_cert.erl#L377 From anotherworldofworld@REDACTED Sun Feb 17 15:36:15 2013 From: anotherworldofworld@REDACTED (Alex toyer) Date: Sun, 17 Feb 2013 20:36:15 +0600 Subject: [erlang-questions] [ANN] Ybot chat robot 0.3.1 released Message-ID: Hello erlagers, Today i released Ybot-0.3.1 version. Ybot is chat robot which inspired with Github Hubot, but built with erlang/otp. New in this version: * Added reconnect timeout option and reconnect ability to all clients --> #33. * hacker_help plugin fixed --> #35 * Added supporting of HipChat. * HTTP transport imrpoved. JSON request support added. * HTTP transport bot-nick parameter added to config. * ybot_manager get_all_transports_pid api added. * Use cowboy web server instead inets httpd for http serving. * Added http PUT request support. Resend request body from http to all runned transports. * echo plugin added. * Flowdock support added. * Skype support added. * Removed http PUT request supporting. Now only POST. * New transport - Talkerapp (http://talkerapp.com/rooms). * Message parsers moved to ybot_parser module. Ybot at github - https://github.com/0xAX/Ybot Ybot plugins repository - https://github.com/0xAX/ybot-contrib Ybot documentation in one place - http://0xax.github.com/Ybot.html if you will have any questions, suggestions, ideas or something else write me to twitter: 0xAX or at github. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gabre@REDACTED Sun Feb 17 18:52:01 2013 From: gabre@REDACTED (=?iso-8859-2?b?SG9zc3r6IA==?= =?iso-8859-2?b?R+Fib3I=?=) Date: Sun, 17 Feb 2013 18:52:01 +0100 Subject: [erlang-questions] Wx Erlang - wxStyledTextCtrl problem Message-ID: <20130217185201.14106pdpcw1kaw1t@webmail.elte.hu> Hello, I tried to write a code browser using wxStyledTextCtrl but I encountered a strange problem. I connected my wxStyledTextCtrl object to 'stc_doubleclick' because I need to know when the user double clicks (and I also need its position, etc...). When I tested my program I saw that on double click no position information was given, and every field of the received wxStyledText record was 0 (anywhere I clicked): {wxStyledText,stc_doubleclick,0,0,0,0,[],0,0,0,0,0,0,0,0,0,0,0,0,[], false,1} The Wx/Scintilla documentation states the following (in section 'Notifications'): SCN_DOUBLECLICK The mouse button was double clicked in editor. The position field is set to the text position of the double click, the line field is set to the line of the double click, and the modifiers field is set to the key modifiers held down in a similar manner to SCN_KEY. My question/problem is that why the fields aren't set? Is this a bug? Thanks, and best regards, Gabor Hosszu From max.lapshin@REDACTED Sun Feb 17 19:45:17 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Sun, 17 Feb 2013 21:45:17 +0300 Subject: [erlang-questions] ** dets: Bug was found when accessing table 'index_001-040-030' Message-ID: Is it a bug in dets or it is a broken data? Should I post any other information such as data file itself? -------------- next part -------------- An HTML attachment was scrubbed... URL: From z@REDACTED Sun Feb 17 20:18:37 2013 From: z@REDACTED (Danil Zagoskin) Date: Sun, 17 Feb 2013 23:18:37 +0400 Subject: [erlang-questions] [ANN] autohelp: include documentation in your modules Message-ID: Hello! I'd like to announce autohelp parse_transform. It's goal is to be non-intrusive but useful. When module is compiled with this parse_transform, functions help/{0,1,2} are added. help/0 shows module description and function list, help/1 shows short descriptions for functions with given name, help/2 shows full description of function with given arity. autohelp tries to avoid conflicts and if something goes wrong prints warning and does not modify AST. It is available on Github: https://github.com/stolen/autohelp Hope you find it useful. -- --------------------------------------------- ????? ???????? | +7 906 064 20 47 | z@REDACTED -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasbean@REDACTED Mon Feb 18 09:07:20 2013 From: hasbean@REDACTED (Yousef A) Date: Mon, 18 Feb 2013 11:07:20 +0300 Subject: [erlang-questions] Process not receiving first message when using driver_send_term()? Message-ID: I've been trying to add features to liberlua, but I ran into this wall. I'm running a port_command on Erlang and receiving it with outputv in the driver, but not all driver_send_terms are received by the calling process. I'm using R15B03 (erts-5.9.3.1) This is the driver entry: static ErlDrvEntry lua_driver_entry = { NULL, /* init */ start, /* startup */ stop, /* shutdown */ NULL, /* output */ NULL, /* ready_input */ NULL, /* ready_output */ "liberlua", /* the name of the driver */ NULL, /* finish */ NULL, /* handle */ NULL, /* control */ NULL, /* timeout */ process, /* outputv */ NULL, /* ready_async */ NULL, /* flush */ NULL, /* call */ NULL, /* event */ ERL_DRV_EXTENDED_MARKER, /* ERL_DRV_EXTENDED_MARKER */ ERL_DRV_EXTENDED_MAJOR_VERSION, /* ERL_DRV_EXTENDED_MAJOR_VERSION */ ERL_DRV_EXTENDED_MINOR_VERSION, /* ERL_DRV_EXTENDED_MINOR_VERSION */ ERL_DRV_FLAG_USE_PORT_LOCKING /* ERL_DRV_FLAGs */ }; And the process function: static void process(ErlDrvData handle, ErlIOVec *ev) { lua_drv_t *driver_data = (lua_drv_t*) handle; ErlDrvTermData spec[] = { ERL_DRV_ATOM, driver_mk_atom((char*) "testing") }; //send 3 requests and see how many the process receives. driver_send_term(driver_data->port, driver_caller(driver_data->port), spec, sizeof(spec) / sizeof(spec[0])); driver_send_term(driver_data->port, driver_caller(driver_data->port), spec, sizeof(spec) / sizeof(spec[0])); driver_send_term(driver_data->port, driver_caller(driver_data->port), spec, sizeof(spec) / sizeof(spec[0])); } And for Erlang, it's a gen_server: handle_info(Info, State) -> io:format("~n~n========game sent bogus msg============~n ~p ~n~n", [Info]), {noreply, State}. When run, this is what the Erlang process prints out: ========game sent bogus msg============ testing ========game sent bogus msg============ testing And that's it, only 2 messages when 3 driver_send_term() requests are called. I must have missed something, because I don't see anyone else complaining about this problem. Thanks! --Yousef -------------- next part -------------- An HTML attachment was scrubbed... URL: From hasbean@REDACTED Mon Feb 18 09:33:04 2013 From: hasbean@REDACTED (Yousef A) Date: Mon, 18 Feb 2013 11:33:04 +0300 Subject: [erlang-questions] Process not receiving first message when using driver_send_term()? In-Reply-To: References: Message-ID: Please ignore this. Apparently, lua.erl has a "receive" part whenever I run lua:func() which explains why I keep missing the first one, duh. On Mon, Feb 18, 2013 at 11:07 AM, Yousef A wrote: > I've been trying to add features to liberlua, but I ran into this wall. > I'm running a port_command on Erlang and receiving it with outputv in the > driver, but not all driver_send_terms are received by the calling process. > I'm using R15B03 (erts-5.9.3.1) > > This is the driver entry: > > static ErlDrvEntry lua_driver_entry = { > > NULL, /* init */ > start, /* startup */ > stop, /* shutdown */ > NULL, /* output */ > NULL, /* ready_input */ > NULL, /* ready_output */ > "liberlua", /* the name of the driver */ > NULL, /* finish */ > NULL, /* handle */ > NULL, /* control */ > NULL, /* timeout */ > process, /* outputv */ > NULL, /* ready_async */ > NULL, /* flush */ > NULL, /* call */ > NULL, /* event */ > ERL_DRV_EXTENDED_MARKER, /* ERL_DRV_EXTENDED_MARKER */ > ERL_DRV_EXTENDED_MAJOR_VERSION, /* ERL_DRV_EXTENDED_MAJOR_VERSION */ > ERL_DRV_EXTENDED_MINOR_VERSION, /* ERL_DRV_EXTENDED_MINOR_VERSION */ > ERL_DRV_FLAG_USE_PORT_LOCKING /* ERL_DRV_FLAGs */ > }; > > And the process function: > > static void process(ErlDrvData handle, ErlIOVec *ev) > { > lua_drv_t *driver_data = (lua_drv_t*) handle; > > ErlDrvTermData spec[] = { > ERL_DRV_ATOM, driver_mk_atom((char*) "testing") > }; > > > //send 3 requests and see how many the process receives. > > driver_send_term(driver_data->port, driver_caller(driver_data->port), > spec, sizeof(spec) / sizeof(spec[0])); > driver_send_term(driver_data->port, driver_caller(driver_data->port), > spec, sizeof(spec) / sizeof(spec[0])); > driver_send_term(driver_data->port, driver_caller(driver_data->port), > spec, sizeof(spec) / sizeof(spec[0])); > > } > > And for Erlang, it's a gen_server: > > handle_info(Info, State) -> > io:format("~n~n========game sent bogus msg============~n ~p ~n~n", > [Info]), > {noreply, State}. > > > When run, this is what the Erlang process prints out: > > ========game sent bogus msg============ > testing > > > > ========game sent bogus msg============ > testing > > > > > And that's it, only 2 messages when 3 driver_send_term() requests are > called. I must have missed something, because I don't see anyone else > complaining about this problem. > > Thanks! > > --Yousef > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dangud@REDACTED Mon Feb 18 10:57:39 2013 From: dangud@REDACTED (Dan Gudmundsson) Date: Mon, 18 Feb 2013 10:57:39 +0100 Subject: [erlang-questions] Wx Erlang - wxStyledTextCtrl problem In-Reply-To: <20130217185201.14106pdpcw1kaw1t@webmail.elte.hu> References: <20130217185201.14106pdpcw1kaw1t@webmail.elte.hu> Message-ID: Sounds like a wxWidgets bug, you can workaround it by explicitly ask for mousestate. /Dan On Sun, Feb 17, 2013 at 6:52 PM, Hossz? G?bor wrote: > Hello, > > I tried to write a code browser using wxStyledTextCtrl but I encountered a > strange problem. > I connected my wxStyledTextCtrl object to 'stc_doubleclick' because I need > to know when the user double clicks (and I also need its position, etc...). > When I tested my program I saw that on double click no position information > was given, and every field of the received wxStyledText record was 0 > (anywhere I clicked): > {wxStyledText,stc_doubleclick,0,0,0,0,[],0,0,0,0,0,0,0,0,0,0,0,0,[], > false,1} > > The Wx/Scintilla documentation states the following (in section > 'Notifications'): > SCN_DOUBLECLICK > The mouse button was double clicked in editor. The position field is set to > the text position of the double click, the line field is set to the line of > the double click, and the modifiers field is set to the key modifiers held > down in a similar manner to SCN_KEY. > > My question/problem is that why the fields aren't set? Is this a bug? > > Thanks, and best regards, > Gabor Hosszu > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ingela.andin@REDACTED Mon Feb 18 11:10:11 2013 From: ingela.andin@REDACTED (Ingela Andin) Date: Mon, 18 Feb 2013 11:10:11 +0100 Subject: [erlang-questions] Update to public_key in anticipation of CRL support In-Reply-To: <75FE6332-DAC8-4B19-AECB-E5FAAB288E57@redops.org> References: <75FE6332-DAC8-4B19-AECB-E5FAAB288E57@redops.org> Message-ID: Hi! Sigh, why can they not just follow the PKCS standard. It is of course possible to let public_key recognise the other oids. We will look into this. CRL support in public_key is included in R16 although we have not had time to integrate the support in SSL yet. Regards Ingela Erlang/OTP team - Ericsson AB 2013/2/17, Drew Varner : > It looks like CRL support will enter Erlang in the future [1]. CRLs > sometimes use the signature algorithm ID 1.3.14.3.2.29 for SHA instead of > 1.2.840.113549.1.1.5 [2][3]. Can we update pubkey_cert.erl to understand > this OID [4], even though CRL functionality isn't in place yet? It might be > useful in other situations also. > > Thanks, > Drew > > [1] http://erlang.org/pipermail/erlang-questions/2011-August/060639.html > [2] > http://javadoc.iaik.tugraz.at/iaik_jce/current/iaik/asn1/structures/AlgorithmID.html > [3] https://bugzilla.mozilla.org/show_bug.cgi?id=405966 > [4] > https://github.com/erlang/otp/blob/maint/lib/public_key/src/pubkey_cert.erl#L377 > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From jinni.park@REDACTED Mon Feb 18 12:55:12 2013 From: jinni.park@REDACTED (Park, Sungjin) Date: Mon, 18 Feb 2013 20:55:12 +0900 Subject: [erlang-questions] About ssl connection speed. In-Reply-To: References: Message-ID: I used {backlog, 4096} as one of the options for the listener. Did I still miss something? On Fri, Feb 15, 2013 at 12:16 AM, Paul Oliver wrote: > Hi there, > > Check this thread > http://erlang.org/pipermail/erlang-questions/2012-April/065811.html > > Cheers, > Paul. > > > On Thu, Feb 14, 2013 at 3:10 AM, Park, Sungjin wrote: > >> Hello guys, >> >> I'm experiencing that erlang server's ssl listener is significantly >> slower - by a factor of hundreds - than plain tcp listener. To give you >> some numbers, my server accepts less than 100 ssl clients per second while >> it accepts more than 10k tcp clients per second with the same service logic >> behind save for the transport types. I tested the same scenario with a >> server written in c for comparison and got numbers about 1k/s in ssl and >> 10k in tcp. >> >> I want to make sure if this is inevitable as long as I'm using erlang or >> not. >> >> >> -- >> Park, Sungjin >> >> ------------------------------------------------------------------------------------------------------------------- >> Peculiar travel suggestions are dancing lessons from god. >> -- The Books of Bokonon >> >> ------------------------------------------------------------------------------------------------------------------- >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- Park, Sungjin ------------------------------------------------------------------------------------------------------------------- Peculiar travel suggestions are dancing lessons from god. -- The Books of Bokonon ------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.sabil@REDACTED Mon Feb 18 13:29:24 2013 From: ali.sabil@REDACTED (Ali Sabil) Date: Mon, 18 Feb 2013 13:29:24 +0100 Subject: [erlang-questions] About ssl connection speed. In-Reply-To: References: Message-ID: Did you check wich ciphers are being used in both scenarios? DHE ciphers are notably slow. You might want to pass the {ciphers, Ciphers} to the ssl listener, where Ciphers is: Ciphers = lists:filter(fun ({dhe_rsa, _, _}) -> false; ({dhe_dss, _, _}) -> false; (_) -> true end, ssl:cipher_suites()), Best, Ali On Mon, Feb 18, 2013 at 12:55 PM, Park, Sungjin wrote: > I used {backlog, 4096} as one of the options for the listener. > Did I still miss something? > > > On Fri, Feb 15, 2013 at 12:16 AM, Paul Oliver wrote: > >> Hi there, >> >> Check this thread >> http://erlang.org/pipermail/erlang-questions/2012-April/065811.html >> >> Cheers, >> Paul. >> >> >> On Thu, Feb 14, 2013 at 3:10 AM, Park, Sungjin wrote: >> >>> Hello guys, >>> >>> I'm experiencing that erlang server's ssl listener is significantly >>> slower - by a factor of hundreds - than plain tcp listener. To give you >>> some numbers, my server accepts less than 100 ssl clients per second while >>> it accepts more than 10k tcp clients per second with the same service logic >>> behind save for the transport types. I tested the same scenario with a >>> server written in c for comparison and got numbers about 1k/s in ssl and >>> 10k in tcp. >>> >>> I want to make sure if this is inevitable as long as I'm using erlang or >>> not. >>> >>> >>> -- >>> Park, Sungjin >>> >>> ------------------------------------------------------------------------------------------------------------------- >>> Peculiar travel suggestions are dancing lessons from god. >>> -- The Books of Bokonon >>> >>> ------------------------------------------------------------------------------------------------------------------- >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > -- > Park, Sungjin > > ------------------------------------------------------------------------------------------------------------------- > Peculiar travel suggestions are dancing lessons from god. > -- The Books of Bokonon > > ------------------------------------------------------------------------------------------------------------------- > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.bolinder@REDACTED Mon Feb 18 13:51:13 2013 From: hans.bolinder@REDACTED (Hans Bolinder) Date: Mon, 18 Feb 2013 12:51:13 +0000 Subject: [erlang-questions] ** dets: Bug was found when accessing table 'index_001-040-030' In-Reply-To: References: Message-ID: <56466BD70414EA48969B4064696CF28C03D6F8@ESESSMB207.ericsson.se> ________________________________ [Max Lapshin:] > Is it a bug in dets or it is a broken data? It's a bug in Dets. > Should I post any other information such as data file itself? Yes, please. You can mail me directly. Best regards, Hans Bolinder, Erlang/OTP, Ericsson AB -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.bolinder@REDACTED Mon Feb 18 14:11:19 2013 From: hans.bolinder@REDACTED (Hans Bolinder) Date: Mon, 18 Feb 2013 13:11:19 +0000 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <510BDD44.20007@aleynikov.org> References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com>, <510BDD44.20007@aleynikov.org> Message-ID: <56466BD70414EA48969B4064696CF28C03D711@ESESSMB207.ericsson.se> Hi Serge, If I'm not mistaken, it's a known limitation of EDoc. See edoc(3), read_source/2, the preprocess option: Note: comments in included files will not be available to EDoc, even with this option enabled. Best regards, Hans Bolinder, Erlang/OTP, Ericsson AB From serge@REDACTED Mon Feb 18 14:19:14 2013 From: serge@REDACTED (Serge Aleynikov) Date: Mon, 18 Feb 2013 08:19:14 -0500 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <56466BD70414EA48969B4064696CF28C03D711@ESESSMB207.ericsson.se> References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com>, <510BDD44.20007@aleynikov.org> <56466BD70414EA48969B4064696CF28C03D711@ESESSMB207.ericsson.se> Message-ID: <51222A52.9060104@aleynikov.org> Hans, The example I posted below didn't include any hrl files (if that is what you are referring to), and my question was why the comments immediately following the type proto() were parsed correctly, and the comments following the type options() didn't get processed by edoc. I don't think the limitation you are talking about explains that oddity. Serge On 2/18/2013 8:11 AM, Hans Bolinder wrote: > Hi Serge, > > If I'm not mistaken, it's a known limitation of EDoc. See edoc(3), read_source/2, the preprocess option: > > Note: comments in included files will not be available to EDoc, even > with this option enabled. > > Best regards, > > Hans Bolinder, Erlang/OTP, Ericsson AB > On 2/1/2013 10:20 AM, Serge Aleynikov wrote: > I have a similar question. In the following test module the generated > documentation for proto() type includes proper comment, but the > options() doc doesn't include comment. It seems to be immaterial if "%% > Protocol options." appear on the same line as the ending "]." of the > options() type or the next line. Why is the edoc generator doesn't > include comment for the options() type? > > -module(test). > -export([t/2]). > > -type proto() :: tcp | ssl. > %% Protocol type. > > -type options() :: [ > {server, Server::string()} > | {port, Port::integer()} > ]. > %% Protocol options. > %% * Server - server to connect to (no MX lookup) > %% * Relay - domain to do MX lookup of list of servers > > -spec t(Proto :: proto(), Options :: options()) -> ok. > t(_Proto, _Options) -> ok. > > > Segment of relevant generated docs: > >

options()

> 

options() =
>     [{server, Server :: string()} | {port, Port :: integer()}]

> >

proto()

> 

proto() = tcp | ssl

>

Protocol type.

From lukas@REDACTED Mon Feb 18 14:25:47 2013 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 18 Feb 2013 14:25:47 +0100 Subject: [erlang-questions] Slow line oriented IO In-Reply-To: References: Message-ID: Great! All tests over the weekend seem to have passed for the patch so it will be part of R16B. Lukas On Sat, Feb 16, 2013 at 12:57 AM, Hynek Vychodil wrote: > It works like charm. I have patched R15B03 ad with 1MB read ahead it > is unbelievable fast > > 20> timer:tc(line_read_test,read_std,["test.txt"]). > {30732,2408} > > It is even four times faster than port version > > 24> timer:tc(line_read_test,read_port,["test.txt"]). > {135676,2408} > > It is also faster without async threads. Measurements above was with > +A 10. Without async threads: > > 2> timer:tc(line_read_test,read_std,["test.txt"]). > {19322,2408} > > I was expected time around 100ms but current behavior is far beyond > expectation. I hope this patch will find way into upcoming R16. It > seems to speed up line oriented io for "normal" files two times also. > > Thanks a lot > Hynek Vychodil > > On Fri, Feb 15, 2013 at 4:58 PM, Lukas Larsson > wrote: > > Oh, and here are the benchmarks for me now: > > > > R15B03: > > > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > > {21307874,14448} > > > > R16B with patch: > > > > 1> timer:tc(line_read_test,read_std,["test.txt"]). > > {91885,14448} > > > > It's not every day you find a two line change which increases speed by a > > factor of 232! > > > > Lukas > > > > > > On Fri, Feb 15, 2013 at 4:42 PM, Lukas Larsson < > lukas@REDACTED> > > wrote: > >> > >> aha, you needed to have very long lines in the file to provoke the > >> behaviour. The file I was experimenting on had the same line length all > the > >> time. > >> > >> I think I've found the culprit. Aparently the read_ahead option was > >> ignored after the first buffer had been consumed. Try the patch below > and > >> see if it works for you: > >> > >> From 0884480691e047801cef117b19151a7d34c9e438 Mon Sep 17 00:00:00 2001 > >> From: Lukas Larsson > >> Date: Fri, 15 Feb 2013 16:37:52 +0100 > >> Subject: [PATCH] invoke_read_line now remembers read_ahead settings > >> > >> --- > >> erts/emulator/drivers/common/efile_drv.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/erts/emulator/drivers/common/efile_drv.c > >> b/erts/emulator/drivers/common/efile_drv.c > >> index 2279fec..d81f116 100644 > >> --- a/erts/emulator/drivers/common/efile_drv.c > >> +++ b/erts/emulator/drivers/common/efile_drv.c > >> @@ -1160,6 +1160,8 @@ static void invoke_read_line(void *data) > >> /* Need more place */ > >> ErlDrvSizeT need = (d->c.read_line.read_size >= > >> DEFAULT_LINEBUF_SIZE) ? > >> d->c.read_line.read_size + DEFAULT_LINEBUF_SIZE : > >> DEFAULT_LINEBUF_SIZE; > >> + if (need < (d->c.read_line.binp)->orig_size) > >> + need = (d->c.read_line.binp)->orig_size; > >> ErlDrvBinary *newbin = driver_alloc_binary(need); > >> if (newbin == NULL) { > >> d->result_ok = 0; > >> -- > >> 1.7.10.4 > >> > >> > >> > >> > >> On Fri, Feb 15, 2013 at 1:29 AM, Hynek Vychodil < > vychodil.hynek@REDACTED> > >> wrote: > >>> > >>> Using async thread drops time down to 4.6s which is still 20 times > >>> slower than mine simple wrapper. > >>> > >>> On Thu, Feb 14, 2013 at 4:25 PM, Lukas Larsson > wrote: > >>> > Hello! > >>> > > >>> > If you start erlang with an async thread (erl +A 1) you will see > >>> > radically > >>> > different behaviour. For me the time it takes to use read_line drops > >>> > from > >>> > 58s to 2s. When doing file i/o you should always have some async > >>> > threads to > >>> > help you do the work, this is why in R16B we changed the default > from 0 > >>> > to > >>> > 10 async threads. > >>> > > >>> > Lukas > >>> > > >>> > > >>> > On Thu, Feb 14, 2013 at 3:46 PM, Hynek Vychodil > >>> > > >>> > wrote: > >>> >> > >>> >> Hello, > >>> >> I know it was been already discussed here in list and it is also > >>> >> recurring > >>> >> topic for at least five years. But anyway I have been bitten by it > >>> >> again > >>> >> and > >>> >> also found pretty pathological case. I have 30MB text file and it > has > >>> >> a > >>> >> few > >>> >> near to 1MB lines there. (I can provide file with same line lengths > if > >>> >> somebody interested.) What I have been observed is that reading this > >>> >> file > >>> >> using raw file:read_line/1 takes 51s! For comparison I have tried > some > >>> >> different approaches and what I got (line_read_test:read_std/1 is > >>> >> using > >>> >> file:read_line/1): > >>> >> > >>> >> 1> timer:tc(line_read_test,read_std,["test.txt"]). > >>> >> {51028105,2408} > >>> >> 2> timer:tc(line_read_test,read,["test.txt"]). > >>> >> {226220,2408} > >>> >> 3> timer:tc(line_read_test,read_port,["test.txt"]). > >>> >> {139388,2408} > >>> >> > >>> >> $time perl -nE'$i++}{say $i' test.txt > >>> >> 2408 > >>> >> > >>> >> real 0m0.053s > >>> >> user 0m0.044s > >>> >> sys 0m0.008s > >>> >> > >>> >> $ time wc -l test.txt > >>> >> 2408 test.txt > >>> >> > >>> >> real 0m0.013s > >>> >> user 0m0.004s > >>> >> sys 0m0.008s > >>> >> > >>> >> $ time ./a.out test.txt > >>> >> 2408 > >>> >> > >>> >> real 0m0.020s > >>> >> user 0m0.012s > >>> >> sys 0m0.008s > >>> >> > >>> >> It means erlang should be at least 225 times faster > >>> >> (line_read_test:read/1 > >>> >> which has flow control). Erlang can be 350 times faster > >>> >> (line_read_test:read_port/1 without flow control). Another high > level > >>> >> language (perl) is almost thousand times faster. Special C program > is > >>> >> almost > >>> >> four thousands times faster and old good glibc is two and half > >>> >> thousands > >>> >> times faster. Come on guys it is not even fun when simple (and > wrong) > >>> >> erlang > >>> >> wrapper around standard module is more than two order of magnitude > >>> >> faster. > >>> >> From mine experience when there is something two orders of magnitude > >>> >> slower > >>> >> it tells me there is something damn wrong. I have looked into > >>> >> efile_drv.c > >>> >> and > >>> >> ti is unfortunately far beyond mine C skill but if simple buffering > >>> >> and > >>> >> binary:match/2 can outperform it 225 times there has to be something > >>> >> rotten > >>> >> in this code. > >>> >> > >>> >> I have also experimented with read_ahead option in file:open and > >>> >> changing > >>> >> to > >>> >> less value makes thing worse. > >>> >> > >>> >> Just to make grasp how bad it is, in same time I'm able sort 150 > >>> >> million > >>> >> 64bit values (1.2GB of data) three times (one CPU core same HW). It > is > >>> >> not > >>> >> in > >>> >> flow control, mine simple wrapper does flow control too. It can't > make > >>> >> current code less intrusive, if it consumes 100% CPU for 51s instead > >>> >> of > >>> >> 226ms > >>> >> then it will definitely affect whole server. It is not in concurrent > >>> >> access, > >>> >> mine code allows concurrent access too. Admitting there is something > >>> >> broken > >>> >> is first step to fixing it. I hope I helped at least in this way. > >>> >> > >>> >> With best regards > >>> >> Hynek Vychodil > >>> >> > >>> >> _______________________________________________ > >>> >> erlang-questions mailing list > >>> >> erlang-questions@REDACTED > >>> >> http://erlang.org/mailman/listinfo/erlang-questions > >>> >> > >>> > > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >> > >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.bolinder@REDACTED Mon Feb 18 15:03:07 2013 From: hans.bolinder@REDACTED (Hans Bolinder) Date: Mon, 18 Feb 2013 14:03:07 +0000 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <51222A52.9060104@aleynikov.org> References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com>, <510BDD44.20007@aleynikov.org> <56466BD70414EA48969B4064696CF28C03D711@ESESSMB207.ericsson.se>, <51222A52.9060104@aleynikov.org> Message-ID: <56466BD70414EA48969B4064696CF28C03D73B@ESESSMB207.ericsson.se> Hi, My mistake. Sorry. The answer was meant to be sent to Nico Kruber. I'll send it again, in the correct thread. BR From magnus@REDACTED Mon Feb 18 15:12:30 2013 From: magnus@REDACTED (Magnus Henoch) Date: Mon, 18 Feb 2013 14:12:30 +0000 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <510BDD44.20007@aleynikov.org> (Serge Aleynikov's message of "Fri, 01 Feb 2013 10:20:36 -0500") References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com> <510BDD44.20007@aleynikov.org> Message-ID: Serge Aleynikov writes: > I have a similar question. In the following test module the generated > documentation for proto() type includes proper comment, but the > options() doc doesn't include comment. It seems to be immaterial if "%% > Protocol options." appear on the same line as the ending "]." of the > options() type or the next line. Why is the edoc generator doesn't > include comment for the options() type? I played with your example a bit, and found that the documentation does get included if the closing square bracket is on the same line as the second alternative: -module(test). -export([t/2]). -type proto() :: tcp | ssl. %% Protocol type. -type options() :: [ {server, Server::string()} | {port, Port::integer()}]. %% Protocol options. %% * Server - server to connect to (no MX lookup) %% * Relay - domain to do MX lookup of list of servers -spec t(Proto :: proto(), Options :: options()) -> ok. t(_Proto, _Options) -> ok. Tested with edoc from R15B03. Hope this helps someone find the bug... Regards, Magnus From serge@REDACTED Mon Feb 18 16:03:27 2013 From: serge@REDACTED (Serge Aleynikov) Date: Mon, 18 Feb 2013 10:03:27 -0500 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com> <510BDD44.20007@aleynikov.org> Message-ID: <512242BF.3030204@aleynikov.org> Thanks for the workaround. Does sound like a bug in edoc. On 2/18/2013 9:12 AM, Magnus Henoch wrote: > Serge Aleynikov writes: > >> I have a similar question. In the following test module the generated >> documentation for proto() type includes proper comment, but the >> options() doc doesn't include comment. It seems to be immaterial if "%% >> Protocol options." appear on the same line as the ending "]." of the >> options() type or the next line. Why is the edoc generator doesn't >> include comment for the options() type? > > I played with your example a bit, and found that the documentation does > get included if the closing square bracket is on the same line as the > second alternative: > > -module(test). > -export([t/2]). > > -type proto() :: tcp | ssl. > %% Protocol type. > > -type options() :: [ > {server, Server::string()} > | {port, Port::integer()}]. > %% Protocol options. > %% * Server - server to connect to (no MX lookup) > %% * Relay - domain to do MX lookup of list of servers > > -spec t(Proto :: proto(), Options :: options()) -> ok. > t(_Proto, _Options) -> ok. > > Tested with edoc from R15B03. Hope this helps someone find the bug... > > Regards, > Magnus > From jeremy@REDACTED Mon Feb 18 23:31:18 2013 From: jeremy@REDACTED (Jeremy Ong) Date: Mon, 18 Feb 2013 14:31:18 -0800 Subject: [erlang-questions] Multiple read and write ports to an external process Message-ID: I am interested in using Erlang ports for process interop. Currently, `open_port` only seems to allow spawning communication on 1 read and 1 write port. Is there a reason we are limited to just these 2 file descriptors? It seems like it could be advantageous to customize the fd usage more (for example, forward errors or analytics data on fd 3). Is there any built-in way of doing this without resorting to writing nifs? Cheers, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From mediocregopher@REDACTED Tue Feb 19 00:50:26 2013 From: mediocregopher@REDACTED (Brian Picciano) Date: Mon, 18 Feb 2013 18:50:26 -0500 Subject: [erlang-questions] Threads with outdated list of links Message-ID: I hope it's not bad form to post this to two places at once. I already put this on stack overflow but I figured I might get a better response here. Anyway: I have a process that lots of threads make links to in order to associate themselves with this process. I made the system this way so that at any time I can easily see who is associated to the central process without having to also keep track of any lists inside of the application itself. I can simply do process_info(self(),links) and erlang keeps track of which processes are still alive or not, etc.... At least, I thought so, until I found out that the list being returned on this thread isn't accurate at this moment: % P is my central pid (node@REDACTED)212> P. <0.803.0> % Get all the links to it (node@REDACTED)213> {links,L} = process_info(P,links). {links,[<0.29179.155>,<0.6492.250>,<0.29990.293>|...]} % Counting out all the links (node@REDACTED)214> length(L). 31154 % Filtering out all of the dead processes, put them in Lf (node@REDACTED)215> Lf = lists:filter(fun(Pid) -> try process_info(Pid,links) of {links,_} -> true; _ -> false catch _:_ -> false end end, L). [<0.29179.155>,<0.6492.250>,<0.29990.293>,<0.23619.530>|...] % Lf is only about half the size, half the linked processes are dead! (node@REDACTED)216> length(Lf). 15654 % Proof that the links haven't changed in the interim (node@REDACTED)217> {links,L} = process_info(P,links). {links,[<0.29179.155>,<0.6492.250>,<0.29990.293>|...]} The only thing I can think of that would cause this is network connectivity issues, since some of the links may come from threads on a node on another machine. Is that a possible explanation for this? And is there a way to make the thread clean up its link list? -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.lundin@REDACTED Tue Feb 19 09:18:49 2013 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Tue, 19 Feb 2013 09:18:49 +0100 Subject: [erlang-questions] Threads with outdated list of links In-Reply-To: References: Message-ID: Hi, process_info does only operate on local processes. Your filtering does not work as you expect because you will get an exception (bad argument) on every remote process. This is probably the explanation you are seeking for. /Kenneth, Erlang/OTP Ericsson On 2/19/13, Brian Picciano wrote: > I hope it's not bad form to post this to two places at once. I already put > this on stack overflow but I figured I might get a better response here. > Anyway: > > I have a process that lots of threads make links to in order to associate > themselves with this process. I made the system this way so that at any > time I can easily see who is associated to the central process without > having to also keep track of any lists inside of the application itself. I > can simply do process_info(self(),links) and erlang keeps track of which > processes are still alive or not, etc.... > > At least, I thought so, until I found out that the list being returned on > this thread isn't accurate at this moment: > > % P is my central pid > (node@REDACTED)212> P. > <0.803.0> > > % Get all the links to it > (node@REDACTED)213> {links,L} = process_info(P,links). > {links,[<0.29179.155>,<0.6492.250>,<0.29990.293>|...]} > > % Counting out all the links > (node@REDACTED)214> length(L). > 31154 > > % Filtering out all of the dead processes, put them in Lf > (node@REDACTED)215> Lf = lists:filter(fun(Pid) -> try process_info(Pid,links) > of {links,_} -> true; _ -> false catch _:_ -> false end end, L). > [<0.29179.155>,<0.6492.250>,<0.29990.293>,<0.23619.530>|...] > > % Lf is only about half the size, half the linked processes are dead! > (node@REDACTED)216> length(Lf). > 15654 > > % Proof that the links haven't changed in the interim > (node@REDACTED)217> {links,L} = process_info(P,links). > {links,[<0.29179.155>,<0.6492.250>,<0.29990.293>|...]} > > > The only thing I can think of that would cause this is network connectivity > issues, since some of the links may come from threads on a node on another > machine. Is that a possible explanation for this? And is there a way to > make the thread clean up its link list? > From hans.bolinder@REDACTED Tue Feb 19 16:44:19 2013 From: hans.bolinder@REDACTED (Hans Bolinder) Date: Tue, 19 Feb 2013 15:44:19 +0000 Subject: [erlang-questions] edoc: documenting "-type" In-Reply-To: <510BDD44.20007@aleynikov.org> References: <4E1D3FED.4070409@gmail.com> <781d5e3c-3e04-4aa6-b98d-7fae61652b4d@googlegroups.com> <18234ee1-1769-4625-90ae-907d2f9ddb0d@googlegroups.com>, <510BDD44.20007@aleynikov.org> Message-ID: <56466BD70414EA48969B4064696CF28C03D8CB@ESESSMB207.ericsson.se> Hi, There is a patch at https://github.com/uabboli/otp/commit/f4db123786408d23c1497fbc4a875d47f304c64d. It will hopefully make it into the upcoming R16B release. Best regards, Hans Bolinder, Erlang/OTP, Ericsson AB From anthonym@REDACTED Tue Feb 19 17:27:51 2013 From: anthonym@REDACTED (Anthony Molinaro) Date: Tue, 19 Feb 2013 08:27:51 -0800 Subject: [erlang-questions] Using -callback with libraries supporting R14 and higher Message-ID: <20130219162751.GA41097@alumni.caltech.edu> Hi, I recently was asked to add -callback support to a library I maintain because someone was getting warnings from dialyzer. Upon looking into it I don't see any way to support both R14 and R15. The -callback attribute seems to cause syntax errors with the R14 compiler. If you include both with R15 you get an error about how you can't use them together. I know it's possible using a -D flag and some conditional macros, but that requires integration with the build system (and despite its wide appeal, rebar is not the only game in town, some people use EMakefiles, some GNU autotools, etc). Does OTP provide some sort of macro definition, similar to say ?MODULE which provides the version of OTP? If not how are other library maintainers providing versions for multiple incompatible OTP releases? -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro From mononcqc@REDACTED Tue Feb 19 18:13:00 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 19 Feb 2013 12:13:00 -0500 Subject: [erlang-questions] Using -callback with libraries supporting R14 and higher In-Reply-To: <20130219162751.GA41097@alumni.caltech.edu> References: <20130219162751.GA41097@alumni.caltech.edu> Message-ID: <20130219171259.GB88581@ferdmbp.local> First of all, you shouldn't use the OTP version to define your macros -- this is a release name, and it likely will change with custom releases. You can detect compiler or kernel versions easily if what you want to do is define macros. Just make a quick call to: erl -noshell -eval 'io:format("~s~n",[element(3, lists:keyfind(kernel,1,application:which_applications()))]),init:stop().' This will tend to break when you don't control how to build the library, though. At this point what you might want to do is use a parse transform that will do the call at compile time to determine what version of the compiler is being used right now, and filter out the attributes you do not want to keep. I've written a quick one that seems to work (didn't test it extensively). Based on the current ERTS version, it will either drop the behaviour_info/1 and behavior_info/1 functions (and remove them from the export lists), or remove the -spec arguments. The script is also visible at https://gist.github.com/ferd/4987814, and if people are interested, I'll make it a minimal library application available from my github account so people can use it with rebar or whatever dependency management system they have: -module(behaviour_parse). -export([parse_transform/2]). -define(CUTOFF, "2.15"). % R15B parse_transform(AST, _Opts) -> [Vsn] = [X || {kernel,_,X} <- application:which_applications()], walk_ast(if Vsn >= ?CUTOFF -> spec; Vsn < ?CUTOFF -> func end, AST). walk_ast(_, []) -> []; walk_ast(func, [{attribute, _Ln, callback, _}|T]) -> walk_ast(func, T); walk_ast(spec, [{attribute, Ln, export, List}|T]) -> [{attribute, Ln, export, List -- [{behaviour_info,1},{behavior_info,1}]} | walk_ast(spec, T)]; walk_ast(spec, [{function, _Ln, behaviour_info,1,_Clauses}|T]) -> walk_ast(spec, T); walk_ast(spec, [{function, _Ln, behavior_info,1,_Clauses}|T]) -> walk_ast(spec, T); walk_ast(Type, [H|T]) -> [H|walk_ast(Type, T)]. Just include it in a file with: -compile({parse_transform, behaviour_parse}). And it should do its thing. Regards, Fred. On 02/19, Anthony Molinaro wrote: > Hi, > > I recently was asked to add -callback support to a library I maintain > because someone was getting warnings from dialyzer. Upon looking into > it I don't see any way to support both R14 and R15. > > The -callback attribute seems to cause syntax errors with the R14 compiler. > If you include both with R15 you get an error about how you can't use them > together. > > I know it's possible using a -D flag and some conditional macros, but that > requires integration with the build system (and despite its wide appeal, > rebar is not the only game in town, some people use EMakefiles, some > GNU autotools, etc). > > Does OTP provide some sort of macro definition, similar to say ?MODULE > which provides the version of OTP? If not how are other library > maintainers providing versions for multiple incompatible OTP releases? > > -Anthony > > -- > ------------------------------------------------------------------------ > Anthony Molinaro > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mononcqc@REDACTED Tue Feb 19 18:30:16 2013 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 19 Feb 2013 12:30:16 -0500 Subject: [erlang-questions] Using -callback with libraries supporting R14 and higher In-Reply-To: <20130219171259.GB88581@ferdmbp.local> References: <20130219162751.GA41097@alumni.caltech.edu> <20130219171259.GB88581@ferdmbp.local> Message-ID: <20130219173016.GC88581@ferdmbp.local> Argh, disregard the entire parse transform idea. I had forgotten that versions of Erlang prior to R15 didn't actually support the syntax required for -callback arguments. The code that uses them will still fail there, before the parse transform can run. My bad, Fred. On 02/19, Fred Hebert wrote: > First of all, you shouldn't use the OTP version to define your macros -- > this is a release name, and it likely will change with custom releases. > > You can detect compiler or kernel versions easily if what you want to do > is define macros. Just make a quick call to: > > erl -noshell -eval 'io:format("~s~n",[element(3, lists:keyfind(kernel,1,application:which_applications()))]),init:stop().' > > This will tend to break when you don't control how to build the library, > though. > > At this point what you might want to do is use a parse transform > that will do the call at compile time to determine what version of the > compiler is being used right now, and filter out the attributes you do > not want to keep. > > I've written a quick one that seems to work (didn't test it > extensively). Based on the current ERTS version, it will either drop the > behaviour_info/1 and behavior_info/1 functions (and remove them from the > export lists), or remove the -spec arguments. The script is also visible > at https://gist.github.com/ferd/4987814, and if people are interested, > I'll make it a minimal library application available from my github > account so people can use it with rebar or whatever dependency > management system they have: > > -module(behaviour_parse). > -export([parse_transform/2]). > -define(CUTOFF, "2.15"). % R15B > > parse_transform(AST, _Opts) -> > [Vsn] = [X || {kernel,_,X} <- application:which_applications()], > walk_ast(if Vsn >= ?CUTOFF -> spec; > Vsn < ?CUTOFF -> func > end, > AST). > > walk_ast(_, []) -> > []; > walk_ast(func, [{attribute, _Ln, callback, _}|T]) -> > walk_ast(func, T); > walk_ast(spec, [{attribute, Ln, export, List}|T]) -> > [{attribute, Ln, export, List -- [{behaviour_info,1},{behavior_info,1}]} > | walk_ast(spec, T)]; > walk_ast(spec, [{function, _Ln, behaviour_info,1,_Clauses}|T]) -> > walk_ast(spec, T); > walk_ast(spec, [{function, _Ln, behavior_info,1,_Clauses}|T]) -> > walk_ast(spec, T); > walk_ast(Type, [H|T]) -> > [H|walk_ast(Type, T)]. > > > Just include it in a file with: > > -compile({parse_transform, behaviour_parse}). > > And it should do its thing. > > Regards, > Fred. > > On 02/19, Anthony Molinaro wrote: > > Hi, > > > > I recently was asked to add -callback support to a library I maintain > > because someone was getting warnings from dialyzer. Upon looking into > > it I don't see any way to support both R14 and R15. > > > > The -callback attribute seems to cause syntax errors with the R14 compiler. > > If you include both with R15 you get an error about how you can't use them > > together. > > > > I know it's possible using a -D flag and some conditional macros, but that > > requires integration with the build system (and despite its wide appeal, > > rebar is not the only game in town, some people use EMakefiles, some > > GNU autotools, etc). > > > > Does OTP provide some sort of macro definition, similar to say ?MODULE > > which provides the version of OTP? If not how are other library > > maintainers providing versions for multiple incompatible OTP releases? > > > > -Anthony > > > > -- > > ------------------------------------------------------------------------ > > Anthony Molinaro > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions From tyron.zerafa@REDACTED Tue Feb 19 18:59:04 2013 From: tyron.zerafa@REDACTED (Tyron Zerafa) Date: Tue, 19 Feb 2013 18:59:04 +0100 Subject: [erlang-questions] Dynamic Code Loading in ERTS Message-ID: Does anyone know of a way in which one can execute a function's abstract syntax tree WITHOUT using erl_eval? The only (naive) way I have managed to come up with so far is to write the function to a file, compile and load it? Is there a practical way which avoids this overhead (of writing to a file) allowing code to be loaded directly in the ERTS?? Thanks for all your help :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob@REDACTED Tue Feb 19 19:08:02 2013 From: bob@REDACTED (Bob Ippolito) Date: Tue, 19 Feb 2013 10:08:02 -0800 Subject: [erlang-questions] Dynamic Code Loading in ERTS In-Reply-To: References: Message-ID: http://www.erlang.org/doc/man/code.html#load_binary-3 On Tuesday, February 19, 2013, Tyron Zerafa wrote: > Does anyone know of a way in which one can execute a function's abstract > syntax tree WITHOUT using erl_eval? > > The only (naive) way I have managed to come up with so far is to write the > function to a file, compile and load it? Is there a practical way which > avoids this overhead (of writing to a file) allowing code to be loaded > directly in the ERTS?? > > Thanks for all your help :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.joseph.davis@REDACTED Tue Feb 19 19:14:23 2013 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Tue, 19 Feb 2013 12:14:23 -0600 Subject: [erlang-questions] Dynamic Code Loading in ERTS In-Reply-To: References: Message-ID: Tyron, Bob's also a got a pretty easy to understand example in mochiglobal [1] that helped me pick up on some of the compilation parts. [1] https://github.com/mochi/mochiweb/blob/master/src/mochiglobal.erl On Tue, Feb 19, 2013 at 12:08 PM, Bob Ippolito wrote: > http://www.erlang.org/doc/man/code.html#load_binary-3 > > > On Tuesday, February 19, 2013, Tyron Zerafa wrote: >> >> Does anyone know of a way in which one can execute a function's abstract >> syntax tree WITHOUT using erl_eval? >> >> The only (naive) way I have managed to come up with so far is to write the >> function to a file, compile and load it? Is there a practical way which >> avoids this overhead (of writing to a file) allowing code to be loaded >> directly in the ERTS?? >> >> Thanks for all your help :) > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From carlsson.richard@REDACTED Tue Feb 19 20:20:22 2013 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 19 Feb 2013 20:20:22 +0100 Subject: [erlang-questions] Jesse - a JSON schema validator in Erlang Message-ID: <5123D076.8080800@gmail.com> As part of our ongoing effort to publish some of Klarna's libraries as open source, here's the latest addition to the bunch: a JSON schema validator in Erlang, by Alexander Dergachev. https://github.com/klarna/jesse /Richard From wallentin.dahlberg@REDACTED Tue Feb 19 22:32:10 2013 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Tue, 19 Feb 2013 22:32:10 +0100 Subject: [erlang-questions] Dynamic Code Loading in ERTS In-Reply-To: References: Message-ID: A source for examples is often the test suites. In this particular case it could be hard to find. https://github.com/erlang/otp/blob/master/erts/emulator/test/code_parallel_load_SUITE.erl Uses dynamic a dynamic generate and load approach. Not really meant as a tutorial but could give a few pointers. =) // Bj?rn-Egil 2013/2/19 Paul Davis > Tyron, > > Bob's also a got a pretty easy to understand example in mochiglobal > [1] that helped me pick up on some of the compilation parts. > > [1] https://github.com/mochi/mochiweb/blob/master/src/mochiglobal.erl > > On Tue, Feb 19, 2013 at 12:08 PM, Bob Ippolito wrote: > > http://www.erlang.org/doc/man/code.html#load_binary-3 > > > > > > On Tuesday, February 19, 2013, Tyron Zerafa wrote: > >> > >> Does anyone know of a way in which one can execute a function's abstract > >> syntax tree WITHOUT using erl_eval? > >> > >> The only (naive) way I have managed to come up with so far is to write > the > >> function to a file, compile and load it? Is there a practical way which > >> avoids this overhead (of writing to a file) allowing code to be loaded > >> directly in the ERTS?? > >> > >> Thanks for all your help :) > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Tue Feb 19 23:59:07 2013 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 19 Feb 2013 14:59:07 -0800 Subject: [erlang-questions] [ANN] CloudI 1.2.0 Released! Message-ID: <512403BB.5080403@gmail.com> Download 1.2.0 from http://sourceforge.net/projects/cloudi/files/latest/download What is CloudI? CloudI (http://cloudi.org) is meant to be a "universal integrator" using an Erlang core, with its actor model, to pursue efficiency, fault-tolerance and scalability. The CloudI API provides a minimal interface to communicate among actors that are called services, so programming language agnostic, database agnostic, and messaging bus agnostic integration can occur. CloudI currently integrates with the programming languages Erlang, C/C++, Java, Python, and Ruby, the databases PostgreSQL, MySQL, couchdb, memcached, and tokyotyrant, the messaging bus ZeroMQ and the internal CloudI messaging. HTTP is also supported for easy integration with cowboy. If anyone is willing to get involved, don't hesitate to contact me or start looking at the code. What is CloudI? summary: (short answer) "An application server that efficiently integrates with many languages, many databases, and many messaging buses in a way that is both scalable and fault-tolerant." (shorter answer) "A rock-solid transaction processing system for flexible software development." (shortest answer) "A Cloud at the lowest level." About the release: This release includes both a Quick Start and an API doc. The Quick Start can be found directly on http://cloudi.org (or within doc/README.markdown) and the API doc is on http://cloudi.org/api.html (and at doc/html/api.html). Any feedback on the documentation is helpful. Aside from the documentation, the release mainly focused on bugfixes and refactoring. misultin needed to be removed due to problems with its parameterized modules usage while cowboy has more integration (now using 0.8.0). Due to confusion, all mentions of "job" within CloudI have been changed to "service" to help avoid communication problems in the documentation. This impacts what was previously the cloudi_job behavior, which is now the cloudi_service behavior. So, internal services (services using the cloudi_service Erlang behavior) need the behavior name and the callback names updated, as part of this release. If you run into any problems, please don't hesitate to report them (here https://github.com/okeuday/CloudI/issues/new ). The latest news is below: * Erlang >= R15B01 is now required due to callback/behavior changes * misultin was removed, cowboy now provides better functionality (misultin was also broken due to parameterized modules usage) cowboy was updated to 0.8.0 * The documentation and CloudI source code changes so that "job" is no longer used to refer to services. Now "service" is used consistently to help prevent confusion. This change means that Erlang services (i.e., internal services) must now use the cloudi_service behavior instead of the cloudi_job behavior and the callback names reflect the same change. The CloudI Job API is now referred to as the CloudI Service API. * A Quick Start and API documentation was added to help people learn about CloudI * A minor CloudI API change occured that affects Erlang services: cloudi_service:recv_async now always returns {ResponseInfo, Response, TransId} to be consistent with functionality in external services * Important bugfixes and improvements are included in this release * cloudi_service:unsubscribe will now properly unsubscribe a single instance of the service name pattern * Many improvements to internal services and the python_c external services * the ZeroMQ integration service has been fixed * the CloudI Service API had some minor issues fixed * the flood and messaging tests had some fixes * the bug with ACL string usage (from 1.1.1) was fixed * The request_timeout_adjustment service configuration option was added to make all service requests have their timeout decremented by the service request handler's execution time (defaults to false) * ZeroMQ v3 is now the default version if it is enabled (version 3.2.2 is used, if it isn't already installed) Please mention any problems, issues, or ideas! Thanks, Michael From essen@REDACTED Wed Feb 20 01:06:17 2013 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Wed, 20 Feb 2013 01:06:17 +0100 Subject: [erlang-questions] Using -callback with libraries supporting R14 and higher Message-ID: Learn to say "no" to your users, in this case with a reason similar to "we only support behaviours in R15+". I occasionally receive bug reports about Ranch not building in R14, my answer is "upgrade or fork and fix on your own". May sound harsh, but there's more important work to do, you have to draw the line somewhere. Nobody complained so far. Anthony Molinaro wrote: >Hi, > >I recently was asked to add -callback support to a library I maintain >because someone was getting warnings from dialyzer. Upon looking into >it I don't see any way to support both R14 and R15. > >The -callback attribute seems to cause syntax errors with the R14 compiler. >If you include both with R15 you get an error about how you can't use them >together. > >I know it's possible using a -D flag and some conditional macros, but that >requires integration with the build system (and despite its wide appeal, >rebar is not the only game in town, some people use EMakefiles, some >GNU autotools, etc). > >Does OTP provide some sort of macro definition, similar to say ?MODULE >which provides the version of OTP? If not how are other library >maintainers providing versions for multiple incompatible OTP releases? > >-Anthony > >-- >------------------------------------------------------------------------ >Anthony Molinaro >_______________________________________________ >erlang-questions mailing list >erlang-questions@REDACTED >http://erlang.org/mailman/listinfo/erlang-questions From anthonym@REDACTED Wed Feb 20 02:49:35 2013 From: anthonym@REDACTED (Anthony Molinaro) Date: Tue, 19 Feb 2013 17:49:35 -0800 Subject: [erlang-questions] Using -callback with libraries supporting R14 and higher In-Reply-To: References: Message-ID: <2AFBE5C7-5151-4494-A82A-16B7D415047C@alumni.caltech.edu> That's all fine adn good for some projects (especially ones essentially owned by one person), this particular one is for thrift so just forcing users to upgrade is not necessarily possible. However in the case of thrift I use rebar to build the erlang bindings so should be able to do some sort of detection and setting of a -D flag. I was mostly just wondering what others might be doing. Also in this case the complaint was "the dialyzer gives me a warning about callbacks not being defined" so my advice was use the dialyzer option to turn off that warning and plan to re-enable it after r14 is officially deprecated by Ericsson (which is about the time r19 comes out I think, not sure if it was 5 versions back or not). It's probably a more conservative answer, but I want to support the largest user base possible, -Anthony On Feb 19, 2013, at 4:06 PM, Lo?c Hoguin wrote: > Learn to say "no" to your users, in this case with a reason similar to "we only support behaviours in R15+". I occasionally receive bug reports about Ranch not building in R14, my answer is "upgrade or fork and fix on your own". May sound harsh, but there's more important work to do, you have to draw the line somewhere. Nobody complained so far. > > Anthony Molinaro wrote: > >> Hi, >> >> I recently was asked to add -callback support to a library I maintain >> because someone was getting warnings from dialyzer. Upon looking into >> it I don't see any way to support both R14 and R15. >> >> The -callback attribute seems to cause syntax errors with the R14 compiler. >> If you include both with R15 you get an error about how you can't use them >> together. >> >> I know it's possible using a -D flag and some conditional macros, but that >> requires integration with the build system (and despite its wide appeal, >> rebar is not the only game in town, some people use EMakefiles, some >> GNU autotools, etc). >> >> Does OTP provide some sort of macro definition, similar to say ?MODULE >> which provides the version of OTP? If not how are other library >> maintainers providing versions for multiple incompatible OTP releases? >> >> -Anthony >> >> -- >> ------------------------------------------------------------------------ >> Anthony Molinaro >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From watson.timothy@REDACTED Wed Feb 20 10:04:47 2013 From: watson.timothy@REDACTED (Tim Watson) Date: Wed, 20 Feb 2013 09:04:47 +0000 Subject: [erlang-questions] Using -callback with libraries supporting R14 and higher In-Reply-To: <2AFBE5C7-5151-4494-A82A-16B7D415047C@alumni.caltech.edu> References: <2AFBE5C7-5151-4494-A82A-16B7D415047C@alumni.caltech.edu> Message-ID: <5BC49D05-7D45-4B09-809F-A4611EEC3B00@gmail.com> On 20 Feb 2013, at 01:49, Anthony Molinaro wrote: > It's probably a more conservative answer, but I want to support the largest user base possible, > At RabbitMQ we support >= R12B03, so we turn off specs using a macro unless a supported version is being used to compile. The macro is passed via the makefile to erlc - I think this is a pretty common approach, although I realise it's dependent on Make to some extent. Cheers, Tim From erlang@REDACTED Wed Feb 20 12:31:18 2013 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 20 Feb 2013 12:31:18 +0100 Subject: [erlang-questions] Jesse - a JSON schema validator in Erlang In-Reply-To: <5123D076.8080800@gmail.com> References: <5123D076.8080800@gmail.com> Message-ID: I didn't even know there were JSON schemas - this is excellent - one small step nearer to defining *protocols* :-) /Joe On Tue, Feb 19, 2013 at 8:20 PM, Richard Carlsson < carlsson.richard@REDACTED> wrote: > As part of our ongoing effort to publish some of Klarna's libraries as > open source, here's the latest addition to the bunch: a JSON schema > validator in Erlang, by Alexander Dergachev. > > https://github.com/klarna/**jesse > > /Richard > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingham.k@REDACTED Wed Feb 20 17:18:55 2013 From: ingham.k@REDACTED (Igor Karymov) Date: Wed, 20 Feb 2013 08:18:55 -0800 (PST) Subject: [erlang-questions] Jesse - a JSON schema validator in Erlang In-Reply-To: References: <5123D076.8080800@gmail.com> Message-ID: Another approach of scheme definition that can work with json: https://github.com/alavrik/piqi by Anton Lavrik. However json-scheme is more popular standar. -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Feb 20 19:30:20 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Wed, 20 Feb 2013 18:30:20 +0000 Subject: [erlang-questions] Jesse - a JSON schema validator in Erlang In-Reply-To: References: <5123D076.8080800@gmail.com> Message-ID: On Wed, Feb 20, 2013 at 4:18 PM, Igor Karymov wrote: > Another approach of scheme definition that can work with json: > https://github.com/alavrik/piqi Piqi is cool, but if you have external clients, then .xsd, .proto and json-scheme (in descending popularity) are the ways to go. .piqi format for objective-c programmer to communicate json.. what? So not a really good use case. I once used piqi for protobuffer-only communication. Wrote my piqi files, asked piqi to convert them to .proto, and gave these to the third party. Just because I found to writing .piqi than protobuffers easier. In the end, I would love a piqi-to-xsd and piqi-to-json-schema converters. That would allow to use piqi in corporate environments (a.k.a. where xml and xsd are mandatory), and with more flexible clients that use json, which you could pass the JSON schema, requiring much less manual documentation. Though I realize that Anton has much more interesting and useful things to do than writing piqi-any converters (piqic erlang!), this would be a nice addition that would enable piqi usage in some environments. Hell, shouldn't be too hard to implement. Maybe I will do that myself some day. Igor, you raised a valid point. But it's not there.. Yet. :) -- Motiejus Jak?tys From alavrik@REDACTED Thu Feb 21 07:18:13 2013 From: alavrik@REDACTED (Anton Lavrik) Date: Thu, 21 Feb 2013 00:18:13 -0600 Subject: [erlang-questions] Jesse - a JSON schema validator in Erlang In-Reply-To: References: <5123D076.8080800@gmail.com> Message-ID: On Wed, Feb 20, 2013 at 12:30 PM, Motiejus Jak?tys wrote: > On Wed, Feb 20, 2013 at 4:18 PM, Igor Karymov wrote: >> Another approach of scheme definition that can work with json: >> https://github.com/alavrik/piqi Piqi does support working from JSON from Erlang and does it really well. For example, when it finds a problem, it returns a description of it and points to the exact line:column location in the input. Because it is more of a data serialization system, Piqi doesn't support a jesse-like interface for validating JSON dynamically, but it can be added. If anybody is interested, please let me know. > In the end, I would love a piqi-to-xsd and piqi-to-json-schema > converters. That would allow to use piqi in corporate environments > (a.k.a. where xml and xsd are mandatory), and with more flexible > clients that use json, which you could pass the JSON schema, requiring > much less manual documentation. piqi to X conversion can be done fairly easily. First, Piqi specification is stable, small and extensible. Essentially, it has a reasonably small but very powerful set of types that can be mapped to pretty much anything. Second, nobody needs to parse .piqi Just run "piqi compile -t json|pb|xml <.piqi>" and read JSON, XML or Protocol Buffers output using any language. (The "piqi" command is a standalone statically linked program). Anton From iostres@REDACTED Thu Feb 21 11:30:07 2013 From: iostres@REDACTED (Ivan Ostres) Date: Thu, 21 Feb 2013 11:30:07 +0100 Subject: [erlang-questions] Diameter Proxy in Erlang? Message-ID: Hi all, I have to build an diameter DCCA proxy and thinking of doing it in Erlang since diameter stack is available in OTP. I have to build this because we have one diameter client which has burned-in set of AVP's (it doesn't send MSCC AVP which is mandatory for billing system) so I have to change an CCR message a bit and send it to OCS - of course some bits need to be changed in answer also. I looked through relay.erl and relay_cb.erl in diameter examples and noticed that it calls the following function with atom 'relay': handle_request(_Pkt, _SvcName, _Peer) -> {relay, [{timeout, 1000}, {extra, [relayed]}]}. Is it possible to change the _Pkt that is being relayed since it is not a part of return statement? Or should I do separate client and server? Please advise, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre@REDACTED Thu Feb 21 12:16:48 2013 From: andre@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Thu, 21 Feb 2013 12:16:48 +0100 Subject: [erlang-questions] Diameter Proxy in Erlang? In-Reply-To: References: Message-ID: Hi Ivan I think the Mod:prepare_request(Packet, SvcName, Peer) is called after the handle_request (when replied with relay), so you should be able to change the request there. BR/Andre On 21 February 2013 11:30, Ivan Ostres wrote: > Hi all, > > I have to build an diameter DCCA proxy and thinking of doing it in Erlang > since diameter stack is available in OTP. I have to build this because we > have one diameter client which has burned-in set of AVP's (it doesn't send > MSCC AVP which is mandatory for billing system) so I have to change an CCR > message a bit and send it to OCS - of course some bits need to be changed in > answer also. > > I looked through relay.erl and relay_cb.erl in diameter examples and noticed > that it calls the following function with atom 'relay': > > handle_request(_Pkt, _SvcName, _Peer) -> > {relay, [{timeout, 1000}, {extra, [relayed]}]}. > > > Is it possible to change the _Pkt that is being relayed since it is not a > part of return statement? > > Or should I do separate client and server? > > Please advise, > Ivan > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From sverker.eriksson@REDACTED Thu Feb 21 19:30:31 2013 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Thu, 21 Feb 2013 19:30:31 +0100 Subject: [erlang-questions] Any tcp related changes in R15B* that might cause this? In-Reply-To: <5108163A.6090403@erix.ericsson.se> References: <20121214211117.GA86501@alumni.caltech.edu> <50CF35FF.6020101@erix.ericsson.se> <20130102172816.GA24637@alumni.caltech.edu> <20130103061626.GB26037@alumni.caltech.edu> <50EAF721.2020002@erix.ericsson.se> <20130108164231.GA40748@alumni.caltech.edu> <50EC6993.9000608@erix.ericsson.se> <20130115064028.GA53613@alumni.caltech.edu> <5108163A.6090403@erix.ericsson.se> Message-ID: <512667C7.60705@erix.ericsson.se> This patch for {exit_on_close,false} will be included in R16B. /Sverker, Erlang/OTP Sverker Eriksson wrote: > Sorry for the late answer, been kind of busy with R16A. > > I think you have found a bug. I did this modification in your test > > ok = inet:setopts (Sock, [{packet, http}, {exit_on_close,false}]), > > and it still close the socket. > > Here is a patch that seems to solve the problem: > > git fetch git://github.com/sverker/otp.git sverk/tcp-exit_on_close-false > > I'm not 100% sure that this fix doesn't has some other unexpected > effect. The inet driver is quite a complex piece of software. > It would be good if you could test this out. > The patch is based on R15B02 but it should be easy to apply on both > R15B03 and R16A. > > /Sverker, Erlang/OTP > > > Anthony Molinaro wrote: >> Hi Sverker, >> >> I cannot get {exit_on_close, false}, to actually change the behavior >> of my test. I set it as an options to gen_tcp:listen/2 but >> gen_tcp:send/2 >> still returns {error,closed}. I also tried setting it at various other >> points with inet:setopts/2, but it never seems to make a difference. >> >> I'm using R15B02, but I'm wondering if you can modify my example, such >> that the gen_tcp:send/2 doesn't fail with {error,closed}. >> >> I reattached it without any modifications. You should be able to >> just compile it and run >> >> gen_tcp_test:test(10000). >> >> To see it fail. >> >> Hopefully you can show me the error of my ways ;) >> >> -Anthony >> >> On Tue, Jan 08, 2013 at 07:46:43PM +0100, Sverker Eriksson wrote: >>> Anthony Molinaro wrote: >>>> I agree that the new behavior of returning an emsgsize tcp_error is >>>> probably correct, however the new behavior also closes the socket. >>>> >>>> Before your changes mochiweb and my test could return a 400 error so >>>> the client would get a response even if they were sending in a bad >>>> request. After the change the client just gets a remote side closed >>>> connection message. >>>> >>>> So I guess the question is, why is the connection being closed? Is >>>> that also something the other packet protocols do? >>>> >>> Yes all packets protocols act the same in this sense. >>> >>> You can use socket option {exit_on_close, false} to only do a >>> half-close and still allow writing to the socket. >>> >>> /Sverker >>> >>>> Thanks for looking at it, >>>> >>>> -Anthony >>>> >>>> On Mon, Jan 07, 2013 at 05:26:09PM +0100, Sverker Eriksson wrote: >>>>> Thanks for the clarification, guys. >>>>> I've look at dc5f7190 and it seems like the change in error >>>>> handling was more or less intentional (by me). >>>>> >>>>> Old behavior was >>>>> 1. Truncate a too long http line and return it as if it was ok. >>>>> 2. Continue parsing rest of the line as if it were a new http line >>>>> which probably (and hopefully) results in a http_error. >>>>> >>>>> I didn't like that; as it returned a partially parsed line as if it >>>>> was ok, reported the error too late and confusing and also (I guess) >>>>> constituted a security bug that could be abused to trick the http >>>>> parsing with long lines. >>>>> >>>>> So I changed it to reject too long lines and return tcp_error, which >>>>> is what happens for all packet protocols when the length of the >>>>> packet is in some way invalid. >>>>> >>>>> The caller has to handle tcp_error's anyway, so it's a case of bad >>>>> fault identification. >>>>> I guess it would be nicer to return {http_error, "Line too long: >>>>> X-Randomaaaaaaaaaaaaaaaaaa..."} >>>>> or something like that. >>>>> I don't have time to fix that now, but don't think it would be too >>>>> hard do if anyone is eager... >>>>> >>>>> >>>>> /Sverker, Erlang/OTP >>>>> >>>>> >>>>> >>>>> Steve Vinoski wrote: >>>>>> Same results with this test: works on commits prior to dc5f7190, >>>>>> fails >>>>>> on dc5f7190. >>>>>> >>>>>> --steve >>>>>> >>>>>> On Thu, Jan 3, 2013 at 1:16 AM, Anthony Molinaro < >>>>>> anthonym@REDACTED> wrote: >>>>>> >>>>>>> Attached is a test which shows the difference using only gen_tcp >>>>>>> calls. >>>>>>> >>>>>>> Here's the output from both >>>>>>> >>>>>>> Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] >>>>>>> [async-threads:0] >>>>>>> [hipe] [kernel-poll:false] >>>>>>> >>>>>>> Eshell V5.8.5 (abort with ^G) >>>>>>> 1> gen_tcp_test:test(1000). >>>>>>> gen_tcp:send ok >>>>>>> {{"HTTP/1.1",200,"OK"}, >>>>>>> [{"date","Wed, 02 Jan 2013 21:01:08 GMT"}, >>>>>>> {"server","Test"}, >>>>>>> {"content-length","31"}, >>>>>>> {"content-type","text/html"}], >>>>>>> "Hello"} >>>>>>> 2> gen_tcp_test:test(10000). >>>>>>> http_error - R14 >>>>>>> gen_tcp:send ok >>>>>>> {{"HTTP/1.1",400,"Bad Request"}, >>>>>>> [{"date","Wed, 02 Jan 2013 21:03:24 GMT"}, >>>>>>> {"server","Test"}, >>>>>>> {"content-length","0"}], >>>>>>> []} >>>>>>> 3> >>>>>>> >>>>>>> and for R15 >>>>>>> >>>>>>> Erlang R15B02 (erts-5.9.2) [source] [64-bit] [smp:2:2] >>>>>>> [async-threads:0] >>>>>>> [hipe] >>>>>>> [kernel-poll:false] >>>>>>> >>>>>>> Eshell V5.9.2 (abort with ^G) >>>>>>> 1> gen_tcp_test:test(1000). >>>>>>> gen_tcp:send ok >>>>>>> {{"HTTP/1.1",200,"OK"}, >>>>>>> [{"date","Wed, 02 Jan 2013 21:01:08 GMT"}, >>>>>>> {"server","Test"}, >>>>>>> {"content-length","31"}, >>>>>>> {"content-type","text/html"}], >>>>>>> "Hello"} >>>>>>> 2> gen_tcp_test:test(10000). >>>>>>> emsgsize - R15 >>>>>>> gen_tcp:send failed with {error,closed} >>>>>>> socket_closed_remotely >>>>>>> 3> >>>>>>> >>>>>>> Hope it helps, >>>>>>> >>>>>>> -Anthony >>>>>>> >>>>>>> On Wed, Jan 02, 2013 at 12:34:56PM -0500, Steve Vinoski wrote: >>>>>>>> Hi Anthony and Sverker, as I mentioned in a previous email, using >>>>>>> Anthony's >>>>>>>> test case a few weeks ago I bisected between R14B04 and HEAD >>>>>>>> and found >>>>>>>> commit dc5f7190 to be the first commit where the test case fails. >>>>>>>> >>>>>>>> --steve >>>>>>>> >>>>>>>> On Wed, Jan 2, 2013 at 12:28 PM, Anthony Molinaro < >>>>>>>> anthonym@REDACTED> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> The holidays went and interrupted everything, so I'm sorry >>>>>>>>> about the >>>>>>>>> late response. >>>>>>>>> It's unclear if the commit dc5f7190 did it or not, but >>>>>>>>> sometime between >>>>>>>>> R14B04 and R15B02 the behavior did change. I've not isolated it >>>>>>> outside >>>>>>>>> of mochiweb, but I don't think its a mochiweb problem. >>>>>>>>> >>>>>>>>> Here's the pull request which contains the short program I >>>>>>>>> used to >>>>>>>>> reproduce with mochiweb (and work around the new behavior) >>>>>>>>> >>>>>>>>> https://github.com/mochi/mochiweb/pull/91 >>>>>>>>> >>>>>>>>> I'll see if I can't isolate it to just OTP sometime in the next >>>>>>>>> couple of days. >>>>>>>>> >>>>>>>>> -Anthony >>>>>>>>> >>>>>>>>> On Mon, Dec 17, 2012 at 04:10:55PM +0100, Sverker Eriksson wrote: >>>>>>>>>> Have I understood it correctly so far if I say that my commit >>>>>>>>>> dc5f7190 seems to cause a >>>>>>>>>> >>>>>>>>>> {tcp_error,Port,emsgsize} >>>>>>>>>> >>>>>>>>>> while earlier versions caused a >>>>>>>>>> >>>>>>>>>> {http,Port,{http_error,"..."}} >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> for the same scenario with some sort of very long http lines? >>>>>>>>>> >>>>>>>>>> /Sverker, Erlang/OTP >>>>>>>>>> >>>>>>>>>> Anthony Molinaro wrote: >>>>>>>>>>> On Fri, Dec 14, 2012 at 11:32:11AM -0500, Steve Vinoski wrote: >>>>>>>>>>>> Hi Anthony, based on your example, it looks like a >>>>>>>>>>>> follow-up commit >>>>>>>>> that >>>>>>>>>>>> modified some of the changes introduced by my patch >>>>>>>>>>>> introduced the >>>>>>>>> issue >>>>>>>>>>>> you're seeing: commit dc5f7190. You might want to report it >>>>>>>>>>>> over on >>>>>>>>>>>> erlang-bugs. >>>>>>>>>>> Nice catch, I guess that it's actually not enough to return >>>>>>>>>>> error >>>>>>>>>>> for backward compatibility ;) >>>>>>>>>>> >>>>>>>>>>> I think the main thing I'm still not sure of is if the >>>>>>>>>>> decode_packet >>>>>>>>> changes >>>>>>>>>>> are actually the cause. I was able to call decode_packet >>>>>>>>>>> with long >>>>>>>>> headers >>>>>>>>>>> and not actually see an issue. >>>>>>>>>>> >>>>>>>>>>> This >>>>>>>>>>> >>>>>>>>>>> erlang:decode_packet ( httph, >>>>>>>>>>> list_to_binary ([ "X-Random: ", >>>>>>>>>>> [$a || _ <- >>>>>>>>> lists:seq(1,10000)], >>>>>>>>>>> "\r\n\r\n" >>>>>>>>>>> ]), >>>>>>>>>>> []). >>>>>>>>>>> >>>>>>>>>>> Returns the same thing in both R14B04 and R15B02. And as far >>>>>>>>>>> as I >>>>>>> am >>>>>>>>> able >>>>>>>>>>> to tell that is the only call made to decode_packet in >>>>>>>>>>> mochiweb. >>>>>>>>>>> >>>>>>>>>>> I think the actual error is around recvbuf sizing (mochiweb >>>>>>>>>>> uses >>>>>>> 8192 as >>>>>>>>>>> the default, so 10000 is definitely larger than that, >>>>>>>>>>> however, the >>>>>>>>> standard >>>>>>>>>>> libraries must be doing something different as now it returns >>>>>>> emsgsize >>>>>>>>>>> where it did no in the past). I really wish there was a way to >>>>>>> search >>>>>>>>>>> commits on github as then I could search for recvbuf or >>>>>>>>>>> something >>>>>>> like >>>>>>>>>>> that to see what might have changed. >>>>>>>>>>> >>>>>>>>>>> Before I go over to bugs I'll probably try to detangle mochiweb >>>>>>> enough >>>>>>>>>>> to create a small reproducible test case. >>>>>>>>>>> >>>>>>>>>>> I also have a pull request into mochiweb which prevents the >>>>>>>>>>> crash >>>>>>> but >>>>>>>>>>> has the annoying behavior at the moment that it returns an >>>>>>>>>>> error to >>>>>>> the >>>>>>>>>>> client about disconnecting version the 400 error it returned >>>>>>>>>>> before. >>>>>>>>>>> >>>>>>>>>>> https://github.com/mochi/mochiweb/pull/91 >>>>>>>>>>> >>>>>>>>>>> But since the standard library seems to be the one closing the >>>>>>> socket >>>>>>>>>>> there's not much that can be done other than the fix I put >>>>>>>>>>> in place. >>>>>>>>>>> >>>>>>>>>>> Thanks for your help Steve, >>>>>>>>>>> >>>>>>>>>>> -Anthony >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> erlang-questions mailing list >>>>>>>>>> erlang-questions@REDACTED >>>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>> -- >>>>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>>>> Anthony Molinaro < >>>>>>> anthonym@REDACTED> >>>>>>>>> _______________________________________________ >>>>>>>>> erlang-questions mailing list >>>>>>>>> erlang-questions@REDACTED >>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>>>>> >>>>>>> -- >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> Anthony Molinaro >>>>>>> >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From marc@REDACTED Fri Feb 22 09:20:58 2013 From: marc@REDACTED (Marc Worrell) Date: Fri, 22 Feb 2013 09:20:58 +0100 Subject: [erlang-questions] Zotonic interview on TWiT.TV Message-ID: <766EFD92-BE37-4765-A7D5-7D8D8C866192@worrell.nl> Hi, This week two of the Zotonic (the Erlang web framework and CMS) core team members were interviewed on TWiT.TV's FLOSS weekly. Hosts Aaron Newcomb and Dan Lynch had some good and interesting questions. Go and see the interview here: http://twit.tv/show/floss-weekly/243 Or here: http://www.youtube.com/watch?v=igkrFnMZQYI Aaron and Dan were interested in Erlang. I think it is a good idea to get more open source Erlang projects highlighted on the FLOSS weekly podcast. Regards, Marc Worrell From erlang@REDACTED Fri Feb 22 19:31:25 2013 From: erlang@REDACTED (Joe Armstrong) Date: Fri, 22 Feb 2013 19:31:25 +0100 Subject: [erlang-questions] anybody know how to start erlang (cowboy) with launchd on mac. Message-ID: This is a mac mountain lion specific question The subject line says it all. 1) I give some magic command > launchctl load ~/Library/LaunchAgents/joe.experiment.plist This launches erlang/cowboy "on demand" 2) I fetch a web page from "http://localhost:11234" Erlang starts and starts cowboy Anybody know what the magic stuff in joe.experiments.plist is? Cheers /Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Feb 22 20:09:30 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 22 Feb 2013 23:09:30 +0400 Subject: [erlang-questions] Hint erts to preallocate binary of required size Message-ID: erlyvideo has some problems with creating mpeg-ts from mp4. Problem is following: mp4 stores video frames in continuos blocks of 100 kbytes. MPEG-TS requires to split this 100 kbytes into small chunks of about 180 bytes each. Right now it takes about 10 000 microseconds to generate one megabyte of mpeg-ts. First problem: how can I understand if process copies these small chunks into heap or they are still subbinary? Second idea: can it be a good idea to tell erts that I want to allocate new binary of 140 kbytes size and append small chunks to it with only one copying? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Fri Feb 22 20:18:57 2013 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 22 Feb 2013 21:18:57 +0200 Subject: [erlang-questions] Hint erts to preallocate binary of required size In-Reply-To: References: Message-ID: <8F819E43-68FF-4C0C-B1D2-70DA32EC983C@gmail.com> Hello Max, > First problem: how can I understand if process copies these small chunks into heap or they are still subbinary? http://erldocs.com/R14B02/stdlib/binary.html?i=25&search=binary:#referenced_byte_size/1 > Second idea: can it be a good idea to tell erts that I want to allocate new binary of 140 kbytes size and append small chunks to it with only one copying? May be you should use iolist instead of binary? - Dmitry On Feb 22, 2013, at 9:09 PM, Max Lapshin wrote: > erlyvideo has some problems with creating mpeg-ts from mp4. > > Problem is following: mp4 stores video frames in continuos blocks of 100 kbytes. MPEG-TS requires to split this 100 kbytes into small chunks of about 180 bytes each. > > Right now it takes about 10 000 microseconds to generate one megabyte of mpeg-ts. > > First problem: how can I understand if process copies these small chunks into heap or they are still subbinary? > > > Second idea: can it be a good idea to tell erts that I want to allocate new binary of 140 kbytes size and append small chunks to it with only one copying? > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Fri Feb 22 20:44:00 2013 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 22 Feb 2013 23:44:00 +0400 Subject: [erlang-questions] Hint erts to preallocate binary of required size In-Reply-To: <8F819E43-68FF-4C0C-B1D2-70DA32EC983C@gmail.com> References: <8F819E43-68FF-4C0C-B1D2-70DA32EC983C@gmail.com> Message-ID: I'm collecting now iolists and it takes a lot of time. On Fri, Feb 22, 2013 at 11:18 PM, Dmitry Kolesnikov wrote: > Hello Max, > > > First problem: how can I understand if process copies these small chunks > into heap or they are still subbinary? > > > http://erldocs.com/R14B02/stdlib/binary.html?i=25&search=binary:#referenced_byte_size/1 > > > Second idea: can it be a good idea to tell erts that I want to allocate > new binary of 140 kbytes size and append small chunks to it with only one > copying? > > May be you should use iolist instead of binary? > > - Dmitry > > On Feb 22, 2013, at 9:09 PM, Max Lapshin wrote: > > > erlyvideo has some problems with creating mpeg-ts from mp4. > > > > Problem is following: mp4 stores video frames in continuos blocks of 100 > kbytes. MPEG-TS requires to split this 100 kbytes into small chunks of > about 180 bytes each. > > > > Right now it takes about 10 000 microseconds to generate one megabyte of > mpeg-ts. > > > > First problem: how can I understand if process copies these small chunks > into heap or they are still subbinary? > > > > > > Second idea: can it be a good idea to tell erts that I want to allocate > new binary of 140 kbytes size and append small chunks to it with only one > copying? > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francis.joanis@REDACTED Sat Feb 23 00:30:46 2013 From: francis.joanis@REDACTED (Francis Joanis) Date: Fri, 22 Feb 2013 18:30:46 -0500 Subject: [erlang-questions] [ANN] Erlang meetup/Erlounge next Thursday in Ottawa Message-ID: Hi, We'll be having a meetup next week in Ottawa. Details here: http://www.meetup.com/Ottawa-Gatineau-Erlang-Meetup-Group/events/104700322/ Cheers, Francis -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredrikelinder@REDACTED Sat Feb 23 01:23:06 2013 From: fredrikelinder@REDACTED (Fredrik Linder) Date: Fri, 22 Feb 2013 16:23:06 -0800 Subject: [erlang-questions] Can one load non-driver/nif files in Erlang? Message-ID: Hello (A) Is it possible to load .so files in Erlang R15+ that are neither erlang drivers nor erlang nifs? (b) If it is possible, could you provide an example snippet? Thank you /Fredrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Sat Feb 23 10:27:31 2013 From: gleber.p@REDACTED (Gleb Peregud) Date: Sat, 23 Feb 2013 10:27:31 +0100 Subject: [erlang-questions] Can one load non-driver/nif files in Erlang? In-Reply-To: References: Message-ID: No. It is not possible On 23 Feb 2013 01:23, "Fredrik Linder" wrote: > Hello > > (A) Is it possible to load .so files in Erlang R15+ that are neither > erlang drivers nor erlang nifs? > > (b) If it is possible, could you provide an example snippet? > > Thank you > /Fredrik > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Sat Feb 23 11:37:01 2013 From: tony@REDACTED (Tony Rogvall) Date: Sat, 23 Feb 2013 11:37:01 +0100 Subject: [erlang-questions] anybody know how to start erlang (cowboy) with launchd on mac. In-Reply-To: References: Message-ID: <00DD991E-7558-423A-975D-CB20A035D897@rogvall.se> Here is the plist need to start rfzone, I guess you can modify it to start your stuff. ( I probably stole it from somewhere :-) /Tony Label se.seazone.rfzone EnvironmentVariables DYLD_LIBRARY_PATH /opt/local/lib:$DYLD_LIBRARY_PATH ERL_LIBS /Users/%USER%/erlang ProgramArguments /opt/local/bin/erl +Bd -noinput -config /Users/%USER%/erlang/rfzone/sys -sname telld -s rfzone UserName %USER% StandardOutPath /dev/null StandardErrorPath /dev/null RunAtLoad KeepAlive On 22 feb 2013, at 19:31, Joe Armstrong wrote: > This is a mac mountain lion specific question > > The subject line says it all. > > 1) I give some magic command > > > launchctl load ~/Library/LaunchAgents/joe.experiment.plist > > This launches erlang/cowboy "on demand" > > 2) I fetch a web page from "http://localhost:11234" > > Erlang starts and starts cowboy > > Anybody know what the magic stuff in joe.experiments.plist is? > > Cheers > > /Joe > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From aschultz@REDACTED Sat Feb 23 11:49:53 2013 From: aschultz@REDACTED (Andreas Schultz) Date: Sat, 23 Feb 2013 11:49:53 +0100 (CET) Subject: [erlang-questions] Can one load non-driver/nif files in Erlang? In-Reply-To: References: Message-ID: You could write a port_driver or a NIF that would load a .so. Exporting the functions in .so to Erlang is a different story. Andreas -- Dipl. Inform. Andreas Schultz On 23.02.2013, at 01:24, Fredrik Linder wrote: > Hello > > (A) Is it possible to load .so files in Erlang R15+ that are neither erlang drivers nor erlang nifs? > > (b) If it is possible, could you provide an example snippet? > > Thank you > /Fredrik > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From jarimatti.valkonen@REDACTED Sat Feb 23 14:08:19 2013 From: jarimatti.valkonen@REDACTED (Jarimatti Valkonen) Date: Sat, 23 Feb 2013 15:08:19 +0200 Subject: [erlang-questions] anybody know how to start erlang (cowboy) with launchd on mac. In-Reply-To: References: Message-ID: Joe Armstrong kirjoitti 22.2.2013 kello 20.31: > > This launches erlang/cowboy "on demand" > > 2) I fetch a web page from "http://localhost:11234" > > Erlang starts and starts cowboy > > Anybody know what the magic stuff in joe.experiments.plist is? Hi (and apologies for the double post, this was intended to the list) If you want to launch on-demand when a socket connection is made, then AFAIK it's a bit more complicated than just editing the launchd property list. Launchd listens on the socket and passes the socket into the actual program in it's own data structure. The file format documentation (man 5 launchd.plist) specifies how to configure the socket and there's some Apple sample code here: https://developer.apple.com/library/mac/#samplecode/SampleD/Introduction/Intro.html A full example with plist is included in the download, but the sample launchd plist file is not visible in the web site. br, Jarimatti -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sat Feb 23 17:39:15 2013 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 23 Feb 2013 17:39:15 +0100 Subject: [erlang-questions] anybody know how to start erlang (cowboy) with launchd on mac. In-Reply-To: <00DD991E-7558-423A-975D-CB20A035D897@rogvall.se> References: <00DD991E-7558-423A-975D-CB20A035D897@rogvall.se> Message-ID: Brilliant - works like a charm. Thanks a lot I have a few stupid questions - where does erlang standard output go to? - where do launchd error messages go to? - does this keep alive stuff and run at *really* work ie does it start the erlang node on reboot a quick test was pretty encouraging (so you don't need the heartbeat stuff on a mac just use launchd :-) Cheers /Joe On Sat, Feb 23, 2013 at 11:37 AM, Tony Rogvall wrote: > Here is the plist need to start rfzone, I guess you can modify it to start > your stuff. > > ( I probably stole it from somewhere :-) > > /Tony > > > "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> > > > Label > se.seazone.rfzone > EnvironmentVariables > > DYLD_LIBRARY_PATH > /opt/local/lib:$DYLD_LIBRARY_PATH > ERL_LIBS > /Users/%USER%/erlang > > ProgramArguments > > /opt/local/bin/erl > +Bd > -noinput > -config > /Users/%USER%/erlang/rfzone/sys > -sname > telld > -s > rfzone > > UserName > %USER% > StandardOutPath > /dev/null > StandardErrorPath > /dev/null > RunAtLoad > > KeepAlive > > > > > > > > > On 22 feb 2013, at 19:31, Joe Armstrong wrote: > > This is a mac mountain lion specific question > > The subject line says it all. > > 1) I give some magic command > > > launchctl load ~/Library/LaunchAgents/joe.experiment.plist > > This launches erlang/cowboy "on demand" > > 2) I fetch a web page from "http://localhost:11234" > > Erlang starts and starts cowboy > > Anybody know what the magic stuff in joe.experiments.plist is? > > Cheers > > /Joe > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > "Installing applications can lead to corruption over time. Applications > gradually write over each other's libraries, partial upgrades occur, user > and system errors happen, and minute changes may be unnoticeable and > difficult to fix" > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Sat Feb 23 17:41:24 2013 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 23 Feb 2013 17:41:24 +0100 Subject: [erlang-questions] anybody know how to start erlang (cowboy) with launchd on mac. In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 2:08 PM, Jarimatti Valkonen < jarimatti.valkonen@REDACTED> wrote: > Joe Armstrong kirjoitti 22.2.2013 kello 20.31: > > > This launches erlang/cowboy "on demand" > > 2) I fetch a web page from "http://localhost:11234" > > Erlang starts and starts cowboy > > Anybody know what the magic stuff in joe.experiments.plist is? > > > Hi (and apologies for the double post, this was intended to the list) > > If you want to launch on-demand when a socket connection is made, then > AFAIK it's a bit more complicated than just editing the launchd property > list. Launchd listens on the socket and passes the socket into the actual > program in it's own data structure. > UUm sounds tricky - I actually adopted Tony's method. I start a distributed Erlang node with a KeepAlive flag and get it to start cowboy. Not quite the same thing but good enough for my purposes Cheers /Joe > > The file format documentation (man 5 launchd.plist) specifies how to > configure the socket and there's some Apple sample code here: > > > https://developer.apple.com/library/mac/#samplecode/SampleD/Introduction/Intro.html > > A full example with plist is included in the download, but the sample > launchd plist file is not visible in the web site. > > > br, > Jarimatti > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.girondel@REDACTED Sat Feb 23 23:31:17 2013 From: olivier.girondel@REDACTED (Olivier Girondel) Date: Sat, 23 Feb 2013 23:31:17 +0100 Subject: [erlang-questions] Can one load non-driver/nif files in Erlang? In-Reply-To: References: Message-ID: Hi, Can you explain what is your use case ? I have an application that dynamically loads 'plugins' as .so, in pure C The loader is a linked-in driver (.so, erlang wrapper) that dynamically loads other (.so, C plugins) The only problem I found is that the VM dlopens() the driver without RTLD_GLOBAL, which might become a problem if other .so need symbols HTH On Sat, Feb 23, 2013 at 10:27 AM, Gleb Peregud wrote: > No. It is not possible > > On 23 Feb 2013 01:23, "Fredrik Linder" wrote: >> >> Hello >> >> (A) Is it possible to load .so files in Erlang R15+ that are neither >> erlang drivers nor erlang nifs? >> >> (b) If it is possible, could you provide an example snippet? -- Olivier From gabre@REDACTED Mon Feb 25 08:20:21 2013 From: gabre@REDACTED (=?iso-8859-2?b?SG9zc3r6IA==?= =?iso-8859-2?b?R+Fib3I=?=) Date: Mon, 25 Feb 2013 08:20:21 +0100 Subject: [erlang-questions] Wx Modal dialog Message-ID: <20130225082021.200072qj7mjlztf9@webmail.elte.hu> Hello, How is it possible to make a (modal) dialog window in wxErlang? How can I extract the values from the window? (I d like to make a more complex dialog with 2-3 input boxes and other controls.) I ve checked the documentation of wxDialog and noticed that "it freezes wx" (a modal dialog). Regards, Gabor Hosszu From jesper.louis.andersen@REDACTED Mon Feb 25 11:04:09 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 25 Feb 2013 11:04:09 +0100 Subject: [erlang-questions] GADTs / polymorphic data types? In-Reply-To: References: Message-ID: <3CA40777-F581-470D-8E9D-B803C30967E8@erlang-solutions.com> On Feb 13, 2013, at 2:40 AM, Erik S?e S?rensen wrote: > > The form GADTs usually take is tuples-with-fixed-atoms-as-first-element (or just atoms for variants that don't contain data). > Haskell GADTs like > data Either a b = Left a | Right b > data Maybe a = Some a | None > For the record, these are only ADTs, not GADTs. A GADT relaxes a type binder (variable) so it can specialize in certain subtrees of a data type. But this is not needed in Erlang as it is a "mutually consenting adult" language. You will need to handle it yourself. Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From aronisstav@REDACTED Mon Feb 25 11:31:32 2013 From: aronisstav@REDACTED (Stavros Aronis) Date: Mon, 25 Feb 2013 11:31:32 +0100 Subject: [erlang-questions] Letting 'every process run until it is blocked' on an Erlang node. Message-ID: (Also on Stack Overflow: http://stackoverflow.com/questions/15064887/letting-every-process-run-until-it-is-blocked-on-an-erlang-node ) Is it possible to write a function that waits for every process running on an Erlang node to reach a point where it is blocked, waiting for a message? The function should return only when every process is waiting for a message that has not yet been sent to it. Assume that no process is in a time-related suspension (receive with an after clause, timer-related operations etc). The process running this function is, of course, excluded. Obviously wrong answer: `erlang:yield/0`: This gives a chance to every other process to run, but not necessarily until it is blocked. Not 100% correct approach: only_one_not_waiting() -> Running = [P || P <- processes(), process_info(P, status) =/= {status, waiting}], length(Running) == 1 end. everyone_blocked() -> case only_one_not_waiting() of true -> ok; false -> everyone_blocked() end. Ignoring timers, running `only_one_not_waiting/0` repeatedly until it returns true (as `everyone_blocked/0` does) should indicate the desired system state, if this state is eventually reached. However I am not sure how much trust one should put on the return value of `process_info(P, status)`. From poprosturadek@REDACTED Mon Feb 25 12:55:14 2013 From: poprosturadek@REDACTED (=?UTF-8?B?UmFkb3PFgmF3IE1pc2l1aw==?=) Date: Mon, 25 Feb 2013 12:55:14 +0100 Subject: [erlang-questions] RIPEMD 160 in R16B Message-ID: <512B5122.2040102@gmail.com> Hello, is there any possibility to add support for RIPEMD 160 in Erlang's crypto module? There was even some topic on erlang-patches list about this (http://erlang.org/pipermail/erlang-patches/2012-August/003003.html). Also, would it be possible to add option like [hex] (or whatever) to crypto's hash functions to allow them generating hex representations instead of binary ones? I think it would more convenient than manually converting binary into string representations. Last questions I've got: due to adding more and more Unicode support to Erlang, would it be possible in future to pass Unicode data into crypto's functions directly? Now it generates an error and I have to convert it using unicode module before passing into crypto's functions. Regards, Radek From jesper.louis.andersen@REDACTED Mon Feb 25 13:22:21 2013 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 25 Feb 2013 13:22:21 +0100 Subject: [erlang-questions] Letting 'every process run until it is blocked' on an Erlang node. In-Reply-To: References: Message-ID: On Feb 25, 2013, at 11:31 AM, Stavros Aronis wrote: > (Also on Stack Overflow: > http://stackoverflow.com/questions/15064887/letting-every-process-run-until-it-is-blocked-on-an-erlang-node > ) > > Is it possible to write a function that waits for every process > running on an Erlang node to reach a point where it is blocked, > waiting for a message? > I have some code which does so for a set of processes by calling process info for each process as you do. But, I retry until the processes reach a fixpoint in reduction count and status. This is not entirely safe either, but it is better than just looking at the state. And it works for some of my Erlang QuickCheck test cases when considering if they are stable. Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From janhenrynystrom@REDACTED Mon Feb 25 13:30:24 2013 From: janhenrynystrom@REDACTED (Jan Henry Nystrom) Date: Mon, 25 Feb 2013 13:30:24 +0100 Subject: [erlang-questions] Letting 'every process run until it is blocked' on an Erlang node. In-Reply-To: References: Message-ID: Hi, Using tracing with the running flag will let you see when a process was scheduled out and in which function: erlang:trace(new, true, [running]). 0 2> spawn(fun() -> receive X -> X end end). <0.34.0> 3> flush(). Shell got {trace,<0.33.0>,in,{erlang,apply,2}} Shell got {trace,<0.33.0>,out,{io,wait_io_mon_reply,2}} Shell got {trace,<0.33.0>,in,{io,wait_io_mon_reply,2}} Shell got {trace,<0.34.0>,in,{erlang,apply,2}} Shell got {trace,<0.34.0>,out,{erl_eval,receive_clauses,6}} ?. But my hunch is that if you are trying to do this your are doing something fundamentally wrong, but hey without more information it is hard to tell. /Cheers Henry On Feb 25, 2013, at 1:22 PM, Jesper Louis Andersen wrote: > > On Feb 25, 2013, at 11:31 AM, Stavros Aronis wrote: > >> (Also on Stack Overflow: >> http://stackoverflow.com/questions/15064887/letting-every-process-run-until-it-is-blocked-on-an-erlang-node >> ) >> >> Is it possible to write a function that waits for every process >> running on an Erlang node to reach a point where it is blocked, >> waiting for a message? >> > > I have some code which does so for a set of processes by calling process info for each process as you do. But, I retry until the processes reach a fixpoint in reduction count and status. This is not entirely safe either, but it is better than just looking at the state. And it works for some of my Erlang QuickCheck test cases when considering if they are stable. > > Jesper Louis Andersen > Erlang Solutions Ltd., Copenhagen > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From garret.smith@REDACTED Mon Feb 25 17:58:51 2013 From: garret.smith@REDACTED (Garret Smith) Date: Mon, 25 Feb 2013 08:58:51 -0800 Subject: [erlang-questions] Distributed OTP Apps: Failover and Takeover In-Reply-To: References: Message-ID: Hi Andreas, This is due to the failover mechanism you are using, ie marking the app as 'distributed' in sys.config. This makes the kernel start the dist_ac controller to manage your app. Unfortunately, the dist_ac controller is looking for nodedown, not app dieing. Reading carefully on one of the pages your writeup links to ( http://learnyousomeerlang.com/distributed-otp-applications) "The only thing they do is wait for the node of the running application to die. This means that when the node that runs the app dies, another node starts running it instead. The dist_ac controller is also susceptible to net splits. If the network connection between your 2 nodes goes down, the app will be running on both nodes. Once the network heals again, dist_ac doesn't know what to do about it. I see in your example both nodes are on the same machine, so you aren't susceptible to this limitation. I got bit by these problems a while back. I eventually had to roll my own distributed app controller based on gen_leader: https://github.com/garret-smith/gen_leader_revival -Garret Smith On Mon, Feb 11, 2013 at 4:31 AM, Andreas Pauley wrote: > Hi everyone, > > I've made a demo app of mine distributed to test failover and > takeover, after reading the "Distributed OTP Applications" chapter in > Learn you some Erlang. > > The failover and takeover works great if I kill the running beam (eg. > with kill -9). > > However, I tried sending kill signals to the Pids of both my > application behavior and the top supervisor that gets started by the > application. > This crashes the VM, but failover does not happen. > > Is this unsupported, or should I do something to enable failover in > this scenario? > > I've done a more complete writeup with code and output here: > > https://github.com/apauley/dark-overlord#when-processes-die-a-guide-to-the-afterlife > > But in a nutshell, I crash the VM with the commands below, and then > automatic failover to my second node does not happen: > > $ ./rel/overlord/bin/overlord console > Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:8:8] > [async-threads:0] [hipe] [kernel-poll:false] [dtrace] > > 14:03:20.581 overlord@REDACTED [overlord_app] <0.56.0> || Starting app: > normal > 14:03:20.582 overlord@REDACTED [hypnosponge_sup] <0.57.0> || Hello > from the hypnosponge supervisor > (overlord@REDACTED)1> Sup = pid(0, 57, 0). > <0.57.0> > (overlord@REDACTED)2> exit(Sup, kill). > > =ERROR REPORT==== 11-Feb-2013::14:04:46 === > ** Generic server minion_supersup terminating > ** Last message in was {'EXIT',<0.57.0>,killed} > ** When Server state == {state, > {local,minion_supersup}, > simple_one_for_one, > [{child,undefined,minion_makeshift_sup, > {minion_makeshift_sup,start_link,[]}, > temporary,5000,worker, > [minion_makeshift_sup]}], > undefined,1,3,[],minion_supersup,[]} > ** Reason for termination == > ** killed > true > (overlord@REDACTED)3> > =INFO REPORT==== 11-Feb-2013::14:04:46 === > application: overlord > exited: killed > type: permanent > > (overlord@REDACTED)3> {"Kernel pid > > terminated",application_controller,"{application_terminated,overlord,killed}"} > > Crash dump was written to: erl_crash.dump > Kernel pid terminated (application_controller) > ({application_terminated,overlord,killed}) > > -- > http://pauley.org.za/ > http://twitter.com/apauley > http://www.meetup.com/lambda-luminaries/ > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Feb 25 22:24:45 2013 From: erlang@REDACTED (Erik Reitsma) Date: Mon, 25 Feb 2013 22:24:45 +0100 Subject: [erlang-questions] Erlang4Android In-Reply-To: References: <50F313AE.8050303@ernovation.com> Message-ID: <512BD69D.2010807@ernovation.com> Hi David and all, I had not assumed that people would want to use Erlang on Android in distributed mode. However, now that there appears a "market" for this, I will try to get distribution working too. I think that there may be some issues with ownership of the cookie file, and the user id under which Erlang runs. I intend to publish a new version as soon as R16B is out (which should be in a few days). I would prefer to try to get distribution working in that version. I have also worked a bit on a new installer, which makes it possible to install the small version first, and add libraries later. Regards, *Erik. On 02/25/2013 12:45 PM, D?vid Juh?sz wrote: > Hi Erik and all, > > The work that has been done is awesome and I am happy to be able to > run Erlang code on Android devices! > > Unfortunately, I ran into a technical problem trying to run > distributed Erlang programs on Android via SL4A. In root terminal, > distributed mode works smoothly, but in that case I am unable to > create GUI for the program. On the other hand, programs started via > the scripting layer are running without root privileges which causes > some trouble. > > It seems unavoidable that distributed mode must be initialized > programatically using net_kernel:start/1 as SL4A doesn't start Erlang > in distributed mode. The first problem I encountered was the > insufficient privilege to create a cookie file. Having one created > manually, that part of initialization seems to be successful. The next > problem seems to be that epmd isn't getting started. I can start epmd > in daemon mode from the Erlang program using os:cmd/1, so the program > and epmd run under the same user. But the same issue appears: > > {error,{shutdown,{child,undefined,net_sup_dynamic, > {erl_distribution,start_link,[["node@REDACTED"]]}, > permanent,1000,supervisor, > [erl_distribution]}}} > > Can someone give me any pointer how to overcome this issue of using > GUI and distributed mode at the same time? > > Thanks in advance for any help! > > Best wishes, > David > > > 2013. janu?r 13., vas?rnap 21:06:06 UTC+1 id?pontban Erik Reitsma a > k?vetkez?t ?rta: > > Hi all, > > I have put my port of Erlang to Android on Google Code: > http://code.google.com/p/erlang4android/ > > > To install it on your Android device, you should allow > applications from unknown sources, because the app is not > delivered through Google Play (yet). The Android device does *not* > have to be rooted. > > It depends on Scripting Layer for Android (SL4A). So in order to > use it, you should first install SL4A r6 from here: > http://code.google.com/p/android-scripting/downloads/detail?name=sl4a_r6.apk > > Then install the APK for Erlang, this one: > http://code.google.com/p/erlang4android/downloads/detail?name=ErlangForAndroid.apk > > This app is just the installer. Run it to actually install > Erlang/OTP. I have included as much of OTP as might be remotely > useful, so the download is 36.5 MB. Of course you want to use the > megaco and orber applications on your phone! > > You can use the SL4A API through the module android that I have > added. I have added a very simple example, to show you how you can > make a small gui and respond to events. Study the SL4A API to do > other Android things. > > You can run some Erlang code by putting your source in the SL4A > scripts directory on the Android device, which is > /sdcard/sl4a/scripts/. The .erl file should contain and export a > function main/0, which will be called when the script is run. The > .erl file will be compiled to a .beam file automatically, only if > the .beam file does not exist. A minor inconvenience now is, that > if the .erl file is modified, it will not be recompiled unless the > .beam file is removed. The scripts directory will be in the code > path, so if you put other .beam files there, they will be found, > and .erl files there will be compiled too, if necessary. > > Enjoy! > > *Erik. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Tue Feb 26 00:25:29 2013 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Tue, 26 Feb 2013 00:25:29 +0100 Subject: [erlang-questions] Letting 'every process run until it is blocked' on an Erlang node. In-Reply-To: References: Message-ID: 2013/2/25 Jan Henry Nystrom [snip] > > But my hunch is that if you are trying to do this your are doing something > fundamentally wrong, but hey without > more information it is hard to tell. > I must admit to having done something similar. Not in production code (of course!), but as an alternative to "wait a fixed number of milliseconds" in unit and integration tests. As far as I remember, what I did was loop (up to a certain limit) until no processes were running and all message queues were empty. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3@REDACTED Tue Feb 26 01:10:59 2013 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 26 Feb 2013 01:10:59 +0100 Subject: [erlang-questions] Am I leaking memory inside my linked in driver? Message-ID: Hi guys, If I keep calling ei_x_encode_atom(ei_x_buff* x, const char *p)) (http://erlang.org/doc/man/ei.html#ei_x_encode_atom) again and again to create the same atom 'ok' ... am I leaking memory? Yes, I can create the atom 'ok' since the beginning as a static variable and use its reference. But want to be sure I'm leaking memory (or not) in the first case. Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Tue Feb 26 01:40:21 2013 From: serge@REDACTED (Serge Aleynikov) Date: Mon, 25 Feb 2013 19:40:21 -0500 Subject: [erlang-questions] Am I leaking memory inside my linked in driver? In-Reply-To: References: Message-ID: <512C0475.6000907@aleynikov.org> The ei library in C doesn't implements atoms in the same way how Erlang emulator does. I.e. there are no atom structures created or destroyed - the atoms binary representation is encoded in external Erlang binary format to the pointer pointed by ei_x_buff*. So as long as you initialize the ei_x_buff using ei_x_new() and deallocate it with ei_x_free() you are fine. On 2/25/2013 7:10 PM, Zabrane Mickael wrote: > Hi guys, > > If I keep calling ei_x_encode_atom(ei_x_buff* x, const char *p)) > (http://erlang.org/doc/man/ei.html#ei_x_encode_atom) > again and again to create the same atom 'ok' ... am I leaking memory? > > Yes, I can create the atom 'ok' since the beginning as a static variable > and use its reference. > But want to be sure I'm leaking memory (or not) in the first case. > > Regards, > Zabrane > > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From zabrane3@REDACTED Tue Feb 26 06:52:36 2013 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 26 Feb 2013 06:52:36 +0100 Subject: [erlang-questions] Am I leaking memory inside my linked in driver? In-Reply-To: <512C0475.6000907@aleynikov.org> References: <512C0475.6000907@aleynikov.org> Message-ID: Thanks a lot Serge. On Feb 26, 2013, at 1:40 AM, Serge Aleynikov wrote: > [...] > > So as long as you initialize the ei_x_buff using ei_x_new() and > deallocate it with ei_x_free() you are fine. Yup, I'm deallocating memory with ei_x_free() all the time. Final question: is it better in term of performances to use "driver_mk_atom("ok");" instead? I mean, declare all my atoms (as static variables) once at the beginning and use their references. No need for me to deallocate anything in that case, right? Regards, Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Tue Feb 26 14:01:46 2013 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 26 Feb 2013 08:01:46 -0500 Subject: [erlang-questions] Am I leaking memory inside my linked in driver? In-Reply-To: References: <512C0475.6000907@aleynikov.org> Message-ID: <512CB23A.5010807@aleynikov.org> On 2/26/2013 12:52 AM, Zabrane Mickael wrote: > Final question: is it better in term of performances to use > "driver_mk_atom("ok");" instead? > I mean, declare all my atoms (as static variables) once at the beginning > and use their references. > No need for me to deallocate anything in that case, right? The ei_x_encode_atom() and driver_mk_atom() serve different purposes. The first one encodes a string into external Erlang binary format given a string representation of the atom (i.e. serializes it in a wire-level format to be sent elsewhere). It is to be used to assist with communicating with Erlang in a port program running in a separate process, and in a driver using the "output" callback function. The second one driver_mk_atom() is used in drivers that run in the address space of the emulator to create atoms by the emulator. So the answer to your question depends on what it is that you are writing. If you are writing a driver, then yes, use driver_mk_atom() and you don't need to deallocate anything - atoms are global and not garbage collected. If you are writing a port program, then driver_mk_atom() has no meaning for a port program, and is not part of exported ei interface, so you can't use it. From zabrane3@REDACTED Tue Feb 26 14:30:23 2013 From: zabrane3@REDACTED (Zabrane Mickael) Date: Tue, 26 Feb 2013 14:30:23 +0100 Subject: [erlang-questions] Am I leaking memory inside my linked in driver? In-Reply-To: <512CB23A.5010807@aleynikov.org> References: <512C0475.6000907@aleynikov.org> <512CB23A.5010807@aleynikov.org> Message-ID: Crystal clear Serge. I'm writing a driver, so i'll stick with "ei_xxx" for now. Regards, Zabrane On Feb 26, 2013, at 2:01 PM, Serge Aleynikov wrote: > On 2/26/2013 12:52 AM, Zabrane Mickael wrote: >> Final question: is it better in term of performances to use >> "driver_mk_atom("ok");" instead? >> I mean, declare all my atoms (as static variables) once at the beginning >> and use their references. >> No need for me to deallocate anything in that case, right? > > The ei_x_encode_atom() and driver_mk_atom() serve different purposes. > > The first one encodes a string into external Erlang binary format given > a string representation of the atom (i.e. serializes it in a wire-level > format to be sent elsewhere). It is to be used to assist with > communicating with Erlang in a port program running in a separate > process, and in a driver using the "output" callback function. > > The second one driver_mk_atom() is used in drivers that run in the > address space of the emulator to create atoms by the emulator. > > So the answer to your question depends on what it is that you are > writing. If you are writing a driver, then yes, use driver_mk_atom() > and you don't need to deallocate anything - atoms are global and not > garbage collected. > > If you are writing a port program, then driver_mk_atom() has no meaning > for a port program, and is not part of exported ei interface, so you > can't use it. From erlang@REDACTED Tue Feb 26 21:39:43 2013 From: erlang@REDACTED (Erik Reitsma) Date: Tue, 26 Feb 2013 21:39:43 +0100 Subject: [erlang-questions] Erlang4Android In-Reply-To: References: <50F313AE.8050303@ernovation.com> <512BD69D.2010807@ernovation.com> Message-ID: <512D1D8F.4070009@ernovation.com> Hi David & others, I intend to solve this by making it possible to provide extra command line arguments to erl when Erlang is started from SL4A. Unfortunately I cannot set this from SL4A, unless I modify SL4A, which I would very much like to avoid. So, I have to set the command line arguments from the Erlang installer. This means however, that all started shells will be started with the same command line arguments. So, either all start with distribution, or without, unless you change the command line arguments between two starts. I have tried this, and I can start a shell with arguments "-sname x -setcookie ssssssss" (no need for a cookie file this way!). Would this be an acceptable solution for you? *Erik. On 02/25/2013 11:36 PM, Juh?sz D?vid wrote: > Erik, > > thanks for the quick reply! > > 2013/2/25 Erik Reitsma : >> I had not assumed that people would want to use Erlang on Android in >> distributed mode. However, now that there appears a "market" for this, I >> will try to get distribution working too. I think that there may be some >> issues with ownership of the cookie file, and the user id under which Erlang >> runs. > The first problem is the cookie file, indeed. I had to create the file > /data/data/com.ernovation.erlangforandroid/files/erlang/.erlang.cookie, > set privileges to 700 and give it to the user belonging to SL4A > manually in order to overcome error messages pointing explicitly to > the cookie file. After that, there is no explicit sign of any error > (of course, distributed mode doesn't turn on in the background) unless > one checks the value of net_kernel:start/1, to which I referred in my > initial mail. However, I was able to reproduce such values in android > root terminal and even on my laptop when calling net_kernel:start/1 > without running epmd. So I think the problem here is that epmd can't > be reached by the runtime, and that can't be helped with simply > starting epmd by the same user on android. > > Hope that it gives you some pointer to solve the problem. > >> I intend to publish a new version as soon as R16B is out (which should be in >> a few days). I would prefer to try to get distribution working in that >> version. > Of course, that would be fine. Thanks for your work! > > Best regards, > David > >> I have also worked a bit on a new installer, which makes it possible to >> install the small version first, and add libraries later. >> >> Regards, >> *Erik. >> >> On 02/25/2013 12:45 PM, D?vid Juh?sz wrote: >> >> Hi Erik and all, >> >> The work that has been done is awesome and I am happy to be able to run >> Erlang code on Android devices! >> >> Unfortunately, I ran into a technical problem trying to run distributed >> Erlang programs on Android via SL4A. In root terminal, distributed mode >> works smoothly, but in that case I am unable to create GUI for the program. >> On the other hand, programs started via the scripting layer are running >> without root privileges which causes some trouble. >> >> It seems unavoidable that distributed mode must be initialized >> programatically using net_kernel:start/1 as SL4A doesn't start Erlang in >> distributed mode. The first problem I encountered was the insufficient >> privilege to create a cookie file. Having one created manually, that part of >> initialization seems to be successful. The next problem seems to be that >> epmd isn't getting started. I can start epmd in daemon mode from the Erlang >> program using os:cmd/1, so the program and epmd run under the same user. But >> the same issue appears: >> >> {error,{shutdown,{child,undefined,net_sup_dynamic, >> >> {erl_distribution,start_link,[["node@REDACTED"]]}, >> permanent,1000,supervisor, >> [erl_distribution]}}} >> >> Can someone give me any pointer how to overcome this issue of using GUI and >> distributed mode at the same time? >> >> Thanks in advance for any help! >> >> Best wishes, >> David >> >> >> 2013. janu?r 13., vas?rnap 21:06:06 UTC+1 id?pontban Erik Reitsma a >> k?vetkez?t ?rta: >>> Hi all, >>> >>> I have put my port of Erlang to Android on Google Code: >>> http://code.google.com/p/erlang4android/ >>> >>> To install it on your Android device, you should allow applications from >>> unknown sources, because the app is not delivered through Google Play (yet). >>> The Android device does *not* have to be rooted. >>> >>> It depends on Scripting Layer for Android (SL4A). So in order to use it, >>> you should first install SL4A r6 from here: >>> http://code.google.com/p/android-scripting/downloads/detail?name=sl4a_r6.apk >>> Then install the APK for Erlang, this one: >>> http://code.google.com/p/erlang4android/downloads/detail?name=ErlangForAndroid.apk >>> This app is just the installer. Run it to actually install Erlang/OTP. I >>> have included as much of OTP as might be remotely useful, so the download is >>> 36.5 MB. Of course you want to use the megaco and orber applications on your >>> phone! >>> >>> You can use the SL4A API through the module android that I have added. I >>> have added a very simple example, to show you how you can make a small gui >>> and respond to events. Study the SL4A API to do other Android things. >>> >>> You can run some Erlang code by putting your source in the SL4A scripts >>> directory on the Android device, which is /sdcard/sl4a/scripts/. The .erl >>> file should contain and export a function main/0, which will be called when >>> the script is run. The .erl file will be compiled to a .beam file >>> automatically, only if the .beam file does not exist. A minor inconvenience >>> now is, that if the .erl file is modified, it will not be recompiled unless >>> the .beam file is removed. The scripts directory will be in the code path, >>> so if you put other .beam files there, they will be found, and .erl files >>> there will be compiled too, if necessary. >>> >>> Enjoy! >>> >>> *Erik. >> From desired.mta@REDACTED Tue Feb 26 22:13:03 2013 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Tue, 26 Feb 2013 21:13:03 +0000 Subject: [erlang-questions] Erlang4Android In-Reply-To: <512D1D8F.4070009@ernovation.com> References: <50F313AE.8050303@ernovation.com> <512BD69D.2010807@ernovation.com> <512D1D8F.4070009@ernovation.com> Message-ID: On Tue, Feb 26, 2013 at 8:39 PM, Erik Reitsma wrote: > Hi David & others, > > I intend to solve this by making it possible to provide extra command line > arguments to erl when Erlang is started from SL4A. Unfortunately I cannot > set this from SL4A, unless I modify SL4A, which I would very much like to > avoid. So, I have to set the command line arguments from the Erlang > installer. This means however, that all started shells will be started with > the same command line arguments. So, either all start with distribution, or > without, unless you change the command line arguments between two starts. I > have tried this, and I can start a shell with arguments "-sname x -setcookie > ssssssss" (no need for a cookie file this way!). > Would this be an acceptable solution for you? > Maybe you can control environment variables instead? These are equivalent to command-line arguments. ERL_FLAGS, ERL_AFLAGS, ERL_ZFLAGS If not, I would go for non-distributed approach, given that users can "go distributed" by themselves. $ erl Erlang R15B02 (erts-5.9.2) [source] [smp:2:2] [async-threads:0] [kernel-poll:false] Eshell V5.9.2 (abort with ^G) 1> net_kernel:start(['t@REDACTED', longnames]). {ok,<0.34.0>} (t@REDACTED)2> erlang:set_cookie(node(), yadda). true (t@REDACTED)3> Regards, Motiejus From jayson.barley@REDACTED Wed Feb 27 00:45:40 2013 From: jayson.barley@REDACTED (Jayson Barley) Date: Tue, 26 Feb 2013 15:45:40 -0800 Subject: [erlang-questions] messages manipulatio In-Reply-To: References: Message-ID: I did something similar for a gated worker pool. It allowed me to slow the incoming data to only as fast as my workers are able to process it. >From the worker process: handle_cast(get_data, State = #state{data=Data}) -> gp_server:child_data(Data), {noreply, State#state{data=undefined}}; handle_cast(run, State = #state{worker_fun=Fun, data_provider={Module, Function, Args}, data=undefined}) Input = apply(Module, Function, Args), Data = apply(Fun, Input), gp_server:child_ready(self()), {noreply, State#state{data=Data}}. >From the server process: handle_call(get_data, _From, State = #state{ready_children=[Child | Rest], working_children=Working}) -> Reply = gen_server:call(Child, get_data), {reply, Reply, State#state{ready_children=Rest, working_children=[Child | Working]}}. handle_cast({ready, Child}, State = #state{ready_children=Children, working_children=Working}) -> Rest = lists:delete(Child, Working), {noreply, State#state{ready_children=[Child | Children], working_children=Rest}}. It worked well for how I was using it. On Sat, Feb 16, 2013 at 8:01 AM, sasa wrote: > Having thought more, I realized the original approach can be simplified. > This would be the modified version: > > 1. The facade keeps track whether the worker is available or it is working. > 2. Upon receiving a message, the facade joins it with the messages already > in the queue. Then, if the worker is available, it immediately sends data > to it, and clears its own queue. > 3. When the worker receives the notification, it performs the work, then > it notifies the facade that it is available. > 4. When the facade receives the information that the worker is available, > if there is new data available, it will immediately send it to the worker > as explained in 2. > > Thank you for your question, it had made me realized there is place for > simplification. > > On Sat, Feb 16, 2013 at 5:58 AM, Erik S?e S?rensen wrote: > >> I must confess that (at least at this late hour) your Step 2 confuses me. >> Why not just 1+3+4, or indeed *just* step 2? >> >> Regards, >> /Erik >> Den 15/02/2013 16.47 skrev "sasa" : >> >>> Hello, >>> >>> A while ago I encountered the following situation: >>> >>> I had the gen_server base process P which would receive messages, and >>> handle them by sending some data over the network. The messages were coming >>> faster than they were being sent. I established the reason for this was the >>> "randomness" of my network conditions. I also established that sending more >>> messages at once was almost as fast as sending one message, i.e. the >>> network push time wasn't highly dependent on the message size. >>> >>> To tackle this in a generic way, I devised an approach which has served >>> me well in multiple places. I was repeatedly googling whether some similar >>> solution exists, but I couldn't find it. Now, I'm not sure if I have >>> reinvented a wheel, or the approach is not optimal, so I'm asking if you >>> are aware of similar approaches, and are there any faults in this one? >>> >>> The approach I took is following: >>> >>> I split the server in two processes: the "facade" and the worker. The >>> facade acceptes requests from clients, and stores them internally. While >>> the worker is doing its work, new messages are stored in the facade. When >>> the worker is available, it will take all accumulated messages from the >>> facade and process them. >>> >>> These are the steps: >>> 1. The facade receives messages, stores data in its list, and notifies >>> the worker (without sending actual data), that some work is ready. >>> 2. Upon receiving the notification, the worker first flushes its message >>> queue by doing repeated receive ... after 0 as long as there are messages >>> in the queue. >>> 3. Then the worker pulls all messages from the facade. This is a >>> gen_server:call to the facade which will return all messages, and at the >>> same time remove them from its state. >>> 4. Finally, the worker processes all messages. >>> >>> >>> I found this approach useful because the delay on the worker adapts to >>> the incoming message rate. >>> If the worker can handle messages at the incoming rate, everything works >>> without delay. >>> If messages can't be handled at the incoming rate, the worker's delay >>> will increase to accomodate the higher load. In other words, the worker >>> will try to compensate the load by bulk processing messages. Obviously, >>> this is useful only when process_time(N messages) < N * process_time(1 >>> message). >>> >>> Another benefit I found is that I can vary the implementation of the >>> facade i.e. I can store messages using different algorithms. In the first >>> implementation, I stored messages in a list. In one variation, I used hash >>> which allowed me to eliminate duplicate messages. Another variant was >>> truncation of the list, which allowed me to discard old messages if the >>> queue was getting too large. >>> >>> As I said, this has served me well in the production for more than a >>> year, and I have finally found the time to make a generic library out of >>> it. Before putting it public, I'd like to check if there are similar >>> solutions, or alternative approaches? >>> >>> Thanks, and best regards, >>> Sasa >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From baliulia@REDACTED Wed Feb 27 12:04:31 2013 From: baliulia@REDACTED (=?UTF-8?Q?Ignas_Vy=C5=A1niauskas?=) Date: Wed, 27 Feb 2013 12:04:31 +0100 Subject: [erlang-questions] rpc:multicall/5 docs misleading? Message-ID: Hi, the docs for `rpc:multicall/5` claim: > In contrast to an RPC, a multicall is an RPC which is sent concurrently from > one client to multiple servers. This is useful for collecting some > information from a set of nodes, or for calling a function on a set of nodes > to achieve some side effects. It is *semantically the same as iteratively > making a series of RPCs on all the nodes* <..> So, I would expect that, since this works: get_results(Nodes) -> lists:zip(Nodes, [rpc:call(N, erlang, now, []) || N <- Nodes]). this would also work: get_results(Nodes) -> {Results, BadNodes} = rpc:multicall(Nodes, erlang, now, []), lists:zip(Nodes, Results). but the `lists:zip/2` will fail if some node is non-reachable during the multicall since Results will be a shorter list, so in order to really be semantically the same (assuming `Nodes` does not contain duplicate entries), you would need: get_results(Nodes) -> {Results, BadNodes} = rpc:multicall(Nodes, erlang, now, []), FullResults = lists:append( lists:zip(Nodes -- BadNodes, Results), lists:zip(BadNodes, {badrpc, nodedown}) ), %% restore original order [{N, Res} || N <- Nodes, {M, Res} <- FullResults, N == M]. Which, at least to me, is a bit unexpected given the docs. I would suggest either clarifying the docs or, better, changing the implementation so that it becomes more useful. Clearly the returned `BadNodes` are just a leftover from the response from `gen_server:multicall/4` and IMHO useless in this case as it provides no information as to what went wrong and only introduces confusion. -- Ignas From baliulia@REDACTED Wed Feb 27 12:40:50 2013 From: baliulia@REDACTED (=?UTF-8?Q?Ignas_Vy=C5=A1niauskas?=) Date: Wed, 27 Feb 2013 12:40:50 +0100 Subject: [erlang-questions] rpc:multicall/5 docs misleading? In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 12:04 PM, Ignas Vy?niauskas wrote: > lists:zip(BadNodes, {badrpc, nodedown}) This should obviously be [{BN, {badrpc, nodedown}} || BN <- BadNodes] instead. From karol.urbanski@REDACTED Wed Feb 27 13:54:53 2013 From: karol.urbanski@REDACTED (Karol Urbanski) Date: Wed, 27 Feb 2013 13:54:53 +0100 Subject: [erlang-questions] R16A binary packages for CentOS, Debian, Mac, Ubuntu and Windows Message-ID: <20130227125452.GA20418@dex.krakow> Hello, You can find our binary packages for the newest Erlang release (R16B) at http://www.erlang-solutions.com/downloads/download-erlang-otp The systems we provide the packages for are CentOS 6, Mac OS X Snow Leopard and newer, Debian 6.0.3, Ubuntu 12.10 + 12.04 + 11.10 + 10.04, and Windows. More to come soon! Best regards, Karol Urbanski From kenneth.lundin@REDACTED Wed Feb 27 14:09:05 2013 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Wed, 27 Feb 2013 14:09:05 +0100 Subject: [erlang-questions] OTP R16B has been released Message-ID: Erlang/OTP R16B has been released. OTP R16B is a major release with a number of new features, improved characteristics and also some minor incompatibilities. We would like to thank all of you that tested the beta release (R16A) and gave us valuable feedback. See the readme file and the documentation for more details. Some of the highlights are: - Optimized handling of processes in the VM - New internal process table allowing for parallel reads and writes - optimized run queue management - optimized process state changes - "Non-blocking" code loading - New internal port table, and rewrite of scheduling of port tasks. - Dynamic allocation of port structures, allowing the default for maximum ports to be raised to 65536 (from 1024) - Support for UTF-8 encoded Unicode characters in source files. - Inets application: The http client now support HTTPS through a proxy - Asn1 application: Major cleanup of back ends and optimizations of mainly decode for PER and UPER. - The experimental features parameterized modules and packages are removed. An alternative compatible solution for parameterized modules is provided here https://github.com/erlang/pmod_transform. - All built in functions BIF's (even if they are implemented in C as part of the VM) are now visible in the source code of the module they belong to, including their type specs. - The Wx application now compiles and is usable with the unstable development branch of wxWidgets-2.9. This means that wx can now be built on 64 bit MacOsX as well. You can find the README file with more detailed info at http://www.erlang.org/download/otp_src_R16B.readme You can download the full source distribution from http://www.erlang.org/download/otp_src_R16B.tar.gz http://www.erlang.org/download/otp_src_R16B.readme (this file) Note: To unpack the TAR archive you need a GNU TAR compatible program. For installation instructions please read the README that is part of the distribution. You can also find this release at the official Erlang/OTP Git-repository at Github here: https://github.com/erlang/otp tagged *OTP_R16B* The Windows binary distribution can be downloaded from http://www.erlang.org/download/otp_win32_R16B.exe http://www.erlang.org/download/otp_win64_R16B.exe On-line documentation can be found at http://www.erlang.org/doc/. You can also download the complete HTML documentation or the Unix manual files http://www.erlang.org/download/otp_doc_html_R16B.tar.gz http://www.erlang.org/download/otp_doc_man_R16B.tar.gz We also want to thank those that sent us patches, suggestions and bug reports, The Erlang/OTP Team at Ericsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From karol.urbanski@REDACTED Wed Feb 27 14:39:41 2013 From: karol.urbanski@REDACTED (Karol Urbanski) Date: Wed, 27 Feb 2013 14:39:41 +0100 Subject: [erlang-questions] R16B binary packages for CentOS, Debian, Mac, Ubuntu and Windows Message-ID: <20130227133940.GC20418@dex.krakow> Hello, You can find our binary packages for the newest Erlang release (R16B) at http://www.erlang-solutions.com/downloads/download-erlang-otp The systems we provide the packages for are CentOS 6, Mac OS X Snow Leopard and newer, Debian 6.0.3, Ubuntu 12.10 + 12.04 + 11.10 + 10.04, Fedora 17 and Windows. More to come soon! Resend due to a typo in the subject earlier. ;) Best regards, Karol Urbanski From sasa555@REDACTED Wed Feb 27 16:34:46 2013 From: sasa555@REDACTED (sasa) Date: Wed, 27 Feb 2013 16:34:46 +0100 Subject: [erlang-questions] messages manipulatio In-Reply-To: References: Message-ID: >From what I can see, this is more a pooling, but it has similarities to what I described. On Wed, Feb 27, 2013 at 12:45 AM, Jayson Barley wrote: > I did something similar for a gated worker pool. It allowed me to slow the > incoming data to only as fast as my workers are able to process it. > > From the worker process: > handle_cast(get_data, State = #state{data=Data}) -> > gp_server:child_data(Data), > {noreply, State#state{data=undefined}}; > handle_cast(run, State = #state{worker_fun=Fun, data_provider={Module, > Function, Args}, data=undefined}) Input = apply(Module, Function, Args), > Data = apply(Fun, Input), > gp_server:child_ready(self()), > {noreply, State#state{data=Data}}. > > > > From the server process: > handle_call(get_data, _From, State = #state{ready_children=[Child | Rest], > working_children=Working}) -> > Reply = gen_server:call(Child, get_data), > {reply, Reply, State#state{ready_children=Rest, > working_children=[Child | Working]}}. > > handle_cast({ready, Child}, State = #state{ready_children=Children, > working_children=Working}) -> > Rest = lists:delete(Child, Working), > {noreply, State#state{ready_children=[Child | Children], > working_children=Rest}}. > > > > It worked well for how I was using it. > > > On Sat, Feb 16, 2013 at 8:01 AM, sasa wrote: > >> Having thought more, I realized the original approach can be simplified. >> This would be the modified version: >> >> 1. The facade keeps track whether the worker is available or it is >> working. >> 2. Upon receiving a message, the facade joins it with the messages >> already in the queue. Then, if the worker is available, it immediately >> sends data to it, and clears its own queue. >> 3. When the worker receives the notification, it performs the work, then >> it notifies the facade that it is available. >> 4. When the facade receives the information that the worker is available, >> if there is new data available, it will immediately send it to the worker >> as explained in 2. >> >> Thank you for your question, it had made me realized there is place for >> simplification. >> >> On Sat, Feb 16, 2013 at 5:58 AM, Erik S?e S?rensen wrote: >> >>> I must confess that (at least at this late hour) your Step 2 confuses me. >>> Why not just 1+3+4, or indeed *just* step 2? >>> >>> Regards, >>> /Erik >>> Den 15/02/2013 16.47 skrev "sasa" : >>> >>>> Hello, >>>> >>>> A while ago I encountered the following situation: >>>> >>>> I had the gen_server base process P which would receive messages, and >>>> handle them by sending some data over the network. The messages were coming >>>> faster than they were being sent. I established the reason for this was the >>>> "randomness" of my network conditions. I also established that sending more >>>> messages at once was almost as fast as sending one message, i.e. the >>>> network push time wasn't highly dependent on the message size. >>>> >>>> To tackle this in a generic way, I devised an approach which has served >>>> me well in multiple places. I was repeatedly googling whether some similar >>>> solution exists, but I couldn't find it. Now, I'm not sure if I have >>>> reinvented a wheel, or the approach is not optimal, so I'm asking if you >>>> are aware of similar approaches, and are there any faults in this one? >>>> >>>> The approach I took is following: >>>> >>>> I split the server in two processes: the "facade" and the worker. The >>>> facade acceptes requests from clients, and stores them internally. While >>>> the worker is doing its work, new messages are stored in the facade. When >>>> the worker is available, it will take all accumulated messages from the >>>> facade and process them. >>>> >>>> These are the steps: >>>> 1. The facade receives messages, stores data in its list, and notifies >>>> the worker (without sending actual data), that some work is ready. >>>> 2. Upon receiving the notification, the worker first flushes its >>>> message queue by doing repeated receive ... after 0 as long as there are >>>> messages in the queue. >>>> 3. Then the worker pulls all messages from the facade. This is a >>>> gen_server:call to the facade which will return all messages, and at the >>>> same time remove them from its state. >>>> 4. Finally, the worker processes all messages. >>>> >>>> >>>> I found this approach useful because the delay on the worker adapts to >>>> the incoming message rate. >>>> If the worker can handle messages at the incoming rate, everything >>>> works without delay. >>>> If messages can't be handled at the incoming rate, the worker's delay >>>> will increase to accomodate the higher load. In other words, the worker >>>> will try to compensate the load by bulk processing messages. Obviously, >>>> this is useful only when process_time(N messages) < N * process_time(1 >>>> message). >>>> >>>> Another benefit I found is that I can vary the implementation of the >>>> facade i.e. I can store messages using different algorithms. In the first >>>> implementation, I stored messages in a list. In one variation, I used hash >>>> which allowed me to eliminate duplicate messages. Another variant was >>>> truncation of the list, which allowed me to discard old messages if the >>>> queue was getting too large. >>>> >>>> As I said, this has served me well in the production for more than a >>>> year, and I have finally found the time to make a generic library out of >>>> it. Before putting it public, I'd like to check if there are similar >>>> solutions, or alternative approaches? >>>> >>>> Thanks, and best regards, >>>> Sasa >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alessandro.sivieri@REDACTED Wed Feb 27 16:36:58 2013 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Wed, 27 Feb 2013 16:36:58 +0100 Subject: [erlang-questions] Is Valgrind still working? Message-ID: Hi all, the question is the one in the subject: I'm trying to build R15B with Valgrind enabled, but when I execute make valgrind FLAVOR=smp it returns "valgrind is up to date", which is clearly false because there is no beam.valgrind anywhere... so is it still possible to execute Valgrind? In general, what I'm trying to obtain is a profile of the heap during the execution of an Erlang application, because of a comparison I have to do with the same application written in other languages, so if Valgrind is not working I am open to suggestions :) -- Sivieri Alessandro alessandro.sivieri@REDACTED http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Feb 27 16:50:10 2013 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 27 Feb 2013 16:50:10 +0100 Subject: [erlang-questions] Is Valgrind still working? In-Reply-To: References: Message-ID: Hello, We run continuous tests with the valgrind emulator, so it should be working. Are you building from the source tar or from a clone of github? Is ERL_TOP set correctly? What are the build steps (configure, make etc) you do leading up to doing `make valgrind`? Lukas On Wed, Feb 27, 2013 at 4:36 PM, Alessandro Sivieri < alessandro.sivieri@REDACTED> wrote: > Hi all, > > the question is the one in the subject: I'm trying to build R15B with > Valgrind enabled, but when I execute > > make valgrind FLAVOR=smp > > it returns "valgrind is up to date", which is clearly false because there > is no beam.valgrind anywhere... so is it still possible to execute Valgrind? > > In general, what I'm trying to obtain is a profile of the heap during the > execution of an Erlang application, because of a comparison I have to do > with the same application written in other languages, so if Valgrind is not > working I am open to suggestions :) > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://sivieri.wordpress.com/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alessandro.sivieri@REDACTED Wed Feb 27 17:16:40 2013 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Wed, 27 Feb 2013 17:16:40 +0100 Subject: [erlang-questions] Is Valgrind still working? In-Reply-To: References: Message-ID: Ok, looks like the problem is R15B, for R15B01 it works... On Wed, Feb 27, 2013 at 4:50 PM, Lukas Larsson wrote: > Hello, > > We run continuous tests with the valgrind emulator, so it should be > working. Are you building from the source tar or from a clone of github? Is > ERL_TOP set correctly? What are the build steps (configure, make etc) you > do leading up to doing `make valgrind`? > > Lukas > > On Wed, Feb 27, 2013 at 4:36 PM, Alessandro Sivieri < > alessandro.sivieri@REDACTED> wrote: > >> Hi all, >> >> the question is the one in the subject: I'm trying to build R15B with >> Valgrind enabled, but when I execute >> >> make valgrind FLAVOR=smp >> >> it returns "valgrind is up to date", which is clearly false because there >> is no beam.valgrind anywhere... so is it still possible to execute Valgrind? >> >> In general, what I'm trying to obtain is a profile of the heap during the >> execution of an Erlang application, because of a comparison I have to do >> with the same application written in other languages, so if Valgrind is not >> working I am open to suggestions :) >> >> -- >> Sivieri Alessandro >> alessandro.sivieri@REDACTED >> http://sivieri.wordpress.com/ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- Sivieri Alessandro alessandro.sivieri@REDACTED http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas@REDACTED Wed Feb 27 17:23:32 2013 From: lukas@REDACTED (Lukas Larsson) Date: Wed, 27 Feb 2013 17:23:32 +0100 Subject: [erlang-questions] Is Valgrind still working? In-Reply-To: References: Message-ID: indeed. It is probably this fix which solved the issue: https://github.com/erlang/otp/commit/2db25adbf63b6043fba3817c060cf5e32cdec343 Lukas On Wed, Feb 27, 2013 at 5:16 PM, Alessandro Sivieri < alessandro.sivieri@REDACTED> wrote: > Ok, looks like the problem is R15B, for R15B01 it works... > > > On Wed, Feb 27, 2013 at 4:50 PM, Lukas Larsson wrote: > >> Hello, >> >> We run continuous tests with the valgrind emulator, so it should be >> working. Are you building from the source tar or from a clone of github? Is >> ERL_TOP set correctly? What are the build steps (configure, make etc) you >> do leading up to doing `make valgrind`? >> >> Lukas >> >> On Wed, Feb 27, 2013 at 4:36 PM, Alessandro Sivieri < >> alessandro.sivieri@REDACTED> wrote: >> >>> Hi all, >>> >>> the question is the one in the subject: I'm trying to build R15B with >>> Valgrind enabled, but when I execute >>> >>> make valgrind FLAVOR=smp >>> >>> it returns "valgrind is up to date", which is clearly false because >>> there is no beam.valgrind anywhere... so is it still possible to execute >>> Valgrind? >>> >>> In general, what I'm trying to obtain is a profile of the heap during >>> the execution of an Erlang application, because of a comparison I have to >>> do with the same application written in other languages, so if Valgrind is >>> not working I am open to suggestions :) >>> >>> -- >>> Sivieri Alessandro >>> alessandro.sivieri@REDACTED >>> http://sivieri.wordpress.com/ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> >> > > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://sivieri.wordpress.com/ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alessandro.sivieri@REDACTED Wed Feb 27 17:31:34 2013 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Wed, 27 Feb 2013 17:31:34 +0100 Subject: [erlang-questions] Is Valgrind still working? In-Reply-To: References: Message-ID: Yes, that was the problem and the fix works! Thank you very much! On Wed, Feb 27, 2013 at 5:23 PM, Lukas Larsson wrote: > indeed. It is probably this fix which solved the issue: > https://github.com/erlang/otp/commit/2db25adbf63b6043fba3817c060cf5e32cdec343 > > Lukas > > > On Wed, Feb 27, 2013 at 5:16 PM, Alessandro Sivieri < > alessandro.sivieri@REDACTED> wrote: > >> Ok, looks like the problem is R15B, for R15B01 it works... >> >> >> On Wed, Feb 27, 2013 at 4:50 PM, Lukas Larsson wrote: >> >>> Hello, >>> >>> We run continuous tests with the valgrind emulator, so it should be >>> working. Are you building from the source tar or from a clone of github? Is >>> ERL_TOP set correctly? What are the build steps (configure, make etc) you >>> do leading up to doing `make valgrind`? >>> >>> Lukas >>> >>> On Wed, Feb 27, 2013 at 4:36 PM, Alessandro Sivieri < >>> alessandro.sivieri@REDACTED> wrote: >>> >>>> Hi all, >>>> >>>> the question is the one in the subject: I'm trying to build R15B with >>>> Valgrind enabled, but when I execute >>>> >>>> make valgrind FLAVOR=smp >>>> >>>> it returns "valgrind is up to date", which is clearly false because >>>> there is no beam.valgrind anywhere... so is it still possible to execute >>>> Valgrind? >>>> >>>> In general, what I'm trying to obtain is a profile of the heap during >>>> the execution of an Erlang application, because of a comparison I have to >>>> do with the same application written in other languages, so if Valgrind is >>>> not working I am open to suggestions :) >>>> >>>> -- >>>> Sivieri Alessandro >>>> alessandro.sivieri@REDACTED >>>> http://sivieri.wordpress.com/ >>>> >>>> _______________________________________________ >>>> erlang-questions mailing list >>>> erlang-questions@REDACTED >>>> http://erlang.org/mailman/listinfo/erlang-questions >>>> >>>> >>> >> >> >> -- >> Sivieri Alessandro >> alessandro.sivieri@REDACTED >> http://sivieri.wordpress.com/ >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -- Sivieri Alessandro alessandro.sivieri@REDACTED http://sivieri.wordpress.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bchesneau@REDACTED Thu Feb 28 15:22:33 2013 From: bchesneau@REDACTED (Benoit Chesneau) Date: Thu, 28 Feb 2013 15:22:33 +0100 Subject: [erlang-questions] diff/OT library in erlang Message-ID: Hi all, I would like to add some kind of OT [1] to an application here and I wonder if someone has already worked on such library in Erlang? Or even a difflib library like the difflib in Python [2] or an implementation of the Diff Match and Patch algorithm here [3] ? - beno?t [1] http://en.wikipedia.org/wiki/Operational_transformation [2] http://docs.python.org/3/library/difflib.html?highlight=difflib#difflib [3 https://code.google.com/p/google-diff-match-patch/ From co7eb@REDACTED Thu Feb 28 19:42:00 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Thu, 28 Feb 2013 13:42:00 -0500 Subject: [erlang-questions] Hi everyone Message-ID: <000701ce15e3$5020a800$f061f800$@frcuba.co.cu> Hi all!, I would like to introduce myself, I'm Ivan I was here two years ago posting in this mail list, I'm from Cuba I was down for so long and I would like to get active again. I don't know if some of you remember me, I was posting the subject of Pythagorean's numbers and the best way to solve it in Erlang, here is some of your solutions by then. I also was doing a framework called EVO (Extended Visual Otp) to communicate CSharp with Erlang to make client-server applications. I was down for private reasons but I never forgot about Erlang that I have in my defense. Now I would like to post again and continue my learning in Erlang. Thanks and sorry about all this, but I really missed this mailing list. I would like to send specials thanks to Joe Armstrong, Hynek, Willem, Tony, a friend from Spain who lived in Cuba, Vertientes, I don't remember his name, and all of you that make this possible. Best, your friend Ivan. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Tony's improvement of the original form 3. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag1(N) -> L = lists:seq(1,N), [ {A,B,C} || A<- L, B<- L, C<- L, A+B+C =< N, A*A+B*B =:= C*C]. %% Tony's implementation. pythag2(N) -> lists:reverse(pythan2_A(1, N, [])). pythan2_A(A, N, Acc) when A> N -> Acc; pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). pythan2_B(A, B, N, Acc) when A+B> N -> Acc; pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). pythan2_C(A, B, C, N, Acc) when A+B+C> N -> Acc; pythan2_C(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_C(A, B, C+1, N, Acc) end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Hynek's implementation. %% Simpler and about 5% faster version: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag3(N) when is_integer(N) -> pythag3(N,1). pythag3(N, A) when A+2> N -> []; pythag3(N, A) -> pythag3(N, A, 1). pythag3(N, A, B) when A+B+1> N -> pythag3(N, A+1); pythag3(N, A, B) -> pythag3(N, A, B, 1). pythag3(N, A, B, C) when A+B+C> N -> pythag3(N, A, B+1); pythag3(N, A, B, C) when A*A + B*B =:= C*C -> [{A, B, C}|pythag3(N, A, B, C+1)]; pythag3(N, A, B, C) -> pythag3(N, A, B, C+1). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Edmond's implementation using parallelism. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%---- START CODE ---- py2E(Max)-> lists:flatten(lpmap(fun(A) -> forbE(A, 1, [], Max) end, lists:seq(1, Max), ordered)). forbE(A, B, Acc, Max) -> Acc1 = forcE(A, B, 1, Acc, Max), case B< Max of true -> forbE(A, B+1, Acc1, Max); false -> Acc1 end. forcE(A, B, C, Acc, Max) -> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true -> [{A,B,C}|Acc]; _ -> Acc end, case C< Max of true-> forcE(A, B, C+1, Acc1, Max); false-> Acc1 end. pythag2E(N)-> lists:flatten(lpmap(fun(A) -> pythan2_BE(A, 1, N, []) end, lists:seq(1, N), ordered)). pythan2_AE(A, N, Acc) when A> N -> Acc; pythan2_AE(A, N, Acc) -> pythan2_AE(A+1,N,pythan2_BE(A, 1, N, Acc)). pythan2_BE(A, B, N, Acc) when A+B> N -> Acc; pythan2_BE(A, B, N, Acc) -> pythan2_BE(A,B+1,N,pythan2_CE(A, B, 1, N, Acc)). pythan2_CE(A, B, C, N, Acc) when A+B+C> N -> Acc; pythan2_CE(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_CE(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_CE(A, B, C+1, N, Acc) end. %% @spec lpmap(fun(), list(), (atom() = ordered|unordered)) -> list() %% @doc Spawns a process for each element in list L, performs specified %% function F against each in parallel and then returns results either %% same order as L (ordered) or in any order (unordered). %% NB: See also lpmap/4. lpmap(F, L, ordered) -> Ref = erlang:make_ref(), Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- L], lpmap_gather_ordered(Pids, Ref, [], 0, void); lpmap(F, L, unordered) -> Ref = erlang:make_ref(), lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, L), lpmap_gather_unordered(length(L), Ref, [], 0, void). %% @spec lpmap(fun(), integer(), list(), (atom() =ordered|unordered)) -> list() %% @doc Same as lpmap/3 except ensures only a maximum of MaxPs parallel %% processes execute function F at any one time (i.e. first takes MaxPs %% items from list, executes F in parallel against each, then as each %% process returns, spawns another process on next item in L as long as %% active processes are less than MaxPs). %% NB: See also lpmap/4. lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L)> MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- HPids], lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L)> MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, HPids), lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). %% lpmap internal functions lpmap_spawn_link(Parent, Ref, F, I) -> spawn_link(fun() -> Parent ! {self(), Ref, F(I)} end). lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> []; lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> receive {HPid, Ref, Ret} when length(TPids) [H | T] = L, [Ret | lpmap_gather_ordered( lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, H)]), Ref, T, MaxPs, F)]; {HPid, Ref, Ret} -> [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] end. lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> []; lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> receive {_Pid, Ref, Ret} when NPs-1 [H | T] = L, lpmap_spawn_link(self(), Ref, F, H), [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; {_Pid, Ref, Ret} -> [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] end. %%---- END CODE ----- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Willem's implementation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% wpythag2(N) -> L = [{A, A*A} || A<- lists:seq(1,N)], lists:flatten([forAllBs(A, A2, L, N) || {A, A2}<- L]). forAllBs(A, A2, L, N) -> [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N]. forAllCs(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Hynek's new version %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag4(N) when is_integer(N) -> pythag4(N,1). pythag4(N, A) when A+2> N -> []; pythag4(N, A) -> pythag4(N, A, A*A, 1). pythag4(N, A, _A2, B) when A+B+1> N -> pythag4(N, A+1); pythag4(N, A, A2, B) -> pythag4(N, A, A2, B, B*B, 1). pythag4(N, A, A2, B, _B2, C) when A+B+C> N -> pythag4(N, A, A2, B+1); pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; pythag4(N, A, A2, B, B2, C) -> pythag4(N, A, A2, B, B2, C+1). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Willem's implementation in parallel by Hynek %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% wpythag2P(N) -> L = [{A, A*A} || A<- lists:seq(1,N)], % For all A's lists:flatten(lpmap(fun({A, A2}) -> % For all B's in parallel [forAllCsWH(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N] end, L, 2000, ordered)). forAllCsWH(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Morten's implementation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag5(N) when is_integer(N) -> Primes = sieve(N div 2), M1M2s = incorporate_primes([{1,1}], N, Primes), lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2}<- M1M2s, M1> M2, A<- [M1-M2], B<- [2*round(math:sqrt(M1*M2))], C<- [M1+M2], A+B+C =< N])). sieve(N) when is_integer(N) -> erase(), sieve(N,2). sieve(N, K) when K>= N -> [X || X<- lists:seq(2, N), erase(X) == undefined]; sieve(N, K) -> cross_off(K, K, N div K - 1), sieve(N, find_next_in_sieve(K + 1)). cross_off(_K, _Current, 0) -> ok; cross_off(K, Current, Left) -> Next = Current + K, put(Next, out), cross_off(K, Next, Left - 1). find_next_in_sieve(K) -> case get(K) of undefined -> K; _ -> find_next_in_sieve(K+1) end. incorporate_prime(M1M2s, N, P) -> lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2}<- M1M2s]). incorporate_prime_single({M1,M2}, N, P) -> Evens = [{X, Y} || X<- incorporate_prime_even(M1, N, P), Y<- incorporate_prime_even(M2, N, P)], Odds = [{X, Y} || X<- incorporate_prime_odd(M1, N, P), Y<- incorporate_prime_odd(M2, N, P)], Evens ++ Odds. incorporate_prime_even(M, N, P) -> incorporate_prime(M, N, P, []). incorporate_prime_odd(M, N, P) -> incorporate_prime(M * P, N, P, []). incorporate_prime(M, N, _P, Acc) when M> N/2 -> Acc; incorporate_prime(M, N, P, Acc) -> incorporate_prime(M * P * P, N, P, [M|Acc]). incorporate_primes(M1M2s, _N, []) -> M1M2s; incorporate_primes(M1M2s, N, [P|Rest]) -> M1M2s_new = incorporate_prime(M1M2s, N, P), incorporate_primes(M1M2s_new, N, Rest). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Richard's improvement. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py3R(N)-> [{A,B,C} || A <- lists:seq(1, N div 2), B <- lists:seq(1, N - A), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Joe's improvement. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py3a(Max) -> N = Max div 2, [{A,B,C} || A <- lists:seq(1,N+1), B <- lists:seq(1,Max-A), C <- lists:seq(1,Max-A-B), A*A + B*B =:= C*C]. From co7eb@REDACTED Thu Feb 28 22:39:31 2013 From: co7eb@REDACTED (=?iso-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Thu, 28 Feb 2013 16:39:31 -0500 Subject: [erlang-questions] Hi everyone In-Reply-To: References: <000701ce15e3$5020a800$f061f800$@frcuba.co.cu> Message-ID: <000001ce15fc$1d098c70$571ca550$@frcuba.co.cu> Hi Garrett I think remember you, well about Erlang User Group in Havana, well for that I think the time need to pass so hard lol. When I got my degree in the UCI. University of Informatics Sciences of Cuba, I left a few knowledgers in the matter, (I don't know if that word really exits in English "knowledgers"), well some friends of mime that knew about Erlang because we studied together that technology by our own means and we did some things great for us. Because of that was born SPI Team, and I hope some day we will be more than that, just a team, a company maybe, lol. Then we will create the Havana Erlang User Group. PD: another thing I was writing a book about Erlang in spanish, I stopped but know I will continue at the correct time. And I'm going to need any possible help, if you are interesting I could serve of your knowledge. Best, Ivan. -----Mensaje original----- De: Garrett Smith [mailto:g@REDACTED] Enviado el: jueves, 28 de febrero de 2013 16:04 Para: Ivan Carmenates Garc?a Asunto: Re: [erlang-questions] Hi everyone Hi Ivan, Welcome back! I can't comment on the code, but I will say it's so very nice to hear from you! I visited Cuba many years ago (illegally, fuck you US Dept of Treasury!) when computers were banned. It struck me as a terrible tragedy -- something that would set back the Cuban people back decades in the field of computer science. It's very encouraging to see that may not be the case! So when is the Havana Erlang User Group starting up :) Garrett