From essen@REDACTED Mon Oct 1 00:22:15 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 01 Oct 2012 00:22:15 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5068B9B6.6070800@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> Message-ID: <5068C617.2050500@ninenines.eu> Biased Cowboy author here. If I got the wrong idea about another project please correct me. :) On 09/30/2012 11:29 PM, Serge Aleynikov wrote: > So why don't > all the authors of those wonderful alternative libraries just patch the > inets to extend it with something they feel missing so that we only have > one good and documented library to use as the common denominator? Because it's not about what's missing, but about how you use it, how it works and what you can do with it. You could probably classify the servers into two groups, inets+yaws being monolithic servers with many features (that can be ignored/stripped down, at least for yaws) and with a more traditional configuration file based approach (unless you run yaws embedded), and mochiweb+misultin+cowboy being in the lightweight family, being pretty much bare bone HTTP (except mochiweb has a few extra unrelated modules like JSON). Note that mochiweb is in maintenance mode and misultin has been dropped in favor of cowboy. Cowboy itself has a focus on reduced memory use (uses binary), reduced latency, has long-polling and Websocket support (SPDY soon), while trying to stay close to OTP principles (long-polling/websocket connections will be special processes in the next version allowing you to use sys: and friends to debug your code, the handler modules are inspired by gen_server, everything is supervised, and so on). It also has Webmachine-inspired REST support. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From toby@REDACTED Mon Oct 1 02:57:37 2012 From: toby@REDACTED (Toby Thain) Date: Sun, 30 Sep 2012 20:57:37 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5068B9B6.6070800@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> Message-ID: <5068EA81.6000809@telegraphics.com.au> On 30/09/12 5:29 PM, Serge Aleynikov wrote: > Haven't had a need for an HTTP server in Erlang for a while since I last > used inets-based http server five years back, and now examining what > exists in public domain. > > The results are quite surprising - everyone who seems comfortable coding > in Erlang, feels compelled to implement an HTTP server. Previously we > had only one contender to inets - yaws. And pico! http://jungerl.cvs.sourceforge.net/viewvc/jungerl/jungerl/lib/pico/ --Toby > Now the landscape is quite a > mouthful - ... > > Serge > > > (*) http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From mjtruog@REDACTED Mon Oct 1 05:12:53 2012 From: mjtruog@REDACTED (Michael Truog) Date: Sun, 30 Sep 2012 20:12:53 -0700 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5068EA81.6000809@telegraphics.com.au> References: <5068B9B6.6070800@aleynikov.org> <5068EA81.6000809@telegraphics.com.au> Message-ID: <50690A35.2080206@gmail.com> On 09/30/2012 05:57 PM, Toby Thain wrote: > On 30/09/12 5:29 PM, Serge Aleynikov wrote: >> Haven't had a need for an HTTP server in Erlang for a while since I last >> used inets-based http server five years back, and now examining what >> exists in public domain. >> >> The results are quite surprising - everyone who seems comfortable coding >> in Erlang, feels compelled to implement an HTTP server. Previously we >> had only one contender to inets - yaws. > > > And pico! > http://jungerl.cvs.sourceforge.net/viewvc/jungerl/jungerl/lib/pico/ > > And elli! https://github.com/knutin/elli From max.lapshin@REDACTED Mon Oct 1 08:48:42 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Oct 2012 10:48:42 +0400 Subject: [erlang-questions] SublimErl: how to lower amount of code completion? In-Reply-To: References: Message-ID: Is it possible to add completion_skip_erlang_libs to my global config file? In what section? On Tue, Sep 25, 2012 at 7:52 AM, Roberto Ostinelli wrote: > On Mon, Sep 24, 2012 at 6:47 AM, Max Lapshin wrote: >> >> Code completion in SublimErl have some problems: >> it almost doesn't complete with local variable names, but offers me a >> lot of code from corba. >> >> >> Is it possible to setup it so, that local names are first and disable >> some folders for completion strings. >> I hope I will never ever do anything with corba again, so I don't need >> this completion =) > > > https://github.com/ostinelli/SublimErl/commit/b388f39b54231a712675b2c116949f1e9627bde3 > > as per the 'problems', please open an issue for any you might encounter, > specifying exactly what you are referring to, how to reproduce them, and the > intended behavior. may i suggest this list might not be the most appropriate > for these questions ^^_ > > cheers, > > r. From tmr@REDACTED Mon Oct 1 09:20:26 2012 From: tmr@REDACTED (Tomas Morstein) Date: Mon, 01 Oct 2012 09:20:26 +0200 (CEST) Subject: [erlang-questions] eprof bug? In-Reply-To: <1d92a43a-8b85-48c1-a22b-ab2bd65c81cf@zimbra> Message-ID: Hello, I wanted to play with eprof a little, and found that it does not work correctly on my system and anytime when I try to start profiling, it ends up with SIGSEGV: https://gist.github.com/3805252 We tested it on multiple systems with the same version of OTP (R15B) built from source but the problem appears only on my laptop with Linux 2.6.38/x86_64. The configure parameters is what I don't remember at the moment :( Is there anybody aware of such an issue? Best regards, Tom. PS: In meantime, I'll try to compile the new R15B02 (with debugging symbols) and check if it does there too... From watson.timothy@REDACTED Mon Oct 1 09:34:50 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Mon, 1 Oct 2012 08:34:50 +0100 Subject: [erlang-questions] rebar question In-Reply-To: <5066FF9E.5090000@aleynikov.org> References: <5066197B.5020008@aleynikov.org> <76CE4188-BC5E-4DA4-8016-39CD7CDC326D@gmail.com> <5066FF9E.5090000@aleynikov.org> Message-ID: <1C29BBA1-35C6-4E27-B47C-6C5D7807DE43@gmail.com> You can do this by including a rebar.config.script in the project, which is handled with file:script/1 to produce the config. This can read the regular rebar.config (either using file:consult/1 or rebar_config) and modify whatever it likes. An alternative is to write a plugin which hooks into the post_compile: -module(example). post_compile(Conf, _) -> %% move the binar wherever you want ok. On 29 Sep 2012, at 15:03, Serge Aleynikov wrote: > In order to compile a port program using rebar, I need to place the > resulting port program in the architecture-specific folder: > priv/$ARCH/exec-port > > where $ARCH is the result of the call: > erlang:system_info(system_architecture) > > I was thinking of setting ARCH in the pre-compile hook. However, it > doesn't seem that the "port_specs" config clause allows expansion of > environment variables, so if the "ARCH" variable has a value, the > following won't work: > > {port_specs, [{"priv/$ARCH/exec-port", ["c_src/*.cpp"]}]}. > > Is there a work-around I could use to accomplish this? > > On 9/29/2012 5:23 AM, Tim Watson wrote: >> I'm a heavy rebar user and have compiled ports with it before, so give me a shout if you need any assistance. I can also do testing on several unix platforms. >> >> On 28 Sep 2012, at 22:41, Serge Aleynikov wrote: >> >>> Unfortunately I don't have OSx to test this. I can't reproduce your >>> problem on Linux. Perhaps you could check the libtool version this way: >>> >>> $ grep "^macro_version" /usr/bin/libtool >>> macro_version=2.4 >>> >>> and if yours is different, try to upgrade it. >>> >>> In order to simplify the build process I'll try to convert erlexec to >>> use rebar some time next week, which will eliminate autotools-based >>> toolchain. Need to figure out with rebar how to test and optionally >>> enable conditional defines based on presence of special C headers such >>> as "sys/capability.h" when compiling a port program. >>> >>> On 9/28/2012 5:16 PM, Tim Watson wrote:> Well I do, but clearly not the >>> same version as you? >>>> >>>> t4@REDACTED:erlexec $ which libtool >>>> /usr/bin/libtool >>>> t4@REDACTED:erlexec $ ls -la /usr/bin/ | grep libtool >>>> -r-xr-xr-x 1 root wheel 149216 22 May 15:30 libtool >>>> lrwxr-xr-x 1 root wheel 7 22 May 15:30 ranlib -> libtool >>>> t4@REDACTED:erlexec $ libtool --version >>>> libtool: unknown option character `-' in: --version >>>> Usage: libtool -static [-] file [...] [-filelist listfile[,dirname]] >>> [-arch_only arch] [-sacLT] >>>> Usage: libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] >>> [-arch_only arch] [-o output] [-install_name name] >>> [-compatibility_version #] [-current_version #] [-seg1addr 0x#] >>> [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table >>> ] [-seg_addr_table_filename ] [-all_load] >>> [-noall_load] >>>> t4@REDACTED:erlexec $ >>>> >>>> Honestly the osx build tool chain drives me nuts sometimes. :/ >>>> >>>> On 28 Sep 2012, at 17:24, Serge Aleynikov wrote: >>>> >>>>> Is it possible that you don't have libtool installed (missing >>>>> AC_PROG_LIBTOOL)? >>>>> >>>>> I have the following on linux and everything builds fine: >>>>> $ autoconf -V >>>>> autoconf (GNU Autoconf) 2.68 >>>>> $ libtool --version >>>>> libtool (GNU libtool) 2.4 >>>>> $ ./build >>>>> $ make >>>>> $ make install >>>>> $ tree install >>>>> install >>>>> ??? exec-1.0 >>>>> ??? ebin >>>>> ? ??? exec.app >>>>> ? ??? exec_app.beam >>>>> ? ??? exec.beam >>>>> ??? include >>>>> ? ??? exec.hrl >>>>> ??? priv >>>>> ? ??? x86_64-unknown-linux-gnu >>>>> ? ??? bin >>>>> ? ??? exec-port >>>>> ??? src >>>>> ??? exec_app.erl >>>>> ??? exec.erl >>>>> >>>>> >>>>> On 9/28/2012 7:45 AM, Tim Watson wrote:> The project doesn't appear to >>>>> build cleanly on mac os lion: >>>>>> >>>>>> t4@REDACTED:erlexec $ ./configure >>>>>> configure: error: cannot find install-sh, install.sh, or shtool in >>>>> config "."/config >>>>>> t4@REDACTED:erlexec $ ./build >>>>>> Install dir: /usr/local/src/erlang/erlexec/install >>>>>> aclocal >>>>>> autoconf -i >>>>>> configure.ac:19: error: possibly undefined macro: AC_PROG_LIBTOOL >>>>>> If this token and others are legitimate, please use >>> m4_pattern_allow. >>>>>> See the Autoconf documentation. >>>>>> make: *** [configure] Error 1 >>>>>> t4@REDACTED:erlexec $ >>>>>> t4@REDACTED:erlexec $ autoconf -V >>>>>> autoconf (GNU Autoconf) 2.69 >>>>>> Copyright (C) 2012 Free Software Foundation, Inc. >>>>>> License GPLv3+/Autoconf: GNU GPL version 3 or later >>>>>> , >>>>> >>>>>> This is free software: you are free to change and redistribute it. >>>>>> There is NO WARRANTY, to the extent permitted by law. >>>>>> >>>>>> Written by David J. MacKenzie and Akim Demaille. >>>>>> t4@REDACTED:erlexec $ >>>>>> >>>>>> >>>>>> Is there a specific version of autotools required to build and compile >>>>> erlexec? >>>>>> >>>>>> On 27 Sep 2012, at 20:11, Tim Watson wrote: >>>>>> >>>>>>> On 27 Sep 2012, at 19:18, freza@REDACTED wrote: >>>>>>>> Haven't been following this thread closely, but it seems erlexec >>> wasn't >>>>>>>> mentioned yet: >>>>>>>> >>>>>>>> http://code.google.com/p/erlexec/ >>>>>>> >>>>>>> Doesn't appear very well maintained. I might pick it apart and >>>>> rewrite it when native processes get released - it is a cool idea. >>>>> Anyway, for the time being it's not a great help as I'm writing a >>>>> framework to help set up and tear down clusters for distributed testing >>>>> (based largely on common test) so I want to minimise the number of >>>>> dependencies as much as possible. Besides, I'm more interested in >>>>> working on the remote/ssh runner+monitor next, rather than spending >>>>> another age fiddling around with the script runner. >>>>>>> >>>>>>> But yes, erlexec does look interesting and has been on my radar since >>>>> it popped up on this list a few months back, discussing the desire to do >>>>> proper signal handling (such as sending SIGTERM, SIGHUP, etc to the >>>>> external program) without resorting to os:cmd("kill -" ++ Code ++ " " ++ >>>>> OsPid). Again, this is something that linked-in drivers *can* deal with >>>>> to a limited extent, and therefore I suspect that native processes will >>>>> make for a very nice foundation on which to build an alternative to >>>>> open_port/2. >>>>>>> >>>>>>> Also, IIRC there was a cool command line testing framework open >>>>> sourced in the last year which was announced on this list. I can't >>>>> remember what it was called, but it has a port driver that handles the >>>>> the external comms and so on - looked very nice, although that's not >>>>> what I'm trying to do/be as my focus is on getting things in a >>>>> consistent state and monitoring to make sure they stay that way through >>>>> the test run, then tearing down nicely before the next phase kicks off. >>>>>>> >>>>>>> Cheers, >>>>>>> Tim >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>> From erlang@REDACTED Mon Oct 1 10:01:31 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 1 Oct 2012 10:01:31 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5068B9B6.6070800@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> Message-ID: On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov wrote: > Haven't had a need for an HTTP server in Erlang for a while since I last > used inets-based http server five years back, and now examining what > exists in public domain. > > The results are quite surprising - everyone who seems comfortable coding > in Erlang, feels compelled to implement an HTTP server. Previously we > had only one contender to inets - yaws. Now the landscape is quite a > mouthful - inets, yaws, cowboy, musultin, mochiweb, etc. As Steve > Vinoski rightfully points out (*) this doesn't help to pull the blanket > in all directions. From personal experience with inets, I recall that > once I figured out how to use it, it happened to be a very helpful and > powerful HTTP library for building an application server. So why don't > all the authors of those wonderful alternative libraries just patch the > inets to extend it with something they feel missing so that we only have > one good and documented library to use as the common denominator? I can think of several reasons why this does not happen: a) You have to distinguish between functional and non-functional behavior. All web-servers implement the same protocol (HTTP) - so they are functionally equivalent - but their non-functional behavior is very different - one server might be optimized for large numbers of short-lived sessions, another for a few long-lived sessions. One might cache content, another not. So even though they are all http servers they have very different behaviour - this is good. b) Servers have two interfaces - the protocol interface (HTTP) and the interface towards Erlang (the API if you like). As far as I can see most severs have different Erlang APIs. This is bad. c) Inets is shipped with OTP ie built and maintained by the OTP group. This is not really "core business" as far as the OTP group is concerned, better for everybody if non-core applications get taken over by third-part vendors. d) As things mature - people want more in the way of "service" - ie good documentation and the ability to buy support - the problem with software is not so much that of writing it - which can be easy or difficult depending upon the problem but with the "burden of support" Personally I'm quite happy with there being several different http servers, It gives me several to choose from. No server in the world will do exactly what I want so they all need tweaking in one way or another. What I don't like is that all have different APIs - this make swapping between severs a pain. Some degree of standardization of the APIs necessary to setup and manage the servers would be highly desirable. It would also be highly desirable to standardize the relationship between a http uri and and erlang function call. For example http://some.host/erlcall?mod=foo&func=bar&arg1=111 means call foo:bar(...) on the server and return the value as html (or something) Currently the http-server side of things seems ok - what I'm missing is the entire user management cycle - password management, authentication, cookies, forgotten passwords etc. - this needs far more than an http server, it needs a persistent database, security and so on ... Cheers /Joe > Serge > > > (*) http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From cookchrisd@REDACTED Mon Oct 1 10:11:45 2012 From: cookchrisd@REDACTED (Chris Cook) Date: Mon, 1 Oct 2012 09:11:45 +0100 Subject: [erlang-questions] Bad Argument, (Bug or something I'm missing). Message-ID: Morning list, I have, Erlang R15B01 (erts-5.9.1) [source] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9.1 (abort with ^G) 1> List = "89234789234jhk2hk234892789fauky324978". "89234789234jhk2hk234892789fauky324978" 2> <>/binary>>. ** exception error: bad argument But when I write it as this, 3> <<"89234789234jhk2hk234892789fauky324978", <<"-">>/binary>>. <<"89234789234jhk2hk234892789fauky324978-">> I get the result I expected to get from the above 1 & 2. Could someone please explain what is going wrong and why, because I'm very confused with it. Regards Chris Cook. From egil@REDACTED Mon Oct 1 10:36:58 2012 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Mon, 1 Oct 2012 10:36:58 +0200 Subject: [erlang-questions] eprof bug? In-Reply-To: References: Message-ID: <5069562A.1070503@erlang.org> On 2012-10-01 09:20, Tomas Morstein wrote: > Hello, > > I wanted to play with eprof a little, and found that it does not work > correctly on my system and anytime when I try to start profiling, > it ends up with SIGSEGV: > https://gist.github.com/3805252 > > We tested it on multiple systems with the same version of OTP (R15B) > built from source but the problem appears only on my laptop with > Linux 2.6.38/x86_64. The configure parameters is what I don't remember > at the moment :( > > Is there anybody aware of such an issue? If memory serves, there was an issue with eprof (tracing) and HiPE. Is there any chance you are running HiPE compiled libs, i.e. HiPE compiled lists module? If you find the same issue in R15B02, let me know. // Bj?rn-Egil > > Best regards, > Tom. > > > PS: In meantime, I'll try to compile the new R15B02 (with debugging > symbols) and check if it does there too... > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From lukas@REDACTED Mon Oct 1 10:43:53 2012 From: lukas@REDACTED (Lukas Larsson) Date: Mon, 1 Oct 2012 10:43:53 +0200 Subject: [erlang-questions] Bad Argument, (Bug or something I'm missing). In-Reply-To: References: Message-ID: Hi, If you just do 2> L = "a",<>. ** exception error: bad argument You get the same behaviour. When you do <<"a">> it is just syntactic sugar for << <<97>>/binary >>. However when passing the value as a variable you do not get the syntactic sugar help so you have to do this: 3> L = "a",<<(list_to_binary(L))/binary>>. <<"a">> or something to that effect to get the correct result. Hope that makes sense. Lukas On Mon, Oct 1, 2012 at 10:11 AM, Chris Cook wrote: > Morning list, > > I have, > > Erlang R15B01 (erts-5.9.1) [source] [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> List = "89234789234jhk2hk234892789fauky324978". > "89234789234jhk2hk234892789fauky324978" > 2> <>/binary>>. > ** exception error: bad argument > > But when I write it as this, > > 3> <<"89234789234jhk2hk234892789fauky324978", <<"-">>/binary>>. > <<"89234789234jhk2hk234892789fauky324978-">> > > I get the result I expected to get from the above 1 & 2. Could someone > please explain what is going wrong and why, because I'm very confused > with it. > > Regards > > Chris Cook. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From gumm@REDACTED Mon Oct 1 10:46:51 2012 From: gumm@REDACTED (Jesse Gumm) Date: Mon, 1 Oct 2012 03:46:51 -0500 Subject: [erlang-questions] Bad Argument, (Bug or something I'm missing). In-Reply-To: References: Message-ID: Hi, You just need to convert List to a binary first with list_to_binary. The <<"something">> syntax is just a friendly way to make binary literals. But it does not perform automatic casting of lists. Further, the extra << >> you're wrapping around "-" is unnecessary. Example: > L = "123456", > B = list_to_binary(L), > <>. Will yield: <<"123456-">> Hope that helps. Take it easy, -Jesse -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm On Oct 1, 2012 3:11 AM, "Chris Cook" wrote: > Morning list, > > I have, > > Erlang R15B01 (erts-5.9.1) [source] [async-threads:0] [hipe] > [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> List = "89234789234jhk2hk234892789fauky324978". > "89234789234jhk2hk234892789fauky324978" > 2> <>/binary>>. > ** exception error: bad argument > > But when I write it as this, > > 3> <<"89234789234jhk2hk234892789fauky324978", <<"-">>/binary>>. > <<"89234789234jhk2hk234892789fauky324978-">> > > I get the result I expected to get from the above 1 & 2. Could someone > please explain what is going wrong and why, because I'm very confused > with it. > > Regards > > Chris Cook. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvg.mailing@REDACTED Mon Oct 1 12:40:19 2012 From: rvg.mailing@REDACTED (Rudolph van Graan) Date: Mon, 01 Oct 2012 10:40:19 +0000 (GMT) Subject: [erlang-questions] Probably a dumb eunit question Message-ID: <3da54eea-3f22-e180-ad72-ea083afb8f5c@me.com> I have one test case that fails because something it calls crashes. I see this output after running the test: ======================== EUnit ======================== *failed* ::undef ======================================================= ? Failed: 1. ?Skipped: 0. ?Passed: 0. I know WHY it fails. The test consists of this single line of code: ??assertMatch(ok,my_app:install()), If I change the assertMatch to assertError, I get the real reason for the crash: *failed* ::{assertException_failed, ? ? [{module,cache_tests}, ? ? ?{line,45}, ? ? ?{expression,"app : install ( )"}, ? ? ?{pattern,"{ error , ok , [...] }"}, ? ? ?{unexpected_exception, ? ? ? ? ?{error,undef, ? ? ? ? ? ? ?[{util,get_value,[frag_properties,[{...}|...]],[]}, ? ? ? ? ? ? ? {app,'-install/2-fun-0-',4,[{...}|...]}, ? ? ? ? ? ? ? {lists,foldl,3,[...]}, ? ? ? ? ? ? ? {app,install,2,...}, ? ? ? ? ? ? ? {cache_tests,'-start_stop/0-fun-0-',...}, ? ? ? ? ? ? ? {cache_tests,...}, ? ? ? ? ? ? ? {...}|...]}}]} My question is... what am I doing wrong so that I don't get the crash stack and details when I just use assertMatch(...) ? Thanks, R -------------- next part -------------- An HTML attachment was scrubbed... URL: From xapwing@REDACTED Mon Oct 1 14:26:48 2012 From: xapwing@REDACTED (Arie van Wingerden) Date: Mon, 1 Oct 2012 14:26:48 +0200 Subject: [erlang-questions] Offtopic: great kickstarter project Message-ID: Hi, maybe this might interest (some of) you. Maybe it can be used in a similar way as Gumstix with Erlang. http://www.kickstarter.com/projects/adapteva/parallella-a-supercomputer-for-everyone?ref=live Kind regards, Arie -------------- next part -------------- An HTML attachment was scrubbed... URL: From magnus@REDACTED Mon Oct 1 14:27:01 2012 From: magnus@REDACTED (Magnus Henoch) Date: Mon, 01 Oct 2012 13:27:01 +0100 Subject: [erlang-questions] Probably a dumb eunit question In-Reply-To: <3da54eea-3f22-e180-ad72-ea083afb8f5c@me.com> (Rudolph van Graan's message of "Mon, 01 Oct 2012 10:40:19 +0000 (GMT)") References: <3da54eea-3f22-e180-ad72-ea083afb8f5c@me.com> Message-ID: Rudolph van Graan writes: > I have one test case that fails because something it calls crashes. I > see this output after running the test: > > ======================== EUnit ======================== > *failed* > ::undef > ======================================================= > Failed: 1. Skipped: 0. Passed: 0. [...] > My question is... what am I doing wrong so that I don't get the crash > stack and details when I just use assertMatch(...) ? Are you using R15B or R15B01? The lack of stack traces in eunit is a known problem, fixed in R15B02. Regards, Magnus From mononcqc@REDACTED Mon Oct 1 14:57:25 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 01 Oct 2012 08:57:25 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> Message-ID: <50699335.2020501@ferd.ca> I did not have much to say about the rest (except that different interfaces may emerge based on the use cases that the server optimizes for), and I mostly nodded in agreement to most of your points, but the last one I believe is a very bad idea. This is a problem in terms of: 1. *Maintenance*. You don't want to usually give such an obvious window into your server as it somewhat crystallizes your internal application as a public API, what the URI is usually used for. We should strive to have URIs never change (see W3C - Cool URIs don't change ). Unless you're ready not to change the function calls under the hood, I would not use this. 2. *Security*. How do you keep someone from calling http://some.host/erlcall?mod=init&func=stop or http://some.host/erlcall?mod=os&func=cmd&arg1="rm -rf /" ? If you go look at tryerlang.org's repository, you'll see how restrictive Roberto Aloi had to make it to avoid people doing tricky things like passing binaries of funs calling things they shouldn't indirectly just to crash the system. This is absolutely non-trivial as soon as you move out from using a very restrictive white list of valid functions -- at which point you may as well hide them behind URIs. 3. *Clarity*. What are the argument types? Should we treat the '111' as a string? a binary? an integer? An IO List? Should the server force all arguments to be of a given type that needs to be converted later so you need to have some kind of intermediary function doing conversion for you? It's not different from what we get right now, but there's a difference in expectations from the developer. A minor issue, I guess. 4. *RPC and HTTP*. Whether RPC is still a good way to do things is up to debate (look for Steve Vinoski's essays and posts on the topic). Shouldn't the erlcall URI be extended to also handle timeouts? Or are these server options unrelated to the function call? More than that is the idea of how you write things that may depend on content-type (such as RESTful web services). For example, a browser that sends a request to a page in HTML may be served HTML. If Javascript asks for info in JSON, then the same URI may forward JSON. There is an out-of-URL mechanism to deal with different requests and ideas. 5. *HTTP Spec problems*. GET and HEAD requests should be idempotent. PUT, POST, and DELETE can be used in ways that change information on the server. If I use POST to http://some.host/erlcall?mod=foo&func=bar&arg1=111 and it updates data, what should it do when I use GET or HEAD on it? Do we prefer to use the POST body for the requests that are done under the POST method? What if I include both POST arguments and querystring arguments and they are different? More than that, HTTP doesn't specify what happens when you have more than one querystring argument being the same (from memory). It should thus be possible to have arg1 four times in the URL, with arg3 being there only once, and arg2 being entirely missing. Same with the module or function name. How do we make this interface behave at that point? Things are way more complex than what they look like. There are likely more issues with this approach. Ideally, it should be impossible for the user of your server to know what you used to implement it -- this is what should give you the most freedom in terms of implementation when lots of people start using it. At this point I'm thinking we should start using different protocols from HTTP if we're not really willing to respect it, but that's for another discussion. And yes, more user management would definitely be nice. I've written a tiny library to handle passwords themselves, but it's not close enough to be something a framework would use without surrounding support -- it just handles the core of password hashing and verification. It's currently hosted at https://github.com/ferd/erlpass . Then each platform such as Zotonic likely reimplemented their own, although I don't know how moveable they are outside of the project. Regards, Fred. On 12-10-01 4:01 AM, Joe Armstrong wrote: > On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov wrote: > > > It would also be highly desirable to standardize the relationship > between a http uri and > and erlang function call. For example > > > http://some.host/erlcall?mod=foo&func=bar&arg1=111 > > means call foo:bar(...) on the server and return the value as html (or > something) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dale@REDACTED Mon Oct 1 15:32:41 2012 From: dale@REDACTED (Dale Harvey) Date: Mon, 1 Oct 2012 14:32:41 +0100 Subject: [erlang-questions] Erlang http servers In-Reply-To: <50699335.2020501@ferd.ca> References: <5068B9B6.6070800@aleynikov.org> <50699335.2020501@ferd.ca> Message-ID: "The standard library is where modules go to die" http://www.leancrew.com/all-this/2012/04/where-modules-go-to-die/ This is from python world, and is very obviously seen in Erlang. Alternatively https://npmjs.org/ has 15k+ packages, we dont need to add user management and the kitchen sink to inets, we need to remove it from the standard library and promote userland packages as first class citizens (ie pick a tool to install packages and bundle it) On 1 October 2012 13:57, Fred Hebert wrote: > I did not have much to say about the rest (except that different > interfaces may emerge based on the use cases that the server optimizes > for), and I mostly nodded in agreement to most of your points, but the last > one I believe is a very bad idea. > > This is a problem in terms of: > > 1. *Maintenance*. You don't want to usually give such an obvious > window into your server as it somewhat crystallizes your internal > application as a public API, what the URI is usually used for. We should > strive to have URIs never change (see W3C - Cool URIs don't change). Unless you're ready not to change the function calls under the hood, I > would not use this. > 2. *Security*. How do you keep someone from calling > http://some.host/erlcall?mod=init&func=stop or > http://some.host/erlcall?mod=os&func=cmd&arg1="rm -rf /" ? If you go > look at tryerlang.org's repository, you'll see how restrictive Roberto > Aloi had to make it to avoid people doing tricky things like passing > binaries of funs calling things they shouldn't indirectly just to crash the > system. This is absolutely non-trivial as soon as you move out from using a > very restrictive white list of valid functions -- at which point you may as > well hide them behind URIs. > 3. *Clarity*. What are the argument types? Should we treat the '111' > as a string? a binary? an integer? An IO List? Should the server force all > arguments to be of a given type that needs to be converted later so you > need to have some kind of intermediary function doing conversion for you? > It's not different from what we get right now, but there's a difference in > expectations from the developer. A minor issue, I guess. > 4. *RPC and HTTP*. Whether RPC is still a good way to do things is up > to debate (look for Steve Vinoski's essays and posts on the topic). > Shouldn't the erlcall URI be extended to also handle timeouts? Or are these > server options unrelated to the function call? More than that is the idea > of how you write things that may depend on content-type (such as RESTful > web services). For example, a browser that sends a request to a page in > HTML may be served HTML. If Javascript asks for info in JSON, then the same > URI may forward JSON. There is an out-of-URL mechanism to deal with > different requests and ideas. > 5. *HTTP Spec problems*. GET and HEAD requests should be idempotent. > PUT, POST, and DELETE can be used in ways that change information on the > server. If I use POST to > http://some.host/erlcall?mod=foo&func=bar&arg1=111 and it updates > data, what should it do when I use GET or HEAD on it? Do we prefer to use > the POST body for the requests that are done under the POST method? What if > I include both POST arguments and querystring arguments and they are > different? > More than that, HTTP doesn't specify what happens when you have more > than one querystring argument being the same (from memory). It should thus > be possible to have arg1 four times in the URL, with arg3 being there > only once, and arg2 being entirely missing. Same with the module or > function name. How do we make this interface behave at that point? Things > are way more complex than what they look like. > > There are likely more issues with this approach. Ideally, it should be > impossible for the user of your server to know what you used to implement > it -- this is what should give you the most freedom in terms of > implementation when lots of people start using it. At this point I'm > thinking we should start using different protocols from HTTP if we're not > really willing to respect it, but that's for another discussion. > > And yes, more user management would definitely be nice. I've written a > tiny library to handle passwords themselves, but it's not close enough to > be something a framework would use without surrounding support -- it just > handles the core of password hashing and verification. It's currently > hosted at https://github.com/ferd/erlpass . Then each platform such as > Zotonic likely reimplemented their own, although I don't know how moveable > they are outside of the project. > Regards, > Fred. > > > On 12-10-01 4:01 AM, Joe Armstrong wrote: > > On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov > wrote: > > > It would also be highly desirable to standardize the relationship > between a http uri and > and erlang function call. For example > > > http://some.host/erlcall?mod=foo&func=bar&arg1=111 > > > means call foo:bar(...) on the server and return the value as html (or > something) > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.lundin@REDACTED Mon Oct 1 16:30:25 2012 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Mon, 1 Oct 2012 16:30:25 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> Message-ID: I don't agree with everything Joe writes below: On 10/1/12, Joe Armstrong wrote: > On Sun, Sep 30, 2012 at 11:29 PM, Serge Aleynikov > wrote: >> Haven't had a need for an HTTP server in Erlang for a while since I last >> used inets-based http server five years back, and now examining what >> exists in public domain. >> >> The results are quite surprising - everyone who seems comfortable coding >> in Erlang, feels compelled to implement an HTTP server. Previously we >> had only one contender to inets - yaws. Now the landscape is quite a >> mouthful - inets, yaws, cowboy, musultin, mochiweb, etc. As Steve >> Vinoski rightfully points out (*) this doesn't help to pull the blanket >> in all directions. From personal experience with inets, I recall that >> once I figured out how to use it, it happened to be a very helpful and >> powerful HTTP library for building an application server. So why don't >> all the authors of those wonderful alternative libraries just patch the >> inets to extend it with something they feel missing so that we only have >> one good and documented library to use as the common denominator? > I agree with Serge that it is somewhat unfortunate that there are so many initiatives where people write their own implementation of something we already have included in the Erlang/OTP distribution and doing this without even suggesting or asking us if we could incorporate some missing feature or improve the current implementation in various ways. I think it is better to try to improve the existing functionality first and only if that is not feasible make another implementation with a new API. In Erlang/OTP we have keep strict compatibility with many of our APIs which is both a strength and sometimes a burden (when we are not happy with the old API) but there are mostly ways around that if there is strong case for a change. > I can think of several reasons why this does not happen: > > a) You have to distinguish between functional and non-functional behavior. > All web-servers implement the same protocol (HTTP) - so they are > functionally > equivalent - but their non-functional behavior is very different > - one server > might be optimized for large numbers of short-lived sessions, > another for a few long-lived > sessions. One might cache content, another not. So even though they > are > all http servers they have very different behaviour - this is good. > > b) Servers have two interfaces - the protocol interface (HTTP) and > the interface towards > Erlang (the API if you like). As far as I can see most severs > have different Erlang APIs. > This is bad. Would be good if there was a kind of standardized API for HTTP-servers and clients. > > c) Inets is shipped with OTP ie built and maintained by the OTP group. > This is not really > "core business" as far as the OTP group is concerned, better for > everybody if non-core > applications get taken over by third-part vendors. Joe is right reagarding core business in that there are not much use of HTTP-servers in products developed with Erlang inside Ericsson (who is financing ERlang/OTP) and if used the requirements are not very demanding. > > d) As things mature - people want more in the way of "service" - ie > good documentation and the ability > to buy support - the problem with software is not so much that of > writing it - which can be > easy or difficult depending upon the problem but with the "burden of > support" We still actively develop inets with its http client and server and I am not sure that all the reasons to develop alternatives still exist. We are more than happy to receive contributions with bugfixes and new functionality. I think it is of great value to have a relative complete functionality in Erlang/OTP so that the user can implement basic functionality without having to search for separate implementations of, with unknown quality and completeness at least for a beginner. > > Personally I'm quite happy with there being several different http > servers, It gives me several to choose > from. No server in the world will do exactly what I want so they all > need tweaking in one way or another. > > What I don't like is that all have different APIs - this make swapping > between severs a pain. > Some degree of standardization of the APIs necessary to setup and > manage the servers would be > highly desirable. > > It would also be highly desirable to standardize the relationship > between a http uri and > and erlang function call. For example > > > http://some.host/erlcall?mod=foo&func=bar&arg1=111 > > means call foo:bar(...) on the server and return the value as html (or > something) > > Currently the http-server side of things seems ok - what I'm missing is > the > entire user management cycle - password management, authentication, > cookies, > forgotten passwords etc. - this needs far more than an http server, it > needs > a persistent database, security and so on ... > > Cheers > > /Joe > > /Kenneth, Erlang/OTP Ericsson > >> Serge >> >> >> (*) >> http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From roberto@REDACTED Mon Oct 1 16:33:20 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 1 Oct 2012 07:33:20 -0700 Subject: [erlang-questions] SublimErl: how to lower amount of code completion? In-Reply-To: References: Message-ID: Hi Max, not sure what you mean with global config file, if you mean Preferences.sublime-settings that doesn't have sections. Only support is to put it into the SublimErl settings file: https://github.com/ostinelli/SublimErl/blob/master/SublimErl.sublime-settings Regards, r. On Sun, Sep 30, 2012 at 11:48 PM, Max Lapshin wrote: > Is it possible to add completion_skip_erlang_libs to my global config file? > > In what section? > > On Tue, Sep 25, 2012 at 7:52 AM, Roberto Ostinelli > wrote: > > On Mon, Sep 24, 2012 at 6:47 AM, Max Lapshin > wrote: > >> > >> Code completion in SublimErl have some problems: > >> it almost doesn't complete with local variable names, but offers me a > >> lot of code from corba. > >> > >> > >> Is it possible to setup it so, that local names are first and disable > >> some folders for completion strings. > >> I hope I will never ever do anything with corba again, so I don't need > >> this completion =) > > > > > > > https://github.com/ostinelli/SublimErl/commit/b388f39b54231a712675b2c116949f1e9627bde3 > > > > as per the 'problems', please open an issue for any you might encounter, > > specifying exactly what you are referring to, how to reproduce them, and > the > > intended behavior. may i suggest this list might not be the most > appropriate > > for these questions ^^_ > > > > cheers, > > > > r. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From morlac78@REDACTED Mon Oct 1 17:30:44 2012 From: morlac78@REDACTED (Christian Adams) Date: Mon, 1 Oct 2012 17:30:44 +0200 Subject: [erlang-questions] [ANN] erl_i2c v0.1 released Message-ID: Hi all, just a short mail to announce release of erl_i2c v0.1. A small gen_server and c-node to interface hardware (sensors, ..) connected to an i2c-bus. For now just the main-capabilities: - write bytes to a device, register - read bytes from a device, register For further information, src and a rudimentary doc is available at github.com/morlac/erl_i2c Feel free to comment, fork, and contribute :) Enjoy, Christian Adams From essen@REDACTED Mon Oct 1 17:38:52 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 01 Oct 2012 17:38:52 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> Message-ID: <5069B90C.7070504@ninenines.eu> On 10/01/2012 04:30 PM, Kenneth Lundin wrote: > I agree with Serge that it is somewhat unfortunate that there are so > many initiatives > where people write their own implementation of something we already > have included in > the Erlang/OTP distribution and doing this without even suggesting or > asking us if we could > incorporate some missing feature or improve the current implementation > in various ways. It happens a lot. People tend to review available options, and if none fits their needs, write something themselves without consulting any of the authors. Sometimes they're right to do so, because the answer they'll get is obvious, and sometimes not. There's a few reasons this happens for this specific case: * HTTP is simple, it's easier to go ahead and experiment on a new project than on an existing one. * If the experiment succeeds and the implementation has to differ significantly from other projects, it's not worth backporting since you'll just run into a wall of backward compatibility requirements. * OTP has a long release process. Many companies need their fixes to be included quickly to avoid maintaining huge branches for their changes. They generally agree in return to keep track of changes that might happen during the product's development. * Inets is seriously well hidden. I only heard about it after releasing Cowboy. I knew about Yaws/Mochiweb/Misultin but not the inets HTTP server. Many people I talked to didn't seem to know it existed. * Sometimes removing features is an improvement. Inets is big, it's only natural that lightweight alternatives appear. Same happened to Apache, and it's no surprise that Inets is quite similar to Apache from a user point of view. Note that while not everyone uses inets, most servers do use erlang:decode_packet/3 which does most of the job. Cowboy doesn't anymore, because it needed backward incompatible changes to erlang:decode_packet/3. I didn't ask your thoughts, simply because the experiment I did hinted that doing the parsing in full Erlang was a better idea: this change results in improved HiPE optimizations and better reduction management which results in better latency under load. > In Erlang/OTP we have keep strict compatibility with many of our APIs > which is both a strength and sometimes a burden (when we are not happy > with the old API) but there are mostly ways around that if there is > strong case for a change. It's a big strength for people looking for stability and a big burden for people looking for reactivity. Can't really blame the second group for finding solutions to their problems. > We still actively develop inets with its http client and server and I > am not sure that > all the reasons to develop alternatives still exist. But like you said the level of support for applications included in OTP vary widely. It is also not clearly indicated which parts are the focus of recent works (there is a small roadmap for the most important stuff but that's about it), nor can it be deduced from reading tickets in a bug tracker or similar collaboration tool. Which brings me to a very important point: please start using a public bug tracker! It doesn't have to be 100% public (as I'm sure you have some issues that are internal only) but it would help a lot if people could see the existing issues instead of stumbling upon the same bugs again and again. It probably also would help with contributions. > I think it is of great value to have a relative complete functionality > in Erlang/OTP so that > the user can implement basic functionality without having to search > for separate implementations of, with unknown quality and completeness > at least for a beginner. I would prefer Erlang/OTP to focus on the core features and leave the satellite features to third parties. OTP has an HTTP server but doesn't have proper Unicode support. OTP has OpenGL but doesn't have proper timezones support. These 2 examples missing from OTP are some of many things that are too complex for most people to implement, and that would not benefit from having alternatives. Let's take another example. While taking a quick glance at the modules included in OTP I discovered there is a GD implementation in the percept application. I'm pretty sure most people didn't know it was there, as they usually end up using a NIF. It's quite unexpected that it's in OTP. Finding things in OTP unexpectedly is a quite common occurence, though. If the user had a package management application (perhaps working similarly to rebar's deps, = per project/release, no site-wide install by default) along with a package index website, it could quite easily find what he needs. It wouldn't be of unknown quality or completeness, simply because adding a comment and rating system to a package index website is fairly simple to do. Plus, he can make an opinion for himself, instead of just using whatever someone decided for him. You might include a project in OTP that fits your criteria for quality, but it might be crap for someone else. A more democratic system where people choose what they use based on other users' feedback and ratings is not only a lot more powerful but also allows to bring some light on some more obscure projects that are sometimes quite hard to find armed only with Google. It makes sense to have a single choice for big corporate environments, not so much for the rest of the world. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From serge@REDACTED Mon Oct 1 17:39:40 2012 From: serge@REDACTED (Serge Aleynikov) Date: Mon, 01 Oct 2012 11:39:40 -0400 Subject: [erlang-questions] can a program launched with open_port({spawn, Cmd}, Options) remain running after the port closes? In-Reply-To: <76CE4188-BC5E-4DA4-8016-39CD7CDC326D@gmail.com> References: <5066197B.5020008@aleynikov.org> <76CE4188-BC5E-4DA4-8016-39CD7CDC326D@gmail.com> Message-ID: <5069B93C.4070507@aleynikov.org> Thanks for your tip on the dynamic configuration of rebar. Below please find a version of erlexec that uses rebar: https://github.com/saleyn/erlexec Along with its documentation: http://saleyn.github.com/erlexec The old repository http://code.google.com/p/erlexec/ is deprecated. Please let me know if this builds on other OS's besides Linux. Regards, Serge On 9/29/2012 5:23 AM, Tim Watson wrote: > I'm a heavy rebar user and have compiled ports with it before, so give me a shout if you need any assistance. I can also do testing on several unix platforms. > > On 28 Sep 2012, at 22:41, Serge Aleynikov wrote: > >> Unfortunately I don't have OSx to test this. I can't reproduce your >> problem on Linux. Perhaps you could check the libtool version this way: >> >> $ grep "^macro_version" /usr/bin/libtool >> macro_version=2.4 >> >> and if yours is different, try to upgrade it. >> >> In order to simplify the build process I'll try to convert erlexec to >> use rebar some time next week, which will eliminate autotools-based >> toolchain. Need to figure out with rebar how to test and optionally >> enable conditional defines based on presence of special C headers such >> as "sys/capability.h" when compiling a port program. >> >> On 9/28/2012 5:16 PM, Tim Watson wrote:> Well I do, but clearly not the >> same version as you? >>> >>> t4@REDACTED:erlexec $ which libtool >>> /usr/bin/libtool >>> t4@REDACTED:erlexec $ ls -la /usr/bin/ | grep libtool >>> -r-xr-xr-x 1 root wheel 149216 22 May 15:30 libtool >>> lrwxr-xr-x 1 root wheel 7 22 May 15:30 ranlib -> libtool >>> t4@REDACTED:erlexec $ libtool --version >>> libtool: unknown option character `-' in: --version >>> Usage: libtool -static [-] file [...] [-filelist listfile[,dirname]] >> [-arch_only arch] [-sacLT] >>> Usage: libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] >> [-arch_only arch] [-o output] [-install_name name] >> [-compatibility_version #] [-current_version #] [-seg1addr 0x#] >> [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table >> ] [-seg_addr_table_filename ] [-all_load] >> [-noall_load] >>> t4@REDACTED:erlexec $ >>> >>> Honestly the osx build tool chain drives me nuts sometimes. :/ >>> >>> On 28 Sep 2012, at 17:24, Serge Aleynikov wrote: >>> >>>> Is it possible that you don't have libtool installed (missing >>>> AC_PROG_LIBTOOL)? >>>> >>>> I have the following on linux and everything builds fine: >>>> $ autoconf -V >>>> autoconf (GNU Autoconf) 2.68 >>>> $ libtool --version >>>> libtool (GNU libtool) 2.4 >>>> $ ./build >>>> $ make >>>> $ make install >>>> $ tree install >>>> install >>>> ??? exec-1.0 >>>> ??? ebin >>>> ? ??? exec.app >>>> ? ??? exec_app.beam >>>> ? ??? exec.beam >>>> ??? include >>>> ? ??? exec.hrl >>>> ??? priv >>>> ? ??? x86_64-unknown-linux-gnu >>>> ? ??? bin >>>> ? ??? exec-port >>>> ??? src >>>> ??? exec_app.erl >>>> ??? exec.erl >>>> >>>> >>>> On 9/28/2012 7:45 AM, Tim Watson wrote:> The project doesn't appear to >>>> build cleanly on mac os lion: >>>>> >>>>> t4@REDACTED:erlexec $ ./configure >>>>> configure: error: cannot find install-sh, install.sh, or shtool in >>>> config "."/config >>>>> t4@REDACTED:erlexec $ ./build >>>>> Install dir: /usr/local/src/erlang/erlexec/install >>>>> aclocal >>>>> autoconf -i >>>>> configure.ac:19: error: possibly undefined macro: AC_PROG_LIBTOOL >>>>> If this token and others are legitimate, please use >> m4_pattern_allow. >>>>> See the Autoconf documentation. >>>>> make: *** [configure] Error 1 >>>>> t4@REDACTED:erlexec $ >>>>> t4@REDACTED:erlexec $ autoconf -V >>>>> autoconf (GNU Autoconf) 2.69 >>>>> Copyright (C) 2012 Free Software Foundation, Inc. >>>>> License GPLv3+/Autoconf: GNU GPL version 3 or later >>>>> , >>>> >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. >>>>> >>>>> Written by David J. MacKenzie and Akim Demaille. >>>>> t4@REDACTED:erlexec $ >>>>> >>>>> >>>>> Is there a specific version of autotools required to build and compile >>>> erlexec? >>>>> >>>>> On 27 Sep 2012, at 20:11, Tim Watson wrote: >>>>> >>>>>> On 27 Sep 2012, at 19:18, freza@REDACTED wrote: >>>>>>> Haven't been following this thread closely, but it seems erlexec >> wasn't >>>>>>> mentioned yet: >>>>>>> >>>>>>> http://code.google.com/p/erlexec/ >>>>>> >>>>>> Doesn't appear very well maintained. I might pick it apart and >>>> rewrite it when native processes get released - it is a cool idea. >>>> Anyway, for the time being it's not a great help as I'm writing a >>>> framework to help set up and tear down clusters for distributed testing >>>> (based largely on common test) so I want to minimise the number of >>>> dependencies as much as possible. Besides, I'm more interested in >>>> working on the remote/ssh runner+monitor next, rather than spending >>>> another age fiddling around with the script runner. >>>>>> >>>>>> But yes, erlexec does look interesting and has been on my radar since >>>> it popped up on this list a few months back, discussing the desire to do >>>> proper signal handling (such as sending SIGTERM, SIGHUP, etc to the >>>> external program) without resorting to os:cmd("kill -" ++ Code ++ " " ++ >>>> OsPid). Again, this is something that linked-in drivers *can* deal with >>>> to a limited extent, and therefore I suspect that native processes will >>>> make for a very nice foundation on which to build an alternative to >>>> open_port/2. >>>>>> >>>>>> Also, IIRC there was a cool command line testing framework open >>>> sourced in the last year which was announced on this list. I can't >>>> remember what it was called, but it has a port driver that handles the >>>> the external comms and so on - looked very nice, although that's not >>>> what I'm trying to do/be as my focus is on getting things in a >>>> consistent state and monitoring to make sure they stay that way through >>>> the test run, then tearing down nicely before the next phase kicks off. >>>>>> >>>>>> Cheers, >>>>>> Tim >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> erlang-questions mailing list >>>>> erlang-questions@REDACTED >>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>> >>> From kenneth.lundin@REDACTED Mon Oct 1 20:11:09 2012 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Mon, 1 Oct 2012 20:11:09 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5069B90C.7070504@ninenines.eu> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> Message-ID: Den 1 okt 2012 17:39 skrev "Lo?c Hoguin" : > > On 10/01/2012 04:30 PM, Kenneth Lundin wrote: >> >> I agree with Serge that it is somewhat unfortunate that there are so >> many initiatives >> where people write their own implementation of something we already >> have included in >> the Erlang/OTP distribution and doing this without even suggesting or >> asking us if we could >> incorporate some missing feature or improve the current implementation >> in various ways. > > > It happens a lot. People tend to review available options, and if none fits their needs, write something themselves without consulting any of the authors. Sometimes they're right to do so, because the answer they'll get is obvious, and sometimes not. > > There's a few reasons this happens for this specific case: > > * HTTP is simple, it's easier to go ahead and experiment on a new project than on an existing one. > * If the experiment succeeds and the implementation has to differ significantly from other projects, it's not worth backporting since you'll just run into a wall of backward compatibility requirements. > * OTP has a long release process. Many companies need their fixes to be included quickly to avoid maintaining huge branches for their changes. They generally agree in return to keep track of changes that might happen during the product's development. I think we are pretty fast with service releases about every 3rd month and updates on the maint branch in git between that. > * Inets is seriously well hidden. I only heard about it after releasing Cowboy. I knew about Yaws/Mochiweb/Misultin but not the inets HTTP server. Many people I talked to didn't seem to know it existed. I agree on that. I have always thought that the name inets is misleading. > * Sometimes removing features is an improvement. Inets is big, it's only natural that lightweight alternatives appear. Same happened to Apache, and it's no surprise that Inets is quite similar to Apache from a user point of view. I agree here too. The http server included in OTP should have been a lightweight one leaving for others to implement a more complete or specialized one. > > Note that while not everyone uses inets, most servers do use erlang:decode_packet/3 which does most of the job. Cowboy doesn't anymore, because it needed backward incompatible changes to erlang:decode_packet/3. I didn't ask your thoughts, simply because the experiment I did hinted that doing the parsing in full Erlang was a better idea: this change results in improved HiPE optimizations and better reduction management which results in better latency under load. Interesting, would be good if you could give us more info on this. The http parsing in decode_packet is there to improve performance. If it is not ( without using HiPE) we should consider removing it! > > >> In Erlang/OTP we have keep strict compatibility with many of our APIs >> which is both a strength and sometimes a burden (when we are not happy >> with the old API) but there are mostly ways around that if there is >> strong case for a change. > > > It's a big strength for people looking for stability and a big burden for people looking for reactivity. Can't really blame the second group for finding solutions to their problems. I am not blaming people for implementing their own solution, it is good to have a lot to choose from. But I still think the stability is very important for the majority of users making systems with a longer life cycle than a year. Our experience is that we have to be very compatible in order to convince our users (often big projects) to step up to our latest release. By this we can also stop supporting older versions. > > >> We still actively develop inets with its http client and server and I >> am not sure that >> all the reasons to develop alternatives still exist. > > > But like you said the level of support for applications included in OTP vary widely. It is also not clearly indicated which parts are the focus of recent works (there is a small roadmap for the most important stuff but that's about it), nor can it be deduced from reading tickets in a bug tracker or similar collaboration tool. Yes we plan to improve in this area. I would say that the level of support is equally high for all applications we regard as important and heavily used by Ericsson and or the external Erlang community. > > Which brings me to a very important point: please start using a public bug tracker! It doesn't have to be 100% public (as I'm sure you have some issues that are internal only) but it would help a lot if people could see the existing issues instead of stumbling upon the same bugs again and again. It probably also would help with contributions. We have discussed this as well. > > >> I think it is of great value to have a relative complete functionality >> in Erlang/OTP so that >> the user can implement basic functionality without having to search >> for separate implementations of, with unknown quality and completeness >> at least for a beginner. > > > I would prefer Erlang/OTP to focus on the core features and leave the satellite features to third parties. OTP has an HTTP server but doesn't have proper Unicode support. OTP has OpenGL but doesn't have proper timezones support. These 2 examples missing from OTP are some of many things that are too complex for most people to implement, and that would not benefit from having alternatives. What is missing when it comes to timezones? Has it been mentioned in another thread? I think we rely on the OS configuration here. > > Let's take another example. While taking a quick glance at the modules included in OTP I discovered there is a GD implementation in the percept application. I'm pretty sure most people didn't know it was there, as they usually end up using a NIF. It's quite unexpected that it's in OTP. It was developed as part of Percept just to have something to use there. We don't regard it complete or general enough to be supported as an app of its own. > > Finding things in OTP unexpectedly is a quite common occurence, though. I think most things are documented but the docs could have better search facilities. > > If the user had a package management application (perhaps working similarly to rebar's deps, = per project/release, no site-wide install by default) along with a package index website, it could quite easily find what he needs. It wouldn't be of unknown quality or completeness, simply because adding a comment and rating system to a package index website is fairly simple to do. I agree that a package management system would be good. There are initiatives in that direction. > > Plus, he can make an opinion for himself, instead of just using whatever someone decided for him. You might include a project in OTP that fits your criteria for quality, but it might be crap for someone else. A more democratic system where people choose what they use based on other users' feedback and ratings is not only a lot more powerful but also allows to bring some light on some more obscure projects that are sometimes quite hard to find armed only with Google. > > It makes sense to have a single choice for big corporate environments, not so much for the rest of the world. > Striving in that direction. > -- > Lo?c Hoguin > > Erlang Cowboy > Nine Nines > http://ninenines.eu /Kenneth, Erlang/OTP Ericsson -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Oct 1 21:52:53 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Mon, 01 Oct 2012 21:52:53 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> Message-ID: <5069F495.7050907@ninenines.eu> On 10/01/2012 08:11 PM, Kenneth Lundin wrote: > > * OTP has a long release process. Many companies need their fixes to > be included quickly to avoid maintaining huge branches for their > changes. They generally agree in return to keep track of changes that > might happen during the product's development. > I think we are pretty fast with service releases about every 3rd month > and updates on the maint branch in git between that. But doesn't that imply upgrading the whole of OTP? If it does, that means more testing and more chances for new issues. If you can build a release upgrade with one app from one OTP version and the rest from a previous version, I'd like to know how. > > Note that while not everyone uses inets, most servers do use > erlang:decode_packet/3 which does most of the job. Cowboy doesn't > anymore, because it needed backward incompatible changes to > erlang:decode_packet/3. I didn't ask your thoughts, simply because the > experiment I did hinted that doing the parsing in full Erlang was a > better idea: this change results in improved HiPE optimizations and > better reduction management which results in better latency under load. > > Interesting, would be good if you could give us more info on this. The > http parsing in > decode_packet is there to improve performance. If it is not ( without > using HiPE) we should consider removing it! Between now and then binaries appeared, got performance improvements, and considering Cowboy is full binary (no conversion to atom for methods or header names or anything) and only receives requests, not responses, that means it can skip many steps that decode_packet/3 can't. Binaries also mean you can parse the request line and all the headers using a single match context which seems to be very fast (good job!) and I'm guessing is the main reason for improved performance under HiPE (I'm about to start reading the HiPE papers to understand that better). > >> In Erlang/OTP we have keep strict compatibility with many of our APIs > >> which is both a strength and sometimes a burden (when we are not happy > >> with the old API) but there are mostly ways around that if there is > >> strong case for a change. > > > > > > It's a big strength for people looking for stability and a big burden > for people looking for reactivity. Can't really blame the second group > for finding solutions to their problems. > > I am not blaming people I used a word too strong there. :) > for implementing their own solution, it is good > to have a lot to choose from. But I still think the stability is very > important for the majority of users making systems with a longer life > cycle than a year. > Our experience is that we have to be very compatible in order to > convince our > users (often big projects) to step up to our latest release. > By this we can also stop supporting older versions. Yes and I wouldn't want that to stop, that's a strong point. In fact having both ways is great, it means you can make sure the important parts of your system are stable while the more bleeding-edge parts are kept, well, bleeding-edge. > >> I think it is of great value to have a relative complete functionality > >> in Erlang/OTP so that > >> the user can implement basic functionality without having to search > >> for separate implementations of, with unknown quality and completeness > >> at least for a beginner. > > > > > > I would prefer Erlang/OTP to focus on the core features and leave the > satellite features to third parties. OTP has an HTTP server but doesn't > have proper Unicode support. OTP has OpenGL but doesn't have proper > timezones support. These 2 examples missing from OTP are some of many > things that are too complex for most people to implement, and that would > not benefit from having alternatives. > > What is missing when it comes to timezones? Has it been mentioned in > another thread? > I think we rely on the OS configuration here. Date/time conversion from one timezone to another. For example if your data is saved with UTC time but the user is in Paris you want to convert it to Europe/Paris. Perhaps it exists, but I haven't found it. I think there's a lot of work to be done with regards to localization and internationalization support, but I haven't had enough time to think about it in details yet. Unicode (not just Unicode though, but a native string type abstracting encodings, EEP coming in a few weeks), timezones, graphical drawing libraries are pretty much what I find lacks the most for the web side of things. Graphical drawing libraries are fine outside of OTP for a while (until there's one that can do near everything and do it well) though. Thanks for the info on package management and bug tracker. Hope we'll be involved in the process of creating the former. And thanks for your nice reply. :) -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From roberto@REDACTED Mon Oct 1 22:25:15 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 1 Oct 2012 13:25:15 -0700 Subject: [erlang-questions] [ANN] SublimErl v0.4 Message-ID: Dear list, SublimErl v0.4 has been released. The major feature update is code auto-indenting, i.e. indent code automatically with the press of a key. For those who don't know, SublimErl is a Sublime Text 2 plugin for Erlang. More info here: https://github.com/ostinelli/SublimErl/tree/0.4 All other features are still supported: - Benefit from *Code Completion* ( all Erlang libs + your current project ) - Run *Eunit* tests ( all tests from file / single test ) - Run *Common Tests* ( single file ) - Run *Dialyzer* tests ( single file ) Cheers, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Mon Oct 1 22:29:04 2012 From: freza@REDACTED (freza@REDACTED) Date: Mon, 1 Oct 2012 16:29:04 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5069B90C.7070504@ninenines.eu> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> Message-ID: <20121001202904.GA10476@circlewave.net> On Mon, Oct 01, 2012 at 05:38:52PM +0200, Lo?c Hoguin wrote: > Finding things in OTP unexpectedly is a quite common occurence, though. Not OTP's fault, anybody can do (for example): $ mandb ~/R15B01/lib/erlang/man [... noise ...] $ apropos -M ~/R15B01/lib/erlang/man http http_uri (3) - URI utility module httpc (3) - An HTTP/1.1 client httpd (3) - An implementation of an HTTP 1.1 compliant Web server, as defined in RFC 2616. httpd_conf (3) - Configuration utility functions to be used by the Erlang Web server API programmer. httpd_socket (3) - Communication utility functions to be used by the Erlang Web server API programmer. httpd_util (3) - Miscellaneous utility functions to be used when implementing Erlang Web server API modules. Standard functionality, really. And there's always find(1) and grep(1) to assist with detective work -- not perfect, but quite efficient anyways. BR, -- Jachym From max.lapshin@REDACTED Tue Oct 2 07:09:13 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Tue, 2 Oct 2012 09:09:13 +0400 Subject: [erlang-questions] [ANN] Cowboy 0.6.1 In-Reply-To: References: <502254C6.1040407@ninenines.eu> <3F3CC0C2-B801-48D9-BA35-B2D0693E89EE@gmail.com> <5022745A.2080302@ninenines.eu> <50228834.9000601@ninenines.eu> <41123609-B62E-44C9-BCD5-7FCA2BE934FC@gmail.com> Message-ID: Hi, Loic. How would it be possible to do this thing in new cowboy: https://github.com/erlyvideo/flussonic/blob/master/apps/flussonic/src/mpegts_handler.erl#L61 I need to send stream via HTTP without end and without chunked response. From bsvancara@REDACTED Tue Oct 2 14:40:16 2012 From: bsvancara@REDACTED (Bohuslav Svancara) Date: Tue, 2 Oct 2012 14:40:16 +0200 Subject: [erlang-questions] C:\Program Files\erlx.x.x\bin - yes or no? In-Reply-To: References: <5056FEDD.6070700@rabbitmq.com> <505726CA.7070409@rabbitmq.com> <505846BA.4000303@rabbitmq.com> <50585161.7090706@rabbitmq.com> Message-ID: The same problem here. Windows 7 Prof 64-bit. This installer: http://www.erlang.org/download/otp_win64_R15B02.exe skips installation of vcredist_x64.exe. This is its log: Output folder: C:\Program Files\erl5.9.2 Extract: Install.ini Extract: Install.exe... 100% Output folder: C:\Program Files\erl5.9.2\releases Output folder: C:\Program Files\erl5.9.2\releases\R15B02 ... This installer: http://www.erlang.org/download/otp_win64_R15B01.exe Does not skip this step and installs vcredist_x64.exe. This is its log: Output folder: C:\Program Files\erl5.9.1 Extract: vcredist_x64.exe Execute: "C:\Program Files\erl5.9.1\vcredist_x64.exe" Output folder: C:\Program Files\erl5.9.1 Extract: Install.ini Extract: Install.exe... 100% Output folder: C:\Program Files\erl5.9.1\releases Output folder: C:\Program Files\erl5.9.1\releases\R15B01 .... *otp_win64_R15B02.exe* does not create correct directory structure - see a picture. But if one installs *otp_win64_R15B01.exe* before *otp_win64_R15B02.exe*, *otp_win64_R15B02.exe* creates correct directory structure (!) and runs fine then. Thank you. Bob 2012/9/18 Lukas Larsson > Hi, > > I've found the bug, however because of the way our build procedures > work it will take me a day to build and test the new release, hope > that is ok. > > You are absolutely right about installing win64 on win32, I'll put a > ticket in our backlog to fix that issue. > > Lukas > > On Tue, Sep 18, 2012 at 12:48 PM, Simon MacMullen > wrote: > > ...although there is still the installing-win64-on-win32 thing. You may > very > > well say "user error", but it would be really nice if the installer > didn't > > look like it had succeeded :-) > > > > Cheers, Simon > > > > > > On 18/09/12 11:34, Lukas Larsson wrote: > >> > >> Hi, > >> > >> The .dll should have been part of it, however when fixing something > >> else we apparently broke this functionality. R15B01 works as it > >> should, we'll build a new installer for R15B02 after lunch and get it > >> to you as soon as we can. > >> > >> Thanks for reporting this error! > >> > >> Lukas > >> > >> On Tue, Sep 18, 2012 at 12:02 PM, Simon MacMullen > >> wrote: > >>> > >>> On 17/09/12 17:25, Lukas Larsson wrote: > >>>> > >>>> > >>>> have you tried running Install.exe manually in cmd? > >>> > >>> > >>> > >>> "The program can't start because MSVCR100.dll is missing from your > >>> computer. > >>> Try reinstalling the program to fix this problem." > >>> > >>> So we have a culprit. > >>> > >>> I guess this is a shared library that can be installed by tons of > things, > >>> but I was installing on a brand new system. Is it possible to > distribute > >>> this library with Erlang? If not, is it possible to detect its absence > in > >>> the installer and pop up an error message telling the user to download > >>> it? > >>> :-) > >>> > >>> BTW, I also get the same symptoms installing win64 on a win32 system. > >>> Yes, > >>> of course you shouldn't do that, but again the failure is unobvious - > you > >>> just get a partial installation and no error message. > >>> > >>> > >>> Cheers, Simon > >>> > >>> -- > >>> Simon MacMullen > >>> RabbitMQ, VMware > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > > -- > > Simon MacMullen > > RabbitMQ, VMware > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: erl592dirs.png Type: image/png Size: 8407 bytes Desc: not available URL: From lukas@REDACTED Tue Oct 2 14:54:09 2012 From: lukas@REDACTED (Lukas Larsson) Date: Tue, 2 Oct 2012 14:54:09 +0200 Subject: [erlang-questions] C:\Program Files\erlx.x.x\bin - yes or no? In-Reply-To: References: <5056FEDD.6070700@rabbitmq.com> <505726CA.7070409@rabbitmq.com> <505846BA.4000303@rabbitmq.com> <50585161.7090706@rabbitmq.com> Message-ID: Hi! If you are having the same problem you want to download http://www.erlang.org/download/otp_win64_R15B02_with_MSVCR100_installer_fix.exe I just realized that http://www.erlang.org/download.html had not been updated with this installer, that should be fixed now. Lukas On Tue, Oct 2, 2012 at 2:40 PM, Bohuslav Svancara wrote: > The same problem here. > > Windows 7 Prof 64-bit. > > This installer: http://www.erlang.org/download/otp_win64_R15B02.exe > skips installation of vcredist_x64.exe. > > This is its log: > > Output folder: C:\Program Files\erl5.9.2 > Extract: Install.ini > Extract: Install.exe... 100% > Output folder: C:\Program Files\erl5.9.2\releases > Output folder: C:\Program Files\erl5.9.2\releases\R15B02 > ... > > This installer: http://www.erlang.org/download/otp_win64_R15B01.exe > Does not skip this step and installs vcredist_x64.exe. > > This is its log: > > Output folder: C:\Program Files\erl5.9.1 > Extract: vcredist_x64.exe > Execute: "C:\Program Files\erl5.9.1\vcredist_x64.exe" > Output folder: C:\Program Files\erl5.9.1 > Extract: Install.ini > Extract: Install.exe... 100% > Output folder: C:\Program Files\erl5.9.1\releases > Output folder: C:\Program Files\erl5.9.1\releases\R15B01 > .... > > > otp_win64_R15B02.exe does not create correct directory structure - see a > picture. > But if one installs otp_win64_R15B01.exe before otp_win64_R15B02.exe, > otp_win64_R15B02.exe creates correct directory structure (!) and runs fine > then. > > Thank you. > Bob > > 2012/9/18 Lukas Larsson >> >> Hi, >> >> I've found the bug, however because of the way our build procedures >> work it will take me a day to build and test the new release, hope >> that is ok. >> >> You are absolutely right about installing win64 on win32, I'll put a >> ticket in our backlog to fix that issue. >> >> Lukas >> >> On Tue, Sep 18, 2012 at 12:48 PM, Simon MacMullen >> wrote: >> > ...although there is still the installing-win64-on-win32 thing. You may >> > very >> > well say "user error", but it would be really nice if the installer >> > didn't >> > look like it had succeeded :-) >> > >> > Cheers, Simon >> > >> > >> > On 18/09/12 11:34, Lukas Larsson wrote: >> >> >> >> Hi, >> >> >> >> The .dll should have been part of it, however when fixing something >> >> else we apparently broke this functionality. R15B01 works as it >> >> should, we'll build a new installer for R15B02 after lunch and get it >> >> to you as soon as we can. >> >> >> >> Thanks for reporting this error! >> >> >> >> Lukas >> >> >> >> On Tue, Sep 18, 2012 at 12:02 PM, Simon MacMullen >> >> wrote: >> >>> >> >>> On 17/09/12 17:25, Lukas Larsson wrote: >> >>>> >> >>>> >> >>>> have you tried running Install.exe manually in cmd? >> >>> >> >>> >> >>> >> >>> "The program can't start because MSVCR100.dll is missing from your >> >>> computer. >> >>> Try reinstalling the program to fix this problem." >> >>> >> >>> So we have a culprit. >> >>> >> >>> I guess this is a shared library that can be installed by tons of >> >>> things, >> >>> but I was installing on a brand new system. Is it possible to >> >>> distribute >> >>> this library with Erlang? If not, is it possible to detect its absence >> >>> in >> >>> the installer and pop up an error message telling the user to download >> >>> it? >> >>> :-) >> >>> >> >>> BTW, I also get the same symptoms installing win64 on a win32 system. >> >>> Yes, >> >>> of course you shouldn't do that, but again the failure is unobvious - >> >>> you >> >>> just get a partial installation and no error message. >> >>> >> >>> >> >>> Cheers, Simon >> >>> >> >>> -- >> >>> Simon MacMullen >> >>> RabbitMQ, VMware >> >>> _______________________________________________ >> >>> erlang-questions mailing list >> >>> erlang-questions@REDACTED >> >>> http://erlang.org/mailman/listinfo/erlang-questions >> > >> > >> > >> > -- >> > Simon MacMullen >> > RabbitMQ, VMware >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From Sergey_Zhemzhitsky@REDACTED Tue Oct 2 15:55:58 2012 From: Sergey_Zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Tue, 2 Oct 2012 13:55:58 +0000 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. Message-ID: <06139A918ACCA041BF46A0F36940C7FA4FE49735@exch-mbx1.msk.trd.ru> Hello erlang gurus, Currently I?ve faced with the following issue with termination of threads created in port drivers. I?m creating a separate thread in the erlang port driver that sends terms to some erlang process. -------------------------------------- The thread is created like this: ErlDrvTid threadId; if(erl_drv_thread_create(?thread name?, &threadId, threadFunc, threadData, NULL)) { // log error & exit } driverData->tid = threadId; Driver data looks like this: struct DriverData { ErlDrvPort port; ErlDrvTid tid; int isRunning; } -------------------------------------- Thread function looks like this DriverData* data = (DriverData*) threadData data. isRunning = 1; while(data.isRunning) { doSomething(); } -------------------------------------- When I?d like to stop this thread from one of the driver callbacks I do something like this: void handleInput(ErlDrvData handle, char * buf, ErlDrvSizeT len) { DriverData* data = (DriverData*) handle; int command = parseBuf(buf, len); if (command == STOP) { data.isRunning = 0; if(erl_drv_thread_join(data->tid, NULL)) { // here is the place where EDEADLK happens // report error } } else { ? } } >From time to time erl_drv_thread_join returns error EDEADLK=35, i.e. the current thread (scheduler thread) tries to join itself. According to the documentation ?A Thread identifier may be reused very quickly after a thread has terminated. Therefore, if a thread corresponding to one of the involved thread identifiers has terminated since the thread identifier was saved, the result of erl_drv_equal_tids() might not give the expected result.? I suppose that thread terminates earlier then erl_drv_thread_join call happens, so ErlDrvTid is already reused. So the question is how to use erl_drv_thread_join properly and how to guarantee that the saved ErlDrvTid value points to the same data that was returned from erl_drv_thread_create? Best Regards, Sergey _______________________________________________________ CONFIDENTIALITY NOTICE: This email and any files attached to it may be conf idential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas@REDACTED Tue Oct 2 16:18:25 2012 From: thomas@REDACTED (Thomas Allen) Date: Tue, 2 Oct 2012 10:18:25 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: Message-ID: <0cc53c319c28f5f088a4039eb8a3daa8.squirrel@oinksoft.com> > "The standard library is where modules go to die" > http://www.leancrew.com/all-this/2012/04/where-modules-go-to-die/ > This is from python world, and is very obviously seen in Erlang. Could you cite some examples of third-party libraries being absorbed into the Erlang/OTP distribution and "dying"? Perhaps this is not as obvious to everybody as you think. > Alternatively https://npmjs.org/ has 15k+ packages, we dont need to add > user management and the kitchen sink to inets, we need to remove it from > the standard library and promote userland packages as first class citizens > (ie pick a tool to install packages and bundle it) I couldn't disagree with that much more ... npm/Node.js is cluttered with packages of dubious quality that are not maintained in the slightest, and while this "Wild West" they prefer may seem exciting, I do not think it bodes well for the long-term stability of projects that depend on these packages. If some particular functionality becomes used in a large body of Erlang projects, it should certainly be brought into the stdlib. Per your Python reference, one need only look at the "json" module for an example where bringing something into the stdlib has made deployment and maintenance of Python code-bases simpler -- no more installing the "simplejson" package, docs right there on docs.python.org, and JSON earning first-class status as a serialization format for Python objects. To advocate actually removing things from the Erlang stdlib, potentially breaking working code on a large scale, is madness to me. Thomas Allen From comptekki@REDACTED Tue Oct 2 16:51:00 2012 From: comptekki@REDACTED (Wes James) Date: Tue, 2 Oct 2012 08:51:00 -0600 Subject: [erlang-questions] sending message to more than one node Message-ID: With {process_name, node@REDACTED} ! <<"message">> I can send a message to one node. What if I want to send to more than one node? Do I make a list of nodes and then loop through that list and send a message to each? Thanks, wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Tue Oct 2 16:54:42 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Tue, 2 Oct 2012 17:54:42 +0300 Subject: [erlang-questions] sending message to more than one node In-Reply-To: References: Message-ID: Hi, rpc:multi_server_call http://erldocs.com/R14B02/kernel/rpc.html?i=22&search=rpc#multi_server_call/2 - Dmitry On Oct 2, 2012, at 5:51 PM, Wes James wrote: > With {process_name, node@REDACTED} ! <<"message">> I can send a message to one node. What if I want to send to more than one node? Do I make a list of nodes and then loop through that list and send a message to each? > > Thanks, > > wes > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From comptekki@REDACTED Tue Oct 2 16:57:27 2012 From: comptekki@REDACTED (Wes James) Date: Tue, 2 Oct 2012 08:57:27 -0600 Subject: [erlang-questions] sending message to more than one node In-Reply-To: References: Message-ID: ok thanks, I'll take a look. Wes On Tue, Oct 2, 2012 at 8:54 AM, Dmitry Kolesnikov wrote: > Hi, > > rpc:multi_server_call > > http://erldocs.com/R14B02/kernel/rpc.html?i=22&search=rpc#multi_server_call/2 > > > - Dmitry > > On Oct 2, 2012, at 5:51 PM, Wes James wrote: > > > With {process_name, node@REDACTED} ! <<"message">> I can send a message to > one node. What if I want to send to more than one node? Do I make a list > of nodes and then loop through that list and send a message to each? > > > > Thanks, > > > > wes > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Tue Oct 2 17:52:37 2012 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 02 Oct 2012 11:52:37 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5069F495.7050907@ninenines.eu> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> Message-ID: <506B0DC5.6010809@aleynikov.org> I agree with Joe that having a design of the HTTP server architecture included in OTP that would offer standardized internal API and allow developers to either replace selected functionality via providing callback modules or swap the HTTP servers easily would be a great plus. Concerning inets, what's really strange is that whenever we see some benchmarks posted comparing performance of different HTTP implementations, inets is never included there, though I would think, it would make more sense in terms of showing how a custom implementation compares to what's included in OTP. Does it have to do something with what Lo?c referred to as inets being well hidden in OTP? I doubt, but if not that, then what? There are not even benchmarks comparing yaws to inets, and despite the fact that both offer a rich set of functionality make one wonder was was the idea of writing yaws in the first place given existence of inets' httpd. Diversity is good, but at some point it creates a burden for the user who needs to make a selection of an HTTP server for his project. Instead of solving the application-related problem, now the task becomes studying several implementations to see which one offers a feature-set best suited for the problem being solved, and the answer to that question is not always straightforward. Perhaps a good initiative from the OTP team would be to spec out the open architecture of an HTTP server and welcome feedback from authors of current http libraries, so that the successor of inets would be flexible enough to accommodate building light-weight and heavy-weight HTTP servers based on the supported code base included in OTP? This would also open a possibility of diversity for library writers replacing/enhancing parts to suit their needs. Serge On 10/1/2012 3:52 PM, Lo?c Hoguin wrote: > On 10/01/2012 08:11 PM, Kenneth Lundin wrote: >> > * OTP has a long release process. Many companies need their fixes to >> be included quickly to avoid maintaining huge branches for their >> changes. They generally agree in return to keep track of changes that >> might happen during the product's development. >> I think we are pretty fast with service releases about every 3rd month >> and updates on the maint branch in git between that. > > But doesn't that imply upgrading the whole of OTP? If it does, that > means more testing and more chances for new issues. If you can build a > release upgrade with one app from one OTP version and the rest from a > previous version, I'd like to know how. > >> > Note that while not everyone uses inets, most servers do use >> erlang:decode_packet/3 which does most of the job. Cowboy doesn't >> anymore, because it needed backward incompatible changes to >> erlang:decode_packet/3. I didn't ask your thoughts, simply because the >> experiment I did hinted that doing the parsing in full Erlang was a >> better idea: this change results in improved HiPE optimizations and >> better reduction management which results in better latency under load. >> >> Interesting, would be good if you could give us more info on this. The >> http parsing in >> decode_packet is there to improve performance. If it is not ( without >> using HiPE) we should consider removing it! > > Between now and then binaries appeared, got performance improvements, > and considering Cowboy is full binary (no conversion to atom for methods > or header names or anything) and only receives requests, not responses, > that means it can skip many steps that decode_packet/3 can't. > > Binaries also mean you can parse the request line and all the headers > using a single match context which seems to be very fast (good job!) and > I'm guessing is the main reason for improved performance under HiPE (I'm > about to start reading the HiPE papers to understand that better). > >> >> In Erlang/OTP we have keep strict compatibility with many of our APIs >> >> which is both a strength and sometimes a burden (when we are not >> happy >> >> with the old API) but there are mostly ways around that if there is >> >> strong case for a change. >> > >> > >> > It's a big strength for people looking for stability and a big burden >> for people looking for reactivity. Can't really blame the second group >> for finding solutions to their problems. >> >> I am not blaming people > > I used a word too strong there. :) > >> for implementing their own solution, it is good >> to have a lot to choose from. But I still think the stability is very >> important for the majority of users making systems with a longer life >> cycle than a year. >> Our experience is that we have to be very compatible in order to >> convince our >> users (often big projects) to step up to our latest release. >> By this we can also stop supporting older versions. > > Yes and I wouldn't want that to stop, that's a strong point. In fact > having both ways is great, it means you can make sure the important > parts of your system are stable while the more bleeding-edge parts are > kept, well, bleeding-edge. > >> >> I think it is of great value to have a relative complete >> functionality >> >> in Erlang/OTP so that >> >> the user can implement basic functionality without having to search >> >> for separate implementations of, with unknown quality and >> completeness >> >> at least for a beginner. >> > >> > >> > I would prefer Erlang/OTP to focus on the core features and leave the >> satellite features to third parties. OTP has an HTTP server but doesn't >> have proper Unicode support. OTP has OpenGL but doesn't have proper >> timezones support. These 2 examples missing from OTP are some of many >> things that are too complex for most people to implement, and that would >> not benefit from having alternatives. >> >> What is missing when it comes to timezones? Has it been mentioned in >> another thread? >> I think we rely on the OS configuration here. > > Date/time conversion from one timezone to another. For example if your > data is saved with UTC time but the user is in Paris you want to convert > it to Europe/Paris. > > Perhaps it exists, but I haven't found it. > > I think there's a lot of work to be done with regards to localization > and internationalization support, but I haven't had enough time to think > about it in details yet. Unicode (not just Unicode though, but a native > string type abstracting encodings, EEP coming in a few weeks), > timezones, graphical drawing libraries are pretty much what I find lacks > the most for the web side of things. Graphical drawing libraries are > fine outside of OTP for a while (until there's one that can do near > everything and do it well) though. > > > > Thanks for the info on package management and bug tracker. Hope we'll be > involved in the process of creating the former. And thanks for your nice > reply. :) > From roberto@REDACTED Tue Oct 2 18:50:34 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Tue, 2 Oct 2012 09:50:34 -0700 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5068B9B6.6070800@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> Message-ID: On Sun, Sep 30, 2012 at 2:29 PM, Serge Aleynikov wrote: > The results are quite surprising - everyone who seems comfortable coding > in Erlang, feels compelled to implement an HTTP server. > I find this quite offensive to all the people that are putting efforts and releasing their own work open source (and especially in BSD/MIT terms). But I'm sure that you only meant to be provocative. :) I too believed that too many duplication of efforts were going on, and that's the reason why I've stopped misultin development [1] something like 8 months ago. The main result that I've obtained in closing misultin was to push people to concentrate efforts around cowboy and yaws, and at least as far as cowboy goes, this has been a success (judging by the number of products that I've helped switching over). However, I believe everyone should do whatever they feel like. Open source and many projects are the result of a bunch of intelligent people desires to bring their own creatures to life., andI see nothing bad in that. I'm actually sorry to read statements such as "diversity is bad" or "too many options". This. is. open sooooooooooource. [2] r. [1] https://github.com/ostinelli/misultin/blob/master/README.md [2] http://i2.kym-cdn.com/entries/icons/original/000/000/026/Gerard-Butler-This-Is-Sparta.jpg -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge@REDACTED Tue Oct 2 19:23:30 2012 From: serge@REDACTED (Serge Aleynikov) Date: Tue, 02 Oct 2012 13:23:30 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> Message-ID: <506B2312.4090700@aleynikov.org> Roberto, On 10/2/2012 12:50 PM, Roberto Ostinelli wrote: > On Sun, Sep 30, 2012 at 2:29 PM, Serge Aleynikov > wrote: > > The results are quite surprising - everyone who seems comfortable coding > in Erlang, feels compelled to implement an HTTP server. > > > I find this quite offensive to all the people that are putting efforts > and releasing their own work open source (and especially in BSD/MIT > terms). But I'm sure that you only meant to be provocative. :) I am guilty too of having implemented my own small Erlang HTTP server at some point in the past before deciding to switch to using inets! :-) > I too believed that too many duplication of efforts were going on, and > that's the reason why I've stopped misultin development [1] something > like 8 months ago. The main result that I've obtained in closing > misultin was to push people to concentrate efforts around cowboy and > yaws, and at least as far as cowboy goes, this has been a success > (judging by the number of products that I've helped switching over). Concerning misultin, I did previously look at it and thought it was great. Despite the success, your decision to close it in favor of promoting other http server initiatives was very much admirable. > However, I believe everyone should do whatever they feel like. Open > source and many projects are the result of a bunch of intelligent people > desires to bring their own creatures to life., andI see nothing bad in > that. I'm actually sorry to read statements such as "diversity is bad" > or "too many options". > > This. is. open sooooooooooource. [2] See my previous post. I don't think we disagree in terms of what you said above. BR, Serge From essen@REDACTED Tue Oct 2 19:37:14 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Tue, 02 Oct 2012 19:37:14 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <506B0DC5.6010809@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: <506B264A.9080105@ninenines.eu> On 10/02/2012 05:52 PM, Serge Aleynikov wrote: > I agree with Joe that having a design of the HTTP server architecture > included in OTP that would offer standardized internal API and allow > developers to either replace selected functionality via providing > callback modules or swap the HTTP servers easily would be a great plus. This wouldn't have helped Cowboy, as one of the biggest difference is the use of binary instead of strings. There's another solution, which is to write wrappers for other APIs, for example mochicow allows Mochiweb applications to use Cowboy under the hood without having to change their code. This means that the switch can be progressive instead of being brutal like a direct code switch. > Diversity is good, but at some point it creates a burden for the user > who needs to make a selection of an HTTP server for his project. It doesn't! > Instead of solving the application-related problem, now the task becomes > studying several implementations to see which one offers a feature-set > best suited for the problem being solved, and the answer to that > question is not always straightforward. Erlang is wonderful in that it allows you to write your application 100% HTTP-free, and then write another application for the HTTP layer that will be a thin layer running alongside your application. The communication layer can be separate from the business logic. If you go that way then your application's code isn't dependent on the transports used at all, and you can easily switch servers, change protocols, serialization formats or anything in your HTTP layer without this impacting your application in any way. For example, when SPDY support will be added to Cowboy, only the HTTP layer will require a change to make use of it (start_http, which only does HTTP, can be replaced by start_spdy, which does SPDY or HTTP depending on client capabilities). The application behind it shouldn't have to care about that. > Perhaps a good initiative from the OTP team would be to spec out the > open architecture of an HTTP server and welcome feedback from authors of > current http libraries, so that the successor of inets would be flexible > enough to accommodate building light-weight and heavy-weight HTTP > servers based on the supported code base included in OTP? This would > also open a possibility of diversity for library writers > replacing/enhancing parts to suit their needs. More importantly, do people actually want that? -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From moxford@REDACTED Tue Oct 2 20:09:38 2012 From: moxford@REDACTED (Mike Oxford) Date: Tue, 2 Oct 2012 11:09:38 -0700 Subject: [erlang-questions] Erlang http servers In-Reply-To: <506B264A.9080105@ninenines.eu> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> <506B264A.9080105@ninenines.eu> Message-ID: I would even go so far as to propose that almost every single Erlang project, for the foreseeable future will want an HTTP service. I would even extend this to "most project in any realm in any language." Reason: Even if you're not providing an external service, having a web-status console, admin or REST-based health/monitoring/whatever port has a whole lot of value. Right now, if you want external services you have to meld server+protocol+client. The HTTP's strongest suit is that, just like javascript, it's ubiquitous. It may not be optimal, but it's everywhere and it's easy. Would I prefer a world built on client-side(browser!) Erlang, a length-prefixed binary protocol and real sockets (instead of websocket silliness)? Hell yes. But we don't have that, and likely won't anytime soon, so HTTP+javascript continues to be the *lingua franca.* Having too many options is a Bad Thing, IMO. Erlang is already enough of a 'big scary monster in the dark' for a lot of people. If we had a unified, documented, packaged and turnkey solution withing the stdlib so that users could 'drop in and go' it would increase uptake as people at least explored it. The are more web-coders than ever, many with just enough knowledge to be dangerous. Apple vs PC back in the day --- snag all of the low-hanging cheap fruit of PC clones instead of insisting on the white-ivory-tower and then achieve market dominance (which is then yours to lose/squander at your leisure... :) -mox On Tue, Oct 2, 2012 at 10:37 AM, Lo?c Hoguin wrote: > On 10/02/2012 05:52 PM, Serge Aleynikov wrote: > >> I agree with Joe that having a design of the HTTP server architecture >> included in OTP that would offer standardized internal API and allow >> developers to either replace selected functionality via providing >> callback modules or swap the HTTP servers easily would be a great plus. >> > > This wouldn't have helped Cowboy, as one of the biggest difference is the > use of binary instead of strings. > > There's another solution, which is to write wrappers for other APIs, for > example mochicow allows Mochiweb applications to use Cowboy under the hood > without having to change their code. This means that the switch can be > progressive instead of being brutal like a direct code switch. > > > Diversity is good, but at some point it creates a burden for the user >> who needs to make a selection of an HTTP server for his project. >> > > It doesn't! > > > > Instead of solving the application-related problem, now the task becomes > > studying several implementations to see which one offers a feature-set > > best suited for the problem being solved, and the answer to that > > question is not always straightforward. > > Erlang is wonderful in that it allows you to write your application 100% > HTTP-free, and then write another application for the HTTP layer that will > be a thin layer running alongside your application. The communication layer > can be separate from the business logic. > > If you go that way then your application's code isn't dependent on the > transports used at all, and you can easily switch servers, change > protocols, serialization formats or anything in your HTTP layer without > this impacting your application in any way. > > For example, when SPDY support will be added to Cowboy, only the HTTP > layer will require a change to make use of it (start_http, which only does > HTTP, can be replaced by start_spdy, which does SPDY or HTTP depending on > client capabilities). The application behind it shouldn't have to care > about that. > > > Perhaps a good initiative from the OTP team would be to spec out the >> open architecture of an HTTP server and welcome feedback from authors of >> current http libraries, so that the successor of inets would be flexible >> enough to accommodate building light-weight and heavy-weight HTTP >> servers based on the supported code base included in OTP? This would >> also open a possibility of diversity for library writers >> replacing/enhancing parts to suit their needs. >> > > More importantly, do people actually want that? > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > ______________________________**_________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/**listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Tue Oct 2 22:56:47 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Tue, 2 Oct 2012 21:56:47 +0100 Subject: [erlang-questions] R14B04 on Tilera TILEPro64? In-Reply-To: References: Message-ID: <20121002205647.GA3000@precise> On Wed, Nov 16, 2011 at 05:40:01PM +0100, Olivier BOUDEVILLE wrote: > Hi, > > Reading this very insightful message from Bj?rn-Egil: > http://erlang.2086793.n4.nabble.com/How-to-compiler-Erlang-for-the-TILEPro64-tp2119083p2119084.html > I was wondering if some details could be shared about how the newer > (starting from R13B04) cross-compilation build process of the Erlang VM is > to be used now on these Tilera cards. > > (I tried to apply the recipe explained in this thread with R14B04 and > R13B03 without much luck; anyway I suppose a different process is to be > applied now?). Hi, any update on this? I might start playing with Tilera64 on R15B02 soonish, but would like to have a closest starting point possible. Motiejus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From klacke@REDACTED Tue Oct 2 23:29:42 2012 From: klacke@REDACTED (Claes Wikstrom) Date: Tue, 02 Oct 2012 23:29:42 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> Message-ID: <506B5CC6.3080708@hyber.org> On 10/1/12 4:30 PM, Kenneth Lundin wrote: >> > I agree with Serge that it is somewhat unfortunate that there are so > many initiatives > where people write their own implementation of something we already > have included in > the Erlang/OTP distribution and doing this without even suggesting or > asking us if we could > incorporate some missing feature or improve the current implementation > in various ways. Why, the more the merrier, let the best library win. >> b) Servers have two interfaces - the protocol interface (HTTP) and >> the interface towards >> Erlang (the API if you like). As far as I can see most severs >> have different Erlang APIs. >> This is bad. > Would be good if there was a kind of standardized API for HTTP-servers > and clients. Disagree, in todays webdevelopment the DOM is the really important interface, and that is standardized. Switching servers is work, yes, but doable. Actually, the API between the protocol HTTP and the erlang code is pretty much what distinguishes the different servers, this api can be more or less ... well good. Yaws has appmods to do this, other servers have different mechanisms. This is a matter of preference and standardizing on this pretty much removes the need for different servers and we're back on square one - we should all have contributed to inets instead. This would have lead to a worse world. -klacke From wallentin.dahlberg@REDACTED Wed Oct 3 00:29:06 2012 From: wallentin.dahlberg@REDACTED (=?ISO-8859-1?Q?Bj=F6rn=2DEgil_Dahlberg?=) Date: Wed, 3 Oct 2012 00:29:06 +0200 Subject: [erlang-questions] R14B04 on Tilera TILEPro64? In-Reply-To: <20121002205647.GA3000@precise> References: <20121002205647.GA3000@precise> Message-ID: I haven't played around with the Tilera cards since r13b -> r14b:ish so I can't give you any more insight than that mail-thread atm. >From what I remember, the compiling was pretty straight forward though tile-gcc gave a lot more warnings than ordinary gcc. Mostly functions with no prototypes. (seems scary when you first see it but its fine) Look at the xcomp file for hints to configure. I might be the case that the tilera-xcomp.conf file commited to the otp-repo is very tailored to the Tilera card we have, and certainly tailored to the MDE-version. Btw, "xcomp" is a special procedure for cross-compiling OTP. It isn't necessary to cross-compile it this way. You should be able to cross-compile it the normal way, like any other open source project. Also, I did do cross-compilation but the guys at Tilera Corp. uploaded everything to the card and compiled OTP there instead. They didn't bother with cross-compilation. That could be another option. // Bj?rn-Egil 2012/10/2 Motiejus Jak?tys > On Wed, Nov 16, 2011 at 05:40:01PM +0100, Olivier BOUDEVILLE wrote: > > Hi, > > > > Reading this very insightful message from Bj?rn-Egil: > > > http://erlang.2086793.n4.nabble.com/How-to-compiler-Erlang-for-the-TILEPro64-tp2119083p2119084.html > > I was wondering if some details could be shared about how the newer > > (starting from R13B04) cross-compilation build process of the Erlang VM > is > > to be used now on these Tilera cards. > > > > (I tried to apply the recipe explained in this thread with R14B04 and > > R13B03 without much luck; anyway I suppose a different process is to be > > applied now?). > > Hi, > > any update on this? > > I might start playing with Tilera64 on R15B02 soonish, but would like to > have a closest starting point possible. > > Motiejus > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Wed Oct 3 03:07:05 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 02 Oct 2012 21:07:05 -0400 Subject: [erlang-questions] [ANN] Spawnfest 2012 Winners In-Reply-To: References: Message-ID: <506B8FB9.90005@ferd.ca> E-Mail explaining who won and whatnot Greetings everyone After long weeks of deliberation, judges of Spawnfest 2012 have come up with a list of winners and prizes! We encourage you to visit the list at http://spawnfest.com/winners/ . The winners have been announced for a while now, but I've been somewhat too busy/lazy to come bring the good news to erlang-questions. I (and the rest of the organizing committee) would like to congratulate first and foremost, Daniel Luna, lone member of team ChaosMonkey, who won the grand prize for this year. He earns himself $500 of O'Reilly books, a GitHub Silver Plan for a year, a Raspberry Pi Cloudant BigCouch Cluster, and a VMware hoodie. Many great songs shall be written in his honor. Similarly to last year, Spawnfest also received so many generous prizes from sponsors we had no choice but to distribute them to many teams. Other teams were given different prizes including, an iPad from Basho, a ton of VMware hoodies, Github bronze plans, Heroku vouchers, a case of wine straight from Argentina via Inaka Networks, Erlang Solutions Ltd. training courses and tickets to Erlang factories, backpacks, and free months of hosting at GreenQloud. Congratulations to all teams that managed to get some of these! We'd like to thank all participants in this year's Spawnfest. Much fun was had by everyone involved over IRC, Twitter, and Instagram . We'd also like to thank the judges, who took time to try, read, and judge each of the projects submitted, trying to coordinate over continents to let us know who would win. Finally, we'd like to thank our sponsors, who help make this contest what it is through prizes, but also through support during the event itself. Notable mentions going to GitHub giving us all free private repositories for the duration of the contest, and GreenQloud for the VPSes made available to all teams that needed one. We hope we're going to be able to have a third edition of Spawnfest in 2013, and that people from the Erlang community will once again embrace it the way they did in 2011 and this year. And finally, thanks to the sponsors who make this contest possible, through services such as repositories, VPSes, advertisement, and prize giving to name a few: Visit Basho Visit GreenQloud Visit Github Visit Cloudant Visit Inaka Visit Erlang Solutions Visit O'Reilly Visit Heroku Visit vmware Visit RabbitMQ Visit 99s Visit Spawngrid Visit K??i -------------- next part -------------- An HTML attachment was scrubbed... URL: From gumm@REDACTED Wed Oct 3 04:09:38 2012 From: gumm@REDACTED (Jesse Gumm) Date: Tue, 2 Oct 2012 21:09:38 -0500 Subject: [erlang-questions] [ANN] Spawnfest 2012 Winners In-Reply-To: <506B8FB9.90005@ferd.ca> References: <506B8FB9.90005@ferd.ca> Message-ID: Congrats to all! Looking at the winners list, I see a lot of interesting new tools (and the learning thing sounds quite neat), but a distinct lack of games. This is something I feel compelled to rectify! I was seriously bummed that I couldn't participate this year due to prior engagements that weekend. Last year was a total blast. Next year! It's game time! -- Jesse Gumm Owner, Sigma Star Systems 414.940.4866 || sigma-star.com || @jessegumm On Oct 2, 2012 8:07 PM, "Fred Hebert" wrote: > Greetings everyone > > After long weeks of deliberation, judges of Spawnfest 2012 have come up > with a list of winners and prizes! We encourage you to visit the list at > http://spawnfest.com/winners/. > The winners have been announced for a while now, but I've been somewhat too > busy/lazy to come bring the good news to erlang-questions. > > I (and the rest of the organizing committee) would like to congratulate > first and foremost, Daniel Luna, lone member of team ChaosMonkey, who won > the grand prize for this year. He earns himself $500 of O'Reilly books, a > GitHub Silver Plan for a year, a Raspberry Pi Cloudant BigCouch Cluster, > and a VMware hoodie. Many great songs shall be written in his honor. > > Similarly to last year, Spawnfest also received so many generous prizes > from sponsors we had no choice but to distribute them to many teams. Other > teams were given different prizes including, an iPad from Basho, a ton of > VMware hoodies, Github bronze plans, Heroku vouchers, a case of wine > straight from Argentina via Inaka Networks, Erlang Solutions Ltd. training > courses and tickets to Erlang factories, backpacks, and free months of > hosting at GreenQloud. Congratulations to all teams that managed to get > some of these! > > We'd like to thank all participants in this year's Spawnfest. Much fun was > had by everyone involved over IRC, Twitter, and Instagram. > We'd also like to thank the judges, who took time to try, read, and judge > each of the projects submitted, trying to coordinate over continents to let > us know who would win. > > Finally, we'd like to thank our sponsors, who help make this contest what > it is through prizes, but also through support during the event itself. > Notable mentions going to GitHub giving us all free private repositories > for the duration of the contest, and GreenQloud for the VPSes made > available to all teams that needed one. > We hope we're going to be able to have a third edition of Spawnfest in > 2013, and that people from the Erlang community will once again embrace it > the way they did in 2011 and this year. > > And finally, thanks to the sponsors who make this contest possible, > through services such as repositories, VPSes, advertisement, and prize > giving to name a few: > [image: Visit Basho] > [image: Visit GreenQloud] > [image: Visit Github] > [image: Visit Cloudant] > [image: Visit Inaka] > [image: Visit Erlang Solutions] > [image: Visit O'Reilly] > [image: Visit Heroku] > [image: Visit vmware] > [image: Visit RabbitMQ] > [image: Visit 99s] > [image: Visit Spawngrid] > [image: Visit K??i] > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.eugene.turner@REDACTED Wed Oct 3 04:25:40 2012 From: michael.eugene.turner@REDACTED (Michael Turner) Date: Wed, 3 Oct 2012 11:25:40 +0900 Subject: [erlang-questions] Erlang http servers In-Reply-To: <506B0DC5.6010809@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: I recently received a copy of Building Web Applications with Erlang, which is almost entirely about yaws. The discussion of inets on this thread spurred me to look in the index under "i" -- except, oops, the book has no index. It does have an appendix about other webservers and frameworks in Erlang -- but inets is not listed there. "Well-hidden" indeed. -michael turner On Wed, Oct 3, 2012 at 12:52 AM, Serge Aleynikov wrote: > I agree with Joe that having a design of the HTTP server architecture > included in OTP that would offer standardized internal API and allow > developers to either replace selected functionality via providing > callback modules or swap the HTTP servers easily would be a great plus. > > Concerning inets, what's really strange is that whenever we see some > benchmarks posted comparing performance of different HTTP > implementations, inets is never included there, though I would think, it > would make more sense in terms of showing how a custom implementation > compares to what's included in OTP. Does it have to do something with > what Lo?c referred to as inets being well hidden in OTP? I doubt, but > if not that, then what? There are not even benchmarks comparing yaws to > inets, and despite the fact that both offer a rich set of functionality > make one wonder was was the idea of writing yaws in the first place > given existence of inets' httpd. > > Diversity is good, but at some point it creates a burden for the user > who needs to make a selection of an HTTP server for his project. > Instead of solving the application-related problem, now the task becomes > studying several implementations to see which one offers a feature-set > best suited for the problem being solved, and the answer to that > question is not always straightforward. > > Perhaps a good initiative from the OTP team would be to spec out the > open architecture of an HTTP server and welcome feedback from authors of > current http libraries, so that the successor of inets would be flexible > enough to accommodate building light-weight and heavy-weight HTTP > servers based on the supported code base included in OTP? This would > also open a possibility of diversity for library writers > replacing/enhancing parts to suit their needs. > > Serge > > On 10/1/2012 3:52 PM, Lo?c Hoguin wrote: >> On 10/01/2012 08:11 PM, Kenneth Lundin wrote: >>> > * OTP has a long release process. Many companies need their fixes to >>> be included quickly to avoid maintaining huge branches for their >>> changes. They generally agree in return to keep track of changes that >>> might happen during the product's development. >>> I think we are pretty fast with service releases about every 3rd month >>> and updates on the maint branch in git between that. >> >> But doesn't that imply upgrading the whole of OTP? If it does, that >> means more testing and more chances for new issues. If you can build a >> release upgrade with one app from one OTP version and the rest from a >> previous version, I'd like to know how. >> >>> > Note that while not everyone uses inets, most servers do use >>> erlang:decode_packet/3 which does most of the job. Cowboy doesn't >>> anymore, because it needed backward incompatible changes to >>> erlang:decode_packet/3. I didn't ask your thoughts, simply because the >>> experiment I did hinted that doing the parsing in full Erlang was a >>> better idea: this change results in improved HiPE optimizations and >>> better reduction management which results in better latency under load. >>> >>> Interesting, would be good if you could give us more info on this. The >>> http parsing in >>> decode_packet is there to improve performance. If it is not ( without >>> using HiPE) we should consider removing it! >> >> Between now and then binaries appeared, got performance improvements, >> and considering Cowboy is full binary (no conversion to atom for methods >> or header names or anything) and only receives requests, not responses, >> that means it can skip many steps that decode_packet/3 can't. >> >> Binaries also mean you can parse the request line and all the headers >> using a single match context which seems to be very fast (good job!) and >> I'm guessing is the main reason for improved performance under HiPE (I'm >> about to start reading the HiPE papers to understand that better). >> >>> >> In Erlang/OTP we have keep strict compatibility with many of our APIs >>> >> which is both a strength and sometimes a burden (when we are not >>> happy >>> >> with the old API) but there are mostly ways around that if there is >>> >> strong case for a change. >>> > >>> > >>> > It's a big strength for people looking for stability and a big burden >>> for people looking for reactivity. Can't really blame the second group >>> for finding solutions to their problems. >>> >>> I am not blaming people >> >> I used a word too strong there. :) >> >>> for implementing their own solution, it is good >>> to have a lot to choose from. But I still think the stability is very >>> important for the majority of users making systems with a longer life >>> cycle than a year. >>> Our experience is that we have to be very compatible in order to >>> convince our >>> users (often big projects) to step up to our latest release. >>> By this we can also stop supporting older versions. >> >> Yes and I wouldn't want that to stop, that's a strong point. In fact >> having both ways is great, it means you can make sure the important >> parts of your system are stable while the more bleeding-edge parts are >> kept, well, bleeding-edge. >> >>> >> I think it is of great value to have a relative complete >>> functionality >>> >> in Erlang/OTP so that >>> >> the user can implement basic functionality without having to search >>> >> for separate implementations of, with unknown quality and >>> completeness >>> >> at least for a beginner. >>> > >>> > >>> > I would prefer Erlang/OTP to focus on the core features and leave the >>> satellite features to third parties. OTP has an HTTP server but doesn't >>> have proper Unicode support. OTP has OpenGL but doesn't have proper >>> timezones support. These 2 examples missing from OTP are some of many >>> things that are too complex for most people to implement, and that would >>> not benefit from having alternatives. >>> >>> What is missing when it comes to timezones? Has it been mentioned in >>> another thread? >>> I think we rely on the OS configuration here. >> >> Date/time conversion from one timezone to another. For example if your >> data is saved with UTC time but the user is in Paris you want to convert >> it to Europe/Paris. >> >> Perhaps it exists, but I haven't found it. >> >> I think there's a lot of work to be done with regards to localization >> and internationalization support, but I haven't had enough time to think >> about it in details yet. Unicode (not just Unicode though, but a native >> string type abstracting encodings, EEP coming in a few weeks), >> timezones, graphical drawing libraries are pretty much what I find lacks >> the most for the web side of things. Graphical drawing libraries are >> fine outside of OTP for a while (until there's one that can do near >> everything and do it well) though. >> >> >> >> Thanks for the info on package management and bug tracker. Hope we'll be >> involved in the process of creating the former. And thanks for your nice >> reply. :) >> > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Wed Oct 3 10:04:30 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Wed, 3 Oct 2012 10:04:30 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <0cc53c319c28f5f088a4039eb8a3daa8.squirrel@oinksoft.com> References: <0cc53c319c28f5f088a4039eb8a3daa8.squirrel@oinksoft.com> Message-ID: <2828D3DC-ACAA-4349-8175-69DCDC2B1E49@erlang-solutions.com> On Oct 2, 2012, at 4:18 PM, "Thomas Allen" wrote: > > I couldn't disagree with that much more ... npm/Node.js is cluttered with > packages of dubious quality that are not maintained in the slightest, and > while this "Wild West" they prefer may seem exciting, I do not think it > bodes well for the long-term stability of projects that depend on these > packages. > Two ideas spring to mind immediately. First, one should definitely not make the package repository a dump. You want to make package rules. Does it have documentation? Does it have a test suite? Can the test suite be run automatically? Semantic versioning? And so on. Is the package experimental, does it have real world uses? Furthermore, we could definitely adopt the ideas from the haskell-platform: A set of well-defined and stable packages are made into a basis library. Packages from the dump "graduate". Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From yrashk@REDACTED Wed Oct 3 10:11:08 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Wed, 3 Oct 2012 01:11:08 -0700 (PDT) Subject: [erlang-questions] Erlang http servers In-Reply-To: <2828D3DC-ACAA-4349-8175-69DCDC2B1E49@erlang-solutions.com> References: <0cc53c319c28f5f088a4039eb8a3daa8.squirrel@oinksoft.com> <2828D3DC-ACAA-4349-8175-69DCDC2B1E49@erlang-solutions.com> Message-ID: On Wednesday, October 3, 2012 1:04:45 AM UTC-7, Jesper Louis Andersen wrote: > > I couldn't disagree with that much more ... npm/Node.js is cluttered > with > > packages of dubious quality that are not maintained in the slightest, > and > > while this "Wild West" they prefer may seem exciting, I do not think it > > bodes well for the long-term stability of projects that depend on these > > packages. > > > > Two ideas spring to mind immediately. > > First, one should definitely not make the package repository a dump. You > want to make package rules. Does it have documentation? Does it have a test > suite? Can the test suite be run automatically? Semantic versioning? And so > on. Is the package experimental, does it have real world uses? > > Furthermore, we could definitely adopt the ideas from the > haskell-platform: A set of well-defined and stable packages are made into a > basis library. Packages from the dump "graduate". > > I believe in giving people simple and flexible tools and let them build the rules later on. This is basically why, after doing my work on Agner, I recently launched EXPM (http://expm.co, http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/) that is much simpler and much more flexible. EXPM allows to host your own repos very trivially, therefore allowing to build any combinations, including the one suggested by you. So far I am just running a "dump" index at expm.co to bootstrap this system (after all, publishing got magnitudes easier with expm comparing to Agner), but I can very much see how we can eventually add another repository (say, stable.expm.co) that will only "host" graduated packages. Yurii. -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Wed Oct 3 10:18:29 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Wed, 3 Oct 2012 09:18:29 +0100 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <0cc53c319c28f5f088a4039eb8a3daa8.squirrel@oinksoft.com> <2828D3DC-ACAA-4349-8175-69DCDC2B1E49@erlang-solutions.com> Message-ID: <20121003081816.GA5134@precise> On Wed, Oct 03, 2012 at 01:11:08AM -0700, Yurii Rashkovskii wrote: > I believe in giving people simple and flexible tools and let them build the > rules later on. This is basically why, after doing my work on Agner, I > recently launched EXPM (http://expm.co, > http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/) that is much > simpler and much more flexible. EXPM allows to host your own repos very > trivially, therefore allowing to build any combinations, including the one > suggested by you. ... And why are you announcing this in a thread about "Erlang http servers"? Please, make a proper announcement. > So far I am just running a "dump" index at expm.co to bootstrap this system > (after all, publishing got magnitudes easier with expm comparing to Agner), > but I can very much see how we can eventually add another repository (say, > stable.expm.co) that will only "host" graduated packages. From first sight it looks like a clone of PyPi (which, IMHO, is great). Let's see how it goes. Motiejus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From yrashk@REDACTED Wed Oct 3 10:24:04 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Wed, 3 Oct 2012 01:24:04 -0700 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <0cc53c319c28f5f088a4039eb8a3daa8.squirrel@oinksoft.com> <2828D3DC-ACAA-4349-8175-69DCDC2B1E49@erlang-solutions.com> <20121003081816.GA5134@precise> Message-ID: Motiejus, >>> I believe in giving people simple and flexible tools and let them build the >>> rules later on. This is basically why, after doing my work on Agner, I >>> recently launched EXPM (http://expm.co, >>> http://rashkovskii.com/2012/10/01/expm-or-meet-agner-2/) that is much >>> simpler and much more flexible. EXPM allows to host your own repos very >>> trivially, therefore allowing to build any combinations, including the one >>> suggested by you. >> >> ... And why are you announcing this in a thread about "Erlang http >> servers"? Please, make a proper announcement. > I will, I am just fixing things here and there to make sure it doesn't have too many embarrassingly stupid bugs when I make a "proper" announcement. But I do want to involve early adopters who like playing with new stuff (that's why I started talking about it, after a long period of silence on "Agner 2" project) Yurii From ingela.andin@REDACTED Wed Oct 3 22:00:02 2012 From: ingela.andin@REDACTED (Ingela Andin) Date: Wed, 3 Oct 2012 22:00:02 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: Hi! I feel compelled to contribute on the topic of the hidden inets application. Once a upon a time someone in Ericsson wrote an http server that was very much inspired by Apache, which probably seemed sensible at the time but it does not feel like the erlang way of doing things. It was almost not document but used extensively by Ericsson projects, hence very hard to get rid of. Years later several improvements have been made and inets httpd server can be run in a much more erlangish way without Apache config files, even though we still have to support them. The documentation is much improved but still quite sparse. And now I think we come to the negative circle. A lot of pople seem to think HTTP is easy and roll their own and the Ericsson projects are mostly content and do not have so many new requirements. As you know HTTP is not Ericsson core business and most open source users used something else and hence inets have not been given the highest priority. And there has been hardly no user contributions. Most efforts to improve inets has been to the HTTP client as it is more used. Lesson to be learnt it is worth doing things properly from the start because other pople may have to live with your code and bad naming for years and years to come. Regards Ingela Erlang/OTP team - Ericsson AB 2012/10/3 Michael Turner : > I recently received a copy of Building Web Applications with Erlang, > which is almost entirely about yaws. The discussion of inets on this > thread spurred me to look in the index under "i" -- except, oops, the > book has no index. It does have an appendix about other webservers and > frameworks in Erlang -- but inets is not listed there. > > "Well-hidden" indeed. > > -michael turner > > > On Wed, Oct 3, 2012 at 12:52 AM, Serge Aleynikov wrote: >> I agree with Joe that having a design of the HTTP server architecture >> included in OTP that would offer standardized internal API and allow >> developers to either replace selected functionality via providing >> callback modules or swap the HTTP servers easily would be a great plus. >> >> Concerning inets, what's really strange is that whenever we see some >> benchmarks posted comparing performance of different HTTP >> implementations, inets is never included there, though I would think, it >> would make more sense in terms of showing how a custom implementation >> compares to what's included in OTP. Does it have to do something with >> what Lo?c referred to as inets being well hidden in OTP? I doubt, but >> if not that, then what? There are not even benchmarks comparing yaws to >> inets, and despite the fact that both offer a rich set of functionality >> make one wonder was was the idea of writing yaws in the first place >> given existence of inets' httpd. >> >> Diversity is good, but at some point it creates a burden for the user >> who needs to make a selection of an HTTP server for his project. >> Instead of solving the application-related problem, now the task becomes >> studying several implementations to see which one offers a feature-set >> best suited for the problem being solved, and the answer to that >> question is not always straightforward. >> >> Perhaps a good initiative from the OTP team would be to spec out the >> open architecture of an HTTP server and welcome feedback from authors of >> current http libraries, so that the successor of inets would be flexible >> enough to accommodate building light-weight and heavy-weight HTTP >> servers based on the supported code base included in OTP? This would >> also open a possibility of diversity for library writers >> replacing/enhancing parts to suit their needs. >> >> Serge >> >> On 10/1/2012 3:52 PM, Lo?c Hoguin wrote: >>> On 10/01/2012 08:11 PM, Kenneth Lundin wrote: >>>> > * OTP has a long release process. Many companies need their fixes to >>>> be included quickly to avoid maintaining huge branches for their >>>> changes. They generally agree in return to keep track of changes that >>>> might happen during the product's development. >>>> I think we are pretty fast with service releases about every 3rd month >>>> and updates on the maint branch in git between that. >>> >>> But doesn't that imply upgrading the whole of OTP? If it does, that >>> means more testing and more chances for new issues. If you can build a >>> release upgrade with one app from one OTP version and the rest from a >>> previous version, I'd like to know how. >>> >>>> > Note that while not everyone uses inets, most servers do use >>>> erlang:decode_packet/3 which does most of the job. Cowboy doesn't >>>> anymore, because it needed backward incompatible changes to >>>> erlang:decode_packet/3. I didn't ask your thoughts, simply because the >>>> experiment I did hinted that doing the parsing in full Erlang was a >>>> better idea: this change results in improved HiPE optimizations and >>>> better reduction management which results in better latency under load. >>>> >>>> Interesting, would be good if you could give us more info on this. The >>>> http parsing in >>>> decode_packet is there to improve performance. If it is not ( without >>>> using HiPE) we should consider removing it! >>> >>> Between now and then binaries appeared, got performance improvements, >>> and considering Cowboy is full binary (no conversion to atom for methods >>> or header names or anything) and only receives requests, not responses, >>> that means it can skip many steps that decode_packet/3 can't. >>> >>> Binaries also mean you can parse the request line and all the headers >>> using a single match context which seems to be very fast (good job!) and >>> I'm guessing is the main reason for improved performance under HiPE (I'm >>> about to start reading the HiPE papers to understand that better). >>> >>>> >> In Erlang/OTP we have keep strict compatibility with many of our APIs >>>> >> which is both a strength and sometimes a burden (when we are not >>>> happy >>>> >> with the old API) but there are mostly ways around that if there is >>>> >> strong case for a change. >>>> > >>>> > >>>> > It's a big strength for people looking for stability and a big burden >>>> for people looking for reactivity. Can't really blame the second group >>>> for finding solutions to their problems. >>>> >>>> I am not blaming people >>> >>> I used a word too strong there. :) >>> >>>> for implementing their own solution, it is good >>>> to have a lot to choose from. But I still think the stability is very >>>> important for the majority of users making systems with a longer life >>>> cycle than a year. >>>> Our experience is that we have to be very compatible in order to >>>> convince our >>>> users (often big projects) to step up to our latest release. >>>> By this we can also stop supporting older versions. >>> >>> Yes and I wouldn't want that to stop, that's a strong point. In fact >>> having both ways is great, it means you can make sure the important >>> parts of your system are stable while the more bleeding-edge parts are >>> kept, well, bleeding-edge. >>> >>>> >> I think it is of great value to have a relative complete >>>> functionality >>>> >> in Erlang/OTP so that >>>> >> the user can implement basic functionality without having to search >>>> >> for separate implementations of, with unknown quality and >>>> completeness >>>> >> at least for a beginner. >>>> > >>>> > >>>> > I would prefer Erlang/OTP to focus on the core features and leave the >>>> satellite features to third parties. OTP has an HTTP server but doesn't >>>> have proper Unicode support. OTP has OpenGL but doesn't have proper >>>> timezones support. These 2 examples missing from OTP are some of many >>>> things that are too complex for most people to implement, and that would >>>> not benefit from having alternatives. >>>> >>>> What is missing when it comes to timezones? Has it been mentioned in >>>> another thread? >>>> I think we rely on the OS configuration here. >>> >>> Date/time conversion from one timezone to another. For example if your >>> data is saved with UTC time but the user is in Paris you want to convert >>> it to Europe/Paris. >>> >>> Perhaps it exists, but I haven't found it. >>> >>> I think there's a lot of work to be done with regards to localization >>> and internationalization support, but I haven't had enough time to think >>> about it in details yet. Unicode (not just Unicode though, but a native >>> string type abstracting encodings, EEP coming in a few weeks), >>> timezones, graphical drawing libraries are pretty much what I find lacks >>> the most for the web side of things. Graphical drawing libraries are >>> fine outside of OTP for a while (until there's one that can do near >>> everything and do it well) though. >>> >>> >>> >>> Thanks for the info on package management and bug tracker. Hope we'll be >>> involved in the process of creating the former. And thanks for your nice >>> reply. :) >>> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ulf@REDACTED Wed Oct 3 22:23:17 2012 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 3 Oct 2012 22:23:17 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: <94F48AEB-A95F-4809-A904-C5A15C860C7E@feuerlabs.com> As one of the first users of the Inets web server, I would like to point out that, at the time, making it Apache-compatible made perfect sense. It was by no means obvious that it was the best choice to use an Erlang-based web server in a telecom-class product, and Apache was a strong contender. Of course, it would have been nice to have been able to put a lot of effort into the web server already from the start, but it was by no means a priority, as we had much bigger fish to fry with OTP. ;-) The Inets HTTP server did it's job extremely well given our requirements. I cannot recall any issues - which is not to say that there weren't any, but that they were not serious enough to linger in my mind all this time. The main challenge for us on the HTTP front was to create a framework that allowed us to design and maintain over 100 different web pages, with dynamic content, access control and live SNMP hooks. *That* would have been something to contribute to the Open Source community. After a wholesale rewrite in (I think) 1999, it was actually pretty nice, esp. compared to what was needed and expected at the time. BR, Ulf W On 3 Oct 2012, at 22:00, Ingela Andin wrote: > Hi! > > I feel compelled to contribute on the topic of the hidden inets > application. Once a upon a time someone in Ericsson wrote an http > server that was very much inspired by Apache, which probably seemed > sensible at the time but it does not feel like the erlang way of doing > things. It was almost not document but used extensively by > Ericsson projects, hence very hard to get rid of. ... Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com From max.lapshin@REDACTED Wed Oct 3 22:28:31 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Oct 2012 00:28:31 +0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: I think, it is a problem about HTTP, not about user contributions. HTTP is an outstanding protocol. It is not yet-another-silly-protocol like RTMP or LDAP. It is a basic protocol for lots of projects, starting today. For example, Dropbox. It could use rsync 10 years ago, but today it is using only HTTP. And it is a very complicated protocol that includes many modern extensions, like websockets, etc. This is why users require implementation which is very optimized and fresh. It is really hard to write such a basic code of HTTP server, that will fit all users requirements for next 3 years. To understand it better, lets try to think: why there is no SIP + RTP implementation in OTP? Why telecom platform doesn't include main telecom protocol? I think, it is because it is almost impossible to write such a SIP implementation, that will fit all needs. From erlang@REDACTED Wed Oct 3 22:43:21 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 3 Oct 2012 22:43:21 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: On Wed, Oct 3, 2012 at 10:28 PM, Max Lapshin wrote: > I think, it is a problem about HTTP, not about user contributions. > > HTTP is an outstanding protocol. It is not yet-another-silly-protocol > like RTMP or LDAP. > It is a basic protocol for lots of projects, starting today. For > example, Dropbox. It could use rsync 10 years ago, > but today it is using only HTTP. > > > And it is a very complicated protocol that includes many modern > extensions, like websockets, etc. > > This is why users require implementation which is very optimized and > fresh. It is really hard to write such a basic code of HTTP server, > that will fit all users requirements for next 3 years. > > > To understand it better, lets try to think: why there is no SIP + RTP > implementation in OTP? Why telecom platform doesn't include main > telecom protocol? Possibly for the same reason that apple doesn't release the source code for OS-X in X-code :-) /J > I think, it is because it is almost impossible to > write such a SIP implementation, that will fit all needs. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From max.lapshin@REDACTED Wed Oct 3 22:47:37 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Oct 2012 00:47:37 +0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: On Thu, Oct 4, 2012 at 12:43 AM, Joe Armstrong wrote: > > Possibly for the same reason that apple doesn't release the > source code for OS-X in X-code :-) > Yes, I haven't even though about it =))) But indeed: what can be considered as a "rather generic" implementation of SIP? What be useful for all? From ulf@REDACTED Wed Oct 3 23:34:05 2012 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 3 Oct 2012 23:34:05 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: <77ACB514-257F-49F3-B306-B6E472CF0469@feuerlabs.com> I talked in Freiburg 2007 about a SIP stack that could have made a fine addition to OTP, but as Joe says... http://cufp.org/archive/2007/report.pdf (page 4, the actual slides are behind the ACM paywall) BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://www.feuerlabs.com 3 okt 2012 kl. 22:47 skrev Max Lapshin : > On Thu, Oct 4, 2012 at 12:43 AM, Joe Armstrong wrote: >> >> Possibly for the same reason that apple doesn't release the >> source code for OS-X in X-code :-) > > Yes, I haven't even though about it =))) > > But indeed: what can be considered as a "rather generic" implementation of SIP? > What be useful for all? > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Wed Oct 3 23:48:03 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 4 Oct 2012 01:48:03 +0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <77ACB514-257F-49F3-B306-B6E472CF0469@feuerlabs.com> References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> <77ACB514-257F-49F3-B306-B6E472CF0469@feuerlabs.com> Message-ID: "A fortunate turn of events allowed us to revive the early SIP exper- iments, rewrite the software and experiment to find an optimal architecture" I consider that this is exactly what is happening now with http servers in erlang. Take a look at difference between yaws api, inets api and cowboy api. They are really different even in such deep things: * how many processes should serve http request? * fully async parsing of http or using recv to synchronously parse it * use return codes or explicit replying via reply module, etc. This is an experimenting and it is hard to experiment when you need to think about backward compatibility. From ok@REDACTED Thu Oct 4 00:22:42 2012 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 4 Oct 2012 11:22:42 +1300 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: <3789CAA5-157B-4B0B-B92F-87AD2B6AE95D@cs.otago.ac.nz> Part of the issue with inets is that at one's very first look at the documentation it looks as though nobody is bothering to maintain the documentation. Inets is a container for Internet clients and servers. Currently, an client and server, a TFPT client and server, ^^ s/an/a/ and a FTP client has been incorporated into Inets. ^^^ s/has/have/ The HTTP server and client is HTTP 1.1 compliant ^^ s/is/are/ as defined in 2616. ^ s//RFC / ... Each client and server in inets is viewed as service. ^s//a / At the last count, my Swedish vocabulary was a massive 20 words. I couldn't even document how to boil water in Swedish. So I'm not criticising anyone at Ericsson for writing this in almost-English. What I *am* saying is that practically every sentence on the first page of text has something about it that's like a poke in the eye to the point that for me it was less stressful to find something else than to read this manual. The text really needs to be revised by a native speaker of English who is also a good writer. Of course, getting that done professionally costs money, and I paid, oh, it must be at least 0 dollars for Erlang... Making the documentation sources and the tool chain available means that it's possible now to contribute to people who need HTTP (or FTP) services in Erlang by improving the inets documentation, or it will be when the bug I reported (about spaces dropping out) is fixed. From ingela.andin@REDACTED Thu Oct 4 09:56:01 2012 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 4 Oct 2012 09:56:01 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <5069B90C.7070504@ninenines.eu> <5069F495.7050907@ninenines.eu> <506B0DC5.6010809@aleynikov.org> Message-ID: Hi! 2012/10/3, Max Lapshin : > I think, it is a problem about HTTP, not about user contributions. No the problem is not that we do not get user contributions, but the lack of contributions is a symptom of the problem that the for very long the non existing documentation of httpd resulted in that people did not understand how inets webserver works and dismisses it, not because it does not work well, but because they do not understand how it works. I still think the documentation is lacking but it is relativly much better than before. Considering HTTP and all things surrounding it I would still expect there to be other webserves around. To compare there are people using the odbc-application and contributing to it, but there are also a lot of pople that think a more native API is what they need and roll thier own. Point is the application has a life outside of Ericsson (in this particulare case it might not still be here if it had not). > This is why users require implementation which is very optimized and > fresh. It is really hard to write such a basic code of HTTP server, > that will fit all users requirements for next 3 years Inets tries to be a small core that should be easy to extend and customize not a big bang webserver that comes with all the bells and whistles. The big problem with the fist version of inets was not the Apache approch but rather the lack of documentation. To make improvments, extensions etc to an application you need to understand how it is meant to work. So point taken documentation needs further improvments. Regards Ingela Erlang/OTP Team - Ericsson AB From zabrane3@REDACTED Thu Oct 4 10:23:32 2012 From: zabrane3@REDACTED (Zabrane Mickael) Date: Thu, 4 Oct 2012 10:23:32 +0200 Subject: [erlang-questions] Erlang on Google Nexus 7? Message-ID: <0EE13AE7-933D-4ABE-842C-6958DE5DF321@gmail.com> Heu guys, Got a new Google Nexus 7 (rooted). Is there any (recent) Erlang binaries for ARM? Regards, Zabrane From ali.sabil@REDACTED Thu Oct 4 14:33:27 2012 From: ali.sabil@REDACTED (Ali Sabil) Date: Thu, 4 Oct 2012 14:33:27 +0200 Subject: [erlang-questions] RB and remote shells Message-ID: Hi all, I have been using rb in a remote shell, and I am wondering why do I have to go through changing the group leader of the rb_server as described here: http://stackoverflow.com/questions/2355744/running-the-report-browser-rb-for-sasl-error-reports-whilst-in-a-remote-shell Are there any plans or idea on how to fix this behavior? Thanks, Ali From watson.timothy@REDACTED Thu Oct 4 15:09:31 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 4 Oct 2012 14:09:31 +0100 Subject: [erlang-questions] can a program launched with open_port({spawn, Cmd}, Options) remain running after the port closes? In-Reply-To: <5069B93C.4070507@aleynikov.org> References: <5066197B.5020008@aleynikov.org> <76CE4188-BC5E-4DA4-8016-39CD7CDC326D@gmail.com> <5069B93C.4070507@aleynikov.org> Message-ID: <7AF8216C-1171-4DBA-96D0-BED75B87E312@gmail.com> Thanks Serge, I'll take a look and let you know. On 1 Oct 2012, at 16:39, Serge Aleynikov wrote: > Thanks for your tip on the dynamic configuration of rebar. > > Below please find a version of erlexec that uses rebar: > > https://github.com/saleyn/erlexec > > Along with its documentation: > > http://saleyn.github.com/erlexec > > The old repository http://code.google.com/p/erlexec/ is deprecated. > > Please let me know if this builds on other OS's besides Linux. > > Regards, > > Serge > > On 9/29/2012 5:23 AM, Tim Watson wrote: >> I'm a heavy rebar user and have compiled ports with it before, so give me a shout if you need any assistance. I can also do testing on several unix platforms. >> >> On 28 Sep 2012, at 22:41, Serge Aleynikov wrote: >> >>> Unfortunately I don't have OSx to test this. I can't reproduce your >>> problem on Linux. Perhaps you could check the libtool version this way: >>> >>> $ grep "^macro_version" /usr/bin/libtool >>> macro_version=2.4 >>> >>> and if yours is different, try to upgrade it. >>> >>> In order to simplify the build process I'll try to convert erlexec to >>> use rebar some time next week, which will eliminate autotools-based >>> toolchain. Need to figure out with rebar how to test and optionally >>> enable conditional defines based on presence of special C headers such >>> as "sys/capability.h" when compiling a port program. >>> >>> On 9/28/2012 5:16 PM, Tim Watson wrote:> Well I do, but clearly not the >>> same version as you? >>>> >>>> t4@REDACTED:erlexec $ which libtool >>>> /usr/bin/libtool >>>> t4@REDACTED:erlexec $ ls -la /usr/bin/ | grep libtool >>>> -r-xr-xr-x 1 root wheel 149216 22 May 15:30 libtool >>>> lrwxr-xr-x 1 root wheel 7 22 May 15:30 ranlib -> libtool >>>> t4@REDACTED:erlexec $ libtool --version >>>> libtool: unknown option character `-' in: --version >>>> Usage: libtool -static [-] file [...] [-filelist listfile[,dirname]] >>> [-arch_only arch] [-sacLT] >>>> Usage: libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] >>> [-arch_only arch] [-o output] [-install_name name] >>> [-compatibility_version #] [-current_version #] [-seg1addr 0x#] >>> [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table >>> ] [-seg_addr_table_filename ] [-all_load] >>> [-noall_load] >>>> t4@REDACTED:erlexec $ >>>> >>>> Honestly the osx build tool chain drives me nuts sometimes. :/ >>>> >>>> On 28 Sep 2012, at 17:24, Serge Aleynikov wrote: >>>> >>>>> Is it possible that you don't have libtool installed (missing >>>>> AC_PROG_LIBTOOL)? >>>>> >>>>> I have the following on linux and everything builds fine: >>>>> $ autoconf -V >>>>> autoconf (GNU Autoconf) 2.68 >>>>> $ libtool --version >>>>> libtool (GNU libtool) 2.4 >>>>> $ ./build >>>>> $ make >>>>> $ make install >>>>> $ tree install >>>>> install >>>>> ??? exec-1.0 >>>>> ??? ebin >>>>> ? ??? exec.app >>>>> ? ??? exec_app.beam >>>>> ? ??? exec.beam >>>>> ??? include >>>>> ? ??? exec.hrl >>>>> ??? priv >>>>> ? ??? x86_64-unknown-linux-gnu >>>>> ? ??? bin >>>>> ? ??? exec-port >>>>> ??? src >>>>> ??? exec_app.erl >>>>> ??? exec.erl >>>>> >>>>> >>>>> On 9/28/2012 7:45 AM, Tim Watson wrote:> The project doesn't appear to >>>>> build cleanly on mac os lion: >>>>>> >>>>>> t4@REDACTED:erlexec $ ./configure >>>>>> configure: error: cannot find install-sh, install.sh, or shtool in >>>>> config "."/config >>>>>> t4@REDACTED:erlexec $ ./build >>>>>> Install dir: /usr/local/src/erlang/erlexec/install >>>>>> aclocal >>>>>> autoconf -i >>>>>> configure.ac:19: error: possibly undefined macro: AC_PROG_LIBTOOL >>>>>> If this token and others are legitimate, please use >>> m4_pattern_allow. >>>>>> See the Autoconf documentation. >>>>>> make: *** [configure] Error 1 >>>>>> t4@REDACTED:erlexec $ >>>>>> t4@REDACTED:erlexec $ autoconf -V >>>>>> autoconf (GNU Autoconf) 2.69 >>>>>> Copyright (C) 2012 Free Software Foundation, Inc. >>>>>> License GPLv3+/Autoconf: GNU GPL version 3 or later >>>>>> , >>>>> >>>>>> This is free software: you are free to change and redistribute it. >>>>>> There is NO WARRANTY, to the extent permitted by law. >>>>>> >>>>>> Written by David J. MacKenzie and Akim Demaille. >>>>>> t4@REDACTED:erlexec $ >>>>>> >>>>>> >>>>>> Is there a specific version of autotools required to build and compile >>>>> erlexec? >>>>>> >>>>>> On 27 Sep 2012, at 20:11, Tim Watson wrote: >>>>>> >>>>>>> On 27 Sep 2012, at 19:18, freza@REDACTED wrote: >>>>>>>> Haven't been following this thread closely, but it seems erlexec >>> wasn't >>>>>>>> mentioned yet: >>>>>>>> >>>>>>>> http://code.google.com/p/erlexec/ >>>>>>> >>>>>>> Doesn't appear very well maintained. I might pick it apart and >>>>> rewrite it when native processes get released - it is a cool idea. >>>>> Anyway, for the time being it's not a great help as I'm writing a >>>>> framework to help set up and tear down clusters for distributed testing >>>>> (based largely on common test) so I want to minimise the number of >>>>> dependencies as much as possible. Besides, I'm more interested in >>>>> working on the remote/ssh runner+monitor next, rather than spending >>>>> another age fiddling around with the script runner. >>>>>>> >>>>>>> But yes, erlexec does look interesting and has been on my radar since >>>>> it popped up on this list a few months back, discussing the desire to do >>>>> proper signal handling (such as sending SIGTERM, SIGHUP, etc to the >>>>> external program) without resorting to os:cmd("kill -" ++ Code ++ " " ++ >>>>> OsPid). Again, this is something that linked-in drivers *can* deal with >>>>> to a limited extent, and therefore I suspect that native processes will >>>>> make for a very nice foundation on which to build an alternative to >>>>> open_port/2. >>>>>>> >>>>>>> Also, IIRC there was a cool command line testing framework open >>>>> sourced in the last year which was announced on this list. I can't >>>>> remember what it was called, but it has a port driver that handles the >>>>> the external comms and so on - looked very nice, although that's not >>>>> what I'm trying to do/be as my focus is on getting things in a >>>>> consistent state and monitoring to make sure they stay that way through >>>>> the test run, then tearing down nicely before the next phase kicks off. >>>>>>> >>>>>>> Cheers, >>>>>>> Tim >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dmercer@REDACTED Thu Oct 4 15:18:04 2012 From: dmercer@REDACTED (David Mercer) Date: Thu, 4 Oct 2012 08:18:04 -0500 Subject: [erlang-questions] Erlang http servers In-Reply-To: <5068B9B6.6070800@aleynikov.org> References: <5068B9B6.6070800@aleynikov.org> Message-ID: <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> Can someone please summarize this thread for me? If I am going to run a web server in Erlang, which one should I use? Cheers, DBM > -----Original Message----- > From: erlang-questions-bounces@REDACTED [mailto:erlang-questions- > bounces@REDACTED] On Behalf Of Serge Aleynikov > Sent: Sunday, September 30, 2012 16:29 > To: erlang-questions@REDACTED > Subject: [erlang-questions] Erlang http servers > > Haven't had a need for an HTTP server in Erlang for a while since I last used > inets-based http server five years back, and now examining what exists in > public domain. > > The results are quite surprising - everyone who seems comfortable coding in > Erlang, feels compelled to implement an HTTP server. Previously we had > only one contender to inets - yaws. Now the landscape is quite a mouthful - > inets, yaws, cowboy, musultin, mochiweb, etc. As Steve Vinoski rightfully > points out (*) this doesn't help to pull the blanket in all directions. From > personal experience with inets, I recall that once I figured out how to use it, it > happened to be a very helpful and powerful HTTP library for building an > application server. So why don't all the authors of those wonderful > alternative libraries just patch the inets to extend it with something they feel > missing so that we only have one good and documented library to use as the > common denominator? > > Serge > > > (*) http://steve.vinoski.net/blog/2011/05/09/erlang-web-server- > benchmarking > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From thomas@REDACTED Thu Oct 4 15:24:58 2012 From: thomas@REDACTED (thomas@REDACTED) Date: Thu, 4 Oct 2012 09:24:58 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> Message-ID: <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> > Can someone please summarize this thread for me? If I am going to run a > web server in Erlang, which one should I use? Right now, Cowboy seems to be the favorite. The fellow who wrote MochiWeb has said as much in past threads (that he would write MochiWeb like Cowboy if he were to start today). The author of the misultin project shut it down so that new development efforts could be focused on Cowboy and tools that integrate with Cowboy. WebMachine is still popular for writing RESTful web services, but note that Cowboy provides this too. Apache users who don't want to proxy to Cowboy probably want to use Yaws. That is my take. Thomas Allen From yrashk@REDACTED Thu Oct 4 22:45:21 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Thu, 4 Oct 2012 13:45:21 -0700 (PDT) Subject: [erlang-questions] RB and remote shells In-Reply-To: References: Message-ID: Ali, Because remote node's rb_sever is started using that node's supervision tree thus inheriting a group leader on that node. When it prints stuff out, it gets routed to that remote group leader as opposed to your local group leader that can print stuff out on your local terminal. Hope this makes sense. On Thursday, October 4, 2012 5:33:58 AM UTC-7, Ali Sabil wrote: > > Hi all, > > I have been using rb in a remote shell, and I am wondering why do I > have to go through changing the group leader of the rb_server as > described here: > > http://stackoverflow.com/questions/2355744/running-the-report-browser-rb-for-sasl-error-reports-whilst-in-a-remote-shell > > Are there any plans or idea on how to fix this behavior? > > Thanks, > Ali > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Fri Oct 5 06:01:54 2012 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Thu, 04 Oct 2012 23:01:54 -0500 Subject: [erlang-questions] +swt very_low doesn't seem to avoid schedulers getting "stuck" Message-ID: <91994.1349409714@snookles.snookles.com> Hi, all. According to my private mailing list archive, there hasn't been much mention of erl's "+swt" flag since about April 2012. I just witnessed a case of where using "+swt very_low" with Riak 1.2.1rc2, using Erlang/OTP R15B01, get "stuck" where CPU consumption was only 200% on an 8 core AWS instance. The other nodes in that Riak cluster were running at over 600% CPU utilization (on average). When I ran this: {io:format("before..."), erlang:system_flag(schedulers_online, 1), timer:sleep(1000), erlang:system_flag(schedulers_online, 8), io:format("after\n")}. ... then average CPU utilization on that node immediately shot up from 200% to about 760%. I'd heard a rumor that "+swt very_low" was supposed to avoid whatever weird scheduler problem/bug that caused some schedulers to appear as if they weren't active. But I was on site today and witnessed this first-hand and verified that the "+swt very_low" flag was indeed being used. I'm not certain what exact Linux distribution and kernel was used. I'll ask the customer to send me that info so I can forward it to the list. Has anyone else seen this behavior? Unlike Knut Nesheim's report on this list back in February 2012, Riak does not use the halfword emulator. We are using some NIFs, but this customer isn't using the most evil one, the Riak eleveldb NIF library. Instead, they're using the Bitcask backend (which has a NIF component but isn't as evil as eleveldb's NIF) and the merge_index backend (which is pure Erlang). -Scott From wwwxy314@REDACTED Fri Oct 5 07:04:25 2012 From: wwwxy314@REDACTED (www) Date: Fri, 5 Oct 2012 13:04:25 +0800 (CST) Subject: [erlang-questions] help--node core down Message-ID: <4bff2536.1e6c.13a2f4f6e33.Coremail.wwwxy314@163.com> Hi all: Recently,node core down every 3-4 days(centos6.2 + R15B02). I got this error Core was generated by `/usr/local/lib/erlang/erts-5.9.2/bin/beam.smp -P 500000 -K true -- -root /usr/l'. Program terminated with signal 11, Segmentation fault. #0 0x00000000004523ac in bf_get_free_block () Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 openssl-1.0.0-20.el6.x86_64 zlib-1.2.3-27.el6.x86_64 (gdb) where #0 0x00000000004523ac in bf_get_free_block () #1 0x000000000044d968 in mbc_alloc () #2 0x000000000044e567 in do_erts_alcu_alloc () #3 0x000000000044e619 in erts_alcu_alloc_thr_pref () #4 0x00000000004fbff7 in erts_bs_append () #5 0x0000000000527d20 in process_main () #6 0x00000000004a791b in sched_thread_func () #7 0x000000000059f315 in thr_wrapper () #8 0x000000325ba077f1 in start_thread () from /lib64/libpthread.so.0 #9 0x000000325b6e570d in clone () from /lib64/libc.so.6 That's why....I was troubled by it. Thanks. Xia Yong -------------- next part -------------- An HTML attachment was scrubbed... URL: From ali.sabil@REDACTED Fri Oct 5 09:37:27 2012 From: ali.sabil@REDACTED (Ali Sabil) Date: Fri, 5 Oct 2012 09:37:27 +0200 Subject: [erlang-questions] RB and remote shells In-Reply-To: References: Message-ID: Thanks Yurii, I see the issue, my question was more about if it would make sense to change the behavior of rb to have more sensible defaults, for example by inheriting the group leader of the process starting it? On Thu, Oct 4, 2012 at 10:45 PM, Yurii Rashkovskii wrote: > Ali, > > Because remote node's rb_sever is started using that node's supervision tree > thus inheriting a group leader on that node. When it prints stuff out, it > gets routed to that remote group leader as opposed to your local group > leader that can print stuff out on your local terminal. > > Hope this makes sense. > > > On Thursday, October 4, 2012 5:33:58 AM UTC-7, Ali Sabil wrote: >> >> Hi all, >> >> I have been using rb in a remote shell, and I am wondering why do I >> have to go through changing the group leader of the rb_server as >> described here: >> >> http://stackoverflow.com/questions/2355744/running-the-report-browser-rb-for-sasl-error-reports-whilst-in-a-remote-shell >> >> Are there any plans or idea on how to fix this behavior? >> >> Thanks, >> Ali >> _______________________________________________ >> erlang-questions mailing list >> erlang-q...@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions From vinoski@REDACTED Fri Oct 5 13:53:32 2012 From: vinoski@REDACTED (Steve Vinoski) Date: Fri, 5 Oct 2012 07:53:32 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> Message-ID: On Thu, Oct 4, 2012 at 9:24 AM, wrote: >> Can someone please summarize this thread for me? If I am going to run a >> web server in Erlang, which one should I use? > > Right now, Cowboy seems to be the favorite. Favorite among whom? This is a very broad statement that isn't broadly true. There are many Yaws users out there, many Mochiweb users, and I bet there are more than a few who still use Misultin. IMO Cowboy is popular among folks who need a library-like HTTP server and who also like to be leading edge, but not everyone falls into that category, not even close. Also, there seems to be some ongoing rumor that Yaws can't handle library-like HTTP apps, but that is entirely untrue. > WebMachine is still popular for writing RESTful web services, but note > that Cowboy provides this too. This statement is slightly confusing. The phrase "Cowboy provides this" could be taken to mean "Cowboy provides Webmachine", which isn't true. So just to be clear, what you really meant was that Cowboy provides assistance for writing RESTful web services. > Apache users who don't want to proxy to Cowboy probably want to use Yaws. This is another unfair generalization. I've never been an Apache user, yet I've used Yaws for everything from regular websites to embedded systems and it's always given me just what I needed. At about 11 years old and still going strong, Yaws is popular among users who need a system they can count on to provide years of production stability, backwards compatibility, and consistent support, all while keeping up with the latest Web developments such as streaming, long polling, WebSocket, and Server-Sent Events. --steve From bengt.kleberg@REDACTED Fri Oct 5 14:45:58 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Fri, 5 Oct 2012 14:45:58 +0200 Subject: [erlang-questions] beginner: slave:start/3 and -pa in Args not working? Message-ID: <1349441158.4694.57.camel@seasc1027.dyn.rnd.as.sw.ericsson.se> Greetings, Does not a slave node get the -pa argument into the code path? After slave:start/3 with Args set to "-pa ./ebin" I have a remote node. Good. It has the following expected arguments: rpc:call('ise_pool_slave_cluster1@REDACTED', init, get_arguments, []). [{root,["/app/erlang/otp_R15B01/LMWP2/lib/erlang"]}, {progname,["erl"]}, {home,["/home/eleberg"]}, {noshell,[]}, {noinput,[]}, {noshell,[]}, {noinput,[]}, {master,["ise_pool_master_cluster1@REDACTED"]}, {name,["ise_pool_slave_cluster1@REDACTED"]}, {rsh,["ssh"]}, {pa,["./ebin"]}] But the code path is not as expected: S = rpc:call('ise_pool_slave_cluster1@REDACTED', code, get_path, []). lists:member("./ebin", S). false Could anybody come forth with suggestions for what to look at? ebngt From thomas@REDACTED Fri Oct 5 15:04:08 2012 From: thomas@REDACTED (Thomas Allen) Date: Fri, 5 Oct 2012 09:04:08 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> Message-ID: On Fri, October 5, 2012 7:53 am, Steve Vinoski wrote: > Favorite among whom? This is a very broad statement that isn't broadly > true. There are many Yaws users out there, many Mochiweb users, and I > bet there are more than a few who still use Misultin. I'm sorry, I made an unfair generalization. I should've been more specific: Following this mailing list and discussions in the Freenode channel for some time, it seems to me that the Cowboy project has a great deal of momentum. This is not to say that the other tools are somehow unsuitable or will all die out. I certainly didn't intend to slight Yaws, which is a perfectly stable, capable web server, as are the other projects. Perhaps I should have phrased my recommendation in terms of what the others are good for rather than comparing them to Cowboy: You can't go wrong with Yaws or MochiWeb which are proven, stable servers. WebMachine is notably well-suited for writing RESTful services and focuses on HTTP compliance. Misultin is a good product but is no longer actively developed or maintained by the original author, so it is a riskier choice. Thomas Allen From tristan.sloughter@REDACTED Fri Oct 5 16:01:44 2012 From: tristan.sloughter@REDACTED (Tristan Sloughter) Date: Fri, 5 Oct 2012 09:01:44 -0500 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> Message-ID: This thread is still going? How long till it is the longest thread ever on the mailing list? (I suspect those are in the 100s though actually, haha). But how has this not just ended with the understanding: a) Yaws for some, can be embedded or used like a stand alone web server and is known to be stable, fast and production ready b) Cowboy for http library in your app and you like bleeding edge and a smaller dependency -- and now even smaller in it self but requires ranch c) Ellis for those who don't care to support http d) Whatever makes you happy, maybe write your own! And in my opinion Cowboy REST is now a much better Webmachine than Webmachine, due to some changes around handling POST and a few other things. Tristan On Fri, Oct 5, 2012 at 8:04 AM, Thomas Allen wrote: > On Fri, October 5, 2012 7:53 am, Steve Vinoski wrote: > > Favorite among whom? This is a very broad statement that isn't broadly > > true. There are many Yaws users out there, many Mochiweb users, and I > > bet there are more than a few who still use Misultin. > > I'm sorry, I made an unfair generalization. I should've been more > specific: Following this mailing list and discussions in the Freenode > channel for some time, it seems to me that the Cowboy project has a great > deal of momentum. This is not to say that the other tools are somehow > unsuitable or will all die out. > > I certainly didn't intend to slight Yaws, which is a perfectly stable, > capable web server, as are the other projects. Perhaps I should have > phrased my recommendation in terms of what the others are good for rather > than comparing them to Cowboy: You can't go wrong with Yaws or MochiWeb > which are proven, stable servers. WebMachine is notably well-suited for > writing RESTful services and focuses on HTTP compliance. Misultin is a > good product but is no longer actively developed or maintained by the > original author, so it is a riskier choice. > > Thomas Allen > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Fri Oct 5 16:23:35 2012 From: desired.mta@REDACTED (=?UTF-8?Q?Motiejus_Jak=C5=A1tys?=) Date: Fri, 5 Oct 2012 15:23:35 +0100 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> Message-ID: On Fri, Oct 5, 2012 at 3:01 PM, Tristan Sloughter wrote: > This thread is still going? How long till it is the longest thread ever on > the mailing list? (I suspect those are in the 100s though actually, haha). Since you mentioned it. This thread is still quite far from the top ones with my email being the 41'st one: http://m.jakstys.lt/tech/2012/02/erlang-most-popular-subjects/ > But how has this not just ended with the understanding: > > a) Yaws for some, can be embedded or used like a stand alone web server and > is known to be stable, fast and production ready > b) Cowboy for http library in your app and you like bleeding edge and a > smaller dependency -- and now even smaller in it self but requires ranch > c) Ellis for those who don't care to support http > d) Whatever makes you happy, maybe write your own! > > And in my opinion Cowboy REST is now a much better Webmachine than > Webmachine, due to some changes around handling POST and a few other things. At least on 0.6.0 cowboy had unexpectedly horrible performance issues handling big POST requests, like discovered by my colleague[1]. Not sure if this is now fixed. So *always* measure. Always. [1]: https://github.com/yfyf/cowboy_bench -- Motiejus Jak?tys From essen@REDACTED Fri Oct 5 16:28:54 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 05 Oct 2012 16:28:54 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> Message-ID: <506EEEA6.6070205@ninenines.eu> On 10/05/2012 04:23 PM, Motiejus Jak?tys wrote: > At least on 0.6.0 cowboy had unexpectedly horrible performance issues > handling big POST requests, like discovered by my colleague[1]. Not > sure if this is now fixed. So *always* measure. Always. > > [1]: https://github.com/yfyf/cowboy_bench It didn't, it just didn't support Expect: 100-continue. It does in 0.6.1. If there's still something wrong, please open a ticket. Nobody's complaining at the moment. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From dmercer@REDACTED Fri Oct 5 16:36:51 2012 From: dmercer@REDACTED (David Mercer) Date: Fri, 5 Oct 2012 09:36:51 -0500 Subject: [erlang-questions] Erlang http servers In-Reply-To: <506EEEA6.6070205@ninenines.eu> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> <506EEEA6.6070205@ninenines.eu> Message-ID: <001801cda306$dca255d0$95e70170$@gmail.com> Does Cowboy work on Windows? I ask because I went to download it and give it a whirl, but the first line of the Quick Start section says to use Rebar, which does not work on Windows. Please advise. Thank-you. Cheers, DBM > -----Original Message----- > From: erlang-questions-bounces@REDACTED [mailto:erlang-questions- > bounces@REDACTED] On Behalf Of Lo?c Hoguin > Sent: Friday, October 05, 2012 09:29 > To: Motiejus Jak?tys > Cc: erlang-questions@REDACTED > Subject: Re: [erlang-questions] Erlang http servers > > On 10/05/2012 04:23 PM, Motiejus Jak?tys wrote: > > At least on 0.6.0 cowboy had unexpectedly horrible performance issues > > handling big POST requests, like discovered by my colleague[1]. Not > > sure if this is now fixed. So *always* measure. Always. > > > > [1]: https://github.com/yfyf/cowboy_bench > > It didn't, it just didn't support Expect: 100-continue. > > It does in 0.6.1. > > If there's still something wrong, please open a ticket. Nobody's complaining > at the moment. > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From essen@REDACTED Fri Oct 5 16:39:17 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Fri, 05 Oct 2012 16:39:17 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <001801cda306$dca255d0$95e70170$@gmail.com> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> <506EEEA6.6070205@ninenines.eu> <001801cda306$dca255d0$95e70170$@gmail.com> Message-ID: <506EF115.4000309@ninenines.eu> It hasn't been tested on Windows, but it should work, with possibly a few easily fixed bugs in cowboy_static (that's the only Cowboy code dealing with files). I have no idea how to build an Erlang project on Windows. Patches are welcome! On 10/05/2012 04:36 PM, David Mercer wrote: > Does Cowboy work on Windows? I ask because I went to download it and give it a whirl, but the first line of the Quick Start section says to use Rebar, which does not work on Windows. Please advise. Thank-you. > > Cheers, > > DBM > > >> -----Original Message----- >> From: erlang-questions-bounces@REDACTED [mailto:erlang-questions- >> bounces@REDACTED] On Behalf Of Lo?c Hoguin >> Sent: Friday, October 05, 2012 09:29 >> To: Motiejus Jak?tys >> Cc: erlang-questions@REDACTED >> Subject: Re: [erlang-questions] Erlang http servers >> >> On 10/05/2012 04:23 PM, Motiejus Jak?tys wrote: >>> At least on 0.6.0 cowboy had unexpectedly horrible performance issues >>> handling big POST requests, like discovered by my colleague[1]. Not >>> sure if this is now fixed. So *always* measure. Always. >>> >>> [1]: https://github.com/yfyf/cowboy_bench >> >> It didn't, it just didn't support Expect: 100-continue. >> >> It does in 0.6.1. >> >> If there's still something wrong, please open a ticket. Nobody's complaining >> at the moment. >> >> -- >> Lo?c Hoguin >> Erlang Cowboy >> Nine Nines >> http://ninenines.eu >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From tuncer.ayaz@REDACTED Fri Oct 5 16:54:58 2012 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Fri, 5 Oct 2012 16:54:58 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <001801cda306$dca255d0$95e70170$@gmail.com> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> <506EEEA6.6070205@ninenines.eu> <001801cda306$dca255d0$95e70170$@gmail.com> Message-ID: On Fri, Oct 5, 2012 at 4:36 PM, David Mercer wrote: > Does Cowboy work on Windows? I ask because I went to > download it and give it a whirl, but the first line of the Quick > Start section says to use Rebar, which does not work on Windows. > Please advise. Thank-you. rebar does work and is used by Erlang developers on Windows. What specifically didn't work for you? Did you use rebar.cmd? From dch@REDACTED Fri Oct 5 17:37:20 2012 From: dch@REDACTED (Dave Cottlehuber) Date: Fri, 5 Oct 2012 17:37:20 +0200 Subject: [erlang-questions] Erlang http servers In-Reply-To: <001801cda306$dca255d0$95e70170$@gmail.com> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> <506EEEA6.6070205@ninenines.eu> <001801cda306$dca255d0$95e70170$@gmail.com> Message-ID: On 5 October 2012 16:36, David Mercer wrote: > Does Cowboy work on Windows? I ask because I went to download it and give it a whirl, but the first line of the Quick Start section says to use Rebar, which does not work on Windows. Please advise. Thank-you. Just to confirm, rebar *does* work on Windows, even for compiling NIFs. Obviously the C code needs to support an appropriate compiler in the first place, but that's not rebar's fault. So far I have built via rebar snappy, ejson, jiffy, erica, and a whole plethora of other tools & projects, without needing a special unix shell, just erlang & rebar. Many thanks to Basho & Tuncer for making that possible. Lo?c, I'll give cowboy a whirl on Windows sometime next week & will send an issue or patch if reqd. A+ Dave From dmercer@REDACTED Fri Oct 5 18:47:06 2012 From: dmercer@REDACTED (David Mercer) Date: Fri, 5 Oct 2012 11:47:06 -0500 Subject: [erlang-questions] Erlang http servers In-Reply-To: References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> <506EEEA6.6070205@ninenines.eu> <001801cda306$dca255d0$95e70170$@gmail.com> Message-ID: <003c01cda319$0ea1d850$2be588f0$@gmail.com> I'll give it another whirl. Tried it about a year ago and it didn't work. Rebar, that is; haven?t tried Cowboy yet. I'll report back on my experiences so that I don't leave you hanging. Cheers, DBM > -----Original Message----- > From: Dave Cottlehuber [mailto:dch@REDACTED] > Sent: Friday, October 05, 2012 10:37 > To: David Mercer; Lo?c Hoguin; Motiejus Jak?tys; erlang- > questions@REDACTED > Subject: Re: [erlang-questions] Erlang http servers > > On 5 October 2012 16:36, David Mercer wrote: > > Does Cowboy work on Windows? I ask because I went to download it and > give it a whirl, but the first line of the Quick Start section says to use Rebar, > which does not work on Windows. Please advise. Thank-you. > > Just to confirm, rebar *does* work on Windows, even for compiling NIFs. > Obviously the C code needs to support an appropriate compiler in the first > place, but that's not rebar's fault. So far I have built via rebar snappy, ejson, > jiffy, erica, and a whole plethora of other tools & projects, without needing a > special unix shell, just erlang & rebar. > Many thanks to Basho & Tuncer for making that possible. > > Lo?c, I'll give cowboy a whirl on Windows sometime next week & will send an > issue or patch if reqd. > > A+ > Dave From jrosenblum@REDACTED Fri Oct 5 20:35:36 2012 From: jrosenblum@REDACTED (Jim) Date: Fri, 5 Oct 2012 14:35:36 -0400 Subject: [erlang-questions] Erlang http servers In-Reply-To: <003c01cda319$0ea1d850$2be588f0$@gmail.com> References: <5068B9B6.6070800@aleynikov.org> <00ee01cda232$b04fd9a0$10ef8ce0$@gmail.com> <405e4a98748f108b2b4a6ff2a66c5fc5.squirrel@oinksoft.com> <506EEEA6.6070205@ninenines.eu> <001801cda306$dca255d0$95e70170$@gmail.com> <003c01cda319$0ea1d850$2be588f0$@gmail.com> Message-ID: <11E1B8ED-9C06-4331-BA4D-7274D4704AE8@prodigy.net> I am currently using Cowboy, built with rebar, on Windows 7, 64Bit. I did not have to do anything special to build. And, except for issues of my own creation, have had no problems; although I have on,y used the most basic http_handler to date. Sent from my iPad On Oct 5, 2012, at 12:47 PM, "David Mercer" wrote: > I'll give it another whirl. Tried it about a year ago and it didn't work. Rebar, that is; haven?t tried Cowboy yet. > > I'll report back on my experiences so that I don't leave you hanging. > > Cheers, > > DBM > > >> -----Original Message----- >> From: Dave Cottlehuber [mailto:dch@REDACTED] >> Sent: Friday, October 05, 2012 10:37 >> To: David Mercer; Lo?c Hoguin; Motiejus Jak?tys; erlang- >> questions@REDACTED >> Subject: Re: [erlang-questions] Erlang http servers >> >> On 5 October 2012 16:36, David Mercer wrote: >>> Does Cowboy work on Windows? I ask because I went to download it and >> give it a whirl, but the first line of the Quick Start section says to use Rebar, >> which does not work on Windows. Please advise. Thank-you. >> >> Just to confirm, rebar *does* work on Windows, even for compiling NIFs. >> Obviously the C code needs to support an appropriate compiler in the first >> place, but that's not rebar's fault. So far I have built via rebar snappy, ejson, >> jiffy, erica, and a whole plethora of other tools & projects, without needing a >> special unix shell, just erlang & rebar. >> Many thanks to Basho & Tuncer for making that possible. >> >> Lo?c, I'll give cowboy a whirl on Windows sometime next week & will send an >> issue or patch if reqd. >> >> A+ >> Dave > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From kunthar@REDACTED Sat Oct 6 02:17:50 2012 From: kunthar@REDACTED (Kunthar) Date: Sat, 6 Oct 2012 03:17:50 +0300 Subject: [erlang-questions] While i am searching for newest greatest Erlang video Message-ID: Found these: https://www.youtube.com/watch?v=DhXHY851AcQ https://www.youtube.com/watch?v=PNL2MT0vovU https://www.youtube.com/watch?v=QJ1wen9vRoU Erlang coders can follow the (way) really now! -- BR, \|/ Kunthar From bsamuels453@REDACTED Sat Oct 6 05:43:49 2012 From: bsamuels453@REDACTED (Ben Samuels) Date: Fri, 5 Oct 2012 20:43:49 -0700 Subject: [erlang-questions] While i am searching for newest greatest Erlang video In-Reply-To: References: Message-ID: You're doing it wrong if you're not using this video to follow the way http://www.youtube.com/watch?v=uKfKtXYLG78 On Fri, Oct 5, 2012 at 5:17 PM, Kunthar wrote: > Found these: > https://www.youtube.com/watch?v=DhXHY851AcQ > https://www.youtube.com/watch?v=PNL2MT0vovU > https://www.youtube.com/watch?v=QJ1wen9vRoU > > Erlang coders can follow the (way) really now! > > -- > BR, > \|/ Kunthar > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush@REDACTED Sat Oct 6 07:04:01 2012 From: straightflush@REDACTED (AD) Date: Sat, 6 Oct 2012 01:04:01 -0400 Subject: [erlang-questions] While i am searching for newest greatest Erlang video In-Reply-To: References: Message-ID: that was amazing. On Fri, Oct 5, 2012 at 11:43 PM, Ben Samuels wrote: > You're doing it wrong if you're not using this video to follow the way > http://www.youtube.com/watch?v=uKfKtXYLG78 > > > On Fri, Oct 5, 2012 at 5:17 PM, Kunthar wrote: > >> Found these: >> https://www.youtube.com/watch?v=DhXHY851AcQ >> https://www.youtube.com/watch?v=PNL2MT0vovU >> https://www.youtube.com/watch?v=QJ1wen9vRoU >> >> Erlang coders can follow the (way) really now! >> >> -- >> BR, >> \|/ Kunthar >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.virding@REDACTED Sat Oct 6 18:01:11 2012 From: robert.virding@REDACTED (Robert Virding) Date: Sat, 06 Oct 2012 17:01:11 +0100 (BST) Subject: [erlang-questions] While i am searching for newest greatest Erlang video In-Reply-To: Message-ID: <09b45f8b-fdbe-46df-a085-c8318741ec80@knuth> These are perfect for warming up both mind and body before getting into a serious erlang programming session. Helps the mind focus. Robert ----- Original Message ----- > From: "Kunthar" > To: "Erlang" > Sent: Saturday, 6 October, 2012 2:17:50 AM > Subject: [erlang-questions] While i am searching for newest greatest Erlang video > > Found these: > https://www.youtube.com/watch?v=DhXHY851AcQ > https://www.youtube.com/watch?v=PNL2MT0vovU > https://www.youtube.com/watch?v=QJ1wen9vRoU > > Erlang coders can follow the (way) really now! > > -- > BR, > \|/ Kunthar > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From hasan.veldstra@REDACTED Sat Oct 6 18:11:55 2012 From: hasan.veldstra@REDACTED (HH Veldstra) Date: Sat, 6 Oct 2012 17:11:55 +0100 Subject: [erlang-questions] While i am searching for newest greatest Erlang video In-Reply-To: <09b45f8b-fdbe-46df-a085-c8318741ec80@knuth> References: <09b45f8b-fdbe-46df-a085-c8318741ec80@knuth> Message-ID: nothing can beat a surstr?mming sandwich to get in the zone for serious Erlang coding On Sat, Oct 6, 2012 at 5:01 PM, Robert Virding < robert.virding@REDACTED> wrote: > These are perfect for warming up both mind and body before getting into a > serious erlang programming session. Helps the mind focus. > > Robert > > ----- Original Message ----- > > From: "Kunthar" > > To: "Erlang" > > Sent: Saturday, 6 October, 2012 2:17:50 AM > > Subject: [erlang-questions] While i am searching for newest greatest > Erlang video > > > > Found these: > > https://www.youtube.com/watch?v=DhXHY851AcQ > > https://www.youtube.com/watch?v=PNL2MT0vovU > > https://www.youtube.com/watch?v=QJ1wen9vRoU > > > > Erlang coders can follow the (way) really now! > > > > -- > > BR, > > \|/ Kunthar > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmr@REDACTED Sun Oct 7 11:33:23 2012 From: tmr@REDACTED (Tomas Morstein) Date: Sun, 07 Oct 2012 11:33:23 +0200 (CEST) Subject: [erlang-questions] eprof bug? In-Reply-To: <44257a6f-aa49-4048-9353-9f3e4b011d6a@googlegroups.com> Message-ID: <59e89e29-8aee-4579-b170-86fdc62118ad@zimbra> > > I wanted to play with eprof a little, and found that it does not > > work > > correctly on my system and anytime when I try to start profiling, > > it ends up with SIGSEGV: > > https://gist.github.com/3805252 > > > > We tested it on multiple systems with the same version of OTP > > (R15B) > > built from source but the problem appears only on my laptop with > > Linux 2.6.38/x86_64. The configure parameters is what I don't > > remember > > at the moment :( > > > > Is there anybody aware of such an issue? > If memory serves, there was an issue with eprof (tracing) and HiPE. > Is there any chance you are running HiPE compiled libs, i.e. HiPE > compiled lists module? I am not able to prove it now but I think it's possible it was accidentally configured with native libs... > If you find the same issue in R15B02, let me know. It works fine on R15B02! Thanks, Tom From tmr@REDACTED Sun Oct 7 12:32:05 2012 From: tmr@REDACTED (Tomas Morstein) Date: Sun, 07 Oct 2012 12:32:05 +0200 (CEST) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: Hello, We wanted to upgrade our development environment to R15B02 and ended up with a problem of different inheritance behaviour: https://gist.github.com/3846164 The point is simple, let's imagine scenario: - standard module a.erl with several core functions (acts like abstract class) - parametrized module b.erl that extends a.erl to bring all the a's functionality (b has always a single parameter what is proplist) On older version of Erlang/OTP, this worked fine and we were able to b:new ([]) what gave us {b, []}. This does not work on R15B02 anymore :-( We don't want to make a.erl parametrized for many reasons, but I understand it makes some sense to bring a "base" (a) instance with the instance of b. The new behaviour still conforms to Richard Carlsson's paper, because our scenario was not presented there... The problem I see is that only the case of non-parametrized BASE "class" has changed in R15B02... Do we need to completely refactor our code? (add a custom new/1 to each b-level module, turn module 'a' into a parametrized one, etc.) How many changes are expected in future? Is it safe to believe that once we refactor the code, no future version will change something else? Should we think about an alternative of doing inheritance in our custom way otherwise? Thanks, Tom -- www.idea.cz | www.mumps.cz | www.openvms.cz | www.fooboo.org From gopienko@REDACTED Sun Oct 7 15:53:33 2012 From: gopienko@REDACTED (Andrew Gopienko) Date: Sun, 7 Oct 2012 20:53:33 +0700 Subject: [erlang-questions] erl_call problem Message-ID: We are using erl_call for nodes monitoring and sometimes erl_call failed with message 'erl_call: failed to connect to node node_01@REDACTED' Node at this time is alive and works as usually. We found that node at this time returns status 'salive' instead of expected 'sok'. ---------------------------------------------------------------------------------------------- ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT attempt to connect to node_01 ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: -> PORT2_REQ alive=node_01 ip=192.168.1.2 ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: <- PORT2_RESP result=0 (ok) ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: port=52658 ntype=77 proto=0 dist-high=5 dist-low=5 ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT connected to remote recv_status: Sun Oct 7 16:45:00 2012: <- len=6 recv_status: Sun Oct 7 16:45:00 2012: <- buf[0]=s (115) recv_status: Sun Oct 7 16:45:00 2012: <- buf[1]=a (97) recv_status: Sun Oct 7 16:45:00 2012: <- buf[2]=l (108) recv_status: Sun Oct 7 16:45:00 2012: <- buf[3]=i (105) recv_status: Sun Oct 7 16:45:00 2012: <- buf[4]=v (118) recv_status: Sun Oct 7 16:45:00 2012: <- buf[5]=e (101) ex_xconnect: Sun Oct 7 16:45:00 2012: RECV_STATUS failed ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT failed erl_call: failed to connect to node node_01@REDACTED, fd=-1, errno=0, erl_errno=5 ---------------------------------------------------------------------------------------------- In what cases the node returns 'salive' ? Andrew Gopienko -------------- next part -------------- An HTML attachment was scrubbed... URL: From desired.mta@REDACTED Sun Oct 7 18:35:44 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Sun, 7 Oct 2012 17:35:44 +0100 Subject: [erlang-questions] extend parser to allow single quotes in variable names (or: "let's confuse the text editors!") Message-ID: <20121007163429.GA4904@precise> Hi, In Haskell you can have variable names with single quotes. Imagine for a moment you are in a perfect world: oops() -> A = 1024, A' = math:log(A) / math:log(2), io:format("Wow! ~p~n", [A']). What would you think about this extension? Regards, Motiejus From jesper.louis.andersen@REDACTED Sun Oct 7 19:41:59 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 7 Oct 2012 19:41:59 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: Message-ID: <90870E01-32AA-4123-8588-4F5249FE83F6@erlang-solutions.com> On Oct 7, 2012, at 12:32 PM, Tomas Morstein wrote: > Hello, > > We wanted to upgrade our development environment to R15B02 > and ended up with a problem of different inheritance behaviour: > https://gist.github.com/3846164 > This is quite interesting from a perspective of parameterized modules in general. They are not part of the language standard, but they are kept around since people use it. Now, since it is not part of the language standard and an experimental feature, it is subject to change without warning. You just hit such a problem. Here is what may be wrong, one thing at a time: * The code you present really *is* undefined behaviour. It should never have been accepted in the first place and you shouldn't write code like that. We don't know since there is no "right" specification here. * The code works incorrectly, but the regression was not caught by the test suite. * Parametertized modules are experimental, you are on your own and must adapt to the change when it happens. * The code has a well-defined behaviour, but it was never made formal in the way the code generator works on these modules. And so on. I don't know what the correct stance is to take here - but perhaps either Richard or the OTP might know what is right to do. Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From ok@REDACTED Sun Oct 7 23:44:45 2012 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 8 Oct 2012 10:44:45 +1300 Subject: [erlang-questions] extend parser to allow single quotes in variable names (or: "let's confuse the text editors!") In-Reply-To: <20121007163429.GA4904@precise> References: <20121007163429.GA4904@precise> Message-ID: On 8/10/2012, at 5:35 AM, Motiejus Jak?tys wrote: > Hi, > > In Haskell you can have variable names with single quotes. Imagine for a > moment you are in a perfect world: > > oops() -> > A = 1024, > A' = math:log(A) / math:log(2), > io:format("Wow! ~p~n", [A']). > > What would you think about this extension? I would think it was pointless. Haskell Erlang x X x' X1 x'' X2 x''' X3 x'''' X4 x''''' X5 Spot the pattern? Haskell (or at least GHC) allows primes inside identifiers too, but nobody seems to take advantage of that. Consider f'x 'y'. Want to read _that_ fast? Consider f'x' y'. Want to tell them apart fast? SML also allows primes inside and at the end of identifiers. This is much less of a pain in SML because SML writes character literals as #"x" rather than 'x'. There is historically and in good typography a clearly visible distinction between the apostrophe and the superscript I II III IV V: (x' is x prime; x'' is x second; x''' is x third, and the next one is xIV, x fourth, not x''''). It's only the limitations of old typewriters and ASCII that lead us to confuse them. Superscript Roman numerals in Unicode identifiers, fine; using the apostrophe as the quote character for atoms and also in identifiers, not fine. From desired.mta@REDACTED Sun Oct 7 23:50:02 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Sun, 7 Oct 2012 22:50:02 +0100 Subject: [erlang-questions] extend parser to allow single quotes in variable names (or: "let's confuse the text editors!") In-Reply-To: References: <20121007163429.GA4904@precise> Message-ID: <20121007215002.GA7284@precise> On Mon, Oct 08, 2012 at 10:44:45AM +1300, Richard O'Keefe wrote: > > On 8/10/2012, at 5:35 AM, Motiejus Jak?tys wrote: > > > Hi, > > > > In Haskell you can have variable names with single quotes. Imagine for a > > moment you are in a perfect world: > > > > oops() -> > > A = 1024, > > A' = math:log(A) / math:log(2), > > io:format("Wow! ~p~n", [A']). > > > > What would you think about this extension? > > I would think it was pointless. > > Haskell Erlang > x X > x' X1 > x'' X2 > x''' X3 > x'''' X4 > x''''' X5 > > Spot the pattern? > > Haskell (or at least GHC) allows primes inside identifiers too, > but nobody seems to take advantage of that. > > Consider f'x 'y'. Want to read _that_ fast? > Consider f'x' y'. Want to tell them apart fast? Whoosh! This reasoning is just perfect. I never want to see code like this. Thanks. Motiejus From jozsef.berces@REDACTED Mon Oct 8 08:56:42 2012 From: jozsef.berces@REDACTED (=?iso-8859-1?Q?J=F3zsef_B=E9rces?=) Date: Mon, 8 Oct 2012 06:56:42 +0000 Subject: [erlang-questions] old module Message-ID: <7460EBDDCF52084A849D0F271CE059B8010B59@ESESSMB101.ericsson.se> Is there any way to tell if an old version of the module already exists? The return value of code:delete/1 tells that, but that is too late, I would like to know in advance that the module needs purge. Thanks, Jozsef -------------- next part -------------- An HTML attachment was scrubbed... URL: From gopienko@REDACTED Mon Oct 8 09:01:07 2012 From: gopienko@REDACTED (Andrew Gopienko) Date: Mon, 8 Oct 2012 14:01:07 +0700 Subject: [erlang-questions] erl_call problem Message-ID: We are using erl_call for nodes monitoring and sometimes erl_call failed with message 'erl_call: failed to connect to node node_01@REDACTED' Node at this time is alive and works as usually. We found that node at this time returns status 'salive' instead of expected 'sok'. ------------------------------ ---------------------------------------------------------------- ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT attempt to connect to node_01 ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: -> PORT2_REQ alive=node_01 ip=192.168.1.2 ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: <- PORT2_RESP result=0 (ok) ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: port=52658 ntype=77 proto=0 dist-high=5 dist-low=5 ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT connected to remote recv_status: Sun Oct 7 16:45:00 2012: <- len=6 recv_status: Sun Oct 7 16:45:00 2012: <- buf[0]=s (115) recv_status: Sun Oct 7 16:45:00 2012: <- buf[1]=a (97) recv_status: Sun Oct 7 16:45:00 2012: <- buf[2]=l (108) recv_status: Sun Oct 7 16:45:00 2012: <- buf[3]=i (105) recv_status: Sun Oct 7 16:45:00 2012: <- buf[4]=v (118) recv_status: Sun Oct 7 16:45:00 2012: <- buf[5]=e (101) ex_xconnect: Sun Oct 7 16:45:00 2012: RECV_STATUS failed ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT failed erl_call: failed to connect to node node_01@REDACTED, fd=-1, errno=0, erl_errno=5 ---------------------------------------------------------------------------------------------- In what cases the node returns 'salive' ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Mon Oct 8 09:09:10 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 8 Oct 2012 09:09:10 +0200 Subject: [erlang-questions] old module In-Reply-To: <7460EBDDCF52084A849D0F271CE059B8010B59@ESESSMB101.ericsson.se> References: <7460EBDDCF52084A849D0F271CE059B8010B59@ESESSMB101.ericsson.se> Message-ID: erlang:check_old_code(Module) -> boolean() Types: Module = atom() Returns true if the Module has old code, and false otherwise. (How did I find this? - > cd /lib/kernel/src > grep purge *.erl I found it around line 1401 of code_server.erl (it's also in the erlang manual page :-) ) /Joe On Mon, Oct 8, 2012 at 8:56 AM, J?zsef B?rces wrote: > Is there any way to tell if an old version of the module already exists? The > return value of code:delete/1 tells that, but that is too late, I would like > to know in advance that the module needs purge. > > Thanks, > Jozsef > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From erlang@REDACTED Mon Oct 8 12:06:54 2012 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 8 Oct 2012 12:06:54 +0200 Subject: [erlang-questions] erlang parser for javascript literals Message-ID: Does anybody have an Erlang parser for Javascript literals? I have some JSON terms - I have to write them like this: {"id" : 123, "author" : "erlang@REDACTED", "title" : "Some title", "type" : "html", "vsns":[{"vsn":1, "time":"201200908113012", "content":"

This is html

\n
a\nb
"}] } But I'd prefer to write them with far fewer quotes and with embedded line feeds like this: {id : 123, author : "erlang@REDACTED", title : "Some title", type : html, vsns:[{vsn:1, time:"201200908113012", content:"

This is html

a
b
"}] } Or do any of the erlang JSON parsers handle non quoted names in object literals? /Joe From robert.virding@REDACTED Mon Oct 8 12:10:25 2012 From: robert.virding@REDACTED (Robert Virding) Date: Mon, 08 Oct 2012 11:10:25 +0100 (BST) Subject: [erlang-questions] While i am searching for newest greatest Erlang video In-Reply-To: Message-ID: <1fdf857f-30b6-4263-af31-124243e8ba8b@knuth> I have have NEVER gotten into that serious Erlang coding, there are limits. Robert ----- Original Message ----- > From: "HH Veldstra" > To: "Robert Virding" > Cc: "Kunthar" , "Erlang" > > Sent: Saturday, 6 October, 2012 5:11:55 PM > Subject: Re: [erlang-questions] While i am searching for newest > greatest Erlang video > nothing can beat a surstr?mming sandwich to get in the zone for > serious Erlang coding > On Sat, Oct 6, 2012 at 5:01 PM, Robert Virding < > robert.virding@REDACTED > wrote: > > These are perfect for warming up both mind and body before getting > > into a serious erlang programming session. Helps the mind focus. > > > Robert > > > ----- Original Message ----- > > > > From: "Kunthar" < kunthar@REDACTED > > > > > To: "Erlang" < erlang-questions@REDACTED > > > > > Sent: Saturday, 6 October, 2012 2:17:50 AM > > > > Subject: [erlang-questions] While i am searching for newest > > > greatest Erlang video > > > > > > > > Found these: > > > > https://www.youtube.com/watch?v=DhXHY851AcQ > > > > https://www.youtube.com/watch?v=PNL2MT0vovU > > > > https://www.youtube.com/watch?v=QJ1wen9vRoU > > > > > > > > Erlang coders can follow the (way) really now! > > > > > > > > -- > > > > BR, > > > > \|/ Kunthar > > > > _______________________________________________ > > > > erlang-questions mailing list > > > > erlang-questions@REDACTED > > > > http://erlang.org/mailman/listinfo/erlang-questions > > > > > > > _______________________________________________ > > > erlang-questions mailing list > > > erlang-questions@REDACTED > > > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Mon Oct 8 12:41:41 2012 From: dmkolesnikov@REDACTED (dmitry kolesnikov) Date: Mon, 8 Oct 2012 13:41:41 +0300 Subject: [erlang-questions] erlang parser for javascript literals In-Reply-To: References: Message-ID: <-3184479837163322775@unknownmsgid> Hello Joe, Object members without quotes is against JSON RFC. If you skip quotes then this is not JSON any more. I am using jsx parser, one of the best. https://github.com/talentdeficit/jsx Best Regards, Dmitry >-|-|-*> On 8.10.2012, at 13.07, Joe Armstrong wrote: > Does anybody have an Erlang parser for Javascript literals? > > I have some JSON terms - I have to write them like this: > > {"id" : 123, > "author" : "erlang@REDACTED", > "title" : "Some title", > "type" : "html", > "vsns":[{"vsn":1, > "time":"201200908113012", > "content":"

This is html

\n
a\nb
"}] > } > > But I'd prefer to write them with far fewer quotes and with embedded > line feeds like this: > > {id : 123, > author : "erlang@REDACTED", > title : "Some title", > type : html, > vsns:[{vsn:1, > time:"201200908113012", > content:"

This is html

> 
a
> b
>
"}] > } > > Or do any of the erlang JSON parsers handle non quoted names in object literals? > > > /Joe > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From jesper.louis.andersen@REDACTED Mon Oct 8 13:21:55 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 8 Oct 2012 13:21:55 +0200 Subject: [erlang-questions] erlang parser for javascript literals In-Reply-To: <-3184479837163322775@unknownmsgid> References: <-3184479837163322775@unknownmsgid> Message-ID: On Oct 8, 2012, at 12:41 PM, dmitry kolesnikov wrote: > Hello Joe, > > Object members without quotes is against JSON RFC. If you skip quotes > then this is not JSON any more. > This is true, but it does not hold you back from choosing another representation format and then converting from that format into proper JSON later when you need to interact with other systems in a correct manner. Raw JSON is quite hard to write due to some of the rules in the dark corners of JS parsing :) Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From jose.valim@REDACTED Mon Oct 8 13:29:50 2012 From: jose.valim@REDACTED (=?ISO-8859-1?Q?Jos=E9_Valim?=) Date: Mon, 8 Oct 2012 13:29:50 +0200 Subject: [erlang-questions] erlang parser for javascript literals In-Reply-To: References: <-3184479837163322775@unknownmsgid> Message-ID: > This is true, but it does not hold you back from choosing another > representation format and then converting from that format into proper JSON > later when you need to interact with other systems in a correct manner. Raw > JSON is quite hard to write due to some of the rules in the dark corners of > JS parsing :) > If you don't want to parse strictly JSON, YAML is an option that will give you more flexibility than JSON (and the features you asked). There are some available Erlang wrappers for libyaml, but I haven't used them personally: https://github.com/goertzenator/yamler * Jos? Valim * * www.plataformatec.com.br Founder and Lead Developer * -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.romain@REDACTED Mon Oct 8 13:41:36 2012 From: christophe.romain@REDACTED (Christophe Romain) Date: Mon, 8 Oct 2012 13:41:36 +0200 Subject: [erlang-questions] Erlang on Google Nexus 7? In-Reply-To: <0EE13AE7-933D-4ABE-842C-6958DE5DF321@gmail.com> References: <0EE13AE7-933D-4ABE-842C-6958DE5DF321@gmail.com> Message-ID: <20121008114136.GH498@localhost> >Got a new Google Nexus 7 (rooted). >Is there any (recent) Erlang binaries for ARM? you can use CEAN bootstraps here: http://cean.process-one.net/bootstraps/R14B04_linux-armhf_full.tar.gz From bgustavsson@REDACTED Mon Oct 8 14:15:06 2012 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 8 Oct 2012 14:15:06 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: On Sun, Oct 7, 2012 at 12:32 PM, Tomas Morstein wrote: > Hello, > > We wanted to upgrade our development environment to R15B02 > and ended up with a problem of different inheritance behaviour: > https://gist.github.com/3846164 > > The point is simple, let's imagine scenario: > - standard module a.erl with several core functions > (acts like abstract class) > - parametrized module b.erl that extends a.erl > to bring all the a's functionality > (b has always a single parameter what is proplist) > > On older version of Erlang/OTP, this worked fine > and we were able to b:new ([]) what gave us {b, []}. > This does not work on R15B02 anymore :-( > > We don't want to make a.erl parametrized for many reasons, > but I understand it makes some sense to bring a "base" (a) > instance with the instance of b. > The new behaviour still conforms to Richard Carlsson's paper, > because our scenario was not presented there... > > The problem I see is that only the case of non-parametrized > BASE "class" has changed in R15B02... The change is this bug fix: https://github.com/erlang/otp/commit/18943e753f7791312d89943851169e89acece8fc It is correction for a bug that was introduced three years ago in this commit: https://github.com/erlang/otp/commit/a612e99fb5aaa934fe5a8591db0f083d7fa0b20a Had parameterized modules been a supported feature, we would most probably have have test case that would have made the breakage visible. > How many changes are expected in future? Is it safe > to believe that once we refactor the code, no future version > will change something else? No, that is not safe assumption for any experimental feature. An experimental feature may changed or removed at any time if it turns out to be a bad idea, or cause more problems than it solves, or that a proper implementation would need too much effort and too many other changes. > Should we think about an alternative of doing inheritance > in our custom way otherwise? > Yes, I would recommend that. We (the OTP team) has not reached a decision yet, but it does seem that the problems outweigh the advantages. This email lists some of the problems: http://erlang.org/pipermail/erlang-questions/2012-August/068575.html /Bjorn -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From gordon@REDACTED Mon Oct 8 14:24:32 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Mon, 8 Oct 2012 13:24:32 +0100 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: We use Mochiweb which makes extensive use of them. So is the general advice that everybody should migrate away from Mochiweb? Gordon On 8 October 2012 13:15, Bj?rn Gustavsson wrote: > On Sun, Oct 7, 2012 at 12:32 PM, Tomas Morstein wrote: >> Hello, >> >> We wanted to upgrade our development environment to R15B02 >> and ended up with a problem of different inheritance behaviour: >> https://gist.github.com/3846164 >> >> The point is simple, let's imagine scenario: >> - standard module a.erl with several core functions >> (acts like abstract class) >> - parametrized module b.erl that extends a.erl >> to bring all the a's functionality >> (b has always a single parameter what is proplist) >> >> On older version of Erlang/OTP, this worked fine >> and we were able to b:new ([]) what gave us {b, []}. >> This does not work on R15B02 anymore :-( >> >> We don't want to make a.erl parametrized for many reasons, >> but I understand it makes some sense to bring a "base" (a) >> instance with the instance of b. >> The new behaviour still conforms to Richard Carlsson's paper, >> because our scenario was not presented there... >> >> The problem I see is that only the case of non-parametrized >> BASE "class" has changed in R15B02... > > The change is this bug fix: > > https://github.com/erlang/otp/commit/18943e753f7791312d89943851169e89acece8fc > > It is correction for a bug that was introduced three years > ago in this commit: > > https://github.com/erlang/otp/commit/a612e99fb5aaa934fe5a8591db0f083d7fa0b20a > > Had parameterized modules been a supported feature, we > would most probably have have test case that would have > made the breakage visible. > >> How many changes are expected in future? Is it safe >> to believe that once we refactor the code, no future version >> will change something else? > > No, that is not safe assumption for any experimental feature. > An experimental feature may changed or removed at any > time if it turns out to be a bad idea, or cause more problems > than it solves, or that a proper implementation would need > too much effort and too many other changes. > >> Should we think about an alternative of doing inheritance >> in our custom way otherwise? >> > > Yes, I would recommend that. > > We (the OTP team) has not reached a decision yet, but it > does seem that the problems outweigh the advantages. This > email lists some of the problems: > > http://erlang.org/pipermail/erlang-questions/2012-August/068575.html > > /Bjorn > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From enewhuis@REDACTED Mon Oct 8 14:33:50 2012 From: enewhuis@REDACTED (Eric Newhuis) Date: Mon, 8 Oct 2012 07:33:50 -0500 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: <8CD989D3-45D2-4840-8C10-D0B7F1F9D3A0@gmail.com> Is there a good write-up on parameterized module alternatives anywhere? - always include an .hrl and pass a record instance as arg 1 ? On Oct 8, 2012, at 7:24 AM, Gordon Guthrie wrote: > We use Mochiweb which makes extensive use of them. > > So is the general advice that everybody should migrate away from Mochiweb? > > Gordon > > On 8 October 2012 13:15, Bj?rn Gustavsson wrote: >> On Sun, Oct 7, 2012 at 12:32 PM, Tomas Morstein wrote: >>> Hello, >>> >>> We wanted to upgrade our development environment to R15B02 >>> and ended up with a problem of different inheritance behaviour: >>> https://gist.github.com/3846164 >>> >>> The point is simple, let's imagine scenario: >>> - standard module a.erl with several core functions >>> (acts like abstract class) >>> - parametrized module b.erl that extends a.erl >>> to bring all the a's functionality >>> (b has always a single parameter what is proplist) >>> >>> On older version of Erlang/OTP, this worked fine >>> and we were able to b:new ([]) what gave us {b, []}. >>> This does not work on R15B02 anymore :-( >>> >>> We don't want to make a.erl parametrized for many reasons, >>> but I understand it makes some sense to bring a "base" (a) >>> instance with the instance of b. >>> The new behaviour still conforms to Richard Carlsson's paper, >>> because our scenario was not presented there... >>> >>> The problem I see is that only the case of non-parametrized >>> BASE "class" has changed in R15B02... >> >> The change is this bug fix: >> >> https://github.com/erlang/otp/commit/18943e753f7791312d89943851169e89acece8fc >> >> It is correction for a bug that was introduced three years >> ago in this commit: >> >> https://github.com/erlang/otp/commit/a612e99fb5aaa934fe5a8591db0f083d7fa0b20a >> >> Had parameterized modules been a supported feature, we >> would most probably have have test case that would have >> made the breakage visible. >> >>> How many changes are expected in future? Is it safe >>> to believe that once we refactor the code, no future version >>> will change something else? >> >> No, that is not safe assumption for any experimental feature. >> An experimental feature may changed or removed at any >> time if it turns out to be a bad idea, or cause more problems >> than it solves, or that a proper implementation would need >> too much effort and too many other changes. >> >>> Should we think about an alternative of doing inheritance >>> in our custom way otherwise? >>> >> >> Yes, I would recommend that. >> >> We (the OTP team) has not reached a decision yet, but it >> does seem that the problems outweigh the advantages. This >> email lists some of the problems: >> >> http://erlang.org/pipermail/erlang-questions/2012-August/068575.html >> >> /Bjorn >> >> -- >> Bj?rn Gustavsson, Erlang/OTP, Ericsson AB >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bgustavsson@REDACTED Mon Oct 8 14:56:32 2012 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 8 Oct 2012 14:56:32 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: On Mon, Oct 8, 2012 at 2:24 PM, Gordon Guthrie wrote: > We use Mochiweb which makes extensive use of them. Are you sure? What I have understood from previous email messages on this list is that many projects use "tuple modules" (which is an implementation detail in parameterized modules) and not parameterized modules directly. See for instance: http://erlang.org/pipermail/erlang-questions/2012-January/063915.html We are thinking about removing the compiler support for parameterized modules, but keeping the low-level mechanism in erlang:apply/3 and the inheritance hack in the error_handler module. > So is the general advice that everybody should migrate away from Mochiweb? No, it is not. It is a recommendation to think twice about using experimental fetures when writing new code or fixing old code. -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From gordon@REDACTED Mon Oct 8 15:00:34 2012 From: gordon@REDACTED (Gordon Guthrie) Date: Mon, 8 Oct 2012 14:00:34 +0100 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: > Are you sure? Uh, no, I am not sure - I thought I was :( I had better find out. >> So is the general advice that everybody should migrate away from Mochiweb? > > No, it is not. It is a recommendation to think twice about using > experimental fetures when writing new code or fixing old code. Yeah, but the point of building on libraries is so as not to write code at all.. G On 8 October 2012 13:56, Bj?rn Gustavsson wrote: > On Mon, Oct 8, 2012 at 2:24 PM, Gordon Guthrie wrote: >> We use Mochiweb which makes extensive use of them. > > Are you sure? What I have understood from previous email > messages on this list is that many projects use "tuple modules" > (which is an implementation detail in parameterized modules) > and not parameterized modules directly. See for instance: > > http://erlang.org/pipermail/erlang-questions/2012-January/063915.html > > We are thinking about removing the compiler support for parameterized > modules, but keeping the low-level mechanism in erlang:apply/3 and > the inheritance hack in the error_handler module. > >> So is the general advice that everybody should migrate away from Mochiweb? > > No, it is not. It is a recommendation to think twice about using > experimental fetures when writing new code or fixing old code. > > > -- > Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From serge@REDACTED Mon Oct 8 15:07:07 2012 From: serge@REDACTED (Serge Aleynikov) Date: Mon, 08 Oct 2012 09:07:07 -0400 Subject: [erlang-questions] erl_call problem In-Reply-To: References: Message-ID: <5072CFFB.3060205@aleynikov.org> You normally receive 'salive' in the distribution protocol when you try to connect to a server node from another client node that has a name already known to the server node. I.e.: shell1> erl -sname a shell2> erl_call -sname a -h b -a 'timer sleep [60000]' shell3> erl_call -sname a -h b -a 'timer sleep [60000]' You should get a failed 'salive' connection error in shell3. Regards, Serge On 10/8/2012 3:01 AM, Andrew Gopienko wrote: > We are using erl_call for nodes monitoring and sometimes erl_call failed > with message 'erl_call: failed to connect to node node_01@REDACTED > ' > Node at this time is alive and works as usually. > > We found that node at this time returns status 'salive' instead of > expected 'sok'. > > ------------------------------ > ---------------------------------------------------------------- > ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT attempt to connect to > node_01 > ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: -> PORT2_REQ alive=node_01 > ip=192.168.1.2 > ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: <- PORT2_RESP result=0 (ok) > ei_epmd_r4_port: Sun Oct 7 16:45:00 2012: port=52658 ntype=77 > proto=0 dist-high=5 dist-low=5 > ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT connected to remote > recv_status: Sun Oct 7 16:45:00 2012: <- len=6 > recv_status: Sun Oct 7 16:45:00 2012: <- buf[0]=s (115) > recv_status: Sun Oct 7 16:45:00 2012: <- buf[1]=a (97) > recv_status: Sun Oct 7 16:45:00 2012: <- buf[2]=l (108) > recv_status: Sun Oct 7 16:45:00 2012: <- buf[3]=i (105) > recv_status: Sun Oct 7 16:45:00 2012: <- buf[4]=v (118) > recv_status: Sun Oct 7 16:45:00 2012: <- buf[5]=e (101) > ex_xconnect: Sun Oct 7 16:45:00 2012: RECV_STATUS failed > ei_xconnect: Sun Oct 7 16:45:00 2012: -> CONNECT failed > erl_call: failed to connect to node node_01@REDACTED > , fd=-1, errno=0, erl_errno=5 > ---------------------------------------------------------------------------------------------- > > In what cases the node returns 'salive' ? > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From bgustavsson@REDACTED Mon Oct 8 15:12:03 2012 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 8 Oct 2012 15:12:03 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: <8CD989D3-45D2-4840-8C10-D0B7F1F9D3A0@gmail.com> References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> <8CD989D3-45D2-4840-8C10-D0B7F1F9D3A0@gmail.com> Message-ID: On Mon, Oct 8, 2012 at 2:33 PM, Eric Newhuis wrote: > Is there a good write-up on parameterized module alternatives anywhere? > > - always include an .hrl and pass a record instance as arg 1 ? > It depends on what exactly you are using them for. Personally, I would define a fun with two args, where the first argument is a method name and the second argument is a tuple with the arguments. It would be used like this: PM = some_module:new(Var1, Var2, Var3), . . . PM(print, {Arg1,Arg2}), . . . PM(save, Filename) . . . And so on. The function new/3 could be defined like this in the module some_module: -record(vars, {var1,var2,var3}). new(Var1, Var2, Var2) -> Vars = vars{var1=Var1,var2=Var2,var3=Var3}, fun(print, Args) -> print(Args, Vars); (save, Args) -> save(Args, Vars) . . . end. -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bgustavsson@REDACTED Mon Oct 8 15:31:05 2012 From: bgustavsson@REDACTED (=?UTF-8?Q?Bj=C3=B6rn_Gustavsson?=) Date: Mon, 8 Oct 2012 15:31:05 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: On Mon, Oct 8, 2012 at 3:00 PM, Gordon Guthrie wrote: >> No, it is not. It is a recommendation to think twice about using >> experimental fetures when writing new code or fixing old code. > > Yeah, but the point of building on libraries is so as not to write code at all.. We are not doing this to break people's code, but because there is cost to keeping experimental features forever in the language. Hopefully mochiweb will continue to work if we keep the apply/3 and error_handler hacks; if not, the maintainers of Mochiweb and we in the OTP team could probably come up with some solution to keep Mochiweb working. -- Bj?rn Gustavsson, Erlang/OTP, Ericsson AB From bob@REDACTED Mon Oct 8 17:19:14 2012 From: bob@REDACTED (Bob Ippolito) Date: Mon, 8 Oct 2012 08:19:14 -0700 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <20e0aced-8a5b-433e-aaff-678e197c28a9@zimbra> Message-ID: On Mon, Oct 8, 2012 at 6:31 AM, Bj?rn Gustavsson wrote: > On Mon, Oct 8, 2012 at 3:00 PM, Gordon Guthrie wrote: > > >> No, it is not. It is a recommendation to think twice about using > >> experimental fetures when writing new code or fixing old code. > > > > Yeah, but the point of building on libraries is so as not to write code > at all.. > > We are not doing this to break people's code, but because there is > cost to keeping experimental features forever in the language. > > Hopefully mochiweb will continue to work if we keep the apply/3 and > error_handler hacks; if not, the maintainers of Mochiweb and we in > the OTP team could probably come up with some solution to keep > Mochiweb working. The API of mochiweb is a pair of parameterized modules, mochiweb_request and mochiweb_response. It doesn't use tuple module hacks except maybe for some tests. If there's some backwards compatible change that can be done to mochiweb to ensure its future compatibility, I'd be happy to do that. If you're worried about parameterized modules going away and breaking mochiweb, it might be a good time to switch to cowboy or yaws. These two are much more future-proof. -bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmercer@REDACTED Mon Oct 8 17:54:15 2012 From: dmercer@REDACTED (David Mercer) Date: Mon, 8 Oct 2012 10:54:15 -0500 Subject: [erlang-questions] Rebar & Cowboy on Windows 7 Message-ID: <018f01cda56d$2b606b90$822142b0$@gmail.com> Having made the statement a few times over the last few months that Rebar is not compatible with Windows, I was challenged on this. My statement was based on my one-off experience with Rebar one or two years ago, so it was time I retried it. I decided to use the discussion about HTTP servers to try out Cowboy, which requires Rebar. 1. Rebar installed just fine on Windows. I downloaded the zip file from Git Hub and unzipped it into my Erlang lib directory. I ended up with a directory named ?basho-rebar-32e67ef? in my lib, which is kind of nonstandard naming, but I could have renamed it to ?rebar-...? if I had bothered to look up the correct version number to put there. Failing that, I could have just renamed it ?rebar?, but it worked as it was, so I was unconcerned. 2. I ran the Rebar bootstrap.bat, which generated a rebar file and a rebar.cmd file to run it. So far I was quite encouraged, with the provision of a Windows ?bat? file and now a ?cmd? file. It would appear that Rebar was built to work on Windows. J 3. I downloaded Cowboy from Git Hub the same way I did Rebar. The resulting directory was again named ?extend-cowboy-1f9d71c? in the nonstandard way of Git Hub, but it still works. 4. Here I ran into some problems, but it turned out they were mine rather than Windows?s. I tried a ?rebar compile? on Cowboy, but got an error about the dependency on Ranch not being available. It should have been available, since I had previously downloaded it, and even if I hadn?t, the rebar.config told it where it could get it from. Turns out, however, that I have to explicitly tell it to get dependencies. Tuncer Ayaz ? bless his soul ? gave me a hand here and explained that I should use the command ?rebar get-deps?, ?rebar get-deps compile?, or abbreviated to ?rebar g-d com?. With that, it downloaded Ranch and compiled fine. Yay! 5. Interestingly enough, Rebar does not download the dependencies into the Erlang lib directory, but rather into a ?deps? subdirectory of your application. This means erl.exe and werl.exe cannot see it, but this can easily be rectified by either copying the downloaded applications up into the lib directory, or using the ?-pa? option on the command line. If anyone has any suggestions as to which is better for their workflow, please let me know. 6. At this point, Cowboy was working, but I figured I should try to get it to do something useful. I couldn?t find a whole lot of documentation on its API, but I did find that it came with a few examples in the examples subdirectory. I decided to give the static example a whirl, and Rebarred it and then ran it, and it worked. Fine. Out of the box. Yay! 7. Unfortunately, when it served up the test.txt sample file, Chrome treated it as a download rather than text to be displayed in the browser. Usually when I encounter an issue where I want to see what?s coming down the wire, I just telnet to the appropriate port and send a ?GET /? to see what is returned. I don?t think Cowboy handles HTTP 1.0, though I couldn?t find any record of that. At any rate, I just had to do some extra typing in order to send a minimal HTTP 1.1 request, and after some poking around, I discovered my problem was related to the MIME type being sent in the HTTP response. So I added the ?mimetypes? application to the example?s Rebar dependencies, modified the static_app.erl file to use the function mimetypes:path_to_mimes to choose the correct MIME type and then restarted it ? or at least tried to. Try as I may, I could not get it to work for the longest time. In the end, I determined that mimetypes?s lexer was built for Unix only, not Windows, so it was not recognizing the line endings in the provided mime.types file. The solution then was to modify mimetypes?s mimetypes_scan.xrl to recognize a CRLF as a newline, too. I suppose I should figure out how to use Git Hub so I can submit that patch. At any rate, now it all works like champ. So I can confirm that both Rebar and Cowboy work on Windows 7 system. Thank-you all for your work on these. David Mercer Director, Technology & Research MedWorth -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Oct 8 18:07:26 2012 From: essen@REDACTED (=?windows-1252?Q?Lo=EFc_Hoguin?=) Date: Mon, 08 Oct 2012 18:07:26 +0200 Subject: [erlang-questions] Rebar & Cowboy on Windows 7 In-Reply-To: <018f01cda56d$2b606b90$822142b0$@gmail.com> References: <018f01cda56d$2b606b90$822142b0$@gmail.com> Message-ID: <5072FA3E.904@ninenines.eu> Hello, On 10/08/2012 05:54 PM, David Mercer wrote: > 3.I downloaded Cowboy from Git Hub the same way I did Rebar. The > resulting directory was again named ?extend-cowboy-1f9d71c? in the > nonstandard way of Git Hub, but it still works. Note that you typically just include Cowboy as a dependency for your own application. > 4.Here I ran into some problems, but it turned out they were mine rather > than Windows?s. I tried a ?rebar compile? on Cowboy, but got an error > about the dependency on Ranch not being available. It should have been > available, since I had previously downloaded it, and even if I hadn?t, > the /rebar.config/ told it where it could get it from. Turns out, > however, that I have to explicitly tell it to get dependencies. Tuncer > Ayaz ? bless his soul ? gave me a hand here and explained that I should > use the command ?rebar get-deps?, ?rebar get-deps compile?, or > abbreviated to ?rebar g-d com?. With that, it downloaded Ranch and > compiled fine. Yay! It's automated on Linux because of the Makefile, you can just type "make" there. Not sure the same can be done for Windows without a lot of manipulations. > 5.Interestingly enough, Rebar does not download the dependencies into > the Erlang lib directory, but rather into a ?deps? subdirectory of your > application. This means /erl.exe/ and /werl.exe/ cannot see it, but > this can easily be rectified by either copying the downloaded > applications up into the lib directory, or using the ?-pa? option on the > command line. If anyone has any suggestions as to which is better for > their workflow, please let me know. Neither, the proper way is building your own releases. Releases will then include both erl.exe and Cowboy and all the applications required. The examples use -pa only for demonstration purposes, I recommend building releases even while developing. > 6.At this point, Cowboy was working, but I figured I should try to get > it to do something useful. I couldn?t find a whole lot of documentation > on its API, but I did find that it came with a few examples in the > /examples/ subdirectory. I decided to give the static example a whirl, > and Rebarred it and then ran it, and it worked. Fine. Out of the box. > Yay! Can you try "rebar edoc"? It's supposed to generate the API reference. > 7.Unfortunately, when it served up the /test.txt/ sample file, Chrome > treated it as a download rather than text to be displayed in the > browser. Usually when I encounter an issue where I want to see what?s > coming down the wire, I just telnet to the appropriate port and send a > ?GET /? to see what is returned. I don?t think Cowboy handles HTTP 1.0, > though I couldn?t find any record of that. At any rate, I just had to > do some extra typing in order to send a minimal HTTP 1.1 request, and > after some poking around, I discovered my problem was related to the > MIME type being sent in the HTTP response. So I added the ?mimetypes? > application to the example?s Rebar dependencies, modified the > /static_app.erl/ file to use the function /mimetypes:path_to_mimes/ to > choose the correct MIME type and then restarted it ? or at least tried > to. Try as I may, I could not get it to work for the longest time. In > the end, I determined that mimetypes?s lexer was built for Unix only, > not Windows, so it was not recognizing the line endings in the provided > /mime.types/ file. The solution then was to modify mimetypes?s > /mimetypes_scan.xrl/ to recognize a CRLF as a newline, too. I suppose I > should figure out how to use Git Hub so I can submit that patch. At any > rate, now it all works like champ. Please do send a patch, that'd be much appreciated! And yeah the example can probably be improved a little using the mimetypes application. Feel free to send a patch for that, too. > So I can confirm that both Rebar and Cowboy work on Windows 7 system. > Thank-you all for your work on these. Thanks for testing! -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From dmercer@REDACTED Mon Oct 8 19:22:37 2012 From: dmercer@REDACTED (David Mercer) Date: Mon, 8 Oct 2012 12:22:37 -0500 Subject: [erlang-questions] Rebar & Cowboy on Windows 7 In-Reply-To: <5072FA3E.904@ninenines.eu> References: <018f01cda56d$2b606b90$822142b0$@gmail.com> <5072FA3E.904@ninenines.eu> Message-ID: <01dc01cda579$8417f1c0$8c47d540$@gmail.com> On Monday, October 08, 2012, Lo?c Hoguin wrote: > It's automated on Linux because of the Makefile, you can just type "make" > there. Not sure the same can be done for Windows without a lot of > manipulations. In my experience, Makefiles in Windows get all screwed up with the directory separator. I'll give it a whirl, though, and see if anyone's version of make on Windows works. > Neither, the proper way is building your own releases. Releases will then > include both erl.exe and Cowboy and all the applications required. > > The examples use -pa only for demonstration purposes, I recommend building > releases even while developing. Good advice, which is probably what I would do if using Rebar in my development. Not previously having Rebar at my disposal, release-making was a bit over-complicated for some of the stuff I did. > Can you try "rebar edoc"? It's supposed to generate the API reference. "rebar doc" works fine, and I looked at the documentation on Friday; it just didn?t help me. For example: start_http(Ref::any(), NbAcceptors::non_neg_integer(), TransOpts::any(), ProtoOpts::any()) -> {ok, pid()} I am still none the wiser as to what to pass the start_http function. > Please do send a patch, that'd be much appreciated! > > And yeah the example can probably be improved a little using the mimetypes > application. Feel free to send a patch for that, too. Will do, once I figure out the Git Hub process. Is Git Hub now preferred over Bit Bucket? Cheers, DBM From vinoski@REDACTED Mon Oct 8 19:57:13 2012 From: vinoski@REDACTED (Steve Vinoski) Date: Mon, 8 Oct 2012 13:57:13 -0400 Subject: [erlang-questions] Server-Sent Events with Yaws Message-ID: My latest Internet Computing column explores W3C Server-Sent Events and how the Yaws web server supports them: http://steve.vinoski.net/blog/2012/10/08/new-column-server-sent-events-with-yaws/ --steve From klas.johansson@REDACTED Mon Oct 8 22:27:30 2012 From: klas.johansson@REDACTED (Klas Johansson) Date: Mon, 8 Oct 2012 22:27:30 +0200 Subject: [erlang-questions] [ANN] wpi - Controlling the Raspberry Pi using Erlang Message-ID: Hi, Are you interested in letting Erlang control a Raspberry Pi [1]? If so you might be interested in `wpi' which is an Erlang wrapper around the WiringPi library (by Gordon Henderson) which is a Raspberry Pi dialect of the Wiring library which you may have experienced for Arduino. wpi makes it possible to read from and write to GPIO pins, write to LCDs, shift bits in and out and control other devices over serial interfaces or SPI and all this from a Raspberry Pi running Erlang. https://github.com/klajo/wpi Contributions are more than welcome. Thanks to David Haglund who contributed the SPI bus and serial interface parts. Best Regards, Klas [1] http://www.raspberrypi.org/faqs From rustompmody@REDACTED Tue Oct 9 07:54:34 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Tue, 9 Oct 2012 11:24:34 +0530 Subject: [erlang-questions] about erlang versions Message-ID: Erlang noob here... On my debian box when I start erlang I get Erlang R15B01 (erts-5.9.1) ... Since I am following http://learnyousomeerlang.com/content and it suggests some versions wondering how I should interpret these numbers -------------- next part -------------- An HTML attachment was scrubbed... URL: From rapsey@REDACTED Tue Oct 9 08:20:47 2012 From: rapsey@REDACTED (Rapsey) Date: Tue, 9 Oct 2012 08:20:47 +0200 Subject: [erlang-questions] about erlang versions In-Reply-To: References: Message-ID: That is how erlang is versioned. Past releases are listed on this page: http://www.erlang.org/download.html So you are one release away from latest. It is fine. Sergej On Tue, Oct 9, 2012 at 7:54 AM, Rustom Mody wrote: > Erlang noob here... > > On my debian box when I start erlang I get > Erlang R15B01 (erts-5.9.1) ... > > Since I am following http://learnyousomeerlang.com/content > and it suggests some versions wondering how I should interpret these > numbers > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony@REDACTED Tue Oct 9 09:28:24 2012 From: tony@REDACTED (Tony Rogvall) Date: Tue, 9 Oct 2012 09:28:24 +0200 Subject: [erlang-questions] [ANN] wpi - Controlling the Raspberry Pi using Erlang In-Reply-To: References: Message-ID: Thanks! Note that serialGetchar (wiringSerial) will block for (up to) 10s if no character is available. (From function header documentation) And this fact (wiringSerial.c:95) options.c_cc [VMIN] = 0 ; options.c_cc [VTIME] = 100 ; // Ten seconds (100 deciseconds) The effect is that the emulator will be blocked and no other processes may run (for up to 10 s)! (unless there are more cores than one (online)) You may want to use driver_select. Should probably set the VTIME=0 to be sure. A little plug: There is a project using threads to get some exotic uart features, a bit heavy, but is working (even on windows!) (Still lack documentation but has a simple api and is simple to use, nice packeting options for example) https://github.com/tonyrog/uart /Tony On 8 okt 2012, at 22:27, Klas Johansson wrote: > Hi, > > Are you interested in letting Erlang control a Raspberry Pi [1]? If so > you might be interested in `wpi' which is an Erlang wrapper around the > WiringPi library (by Gordon Henderson) which is a Raspberry Pi dialect > of the Wiring library which you may have experienced for Arduino. wpi > makes it possible to read from and write to GPIO pins, write to LCDs, > shift bits in and out and control other devices over serial interfaces > or SPI and all this from a Raspberry Pi running Erlang. > > https://github.com/klajo/wpi > > Contributions are more than welcome. Thanks to David Haglund who > contributed the SPI bus and serial interface parts. > > Best Regards, > Klas > > [1] http://www.raspberrypi.org/faqs > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions "Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix" -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric@REDACTED Tue Oct 9 15:11:22 2012 From: eric@REDACTED (Eric Moritz) Date: Tue, 9 Oct 2012 09:11:22 -0400 Subject: [erlang-questions] erlang parser for javascript literals In-Reply-To: References: <-3184479837163322775@unknownmsgid> Message-ID: If you're writing out data structures by hand, YAML is the way to go. On Mon, Oct 8, 2012 at 7:29 AM, Jos? Valim wrote: > >> This is true, but it does not hold you back from choosing another >> representation format and then converting from that format into proper JSON >> later when you need to interact with other systems in a correct manner. Raw >> JSON is quite hard to write due to some of the rules in the dark corners of >> JS parsing :) > > > If you don't want to parse strictly JSON, YAML is an option that will give > you more flexibility than JSON (and the features you asked). > > There are some available Erlang wrappers for libyaml, but I haven't used > them personally: > > https://github.com/goertzenator/yamler > > > Jos? Valim > www.plataformatec.com.br > Founder and Lead Developer > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From klas.johansson@REDACTED Tue Oct 9 22:40:19 2012 From: klas.johansson@REDACTED (Klas Johansson) Date: Tue, 9 Oct 2012 22:40:19 +0200 Subject: [erlang-questions] [ANN] wpi - Controlling the Raspberry Pi using Erlang In-Reply-To: References: Message-ID: Hi Tony, On Tue, Oct 9, 2012 at 9:28 AM, Tony Rogvall wrote: > Thanks! :-) > Note that serialGetchar (wiringSerial) will block for (up to) 10s if no > character is available. > (From function header documentation) > > And this fact (wiringSerial.c:95) > > options.c_cc [VMIN] = 0 ; > options.c_cc [VTIME] = 100 ; // Ten seconds (100 deciseconds) > > The effect is that the emulator will be blocked and no other processes may > run (for up to 10 s)! > (unless there are more cores than one (online)) > You may want to use driver_select. Should probably set the VTIME=0 to be > sure. Yes that's a very valid point, I noticed that while going through the documentation yesterday. I've updated the documentation to state this and also created an "issue" on github. Thanks for the reminder! > A little plug: > There is a project using threads to get some exotic uart features, a bit > heavy, but is working (even on windows!) > (Still lack documentation but has a simple api and is simple to use, nice > packeting options for example) > https://github.com/tonyrog/uart Looks nice. Thanks, Klas > On 8 okt 2012, at 22:27, Klas Johansson wrote: > > Hi, > > Are you interested in letting Erlang control a Raspberry Pi [1]? If so > you might be interested in `wpi' which is an Erlang wrapper around the > WiringPi library (by Gordon Henderson) which is a Raspberry Pi dialect > of the Wiring library which you may have experienced for Arduino. wpi > makes it possible to read from and write to GPIO pins, write to LCDs, > shift bits in and out and control other devices over serial interfaces > or SPI and all this from a Raspberry Pi running Erlang. > > https://github.com/klajo/wpi > > Contributions are more than welcome. Thanks to David Haglund who > contributed the SPI bus and serial interface parts. > > Best Regards, > Klas > > [1] http://www.raspberrypi.org/faqs > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > "Installing applications can lead to corruption over time. Applications > gradually write over each other's libraries, partial upgrades occur, user > and system errors happen, and minute changes may be unnoticeable and > difficult to fix" > > > From tmr@REDACTED Wed Oct 10 00:54:21 2012 From: tmr@REDACTED (Tomas Morstein) Date: Wed, 10 Oct 2012 00:54:21 +0200 (CEST) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: Message-ID: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> > > Is there a good write-up on parameterized module alternatives > > anywhere? > > > > - always include an .hrl and pass a record instance as arg 1 ? > > > > It depends on what exactly you are using them for. > > Personally, I would define a fun with two args, where the first > argument > is a method name and the second argument is a tuple with the > arguments. I like this idea, it's clean and elegant, but not without limitations :( The problem starts when it comes to projects like ChicagoBoss that we at IDEA love and intensively use every day. CB uses pmods not only for simple_bridge-based web controllers, but also for its sophisticated and well integrated concept of data models being linked with Django templating engine in turn. Although I fully understand reasons why pmods and their eventual pseudo-OOP behaviour (like inheritance) are not so clear as they are implemented today, but in other hand, many excellent projects may disappear in final result... I hope such a "total eclipse" is not going to happen, but our case (over-using the extra unsupported and hackish features) we've just adapted our technology to framework and totally forgot to standards, what's in fact our own mistake. Of course, once something is not well documented/specified/supported, no one should use it seriously, so we accept the situation we ran into. Another thing is that some functionality, especially the case of pmods, is so popular, as everybody knows it because it is really useful, that everybody also uses it. Pmods seems to be a "de facto" standard then. Shouldn't we pay attention since it's used so heavily on many places? Please note that I am not in a position allowed to dictate nor recommend anything. It's just a description of my point of view and I'm pretty sure that no matter on the result of pmod "affair", everybody will have to accept "new rules" finally (or move elsewhere). Tom From straightflush@REDACTED Wed Oct 10 04:02:53 2012 From: straightflush@REDACTED (AD) Date: Tue, 9 Oct 2012 22:02:53 -0400 Subject: [erlang-questions] Cowboy , Concurrency , Tuning Message-ID: hello, I currently have a SockJS websocket implementation sitting on top of cowboy. We have been going through some tuning/concurrency testing and we have improved our results drastically by increasing ERL_MAX_PORTS to 64000 +P parameter to a very high number , around 128000 Kernel tuning With this config on a medium VM we have been able to successfully get to around 60,000 connections, up from 4096 so we are headed in the right direction :-) MY next question is with Cowboy tuning. I notice that the default max_connections parameter on the TCP transport is 1024 if not set, but I am not hitting this issue. What is the right way to tune NBAcceptor pool and the the max_connections setting for the acceptor ? Right now my start_listener is simply cowboy:start_listener(my_http_listener, 200, cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}], cowboy_http_protocol, [{dispatch, Routes}] ). but i have seen some examples with cowboy:start_listener(my_http_listener, 200, cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}, *{max_connections, infinity}* ], cowboy_http_protocol, [{dispatch, Routes}] ). Cowboy tuning seems like the last place i need to look, any thoughts/feedback here on best practices ? Thanks -AD -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody@REDACTED Wed Oct 10 08:37:16 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Wed, 10 Oct 2012 12:07:16 +0530 Subject: [erlang-questions] suitability of erlang Message-ID: Hello We are considering Erlang for a project involving significant data-processing under severe time-crunch I am trying to understand the sentence (from http://www.erlang.org/faq/introduction.html#id49576 ) ---------- The most common class of 'less suitable' problems is characterised by performance being a prime requirement *and* constant-factors having a large effect on performance. ---------- How do I understand the '... and constant-factors...' Thanks Rusi -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulperegud@REDACTED Wed Oct 10 08:41:23 2012 From: paulperegud@REDACTED (Paul Peregud) Date: Wed, 10 Oct 2012 08:41:23 +0200 Subject: [erlang-questions] Cowboy , Concurrency , Tuning In-Reply-To: References: Message-ID: It seems that you are using old Cowboy. I was using it too and I hit a bottleneck in cowboy_listener add_pid/remove_pid mechanism. New Cowboy/Ranch handles that part better. On Wed, Oct 10, 2012 at 4:02 AM, AD wrote: > hello, > > I currently have a SockJS websocket implementation sitting on top of > cowboy. We have been going through some tuning/concurrency testing and we > have improved our results drastically by increasing > > ERL_MAX_PORTS to 64000 > +P parameter to a very high number , around 128000 > Kernel tuning > > With this config on a medium VM we have been able to successfully get to > around 60,000 connections, up from 4096 so we are headed in the right > direction :-) > > MY next question is with Cowboy tuning. I notice that the default > max_connections parameter on the TCP transport is 1024 if not set, but I am > not hitting this issue. What is the right way to tune NBAcceptor pool and > the the max_connections setting for the acceptor ? > > Right now my start_listener is simply > > cowboy:start_listener(my_http_listener, 200, > cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}], > cowboy_http_protocol, [{dispatch, Routes}] > ). > > but i have seen some examples with > > cowboy:start_listener(my_http_listener, 200, > cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}, *{max_connections, > infinity}* ], > cowboy_http_protocol, [{dispatch, Routes}] > ). > > Cowboy tuning seems like the last place i need to look, any > thoughts/feedback here on best practices ? > > Thanks > -AD > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -- Best regards, Paul Peregud +48602112091 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Wed Oct 10 09:10:27 2012 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 10 Oct 2012 09:10:27 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: What would be good is if a list of good uses of pmods could be collected, and honorable workarounds suggested. This could of course be done interactively on this list. Present what you think is code that is not easily done well without pmods, and let the members of this list debate suitable rewrites. If good alternatives are hard to find, maybe that could inform some EEP for a more 'erlangish' extension that renders pmods truly obsolete? BR, Ulf W On 10 okt 2012, at 00:54, Tomas Morstein wrote: >>> ... > > The problem starts when it comes to projects like ChicagoBoss > that we at IDEA love and intensively use every day. > CB uses pmods not only for simple_bridge-based web controllers, > but also for its sophisticated and well integrated concept of > data models being linked with Django templating engine in turn. > > Although I fully understand reasons why pmods and their eventual > pseudo-OOP behaviour (like inheritance) are not so clear as they > are implemented today, but in other hand, many excellent projects > may disappear in final result... > > I hope such a "total eclipse" is not going to happen, but our case > (over-using the extra unsupported and hackish features) we've just > adapted our technology to framework and totally forgot to standards, > what's in fact our own mistake. From essen@REDACTED Wed Oct 10 10:36:17 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Wed, 10 Oct 2012 10:36:17 +0200 Subject: [erlang-questions] Cowboy , Concurrency , Tuning In-Reply-To: References: Message-ID: <50753381.1050603@ninenines.eu> On 10/10/2012 04:02 AM, AD wrote: > hello, Hey, > I currently have a SockJS websocket implementation sitting on top of > cowboy. We have been going through some tuning/concurrency testing and > we have improved our results drastically by increasing > > ERL_MAX_PORTS to 64000 > +P parameter to a very high number , around 128000 > Kernel tuning > > With this config on a medium VM we have been able to successfully get > to around 60,000 connections, up from 4096 so we are headed in the right > direction :-) Well then with the above configuration you're at about the maximum concurrent connections you can get. If you increase ERL_MAX_PORTS and +P even more Cowboy shouldn't have any trouble giving you even better results. > MY next question is with Cowboy tuning. I notice that the default > max_connections parameter on the TCP transport is 1024 if not set, but I > am not hitting this issue. What is the right way to tune NBAcceptor > pool and the the max_connections setting for the acceptor ? NBAcceptor shouldn't be too low but shouldn't be too high either. I found 100 to be a good values but it really depends on how many connections come at once, how many cores you have, how many concurrent connections you expect, and so on. > Right now my start_listener is simply > > cowboy:start_listener(my_http_listener, 200, > cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}], > cowboy_http_protocol, [{dispatch, Routes}] > ). > > but i have seen some examples with > > cowboy:start_listener(my_http_listener, 200, > cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}, > */{max_connections, infinity}/* ], > cowboy_http_protocol, [{dispatch, Routes}] > ). If you're using Cowboy 0.6.1 or below and looking for performance, you should set max_connections to infinity. If you're using master, you can let it at the default value. Websocket connections are not counted in that number. All HTTP connections are though, so you might want to look at cowboy_listener to see how to remove the connection from the count (varies depending on your version). -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From be.dmitry@REDACTED Wed Oct 10 11:18:08 2012 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Wed, 10 Oct 2012 13:18:08 +0400 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: My opinion is that parametrized modules are very nice for abstractions. For example like these: https://github.com/rabbitmq/erlando#monad-transformers Of course it is doable without pmods. But it is doable with some kind of reinventing the wheel. Do we need to have non standard way to do the same thing? -module(my_pmod). -export([new/2, apply/2]). -record(my_pmod, {module, thid}). new(Module, Params) -> #my_pmod{module=Module, this=Params}. apply(#my_pmod{module=Module, this=This}, Fun, Params) -> apply(Module, Fun, Params ++ [This]). -- Dmitry Belyaev On 10.10.2012, at 11:10, Ulf Wiger wrote: > What would be good is if a list of good uses of pmods could > be collected, and honorable workarounds suggested. > > This could of course be done interactively on this list. > Present what you think is code that is not easily done well > without pmods, and let the members of this list debate > suitable rewrites. If good alternatives are hard to find, > maybe that could inform some EEP for a more > 'erlangish' extension that renders pmods truly obsolete? > > BR, > Ulf W > > > On 10 okt 2012, at 00:54, Tomas Morstein wrote: > >>>> ... >> >> The problem starts when it comes to projects like ChicagoBoss >> that we at IDEA love and intensively use every day. >> CB uses pmods not only for simple_bridge-based web controllers, >> but also for its sophisticated and well integrated concept of >> data models being linked with Django templating engine in turn. >> >> Although I fully understand reasons why pmods and their eventual >> pseudo-OOP behaviour (like inheritance) are not so clear as they >> are implemented today, but in other hand, many excellent projects >> may disappear in final result... >> >> I hope such a "total eclipse" is not going to happen, but our case >> (over-using the extra unsupported and hackish features) we've just >> adapted our technology to framework and totally forgot to standards, >> what's in fact our own mistake. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From ulf@REDACTED Wed Oct 10 11:23:52 2012 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 10 Oct 2012 11:23:52 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: Message-ID: <46B630DC-F790-455B-8D94-392352EC0DE3@feuerlabs.com> On 10 Oct 2012, at 08:37, Rustom Mody wrote: > Hello > > We are considering Erlang for a project involving significant data-processing under severe time-crunch > > I am trying to understand the sentence (from http://www.erlang.org/faq/introduction.html#id49576 ) > ---------- > The most common class of 'less suitable' problems is characterised by performance being a prime requirement and constant-factors having a large effect on performance. > ---------- > > How do I understand the '... and constant-factors?' For example, if the problem involves a significant sequential component that needs to be very fast, and cannot be made fast by distributing the work across multiple work units. In some cases, even if the problem can scale, it may be the case that Erlang is simply too slow to ever reach parity with very fast sequential solutions. One such example is the ant colony optimization experiment [1], where the Erlang solution did scale well, but started off being ca 130x slower on one core than the MLTon Standard ML solution. The fact that the Erlang solution got 1.9x speedup with two cores then becomes rather academic. [1] http://eric_rollins.home.mindspring.com/erlangAnt.html This is of course only speaking in general terms. In many cases, small parts of performance-critical parts can be factored out and written in e.g. C, with the rest of the application written in Erlang. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre.graf@REDACTED Wed Oct 10 11:55:30 2012 From: andre.graf@REDACTED (=?ISO-8859-1?Q?Andr=E9_Graf?=) Date: Wed, 10 Oct 2012 11:55:30 +0200 Subject: [erlang-questions] Erlang in distributed systems research Message-ID: Dear list I have the opportunity to give a talk about Erlang to the folks from the database and information systems research group at the University Basel. The group heavily uses Java for the prototypes they develop. We all know that they can benefit from Erlang and that's what I want the take-home message to be. Besides of other things I want to talk about some examples of Erlang in research that might fit the topic. Do you have any pointers on that one? Cheers Andr? From matthias@REDACTED Wed Oct 10 12:15:34 2012 From: matthias@REDACTED (Matthias Lang) Date: Wed, 10 Oct 2012 12:15:34 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: Message-ID: <20121010101533.GA4221@corelatus.se> On Wednesday, October 10, Rustom Mody wrote: > We are considering Erlang for a project involving significant > data-processing under severe time-crunch > > I am trying to understand the sentence (from > http://www.erlang.org/faq/introduction.html#id49576 ) > ---------- > The most common class of 'less suitable' problems is characterised by > performance being a prime requirement *and* constant-factors having a large > effect on performance. > ---------- > How do I understand the '... and constant-factors...' I agree, that statement doesn't make a whole heap of sense. Looking at it now, I'm no longer sure what I was thinking then. Maybe it was "If it matters that your pure-Erlang system runs, say, 4x slower than a competitor's C system, then Erlang alone isn't a good choice". I'll think about it a bit and try and come up with something which makes more sense and isn't a novel. Back to your implied question: If you haven't worked with Erlang before, then picking a small part of what you want to do and implementing it is going to give you a much better feel for performance than replies on a mailing list. E.g. you have no idea whether I have a clue or whether I'm just making stuff up because I like seeing myself on a mailing list. Many (most? all?) systems which use Erlang and which do "significant data-processing" use C to improve performance on the parts where performance is important. E.g. if you're converting JPEG images to PNG, then Erlang is going to be several times slower than C. There's a classic blog post about this: http://www.dadgum.com/james/performance.html in some ways, the post says the opposite of what I wrote. The post was also written before the native-code compiler became more widely used. ---- Maybe also of interest: http://www.codinghorror.com/blog/2008/06/exploring-wide-finder.html ---- Another thing: you write "under severe time-crunch", but that could mean "we need to ship a product in a very short time" or it could mean "the data has to be processed in a very short time". Matt From straightflush@REDACTED Wed Oct 10 12:32:42 2012 From: straightflush@REDACTED (AD) Date: Wed, 10 Oct 2012 06:32:42 -0400 Subject: [erlang-questions] Cowboy , Concurrency , Tuning In-Reply-To: <50753381.1050603@ninenines.eu> References: <50753381.1050603@ninenines.eu> Message-ID: Thanks Loic. Any reason why websocket connections are not counted ? is there a separate pool for max_connections for websockets? On Wed, Oct 10, 2012 at 4:36 AM, Lo?c Hoguin wrote: > On 10/10/2012 04:02 AM, AD wrote: > >> hello, >> > > Hey, > > > I currently have a SockJS websocket implementation sitting on top of >> cowboy. We have been going through some tuning/concurrency testing and >> we have improved our results drastically by increasing >> >> ERL_MAX_PORTS to 64000 >> +P parameter to a very high number , around 128000 >> Kernel tuning >> >> With this config on a medium VM we have been able to successfully get >> to around 60,000 connections, up from 4096 so we are headed in the right >> direction :-) >> > > Well then with the above configuration you're at about the maximum > concurrent connections you can get. If you increase ERL_MAX_PORTS and +P > even more Cowboy shouldn't have any trouble giving you even better results. > > > MY next question is with Cowboy tuning. I notice that the default >> max_connections parameter on the TCP transport is 1024 if not set, but I >> am not hitting this issue. What is the right way to tune NBAcceptor >> pool and the the max_connections setting for the acceptor ? >> > > NBAcceptor shouldn't be too low but shouldn't be too high either. I found > 100 to be a good values but it really depends on how many connections come > at once, how many cores you have, how many concurrent connections you > expect, and so on. > > Right now my start_listener is simply >> >> cowboy:start_listener(my_http_**listener, 200, >> cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}], >> cowboy_http_protocol, [{dispatch, Routes}] >> ). >> >> but i have seen some examples with >> >> cowboy:start_listener(my_http_**listener, 200, >> cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}, >> */{max_connections, infinity}/* ], >> cowboy_http_protocol, [{dispatch, Routes}] >> ). >> > > If you're using Cowboy 0.6.1 or below and looking for performance, you > should set max_connections to infinity. If you're using master, you can let > it at the default value. > > Websocket connections are not counted in that number. All HTTP connections > are though, so you might want to look at cowboy_listener to see how to > remove the connection from the count (varies depending on your version). > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solomon.wzs@REDACTED Wed Oct 10 12:42:45 2012 From: solomon.wzs@REDACTED (Solomon) Date: Wed, 10 Oct 2012 18:42:45 +0800 Subject: [erlang-questions] The short name effect memory used Message-ID: Hi. I bulid a web server with mochiweb, I start server with: *erl -pa ebin edit deps/*/ebin -sname my_webserver_dev -s my_webserver* The server used about 1GB memory. But when I start server with: *erl -pa ebin edit deps/*/ebin -sname my_webserver_ -s my_webserver* It just used 20MB memory. What is the problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Wed Oct 10 13:51:36 2012 From: essen@REDACTED (=?UTF-8?B?TG/Dr2MgSG9ndWlu?=) Date: Wed, 10 Oct 2012 13:51:36 +0200 Subject: [erlang-questions] Cowboy , Concurrency , Tuning In-Reply-To: References: <50753381.1050603@ninenines.eu> Message-ID: <50756148.10204@ninenines.eu> In 0.6.1 there is a separate pool for websockets but it was never fully implemented (never requested) so you can't easily set the limit. Instead in master this was removed, they just aren't counted, and the application can just decide how to handle it. The point is that we want to limit connections which are going to use CPU but not the ones that are going to spend most of their time in a receive loop, probably hibernating. On 10/10/2012 12:32 PM, AD wrote: > Thanks Loic. Any reason why websocket connections are not counted ? is > there a separate pool for max_connections for websockets? > > On Wed, Oct 10, 2012 at 4:36 AM, Lo?c Hoguin > wrote: > > On 10/10/2012 04:02 AM, AD wrote: > > hello, > > > Hey, > > > I currently have a SockJS websocket implementation sitting on > top of > cowboy. We have been going through some tuning/concurrency > testing and > we have improved our results drastically by increasing > > ERL_MAX_PORTS to 64000 > +P parameter to a very high number , around 128000 > Kernel tuning > > With this config on a medium VM we have been able to > successfully get > to around 60,000 connections, up from 4096 so we are headed in > the right > direction :-) > > > Well then with the above configuration you're at about the maximum > concurrent connections you can get. If you increase ERL_MAX_PORTS > and +P even more Cowboy shouldn't have any trouble giving you even > better results. > > > MY next question is with Cowboy tuning. I notice that the > default > max_connections parameter on the TCP transport is 1024 if not > set, but I > am not hitting this issue. What is the right way to tune NBAcceptor > pool and the the max_connections setting for the acceptor ? > > > NBAcceptor shouldn't be too low but shouldn't be too high either. I > found 100 to be a good values but it really depends on how many > connections come at once, how many cores you have, how many > concurrent connections you expect, and so on. > > Right now my start_listener is simply > > cowboy:start_listener(my_http___listener, 200, > cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}], > cowboy_http_protocol, [{dispatch, Routes}] > ). > > but i have seen some examples with > > cowboy:start_listener(my_http___listener, 200, > cowboy_tcp_transport, [{port, ?COWBOY_HTTP_PORT}, > */{max_connections, infinity}/* ], > cowboy_http_protocol, [{dispatch, Routes}] > ). > > > If you're using Cowboy 0.6.1 or below and looking for performance, > you should set max_connections to infinity. If you're using master, > you can let it at the default value. > > Websocket connections are not counted in that number. All HTTP > connections are though, so you might want to look at cowboy_listener > to see how to remove the connection from the count (varies depending > on your version). > > -- > Lo?c Hoguin > Erlang Cowboy > Nine Nines > http://ninenines.eu > > -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From bengt.kleberg@REDACTED Wed Oct 10 14:41:40 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Wed, 10 Oct 2012 14:41:40 +0200 Subject: [erlang-questions] beginner: How to debug run_erl pipe problems? Message-ID: <1349872900.4003.97.camel@sekic1152.rnd.ki.sw.ericsson.se> Greetings, I can start Erlang with run_erl, but there are no pipes in /tmp/. How do I find out what goes wrong? Computer: Linux sekic1152 2.6.27.42-0.1-default #1 SMP 2010-01-06 16:07:25 +0100 x86_64 x86_64 x86_64 GNU/Linux Erlang: (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 5.9.1.2 bengt From gianfranco.alongi@REDACTED Wed Oct 10 14:46:43 2012 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Wed, 10 Oct 2012 14:46:43 +0200 Subject: [erlang-questions] beginner: How to debug run_erl pipe problems? In-Reply-To: <1349872900.4003.97.camel@sekic1152.rnd.ki.sw.ericsson.se> References: <1349872900.4003.97.camel@sekic1152.rnd.ki.sw.ericsson.se> Message-ID: Do you see nothing in the logs directory? Also, how do you issue the command? On Wed, Oct 10, 2012 at 2:41 PM, Bengt Kleberg wrote: > Greetings, > > I can start Erlang with run_erl, but there are no pipes in /tmp/. How do > I find out what goes wrong? > > Computer: Linux sekic1152 2.6.27.42-0.1-default #1 SMP 2010-01-06 > 16:07:25 +0100 x86_64 x86_64 x86_64 GNU/Linux > > Erlang: (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 5.9.1.2 > > > bengt > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bengt.kleberg@REDACTED Wed Oct 10 14:50:10 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Wed, 10 Oct 2012 14:50:10 +0200 Subject: [erlang-questions] beginner: How to debug run_erl pipe problems? In-Reply-To: References: <1349872900.4003.97.camel@sekic1152.rnd.ki.sw.ericsson.se> Message-ID: <1349873410.4003.100.camel@sekic1152.rnd.ki.sw.ericsson.se> The command: run_erl -daemon /tmp/ /var/tmp/ "exec erl -detached" The log dir: -rw-rw-r-- 1 eleberg avall 246 Oct 10 14:34 run_erl.log -rw-rw-r-- 1 eleberg avall 130 Oct 10 14:34 erlang.log.1 No information about errors or pipes in the files. bengt On Wed, 2012-10-10 at 14:46 +0200, Gianfranco Alongi wrote: > Do you see nothing in the logs directory? > > Also, how do you issue the command? > > On Wed, Oct 10, 2012 at 2:41 PM, Bengt Kleberg > wrote: > > Greetings, > > > > I can start Erlang with run_erl, but there are no pipes in /tmp/. How do > > I find out what goes wrong? > > > > Computer: Linux sekic1152 2.6.27.42-0.1-default #1 SMP 2010-01-06 > > 16:07:25 +0100 x86_64 x86_64 x86_64 GNU/Linux > > > > Erlang: (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 5.9.1.2 > > > > > > bengt > > > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions From gianfranco.alongi@REDACTED Wed Oct 10 15:01:07 2012 From: gianfranco.alongi@REDACTED (Gianfranco Alongi) Date: Wed, 10 Oct 2012 15:01:07 +0200 Subject: [erlang-questions] beginner: How to debug run_erl pipe problems? In-Reply-To: <1349873410.4003.100.camel@sekic1152.rnd.ki.sw.ericsson.se> References: <1349872900.4003.97.camel@sekic1152.rnd.ki.sw.ericsson.se> <1349873410.4003.100.camel@sekic1152.rnd.ki.sw.ericsson.se> Message-ID: Try without detached. On Wed, Oct 10, 2012 at 2:50 PM, Bengt Kleberg wrote: > The command: > run_erl -daemon /tmp/ /var/tmp/ "exec erl -detached" > > The log dir: > -rw-rw-r-- 1 eleberg avall 246 Oct 10 14:34 run_erl.log > -rw-rw-r-- 1 eleberg avall 130 Oct 10 14:34 erlang.log.1 > > No information about errors or pipes in the files. > > bengt > > On Wed, 2012-10-10 at 14:46 +0200, Gianfranco Alongi wrote: >> Do you see nothing in the logs directory? >> >> Also, how do you issue the command? >> >> On Wed, Oct 10, 2012 at 2:41 PM, Bengt Kleberg >> wrote: >> > Greetings, >> > >> > I can start Erlang with run_erl, but there are no pipes in /tmp/. How do >> > I find out what goes wrong? >> > >> > Computer: Linux sekic1152 2.6.27.42-0.1-default #1 SMP 2010-01-06 >> > 16:07:25 +0100 x86_64 x86_64 x86_64 GNU/Linux >> > >> > Erlang: (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 5.9.1.2 >> > >> > >> > bengt >> > >> > >> > _______________________________________________ >> > erlang-questions mailing list >> > erlang-questions@REDACTED >> > http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From bengt.kleberg@REDACTED Wed Oct 10 15:20:44 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Wed, 10 Oct 2012 15:20:44 +0200 Subject: [erlang-questions] beginner: How to debug run_erl pipe problems? In-Reply-To: References: <1349872900.4003.97.camel@sekic1152.rnd.ki.sw.ericsson.se> <1349873410.4003.100.camel@sekic1152.rnd.ki.sw.ericsson.se> Message-ID: <1349875244.4003.101.camel@sekic1152.rnd.ki.sw.ericsson.se> That works. Thank you. bengt On Wed, 2012-10-10 at 15:01 +0200, Gianfranco Alongi wrote: > Try without detached. > > On Wed, Oct 10, 2012 at 2:50 PM, Bengt Kleberg > wrote: > > The command: > > run_erl -daemon /tmp/ /var/tmp/ "exec erl -detached" > > > > The log dir: > > -rw-rw-r-- 1 eleberg avall 246 Oct 10 14:34 run_erl.log > > -rw-rw-r-- 1 eleberg avall 130 Oct 10 14:34 erlang.log.1 > > > > No information about errors or pipes in the files. > > > > bengt > > > > On Wed, 2012-10-10 at 14:46 +0200, Gianfranco Alongi wrote: > >> Do you see nothing in the logs directory? > >> > >> Also, how do you issue the command? > >> > >> On Wed, Oct 10, 2012 at 2:41 PM, Bengt Kleberg > >> wrote: > >> > Greetings, > >> > > >> > I can start Erlang with run_erl, but there are no pipes in /tmp/. How do > >> > I find out what goes wrong? > >> > > >> > Computer: Linux sekic1152 2.6.27.42-0.1-default #1 SMP 2010-01-06 > >> > 16:07:25 +0100 x86_64 x86_64 x86_64 GNU/Linux > >> > > >> > Erlang: (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 5.9.1.2 > >> > > >> > > >> > bengt > >> > > >> > > >> > _______________________________________________ > >> > erlang-questions mailing list > >> > erlang-questions@REDACTED > >> > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions From yann.secq@REDACTED Wed Oct 10 15:27:47 2012 From: yann.secq@REDACTED (Yann SECQ) Date: Wed, 10 Oct 2012 15:27:47 +0200 Subject: [erlang-questions] Erlang in distributed systems research In-Reply-To: References: Message-ID: <507577D3.30405@univ-lille1.fr> On 10/10/12 11:55 AM, Andr? Graf wrote: > Dear list >... > Besides of other things I want to talk about some examples of Erlang > in research that might fit the topic. Do you have any pointers on that > one? Dear M. Graf, some works have been done in the past in the field of multi-agent systems: - http://www.erlang.se/euc/05/Santoro.pdf - https://github.com/gleber/exat But, I do not know a lot of others examples (at least within MAS domain). Some publications on Scala by Oderski makes references to erlang, but I'm not sure it fulfills your request. Cheers, yann. -- Contact: yann.secq{at}univ-lille1.fr | www.lifl.fr/~secq www.ouverture-independance.fr | www.sauvonsluniversite.com -- "Ne d?sesp?rez jamais. Faites infuser davantage.", Henri Michaux From desired.mta@REDACTED Wed Oct 10 16:08:50 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Wed, 10 Oct 2012 15:08:50 +0100 Subject: [erlang-questions] [PATCH] Fix R15B02 cross-compilation for TileraMDE-3.0.1.125620 Message-ID: <20121010140818.GA17965@precise> Applying the patch below let me cross-compile and run R15B02 on Tilera64 using TileraMDE-3.0.1.125620. In order to prepare a clean patch, I have to know what do the options -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off do on the machines that support it? Also, what is the meaning of -MALLOC_USE_HASH(1)? Overall, thumbs up to the OTP team for making cross compilation very easy. :) --- -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off tile-gcc 4.4.3 does accept these options: cc1: error: unrecognized command line option "-WOPT:lpre=off:spre=off:epre=off" MALLOC_USE_HASH is not defined anywhere in the mentioned build environment. --- erts/emulator/Makefile.in | 1 - erts/emulator/sys/unix/sys.c | 6 ------ 2 files changed, 7 deletions(-) diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index 985ef72..644ca18 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -633,7 +633,6 @@ endif ifneq ($(filter tile-%,$(TARGET)),) $(OBJDIR)/beam_emu.o: beam/beam_emu.c $(CC) $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) \ - -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off \ $(INCLUDES) -c $< -o $@ else # Usually the same as the default rule, but certain platforms (e.g. win32) mix diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index c1fa00b..4c168b6 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -398,12 +398,6 @@ void sys_tty_reset(int exit_code) } } -#ifdef __tile__ -/* Direct malloc to spread memory around the caches of multiple tiles. */ -#include -MALLOC_USE_HASH(1); -#endif - #ifdef USE_THREADS #ifdef ERTS_THR_HAVE_SIG_FUNCS -- 1.7.9.5 From olivier.boudeville@REDACTED Wed Oct 10 16:50:31 2012 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Wed, 10 Oct 2012 16:50:31 +0200 Subject: [erlang-questions] RE Erlang in distributed systems research In-Reply-To: Message-ID: Hi Andr?, It is certainly a shameless plug, but Sim-Diasca (http://www.sim-diasca.com), which is a parallel and distributed discrete simulation engine entirely written in Erlang, seems to correspond. We indeed used it to perform the detailed simulation of industrial distributed systems that are both complex and of very large-scale: we applied Sim-Diasca to smart metering infrastructures, as discussed in http://www.cleveronline.org/ So I suppose it could be another fairly good example. Hope this helps, Best regards, Olivier Boudeville. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 andre.graf@REDACTED Envoy? par : erlang-questions-bounces@REDACTED 10/10/2012 11:55 A erlang-questions@REDACTED cc Objet [erlang-questions] Erlang in distributed systems research Dear list I have the opportunity to give a talk about Erlang to the folks from the database and information systems research group at the University Basel. The group heavily uses Java for the prototypes they develop. We all know that they can benefit from Erlang and that's what I want the take-home message to be. Besides of other things I want to talk about some examples of Erlang in research that might fit the topic. Do you have any pointers on that one? Cheers Andr? _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From egil@REDACTED Wed Oct 10 17:09:10 2012 From: egil@REDACTED (=?ISO-8859-1?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Wed, 10 Oct 2012 17:09:10 +0200 Subject: [erlang-questions] [PATCH] Fix R15B02 cross-compilation for TileraMDE-3.0.1.125620 In-Reply-To: <20121010140818.GA17965@precise> References: <20121010140818.GA17965@precise> Message-ID: <50758F96.5090200@erlang.org> On 2012-10-10 16:08, Motiejus Jaks(tys wrote: > Applying the patch below let me cross-compile and run R15B02 on Tilera64 > using TileraMDE-3.0.1.125620. > > In order to prepare a clean patch, I have to know what do the options > -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off do on the machines that > support it? Also, what is the meaning of -MALLOC_USE_HASH(1)? I am quoting from Tilera here: "'malloc_use_hash' only affects malloc (specifically, it affects the flags it passes to 'mmap' when it allocates more memory from Linux). You can also explicitly malloc memory with various caching attributes using custom mspaces. The 'hash_default' argument in the .hvconfig file affects 'mmap' calls that do not specify homing information. Because malloc*does* specify homing information, at least when you are pthreaded, 'hash_default' is ignored and malloc_use_hash is required to get hash-for-home." This was something included in TileraMDE 2.0 beta (perhaps removed since then). I found it in TileraMDE-2.1.2/tile64/tile/usr/include/malloc.h though. I guess it would be safe to remove "-OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off ". It is an optimization for tile-cc when compiling on target machine. An earlier comment in Makefile.in seems to have been removed, it said: +ifneq ($(filter tile-%,$(TARGET)),) +# Some tile-cc optimizations take pathologically long for beam_emu.c, +# so disable them. +$(OBJDIR)/beam_emu.o: beam/beam_emu.c + $(CC) $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) \ + -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off \ + $(INCLUDES) -c $< -o $@ +endif Patch was supplied by Tilera Corp. Regards, Bj?rn-Egil Dahlberg Erlang/OTP > > Overall, thumbs up to the OTP team for making cross compilation very easy. > :) > > --- > > -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off > tile-gcc 4.4.3 does accept these options: > > cc1: error: unrecognized command line option > "-WOPT:lpre=off:spre=off:epre=off" > > MALLOC_USE_HASH is not defined anywhere in the mentioned build > environment. > --- > erts/emulator/Makefile.in | 1 - > erts/emulator/sys/unix/sys.c | 6 ------ > 2 files changed, 7 deletions(-) > > diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in > index 985ef72..644ca18 100644 > --- a/erts/emulator/Makefile.in > +++ b/erts/emulator/Makefile.in > @@ -633,7 +633,6 @@ endif > ifneq ($(filter tile-%,$(TARGET)),) > $(OBJDIR)/beam_emu.o: beam/beam_emu.c > $(CC) $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) \ > - -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off \ > $(INCLUDES) -c $< -o $@ > else > # Usually the same as the default rule, but certain platforms (e.g. win32) mix > diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c > index c1fa00b..4c168b6 100644 > --- a/erts/emulator/sys/unix/sys.c > +++ b/erts/emulator/sys/unix/sys.c > @@ -398,12 +398,6 @@ void sys_tty_reset(int exit_code) > } > } > > -#ifdef __tile__ > -/* Direct malloc to spread memory around the caches of multiple tiles. */ > -#include > -MALLOC_USE_HASH(1); > -#endif > - > #ifdef USE_THREADS > > #ifdef ERTS_THR_HAVE_SIG_FUNCS -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody@REDACTED Wed Oct 10 18:06:31 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Wed, 10 Oct 2012 21:36:31 +0530 Subject: [erlang-questions] suitability of erlang In-Reply-To: <20121010101533.GA4221@corelatus.se> References: <20121010101533.GA4221@corelatus.se> Message-ID: On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang wrote: > On Wednesday, October 10, Rustom Mody wrote: > > Another thing: you write "under severe time-crunch", but that could > mean "we need to ship a product in a very short time" or it could mean > "the data has to be processed in a very short time". > > Matt > Well mainly the latter But the former is also true. (When is it ever not :-)) My concern is that there is some desire to try out erlang which may be fine... However I am wondering whether the difference between concurrency and parallelism has not been appreciated. http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ is a haskell implementers view of the terminology I wonder where erlang fits in on this Rusi -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Wed Oct 10 18:47:54 2012 From: ulf@REDACTED (Ulf Wiger) Date: Wed, 10 Oct 2012 18:47:54 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: <85AD019E-3FFD-4FC2-B3CB-C9B1613B22AC@feuerlabs.com> Simon Marlow is a very smart guy, and knows what he's talking about. Erlang doesn't really target the domain of parallelism, but to some extent, you can roll your own. Your mileage will depend in part on how heavy your jobs are, and there are some other interesting differences between the Haskell approach and Erlang's approach. - While you can do something similar to Erlang-style concurrency in Haskell, to a certain extent this has been a research area. Nowadays, you can play with Cloud Haskell [1], which closely mimicks Erlang's semantics (well, actually a proposed semantics for 'future Erlang'). Not least exception handling has long been a tricky issue to figure out, due to Haskell's strict type system. - Haskell's main focus so far (I believe) has been what Simon calls deterministic parallelism. There is quite some promise that Haskell's type system will keep you sane when trying this style of programming, whereas in Erlang, you tend to end up treating it as an explicit coordination problem - quite doable, and better than in most other languages, but more error-prone than it could be in Haskell. - Erlang's origin is telecoms, where systems are supposed to serve and evolve for many years in the field. This makes it important to support code evolution - given telecom uptime requirements, notably code evolution in a _running system_. Erlang's support for in-service code change is difficult to reconcile with a strict static type system, and I believe this feature is also absent from Cloud Haskell (at least I don't see it mentioned). If you're building server systems, this is a feature to die for. - Haskell is a superb research language, and it has seen some success in commercial settings, but primarily in areas where it has been possible to maintain a very high skill level across the board in the design teams (I'm speculating here; I haven't actually verified it). Erlang, in contrast, has been picked up by companies across the globe and used in commercial products pretty much after self-study and experimentation. You might say (although (the) Simon(s) might flog me for this), that Erlang is quite easy to get into, whereas Haskell becomes easy once you've finished your PhD in Functional Programming. :) - So basically, some of the smartest programmers in the world choose Haskell as their weapon, and much of the cutting-edge programming language research today is done in Haskell. Erlang is a great craftsman's tool for building robust distributed systems, with many of the niceties of Haskell, but much less of the academic rigour. When deep in the trenches, this is not always a bad thing. ;-) BR, Ulf W [1] http://hackage.haskell.org/package/distributed-process On 10 Oct 2012, at 18:06, Rustom Mody wrote: > On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang wrote: > On Wednesday, October 10, Rustom Mody wrote: > > Another thing: you write "under severe time-crunch", but that could > mean "we need to ship a product in a very short time" or it could mean > "the data has to be processed in a very short time". > > Matt > > Well mainly the latter > But the former is also true. (When is it ever not :-)) > My concern is that there is some desire to try out erlang which may be fine... However I am wondering whether the difference between concurrency and parallelism has not been appreciated. > http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ > is a haskell implementers view of the terminology > I wonder where erlang fits in on this > > Rusi > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From raould@REDACTED Wed Oct 10 18:55:27 2012 From: raould@REDACTED (Raoul Duke) Date: Wed, 10 Oct 2012 09:55:27 -0700 Subject: [erlang-questions] Erlang in distributed systems research In-Reply-To: References: Message-ID: very ivory tower at the moment, but i am excited about: https://www.google.com/search?q=clusterken From raould@REDACTED Wed Oct 10 18:57:10 2012 From: raould@REDACTED (Raoul Duke) Date: Wed, 10 Oct 2012 09:57:10 -0700 Subject: [erlang-questions] Erlang in distributed systems research In-Reply-To: <507577D3.30405@univ-lille1.fr> References: <507577D3.30405@univ-lille1.fr> Message-ID: On Wed, Oct 10, 2012 at 6:27 AM, Yann SECQ wrote: > But, I do not know a lot of others examples (at least within > MAS domain). Some publications on Scala by Oderski makes > references to erlang, but I'm not sure it fulfills your request. https://www.google.com/search?q=erlang+vs.+c%2B%2B+telecommunications might be of relevance? if people didn't already know about it... From erlang@REDACTED Wed Oct 10 20:30:32 2012 From: erlang@REDACTED (Joe Armstrong) Date: Wed, 10 Oct 2012 20:30:32 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody wrote: > On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang > wrote: >> >> On Wednesday, October 10, Rustom Mody wrote: > > >> >> Another thing: you write "under severe time-crunch", but that could >> mean "we need to ship a product in a very short time" or it could mean >> "the data has to be processed in a very short time". >> >> Matt > > > Well mainly the latter > But the former is also true. (When is it ever not :-)) > My concern is that there is some desire to try out erlang which may be > fine... However I am wondering whether the difference between concurrency > and parallelism has not been appreciated. Very true - I have given several lectures on this ... many people use the words concurrent and parallel as synonyms - but there is a real difference. In a nutshell - parallel has to do with real hardware than can do several things "at the same time" whereas concurrency is a software structuring abstraction which pretends that different parts of the software can run at the same time (if suitable hardware were available) Parallel has to do with hardware - on a single cpu you can write your programs using pmap (instead of map) and parBegin and parEnd and it won't make any difference since the machine is still only doing one instruction at a time. It's only because machine are fast that we think they do several things in parallel - in the single CPU + time sharing case if we slowed the clock down we would see that no parallelism is involved - the CPU does one thing a time - even when executing concurrent programs :-) There are only a few real sources of true parallelism - multi-cores, what happens in the pipeline, and how data is fetched/stored Multicores do give you true parallelism, but you must be careful to about the granularity of computation - ie it should not be more work to shift the work to a remote CPU than do it locally. Pipeline messing is old for compiler writers. Data fetching and storing can be done in parallel, this needs a bit of thought. Parallel programs are inherently non portable - in the sense that they specifically depend upon details of the hardware. Concurrent programs are program written using a concurrency abstraction. In Erlang processes are the unti of concurrency - processes are pretty coarse-grain things ie they take a wee bit of effort to get going, so they actually map well onto multi-cores, even if you can detect parallelism in your program (easy in a pure FP, just evaluate all the arguments to a function in parallel) - it's pretty difficult (ie an open research problem) to map this onto the available hardware. Even if you know how long computations take, and you know what resources you have to solve the computations, then mapping the computations onto the resources involves solving the knapsack problem which is NP hard. It you write program in Erlang you at least have a head start over sequential code since the programmer has at least said (using processes) which activities should be run in parallel. Optimally scheduling these is NP hard - but non-optimal first on demand, best effort scheduling works pretty well for non-pathological code. My 10 c. /Joe > http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ > is a haskell implementers view of the terminology > I wonder where erlang fits in on this > > Rusi > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From rickard@REDACTED Wed Oct 10 21:12:23 2012 From: rickard@REDACTED (Rickard Green) Date: Wed, 10 Oct 2012 21:12:23 +0200 Subject: [erlang-questions] +swt very_low doesn't seem to avoid schedulers getting Message-ID: <5075C897.5070002@erlang.org> > Hi, all. According to my private mailing list archive, there hasn't > been much mention of erl's "+swt" flag since about April 2012. > > I just witnessed a case of where using "+swt very_low" with Riak > 1.2.1rc2, using Erlang/OTP R15B01, get "stuck" where CPU consumption was > only 200% on an 8 core AWS instance. The other nodes in that Riak > cluster were running at over 600% CPU utilization (on average). > > When I ran this: > > {io:format("before..."), erlang:system_flag(schedulers_online, 1), > timer:sleep(1000), erlang:system_flag(schedulers_online, 8), > io:format("after\n")}. > > ... then average CPU utilization on that node immediately shot up from > 200% to about 760%. This is very much expected. Since you have work that can load 2 schedulers full time and shuts down all but one, the run-queue will grow. When you later release all schedulers, there will be lots of work to pick from. > > I'd heard a rumor that "+swt very_low" was supposed to avoid whatever > weird scheduler problem/bug that caused some schedulers to appear as if > they weren't active. But I was on site today and witnessed this > first-hand and verified that the "+swt very_low" flag was indeed being > used. > The runtime system tries to compact the load on as few schedulers as possible without getting run-queues that build up. The runtime system wont wake up new schedulers unless some overload has accumulated. This overload either show up as a quickly growing run-queue or a small run-queue over a longer time. The +swt flags sets the threshold that is used for determining when enough overload has accumulated to wake up another scheduler. This compaction of load onto fewer schedulers is there in order to reduce communication overhead when there aren't enough work to fully utilize all schedulers. The performance gain of this compaction depends on the hardware. We have gotten reports about problems with this functionality, but we have not found any bugs in this functionality. We have only found that it behaves as expected. That is, if more schedulers aren't woken this is due to not enough accumulated overload. The +swt switch was introduced in order to give the user the possibility do define what is enough overload for his or her taste. The currently used wakeup strategy is very quick to forget about previously accumulated overload that has disappeared. Maybe even too quick for my taste when "+swt very_low" is used. I've therefore implemented an alternative strategy that most likely will be the default in R16. As of R15B02 you can try this strategy out by passing "+sws proposal" as a command line argument. In combination with "+swt very_low", the runtime system should be even more eager to wake up schedulers than when only using "+swt very_low". > I'm not certain what exact Linux distribution and kernel was used. I'll > ask the customer to send me that info so I can forward it to the list. > > Has anyone else seen this behavior? Unlike Knut Nesheim's report on > this list back in February 2012, Riak does not use the halfword > emulator. We are using some NIFs, but this customer isn't using the > most evil one, the Riak eleveldb NIF library. Instead, they're using > the Bitcask backend (which has a NIF component but isn't as evil as > eleveldb's NIF) and the merge_index backend (which is pure Erlang). > > -Scott Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. From rickard@REDACTED Wed Oct 10 21:12:33 2012 From: rickard@REDACTED (Rickard Green) Date: Wed, 10 Oct 2012 21:12:33 +0200 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. Message-ID: <5075C8A1.9020706@erlang.org> >>From time to time erl_drv_thread_join returns error EDEADLK=35, i.e. the current thread (scheduler thread) tries to join itself. > > According to the documentation ?A Thread identifier may be reused very quickly after a thread has terminated. Therefore, if a thread corresponding to one of the involved thread identifiers has terminated since the thread identifier was saved, the result of erl_drv_equal_tids() might not give the expected result.? > > I suppose that thread terminates earlier then erl_drv_thread_join call happens, so ErlDrvTid is already reused. > This reuse will not cause problems for erl_drv_thread_join() as long as it is used correctly (a tid wont be reused until after the thread has been joined). erl_drv_thread_join() will also refuse to join threads not created by erl_drv_thread_create(), and would in case the scheduler thread tried to join itself fail with EINVAL. > So the question is how to use erl_drv_thread_join properly and how to guarantee that the saved ErlDrvTid value points to the same data that was returned from erl_drv_thread_create? > It is important that the thread is joined once and *only* once. Are you sure that you don't do two calls to erl_drv_thread_join() for the same thread? Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. From enewhuis@REDACTED Wed Oct 10 21:37:37 2012 From: enewhuis@REDACTED (Eric Newhuis) Date: Wed, 10 Oct 2012 14:37:37 -0500 Subject: [erlang-questions] Bounded Buffer Problems/Solutions Message-ID: <022F4CD3-4A8F-4B38-8A5B-41A484B00A72@gmail.com> What are the generally accepted solutions for when producers need to be throttled? gen_server as consumer or otherwise? Are there any good solutions or philosophy for the bounded buffer problem in Erlang? ..flow control? Freaky out-of-band communication at a distance to limit messages sent? Use the idea of rope: Message in reverse to a producer shim that he is allowed to send N more messages before he needs to ask for the right to send more? Other? Are there common Erlang idioms for monitoring execution time at the gen_server level so that an API can respond upward with a policy of "the system is currently busier than a one legged man in a kicking contest?" From charleshixsn@REDACTED Thu Oct 11 00:55:46 2012 From: charleshixsn@REDACTED (Charles Hixson) Date: Wed, 10 Oct 2012 15:55:46 -0700 Subject: [erlang-questions] Process Dictionary limitations?? Message-ID: <5075FCF2.6060502@earthlink.net> I'm choosing a language to implement a ... well, neural network is wrong, and so is cellular automaton, but it gives the idea. Anyway, I'm going to need, in each cell, a few stateful items, e.g. activation level. When I look at what Erlang can do, I see that the Process Dictionary looks as if it would serve my needs, but then I am immediately warned not to use it, that it will cause bugs. These stateful terms will not be exported from the cell within which they are resident. Is this still likely to cause problems? Is there some better approach to maintaining state? (I can't just generate a new process, because other cells will need to know how to access this one, or to test that it has been rolled out.) -- Charles Hixson From mjtruog@REDACTED Thu Oct 11 01:21:29 2012 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 10 Oct 2012 16:21:29 -0700 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <5075FCF2.6060502@earthlink.net> References: <5075FCF2.6060502@earthlink.net> Message-ID: <507602F9.7080005@gmail.com> On 10/10/2012 03:55 PM, Charles Hixson wrote: > I'm choosing a language to implement a ... well, neural network is wrong, and so is cellular automaton, but it gives the idea. Anyway, I'm going to need, in each cell, a few stateful items, e.g. activation level. > > When I look at what Erlang can do, I see that the Process Dictionary looks as if it would serve my needs, but then I am immediately warned not to use it, that it will cause bugs. These stateful terms will not be exported from the cell within which they are resident. Is this still likely to cause problems? Is there some better approach to maintaining state? (I can't just generate a new process, because other cells will need to know how to access this one, or to test that it has been rolled out.) > This explains some basics about the process dictionary: http://www.erlang.org/course/advanced.html#dict Quoted below: * Destroys referential transparency * Makes debugging difficult * Survives Catch/Throw So, it is much better to use variables, so side-effects are more explicit (i.e., function variables). This is the equivalent to the State variable of a gen_server behaviour (http://www.erlang.org/doc/man/gen_server.html). Depending on the expected state-handling, you might want a gen_server, a gen_event, or a gen_fsm for each cell. Otherwise, if you want to avoid OTP behaviour usage, you could just do plain Erlang code, but your code might then be more error-prone (especially since you are asking this question). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Thu Oct 11 01:35:28 2012 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 10 Oct 2012 16:35:28 -0700 Subject: [erlang-questions] Bounded Buffer Problems/Solutions In-Reply-To: <022F4CD3-4A8F-4B38-8A5B-41A484B00A72@gmail.com> References: <022F4CD3-4A8F-4B38-8A5B-41A484B00A72@gmail.com> Message-ID: <50760640.3010807@gmail.com> On 10/10/2012 12:37 PM, Eric Newhuis wrote: > What are the generally accepted solutions for when producers need to be throttled? gen_server as consumer or otherwise? > > Are there any good solutions or philosophy for the bounded buffer problem in Erlang? ..flow control? Freaky out-of-band communication at a distance to limit messages sent? Use the idea of rope: Message in reverse to a producer shim that he is allowed to send N more messages before he needs to ask for the right to send more? Other? > > Are there common Erlang idioms for monitoring execution time at the gen_server level so that an API can respond upward with a policy of "the system is currently busier than a one legged man in a kicking contest?" I haven't seen a generic chunk of Erlang code that would handle all throttling needs because I think it depends on the situation, so the source code is generally ad-hoc. However, I do think that in the situation that many Erlang process act as producers with 1 or more Erlang process acting as a consumer, you can use the process dictionary in a valid way (that most people wouldn't be too angry about). If you use a unique atom as the process dictionary key in each producing process that contains a timestamp/count tuple, you are able to do a simple rate limiter that is non-intrusive to the other source code. I have done this for async logging, to prevent flooding processes here: https://github.com/okeuday/CloudI/blob/master/src/lib/cloudi/src/cloudi_logger.erl#L362 Using the process dictionary helps to avoid checking with a message, in the consuming process. Also, checking in the producing process could be erroneous if all producing cases are not handled properly. Throttling generally wants temporary storage which makes it a problem better suited for the process dictionary than ets (and you can avoid global locking on ets tables). From charleshixsn@REDACTED Thu Oct 11 03:40:29 2012 From: charleshixsn@REDACTED (Charles Hixson) Date: Wed, 10 Oct 2012 18:40:29 -0700 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <507602F9.7080005@gmail.com> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> Message-ID: <5076238D.1080004@earthlink.net> On 10/10/2012 04:21 PM, Michael Truog wrote: > On 10/10/2012 03:55 PM, Charles Hixson wrote: >> I'm choosing a language to implement a ... well, neural network is >> wrong, and so is cellular automaton, but it gives the idea. Anyway, >> I'm going to need, in each cell, a few stateful items, e.g. >> activation level. >> >> When I look at what Erlang can do, I see that the Process Dictionary >> looks as if it would serve my needs, but then I am immediately warned >> not to use it, that it will cause bugs. These stateful terms will >> not be exported from the cell within which they are resident. Is >> this still likely to cause problems? Is there some better approach >> to maintaining state? (I can't just generate a new process, because >> other cells will need to know how to access this one, or to test that >> it has been rolled out.) >> > This explains some basics about the process dictionary: > http://www.erlang.org/course/advanced.html#dict > Quoted below: > > * Destroys referential transparency > * Makes debugging difficult > * Survives Catch/Throw > > So, it is much better to use variables, so side-effects are more > explicit (i.e., function variables). This is the equivalent to the > State variable of a gen_server behaviour > (http://www.erlang.org/doc/man/gen_server.html). Depending on the > expected state-handling, you might want a gen_server, a gen_event, or > a gen_fsm for each cell. Otherwise, if you want to avoid OTP > behaviour usage, you could just do plain Erlang code, but your code > might then be more error-prone (especially since you are asking this > question). > Thank you. That confirms the recommendation against using the process directory, though I will admit that I can't see any way that your proposed alternatives could replace it. -- Charles Hixson -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Thu Oct 11 04:07:58 2012 From: mjtruog@REDACTED (Michael Truog) Date: Wed, 10 Oct 2012 19:07:58 -0700 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <5076238D.1080004@earthlink.net> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> Message-ID: <507629FE.4080003@gmail.com> On 10/10/2012 06:40 PM, Charles Hixson wrote: > On 10/10/2012 04:21 PM, Michael Truog wrote: >> On 10/10/2012 03:55 PM, Charles Hixson wrote: >>> I'm choosing a language to implement a ... well, neural network is wrong, and so is cellular automaton, but it gives the idea. Anyway, I'm going to need, in each cell, a few stateful items, e.g. activation level. >>> >>> When I look at what Erlang can do, I see that the Process Dictionary looks as if it would serve my needs, but then I am immediately warned not to use it, that it will cause bugs. These stateful terms will not be exported from the cell within which they are resident. Is this still likely to cause problems? Is there some better approach to maintaining state? (I can't just generate a new process, because other cells will need to know how to access this one, or to test that it has been rolled out.) >>> >> This explains some basics about the process dictionary: http://www.erlang.org/course/advanced.html#dict >> Quoted below: >> >> * Destroys referential transparency >> * Makes debugging difficult >> * Survives Catch/Throw >> >> So, it is much better to use variables, so side-effects are more explicit (i.e., function variables). This is the equivalent to the State variable of a gen_server behaviour (http://www.erlang.org/doc/man/gen_server.html). Depending on the expected state-handling, you might want a gen_server, a gen_event, or a gen_fsm for each cell. Otherwise, if you want to avoid OTP behaviour usage, you could just do plain Erlang code, but your code might then be more error-prone (especially since you are asking this question). >> > Thank you. That confirms the recommendation against using the process directory, though I will admit that I can't see any way that your proposed alternatives could replace it. > Usually the State variable in the OTP behaviours is always a record, which is preprocessing syntactical sugar for tuples (similar to a struct in C, conceptually). So, it provides constant time (O(1)) access to tuple elements based on field names (when reading elements, setting elements has more overhead... all memory is copied since variables are immutable in Erlang). If you need dynamic field names, there are many options for various key-value data structures. In OTP, dict, gb_trees, orddict (and array if your key type is always an integer). If you need to use dynamic strings as key values, I have a trie here https://github.com/okeuday/trie. However, it is best (most typical and maintainable) if you can rely on a record (tuple) for process state. Within the state record, you can always define any dynamic lookups you might need or other data structures you wish to utilize. Hope this helps make it clear that the key-value data structures available in separate modules (either within OTP or external to OTP) help to make sure you can create key-value lookups within Erlang code without utilizing the process dictionary. The other direction you can go is to use ets which provides global storage in Erlang, managed by a process. However, it is best to avoid global variables whenever possible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Thu Oct 11 05:38:51 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Wed, 10 Oct 2012 20:38:51 -0700 Subject: [erlang-questions] CT take a long time to warm up In-Reply-To: References: <51E69ECE-7A83-4A91-9F2B-DB6EC47668F3@gmail.com> <20120926163945.GA8916@precise> Message-ID: Ok, I finally managed to track this down but I'm totally unsure why this happens. Motiejus, you were right. When I run REBAR, the process automountd tries to connect on port 111 (sunrpc) on my local NAT. When it times out, the CT tests are actually run. If I simply block this process with a firewall, tests start immediately. I'm at disposal if anyone wants to dig further (tuncer?) r. On Wed, Sep 26, 2012 at 1:38 PM, Tuncer Ayaz wrote: > On Wed, Sep 26, 2012 at 9:44 PM, Motiejus Jak?tys wrote: > > On Wed, Sep 26, 2012 at 8:40 PM, Roberto Ostinelli wrote: > > > > How do you run it? Consistent 20 seconds sounds like a DNS > > > > timeout issue (stab in the dark). > > > > > > > > > rebar ct_suites=mymodule > > > > Hi, > > adding the list. > > > > Please do > > > > $TRACE_F rebar ct_suites=mymodule > > Make that > > $TRACE_F rebar suites=mymodule > > and also consider running rebar with fprof: > > $TRACE_F rebar -p suites=mymodule > or > $TRACE_F rebar --profile suites=mymodule > > , but make sure you update rebar first to avoid a faulty > call to fprof:apply/2. > > If it's caused by name lookup, you can try a custom ERL_INETRC file: > http://www.erlang.org/doc/apps/erts/inet_cfg.html > > > and replace $TRACE_F with Max OS C equivalent of "strace -f". > > Then see what it does just before the 20 second pause. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiish@REDACTED Thu Oct 11 06:20:28 2012 From: efiish@REDACTED (Junior White) Date: Thu, 11 Oct 2012 12:20:28 +0800 Subject: [erlang-questions] Bad Argument, (Bug or something I'm missing). In-Reply-To: References: Message-ID: hi, This behave makes me confused too. On Mon, Oct 1, 2012 at 4:46 PM, Jesse Gumm wrote: > Hi, > > You just need to convert List to a binary first with list_to_binary. > > The <<"something">> syntax is just a friendly way to make binary literals. > But it does not perform automatic casting of lists. > > Further, the extra << >> you're wrapping around "-" is unnecessary. > > Example: > > > L = "123456", > > B = list_to_binary(L), > > <>. > > Will yield: > > <<"123456-">> > > Hope that helps. > > Take it easy, > > -Jesse > > -- > Jesse Gumm > Owner, Sigma Star Systems > 414.940.4866 || sigma-star.com || @jessegumm > On Oct 1, 2012 3:11 AM, "Chris Cook" wrote: > >> Morning list, >> >> I have, >> >> Erlang R15B01 (erts-5.9.1) [source] [async-threads:0] [hipe] >> [kernel-poll:false] >> >> Eshell V5.9.1 (abort with ^G) >> 1> List = "89234789234jhk2hk234892789fauky324978". >> "89234789234jhk2hk234892789fauky324978" >> 2> <>/binary>>. >> ** exception error: bad argument >> >> But when I write it as this, >> >> 3> <<"89234789234jhk2hk234892789fauky324978", <<"-">>/binary>>. >> <<"89234789234jhk2hk234892789fauky324978-">> >> >> I get the result I expected to get from the above 1 & 2. Could someone >> please explain what is going wrong and why, because I'm very confused >> with it. >> >> Regards >> >> Chris Cook. >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Thu Oct 11 06:31:44 2012 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 11 Oct 2012 17:31:44 +1300 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <507629FE.4080003@gmail.com> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> Message-ID: You can think of the process dictionary this way: If there is a function clause f(X, Y, Z) -> A = g(X, Y), B = h(Z, A), q(A, B) replace it by f(X, Y, Z, D0) -> {A, D1} = g(X, Y, D0), {B, D2} = h(Z, A, D1), q(A, B, D2) and replace get(K) by {get(K, D), D} and put(K, V) by {V, put(K, V, D)} I've omitted exception handling, but it's not actually all that different. Tedious rather than difficult. So there's an important sense in which it doesn't spoil the functional purity of the language. (I got this idea from a Xerox blue-and-white CS technical report giving a functional semantics for Euclid.) There are bad things that can happen in imperative languages that still can't happen in Erlang. One big bad thing *can* happen. If you call a function you haven't read, you do not know what it is going to do to the process dictionary. It could change the value associated with any key; it could delete any key; it could add any key->value mapping. Even specifications don't help, because although there have been several type systems that include effects, the type system Erlang now uses is not one of them. From desired.mta@REDACTED Thu Oct 11 08:45:48 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Thu, 11 Oct 2012 08:45:48 +0200 Subject: [erlang-questions] [PATCH] Fix R15B02 cross-compilation for TileraMDE-3.0.1.125620 In-Reply-To: <50758F96.5090200@erlang.org> References: <20121010140818.GA17965@precise> <50758F96.5090200@erlang.org> Message-ID: <20121011064144.GA3140@precise> On Wed, Oct 10, 2012 at 05:09:10PM +0200, Bj?rn-Egil Dahlberg wrote: > On 2012-10-10 16:08, Motiejus Jaks(tys wrote: > >Applying the patch below let me cross-compile and run R15B02 on Tilera64 > >using TileraMDE-3.0.1.125620. > > > >In order to prepare a clean patch, I have to know what do the options > >-OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off do on the machines that > >support it? Also, what is the meaning of -MALLOC_USE_HASH(1)? > > I am quoting from Tilera here: Is documentation open? Would it be possible to get a link/reference? Maybe there is any documentation on this feature in "compatibility notes" or "migration guide" of some kind? I could ask Tilera for this. > "'malloc_use_hash' only affects malloc (specifically, it affects the > flags it passes to 'mmap' when it allocates more memory from Linux). > You can also explicitly malloc memory with various caching attributes > using custom mspaces. > > The 'hash_default' argument in the .hvconfig file affects 'mmap' calls > that do not specify homing information. Because malloc*does* specify > homing information, at least when you are pthreaded, 'hash_default' is > ignored and malloc_use_hash is required to get hash-for-home." > > > This was something included in TileraMDE 2.0 beta (perhaps removed > since then). > I found it in TileraMDE-2.1.2/tile64/tile/usr/include/malloc.h though. All right. So it was there, and was removed. I think it would make sense to conditionalize it during configuration phase. But how to test for its existence/absence? > I guess it would be safe to remove "-OPT:Olimit=0 > -WOPT:lpre=off:spre=off:epre=off ". > It is an optimization for tile-cc when compiling on target machine. > An earlier comment in Makefile.in seems to have been removed, it said: > > +ifneq ($(filter tile-%,$(TARGET)),) > +# Some tile-cc optimizations take pathologically long for beam_emu.c, > +# so disable them. > +$(OBJDIR)/beam_emu.o: beam/beam_emu.c > + $(CC) $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) \ > + -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off \ > + $(INCLUDES) -c $< -o $@ > +endif Can you verify that it compiles in "reasonable time" on your system with these flags turned off? > Patch was supplied by Tilera Corp. That is fantastic. Motiejus From watson.timothy@REDACTED Thu Oct 11 10:45:11 2012 From: watson.timothy@REDACTED (Tim Watson) Date: Thu, 11 Oct 2012 09:45:11 +0100 Subject: [erlang-questions] CT take a long time to warm up In-Reply-To: References: <51E69ECE-7A83-4A91-9F2B-DB6EC47668F3@gmail.com> <20120926163945.GA8916@precise> Message-ID: <47C3E91A-34B8-4A3A-AE8C-B38C9195DE96@gmail.com> Whoa, this is fascinating Roberto. Did you try running common test from the shell by hand and see if it still happens, i.e., by running something like this: ct:run_test([{logdir, LogDir}, {label, Label}, {auto_compile, true}, {suite, Suites}]). Does the same thing happen? On 11 Oct 2012, at 04:38, Roberto Ostinelli wrote: > Ok, > > I finally managed to track this down but I'm totally unsure why this happens. Motiejus, you were right. > > When I run REBAR, the process automountd tries to connect on port 111 (sunrpc) on my local NAT. When it times out, the CT tests are actually run. If I simply block this process with a firewall, tests start immediately. > > I'm at disposal if anyone wants to dig further (tuncer?) > > r. > > On Wed, Sep 26, 2012 at 1:38 PM, Tuncer Ayaz wrote: > On Wed, Sep 26, 2012 at 9:44 PM, Motiejus Jak?tys wrote: > > On Wed, Sep 26, 2012 at 8:40 PM, Roberto Ostinelli wrote: > > > > How do you run it? Consistent 20 seconds sounds like a DNS > > > > timeout issue (stab in the dark). > > > > > > > > > rebar ct_suites=mymodule > > > > Hi, > > adding the list. > > > > Please do > > > > $TRACE_F rebar ct_suites=mymodule > > Make that > > $TRACE_F rebar suites=mymodule > > and also consider running rebar with fprof: > > $TRACE_F rebar -p suites=mymodule > or > $TRACE_F rebar --profile suites=mymodule > > , but make sure you update rebar first to avoid a faulty > call to fprof:apply/2. > > If it's caused by name lookup, you can try a custom ERL_INETRC file: > http://www.erlang.org/doc/apps/erts/inet_cfg.html > > > and replace $TRACE_F with Max OS C equivalent of "strace -f". > > Then see what it does just before the 20 second pause. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP using GPGMail URL: From attila.r.nohl@REDACTED Thu Oct 11 11:20:52 2012 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Thu, 11 Oct 2012 11:20:52 +0200 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <5075FCF2.6060502@earthlink.net> References: <5075FCF2.6060502@earthlink.net> Message-ID: 2012/10/11 Charles Hixson : > I'm choosing a language to implement a ... well, neural network is wrong, > and so is cellular automaton, but it gives the idea. Anyway, I'm going to > need, in each cell, a few stateful items, e.g. activation level. > > When I look at what Erlang can do, I see that the Process Dictionary looks > as if it would serve my needs, but then I am immediately warned not to use > it, that it will cause bugs. These stateful terms will not be exported from > the cell within which they are resident. Is this still likely to cause > problems? Is there some better approach to maintaining state? (I can't > just generate a new process, because other cells will need to know how to > access this one, or to test that it has been rolled out.) I don't really understand why you can't generate a new process for each cell - just send a message to the neighbouring cells that there's a new cell. I think each cell needs to know its neighbours anyway. From egil@REDACTED Thu Oct 11 12:10:39 2012 From: egil@REDACTED (=?windows-1252?Q?Bj=F6rn-Egil_Dahlberg?=) Date: Thu, 11 Oct 2012 12:10:39 +0200 Subject: [erlang-questions] [PATCH] Fix R15B02 cross-compilation for TileraMDE-3.0.1.125620 In-Reply-To: <20121011064144.GA3140@precise> References: <20121010140818.GA17965@precise> <50758F96.5090200@erlang.org> <20121011064144.GA3140@precise> Message-ID: <50769B1F.5090305@erlang.org> On 2012-10-11 08:45, Motiejus Jak?tys wrote: > On Wed, Oct 10, 2012 at 05:09:10PM +0200, Bj?rn-Egil Dahlberg wrote: >> On 2012-10-10 16:08, Motiejus Jaks(tys wrote: >>> Applying the patch below let me cross-compile and run R15B02 on Tilera64 >>> using TileraMDE-3.0.1.125620. >>> >>> In order to prepare a clean patch, I have to know what do the options >>> -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off do on the machines that >>> support it? Also, what is the meaning of -MALLOC_USE_HASH(1)? >> I am quoting from Tilera here: > Is documentation open? Would it be possible to get a link/reference? > Maybe there is any documentation on this feature in "compatibility > notes" or "migration guide" of some kind? I could ask Tilera for this. I'm sorry. It's not possible to make that specific information public. I was quoting from a private mail discussion I had with people at Tilera in 2008. >> "'malloc_use_hash' only affects malloc (specifically, it affects the >> flags it passes to 'mmap' when it allocates more memory from Linux). >> You can also explicitly malloc memory with various caching attributes >> using custom mspaces. >> >> The 'hash_default' argument in the .hvconfig file affects 'mmap' calls >> that do not specify homing information. Because malloc*does* specify >> homing information, at least when you are pthreaded, 'hash_default' is >> ignored and malloc_use_hash is required to get hash-for-home." >> >> >> This was something included in TileraMDE 2.0 beta (perhaps removed >> since then). >> I found it in TileraMDE-2.1.2/tile64/tile/usr/include/malloc.h though. > All right. So it was there, and was removed. > > I think it would make sense to conditionalize it during configuration > phase. But how to test for its existence/absence? How about doing this instead? diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index 37dfcb1..176a5af 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -408,8 +408,10 @@ void sys_tty_reset(int exit_code) #ifdef __tile__ /* Direct malloc to spread memory around the caches of multiple tiles. */ #include +#if defined(MALLOC_USE_HASH) MALLOC_USE_HASH(1); #endif +#endif #ifdef USE_THREADS >> I guess it would be safe to remove "-OPT:Olimit=0 >> -WOPT:lpre=off:spre=off:epre=off ". >> It is an optimization for tile-cc when compiling on target machine. >> An earlier comment in Makefile.in seems to have been removed, it said: >> >> +ifneq ($(filter tile-%,$(TARGET)),) >> +# Some tile-cc optimizations take pathologically long for beam_emu.c, >> +# so disable them. >> +$(OBJDIR)/beam_emu.o: beam/beam_emu.c >> + $(CC) $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) \ >> + -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off \ >> + $(INCLUDES) -c $< -o $@ >> +endif > Can you verify that it compiles in "reasonable time" on your system with > these flags turned off? Actually, no. =) This was a problem Tilera had when they compiled in their environment, i.e. uploading all code to the card and compiling it there (With their specific cards). Here at OTP we cross-compiled everything and did not see this problem. Hopefully this was a problem that has been remedied since then though. Also, tag - you?re it! It would be greatly appreciated if you could summarize problems and solution you have found for building and using Erlang on the Tilera card or if you find something we can change to make anyones experience just a bit easier. I don't know exactly where we should put this documentation but my suggestion would either be in the documentation for cross compiling or as a specific page on the GitHub Wiki. If the information arrives, we will surely find a place to put it. =) // Bj?rn-Egil From dch@REDACTED Thu Oct 11 13:41:48 2012 From: dch@REDACTED (Dave Cottlehuber) Date: Thu, 11 Oct 2012 13:41:48 +0200 Subject: [erlang-questions] CT take a long time to warm up In-Reply-To: <47C3E91A-34B8-4A3A-AE8C-B38C9195DE96@gmail.com> References: <51E69ECE-7A83-4A91-9F2B-DB6EC47668F3@gmail.com> <20120926163945.GA8916@precise> <47C3E91A-34B8-4A3A-AE8C-B38C9195DE96@gmail.com> Message-ID: On 11 October 2012 10:45, Tim Watson wrote: > Whoa, this is fascinating Roberto. Did you try running common test from the shell by hand and see if it still happens, i.e., by running something like this: > > ct:run_test([{logdir, LogDir}, > {label, Label}, > {auto_compile, true}, > {suite, Suites}]). > > Does the same thing happen? > > On 11 Oct 2012, at 04:38, Roberto Ostinelli wrote: > >> Ok, >> >> I finally managed to track this down but I'm totally unsure why this happens. Motiejus, you were right. >> >> When I run REBAR, the process automountd tries to connect on port 111 (sunrpc) on my local NAT. When it times out, the CT tests are actually run. If I simply block this process with a firewall, tests start immediately. >> >> I'm at disposal if anyone wants to dig further (tuncer?) >> >> r. Is anything in PATH or ERL_LIBS & similar environment vars referring to a non-existent mountpoint? A+ Dave From sergey_zhemzhitsky@REDACTED Thu Oct 11 14:21:12 2012 From: sergey_zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Thu, 11 Oct 2012 12:21:12 +0000 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. In-Reply-To: <5075C8A1.9020706@erlang.org> References: <5075C8A1.9020706@erlang.org> Message-ID: <06139A918ACCA041BF46A0F36940C7FA4FE9135A@exch-mbx2.msk.trd.ru> Hello Rickard, Thanks a lot for clarification. I'm doing erl_drv_thread_join() only once and only for threads created by erl_drv_thread_create(). The error only happens from time to time in the following scenario: 1. Erlang process, I send messages from the ddriver to, dies for some reason 2. Driver callback process_exit fires. 3. From the driver thread I'm trying to send the final message to the dead erlang process with driver_send_term 4. As the erlang process is dead, driver_send_term fails 5. As driver_send_term fails I'm doing driver_failure_atom to kill linked process. Here two things happen: 6.1 (*stop)(ErlDrvData drv_data) executes as linked process has been killed by driver_failure_atom 6.2 Thread, started in the driver, exists When (*stop)(ErlDrvData drv_data) executes erl_drv_thread_join is called for all the threads started in the driver. So something wrong happens that leads to tid reuse. Best Regards, Sergey -----Original Message----- From: Rickard Green [mailto:rickard@REDACTED] Sent: Wednesday, October 10, 2012 11:13 PM To: Zhemzhitsky Sergey Cc: Erlang Questions Subject: Re: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. >>From time to time erl_drv_thread_join returns error EDEADLK=35, i.e. the current thread (scheduler thread) tries to join itself. > > According to the documentation ?A Thread identifier may be reused very quickly after a thread has terminated. Therefore, if a thread corresponding to one of the involved thread identifiers has terminated since the thread identifier was saved, the result of erl_drv_equal_tids() might not give the expected result.? > > I suppose that thread terminates earlier then erl_drv_thread_join call happens, so ErlDrvTid is already reused. > This reuse will not cause problems for erl_drv_thread_join() as long as it is used correctly (a tid wont be reused until after the thread has been joined). erl_drv_thread_join() will also refuse to join threads not created by erl_drv_thread_create(), and would in case the scheduler thread tried to join itself fail with EINVAL. > So the question is how to use erl_drv_thread_join properly and how to guarantee that the saved ErlDrvTid value points to the same data that was returned from erl_drv_thread_create? > It is important that the thread is joined once and *only* once. Are you sure that you don't do two calls to erl_drv_thread_join() for the same thread? Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. _______________________________________________________ CONFIDENTIALITY NOTICE: This email and any files attached to it may be conf idential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email. From rickard@REDACTED Thu Oct 11 14:35:23 2012 From: rickard@REDACTED (Rickard Green) Date: Thu, 11 Oct 2012 14:35:23 +0200 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. In-Reply-To: <06139A918ACCA041BF46A0F36940C7FA4FE9135A@exch-mbx2.msk.trd.ru> References: <5075C8A1.9020706@erlang.org> <06139A918ACCA041BF46A0F36940C7FA4FE9135A@exch-mbx2.msk.trd.ru> Message-ID: <5076BD0B.4040907@erlang.org> The tid cannot be prematurely reused in this case. The EDEADLK is due to some other reason. Are you calling driver_failure_atom() from your thread? That would explain this issue. Regards, Rickard On 10/11/2012 02:21 PM, Zhemzhitsky Sergey wrote: > Hello Rickard, > > Thanks a lot for clarification. > > I'm doing erl_drv_thread_join() only once and only for threads created by erl_drv_thread_create(). > > The error only happens from time to time in the following scenario: > > 1. Erlang process, I send messages from the ddriver to, dies for some reason > 2. Driver callback process_exit fires. > 3. From the driver thread I'm trying to send the final message to the dead erlang process with driver_send_term > 4. As the erlang process is dead, driver_send_term fails > 5. As driver_send_term fails I'm doing driver_failure_atom to kill linked process. > > Here two things happen: > > 6.1 (*stop)(ErlDrvData drv_data) executes as linked process has been killed by driver_failure_atom > 6.2 Thread, started in the driver, exists > > When (*stop)(ErlDrvData drv_data) executes erl_drv_thread_join is called for all the threads started in the driver. > > So something wrong happens that leads to tid reuse. > > Best Regards, > Sergey > > > -----Original Message----- > From: Rickard Green [mailto:rickard@REDACTED] > Sent: Wednesday, October 10, 2012 11:13 PM > To: Zhemzhitsky Sergey > Cc: Erlang Questions > Subject: Re: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. > > >> >From time to time erl_drv_thread_join returns error EDEADLK=35, i.e. the current thread (scheduler thread) tries to join itself. >> >> According to the documentation ?A Thread identifier may be reused very quickly after a thread has terminated. Therefore, if a thread corresponding to one of the involved thread identifiers has terminated since the thread identifier was saved, the result of erl_drv_equal_tids() might not give the expected result.? >> >> I suppose that thread terminates earlier then erl_drv_thread_join call happens, so ErlDrvTid is already reused. >> > > This reuse will not cause problems for erl_drv_thread_join() as long as it is used correctly (a tid wont be reused until after the thread has been joined). erl_drv_thread_join() will also refuse to join threads not created by erl_drv_thread_create(), and would in case the scheduler thread tried to join itself fail with EINVAL. > >> So the question is how to use erl_drv_thread_join properly and how to guarantee that the saved ErlDrvTid value points to the same data that was returned from erl_drv_thread_create? >> > > It is important that the thread is joined once and *only* once. Are you sure that you don't do two calls to erl_drv_thread_join() for the same thread? > > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB. > > _______________________________________________________ > CONFIDENTIALITY NOTICE: This email and any files attached to it may be conf idential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email. > -- Rickard Green, Erlang/OTP, Ericsson AB. From sergey_zhemzhitsky@REDACTED Thu Oct 11 15:11:14 2012 From: sergey_zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Thu, 11 Oct 2012 13:11:14 +0000 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. In-Reply-To: <5076BD0B.4040907@erlang.org> References: <5075C8A1.9020706@erlang.org> <06139A918ACCA041BF46A0F36940C7FA4FE9135A@exch-mbx2.msk.trd.ru> <5076BD0B.4040907@erlang.org> Message-ID: <06139A918ACCA041BF46A0F36940C7FA4FE91411@exch-mbx2.msk.trd.ru> Hi Rickard, >> Are you calling driver_failure_atom() from your thread? That would explain this issue. Exactly. Could you explain a little bit more why it can cause this issue (I suppose it's due to it is not thread safe)? Best Regards, Sergey Zhemzhitsky Phone. +7 495 2580500 ext. 1246 -----Original Message----- From: Rickard Green [mailto:rickard@REDACTED] Sent: Thursday, October 11, 2012 4:35 PM To: Zhemzhitsky Sergey Cc: Erlang Questions Subject: Re: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. The tid cannot be prematurely reused in this case. The EDEADLK is due to some other reason. Are you calling driver_failure_atom() from your thread? That would explain this issue. Regards, Rickard On 10/11/2012 02:21 PM, Zhemzhitsky Sergey wrote: > Hello Rickard, > > Thanks a lot for clarification. > > I'm doing erl_drv_thread_join() only once and only for threads created by erl_drv_thread_create(). > > The error only happens from time to time in the following scenario: > > 1. Erlang process, I send messages from the ddriver to, dies for some > reason 2. Driver callback process_exit fires. > 3. From the driver thread I'm trying to send the final message to the > dead erlang process with driver_send_term 4. As the erlang process is > dead, driver_send_term fails 5. As driver_send_term fails I'm doing driver_failure_atom to kill linked process. > > Here two things happen: > > 6.1 (*stop)(ErlDrvData drv_data) executes as linked process has been > killed by driver_failure_atom > 6.2 Thread, started in the driver, exists > > When (*stop)(ErlDrvData drv_data) executes erl_drv_thread_join is called for all the threads started in the driver. > > So something wrong happens that leads to tid reuse. > > Best Regards, > Sergey > > > -----Original Message----- > From: Rickard Green [mailto:rickard@REDACTED] > Sent: Wednesday, October 10, 2012 11:13 PM > To: Zhemzhitsky Sergey > Cc: Erlang Questions > Subject: Re: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. > > >> >From time to time erl_drv_thread_join returns error EDEADLK=35, i.e. the current thread (scheduler thread) tries to join itself. >> >> According to the documentation ?A Thread identifier may be reused very quickly after a thread has terminated. Therefore, if a thread corresponding to one of the involved thread identifiers has terminated since the thread identifier was saved, the result of erl_drv_equal_tids() might not give the expected result.? >> >> I suppose that thread terminates earlier then erl_drv_thread_join call happens, so ErlDrvTid is already reused. >> > > This reuse will not cause problems for erl_drv_thread_join() as long as it is used correctly (a tid wont be reused until after the thread has been joined). erl_drv_thread_join() will also refuse to join threads not created by erl_drv_thread_create(), and would in case the scheduler thread tried to join itself fail with EINVAL. > >> So the question is how to use erl_drv_thread_join properly and how to guarantee that the saved ErlDrvTid value points to the same data that was returned from erl_drv_thread_create? >> > > It is important that the thread is joined once and *only* once. Are you sure that you don't do two calls to erl_drv_thread_join() for the same thread? > > Regards, > Rickard > -- > Rickard Green, Erlang/OTP, Ericsson AB. > > _______________________________________________________ > CONFIDENTIALITY NOTICE: This email and any files attached to it may be conf idential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email. > -- Rickard Green, Erlang/OTP, Ericsson AB. From enewhuis@REDACTED Thu Oct 11 15:14:18 2012 From: enewhuis@REDACTED (Eric Newhuis) Date: Thu, 11 Oct 2012 08:14:18 -0500 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> Message-ID: <76668491-F22B-4CBE-A41B-351439E24C2A@gmail.com> Heheh when may we expect the Emacs refactoring macro for this? LOL On Oct 10, 2012, at 11:31 PM, Richard O'Keefe wrote: > You can think of the process dictionary this way: > > If there is a function clause > > f(X, Y, Z) -> A = g(X, Y), B = h(Z, A), q(A, B) > > replace it by > > f(X, Y, Z, D0) -> > {A, D1} = g(X, Y, D0), > {B, D2} = h(Z, A, D1), > q(A, B, D2) > > and replace > > get(K) > > by > > {get(K, D), D} > > and > put(K, V) > by > {V, put(K, V, D)} > > I've omitted exception handling, but it's not actually > all that different. Tedious rather than difficult. > > So there's an important sense in which it doesn't spoil the > functional purity of the language. (I got this idea from a > Xerox blue-and-white CS technical report giving a functional > semantics for Euclid.) There are bad things that can happen > in imperative languages that still can't happen in Erlang. > > One big bad thing *can* happen. If you call a function you > haven't read, you do not know what it is going to do to the > process dictionary. It could change the value associated > with any key; it could delete any key; it could add any > key->value mapping. Even specifications don't help, > because although there have been several type systems that > include effects, the type system Erlang now uses is not one > of them. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From be.dmitry@REDACTED Thu Oct 11 15:48:34 2012 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Thu, 11 Oct 2012 17:48:34 +0400 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> Message-ID: <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> Isn't this feature, I mean process dictionaries, much worse for functional nature of Erlang than parameterized modules? Why aren't there any words about removing pdics but there are about functional pmods? -- Dmitry Belyaev On 11.10.2012, at 8:31, Richard O'Keefe wrote: > You can think of the process dictionary this way: > > If there is a function clause > > f(X, Y, Z) -> A = g(X, Y), B = h(Z, A), q(A, B) > > replace it by > > f(X, Y, Z, D0) -> > {A, D1} = g(X, Y, D0), > {B, D2} = h(Z, A, D1), > q(A, B, D2) > > and replace > > get(K) > > by > > {get(K, D), D} > > and > put(K, V) > by > {V, put(K, V, D)} > > I've omitted exception handling, but it's not actually > all that different. Tedious rather than difficult. > > So there's an important sense in which it doesn't spoil the > functional purity of the language. (I got this idea from a > Xerox blue-and-white CS technical report giving a functional > semantics for Euclid.) There are bad things that can happen > in imperative languages that still can't happen in Erlang. > > One big bad thing *can* happen. If you call a function you > haven't read, you do not know what it is going to do to the > process dictionary. It could change the value associated > with any key; it could delete any key; it could add any > key->value mapping. Even specifications don't help, > because although there have been several type systems that > include effects, the type system Erlang now uses is not one > of them. > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rickard@REDACTED Thu Oct 11 16:13:08 2012 From: rickard@REDACTED (Rickard Green) Date: Thu, 11 Oct 2012 16:13:08 +0200 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. In-Reply-To: <06139A918ACCA041BF46A0F36940C7FA4FE91411@exch-mbx2.msk.trd.ru> References: <5075C8A1.9020706@erlang.org> <06139A918ACCA041BF46A0F36940C7FA4FE9135A@exch-mbx2.msk.trd.ru> <5076BD0B.4040907@erlang.org> <06139A918ACCA041BF46A0F36940C7FA4FE91411@exch-mbx2.msk.trd.ru> Message-ID: <5076D3F4.6030701@erlang.org> On 10/11/2012 03:11 PM, Zhemzhitsky Sergey wrote: > Hi Rickard, > >>> Are you calling driver_failure_atom() from your thread? That would explain this issue. > > Exactly. Could you explain a little bit more why it can cause this issue (I suppose it's due to it is not thread safe)? > Yes, it is because driver_failure_atom() is not thread safe. Apart from misc thread unsafe modifications of the port internal state, driver_failure_atom() can call the stop() callback which means that your thread will end up trying to join with itself. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. From bengt.kleberg@REDACTED Thu Oct 11 16:15:02 2012 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Thu, 11 Oct 2012 16:15:02 +0200 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> Message-ID: <1349964902.4727.29.camel@sekic1152.rnd.ki.sw.ericsson.se> Greetings, The difference is that the process dictionary is not an experimental feature. bengt On Thu, 2012-10-11 at 17:48 +0400, Dmitry Belyaev wrote: > Isn't this feature, I mean process dictionaries, much worse for functional nature of Erlang than parameterized modules? > > Why aren't there any words about removing pdics but there are about functional pmods? > From sergey_zhemzhitsky@REDACTED Thu Oct 11 16:17:25 2012 From: sergey_zhemzhitsky@REDACTED (Zhemzhitsky Sergey) Date: Thu, 11 Oct 2012 14:17:25 +0000 Subject: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. In-Reply-To: <5076D3F4.6030701@erlang.org> References: <5075C8A1.9020706@erlang.org> <06139A918ACCA041BF46A0F36940C7FA4FE9135A@exch-mbx2.msk.trd.ru> <5076BD0B.4040907@erlang.org> <06139A918ACCA041BF46A0F36940C7FA4FE91411@exch-mbx2.msk.trd.ru> <5076D3F4.6030701@erlang.org> Message-ID: <06139A918ACCA041BF46A0F36940C7FA4FE914D3@exch-mbx2.msk.trd.ru> Thanks a lot. Now it's clear enough. Best Regards, Sergey -----Original Message----- From: Rickard Green [mailto:rickard@REDACTED] Sent: Thursday, October 11, 2012 6:13 PM To: Zhemzhitsky Sergey Cc: Erlang Questions Subject: Re: [erlang-questions] Port Driver. erl_drv_thread_join and ErlDrvTid reuse. On 10/11/2012 03:11 PM, Zhemzhitsky Sergey wrote: > Hi Rickard, > >>> Are you calling driver_failure_atom() from your thread? That would explain this issue. > > Exactly. Could you explain a little bit more why it can cause this issue (I suppose it's due to it is not thread safe)? > Yes, it is because driver_failure_atom() is not thread safe. Apart from misc thread unsafe modifications of the port internal state, driver_failure_atom() can call the stop() callback which means that your thread will end up trying to join with itself. Regards, Rickard -- Rickard Green, Erlang/OTP, Ericsson AB. _______________________________________________________ CONFIDENTIALITY NOTICE: This email and any files attached to it may be conf idential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email. From rustompmody@REDACTED Thu Oct 11 16:17:56 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Thu, 11 Oct 2012 19:47:56 +0530 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On Thu, Oct 11, 2012 at 12:00 AM, Joe Armstrong wrote: > On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody > wrote: > > On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang > > wrote: > >> > >> On Wednesday, October 10, Rustom Mody wrote: > > > > > >> > >> Another thing: you write "under severe time-crunch", but that could > >> mean "we need to ship a product in a very short time" or it could mean > >> "the data has to be processed in a very short time". > >> > >> Matt > > > > > > Well mainly the latter > > But the former is also true. (When is it ever not :-)) > > My concern is that there is some desire to try out erlang which may be > > fine... However I am wondering whether the difference between concurrency > > and parallelism has not been appreciated. > > Very true - I have given several lectures on this ... many people use the > words concurrent and parallel as synonyms - but there is a real difference. > > In a nutshell - parallel has to do with real hardware than can do > several things "at the same time" > whereas concurrency is a software structuring abstraction which > pretends that different > parts of the software can run at the same time (if suitable hardware > were available) > > Parallel has to do with hardware - on a single cpu you can write your > programs > using pmap (instead of map) and parBegin and parEnd and it won't make > any difference > since the machine is still only doing one instruction at a time. > > It's only because machine are fast that we think they do several > things in parallel - > in the single CPU + time sharing case if we slowed the clock down we would > see > that no parallelism is involved - the CPU does one thing a time - even > when executing > concurrent programs :-) > > There are only a few real sources of true parallelism - multi-cores, > what happens in the pipeline, and how data is fetched/stored > > Multicores do give you true parallelism, but you must be careful to > about the granularity > of computation - ie it should not be more work to shift the work to a > remote CPU than do it > locally. Pipeline messing is old for compiler writers. Data fetching > and storing can be done in > parallel, this needs a bit of thought. > > Parallel programs are inherently non portable - in the sense that they > specifically depend > upon details of the hardware. > > Concurrent programs are program written using a concurrency > abstraction. In Erlang > processes are the unti of concurrency - processes are pretty coarse-grain > things > ie they take a wee bit of effort to get going, so they actually map > well onto multi-cores, > even if you can detect parallelism in your program (easy in a pure FP, > just evaluate > all the arguments to a function in parallel) - it's pretty difficult > (ie an open research problem) > to map this onto the available hardware. > > Even if you know how long computations take, and you know what > resources you have > to solve the computations, then mapping the computations onto the > resources involves > solving the knapsack problem which is NP hard. > > It you write program in Erlang you at least have a head start over > sequential code > since the programmer has at least said (using processes) which > activities should be run in parallel. > Optimally scheduling these is NP hard - but non-optimal first on > demand, best effort scheduling > works pretty well for non-pathological code. > > My 10 c. > > /Joe Thanks Joe for a detailed answer. I have one more question that sits somewhere between parallelism and concurrency. In the 'normal' world (ie C++, Java, Python) there is a general recommendation that converting a threaded solution to an event-driven solution usually speeds up the program and removes hard-to-find bugs. And at the heart of an event-driven system is usually an FSM. Now I am particularly interested in the combined (regular-exp+FSM) approach that ragel provides: http://www.complang.org/ragel/ In particular, here is the use of ragel to write an http server: http://www.zedshaw.com/essays/ragel_state_charts.html Now since ragel is more or less language agnostic -- it has backends for generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make sense? Rusi -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Thu Oct 11 16:44:05 2012 From: ulf@REDACTED (Ulf Wiger) Date: Thu, 11 Oct 2012 16:44:05 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: <59F982A0-C8C2-471D-8C5E-AF17E68769C2@feuerlabs.com> On 11 Oct 2012, at 16:17, Rustom Mody wrote: > I have one more question that sits somewhere between parallelism and concurrency. > In the 'normal' world (ie C++, Java, Python) there is a general recommendation that converting a threaded solution to an event-driven solution usually speeds up the program and removes hard-to-find bugs. > And at the heart of an event-driven system is usually an FSM. > Now I am particularly interested in the combined (regular-exp+FSM) approach that ragel provides:http://www.complang.org/ragel/ > > In particular, here is the use of ragel to write an http server: http://www.zedshaw.com/essays/ragel_state_charts.html > > Now since ragel is more or less language agnostic -- it has backends for generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make sense? As far as I can tell, Ragel is mainly for parser FSMs. As such, it seems great, but I doubt whether it would add value to Erlang as a way to specify concurrent (message-passing) state machines. The thing it lacks that Erlang has, is selective message reception. This allows Erlang to ignore a message in its inbox, and keep waiting for the desired message. In coordination problems, this is a great way to avoid complexity explosion. I gave a talk about that a while ago: http://www.infoq.com/presentations/Death-by-Accidental-Complexity Basically, you _could_ generate Erlang FSM code from Ragel code, I'm sure, but I think this would mainly be useful for porting existing Ragel models over to Erlang. Ultimately, I think it would still be better to do the 'porting' by hand, to give you a chance to think through the problem in a more Erlang-ish style. The FSMs that are easy to write in Ragel, also ought to be easy to write in Erlang. If I missed some clever device in Ragel that handles this, please correct me. BR, Ulf W Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Thu Oct 11 17:48:07 2012 From: charleshixsn@REDACTED (Charles Hixson) Date: Thu, 11 Oct 2012 08:48:07 -0700 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: References: <5075FCF2.6060502@earthlink.net> Message-ID: <5076EA37.2030903@earthlink.net> On 10/11/2012 02:20 AM, Attila Rajmund Nohl wrote: > 2012/10/11 Charles Hixson: >> I'm choosing a language to implement a ... well, neural network is wrong, >> and so is cellular automaton, but it gives the idea. Anyway, I'm going to >> need, in each cell, a few stateful items, e.g. activation level. >> >> When I look at what Erlang can do, I see that the Process Dictionary looks >> as if it would serve my needs, but then I am immediately warned not to use >> it, that it will cause bugs. These stateful terms will not be exported from >> the cell within which they are resident. Is this still likely to cause >> problems? Is there some better approach to maintaining state? (I can't >> just generate a new process, because other cells will need to know how to >> access this one, or to test that it has been rolled out.) > I don't really understand why you can't generate a new process for > each cell - just send a message to the neighbouring cells that there's > a new cell. I think each cell needs to know its neighbours anyway. > Because lots of other processes would have links to the cell that were the process id. (There isn't really any other way to link to an active process.) It doesn't know who has these links. "Neighbors" isn't the right way to think about it, as they aren't neighbors in any meaningful sense of the term. (The links are one way.) Perhaps I should have called it a weighted directed graph, but that isn't quite the right model either. But with that analogy the weights need to be adjustable. Or I could have called it a neural net, but that also isn't quite the right model, at least as I understand it. Each cell needs to know the cells that it links to, and the cell that most recently linked to it. And it's activation level. And the weight of each link to a "following" cell. The activation levels and weights need to be able to change, but should not be visible outside the cell (though the cell should be able to receive messages that causes it to change these values). It's also possible that eventually I might need some "regional values", that would do things like adjust the sensitivity of all cells in a region to being activated, but I can see a way to do that with message passing (though it does add significantly to the overhead). OTOH, maybe I'll never need these "emotional variations". (So far I don't have a well-defined idea of what "region" would mean.) -- Charles Hixson From max.lapshin@REDACTED Thu Oct 11 18:01:55 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 11 Oct 2012 20:01:55 +0400 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> Message-ID: On Thu, Oct 11, 2012 at 5:48 PM, Dmitry Belyaev wrote: > Isn't this feature, I mean process dictionaries, much worse for functional nature of Erlang than parameterized modules? > > Why aren't there any words about removing pdics but there are about functional pmods? > Damn! Why do you want to remove any feature, you are scared with? Are you using some library that is using pdict and it spoils your life? I think no. Process dictionary is very, very convenient tool. It is the only way to get information from process without sending message to it. It is the only way to get information about process, whose message queue len is more than several thousands of messages and you ask, when OTP will remove it? From raould@REDACTED Thu Oct 11 18:38:19 2012 From: raould@REDACTED (Raoul Duke) Date: Thu, 11 Oct 2012 09:38:19 -0700 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On Thu, Oct 11, 2012 at 7:17 AM, Rustom Mody wrote: > In the 'normal' world (ie C++, Java, Python) there is a general > recommendation that converting a threaded solution to an event-driven > solution usually speeds up the program and removes hard-to-find bugs. please note that you are talking about problems that exist in those paradigms. there are other paradigms that do not have the exact same problems (and of course then have their own problems :-). i suspect The Erlang Way is different enough that it is probably a mistake to just take something for the usual crowd of programming language paradigms, and dump it onto erlang. sincerely. From mjtruog@REDACTED Thu Oct 11 18:58:35 2012 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 11 Oct 2012 09:58:35 -0700 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <5076EA37.2030903@earthlink.net> References: <5075FCF2.6060502@earthlink.net> <5076EA37.2030903@earthlink.net> Message-ID: <5076FABB.7060605@gmail.com> On 10/11/2012 08:48 AM, Charles Hixson wrote: > On 10/11/2012 02:20 AM, Attila Rajmund Nohl wrote: >> 2012/10/11 Charles Hixson: >>> I'm choosing a language to implement a ... well, neural network is wrong, >>> and so is cellular automaton, but it gives the idea. Anyway, I'm going to >>> need, in each cell, a few stateful items, e.g. activation level. >>> >>> When I look at what Erlang can do, I see that the Process Dictionary looks >>> as if it would serve my needs, but then I am immediately warned not to use >>> it, that it will cause bugs. These stateful terms will not be exported from >>> the cell within which they are resident. Is this still likely to cause >>> problems? Is there some better approach to maintaining state? (I can't >>> just generate a new process, because other cells will need to know how to >>> access this one, or to test that it has been rolled out.) >> I don't really understand why you can't generate a new process for >> each cell - just send a message to the neighbouring cells that there's >> a new cell. I think each cell needs to know its neighbours anyway. >> > Because lots of other processes would have links to the cell that were the process id. (There isn't really any other way to link to an active process.) It doesn't know who has these links. "Neighbors" isn't the right way to think about it, as they aren't neighbors in any meaningful sense of the term. (The links are one way.) > > Perhaps I should have called it a weighted directed graph, but that isn't quite the right model either. But with that analogy the weights need to be adjustable. Or I could have called it a neural net, but that also isn't quite the right model, at least as I understand it. > > Each cell needs to know the cells that it links to, and the cell that most recently linked to it. And it's activation level. And the weight of each link to a "following" cell. The activation levels and weights need to be able to change, but should not be visible outside the cell (though the cell should be able to receive messages that causes it to change these values). > > It's also possible that eventually I might need some "regional values", that would do things like adjust the sensitivity of all cells in a region to being activated, but I can see a way to do that with message passing (though it does add significantly to the overhead). OTOH, maybe I'll never need these "emotional variations". (So far I don't have a well-defined idea of what "region" would mean.) > You mentioned the variable number suffix as an emacs problem, but there is a parse transform that can help hide the variable number suffixes for you, here: https://github.com/spawngrid/seqbind . However, it is probably best to avoid parse transforms while learning Erlang, since the source code may not match examples. The weighted directed graph or neural net type of application is probably best done with an Erlang process per cell with messages inbetween. That approach matches an actor model, and Erlang. A tutorial that should be helpful is here: http://www.trapexit.org/Erlang_and_Neural_Networks From desired.mta@REDACTED Thu Oct 11 19:15:23 2012 From: desired.mta@REDACTED (Motiejus =?utf-8?Q?Jak=C5=A1tys?=) Date: Thu, 11 Oct 2012 19:15:23 +0200 Subject: [erlang-questions] [PATCH] Fix R15B02 cross-compilation for TileraMDE-3.0.1.125620 In-Reply-To: <50769B1F.5090305@erlang.org> References: <20121010140818.GA17965@precise> <50758F96.5090200@erlang.org> <20121011064144.GA3140@precise> <50769B1F.5090305@erlang.org> Message-ID: <20121011171523.GA3282@precise> On Thu, Oct 11, 2012 at 12:10:39PM +0200, Bj?rn-Egil Dahlberg wrote: > On 2012-10-11 08:45, Motiejus Jak?tys wrote: > >On Wed, Oct 10, 2012 at 05:09:10PM +0200, Bj?rn-Egil Dahlberg wrote: > >>"'malloc_use_hash' only affects malloc (specifically, it affects the > >>flags it passes to 'mmap' when it allocates more memory from Linux). > >>You can also explicitly malloc memory with various caching attributes > >>using custom mspaces. > >> > >I think it would make sense to conditionalize it during configuration > >phase. But how to test for its existence/absence? > How about doing this instead? > > diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c > index 37dfcb1..176a5af 100644 > --- a/erts/emulator/sys/unix/sys.c > +++ b/erts/emulator/sys/unix/sys.c > @@ -408,8 +408,10 @@ void sys_tty_reset(int exit_code) > #ifdef __tile__ > /* Direct malloc to spread memory around the caches of multiple tiles. */ > #include > +#if defined(MALLOC_USE_HASH) > MALLOC_USE_HASH(1); > #endif > +#endif > #ifdef USE_THREADS Oh, it was a macro. Of course. :-) > >>I guess it would be safe to remove "-OPT:Olimit=0 > >>-WOPT:lpre=off:spre=off:epre=off ". > >>It is an optimization for tile-cc when compiling on target machine. > >>An earlier comment in Makefile.in seems to have been removed, it said: > >> > >>+ifneq ($(filter tile-%,$(TARGET)),) > >>+# Some tile-cc optimizations take pathologically long for beam_emu.c, > >>+# so disable them. > >>+$(OBJDIR)/beam_emu.o: beam/beam_emu.c > >>+ $(CC) $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) \ > >>+ -OPT:Olimit=0 -WOPT:lpre=off:spre=off:epre=off \ > >>+ $(INCLUDES) -c $< -o $@ > >>+endif > >Can you verify that it compiles in "reasonable time" on your system with > >these flags turned off? > Also, tag - you?re it! Did not understand the above sentence. > It would be greatly appreciated if you could summarize problems and > solution you have found for building and using Erlang on the Tilera > card or if you find something we can change to make anyones > experience just a bit easier. I don't know exactly where we should > put this documentation but my suggestion would either be in the > documentation for cross compiling or as a specific page on the > GitHub Wiki. If the information arrives, we will surely find a place > to put it. =) When one is doing ./otp_build autoconf[*] ./otp_build configure --xcomp-conf=FILE make Then everything in documentation makes sense and is really clear. However, if one wants to cross-compile using ./configure[?], then you have to know the cpu-vendor-os triplet. I got it while invoking GCC on the target machine: $ tile-monitor --resume --pci --resume --root -- gcc -v Using built-in specs. Target: tile-unknown-linux-gnu ... Is there a better way? Anyway, this or the better way, I think, could be documented. If people run into compilation problems, usually it is not Erlang specific. It's a missing macro, invalid compiler flag, missing function during linking.. Standard stuff which is common to any C project. I am afraid it does not make sense to add help to the Erlang building documentation. Like "try removing stuff that is complaining and see what happens" (the way I did) is too easy to be misinterpreted, and whilst if you provide more aid you just repeat general C knowledge. Sadly, specific questions are asked, without a reply[1]. Anyway, regarding code. Should I "prepare" a new patch with an updated commit message, or what you have is good enough to be merged? Regards, Motiejus [*]: I am building from git source. [?]: I used ./otp_build only, and this is speculation. Not sure if would work. [1]: http://erlang.org/pipermail/erlang-questions/2011-October/061727.html From erlang@REDACTED Thu Oct 11 19:42:23 2012 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 11 Oct 2012 19:42:23 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On Thu, Oct 11, 2012 at 4:17 PM, Rustom Mody wrote: > > > On Thu, Oct 11, 2012 at 12:00 AM, Joe Armstrong wrote: >> >> On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody >> wrote: >> > On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang >> > wrote: >> >> >> >> On Wednesday, October 10, Rustom Mody wrote: >> > >> > >> >> >> >> Another thing: you write "under severe time-crunch", but that could >> >> mean "we need to ship a product in a very short time" or it could mean >> >> "the data has to be processed in a very short time". >> >> >> >> Matt >> > >> > >> > Well mainly the latter >> > But the former is also true. (When is it ever not :-)) >> > My concern is that there is some desire to try out erlang which may be >> > fine... However I am wondering whether the difference between >> > concurrency >> > and parallelism has not been appreciated. >> >> Very true - I have given several lectures on this ... many people use the >> words concurrent and parallel as synonyms - but there is a real >> difference. >> >> In a nutshell - parallel has to do with real hardware than can do >> several things "at the same time" >> whereas concurrency is a software structuring abstraction which >> pretends that different >> parts of the software can run at the same time (if suitable hardware >> were available) >> >> Parallel has to do with hardware - on a single cpu you can write your >> programs >> using pmap (instead of map) and parBegin and parEnd and it won't make >> any difference >> since the machine is still only doing one instruction at a time. >> >> It's only because machine are fast that we think they do several >> things in parallel - >> in the single CPU + time sharing case if we slowed the clock down we would >> see >> that no parallelism is involved - the CPU does one thing a time - even >> when executing >> concurrent programs :-) >> >> There are only a few real sources of true parallelism - multi-cores, >> what happens in the pipeline, and how data is fetched/stored >> >> Multicores do give you true parallelism, but you must be careful to >> about the granularity >> of computation - ie it should not be more work to shift the work to a >> remote CPU than do it >> locally. Pipeline messing is old for compiler writers. Data fetching >> and storing can be done in >> parallel, this needs a bit of thought. >> >> Parallel programs are inherently non portable - in the sense that they >> specifically depend >> upon details of the hardware. >> >> Concurrent programs are program written using a concurrency >> abstraction. In Erlang >> processes are the unti of concurrency - processes are pretty coarse-grain >> things >> ie they take a wee bit of effort to get going, so they actually map >> well onto multi-cores, >> even if you can detect parallelism in your program (easy in a pure FP, >> just evaluate >> all the arguments to a function in parallel) - it's pretty difficult >> (ie an open research problem) >> to map this onto the available hardware. >> >> Even if you know how long computations take, and you know what >> resources you have >> to solve the computations, then mapping the computations onto the >> resources involves >> solving the knapsack problem which is NP hard. >> >> It you write program in Erlang you at least have a head start over >> sequential code >> since the programmer has at least said (using processes) which >> activities should be run in parallel. >> Optimally scheduling these is NP hard - but non-optimal first on >> demand, best effort scheduling >> works pretty well for non-pathological code. >> >> My 10 c. >> >> /Joe > > > Thanks Joe for a detailed answer. > > I have one more question that sits somewhere between parallelism and > concurrency. > In the 'normal' world (ie C++, Java, Python) there is a general > recommendation that converting a threaded solution to an event-driven > solution usually speeds up the program and removes hard-to-find bugs. > And at the heart of an event-driven system is usually an FSM. > Now I am particularly interested in the combined (regular-exp+FSM) approach > that ragel provides: http://www.complang.org/ragel/ > > In particular, here is the use of ragel to write an http server: > http://www.zedshaw.com/essays/ragel_state_charts.html > > Now since ragel is more or less language agnostic -- it has backends for > generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make > sense? I have no idea - you haven't told me what problem you want to solve - you've told me the technique you wish to use to solve the problem. This is a bit "back to front" - I'd like to *start* with the the problem then figure out how to solve it - not *start* with the idea that you want FSMs and events and so on and not tell me what the problem is. I can't possibly recommend any technology without knowing what the problem is. Regarding FSM in erlang - these are easy: state1() -> receive event1 -> state2(); event2 -> state3() end. state2() -> receive event3 -> state4() .. end. whatever - not really worth getting grey hair over Event driven programming is with Erlang spectacles on - not a wonderful idea. Events are a poor substitue for the lack of concurrency and non-blocking reads. If I had a language with a) only one thread of execution and b) blocking I/O then you know what I'd do - I'd invent event based programming. Let me explain what I think happened ... Once upon a time there was a sequential processor, that could run only one sequential program. The program did the following a: start -> read -> compute -> write -> compute -> read -> compute ... -> stop ie it interleaved computation with reads and writes. If the reads blocked - who cared - it's only doing one thing - a consequence of this architecture was "the program should not crash" - since there is only one thread if it crashes you have a BIG problem - so we invent "defensive programming" Now we want to do TWO or more things at the same time: a: start -> read -> compute -> read -> compute -> read -> compute ... -> stop b: start -> read -> read -> write -> compute -> read ... Now - horrors - read stops the system - it blocks, and writes are problematic we two things might write the same object at the same time. Enter - the big global write lock and event based programming We transform a: into a: start -> read and on_read_complete(a1) a1: compute -> read and on_read_complete(a2) a2: compute -> read and on_read_complete(a3) etc. and the same for b - the result is the software is a big mess look at Javascript $('#result').load('ajax/test.html', function() { alert('Load was performed.'); }); This is typical jquery javascript code load("blaa", fun) means read fun and when the read completes call fun. Now look what happened: Simple code like a: start -> read -> computer -> write -> read -> got broken into a mess of callbacks and you need a TOOL to help you - enter FSM modeling the callback style a -> ... read on-read_complete(a1) becomes a total mess and needs a graphic tool to solve the problem that should never occurred in the first place. If read is NON blocking (ie does not block other processes) then you don't need callbacks. What about crashing? When you have ONE thread crashing is a BIG DEAL But now we have millions of thread - who cares if one crashes - nobody - "let it crash" But wait a moment - fault tolerance then? - "let some other guy fix the error" Who? some other process. Which process? - enter the link - "the linked processess" So now we can chuck the idea of defensive programming - don't do it. Bye bye to 80% of your code that did error checking :-) Erlang encourages a different mind-set - you can have hundeds and of thousands of processes each running sequential easy to write code a: start -> read -> compute -> read -> read -> write and not a FSM or event framework on the horizon Try it - since (again) I don't know what your problem is I can't really make more than generic observation - but from the Erlang point of view events and so on are a very "70's" way of programming. Cheers /Joe > > Rusi > > From daniel@REDACTED Thu Oct 11 19:43:34 2012 From: daniel@REDACTED (Daniel Luna) Date: Thu, 11 Oct 2012 13:43:34 -0400 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On 11 October 2012 12:38, Raoul Duke wrote: > On Thu, Oct 11, 2012 at 7:17 AM, Rustom Mody wrote: >> In the 'normal' world (ie C++, Java, Python) there is a general >> recommendation that converting a threaded solution to an event-driven >> solution usually speeds up the program and removes hard-to-find bugs. > > please note that you are talking about problems that exist in those > paradigms. there are other paradigms that do not have the exact same > problems (and of course then have their own problems :-). i suspect > The Erlang Way is different enough that it is probably a mistake to > just take something for the usual crowd of programming language > paradigms, and dump it onto erlang. Let me pitch in an agreement here and on top of this mention that there are several reasons the "general recommendation" doesn't even make sense for Erlang: 1. there are no threads in Erlang 2. the standard programming patterns in Erlang (gen_server etc) are already very event driven 3. the paradigm for Erlang in general is different from C++ or Java Cheers, Daniel From erlang@REDACTED Thu Oct 11 19:54:07 2012 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 11 Oct 2012 19:54:07 +0200 Subject: [erlang-questions] Bad Argument, (Bug or something I'm missing). In-Reply-To: References: Message-ID: 1> L = "hello". "hello" 2> << L, " world" >>. ** exception error: bad argument 3> 3> B = <<"hello">>. <<"hello">> 4> << B, " world" >>. ** exception error: bad argument 5> << B/binary, " world" >>. <<"hello world">> << X1 X2 ... >> constructs a binary. The X's can be literal strings "....." or Var/Descriptor. Its described in section 7.16 of the erlang reference manual http://www.erlang.org/doc/reference_manual/expressions.html#id77949 Cheers /Joe On Mon, Oct 1, 2012 at 10:11 AM, Chris Cook wrote: > Morning list, > > I have, > > Erlang R15B01 (erts-5.9.1) [source] [async-threads:0] [hipe] [kernel-poll:false] > > Eshell V5.9.1 (abort with ^G) > 1> List = "89234789234jhk2hk234892789fauky324978". > "89234789234jhk2hk234892789fauky324978" > 2> <>/binary>>. > ** exception error: bad argument > > But when I write it as this, > > 3> <<"89234789234jhk2hk234892789fauky324978", <<"-">>/binary>>. > <<"89234789234jhk2hk234892789fauky324978-">> > > I get the result I expected to get from the above 1 & 2. Could someone > please explain what is going wrong and why, because I'm very confused > with it. > > Regards > > Chris Cook. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From be.dmitry@REDACTED Thu Oct 11 19:55:17 2012 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Thu, 11 Oct 2012 21:55:17 +0400 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> Message-ID: I don't want any feature to be removed. Process dictionaries are useful sometimes. So are parameterized modules. The fact that sometimes people use those features improperly doesn't mean the feature is bad. I hope no features we currently have in the language will be removed. Even experimental ones. -- Dmitry Belyaev On 11.10.2012, at 20:01, Max Lapshin wrote: > On Thu, Oct 11, 2012 at 5:48 PM, Dmitry Belyaev wrote: >> Isn't this feature, I mean process dictionaries, much worse for functional nature of Erlang than parameterized modules? >> >> Why aren't there any words about removing pdics but there are about functional pmods? >> > > Damn! > > Why do you want to remove any feature, you are scared with? > Are you using some library that is using pdict and it spoils your > life? I think no. > > Process dictionary is very, very convenient tool. It is the only way > to get information from process without sending message to it. It is > the only way to get information about process, whose message queue len > is more than several thousands of messages and you ask, when OTP will > remove it? From erlang@REDACTED Thu Oct 11 20:08:26 2012 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 11 Oct 2012 20:08:26 +0200 Subject: [erlang-questions] Bounded Buffer Problems/Solutions In-Reply-To: <022F4CD3-4A8F-4B38-8A5B-41A484B00A72@gmail.com> References: <022F4CD3-4A8F-4B38-8A5B-41A484B00A72@gmail.com> Message-ID: On Wed, Oct 10, 2012 at 9:37 PM, Eric Newhuis wrote: > What are the generally accepted solutions for when producers need to be throttled? gen_server as consumer or otherwise? > > Are there any good solutions or philosophy for the bounded buffer problem in Erlang? ..flow control? Freaky out-of-band communication at a distance to limit messages sent? Use the idea of rope: Message in reverse to a producer shim that he is allowed to send N more messages before he needs to ask for the right to send more? Other? Actually no - we noticed years ago (round 1990) that if producers ran faster than consumers you got a problem. So it this case we made everything synchronous. Send a message, wait for a reply ad nauseam. This is used in gen_servers and virtually everywhere. It doesn't seem to matter I guess if the CPU is always busy we're happy bunnies. Ok so you wait a bit longer, bit something else gets done while you're waiting. Now you could (say) require an ack after every ten messages - and send up to ten messages without requiring an ack or something but things actually go surprisingly well without this ... > > Are there common Erlang idioms for monitoring execution time at the gen_server level so that an API can respond upward with a policy of "the system is currently busier than a one legged man in a kicking contest?" No - easy to implement though. My approach has always been take the simplest solution that works - then measure and only fix things if they are broken. So we've built massive systems with synchronous client server RPCs all over the place then we tune them - most often it's the time to parse inputs or the database that's the problem. Getting data in and out of the system and computation seem to be the problem areas the interprocess message passing is very fast compared to this - so complex buffering seems not to be necessary. I'm not saying you never need it - just that I've never seen it myself. Cheers /Joe > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From rustompmody@REDACTED Thu Oct 11 20:47:34 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Fri, 12 Oct 2012 00:17:34 +0530 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong wrote: > I have no idea - you haven't told me what problem you want to solve - > you've told me the > technique you wish to use to solve the problem. > > This is a bit "back to front" - I'd like to *start* with the the > problem then figure out how to solve > it - not *start* with the idea that you want FSMs and events and so on > and not tell me what the > problem is. > Legacy system in C -- Sits on top of an event-oriented middleware. Since the C system is at core FSM-oriented, it is quite unstructured (from the normal programming POV) Ideally one should throw out the mess and start fresh with an appropriate technology, perhaps Erlang. Realistically, the stakeholders always prefer evolution to revolution. If not for anything else, just for unraveling the old-code semantics some re-engineering is inevitably required -------------- next part -------------- An HTML attachment was scrubbed... URL: From erlang@REDACTED Thu Oct 11 23:07:21 2012 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 11 Oct 2012 23:07:21 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: On Thu, Oct 11, 2012 at 8:47 PM, Rustom Mody wrote: > On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong wrote: >> >> I have no idea - you haven't told me what problem you want to solve - >> you've told me the >> technique you wish to use to solve the problem. >> >> This is a bit "back to front" - I'd like to *start* with the the >> problem then figure out how to solve >> it - not *start* with the idea that you want FSMs and events and so on >> and not tell me what the >> problem is. > > > Legacy system in C -- Sits on top of an event-oriented middleware. > Since the C system is at core FSM-oriented, it is quite unstructured (from > the normal programming POV) Ohhhhh - Ummm - tricky - no easy thing to do here. New technologies have their best chance a) immediately after a disaster or b) at the start of a new project. Cheers /Joe > > Ideally one should throw out the mess and start fresh with an appropriate > technology, perhaps Erlang. > Realistically, the stakeholders always prefer evolution to revolution. > If not for anything else, just for unraveling the old-code semantics some > re-engineering is inevitably required From emmiller@REDACTED Thu Oct 11 23:52:55 2012 From: emmiller@REDACTED (Evan Miller) Date: Thu, 11 Oct 2012 16:52:55 -0500 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: On Wed, Oct 10, 2012 at 2:10 AM, Ulf Wiger wrote: > What would be good is if a list of good uses of pmods could > be collected, and honorable workarounds suggested. > > This could of course be done interactively on this list. > Present what you think is code that is not easily done well > without pmods, and let the members of this list debate > suitable rewrites. If good alternatives are hard to find, > maybe that could inform some EEP for a more > 'erlangish' extension that renders pmods truly obsolete? I will take the bait. As mentioned Chicago Boss uses parameterized modules in a few places -- in SimpleBridge (the connector to Mochiweb et al) and, more importantly, in custom controller code and in BossDB[1]. Before diving into technical details, I will note that if support for pmods is removed in Erlang core, then essentially every user of CB will have to rewrite all of their model code and database accesses, and make significant modifications to their controller code. I don't have statistics on the number of deployments, but we do have 284 members on our mailing list, and the project is one of the ten most-watched Erlang projects on GitHub. So even though pmods are "experimental", a lot of developers are watching the fate of pmods very closely. I designed BossDB around parameterized modules because pmods allow one to develop with a familiar ORM pattern in Erlang. I wanted a record-oriented API that was simple, succinct, and comprehensible. With pmods I was able to succeed in these goals. For example, to create a new record: Person = person:new(id, "Joe", "Armstrong") To save it: case Person:save() of {ok, SavedPerson} -> % do something ... {error, ValidationErrors} -> % do something else ... end To access an attribute: Person:first_name(). "Joe" Person:last_name(). "Armstrong" On the inside, one can access the "internal" parameters directly to construct additional accessors: -module(person, [Id, FirstName, LastName]). full_name() -> FirstName ++ " " ++ LastName. Or perform validation: validation_tests() -> [{fun() -> length(FirstName) > 0, "First name must be non-empty!"}, {fun() -> length(LastName) > 0, "Last name must be non-empty!"}]. In world without pmods, these APIs would be more difficult to read and the custom code would be more difficult to write. For example, the accessors would look like: person:first_name(Person) person:last_name(Person) The custom accessor would look like: full_name(Person) -> first_name(Person) ++ " " ++ last_name(Person). You'd then need to define ways of accessing the internal versions of variables versus the accessed ones (which might be cleaned up for the view, converted to a list, etc). Right now, with pmods, the distinction is simple -- inside your module you can use Variable to get the internal version, use THIS:variable() to get the externally-visible version. To experienced Erlang programmers, the pmod-free code above hardly represent the worst treachery against simplicity. But here, I personally do not find the pure-functional style of coding to be preferable to the pmod way of doing things. For the class of Erlang projects that I work on, the most important thing is for code to be readable. I think it is the code readability that has attracted other users to use Chicago Boss for their own projects. I realize that the use of pmods comes at a cost (e.g. poor dialyzer support), but to be honest, I really don't care. If I can read the code and if it passes the test suite, I will ship it. I would of course prefer that pmods be promoted to first-class citizenship so that we could enjoy the fruits of static analysis, but given the choice between pmods and the dialyzer for a database API, I will choose continue to choose pmods. I have focused the discussion here on BossDB, but there are other times where a pmod is the most convenient way to define callback behavior. You get a few variables "for free" in the parameter list, so the callback writer does not have to accept the same list of variables in every single function. For this reason the web controllers in Chicago Boss are pmods with request information as one variable and the current session ID as another. I am sorry if this set of choices offends anyone's functional sensibilities, but I think everyone should be able to code in a style that is most productive for them. For me that style is Erlang plus judicious use of parameterized modules. I am happy for you to put all the warning signs that you want around the pmod documentation, and to recount all the horrible pmod code that you've had to disentangle as you're swapping war stories around the campfire, but I hope that parameterized modules will exist in future versions of Erlang so that people who find them the be a good solution for a particular class of problems can continue to use them. Respectfully Evan 1. https://github.com/evanmiller/boss_db > > BR, > Ulf W > > > On 10 okt 2012, at 00:54, Tomas Morstein wrote: > >>>> ... >> >> The problem starts when it comes to projects like ChicagoBoss >> that we at IDEA love and intensively use every day. >> CB uses pmods not only for simple_bridge-based web controllers, >> but also for its sophisticated and well integrated concept of >> data models being linked with Django templating engine in turn. >> >> Although I fully understand reasons why pmods and their eventual >> pseudo-OOP behaviour (like inheritance) are not so clear as they >> are implemented today, but in other hand, many excellent projects >> may disappear in final result... >> >> I hope such a "total eclipse" is not going to happen, but our case >> (over-using the extra unsupported and hackish features) we've just >> adapted our technology to framework and totally forgot to standards, >> what's in fact our own mistake. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Evan Miller http://www.evanmiller.org/ From chaos.a.d88@REDACTED Fri Oct 12 00:41:33 2012 From: chaos.a.d88@REDACTED (Anatoly Kanivetskiy) Date: Thu, 11 Oct 2012 15:41:33 -0700 (PDT) Subject: [erlang-questions] enif_send and non-SMP issues (enif_send: env==NULL on non-SMP VMAborted) Message-ID: <10116a6f-30b9-4bcf-9ce7-b758ae99feb0@googlegroups.com> Hello, sirs! Recently, under some dramatic circumstances, I have discovered that NIF's are doomed to brutal termination, when enif_send is called from spawned threads. It forces me to just throw an exception during NIF initialization process, whenever erlang:system_info(smp_support) returns false. As I understood, there is no general solution for this problem, and it can be tricky to gracefully handle this, since usually it forces you to think about two threading modes of your NIF at the same time. So, the question to erlang masters: what will be your advice to all developers, which once will find that their nif's are not more cpu-topology-agnostic, and their applications fails to run on single-core environments? -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Fri Oct 12 03:52:59 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 11 Oct 2012 18:52:59 -0700 Subject: [erlang-questions] CT take a long time to warm up In-Reply-To: <47C3E91A-34B8-4A3A-AE8C-B38C9195DE96@gmail.com> References: <51E69ECE-7A83-4A91-9F2B-DB6EC47668F3@gmail.com> <20120926163945.GA8916@precise> <47C3E91A-34B8-4A3A-AE8C-B38C9195DE96@gmail.com> Message-ID: On Thu, Oct 11, 2012 at 1:45 AM, Tim Watson wrote: > Whoa, this is fascinating Roberto. Did you try running common test from > the shell by hand and see if it still happens, i.e., by running something > like this: > > ct:run_test([{logdir, LogDir}, > {label, Label}, > {auto_compile, true}, > {suite, Suites}]). > > Does the same thing happen? Tried: ct_run -suite mymodule_SUITE same thing happened. r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Oct 12 03:53:25 2012 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 12 Oct 2012 14:53:25 +1300 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> <6560AE95-5912-4C00-9302-2B329BD1074B@gmail.com> Message-ID: On 12/10/2012, at 2:48 AM, Dmitry Belyaev wrote: > Isn't this feature, I mean process dictionaries, much worse for functional nature of Erlang than parameterized modules? "Much worse"? I'm not sure how you would measure that. Processes and message passing (which let you simulate shared mutable variables) clearly cancel out properties that you would otherwise expect a functional language to enjoy, while the process dictionary and parameterised modules do not. > > Why aren't there any words about removing pdics but there are about functional pmods? Because the process dictionary interface has been a documented official this-isn't-going-away part of Erlang for a long time, whereas parameterised modules have always been this-is-experimental-use-at-your-own-risk- but-we-TOLD-you sort of thing. Modules with parameters in ML ('functors') are a very powerful structuring tool, but they go with a number of things Erlang is lacking, like nested modules, module types ('signatures'), and a formal semantics for the things, and still had several major problems that required repeated redesign. It may not be without significance that the Haskell designers, willing to experiment boldly in so many ways, refuse to take one tiny step in that direction. The ability to use frames as quasi-modules (if/when we get frames) will provide an alternative. From roberto@REDACTED Fri Oct 12 03:54:07 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Thu, 11 Oct 2012 18:54:07 -0700 Subject: [erlang-questions] CT take a long time to warm up In-Reply-To: References: <51E69ECE-7A83-4A91-9F2B-DB6EC47668F3@gmail.com> <20120926163945.GA8916@precise> <47C3E91A-34B8-4A3A-AE8C-B38C9195DE96@gmail.com> Message-ID: > > Is anything in PATH or ERL_LIBS & similar environment vars referring > to a non-existent mountpoint? > > A+ > Dave $ echo $PATH /opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin $ERL_LIBS $ nothing much in there. r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Oct 12 04:17:14 2012 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 12 Oct 2012 15:17:14 +1300 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: From my perspective, the event loop approach has always been a bug-ridden nightmare. The idea of turning a multi-thread program into an event loop one in order to *remove* bugs is, wow, magno bizarro. I mean, it's like trying to cope with mice in the house by releasing a lot of large hungry rats. Yes, they'll eat up the mice, but then you have worse problems. People who recommend rats (I mean event loops) are thinking about threads in languages that don't really understand them. It's hard to imagine a Concurrent Pascal, Occam, or Ada programmer thinking their program would be improved by turning it inside out and back to front. But then, Concurrent Pascal, Occam, and Ada compilers have a clue about what's going on, and prevent you writing code that has data races and so on in the first place. If you want high performance, then in a multicore world turning threads into an event loop is the very worst thing you can do, because that one thread is going to run on one core and the others will sit idle. Again, whatever validity the advice may once have had was limited to single-core cpus with heavy-weight threads and high thread switching overheads. Not Erlang. From mjtruog@REDACTED Fri Oct 12 05:53:20 2012 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 11 Oct 2012 20:53:20 -0700 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: <50779430.2080609@gmail.com> On 10/11/2012 11:47 AM, Rustom Mody wrote: > On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong > wrote: > > I have no idea - you haven't told me what problem you want to solve - > you've told me the > technique you wish to use to solve the problem. > > This is a bit "back to front" - I'd like to *start* with the the > problem then figure out how to solve > it - not *start* with the idea that you want FSMs and events and so on > and not tell me what the > problem is. > > > Legacy system in C -- Sits on top of an event-oriented middleware. > Since the C system is at core FSM-oriented, it is quite unstructured (from the normal programming POV) > > Ideally one should throw out the mess and start fresh with an appropriate technology, perhaps Erlang. > Realistically, the stakeholders always prefer evolution to revolution. > If not for anything else, just for unraveling the old-code semantics some re-engineering is inevitably required > There are 4 main ways to integrate Erlang and C/C++ source code: - NIF - port driver - port - cnode The NIF and port driver approach can expose the Erlang VM to any memory problems the C code creates, though it is the most efficient. Usually legacy C or C++ code is best integrated with a port, which is ran as a separate OS process communicating with pipes. I have an open source project here: http://cloudi.org, which can simplify this integration and also provide natural scalability for your legacy source code if: you use the CloudI API (in this case, in C or C++) to communicate with an Erlang CloudI service that handles your external communication that needs to scale. Then you can easily migrate parts of your system, or the whole system, over to Erlang, which will naturally improve your testing, debugging, scalability, reliability, concurrency and the source code size. The integration is similar to integrating with an Erlang port, but it provides features that prevent integration problems, like: - stdout/stderr handling - automatic OS process failure handling (similar to supervisor restarts) - request UUIDs - requests as transactions (i.e., each request receives a response asynchronously) - timeouts are enforced for soft-realtime constraints on the external software - native language integration (both C and C++ have separate interfaces which provide the CloudI API) As with all open source projects, it needs more documentation, but tell me if you have any questions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From codeithere@REDACTED Fri Oct 12 06:03:13 2012 From: codeithere@REDACTED (Code Box) Date: Thu, 11 Oct 2012 21:03:13 -0700 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop Message-ID: ** Reason for termination == ** {timeout,{gen_server,call,[thetime,gettime]}} =CRASH REPORT==== 2012-10-09 05:37:04 UTC === crasher: initial call: process_listener:-init/1-fun-2-/0 pid: <0.12376.513> registered_name: [] exception exit: {timeout,{gen_server,call,[thetime,gettime]}} in function gen_server:terminate/6 ancestors: [incoming_req_processor,incoming_sup,top_process_sup, <0.52.0>] messages: [] links: [] dictionary: [{random_seed,{23375,22820,17046}}] trap_exit: true status: running heap_size: 6765 stack_size: 24 reductions: 1646842 neighbours: I am seeing a lot of these messages in my Crash Reports. Once this reaches this it goes into this crash loop for quite a while. I am not sure how to debug this error. These timeouts are really annoying. Can some one help me understand the root cause of this? Why does my genserver calls are facing timeouts ? Is it that my erlang VM is slow if so why ? How can i debug this issue to get to the root cause of it ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Oct 12 06:14:05 2012 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 11 Oct 2012 21:14:05 -0700 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop In-Reply-To: References: Message-ID: <5077990D.8000207@gmail.com> On 10/11/2012 09:03 PM, Code Box wrote: > ** Reason for termination == > ** {timeout,{gen_server,call,[thetime,gettime]}} > > =CRASH REPORT==== 2012-10-09 05:37:04 UTC === > crasher: > initial call: process_listener:-init/1-fun-2-/0 > pid: <0.12376.513> > registered_name: [] > exception exit: {timeout,{gen_server,call,[thetime,gettime]}} > in function gen_server:terminate/6 > ancestors: [incoming_req_processor,incoming_sup,top_process_sup, > <0.52.0>] > messages: [] > links: [] > dictionary: [{random_seed,{23375,22820,17046}}] > trap_exit: true > status: running > heap_size: 6765 > stack_size: 24 > reductions: 1646842 > neighbours: > > I am seeing a lot of these messages in my Crash Reports. Once this reaches this it goes into this crash loop for quite a while. I am not sure how to debug this error. These timeouts are really annoying. Can some one help me understand the root cause of this? > > Why does my genserver calls are facing timeouts ? Is it that my erlang VM is slow if so why ? How can i debug this issue to get to the root cause of it ? > If you look at gen_server:call/2 at http://www.erlang.org/doc/man/gen_server.html it shows the default Timeout is 5000 milliseconds (5 seconds). Your gen_server process must have been processing for longer than 5 seconds while a gen_server:call/2 message was waiting in the process message queue, to cause the timeout exception. So, it isn't the Erlang VM being slow, it is just an Erlang process that is overloaded (i.e., the "thetime" registered process). -------------- next part -------------- An HTML attachment was scrubbed... URL: From charleshixsn@REDACTED Thu Oct 11 17:35:50 2012 From: charleshixsn@REDACTED (Charles Hixson) Date: Thu, 11 Oct 2012 08:35:50 -0700 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> Message-ID: <5076E756.7050202@earthlink.net> On 10/10/2012 09:31 PM, Richard O'Keefe wrote: > You can think of the process dictionary this way: > > If there is a function clause > > f(X, Y, Z) -> A = g(X, Y), B = h(Z, A), q(A, B) > > replace it by > > f(X, Y, Z, D0) -> > {A, D1} = g(X, Y, D0), > {B, D2} = h(Z, A, D1), > q(A, B, D2) > > and replace > > get(K) > > by > > {get(K, D), D} > > and > put(K, V) > by > {V, put(K, V, D)} > > I've omitted exception handling, but it's not actually > all that different. Tedious rather than difficult. > > So there's an important sense in which it doesn't spoil the > functional purity of the language. (I got this idea from a > Xerox blue-and-white CS technical report giving a functional > semantics for Euclid.) There are bad things that can happen > in imperative languages that still can't happen in Erlang. > > One big bad thing *can* happen. If you call a function you > haven't read, you do not know what it is going to do to the > process dictionary. It could change the value associated > with any key; it could delete any key; it could add any > key->value mapping. Even specifications don't help, > because although there have been several type systems that > include effects, the type system Erlang now uses is not one > of them. > > I thought process dictionaries were "per process". If so, and I don't export the values (which I don't intend to), then how would any other part even have access to them? What these would be used for is things like adjusting the weight of a weighted graph cell, or deciding that a cell was too inactive, and should be rolled out. (I haven't worked out just how to do that, wanting to decide first if I should learn Erlang, or look for another language that could do what I need.) In analogy to C, they would be the equivalent of a static local variable. Or perhaps a C++ class private variable is closer. -- Charles Hixson From codeithere@REDACTED Fri Oct 12 07:05:41 2012 From: codeithere@REDACTED (Code Box) Date: Thu, 11 Oct 2012 22:05:41 -0700 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop In-Reply-To: <5077990D.8000207@gmail.com> References: <5077990D.8000207@gmail.com> Message-ID: Will it not relate to any CPU Stats of my host and also any memory stats of my host that the process is overloaded ? I see CPU % usage as just 50% ? On Thu, Oct 11, 2012 at 9:14 PM, Michael Truog wrote: > ** > On 10/11/2012 09:03 PM, Code Box wrote: > > ** Reason for termination == > ** {timeout,{gen_server,call,[thetime,gettime]}} > > =CRASH REPORT==== 2012-10-09 05:37:04 UTC === > crasher: > initial call: process_listener:-init/1-fun-2-/0 > pid: <0.12376.513> > registered_name: [] > exception exit: {timeout,{gen_server,call,[thetime,gettime]}} > in function gen_server:terminate/6 > ancestors: [incoming_req_processor,incoming_sup,top_process_sup, > <0.52.0>] > messages: [] > links: [] > dictionary: [{random_seed,{23375,22820,17046}}] > trap_exit: true > status: running > heap_size: 6765 > stack_size: 24 > reductions: 1646842 > neighbours: > > I am seeing a lot of these messages in my Crash Reports. Once this > reaches this it goes into this crash loop for quite a while. I am not sure > how to debug this error. These timeouts are really annoying. Can some one > help me understand the root cause of this? > > Why does my genserver calls are facing timeouts ? Is it that my erlang > VM is slow if so why ? How can i debug this issue to get to the root cause > of it ? > > If you look at gen_server:call/2 at > http://www.erlang.org/doc/man/gen_server.html > it shows the default Timeout is 5000 milliseconds (5 seconds). Your > gen_server process must have been processing for longer than 5 seconds > while a gen_server:call/2 message was waiting in the process message queue, > to cause the timeout exception. So, it isn't the Erlang VM being slow, it > is just an Erlang process that is overloaded (i.e., the "thetime" > registered process). > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Fri Oct 12 07:18:27 2012 From: mjtruog@REDACTED (Michael Truog) Date: Thu, 11 Oct 2012 22:18:27 -0700 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop In-Reply-To: References: <5077990D.8000207@gmail.com> Message-ID: <5077A823.6030207@gmail.com> Well a common problem is to have the process also blocked on its own synchronous call, so that can keep the CPU usage low, since it is spending time mostly idle waiting for 1 or more responses from some other processes. The best way I have seen to deal with this type of timeout problem is to always pass the timeouts in the message like this: gen_server:call(, {, Timeout - DELTA}, Timeout) Where DELTA can be 100 milliseconds. Then the (Timeout-DELTA) value the handle_call sees can be used for any internally synchronous calls. However, then the problem becomes understanding what the cumulative delay might be, if there are multiple synchronous calls used within the process. Ideally, the process is kept simpler, so it doesn't need to try and track many synchronous calls. I am not entirely sure if this is your problem, since it could be latency due to function calls too, if function calls are blocking schedulers or something strange, code loading locking schedulers. Usually those issues aren't as common a concern though. On 10/11/2012 10:05 PM, Code Box wrote: > Will it not relate to any CPU Stats of my host and also any memory stats of my host that the process is overloaded ? I see CPU % usage as just 50% ? > > On Thu, Oct 11, 2012 at 9:14 PM, Michael Truog > wrote: > > On 10/11/2012 09:03 PM, Code Box wrote: >> ** Reason for termination == >> ** {timeout,{gen_server,call,[thetime,gettime]}} >> >> =CRASH REPORT==== 2012-10-09 05:37:04 UTC === >> crasher: >> initial call: process_listener:-init/1-fun-2-/0 >> pid: <0.12376.513> >> registered_name: [] >> exception exit: {timeout,{gen_server,call,[thetime,gettime]}} >> in function gen_server:terminate/6 >> ancestors: [incoming_req_processor,incoming_sup,top_process_sup, >> <0.52.0>] >> messages: [] >> links: [] >> dictionary: [{random_seed,{23375,22820,17046}}] >> trap_exit: true >> status: running >> heap_size: 6765 >> stack_size: 24 >> reductions: 1646842 >> neighbours: >> >> I am seeing a lot of these messages in my Crash Reports. Once this reaches this it goes into this crash loop for quite a while. I am not sure how to debug this error. These timeouts are really annoying. Can some one help me understand the root cause of this? >> >> Why does my genserver calls are facing timeouts ? Is it that my erlang VM is slow if so why ? How can i debug this issue to get to the root cause of it ? >> > If you look at gen_server:call/2 at http://www.erlang.org/doc/man/gen_server.html > it shows the default Timeout is 5000 milliseconds (5 seconds). Your gen_server process must have been processing for longer than 5 seconds while a gen_server:call/2 message was waiting in the process message queue, to cause the timeout exception. So, it isn't the Erlang VM being slow, it is just an Erlang process that is overloaded (i.e., the "thetime" registered process). > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody@REDACTED Fri Oct 12 07:58:52 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Fri, 12 Oct 2012 11:28:52 +0530 Subject: [erlang-questions] suitability of erlang In-Reply-To: <50779430.2080609@gmail.com> References: <20121010101533.GA4221@corelatus.se> <50779430.2080609@gmail.com> Message-ID: On Fri, Oct 12, 2012 at 9:23 AM, Michael Truog wrote: There are 4 main ways to integrate Erlang and C/C++ source code: > - NIF > - port driver > - port > - cnode > > Thanks thats a good set of starting points > The NIF and port driver approach can expose the Erlang VM to any memory > problems the C code creates, though it is the most efficient. Usually > legacy C or C++ code is best integrated with a port, which is ran as a > separate OS process communicating with pipes. I have an open source > project here: http://cloudi.org, which can simplify this integration and > also provide natural scalability for your legacy source code ? > Great! I dont yet know our project or Erlang well enough; yet I expect that this will be a help > if: you use the CloudI API (in this case, in C or C++) to communicate with > an Erlang CloudI service that handles your external communication that > needs to scale. Then you can easily migrate parts of your system, or the > whole system, over to Erlang, which will naturally improve your testing, > debugging, scalability, reliability, concurrency and the source code size. > The integration is similar to integrating with an Erlang port, but it > provides features that prevent integration problems, like: > - stdout/stderr handling > - automatic OS process failure handling (similar to supervisor restarts) > - request UUIDs > - requests as transactions (i.e., each request receives a response > asynchronously) > - timeouts are enforced for soft-realtime constraints on the external > software > - native language integration (both C and C++ have separate interfaces > which provide the CloudI API) > > As with all open source projects, it needs more documentation, but tell me > if you have any questions. > > As we start making headway I would be happy to contribute patches to docs. Maybe even code but I am some way off yet :-) Rusi -- http://blog.languager.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Fri Oct 12 08:22:48 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 12 Oct 2012 10:22:48 +0400 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <5076E756.7050202@earthlink.net> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> <5076E756.7050202@earthlink.net> Message-ID: No. Process dictionary is readable outside of process. It should be used only as a well-known hack where you cannot work without it. For example, it is the only way to collect information about busy process. Think about process dictionary as of the way to make labels for processes. For example in erlyvideo it is used to mark processes like: put(flu_name, {stream, <<"stream1">>}) and thus I can understand, that it is this problematic stream, that consumed 5 GB of ram. From rc.china@REDACTED Fri Oct 12 08:37:37 2012 From: rc.china@REDACTED (raocheng) Date: Fri, 12 Oct 2012 14:37:37 +0800 Subject: [erlang-questions] How to get the start parameter when a gen_fsm exits abnormally? Message-ID: I have created a gen_fsm(module: test_service), which is started as a temporary child of supervisor(module: test_service_sup). Since the instance of test_service can be created dynamically, I use the simple_one_for_one strategy. However, when the FSM(test_service) exit abnormally, we can not get the start parameters for the FSM in SUPERVISOR REPORT: 2012-10-12 14:15:28 =SUPERVISOR REPORT==== Supervisor: {local,test_service_sup} Context: child_terminated Reason: timeout Offender: [{pid,<0.2250.0>},{name,test_service},{mfargs,{test_service,start_link,undefined}},{restart_type,temporary},{shutdown,infinity},{child_type,worker}] So it's hard for us to figure out which test_service instance has failed. Is there any better method ? Thanks in advance! FYI: 1. If test_service is registered as a permanent instead of temporary child of test_service_sup, then when it exits abnormally, the UPERVISOR REPORT will include its start parameters. 2. This is the code: 1) -module(test_service). -behaviour(gen_fsm). start_link(Parameter) -> gen_fsm:start_link(?MODULE, [Parameter], []). init([Parameter]) -> ... {ok, trying, state}. 2) -module(test_service_sup). start_child(Parameter) -> supervisor:start_child(?SERVER, [Parameter]). init([]) -> TestServiceSpec = {test_service, {test_service, start_link, []}, temporary, infinity, worker, [test_service]}, {ok, { {simple_one_for_one, 0, 1}, [TestServiceSpec]} }. -------------- next part -------------- An HTML attachment was scrubbed... URL: From codeithere@REDACTED Fri Oct 12 09:15:12 2012 From: codeithere@REDACTED (Code Box) Date: Fri, 12 Oct 2012 00:15:12 -0700 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop In-Reply-To: <5077A823.6030207@gmail.com> References: <5077990D.8000207@gmail.com> <5077A823.6030207@gmail.com> Message-ID: Thanks for your reply. I really appreciate it. I am sure i do have a lot of load on my server like few thousands requests per second. But the process getting time out is not waiting on any other process that call just does a . So I definitely think it is due to the reason that the process is overloaded and all the other requests to that process are in the process queue are getting time outs. I am trying to prove this looking at the Server metrics around CPU, Memory, IO Stats. Talking about IO Stats i do see a big spike in IO Stats. That could be the reason for other processes blocked till the IO is happening which can cause CPU contention. On Thu, Oct 11, 2012 at 10:18 PM, Michael Truog wrote: > ** > Well a common problem is to have the process also blocked on its own > synchronous call, so that can keep the CPU usage low, since it is spending > time mostly idle waiting for 1 or more responses from some other > processes. The best way I have seen to deal with this type of timeout > problem is to always pass the timeouts in the message like this: > gen_server:call(, {, Timeout - DELTA}, Timeout) > Where DELTA can be 100 milliseconds. Then the (Timeout-DELTA) value the > handle_call sees can be used for any internally synchronous calls. > However, then the problem becomes understanding what the cumulative delay > might be, if there are multiple synchronous calls used within the process. > Ideally, the process is kept simpler, so it doesn't need to try and track > many synchronous calls. > > I am not entirely sure if this is your problem, since it could be latency > due to function calls too, if function calls are blocking schedulers or > something strange, code loading locking schedulers. Usually those issues > aren't as common a concern though. > > > On 10/11/2012 10:05 PM, Code Box wrote: > > Will it not relate to any CPU Stats of my host and also any memory stats > of my host that the process is overloaded ? I see CPU % usage as just 50% ? > > On Thu, Oct 11, 2012 at 9:14 PM, Michael Truog wrote: > > On 10/11/2012 09:03 PM, Code Box wrote: > > ** Reason for termination == > ** {timeout,{gen_server,call,[thetime,gettime]}} > > =CRASH REPORT==== 2012-10-09 05:37:04 UTC === > crasher: > initial call: process_listener:-init/1-fun-2-/0 > pid: <0.12376.513> > registered_name: [] > exception exit: {timeout,{gen_server,call,[thetime,gettime]}} > in function gen_server:terminate/6 > ancestors: [incoming_req_processor,incoming_sup,top_process_sup, > <0.52.0>] > messages: [] > links: [] > dictionary: [{random_seed,{23375,22820,17046}}] > trap_exit: true > status: running > heap_size: 6765 > stack_size: 24 > reductions: 1646842 > neighbours: > > I am seeing a lot of these messages in my Crash Reports. Once this > reaches this it goes into this crash loop for quite a while. I am not sure > how to debug this error. These timeouts are really annoying. Can some one > help me understand the root cause of this? > > Why does my genserver calls are facing timeouts ? Is it that my erlang > VM is slow if so why ? How can i debug this issue to get to the root cause > of it ? > > If you look at gen_server:call/2 at > http://www.erlang.org/doc/man/gen_server.html > it shows the default Timeout is 5000 milliseconds (5 seconds). Your > gen_server process must have been processing for longer than 5 seconds > while a gen_server:call/2 message was waiting in the process message queue, > to cause the timeout exception. So, it isn't the Erlang VM being slow, it > is just an Erlang process that is overloaded (i.e., the "thetime" > registered process). > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ok@REDACTED Fri Oct 12 09:28:18 2012 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 12 Oct 2012 20:28:18 +1300 Subject: [erlang-questions] Process Dictionary limitations?? In-Reply-To: <5076E756.7050202@earthlink.net> References: <5075FCF2.6060502@earthlink.net> <507602F9.7080005@gmail.com> <5076238D.1080004@earthlink.net> <507629FE.4080003@gmail.com> <5076E756.7050202@earthlink.net> Message-ID: On 12/10/2012, at 4:35 AM, Charles Hixson wrote: > >> > I thought process dictionaries were "per process". They are. > If so, and I don't export the values (which I don't intend to), then how would any other part even have access to them? I'm not sure what you mean by ``export''. process_info(Pid, dictionary) will let you read the process dictionary of another process. This is of course meant as a debugging tool, and if it bothers you, process_flag(sensitive, true) will let you say ``this process's dictionary is NOT to be revealed.'' The transformation I sketched didn't allow for debugging tools. > > What these would be used for is things like adjusting the weight of a weighted graph cell, or deciding that a cell was too inactive, and should be rolled out. (I haven't worked out just how to do that, wanting to decide first if I should learn Erlang, or look for another language that could do what I need.) In analogy to C, they would be the equivalent of a static local variable. Or perhaps a C++ class private variable is closer. Since static local variables and class private variables are per-program, not per-process, neither would seem to be a good analogue. I get this uneasy feeling that you are letting language features drive design. From mjtruog@REDACTED Fri Oct 12 09:31:13 2012 From: mjtruog@REDACTED (Michael Truog) Date: Fri, 12 Oct 2012 00:31:13 -0700 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop In-Reply-To: References: <5077990D.8000207@gmail.com> <5077A823.6030207@gmail.com> Message-ID: <5077C741.6040706@gmail.com> Ok, if you are experiencing latency with file io, make sure you have async thread pool threads set on the Erlang VM with something like: erl +A 5 If it is related to the async thread pool, the job queue is not shared between the threads... it is a queue per thread, so the size of the async thread pool can impact the wait time... meaning that file io can take longer if the async thread pool is smaller, but you normally don't need a large number of async threads started. If it is socket stuff, it might be related to the encoding, but there are many possibilities down that road. On 10/12/2012 12:15 AM, Code Box wrote: > Thanks for your reply. I really appreciate it. I am sure i do have a lot of load on my server like few thousands requests per second. But the process getting time out is not waiting on any other process that call just does a . So I definitely think it is due to the reason that the process is overloaded and all the other requests to that process are in the process queue are getting time outs. I am trying to prove this looking at the Server metrics around CPU, Memory, IO Stats. Talking about IO Stats i do see a big spike in IO Stats. That could be the reason for other processes blocked till the IO is happening which can cause CPU contention. > > > > On Thu, Oct 11, 2012 at 10:18 PM, Michael Truog > wrote: > > Well a common problem is to have the process also blocked on its own synchronous call, so that can keep the CPU usage low, since it is spending time mostly idle waiting for 1 or more responses from some other processes. The best way I have seen to deal with this type of timeout problem is to always pass the timeouts in the message like this: > gen_server:call(, {, Timeout - DELTA}, Timeout) > Where DELTA can be 100 milliseconds. Then the (Timeout-DELTA) value the handle_call sees can be used for any internally synchronous calls. However, then the problem becomes understanding what the cumulative delay might be, if there are multiple synchronous calls used within the process. Ideally, the process is kept simpler, so it doesn't need to try and track many synchronous calls. > > I am not entirely sure if this is your problem, since it could be latency due to function calls too, if function calls are blocking schedulers or something strange, code loading locking schedulers. Usually those issues aren't as common a concern though. > > > On 10/11/2012 10:05 PM, Code Box wrote: >> Will it not relate to any CPU Stats of my host and also any memory stats of my host that the process is overloaded ? I see CPU % usage as just 50% ? >> >> On Thu, Oct 11, 2012 at 9:14 PM, Michael Truog > wrote: >> >> On 10/11/2012 09:03 PM, Code Box wrote: >>> ** Reason for termination == >>> ** {timeout,{gen_server,call,[thetime,gettime]}} >>> >>> =CRASH REPORT==== 2012-10-09 05:37:04 UTC === >>> crasher: >>> initial call: process_listener:-init/1-fun-2-/0 >>> pid: <0.12376.513> >>> registered_name: [] >>> exception exit: {timeout,{gen_server,call,[thetime,gettime]}} >>> in function gen_server:terminate/6 >>> ancestors: [incoming_req_processor,incoming_sup,top_process_sup, >>> <0.52.0>] >>> messages: [] >>> links: [] >>> dictionary: [{random_seed,{23375,22820,17046}}] >>> trap_exit: true >>> status: running >>> heap_size: 6765 >>> stack_size: 24 >>> reductions: 1646842 >>> neighbours: >>> >>> I am seeing a lot of these messages in my Crash Reports. Once this reaches this it goes into this crash loop for quite a while. I am not sure how to debug this error. These timeouts are really annoying. Can some one help me understand the root cause of this? >>> >>> Why does my genserver calls are facing timeouts ? Is it that my erlang VM is slow if so why ? How can i debug this issue to get to the root cause of it ? >>> >> If you look at gen_server:call/2 at http://www.erlang.org/doc/man/gen_server.html >> it shows the default Timeout is 5000 milliseconds (5 seconds). Your gen_server process must have been processing for longer than 5 seconds while a gen_server:call/2 message was waiting in the process message queue, to cause the timeout exception. So, it isn't the Erlang VM being slow, it is just an Erlang process that is overloaded (i.e., the "thetime" registered process). >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anita.wong@REDACTED Fri Oct 12 09:48:35 2012 From: anita.wong@REDACTED (Anita Wong) Date: Fri, 12 Oct 2012 15:48:35 +0800 Subject: [erlang-questions] gen_tcp receive extremely high CPU Message-ID: Hi All, Sorry that I'm an Erlang newbie and may make stupid question. But please help me to solve the issue. I have written an Erlang server to replace the one I'm using with Node.js, which ate all my memory and I'm praying that Erlang could be a way out. The server works properly under unit test and internal testing, but face a high CPU usage in stress test. After trimming down, I found that the CPU burst was due to the TCP receive from clients. *receiveClientPacket(Sock) ->* * inet:setopts(Sock, [{active, once}, {buffer, ?CLIENTHEARTBEATSIZE}]),* * receive* * {tcp, Sock, Data} ->* * {ok, Data}**;* * {tcp_closed, Sock} ->* * {error, closed}* * after ?CLIENTRECCEIVETIMEOUT ->* * {error, timeout}* * end.* I tried making the process sleep for 10 hours at the beginning of the function (to prevent it from calling receive), the CPU didn't burst at all. Therefore I conclude that the burst of CPU is due to TCP receive. (Please correct me if I made any mistake) Here are information about my stress test: 1. start the Erlang server with: 1. erl +zdbbl 2097151 -K true +A 128 +P 5000000 2. connect 5000 clients to the Erlang server 3. each connected client sends a 2 byte data to the server every 1 min 4. after all the connections is done, (i.e. only the 2 byte data per min are performing), the CPU burst to ~30%sy (from "top") I'm using an Amazon Linux AMI (large instance, 64-bit) for the Erlang server. Is the burst due to the linux? As I have no idea how the system will use up the CPU. Or is it my poor code's problem? (I believe so...) In real situation, our servers don't only receive ping pong, but also messages, which is a lot more loading... This is only the first step... Millions of thanks to anyone who can save me. Anita~* ~~~~~~~~~~~~~~~~~~~~~~~ Information about large instance (for reference): 7.5 GB memory 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) 850 GB instance storage 64-bit platform I/O Performance: High -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleber.p@REDACTED Fri Oct 12 09:55:17 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 12 Oct 2012 09:55:17 +0200 Subject: [erlang-questions] gen_tcp receive extremely high CPU In-Reply-To: References: Message-ID: On Fri, Oct 12, 2012 at 9:48 AM, Anita Wong wrote: > Hi All, > > Sorry that I'm an Erlang newbie and may make stupid question. But please > help me to solve the issue. > > I have written an Erlang server to replace the one I'm using with Node.js, > which ate all my memory and I'm praying that Erlang could be a way out. > The server works properly under unit test and internal testing, but face a > high CPU usage in stress test. > > After trimming down, I found that the CPU burst was due to the TCP receive > from clients. > > receiveClientPacket(Sock) -> > inet:setopts(Sock, [{active, once}, {buffer, ?CLIENTHEARTBEATSIZE}]), > receive > {tcp, Sock, Data} -> > {ok, Data}; > {tcp_closed, Sock} -> > {error, closed} > after ?CLIENTRECCEIVETIMEOUT -> > {error, timeout} > end. > > > I tried making the process sleep for 10 hours at the beginning of the > function (to prevent it from calling receive), the CPU didn't burst at all. > Therefore I conclude that the burst of CPU is due to TCP receive. (Please > correct me if I made any mistake) > > Here are information about my stress test: > > start the Erlang server with: > > erl +zdbbl 2097151 -K true +A 128 +P 5000000 > > connect 5000 clients to the Erlang server > each connected client sends a 2 byte data to the server every 1 min > after all the connections is done, (i.e. only the 2 byte data per min are > performing), the CPU burst to ~30%sy (from "top") > > > I'm using an Amazon Linux AMI (large instance, 64-bit) for the Erlang > server. Is the burst due to the linux? As I have no idea how the system will > use up the CPU. Or is it my poor code's problem? (I believe so...) > > In real situation, our servers don't only receive ping pong, but also > messages, which is a lot more loading... This is only the first step... > > Millions of thanks to anyone who can save me. > > Anita~* > > ~~~~~~~~~~~~~~~~~~~~~~~ > > Information about large instance (for reference): > 7.5 GB memory > 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) > 850 GB instance storage > 64-bit platform > I/O Performance: High Hello Anita Please try rewriting the code to use {active, false} and gen_tcp:recv instead and see if it helps. Also high CPU usage might be due to lock contention of the VM - try running a VM with lock-counters enabled ( http://www.erlang.org/doc/apps/tools/lcnt_chapter.html ) to see if there's some lock which is contended. But the most important question - is 30% CPU burst really a problem? 30% is usually not a big CPU usage, unless you need some sort of real time guarantees. Also Erlang VM uses spin locks and sometimes it uses more CPU than similar code in other language, but spin locks are used to reduce latency. Best regards, Gleb Peregud From gleber.p@REDACTED Fri Oct 12 10:04:23 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Fri, 12 Oct 2012 10:04:23 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: On Thu, Oct 11, 2012 at 11:52 PM, Evan Miller wrote: > I designed BossDB around parameterized modules because pmods allow one > to develop with a familiar ORM pattern in Erlang. I wanted a > record-oriented API that was simple, succinct, and comprehensible. > With pmods I was able to succeed in these goals. For example, to > create a new record: > > Person = person:new(id, "Joe", "Armstrong") > > To save it: > > case Person:save() of > {ok, SavedPerson} -> % do something ... > {error, ValidationErrors} -> % do something else ... > end > > To access an attribute: > > Person:first_name(). > "Joe" > > Person:last_name(). > "Armstrong" I believe that someone from OTP team mentioned that the parametrized modules are to be deleted, but tuple modules will stay. Essentially this will mean that instead of writing: -module(person, [FirstName, LastName]). -compile(export_all). first_name() -> FirstName. You will have to write: -module(person). -record(person, {first_name, last_name}). -compile(export_all). new(FirstName, LastName) -> #person{first_name = FirstName, last_name = LastName}. first_name(Person) -> Perosn#person.first_name. but you will still be able to do: Person = person:new("Joe", "Armstrong"). Person:first_name(). "Joe" case Person:save() of {ok, SavedPerson} -> % do something ... {error, ValidationErrors} -> % do something else ... end. So from point of view of readability there will be no change, althought the amount of code in the model module (person.erl) will slightly increase. Please correct me if I am wrong. Cheers, Gleb From rapsey@REDACTED Fri Oct 12 10:16:19 2012 From: rapsey@REDACTED (Rapsey) Date: Fri, 12 Oct 2012 10:16:19 +0200 Subject: [erlang-questions] gen_tcp receive extremely high CPU In-Reply-To: References: Message-ID: Yeah 30% is really not that much. There is one optimization you can do however. If you have an after block, this means erlang will be setting up a new timer for every packet. I would use erlang:send_after and have a constant timer that only gets executed every X seconds. When it comes to serving network traffic erlang will surely work very well. I would recommend you take a look at the cowboy server and build on that (can be used as a http server or just plain TCP). Sergej On Fri, Oct 12, 2012 at 9:55 AM, Gleb Peregud wrote: > On Fri, Oct 12, 2012 at 9:48 AM, Anita Wong > wrote: > > Hi All, > > > > Sorry that I'm an Erlang newbie and may make stupid question. But please > > help me to solve the issue. > > > > I have written an Erlang server to replace the one I'm using with > Node.js, > > which ate all my memory and I'm praying that Erlang could be a way out. > > The server works properly under unit test and internal testing, but face > a > > high CPU usage in stress test. > > > > After trimming down, I found that the CPU burst was due to the TCP > receive > > from clients. > > > > receiveClientPacket(Sock) -> > > inet:setopts(Sock, [{active, once}, {buffer, ?CLIENTHEARTBEATSIZE}]), > > receive > > {tcp, Sock, Data} -> > > {ok, Data}; > > {tcp_closed, Sock} -> > > {error, closed} > > after ?CLIENTRECCEIVETIMEOUT -> > > {error, timeout} > > end. > > > > > > I tried making the process sleep for 10 hours at the beginning of the > > function (to prevent it from calling receive), the CPU didn't burst at > all. > > Therefore I conclude that the burst of CPU is due to TCP receive. (Please > > correct me if I made any mistake) > > > > Here are information about my stress test: > > > > start the Erlang server with: > > > > erl +zdbbl 2097151 -K true +A 128 +P 5000000 > > > > connect 5000 clients to the Erlang server > > each connected client sends a 2 byte data to the server every 1 min > > after all the connections is done, (i.e. only the 2 byte data per min are > > performing), the CPU burst to ~30%sy (from "top") > > > > > > I'm using an Amazon Linux AMI (large instance, 64-bit) for the Erlang > > server. Is the burst due to the linux? As I have no idea how the system > will > > use up the CPU. Or is it my poor code's problem? (I believe so...) > > > > In real situation, our servers don't only receive ping pong, but also > > messages, which is a lot more loading... This is only the first step... > > > > Millions of thanks to anyone who can save me. > > > > Anita~* > > > > ~~~~~~~~~~~~~~~~~~~~~~~ > > > > Information about large instance (for reference): > > 7.5 GB memory > > 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) > > 850 GB instance storage > > 64-bit platform > > I/O Performance: High > > Hello Anita > > Please try rewriting the code to use {active, false} and gen_tcp:recv > instead and see if it helps. Also high CPU usage might be due to lock > contention of the VM - try running a VM with lock-counters enabled ( > http://www.erlang.org/doc/apps/tools/lcnt_chapter.html ) to see if > there's some lock which is contended. > > But the most important question - is 30% CPU burst really a problem? > 30% is usually not a big CPU usage, unless you need some sort of real > time guarantees. Also Erlang VM uses spin locks and sometimes it uses > more CPU than similar code in other language, but spin locks are used > to reduce latency. > > Best regards, > Gleb Peregud > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomas.morstein@REDACTED Fri Oct 12 10:42:51 2012 From: tomas.morstein@REDACTED (Tomas Morstein) Date: Fri, 12 Oct 2012 10:42:51 +0200 (CEST) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: Message-ID: <843a1fc4-1adc-4f7b-9ce1-8d7decdd6cab@zimbra> > -module(person). > -record(person, {first_name, last_name}). > -compile(export_all). > new(FirstName, LastName) -> > #person{first_name = FirstName, last_name = LastName}. > first_name(Person) -> > Perosn#person.first_name. Ha, that's what I didn't know about! With this feature, I can start to sleep without nightmares again :-) > So from point of view of readability there will be no change, > althought the amount of code in the model module (person.erl) will > slightly increase. What's not so big deal since both boss_db and iodb models are preprocessed (partially generated) by model-compiler. So only what we will have to do is to change model compiler. What's more, I believe boss_db could remain backward compatible because it shouldn't be so problematic to transform -model (person, [FirstName, LastName]) to -model (person). -record (person, {first_name, last_name}). during the preprocessing, before passing to the Erlang compiler itself. With our IODB model compiler, we already do something similar but in another direction -- we define IODB data fields through attributes that are in turn used to generate access functions. Each module that is not wrote as parametrized is automatically converted to pmod by preprocessor. It is also automatically extended by another module with abstract model logic, what's the real problem why I have started this thread ;-) From anita.wong@REDACTED Fri Oct 12 10:44:18 2012 From: anita.wong@REDACTED (Anita Wong) Date: Fri, 12 Oct 2012 16:44:18 +0800 Subject: [erlang-questions] gen_tcp receive extremely high CPU In-Reply-To: References: Message-ID: Thanks for the suggestions. 30% isn't much, but when I establish 50000 connections, the CPU becomes %100. When I simulate some messages, the CPU becomes 180%, and everything jammed. Our production Node.js server serving 70000 connections with ping-pong and messages, most of the time got <5% CPU. That's the reason why I think 30% is a lot, (and it won't stop until I stop the ping pong). I've tried removing the "after" as Sergej said, no difference. I'm reading the doc about the "lock" thing, praying that it can help... I've even tried running the test on http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles, but it gives same result? I think it may be a configuration / compile problem... Anita WONG Senior System Engineer let's try TalkBox. my username is "anita" TalkBox Limited | www.talkboxapp.com D: +852.3526.0238 | M: +852.9821.5819 E: anita.wong@REDACTED Facebook: TalkBox App | Twitter: TalkBoxApp | Weibo: TalkBox On 12 Oct, 2012, at 4:16 PM, Rapsey wrote: > Yeah 30% is really not that much. There is one optimization you can do however. If you have an after block, this means erlang will be setting up a new timer for every packet. I would use erlang:send_after and have a constant timer that only gets executed every X seconds. > > When it comes to serving network traffic erlang will surely work very well. I would recommend you take a look at the cowboy server and build on that (can be used as a http server or just plain TCP). > > > Sergej > > On Fri, Oct 12, 2012 at 9:55 AM, Gleb Peregud wrote: > On Fri, Oct 12, 2012 at 9:48 AM, Anita Wong wrote: > > Hi All, > > > > Sorry that I'm an Erlang newbie and may make stupid question. But please > > help me to solve the issue. > > > > I have written an Erlang server to replace the one I'm using with Node.js, > > which ate all my memory and I'm praying that Erlang could be a way out. > > The server works properly under unit test and internal testing, but face a > > high CPU usage in stress test. > > > > After trimming down, I found that the CPU burst was due to the TCP receive > > from clients. > > > > receiveClientPacket(Sock) -> > > inet:setopts(Sock, [{active, once}, {buffer, ?CLIENTHEARTBEATSIZE}]), > > receive > > {tcp, Sock, Data} -> > > {ok, Data}; > > {tcp_closed, Sock} -> > > {error, closed} > > after ?CLIENTRECCEIVETIMEOUT -> > > {error, timeout} > > end. > > > > > > I tried making the process sleep for 10 hours at the beginning of the > > function (to prevent it from calling receive), the CPU didn't burst at all. > > Therefore I conclude that the burst of CPU is due to TCP receive. (Please > > correct me if I made any mistake) > > > > Here are information about my stress test: > > > > start the Erlang server with: > > > > erl +zdbbl 2097151 -K true +A 128 +P 5000000 > > > > connect 5000 clients to the Erlang server > > each connected client sends a 2 byte data to the server every 1 min > > after all the connections is done, (i.e. only the 2 byte data per min are > > performing), the CPU burst to ~30%sy (from "top") > > > > > > I'm using an Amazon Linux AMI (large instance, 64-bit) for the Erlang > > server. Is the burst due to the linux? As I have no idea how the system will > > use up the CPU. Or is it my poor code's problem? (I believe so...) > > > > In real situation, our servers don't only receive ping pong, but also > > messages, which is a lot more loading... This is only the first step... > > > > Millions of thanks to anyone who can save me. > > > > Anita~* > > > > ~~~~~~~~~~~~~~~~~~~~~~~ > > > > Information about large instance (for reference): > > 7.5 GB memory > > 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) > > 850 GB instance storage > > 64-bit platform > > I/O Performance: High > > Hello Anita > > Please try rewriting the code to use {active, false} and gen_tcp:recv > instead and see if it helps. Also high CPU usage might be due to lock > contention of the VM - try running a VM with lock-counters enabled ( > http://www.erlang.org/doc/apps/tools/lcnt_chapter.html ) to see if > there's some lock which is contended. > > But the most important question - is 30% CPU burst really a problem? > 30% is usually not a big CPU usage, unless you need some sort of real > time guarantees. Also Erlang VM uses spin locks and sometimes it uses > more CPU than similar code in other language, but spin locks are used > to reduce latency. > > Best regards, > Gleb Peregud > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anita.wong@REDACTED Fri Oct 12 12:24:42 2012 From: anita.wong@REDACTED (Anita Wong) Date: Fri, 12 Oct 2012 18:24:42 +0800 Subject: [erlang-questions] gen_tcp receive extremely high CPU In-Reply-To: References: Message-ID: I think this is not the correct question.. I think I need to open another topic, thanks guys! From be.dmitry@REDACTED Fri Oct 12 12:39:53 2012 From: be.dmitry@REDACTED (Dmitry Belyaev) Date: Fri, 12 Oct 2012 14:39:53 +0400 Subject: [erlang-questions] gen_tcp receive extremely high CPU In-Reply-To: References: Message-ID: <69E249CB-A789-4830-9D0A-CC5684E3FC2B@gmail.com> Hi, Anita. Could you tell us the value of CLIENTHEARTBEATSIZE macro? And why do you set the buffer socket option? -- Dmitry Belyaev On 12.10.2012, at 11:48, Anita Wong wrote: > Hi All, > > Sorry that I'm an Erlang newbie and may make stupid question. But please help me to solve the issue. > > I have written an Erlang server to replace the one I'm using with Node.js, which ate all my memory and I'm praying that Erlang could be a way out. > The server works properly under unit test and internal testing, but face a high CPU usage in stress test. > > After trimming down, I found that the CPU burst was due to the TCP receive from clients. > receiveClientPacket(Sock) -> > inet:setopts(Sock, [{active, once}, {buffer, ?CLIENTHEARTBEATSIZE}]), > receive > {tcp, Sock, Data} -> > {ok, Data}; > {tcp_closed, Sock} -> > {error, closed} > after ?CLIENTRECCEIVETIMEOUT -> > {error, timeout} > end. > > I tried making the process sleep for 10 hours at the beginning of the function (to prevent it from calling receive), the CPU didn't burst at all. > Therefore I conclude that the burst of CPU is due to TCP receive. (Please correct me if I made any mistake) > > Here are information about my stress test: > start the Erlang server with: > erl +zdbbl 2097151 -K true +A 128 +P 5000000 > connect 5000 clients to the Erlang server > each connected client sends a 2 byte data to the server every 1 min > after all the connections is done, (i.e. only the 2 byte data per min are performing), the CPU burst to ~30%sy (from "top") > > I'm using an Amazon Linux AMI (large instance, 64-bit) for the Erlang server. Is the burst due to the linux? As I have no idea how the system will use up the CPU. Or is it my poor code's problem? (I believe so...) > > In real situation, our servers don't only receive ping pong, but also messages, which is a lot more loading... This is only the first step... > > Millions of thanks to anyone who can save me. > > Anita~* > > ~~~~~~~~~~~~~~~~~~~~~~~ > > Information about large instance (for reference): > 7.5 GB memory > 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) > 850 GB instance storage > 64-bit platform > I/O Performance: High > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattevans123@REDACTED Fri Oct 12 18:40:22 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Fri, 12 Oct 2012 12:40:22 -0400 Subject: [erlang-questions] Timeout Erlang GenServer Crash Loop In-Reply-To: <5077C741.6040706@gmail.com> References: <5077990D.8000207@gmail.com> <5077A823.6030207@gmail.com>, , <5077C741.6040706@gmail.com> Message-ID: It's hard to answer without knowing what your code is doing (i.e. maybe there is an inefficiency somewhere). However, a common design pattern if your gen_server is doing complex work is to spawn another process to do this task. If you are running multicore the work will be distributed over the different cores: e.g. handle_call({some_operation,Data}, From, State) -> spawn(fun() -> Rsp = do_lots_of_work(), gen_server:reply(From,Rsp) end), {noreply,State}; Date: Fri, 12 Oct 2012 00:31:13 -0700 From: mjtruog@REDACTED To: codeithere@REDACTED CC: erlang-questions@REDACTED Subject: Re: [erlang-questions] Timeout Erlang GenServer Crash Loop Ok, if you are experiencing latency with file io, make sure you have async thread pool threads set on the Erlang VM with something like: erl +A 5 If it is related to the async thread pool, the job queue is not shared between the threads... it is a queue per thread, so the size of the async thread pool can impact the wait time... meaning that file io can take longer if the async thread pool is smaller, but you normally don't need a large number of async threads started. If it is socket stuff, it might be related to the encoding, but there are many possibilities down that road. On 10/12/2012 12:15 AM, Code Box wrote: Thanks for your reply. I really appreciate it. I am sure i do have a lot of load on my server like few thousands requests per second. But the process getting time out is not waiting on any other process that call just does a . So I definitely think it is due to the reason that the process is overloaded and all the other requests to that process are in the process queue are getting time outs. I am trying to prove this looking at the Server metrics around CPU, Memory, IO Stats. Talking about IO Stats i do see a big spike in IO Stats. That could be the reason for other processes blocked till the IO is happening which can cause CPU contention. On Thu, Oct 11, 2012 at 10:18 PM, Michael Truog wrote: Well a common problem is to have the process also blocked on its own synchronous call, so that can keep the CPU usage low, since it is spending time mostly idle waiting for 1 or more responses from some other processes. The best way I have seen to deal with this type of timeout problem is to always pass the timeouts in the message like this: gen_server:call(, {, Timeout - DELTA}, Timeout) Where DELTA can be 100 milliseconds. Then the (Timeout-DELTA) value the handle_call sees can be used for any internally synchronous calls. However, then the problem becomes understanding what the cumulative delay might be, if there are multiple synchronous calls used within the process. Ideally, the process is kept simpler, so it doesn't need to try and track many synchronous calls. I am not entirely sure if this is your problem, since it could be latency due to function calls too, if function calls are blocking schedulers or something strange, code loading locking schedulers. Usually those issues aren't as common a concern though. On 10/11/2012 10:05 PM, Code Box wrote: Will it not relate to any CPU Stats of my host and also any memory stats of my host that the process is overloaded ? I see CPU % usage as just 50% ? On Thu, Oct 11, 2012 at 9:14 PM, Michael Truog wrote: On 10/11/2012 09:03 PM, Code Box wrote: ** Reason for termination == ** {timeout,{gen_server,call,[thetime,gettime]}} =CRASH REPORT==== 2012-10-09 05:37:04 UTC === crasher: initial call: process_listener:-init/1-fun-2-/0 pid: <0.12376.513> registered_name: [] exception exit: {timeout,{gen_server,call,[thetime,gettime]}} in function gen_server:terminate/6 ancestors: [incoming_req_processor,incoming_sup,top_process_sup, <0.52.0>] messages: [] links: [] dictionary: [{random_seed,{23375,22820,17046}}] trap_exit: true status: running heap_size: 6765 stack_size: 24 reductions: 1646842 neighbours: I am seeing a lot of these messages in my Crash Reports. Once this reaches this it goes into this crash loop for quite a while. I am not sure how to debug this error. These timeouts are really annoying. Can some one help me understand the root cause of this? Why does my genserver calls are facing timeouts ? Is it that my erlang VM is slow if so why ? How can i debug this issue to get to the root cause of it ? If you look at gen_server:call/2 at http://www.erlang.org/doc/man/gen_server.html it shows the default Timeout is 5000 milliseconds (5 seconds). Your gen_server process must have been processing for longer than 5 seconds while a gen_server:call/2 message was waiting in the process message queue, to cause the timeout exception. So, it isn't the Erlang VM being slow, it is just an Erlang process that is overloaded (i.e., the "thetime" registered process). _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From yrashk@REDACTED Fri Oct 12 19:35:26 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Fri, 12 Oct 2012 10:35:26 -0700 (PDT) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: >From what I learned earlier this year, tuple modules have a bleak future as well: === [1] > But I was wondering if there's any word out about the fate of tuple > modules? The ones like {erlang}:element(1). Are they expected to be > kept around? (I certainly hope they are :) > No, we don't expect to keep them. As pointed out by others, "tuple modules" is just an implementation detail of parameterized modules (which is an experimental feature). We have not reached a decision yet, but in the future we expect we will do one of two things: * Remove parameterized modules completely from the compiler and run-time system. OR * Implement parameterized modules properly by creating a new data type, both to allow better type checking by Dialyzer, and to possibly improve performance. /Bj?rn ==== On Friday, October 12, 2012 1:04:52 AM UTC-7, Gleb Peregud wrote: > > I believe that someone from OTP team mentioned that the parametrized > modules are to be deleted, but tuple modules will stay. Essentially > this will mean that instead of writing: > > -module(person, [FirstName, LastName]). > -compile(export_all). > first_name() -> > FirstName. > > You will have to write: > > -module(person). > -record(person, {first_name, last_name}). > -compile(export_all). > new(FirstName, LastName) -> > #person{first_name = FirstName, last_name = LastName}. > first_name(Person) -> > Perosn#person.first_name. > > but you will still be able to do: > > Person = person:new("Joe", "Armstrong"). > Person:first_name(). > "Joe" > case Person:save() of > {ok, SavedPerson} -> % do something ... > {error, ValidationErrors} -> % do something else ... > end. > > So from point of view of readability there will be no change, > althought the amount of code in the model module (person.erl) will > slightly increase. > > Please correct me if I am wrong. > > Cheers, > Gleb > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yrashk@REDACTED Fri Oct 12 19:41:15 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Fri, 12 Oct 2012 10:41:15 -0700 (PDT) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: What is even more sad, I've been told that tuple modules will likely be killed in the name of performance. Earlier this year I conducted an experiment and sent its proceedings to Bj?rn and this mailing list, but I have never heard back from him. Here it is again: "I actually just did a totally unscientific test. I took the maint branch, and ran 10,000,000 Mod:fn() calls where Mod was passed from another function call and was just an atom. I ran it on maint, and on my Mac Pro timer:tc clocked 446983 (that's 0.446 seconds). After that, I brutally removed tuple call support from apply and fixed_apply BIFs in beam_emu.c, making `if (is_not_atom(module))` go straight to error label. I also removed `this` variable which was used in those checks because there was no need for it anymore. I re-ran the test, and timer:tc clocked 440680 and around. So, the runtime cost difference was about 6000 microseconds, or 0.006 seconds, for 10 million calls. If I am not mistaking, this is a 1.35% speedup on something as low cost as about 0.0446 microseconds per call (or 0.0440 microseconds per call without support for tuple calls). Is it this big of a difference to support the removal? What if we made tuple calls a documented feature? I'd be glad to write an EEP to support it." On Friday, October 12, 2012 10:35:26 AM UTC-7, Yurii Rashkovskii wrote: > > From what I learned earlier this year, tuple modules have a bleak future > as well: > > === > > [1] > But I was wondering if there's any word out about the fate of tuple > > modules? The ones like {erlang}:element(1). Are they expected to be > > kept around? (I certainly hope they are :) > > > > No, we don't expect to keep them. > > As pointed out by others, "tuple modules" is just an implementation > detail of parameterized modules (which is an experimental feature). > > We have not reached a decision yet, but in the future we expect > we will do one of two things: > > * Remove parameterized modules completely from the compiler > and run-time system. > > OR > > * Implement parameterized modules properly by creating a new > data type, both to allow better type checking by Dialyzer, and > to possibly improve performance. > > /Bj?rn > ==== > > On Friday, October 12, 2012 1:04:52 AM UTC-7, Gleb Peregud wrote: >> >> I believe that someone from OTP team mentioned that the parametrized >> modules are to be deleted, but tuple modules will stay. Essentially >> this will mean that instead of writing: >> >> -module(person, [FirstName, LastName]). >> -compile(export_all). >> first_name() -> >> FirstName. >> >> You will have to write: >> >> -module(person). >> -record(person, {first_name, last_name}). >> -compile(export_all). >> new(FirstName, LastName) -> >> #person{first_name = FirstName, last_name = LastName}. >> first_name(Person) -> >> Perosn#person.first_name. >> >> but you will still be able to do: >> >> Person = person:new("Joe", "Armstrong"). >> Person:first_name(). >> "Joe" >> case Person:save() of >> {ok, SavedPerson} -> % do something ... >> {error, ValidationErrors} -> % do something else ... >> end. >> >> So from point of view of readability there will be no change, >> althought the amount of code in the model module (person.erl) will >> slightly increase. >> >> Please correct me if I am wrong. >> >> Cheers, >> Gleb >> _______________________________________________ >> erlang-questions mailing list >> erlang-q...@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Fri Oct 12 19:46:40 2012 From: ulf@REDACTED (Ulf Wiger) Date: Fri, 12 Oct 2012 19:46:40 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: <1FBAD314-FF4F-4313-81EE-B30C675CA41E@feuerlabs.com> Oh, well? :) OTOH, Bj?rn Gustavsson himself said earlier (8 Oct): > Are you sure? What I have understood from previous email > messages on this list is that many projects use "tuple modules" > (which is an implementation detail in parameterized modules) > and not parameterized modules directly. See for instance: > > http://erlang.org/pipermail/erlang-questions/2012-January/063915.html > > We are thinking about removing the compiler support for parameterized > modules, but keeping the low-level mechanism in erlang:apply/3 and > the inheritance hack in the error_handler module. Just for fun, FWIW, I wrote a little parse transform that gives roughly the same functionality as parameterized modules, but without relying on the special syntax. https://github.com/uwiger/parse_trans/commit/ae7163f3dbc2d3cbf23fb9a796b7f2e576dab09a Note how it's in parse_trans/examples/, so it's mainly for fun. It does seem to work, though. BR, Ulf W On 12 Oct 2012, at 19:35, Yurii Rashkovskii wrote: > From what I learned earlier this year, tuple modules have a bleak future as well: > > === > > [1] > But I was wondering if there's any word out about the fate of tuple > > modules? The ones like {erlang}:element(1). Are they expected to be > > kept around? (I certainly hope they are :) > > > > No, we don't expect to keep them. > > [?] > > /Bj?rn Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Fri Oct 12 21:15:21 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 12 Oct 2012 22:15:21 +0300 Subject: [erlang-questions] binary, ets and memory... Message-ID: Hello, Recently, my system starts to swap. The investigation has indicated that memory consumption was almost twice more then I've expected and to be honest, I've confused why it so... I am talking about Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] So, I do have two processes * first process handles tcp/ip socket I/O. received data is pushed to second process * second process splits binaries binary:split(Buf, [<<$,>>], [global]), parses data and makes list of tuple. When list of tuples is ready, it folds tuples into ets table ets:new(cache, [named_table, ordered_set, public]), lists:foldl( fun({A, B}, Acc) -> ets:insert(cache, {A, B}) end, true, List ). I do have about 6M tuples, where first element is SHA1 signature, second element is integer. Receiver process pushes 100 tuples per time. I hope you got rough idea. When cache is populated, I do have following memory usage, it looks suspicious for me: {total,1179235080}, {processes,2373638}, {processes_used,2373570}, {system,1176861442}, {atom,264505}, {atom_used,253241}, {binary,434761000}, <-- this looks strange for me. Why binaries are left in heap and in ets? {code,6521469}, {ets,732409416} If I change my implementation to lists:foldl( fun({A, B}, Acc) -> ets:insert(cache, {binary:copy(A), B}) end, true, List ). then memory utilization is on the par with my estimates {total,701448856}, {processes,2405251}, {processes_used,2405170}, {system,699043605}, {atom,264505}, {atom_used,253241}, {binary,2686280}, {code,6521493}, {ets,686663080} Best Regards, Dmitry From silent_vendetta@REDACTED Fri Oct 12 21:27:03 2012 From: silent_vendetta@REDACTED (Chris Hicks) Date: Fri, 12 Oct 2012 12:27:03 -0700 Subject: [erlang-questions] binary, ets and memory... In-Reply-To: References: Message-ID: I could be wrong but I'm going to take a guess and say that in the first implementation the whole binary is being kept around and never destroyed. I think what's happening is you are getting a reference to part of a larger binary and passing that around, but the larger binary is sticking around since part of it is still being used. Copying the part you need, and thus creating an entirely new binary, is probably allowing all references to that large binary to disappear so that it can be GC'd. That's a rather naive guess based on what I know about how binaries work. Can anyone else back that up or tell me I'm wrong? Chris > From: dmkolesnikov@REDACTED > Date: Fri, 12 Oct 2012 22:15:21 +0300 > To: erlang-questions@REDACTED > Subject: [erlang-questions] binary, ets and memory... > > Hello, > > Recently, my system starts to swap. The investigation has indicated that memory consumption was almost twice more then I've expected and to be honest, I've confused why it so... > > I am talking about Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] > > So, I do have two processes > * first process handles tcp/ip socket I/O. received data is pushed to second process > * second process splits binaries binary:split(Buf, [<<$,>>], [global]), parses data and makes list of tuple. When list of tuples is ready, it folds tuples into ets table > ets:new(cache, [named_table, ordered_set, public]), > lists:foldl( > fun({A, B}, Acc) -> ets:insert(cache, {A, B}) end, > true, > List > ). > > I do have about 6M tuples, where first element is SHA1 signature, second element is integer. Receiver process pushes 100 tuples per time. I hope you got rough idea. > > When cache is populated, I do have following memory usage, it looks suspicious for me: > {total,1179235080}, > {processes,2373638}, > {processes_used,2373570}, > {system,1176861442}, > {atom,264505}, > {atom_used,253241}, > {binary,434761000}, <-- this looks strange for me. Why binaries are left in heap and in ets? > {code,6521469}, > {ets,732409416} > > If I change my implementation to > lists:foldl( > fun({A, B}, Acc) -> ets:insert(cache, {binary:copy(A), B}) end, > true, > List > ). > then memory utilization is on the par with my estimates > {total,701448856}, > {processes,2405251}, > {processes_used,2405170}, > {system,699043605}, > {atom,264505}, > {atom_used,253241}, > {binary,2686280}, > {code,6521493}, > {ets,686663080} > > Best Regards, > Dmitry > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.lundin@REDACTED Fri Oct 12 21:36:46 2012 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Fri, 12 Oct 2012 21:36:46 +0200 Subject: [erlang-questions] Fwd: Re: (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: <1FBAD314-FF4F-4313-81EE-B30C675CA41E@feuerlabs.com> References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> <1FBAD314-FF4F-4313-81EE-B30C675CA41E@feuerlabs.com> Message-ID: End of speculation, we have made a number of decisions for the upcoming R16B ( end of February). We will describe this in more detail on this list and on erlang.org next week. Parameterized modules is part of this decision, and there is no need to worry. The code using them today will continue to work. But as said more details will come soon. /Kenneth, Erlang/OTP Ericsson ---------- Vidarebefordrat meddelande ---------- Fr?n: "Ulf Wiger" Datum: 12 okt 2012 19:46 ?mne: Re: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? Till: "Yurii Rashkovskii" Kopia: "Tomas Morstein" , , "erlang-questions@REDACTED" Oh, well? :) OTOH, Bj?rn Gustavsson himself said earlier (8 Oct): Are you sure? What I have understood from previous email messages on this list is that many projects use "tuple modules" (which is an implementation detail in parameterized modules) and not parameterized modules directly. See for instance: http://erlang.org/pipermail/erlang-questions/2012-January/063915.html We are thinking about removing the compiler support for parameterized modules, but keeping the low-level mechanism in erlang:apply/3 and the inheritance hack in the error_handler module. Just for fun, FWIW, I wrote a little parse transform that gives roughly the same functionality as parameterized modules, but without relying on the special syntax. https://github.com/uwiger/parse_trans/commit/ae7163f3dbc2d3cbf23fb9a796b7f2e576dab09a Note how it's in parse_trans/examples/, so it's mainly for fun. It does seem to work, though. BR, Ulf W On 12 Oct 2012, at 19:35, Yurii Rashkovskii wrote: >From what I learned earlier this year, tuple modules have a bleak future as well: === [1] > But I was wondering if there's any word out about the fate of tuple > modules? The ones like {erlang}:element(1). Are they expected to be > kept around? (I certainly hope they are :) > No, we don't expect to keep them. [?] /Bj?rn Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. http://feuerlabs.com _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Fri Oct 12 21:42:25 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Fri, 12 Oct 2012 22:42:25 +0300 Subject: [erlang-questions] binary, ets and memory... In-Reply-To: References: Message-ID: this make sense for me, especially taking into account that binary:split returns references to subject but I always thought that ets copies data. But here it seems that data is copied but reference counter to parts is not decreased... - Dmitry On Oct 12, 2012, at 10:27 PM, Chris Hicks wrote: > I could be wrong but I'm going to take a guess and say that in the first implementation the whole binary is being kept around and never destroyed. I think what's happening is you are getting a reference to part of a larger binary and passing that around, but the larger binary is sticking around since part of it is still being used. Copying the part you need, and thus creating an entirely new binary, is probably allowing all references to that large binary to disappear so that it can be GC'd. > > That's a rather naive guess based on what I know about how binaries work. Can anyone else back that up or tell me I'm wrong? > > Chris > > > From: dmkolesnikov@REDACTED > > Date: Fri, 12 Oct 2012 22:15:21 +0300 > > To: erlang-questions@REDACTED > > Subject: [erlang-questions] binary, ets and memory... > > > > Hello, > > > > Recently, my system starts to swap. The investigation has indicated that memory consumption was almost twice more then I've expected and to be honest, I've confused why it so... > > > > I am talking about Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] > > > > So, I do have two processes > > * first process handles tcp/ip socket I/O. received data is pushed to second process > > * second process splits binaries binary:split(Buf, [<<$,>>], [global]), parses data and makes list of tuple. When list of tuples is ready, it folds tuples into ets table > > ets:new(cache, [named_table, ordered_set, public]), > > lists:foldl( > > fun({A, B}, Acc) -> ets:insert(cache, {A, B}) end, > > true, > > List > > ). > > > > I do have about 6M tuples, where first element is SHA1 signature, second element is integer. Receiver process pushes 100 tuples per time. I hope you got rough idea. > > > > When cache is populated, I do have following memory usage, it looks suspicious for me: > > {total,1179235080}, > > {processes,2373638}, > > {processes_used,2373570}, > > {system,1176861442}, > > {atom,264505}, > > {atom_used,253241}, > > {binary,434761000}, <-- this looks strange for me. Why binaries are left in heap and in ets? > > {code,6521469}, > > {ets,732409416} > > > > If I change my implementation to > > lists:foldl( > > fun({A, B}, Acc) -> ets:insert(cache, {binary:copy(A), B}) end, > > true, > > List > > ). > > then memory utilization is on the par with my estimates > > {total,701448856}, > > {processes,2405251}, > > {processes_used,2405170}, > > {system,699043605}, > > {atom,264505}, > > {atom_used,253241}, > > {binary,2686280}, > > {code,6521493}, > > {ets,686663080} > > > > Best Regards, > > Dmitry > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah@REDACTED Fri Oct 12 22:34:06 2012 From: noah@REDACTED (Noah Diewald) Date: Fri, 12 Oct 2012 15:34:06 -0500 Subject: [erlang-questions] Fwd: Re: (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> <1FBAD314-FF4F-4313-81EE-B30C675CA41E@feuerlabs.com> Message-ID: <50787EBE.2050306@diewald.me> There seems to be some problems with the RSS feed. It doesn't work in my favorite programs: http://feedvalidator.org/check.cgi?url=http://www.erlang.org/rss/news On 10/12/2012 02:36 PM, Kenneth Lundin wrote: > End of speculation, we have made a number of decisions for the upcoming > R16B ( end of February). We will describe this in more detail on this > list and on erlang.org > next week. Parameterized modules is part of this decision, and there is > no need to worry. The code using them today will continue to work. > > But as said more details will come soon. > > /Kenneth, Erlang/OTP Ericsson > > ---------- Vidarebefordrat meddelande ---------- > Fr?n: "Ulf Wiger" > > Datum: 12 okt 2012 19:46 > ?mne: Re: [erlang-questions] (Non)Parametrized modules, inheritance, and > R15B02 issues? > Till: "Yurii Rashkovskii" > > Kopia: "Tomas Morstein" >, > >, > "erlang-questions@REDACTED " > > > > > Oh, well? :) > > OTOH, Bj?rn Gustavsson himself said earlier (8 Oct): > >> Are you sure? What I have understood from previous email >> messages on this list is that many projects use "tuple modules" >> (which is an implementation detail in parameterized modules) >> and not parameterized modules directly. See for instance: >> >> http://erlang.org/pipermail/erlang-questions/2012-January/063915.html >> >> We are thinking about removing the compiler support for parameterized >> modules, but keeping the low-level mechanism in erlang:apply/3 and >> the inheritance hack in the error_handler module. > > Just for fun, FWIW, I wrote a little parse transform that gives > roughly the same functionality as parameterized modules, > but without relying on the special syntax. > > https://github.com/uwiger/parse_trans/commit/ae7163f3dbc2d3cbf23fb9a796b7f2e576dab09a > > Note how it's in parse_trans/examples/, so it's mainly for fun. > It does seem to work, though. > > BR, > Ulf W > > On 12 Oct 2012, at 19:35, Yurii Rashkovskii wrote: > >> From what I learned earlier this year, tuple modules have a bleak >> future as well: >> >> === >> >> [1] > But I was wondering if there's any word out about the fate of tuple >> > modules? The ones like {erlang}:element(1). Are they expected to be >> > kept around? (I certainly hope they are :) >> > >> >> No, we don't expect to keep them. >> >> [?] >> >> /Bj?rn > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 551 bytes Desc: OpenPGP digital signature URL: From yrashk@REDACTED Sat Oct 13 08:28:30 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Fri, 12 Oct 2012 23:28:30 -0700 (PDT) Subject: [erlang-questions] Fwd: Re: (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> <1FBAD314-FF4F-4313-81EE-B30C675CA41E@feuerlabs.com> Message-ID: Sounds great! Hope tuple modules are also part of this decision :) Anyway ? a final resolution is a nice thing to have :) On Friday, October 12, 2012 12:36:55 PM UTC-7, Kenneth Lundin wrote: > > End of speculation, we have made a number of decisions for the upcoming > R16B ( end of February). We will describe this in more detail on this list > and on erlang.org > next week. Parameterized modules is part of this decision, and there is > no need to worry. The code using them today will continue to work. > > But as said more details will come soon. > > /Kenneth, Erlang/OTP Ericsson > ---------- Vidarebefordrat meddelande ---------- > Fr?n: "Ulf Wiger" > > Datum: 12 okt 2012 19:46 > ?mne: Re: [erlang-questions] (Non)Parametrized modules, inheritance, and > R15B02 issues? > Till: "Yurii Rashkovskii" > > Kopia: "Tomas Morstein" >, < > erlang-pr...@REDACTED >, "erlang-q...@REDACTED" > > > > > Oh, well? :) > > OTOH, Bj?rn Gustavsson himself said earlier (8 Oct): > > Are you sure? What I have understood from previous email > messages on this list is that many projects use "tuple modules" > (which is an implementation detail in parameterized modules) > and not parameterized modules directly. See for instance: > > http://erlang.org/pipermail/erlang-questions/2012-January/063915.html > > We are thinking about removing the compiler support for parameterized > modules, but keeping the low-level mechanism in erlang:apply/3 and > the inheritance hack in the error_handler module. > > > Just for fun, FWIW, I wrote a little parse transform that gives > roughly the same functionality as parameterized modules, > but without relying on the special syntax. > > > https://github.com/uwiger/parse_trans/commit/ae7163f3dbc2d3cbf23fb9a796b7f2e576dab09a > > Note how it's in parse_trans/examples/, so it's mainly for fun. > It does seem to work, though. > > BR, > Ulf W > > On 12 Oct 2012, at 19:35, Yurii Rashkovskii wrote: > > From what I learned earlier this year, tuple modules have a bleak future > as well: > > === > > [1] > But I was wondering if there's any word out about the fate of tuple > > modules? The ones like {erlang}:element(1). Are they expected to be > > kept around? (I certainly hope they are :) > > > > No, we don't expect to keep them. > > [?] > > /Bj?rn > > > Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc. > http://feuerlabs.com > > > > > _______________________________________________ > erlang-questions mailing list > erlang-q...@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hd2010@REDACTED Sat Oct 13 20:45:21 2012 From: hd2010@REDACTED (Henning Diedrich) Date: Sat, 13 Oct 2012 13:45:21 -0500 Subject: [erlang-questions] Why Erlang? GDC Online 2012 Message-ID: <615E7E5D-B00A-4A24-9569-C2B39EA99399@eonblast.com> Hi list, the slides of my talk at the last GDC Online in Austin, last Wednesday, are up at http://www.slideshare.net/eonblast/why-erlang-gdc-online-2012 It's an intro to Erlang, including for management level folks with past programming experience. But also get-your-toes-wet source examples in its second part. The slides are verbose in an attempt to make them useful for Slideshare-only consumption. Please let me know if you find any errors or grave omissions. Big thanks again to Robert, Joe, Ulf and Felix for their feedback and input. Thanks, Henning From gleber.p@REDACTED Sat Oct 13 21:17:53 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Sat, 13 Oct 2012 21:17:53 +0200 Subject: [erlang-questions] Why Erlang? GDC Online 2012 In-Reply-To: <615E7E5D-B00A-4A24-9569-C2B39EA99399@eonblast.com> References: <615E7E5D-B00A-4A24-9569-C2B39EA99399@eonblast.com> Message-ID: Can you give a link for an announcement of "cures" of strings and records? I must have missed them here at mailing list Cheers, Gleb On Saturday, October 13, 2012, Henning Diedrich wrote: > Hi list, > > the slides of my talk at the last GDC Online in Austin, last Wednesday, > are up at > > http://www.slideshare.net/eonblast/why-erlang-gdc-online-2012 > > It's an intro to Erlang, including for management level folks with past > programming experience. But also get-your-toes-wet source examples in its > second part. > > The slides are verbose in an attempt to make them useful for > Slideshare-only consumption. > > Please let me know if you find any errors or grave omissions. > > Big thanks again to Robert, Joe, Ulf and Felix for their feedback and > input. > > Thanks, > Henning > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Sat Oct 13 22:45:01 2012 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Sat, 13 Oct 2012 22:45:01 +0200 Subject: [erlang-questions] Problem running meck Message-ID: <5079D2CD.9070909@gmail.com> Hi, I have created a rebar.config for my chronos project - https://github.com/lehoff/chronos.git - but I am having problems getting my example to work. I am starting my shell like this: | erl -pz examples -pz deps/*/ebin -pz ebin|| | Then I am trying to run the ping_test:run_test/0, but it fails misserably: 1> ping_test:run_test(). ** exception exit: undef in function chronos_meck_original:start_link/1 called as chronos_meck_original:start_link(ping_timer_server) in call from meck:exec/5 (src/meck.erl, line 418) in call from chronos:start_link/1 called as chronos:start_link(ping_timer_server) in call from ping:init/1 (examples/ping.erl, line 47) in call from gen_server:init_it/6 (gen_server.erl, line 304) in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 227) =ERROR REPORT==== 13-Oct-2012::22:28:13 === ** Generic server chronos_meck terminating ** Last message in was {'EXIT',<0.31.0>, {undef, [{chronos_meck_original,start_link, [ping_timer_server], []}, {meck,exec,5, [{file,"src/meck.erl"},{line,418}]}, {chronos,start_link,[ping_timer_server],[]}, {ping,init,1, [{file,"examples/ping.erl"},{line,47}]}, {gen_server,init_it,6, [{file,"gen_server.erl"},{line,304}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]}} ** When Server state == {state,chronos, {dict,10,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[], [[{code_change,3}|passthrough]], [[{handle_info,2}|passthrough]], [],[], [[{start_link,1}|passthrough], [{stop_timer,2}|passthrough]], [], [[{start_timer,4}| #Fun], [{handle_cast,2}|passthrough]], [[{handle_call,3}|passthrough], [{terminate,2}|passthrough]], [], [[{stop,1}|passthrough]], [],[],[], [[{init,1}|passthrough]]}}}, false, [{<0.44.0>, {chronos,start_link,[ping_timer_server]}, error,undef, [{chronos_meck_original,start_link, [ping_timer_server], []}, {meck,exec,5, [{file,"src/meck.erl"},{line,418}]}, {chronos,start_link,[ping_timer_server]}, {ping,init,1, [{file,"examples/ping.erl"},{line,47}]}, {gen_server,init_it,6, [{file,"gen_server.erl"},{line,304}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]}], {false,no_binary}, false} ** Reason for termination == ** {'module could not be loaded', [{chronos_meck_original,start_link,[ping_timer_server],[]}, {meck,exec,5,[{file,"src/meck.erl"},{line,418}]}, {chronos,start_link,[ping_timer_server],[]}, {ping,init,1,[{file,"examples/ping.erl"},{line,47}]}, {gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]} 2> It seems that the mocking is not working as it should, but I cannot figure out what is the problem. I have also tried to run things from the ebin directory, but it still fails. Any advice on how to get this to work is most welcome! Cheers, ___ /orben -- http://www.linkedin.com/in/torbenhoffmann -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Sun Oct 14 01:23:54 2012 From: steven.charles.davis@REDACTED (Steve Davis) Date: Sat, 13 Oct 2012 18:23:54 -0500 Subject: [erlang-questions] Getting attributes Message-ID: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> Hi all, Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. Each coordinate has a mutable attribute {{x, y}, attribute} If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? What is the correct solution to this without storing everything in "Oracle" and using relational queries? Can this be sensibly maintained in a KV store? /s From steven.charles.davis@REDACTED Sun Oct 14 01:38:32 2012 From: steven.charles.davis@REDACTED (Steve Davis) Date: Sat, 13 Oct 2012 18:38:32 -0500 Subject: [erlang-questions] Getting attributes In-Reply-To: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> Message-ID: <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> I understated the issue: The position of the entity can also change: {id, {x,y}, attribute} So, in fact, the id is the only constant and the pos and attribute are independently mutable. On Oct 13, 2012, at 6:23 PM, Steve Davis wrote: > Hi all, > > Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. > > Each coordinate has a mutable attribute {{x, y}, attribute} > > If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? > > What is the correct solution to this without storing everything in "Oracle" and using relational queries? > > Can this be sensibly maintained in a KV store? > > /s From moxford@REDACTED Sun Oct 14 03:52:17 2012 From: moxford@REDACTED (Mike Oxford) Date: Sat, 13 Oct 2012 18:52:17 -0700 Subject: [erlang-questions] Getting attributes In-Reply-To: <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> Message-ID: Sure; you have two ways of doing this. 1) Key = {x,y} and just make queries for each surrounding node to get the attributes. This requires more read calls. 2) Key = {x,y} and you COPY all of the data from the surrounding nodes into it. This requires only 1 read call, but much more work on the write-side as things move around. On a heavy-write system, #2 will spend a lot of time blocked as everything is updating everything around it when it moves (from cluster + to cluster.) #1 takes more reads which are relatively cheap. It depends on your read/write profile balance. On a very heavy "read" system where thing do not move often, #2 will be best. On a system with lots of moving around, #1 is better. Where that boundary line lies is specific to your own application's patterns and this figuring out this pattern will dictate which method is best (or even if you really want to go KV for this.) Profile it, even if your first-pass profile is just on paper/pencil. -mox On Sat, Oct 13, 2012 at 4:38 PM, Steve Davis wrote: > I understated the issue: > > The position of the entity can also change: > {id, {x,y}, attribute} > > So, in fact, the id is the only constant and the pos and attribute are > independently mutable. > > > > On Oct 13, 2012, at 6:23 PM, Steve Davis > wrote: > > > Hi all, > > > > Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. > > > > Each coordinate has a mutable attribute {{x, y}, attribute} > > > > If I need to query: What are the attributes of positions "next to" my > position, i.e: x +- 1, y +- 1? > > > > What is the correct solution to this without storing everything in > "Oracle" and using relational queries? > > > > Can this be sensibly maintained in a KV store? > > > > /s > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrosenblum@REDACTED Sun Oct 14 04:09:16 2012 From: jrosenblum@REDACTED (Jim) Date: Sat, 13 Oct 2012 22:09:16 -0400 Subject: [erlang-questions] Getting attributes In-Reply-To: References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> Message-ID: <69447ED7-E2A5-4B49-9767-14A0DB7DBE2A@prodigy.net> How about use two index columns consisting if the bit representation of the x and y coordinates as in {key, attribute, 000000000001, 000000000010}. Use mnesia and just do the math on the bits to calculate the next select Depending on your problem, It's possible for the key to be {bitstring1,bitstring2} Sent from my iPhone On Oct 13, 2012, at 9:52 PM, Mike Oxford wrote: > Sure; you have two ways of doing this. > 1) Key = {x,y} and just make queries for each surrounding node to get the attributes. This requires more read calls. > 2) Key = {x,y} and you COPY all of the data from the surrounding nodes into it. This requires only 1 read call, but much more work on the write-side as things move around. > > On a heavy-write system, #2 will spend a lot of time blocked as everything is updating everything around it when it moves (from cluster + to cluster.) > #1 takes more reads which are relatively cheap. > > It depends on your read/write profile balance. On a very heavy "read" system where thing do not move often, #2 will be best. On a system with lots of moving around, #1 is better. Where that boundary line lies is specific to your own application's patterns and this figuring out this pattern will dictate which method is best (or even if you really want to go KV for this.) Profile it, even if your first-pass profile is just on paper/pencil. > > -mox > > On Sat, Oct 13, 2012 at 4:38 PM, Steve Davis wrote: >> I understated the issue: >> >> The position of the entity can also change: >> {id, {x,y}, attribute} >> >> So, in fact, the id is the only constant and the pos and attribute are independently mutable. >> >> >> >> On Oct 13, 2012, at 6:23 PM, Steve Davis wrote: >> >> > Hi all, >> > >> > Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. >> > >> > Each coordinate has a mutable attribute {{x, y}, attribute} >> > >> > If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? >> > >> > What is the correct solution to this without storing everything in "Oracle" and using relational queries? >> > >> > Can this be sensibly maintained in a KV store? >> > >> > /s >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sun Oct 14 08:55:28 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 14 Oct 2012 09:55:28 +0300 Subject: [erlang-questions] Getting attributes In-Reply-To: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> Message-ID: <8B0010F4-8F10-419B-89DF-FC5DE00B14DF@gmail.com> Yes, You can use z-cuve to handle it within key/value store. http://en.wikipedia.org/wiki/Z-order_curve alternative approach, is R-tree but 64k-by-64k might be too much to keep it in-memory. Therefore, MySQL + RTRee index should help you. - Dmitry On Oct 14, 2012, at 2:23 AM, Steve Davis wrote: > Hi all, > > Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. > > Each coordinate has a mutable attribute {{x, y}, attribute} > > If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? > > What is the correct solution to this without storing everything in "Oracle" and using relational queries? > > Can this be sensibly maintained in a KV store? > > /s > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From hd2010@REDACTED Sun Oct 14 10:41:12 2012 From: hd2010@REDACTED (Henning Diedrich) Date: Sun, 14 Oct 2012 03:41:12 -0500 Subject: [erlang-questions] Why Erlang? GDC Online 2012 In-Reply-To: References: <615E7E5D-B00A-4A24-9569-C2B39EA99399@eonblast.com> Message-ID: Could be here: http://www.erlang-factory.com/conference/ErlangUserConference2012/speakers/KennethLundin It's my recollection of the presentation "Latest News from the Erlang OTP Group" at the EUC 2012 by Kenneth Lundin. I did not find slides online but from the year before, hinting in that direction, too. See Slide 10 http://www.erlang-factory.com/upload/presentations/487/EUC_stockholm2011.pdf The alternate for Records would be O'Keefe's frames. For the strings I don't know details. Best, Henning On Oct 13, 2012, at 2:17 PM, Gleb Peregud wrote: > Can you give a link for an announcement of "cures" of strings and records? I must have missed them here at mailing list > > Cheers, > Gleb > > On Saturday, October 13, 2012, Henning Diedrich wrote: > Hi list, > > the slides of my talk at the last GDC Online in Austin, last Wednesday, are up at > > http://www.slideshare.net/eonblast/why-erlang-gdc-online-2012 > > It's an intro to Erlang, including for management level folks with past programming experience. But also get-your-toes-wet source examples in its second part. > > The slides are verbose in an attempt to make them useful for Slideshare-only consumption. > > Please let me know if you find any errors or grave omissions. > > Big thanks again to Robert, Joe, Ulf and Felix for their feedback and input. > > Thanks, > Henning > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.charles.davis@REDACTED Sun Oct 14 17:29:52 2012 From: steven.charles.davis@REDACTED (Steve Davis) Date: Sun, 14 Oct 2012 10:29:52 -0500 Subject: [erlang-questions] Getting attributes In-Reply-To: <69447ED7-E2A5-4B49-9767-14A0DB7DBE2A@prodigy.net> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> <69447ED7-E2A5-4B49-9767-14A0DB7DBE2A@prodigy.net> Message-ID: <4BBEF769-4959-4028-AA6A-10BB6AA2D3B1@gmail.com> Thanks to you all for the excellent practical and educational advice. It's got me moving forward again :-) Best regards, /s From toby@REDACTED Sun Oct 14 19:31:20 2012 From: toby@REDACTED (Toby Thain) Date: Sun, 14 Oct 2012 13:31:20 -0400 Subject: [erlang-questions] Getting attributes In-Reply-To: <8B0010F4-8F10-419B-89DF-FC5DE00B14DF@gmail.com> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <8B0010F4-8F10-419B-89DF-FC5DE00B14DF@gmail.com> Message-ID: <507AF6E8.3060908@telegraphics.com.au> On 14/10/12 2:55 AM, Dmitry Kolesnikov wrote: > Yes, > > You can use z-cuve to handle it within key/value store. > http://en.wikipedia.org/wiki/Z-order_curve > > alternative approach, is R-tree but 64k-by-64k might be too much to keep it in-memory. Therefore, MySQL + RTRee index should help you. > If using a relational database, spatial indexing is not required. Simple tuples are quite enough, just put a composite index on (x,y) since all fetches are (constant,constant). And you can get all neighbours in a single statement. --Toby > > - Dmitry > > On Oct 14, 2012, at 2:23 AM, Steve Davis wrote: > >> Hi all, >> >> Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. >> >> Each coordinate has a mutable attribute {{x, y}, attribute} >> >> If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? >> >> What is the correct solution to this without storing everything in "Oracle" and using relational queries? >> >> Can this be sensibly maintained in a KV store? >> >> /s >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From torben.lehoff@REDACTED Sun Oct 14 21:30:07 2012 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Sun, 14 Oct 2012 21:30:07 +0200 Subject: [erlang-questions] Problem running meck In-Reply-To: <5079D2CD.9070909@gmail.com> References: <5079D2CD.9070909@gmail.com> Message-ID: <507B12BF.8000900@gmail.com> Problem solved! After upgrading to R15B02 things started working - thanks to Adam Lindberg for suggesting that! Cheers, ___ /orben On 2012-10-13 22:45, Torben Hoffmann wrote: > Hi, > > I have created a rebar.config for my chronos project - > https://github.com/lehoff/chronos.git - but I am having problems > getting my example to work. > > I am starting my shell like this: > | erl -pz examples -pz deps/*/ebin -pz ebin|| > | > Then I am trying to run the ping_test:run_test/0, but it fails misserably: > 1> ping_test:run_test(). > ** exception exit: undef > in function chronos_meck_original:start_link/1 > called as chronos_meck_original:start_link(ping_timer_server) > in call from meck:exec/5 (src/meck.erl, line 418) > in call from chronos:start_link/1 > called as chronos:start_link(ping_timer_server) > in call from ping:init/1 (examples/ping.erl, line 47) > in call from gen_server:init_it/6 (gen_server.erl, line 304) > in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 227) > > =ERROR REPORT==== 13-Oct-2012::22:28:13 === > ** Generic server chronos_meck terminating > ** Last message in was {'EXIT',<0.31.0>, > {undef, > [{chronos_meck_original,start_link, > [ping_timer_server], > []}, > {meck,exec,5, > [{file,"src/meck.erl"},{line,418}]}, > {chronos,start_link,[ping_timer_server],[]}, > {ping,init,1, > [{file,"examples/ping.erl"},{line,47}]}, > {gen_server,init_it,6, > [{file,"gen_server.erl"},{line,304}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,227}]}]}} > ** When Server state == {state,chronos, > {dict,10,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], > []}, > {{[],[], > [[{code_change,3}|passthrough]], > [[{handle_info,2}|passthrough]], > [],[], > [[{start_link,1}|passthrough], > [{stop_timer,2}|passthrough]], > [], > [[{start_timer,4}| > #Fun], > [{handle_cast,2}|passthrough]], > [[{handle_call,3}|passthrough], > [{terminate,2}|passthrough]], > [], > [[{stop,1}|passthrough]], > [],[],[], > [[{init,1}|passthrough]]}}}, > false, > [{<0.44.0>, > {chronos,start_link,[ping_timer_server]}, > error,undef, > [{chronos_meck_original,start_link, > [ping_timer_server], > []}, > {meck,exec,5, > [{file,"src/meck.erl"},{line,418}]}, > {chronos,start_link,[ping_timer_server]}, > {ping,init,1, > [{file,"examples/ping.erl"},{line,47}]}, > {gen_server,init_it,6, > [{file,"gen_server.erl"},{line,304}]}, > {proc_lib,init_p_do_apply,3, > [{file,"proc_lib.erl"},{line,227}]}]}], > {false,no_binary}, > false} > ** Reason for termination == > ** {'module could not be loaded', > [{chronos_meck_original,start_link,[ping_timer_server],[]}, > {meck,exec,5,[{file,"src/meck.erl"},{line,418}]}, > {chronos,start_link,[ping_timer_server],[]}, > {ping,init,1,[{file,"examples/ping.erl"},{line,47}]}, > {gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}, > {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]} > 2> > > It seems that the mocking is not working as it should, but I cannot > figure out what is the problem. > I have also tried to run things from the ebin directory, but it still > fails. > > Any advice on how to get this to work is most welcome! > > Cheers, > ___ > /orben > -- > http://www.linkedin.com/in/torbenhoffmann -- http://www.linkedin.com/in/torbenhoffmann -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmkolesnikov@REDACTED Sun Oct 14 21:38:13 2012 From: dmkolesnikov@REDACTED (Dmitry Kolesnikov) Date: Sun, 14 Oct 2012 22:38:13 +0300 Subject: [erlang-questions] Getting attributes In-Reply-To: <507AF6E8.3060908@telegraphics.com.au> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <8B0010F4-8F10-419B-89DF-FC5DE00B14DF@gmail.com> <507AF6E8.3060908@telegraphics.com.au> Message-ID: Toby, Unfortunately, composite index on (x,y) does not give you an optimal performance due to B-Tree index nature. Any (x,y) proximity requests are converted into random disk I/O because B-Tree clusters data first component and then by second. This is a reason why z-order curve were invented. It allows to fit spacial data into B-Tree index and provide you sequential disk I/O for proximity queries. - Dmitry On Oct 14, 2012, at 8:31 PM, Toby Thain wrote: > On 14/10/12 2:55 AM, Dmitry Kolesnikov wrote: >> Yes, >> >> You can use z-cuve to handle it within key/value store. >> http://en.wikipedia.org/wiki/Z-order_curve >> >> alternative approach, is R-tree but 64k-by-64k might be too much to keep it in-memory. Therefore, MySQL + RTRee index should help you. >> > > If using a relational database, spatial indexing is not required. Simple tuples are quite enough, just put a composite index on (x,y) since all fetches are (constant,constant). And you can get all neighbours in a single statement. > > --Toby > >> >> - Dmitry >> >> On Oct 14, 2012, at 2:23 AM, Steve Davis wrote: >> >>> Hi all, >>> >>> Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. >>> >>> Each coordinate has a mutable attribute {{x, y}, attribute} >>> >>> If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? >>> >>> What is the correct solution to this without storing everything in "Oracle" and using relational queries? >>> >>> Can this be sensibly maintained in a KV store? >>> >>> /s >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > From toby@REDACTED Sun Oct 14 22:43:39 2012 From: toby@REDACTED (Toby Thain) Date: Sun, 14 Oct 2012 16:43:39 -0400 Subject: [erlang-questions] Getting attributes In-Reply-To: References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <8B0010F4-8F10-419B-89DF-FC5DE00B14DF@gmail.com> <507AF6E8.3060908@telegraphics.com.au> Message-ID: <507B23FB.7020205@telegraphics.com.au> On 14/10/12 3:38 PM, Dmitry Kolesnikov wrote: > Toby, > > Unfortunately, composite index on (x,y) does not give you an optimal performance due to B-Tree index nature. Any (x,y) proximity requests are converted into random disk I/O because B-Tree clusters data first component and then by second. This is a reason why z-order curve were invented. It allows to fit spacial data into B-Tree index and provide you sequential disk I/O for proximity queries. > > - Dmitry > > A good point about locality. Does the OP require this level of performance, though? There is probably low hanging fruit before one gets to the point of tuning RDBMS index types. --Toby From mattevans123@REDACTED Sun Oct 14 22:58:43 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Sun, 14 Oct 2012 16:58:43 -0400 Subject: [erlang-questions] ets:select badarg question (R15B) Message-ID: Hi, I'm running a select with a continuation on a public ets table with approximately 20,000 records. The original call to the select is: ets:select(TableName,[{'$1',[],['$_']}],RecordsToGet), then an ets:select/1 on the continuation. The process that is spawned to walk the table is sitting in a recursive loop doing a select (grabbing about 150 records each time), processing the matching records (which includes possibly deleting some of these records), then sleeping for a few seconds. It takes about 900 seconds to walk through the table. As well as records getting deleted, new records are added to the table during this time. Every now and then I get an exception in the process doing the select/1: Error in process <0.16218.1> on node 'xxxxx@REDACTED' with exit value: {badarg,[{ets,select,[{'table_E0:39:D7:00:11:80',301,6,<<0 bytes>>,[{p_MacEntry....... (truncated here) This is the code (all running in the same process that started the select) obviously the last ets:select is causing the crash: start_aging1(AgeInterval,MaxAge,SelectData) -> start_aging1(AgeInterval,MaxAge,SelectData,0). start_aging1(_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); start_aging1(_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); start_aging1(AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), timer:sleep(AgeInterval), start_aging1(AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). I'm not sure what could be the cause of the select getting a bad arg. Any ideas? Thanks Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Sun Oct 14 23:29:48 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Oct 2012 14:29:48 -0700 Subject: [erlang-questions] application starting in releases Message-ID: Dear list, AFAIK a release will start the dependency applications defined in an application .app file. So for instance, if I specify: {application, myapp, [ {description, ""}, {vsn, "0.1-dev"}, {modules, []}, {registered, [ ]}, {applications, [ kernel, stdlib, compiler, syntax_tools, lager ]}, {mod, {myapp, []}}, {env, []} ]}. When I package all of this as a release, all of the specified applications will be started automatically when I launch myapp. Is this correct? If so, I want to have a developer start script that starts these applications manually, to emulate what the release will launch for me: I do not want to have extra application:start(compiler) code in myapp, since I won't need that once it's packaged. So I'm using this: erl -pa apps/*/ebin -pa deps/*/ebin \ -boot start_sasl \ -config app \ -s application start compiler -s myapp However compiler does not start: 1> application:which_applications(). [{sasl,"SASL CXC 138 11","2.2.1"}, {stdlib,"ERTS CXC 138 10","1.18.2"}, {kernel,"ERTS CXC 138 10","2.15.2"}] But if I manually start it: 2> application:start(compiler). ok Is there something I'm doing wrong? Am I actually supposed to use application:start/1 in myapp anyways? Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From torben.lehoff@REDACTED Sun Oct 14 23:49:22 2012 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Sun, 14 Oct 2012 23:49:22 +0200 Subject: [erlang-questions] [ANN] Chronos v0.0.4 Message-ID: <507B3362.5010602@gmail.com> Hi, I have updated Chronos to v0.0.4. What is Chronos? Chronos is a timer library for Erlang that makes it easier to work with timers. There is an example that shows how to abstract time away from the code when doing testing cases - this will be useful for anyone doing tests with timers. Changes in v0.0.4 * rebarified Chronos. * Added test/chronos_SUITE.erl so that common test can be used to run QuickCheck. * Cleaned up Makefile Where to get it? https://github.com/lehoff/chronos Cheers, ___ /orben -- http://www.linkedin.com/in/torbenhoffmann -------------- next part -------------- An HTML attachment was scrubbed... URL: From eriksoe@REDACTED Sun Oct 14 23:55:33 2012 From: eriksoe@REDACTED (=?ISO-8859-1?Q?Erik_S=F8e_S=F8rensen?=) Date: Sun, 14 Oct 2012 23:55:33 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: Given that the underlying mechanism stays, and only the syntactical conveniences are under threat: to what extent can the current pmod handling be replaced by / emulated with a parse_transform? If it's possible to do it that way, that might provide the best of two worlds: the existence of the pmod feature without having the maintenance burden in the compiler. Den 12/10/2012 10.04 skrev "Gleb Peregud" : > On Thu, Oct 11, 2012 at 11:52 PM, Evan Miller wrote: > > I designed BossDB around parameterized modules because pmods allow one > > to develop with a familiar ORM pattern in Erlang. I wanted a > > record-oriented API that was simple, succinct, and comprehensible. > > With pmods I was able to succeed in these goals. For example, to > > create a new record: > > > > Person = person:new(id, "Joe", "Armstrong") > > > > To save it: > > > > case Person:save() of > > {ok, SavedPerson} -> % do something ... > > {error, ValidationErrors} -> % do something else ... > > end > > > > To access an attribute: > > > > Person:first_name(). > > "Joe" > > > > Person:last_name(). > > "Armstrong" > > I believe that someone from OTP team mentioned that the parametrized > modules are to be deleted, but tuple modules will stay. Essentially > this will mean that instead of writing: > > -module(person, [FirstName, LastName]). > -compile(export_all). > first_name() -> > FirstName. > > You will have to write: > > -module(person). > -record(person, {first_name, last_name}). > -compile(export_all). > new(FirstName, LastName) -> > #person{first_name = FirstName, last_name = LastName}. > first_name(Person) -> > Perosn#person.first_name. > > but you will still be able to do: > > Person = person:new("Joe", "Armstrong"). > Person:first_name(). > "Joe" > case Person:save() of > {ok, SavedPerson} -> % do something ... > {error, ValidationErrors} -> % do something else ... > end. > > So from point of view of readability there will be no change, > althought the amount of code in the model module (person.erl) will > slightly increase. > > Please correct me if I am wrong. > > Cheers, > Gleb > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freza@REDACTED Sun Oct 14 23:45:26 2012 From: freza@REDACTED (freza@REDACTED) Date: Sun, 14 Oct 2012 17:45:26 -0400 Subject: [erlang-questions] application starting in releases In-Reply-To: References: Message-ID: <20121014214526.GA22270@circlewave.net> On Sun, Oct 14, 2012 at 02:29:48PM -0700, Roberto Ostinelli wrote: > If so, I want to have a developer start script that starts these > applications manually, to emulate what the release will launch for me: I do > not want to have extra application:start(compiler) code in myapp, since I > won't need that once it's packaged. > > So I'm using this: > > erl -pa apps/*/ebin -pa deps/*/ebin \ > -boot start_sasl \ > -config app \ > -s application start compiler This ends up doing "application:start([compiler])" which returns an error, init(3) probably ignores return value as it couldn't really know what to expect. Try with -eval instead: erl -boot start_sasl -eval 'application:start(compiler).' BTW I vaguely recall rebar can do this for you somehow? > Am I actually supposed to use application:start/1 in myapp anyways? No, it's quite rare to do that, the only use case I've seen is bringing protocol stacks up and down manually depending on availability of some backend connections (well, lazy man's way to do that anyway). BR, -- Jachym From ok@REDACTED Mon Oct 15 00:38:23 2012 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 15 Oct 2012 11:38:23 +1300 Subject: [erlang-questions] Why Erlang? GDC Online 2012 In-Reply-To: References: <615E7E5D-B00A-4A24-9569-C2B39EA99399@eonblast.com> Message-ID: > On Saturday, October 13, 2012, Henning Diedrich wrote: > Hi list, > > the slides of my talk at the last GDC Online in Austin, last Wednesday, are up at > > http://www.slideshare.net/eonblast/why-erlang-gdc-online-2012 Nice. It's ConWAY's Game of Life, not ConWELL's. How did you get the 3d figurines? Slide 79 was quite impressive. Slide 96: "Start on capitals" should be "Start WITH capitals", "Statements end on" should be "Statements end WITH", and "short variables names" should be "short variable names". Slide 130, "wheter" should be "whether". Slide 134: "Stackless [Python] has a GIL" -- what's a GIL? Slide 150: "tupels" should be "tuples". From yrashk@REDACTED Mon Oct 15 00:43:51 2012 From: yrashk@REDACTED (Yurii Rashkovskii) Date: Sun, 14 Oct 2012 15:43:51 -0700 (PDT) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: <7a99dd01-628d-41b2-af9e-34bb6210d4bb@googlegroups.com> It is still a question whether the underlying mechanism is to stay. I certainly hope it is. On Sunday, October 14, 2012 2:55:43 PM UTC-7, Erik S?e S?rensen wrote: > > Given that the underlying mechanism stays, and only the syntactical > conveniences are under threat: to what extent can the current pmod handling > be replaced by / emulated with a parse_transform? > If it's possible to do it that way, that might provide the best of two > worlds: the existence of the pmod feature without having the maintenance > burden in the compiler. > Den 12/10/2012 10.04 skrev "Gleb Peregud" > >: > >> On Thu, Oct 11, 2012 at 11:52 PM, Evan Miller > >> wrote: >> > I designed BossDB around parameterized modules because pmods allow one >> > to develop with a familiar ORM pattern in Erlang. I wanted a >> > record-oriented API that was simple, succinct, and comprehensible. >> > With pmods I was able to succeed in these goals. For example, to >> > create a new record: >> > >> > Person = person:new(id, "Joe", "Armstrong") >> > >> > To save it: >> > >> > case Person:save() of >> > {ok, SavedPerson} -> % do something ... >> > {error, ValidationErrors} -> % do something else ... >> > end >> > >> > To access an attribute: >> > >> > Person:first_name(). >> > "Joe" >> > >> > Person:last_name(). >> > "Armstrong" >> >> I believe that someone from OTP team mentioned that the parametrized >> modules are to be deleted, but tuple modules will stay. Essentially >> this will mean that instead of writing: >> >> -module(person, [FirstName, LastName]). >> -compile(export_all). >> first_name() -> >> FirstName. >> >> You will have to write: >> >> -module(person). >> -record(person, {first_name, last_name}). >> -compile(export_all). >> new(FirstName, LastName) -> >> #person{first_name = FirstName, last_name = LastName}. >> first_name(Person) -> >> Perosn#person.first_name. >> >> but you will still be able to do: >> >> Person = person:new("Joe", "Armstrong"). >> Person:first_name(). >> "Joe" >> case Person:save() of >> {ok, SavedPerson} -> % do something ... >> {error, ValidationErrors} -> % do something else ... >> end. >> >> So from point of view of readability there will be no change, >> althought the amount of code in the model module (person.erl) will >> slightly increase. >> >> Please correct me if I am wrong. >> >> Cheers, >> Gleb >> _______________________________________________ >> erlang-questions mailing list >> erlang-q...@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Oct 15 00:56:03 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Oct 2012 15:56:03 -0700 Subject: [erlang-questions] starting an app with an app.config file Message-ID: Dear list, I have an application set up with the standard structure: -- myproject rebar.config *app.config* |-- apps |-- myapp |--src ... |--test myapp_SUITE.erl |-- deps |-- dep1 |-- dep2 |-- ... Please note app.config in there. Everything perfectly fine if I start myapp. However, I'm trying to use Common Tests, and I start myapp in myapp_SUITE.erl during init: init_per_suite(Config) -> ok = application:start(myapp), Config. During startup, myapp tries to access config variables: {ok, Port} = application:get_env(myapp, port), This crashes in test, because application:get_env(myapp, port) returns undefined. This basically means that myapp does not load app.config. How can I solve this? Thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmr@REDACTED Mon Oct 15 00:57:18 2012 From: tmr@REDACTED (Tomas Morstein) Date: Mon, 15 Oct 2012 00:57:18 +0200 (CEST) Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: Message-ID: <33182aa7-173b-4be9-8273-492cc6f14399@zimbra> > Given that the underlying mechanism stays, and only the syntactical > conveniences are under threat: to what extent can the current pmod > handling be replaced by / emulated with a parse_transform? I think this is exactly what Ulf meant by the code presented on Github. We also do some transforms in our database project, so we can do one more new easily, but it has some limitations. > If it's possible to do it that way, that might provide the best of > two worlds: the existence of the pmod feature without having the > maintenance burden in the compiler. It's just a few minutes I was remsh-connected to a live system and did some manual data updates of our data objects (based on pmods). AFAIK, this wouldn't be possible with transforms, so transforms are only partial solution, if we'll simply omit their other disadvantages. I'd wait for what OTP team is going to publish within the upcoming week.. From roberto@REDACTED Mon Oct 15 00:57:17 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Oct 2012 15:57:17 -0700 Subject: [erlang-questions] application starting in releases In-Reply-To: <20121014214526.GA22270@circlewave.net> References: <20121014214526.GA22270@circlewave.net> Message-ID: This ends up doing "application:start([compiler])" which returns an error, > init(3) probably ignores return value as it couldn't really know what to > expect. Try with -eval instead: > > erl -boot start_sasl -eval 'application:start(compiler).' > Oh I see. I wanted to avoid `eval` for some reason, but this works perfectly. Thank you. BTW I vaguely recall rebar can do this for you somehow? No idea. Tuncer? :) > Am I actually supposed to use application:start/1 in myapp anyways? > > No, it's quite rare to do that, the only use case I've seen is bringing > protocol stacks up and down manually depending on availability of some > backend connections (well, lazy man's way to do that anyway). Ok this is exactly what I thought. Again, thank you, r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Oct 15 01:32:42 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Oct 2012 16:32:42 -0700 Subject: [erlang-questions] starting an app with an app.config file In-Reply-To: References: Message-ID: To extend: I can ensure that app.config is used: - when in development mode, by configuring it manually: erl -pa apps/*/ebin -pa deps/*/ebin \ -boot start_sasl \ -config app \ -s myapp - when it is packaged as a release (i believe this can be managed by reltool) But how can I set myapp to use it in common tests? On Sun, Oct 14, 2012 at 3:56 PM, Roberto Ostinelli wrote: > Dear list, > > I have an application set up with the standard structure: > > -- myproject > rebar.config > *app.config* > |-- apps > |-- myapp > |--src > ... > |--test > myapp_SUITE.erl > |-- deps > |-- dep1 > |-- dep2 > |-- ... > > > Please note app.config in there. Everything perfectly fine if I start > myapp. > > However, I'm trying to use Common Tests, and I start myapp in > myapp_SUITE.erl during init: > > init_per_suite(Config) -> > ok = application:start(myapp), > Config. > > During startup, myapp tries to access config variables: > > {ok, Port} = application:get_env(myapp, port), > > This crashes in test, because application:get_env(myapp, port) returns > undefined. This basically means that myapp does not load app.config. > > How can I solve this? > > Thank you, > > r. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Oct 15 04:13:23 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Oct 2012 19:13:23 -0700 Subject: [erlang-questions] [ANN] SublimErl v0.5 Message-ID: Dear list, SublimErl v0.5 has been released. A quick video overview of features is available here: http://www.youtube.com/watch?v=KIzxbjlHmu0 It allows you to: - Benefit from Code Completion ( all Erlang libs + your current project ) - Allows you to Auto-Indent your Erlang code - Run Eunit tests ( all tests for module / single test ) - Run Common Tests ( all tests for module ) - Run Dialyzer tests ( single module ) - Goto any exported function of your project easily - Access man pages from the text editor Feedback always welcome. r. -------------- next part -------------- An HTML attachment was scrubbed... URL: From siraaj.khandkar@REDACTED Mon Oct 15 04:41:18 2012 From: siraaj.khandkar@REDACTED (Siraaj Khandkar) Date: Sun, 14 Oct 2012 22:41:18 -0400 Subject: [erlang-questions] Why Erlang? GDC Online 2012 In-Reply-To: References: <615E7E5D-B00A-4A24-9569-C2B39EA99399@eonblast.com> Message-ID: <1658ACAC-75C1-450B-8BEE-ECCA1F1E3292@gmail.com> On Oct 14, 2012, at 6:38 PM, Richard O'Keefe wrote: ... > Slide 134: "Stackless [Python] has a GIL" -- what's a GIL? Global Interpreter Lock -- Siraaj Khandkar .o. ..o ooo From gopienko@REDACTED Mon Oct 15 07:40:24 2012 From: gopienko@REDACTED (Andrew Gopienko) Date: Mon, 15 Oct 2012 12:40:24 +0700 Subject: [erlang-questions] starting an app with an app.config file In-Reply-To: References: Message-ID: >From my eunit tests SaslSpec = [{errlog_type, error}], case application:load({application,sasl,SaslSpec}) of {error,{already_loaded,_}} -> ok; ok -> ok = application:start(sasl), 2012/10/15 Roberto Ostinelli > To extend: I can ensure that app.config is used: > > > - when in development mode, by configuring it manually: > > erl -pa apps/*/ebin -pa deps/*/ebin \ > -boot start_sasl \ > -config app \ > -s myapp > > - when it is packaged as a release (i believe this can be managed by > reltool) > > But how can I set myapp to use it in common tests? > > > > On Sun, Oct 14, 2012 at 3:56 PM, Roberto Ostinelli wrote: > >> Dear list, >> >> I have an application set up with the standard structure: >> >> -- myproject >> rebar.config >> *app.config* >> |-- apps >> |-- myapp >> |--src >> ... >> |--test >> myapp_SUITE.erl >> |-- deps >> |-- dep1 >> |-- dep2 >> |-- ... >> >> >> Please note app.config in there. Everything perfectly fine if I start >> myapp. >> >> However, I'm trying to use Common Tests, and I start myapp in >> myapp_SUITE.erl during init: >> >> init_per_suite(Config) -> >> ok = application:start(myapp), >> Config. >> >> During startup, myapp tries to access config variables: >> >> {ok, Port} = application:get_env(myapp, port), >> >> This crashes in test, because application:get_env(myapp, port) returns >> undefined. This basically means that myapp does not load app.config. >> >> How can I solve this? >> >> Thank you, >> >> r. >> > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto@REDACTED Mon Oct 15 08:09:40 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Oct 2012 23:09:40 -0700 Subject: [erlang-questions] starting an app with an app.config file In-Reply-To: References: Message-ID: What has this to do with my question? My question is how can I use the same app.config file when I start an application from Common Tests. On Sunday, October 14, 2012, Andrew Gopienko wrote: > From my eunit tests > > SaslSpec = [{errlog_type, error}], > case application:load({application,sasl,SaslSpec}) of > {error,{already_loaded,_}} -> ok; > ok -> > ok = application:start(sasl), > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulf@REDACTED Mon Oct 15 08:30:50 2012 From: ulf@REDACTED (Ulf Wiger) Date: Mon, 15 Oct 2012 08:30:50 +0200 Subject: [erlang-questions] (Non)Parametrized modules, inheritance, and R15B02 issues? In-Reply-To: References: <865f8ce0-05c3-4b6e-b31d-b47724912841@zimbra> Message-ID: I illustrated that on this thread just a few days ago. https://github.com/uwiger/parse_trans/blob/master/examples/pmod.erl https://github.com/uwiger/parse_trans/blob/master/examples/ex_pmod.erl BR, Ulf W Ulf Wiger, Feuerlabs, Inc. http://www.feuerlabs.com 14 okt 2012 kl. 23:55 skrev Erik S?e S?rensen : > Given that the underlying mechanism stays, and only the syntactical conveniences are under threat: to what extent can the current pmod handling be replaced by / emulated with a parse_transform? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jesper.louis.andersen@REDACTED Mon Oct 15 10:36:41 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 15 Oct 2012 10:36:41 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: <20121010101533.GA4221@corelatus.se> Message-ID: <7AE6E3F8-BDAE-40B2-830F-7FBC421FF355@erlang-solutions.com> On Oct 10, 2012, at 8:30 PM, Joe Armstrong wrote: > Even if you know how long computations take, and you know what > resources you have > to solve the computations, then mapping the computations onto the > resources involves > solving the knapsack problem which is NP hard. One would think so. But in practice, the Knapsack problem is pseudo-polynomial and there exists very efficient algorithms for its solution. They are so fast that they often find the knapsack before an O(n lg n) sort has run on the items. So this particular problem is solved. Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From jesper.louis.andersen@REDACTED Mon Oct 15 10:45:51 2012 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 15 Oct 2012 10:45:51 +0200 Subject: [erlang-questions] suitability of erlang In-Reply-To: References: Message-ID: <36CF84B4-AAF0-4916-95ED-C1289B90B6BF@erlang-solutions.com> On Oct 10, 2012, at 8:37 AM, Rustom Mody wrote: > Hello > > We are considering Erlang for a project involving significant data-processing under severe time-crunch > > I am trying to understand the sentence (from http://www.erlang.org/faq/introduction.html#id49576 ) > ---------- > The most common class of 'less suitable' problems is characterised by performance being a prime requirement and constant-factors having a large effect on performance. > ---------- > > How do I understand the '... and constant-factors?' I think, but may be wrong, that it refers to the fact that Erlang is an interpreted language, which is also dynamically typed. There is no way you get such a language to have as low a constant factor for a computation in general as a solution baked up in C. A good example is if you want to do an FFT on some data. Since you have the interpreter overhead, there is no way to win here. At all. The constant factor on this piece of code is going to be higher. But there are two things to be said about this: One, a typical program does not spend its time in a small kernel of computation anymore. It is way more complex and has a lot of interaction with the real world. And if there is a small kernel, you can implement the kernel in a different language and get away with it. Two, I've found that a very large factor of performance stems from the ability to try out clever solutions and be clever about what you do. And in my experience, this is easier to achieve in sane, modern languages like ML, Erlang or (to a certain extent) Haskell. Thus, if you can trade a slow solution for a faster one quickly, then you end up with a faster program in the end. Another factor is time constraint, where a higher level language lets you express the problem quicker. But Erlang does take some knowledge to make perform. Jesper Louis Andersen Erlang Solutions Ltd., Copenhagen From gopienko@REDACTED Mon Oct 15 10:52:55 2012 From: gopienko@REDACTED (Andrew Gopienko) Date: Mon, 15 Oct 2012 15:52:55 +0700 Subject: [erlang-questions] starting an app with an app.config file In-Reply-To: References: Message-ID: Just read your config file and set env with application:load before application:start 2012/10/15 Roberto Ostinelli > What has this to do with my question? > > My question is how can I use the same app.config file when I start an > application from Common Tests. > > > > On Sunday, October 14, 2012, Andrew Gopienko wrote: > >> From my eunit tests >> >> SaslSpec = [{errlog_type, error}], >> case application:load({application,sasl,SaslSpec}) of >> {error,{already_loaded,_}} -> ok; >> ok -> >> ok = application:start(sasl), >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Mon Oct 15 11:49:14 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 15 Oct 2012 11:49:14 +0200 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: References: Message-ID: <507BDC1A.4080309@erix.ericsson.se> What options are you using when the table is created? /Sverker, Erlang/OTP Matthew Evans wrote: > Hi, > I'm running a select with a continuation on a public ets table with approximately 20,000 records. > The original call to the select is: ets:select(TableName,[{'$1',[],['$_']}],RecordsToGet), then an ets:select/1 on the continuation. > > > > > > > > > The process that is spawned to walk the table is sitting in a recursive loop doing a select (grabbing about 150 records each time), processing the matching records (which includes possibly deleting some of these records), then sleeping for a few seconds. It takes about 900 seconds to walk through the table. As well as records getting deleted, new records are added to the table during this time. > Every now and then I get an exception in the process doing the select/1: > Error in process <0.16218.1> on node 'xxxxx@REDACTED' with exit value: {badarg,[{ets,select,[{'table_E0:39:D7:00:11:80',301,6,<<0 bytes>>,[{p_MacEntry....... (truncated here) > This is the code (all running in the same process that started the select) obviously the last ets:select is causing the crash: > > > > > > > > > start_aging1(AgeInterval,MaxAge,SelectData) -> > start_aging1(AgeInterval,MaxAge,SelectData,0). > start_aging1(_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> > gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); > start_aging1(_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> > RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), > gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); > start_aging1(AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> > RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), > timer:sleep(AgeInterval), > start_aging1(AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). > > I'm not sure what could be the cause of the select getting a bad arg. > Any ideas? > Thanks > Matt > > ------------------------------------------------------------------------ > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From s.j.thompson@REDACTED Mon Oct 15 13:20:25 2012 From: s.j.thompson@REDACTED (Simon Thompson) Date: Mon, 15 Oct 2012 12:20:25 +0100 Subject: [erlang-questions] Research Associate in Modelling and Testing with QuickCheck Message-ID: <829593DB-4522-4705-ADCB-3316453F3E29@kent.ac.uk> As a part of the EU-funded project PROWESS we're looking for a postdoctoral research associate for 2 years 11 months to work on modelling and testing of web services, in the context of property-based testing using QuickCheck. PROWESS involves not only Kent, but also the universities of Sheffield and A Coru?a, Chalmers University in Gothenburg and the Polytechnic University of Madrid. Industrial participants include Erlang Solutions Ltd, Quviq AB, Interoud and SP, the Technical Research Institute of Sweden. At the core of PROWESS is property-based testing as implemented in the QuickCheck tool, which provides property-based testing (PBT) for Erlang. In PBT, instead of describing a system through a series of unit tests, it is specified by a set of logical properties, which are then tested on randomly generated inputs to the system. For state-based systems, models are described as state machines, and the systems are exercised through randomly generate paths through the machines. This approach has been used very successfully in a number of application areas, most notably in the automotive sector, where QuickCheck models are used to test automotive software written in C for its conformance to the AUTOSAR standard. The Kent team on PROWESS will help to deliver the following results. ? A library of common properties for PBT of web services. ? Tools and techniques for extraction of specifications and models from existing test suites for web services. ? Tools and techniques to present and assess different competing implementations of a specification. ? Approaches to dealing with evolution of properties, models and systems. ? An approach to assessing the quality of properties by means of mutation of implementations (by analogy with mutation testing). ? Mechanisms by which requirements can be related to other artefacts such as properties and models. The successful applicant should have a doctoral qualification in computer science, experience of using functional programming as well as strong core CS sills. They should be able to work independently, and also to liaise effectively with members of the project both at Kent and at other partners. More details about the post, and the application procedure, are at http://www11.i-grasp.com/fe/tpl_kent01.asp?newms=jj&id=37073&aid=14243 If you have any questions about the position, or would like to discuss it informally, please contact me by email. Simon Thompson Simon Thompson | Professor of Logic and Computation School of Computing | University of Kent | Canterbury, CT2 7NF, UK s.j.thompson@REDACTED | M +44 7986 085754 | W www.cs.kent.ac.uk/~sjt From mattevans123@REDACTED Mon Oct 15 13:52:55 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Mon, 15 Oct 2012 07:52:55 -0400 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: <507BDC1A.4080309@erix.ericsson.se> References: , <507BDC1A.4080309@erix.ericsson.se> Message-ID: Hi These are the options ets:new(StationIdTableName,[{keypos,2},named_table,public]), The only odd things I can think of is that during the select records are getting deleted and added. Also, the select operation is run slowly (over 900 seconds) Again, this isn't every time I get the crash. Matt > Date: Mon, 15 Oct 2012 11:49:14 +0200 > From: sverker.eriksson@REDACTED > To: mattevans123@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] ets:select badarg question (R15B) > > What options are you using when the table is created? > > /Sverker, Erlang/OTP > > Matthew Evans wrote: > > Hi, > > I'm running a select with a continuation on a public ets table with approximately 20,000 records. > > The original call to the select is: ets:select(TableName,[{'$1',[],['$_']}],RecordsToGet), then an ets:select/1 on the continuation. > > > > > > > > > > > > > > > > > > The process that is spawned to walk the table is sitting in a recursive loop doing a select (grabbing about 150 records each time), processing the matching records (which includes possibly deleting some of these records), then sleeping for a few seconds. It takes about 900 seconds to walk through the table. As well as records getting deleted, new records are added to the table during this time. > > Every now and then I get an exception in the process doing the select/1: > > Error in process <0.16218.1> on node 'xxxxx@REDACTED' with exit value: {badarg,[{ets,select,[{'table_E0:39:D7:00:11:80',301,6,<<0 bytes>>,[{p_MacEntry....... (truncated here) > > This is the code (all running in the same process that started the select) obviously the last ets:select is causing the crash: > > > > > > > > > > > > > > > > > > start_aging1(AgeInterval,MaxAge,SelectData) -> > > start_aging1(AgeInterval,MaxAge,SelectData,0). > > start_aging1(_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> > > gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); > > start_aging1(_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> > > RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), > > gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); > > start_aging1(AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> > > RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), > > timer:sleep(AgeInterval), > > start_aging1(AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). > > > > I'm not sure what could be the cause of the select getting a bad arg. > > Any ideas? > > Thanks > > Matt > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Mon Oct 15 14:50:34 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 15 Oct 2012 14:50:34 +0200 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: References: , <507BDC1A.4080309@erix.ericsson.se> Message-ID: <507C069A.9090301@erix.ericsson.se> You need to use ets:safe_fixtable/2 to get a "sensible" result from select/match with continuation on unordered tables. Otherwise you may miss records, get double hits and (as in your case) get badarg if the table has shrunken due to deleted record causing the continuation to point out of table bounds. One could argue that '$end_of_table' would be a more suitable result than badarg for this case. I see that the documentation is not clear about this. It describes usage of safe_fixtable together with first/1 and next/2 but it does not state that it also is needed for select/match with continuation. Single call select/match does not need safe_fixtable. /Sverker, Erlang/OTP Matthew Evans wrote: > Hi > These are the options > > > ets:new(StationIdTableName,[{keypos,2},named_table,public]), > The only odd things I can think of is that during the select records are getting deleted and added. Also, the select operation is run slowly (over 900 seconds) > Again, this isn't every time I get the crash. > Matt > > >> Date: Mon, 15 Oct 2012 11:49:14 +0200 >> From: sverker.eriksson@REDACTED >> To: mattevans123@REDACTED >> CC: erlang-questions@REDACTED >> Subject: Re: [erlang-questions] ets:select badarg question (R15B) >> >> What options are you using when the table is created? >> >> /Sverker, Erlang/OTP >> >> Matthew Evans wrote: >> >>> Hi, >>> I'm running a select with a continuation on a public ets table with approximately 20,000 records. >>> The original call to the select is: ets:select(TableName,[{'$1',[],['$_']}],RecordsToGet), then an ets:select/1 on the continuation. >>> >>> >>> >>> >>> >>> >>> >>> >>> The process that is spawned to walk the table is sitting in a recursive loop doing a select (grabbing about 150 records each time), processing the matching records (which includes possibly deleting some of these records), then sleeping for a few seconds. It takes about 900 seconds to walk through the table. As well as records getting deleted, new records are added to the table during this time. >>> Every now and then I get an exception in the process doing the select/1: >>> Error in process <0.16218.1> on node 'xxxxx@REDACTED' with exit value: {badarg,[{ets,select,[{'table_E0:39:D7:00:11:80',301,6,<<0 bytes>>,[{p_MacEntry....... (truncated here) >>> This is the code (all running in the same process that started the select) obviously the last ets:select is causing the crash: >>> >>> >>> >>> >>> >>> >>> >>> >>> start_aging1(AgeInterval,MaxAge,SelectData) -> >>> start_aging1(AgeInterval,MaxAge,SelectData,0). >>> start_aging1(_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); >>> start_aging1(_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> >>> RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); >>> start_aging1(AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> >>> RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), >>> timer:sleep(AgeInterval), >>> start_aging1(AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). >>> >>> I'm not sure what could be the cause of the select getting a bad arg. >>> Any ideas? >>> Thanks >>> Matt >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> erlang-questions mailing list >>> erlang-questions@REDACTED >>> http://erlang.org/mailman/listinfo/erlang-questions >>> >>> > > From mattevans123@REDACTED Mon Oct 15 16:54:11 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Mon, 15 Oct 2012 10:54:11 -0400 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: <507C069A.9090301@erix.ericsson.se> References: , <507BDC1A.4080309@erix.ericsson.se> , <507C069A.9090301@erix.ericsson.se> Message-ID: Many thanks, I'll give that a go. I am however a little worried about having a safe_fixtable "active" on a table for 900 seconds so might need to be a little smarter. Cheers Matt > Date: Mon, 15 Oct 2012 14:50:34 +0200 > From: sverker.eriksson@REDACTED > To: mattevans123@REDACTED > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] ets:select badarg question (R15B) > > You need to use ets:safe_fixtable/2 to get a "sensible" result from > select/match with continuation on unordered tables. Otherwise you may > miss records, get double hits and (as in your case) get badarg if the > table has shrunken due to deleted record causing the continuation to > point out of table bounds. One could argue that '$end_of_table' would be > a more suitable result than badarg for this case. > > I see that the documentation is not clear about this. It describes usage > of safe_fixtable together with first/1 and next/2 but it does not state > that it also is needed for select/match with continuation. > Single call select/match does not need safe_fixtable. > > /Sverker, Erlang/OTP > > Matthew Evans wrote: > > Hi > > These are the options > > > > > > ets:new(StationIdTableName,[{keypos,2},named_table,public]), > > The only odd things I can think of is that during the select records are getting deleted and added. Also, the select operation is run slowly (over 900 seconds) > > Again, this isn't every time I get the crash. > > Matt > > > > > >> Date: Mon, 15 Oct 2012 11:49:14 +0200 > >> From: sverker.eriksson@REDACTED > >> To: mattevans123@REDACTED > >> CC: erlang-questions@REDACTED > >> Subject: Re: [erlang-questions] ets:select badarg question (R15B) > >> > >> What options are you using when the table is created? > >> > >> /Sverker, Erlang/OTP > >> > >> Matthew Evans wrote: > >> > >>> Hi, > >>> I'm running a select with a continuation on a public ets table with approximately 20,000 records. > >>> The original call to the select is: ets:select(TableName,[{'$1',[],['$_']}],RecordsToGet), then an ets:select/1 on the continuation. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> The process that is spawned to walk the table is sitting in a recursive loop doing a select (grabbing about 150 records each time), processing the matching records (which includes possibly deleting some of these records), then sleeping for a few seconds. It takes about 900 seconds to walk through the table. As well as records getting deleted, new records are added to the table during this time. > >>> Every now and then I get an exception in the process doing the select/1: > >>> Error in process <0.16218.1> on node 'xxxxx@REDACTED' with exit value: {badarg,[{ets,select,[{'table_E0:39:D7:00:11:80',301,6,<<0 bytes>>,[{p_MacEntry....... (truncated here) > >>> This is the code (all running in the same process that started the select) obviously the last ets:select is causing the crash: > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> start_aging1(AgeInterval,MaxAge,SelectData) -> > >>> start_aging1(AgeInterval,MaxAge,SelectData,0). > >>> start_aging1(_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> > >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); > >>> start_aging1(_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> > >>> RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), > >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); > >>> start_aging1(AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> > >>> RecordsAgedCount = age_records(Matches,[],MaxAge*1000000,os:timestamp()), > >>> timer:sleep(AgeInterval), > >>> start_aging1(AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). > >>> > >>> I'm not sure what could be the cause of the select getting a bad arg. > >>> Any ideas? > >>> Thanks > >>> Matt > >>> > >>> ------------------------------------------------------------------------ > >>> > >>> _______________________________________________ > >>> erlang-questions mailing list > >>> erlang-questions@REDACTED > >>> http://erlang.org/mailman/listinfo/erlang-questions > >>> > >>> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattevans123@REDACTED Mon Oct 15 17:09:18 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Mon, 15 Oct 2012 11:09:18 -0400 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: References: , , <507BDC1A.4080309@erix.ericsson.se>, , , <507C069A.9090301@erix.ericsson.se>, Message-ID: I'll rephrase that. This is my concern from the ets documentation for safe_fixtable/2: Note that no deleted objects are actually removed from a fixed table until it has been released. If a process fixes a table but never releases it, the memory used by the deleted objects will never be freed. The performance of operations on the table will also degrade significantly. Is there a potential problem on having a safe_fixtable "active" for long periods of time on a table having lots of inserts and deletes? Thanks Matt________________________________ > From: mattevans23@@hotmail.com > To: sverker.eriksson@REDACTED > Date: Mon, 5 Oct 012 0::4::1 -400 > CC: erlang-questions@REDACTED > Subject: Re: [erlang-questions] ets:select badarg question (R5BB) > > Many thanks, > > I'll give that a go. I am however a little worried about having a > safe_fixtable "active" on a table for 00 seconds so might need to be a > little smarter. > > Cheers > > Matt > > > Date: Mon, 5 Oct 012 4::0::4 +200 > > From: sverker.eriksson@REDACTED > > To: mattevans23@@hotmail.com > > CC: erlang-questions@REDACTED > > Subject: Re: [erlang-questions] ets:select badarg question (R5BB) > > > > You need to use ets:safe_fixtable/ to get a "sensible" result from > > select/match with continuation on unordered tables. Otherwise you may > > miss records, get double hits and (as in your case) get badarg if the > > table has shrunken due to deleted record causing the continuation to > > point out of table bounds. One could argue that '$end_of_table' would be > > a more suitable result than badarg for this case. > > > > I see that the documentation is not clear about this. It describes usage > > of safe_fixtable together with first/ and next/ but it does not state > > that it also is needed for select/match with continuation. > > Single call select/match does not need safe_fixtable. > > > > /Sverker, Erlang/OTP > > > > Matthew Evans wrote: > > > Hi > > > These are the options > > > > > > > > > ets:new(StationIdTableName,[{keypos,}},named_table,public]), > > > The only odd things I can think of is that during the select > records are getting deleted and added. Also, the select operation is > run slowly (over 00 seconds) > > > Again, this isn't every time I get the crash. > > > Matt > > > > > > > > >> Date: Mon, 5 Oct 012 1::9::4 +200 > > >> From: sverker.eriksson@REDACTED > > >> To: mattevans23@@hotmail.com > > >> CC: erlang-questions@REDACTED > > >> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) > > >> > > >> What options are you using when the table is created? > > >> > > >> /Sverker, Erlang/OTP > > >> > > >> Matthew Evans wrote: > > >> > > >>> Hi, > > >>> I'm running a select with a continuation on a public ets table > with approximately 0,,00 records. > > >>> The original call to the select is: > ets:select(TableName,[{'$'',[],['$_']}],RecordsToGet), then an > ets:select/ on the continuation. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> The process that is spawned to walk the table is sitting in a > recursive loop doing a select (grabbing about 50 records each time), > processing the matching records (which includes possibly deleting some > of these records), then sleeping for a few seconds. It takes about 00 > seconds to walk through the table. As well as records getting deleted, > new records are added to the table during this time. > > >>> Every now and then I get an exception in the process doing the > select/:: > > >>> Error in process <..6218..>> on node 'xxxxx@REDACTED'' with > exit value: {badarg,[{ets,select,[{'table_E::9::D::0::1::0'',01,,,,<< > bytes>>,[{p_MacEntry....... (truncated here) > > >>> This is the code (all running in the same process that started > the select) obviously the last ets:select is causing the crash: > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> start_aging((AgeInterval,MaxAge,SelectData) -> > > >>> start_aging((AgeInterval,MaxAge,SelectData,)). > > >>> start_aging((_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> > > >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); > > >>> start_aging((_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> > > >>> RecordsAgedCount = > age_records(Matches,[],MaxAge*000000,,os:timestamp()), > > >>> > gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); > > >>> start_aging((AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> > > >>> RecordsAgedCount = > age_records(Matches,[],MaxAge*000000,,os:timestamp()), > > >>> timer:sleep(AgeInterval), > > >>> > start_aging((AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). > > >>> > > >>> I'm not sure what could be the cause of the select getting a bad arg. > > >>> Any ideas? > > >>> Thanks > > >>> Matt > > >>> > > >>> > ------------------------------------------------------------------------ > > >>> > > >>> _______________________________________________ > > >>> erlang-questions mailing list > > >>> erlang-questions@REDACTED > > >>> http://erlang.org/mailman/listinfo/erlang-questions > > >>> > > >>> > > > > > > > > > > _______________________________________________ erlang-questions > mailing list erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From essen@REDACTED Mon Oct 15 17:17:54 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 15 Oct 2012 17:17:54 +0200 Subject: [erlang-questions] Process/FD leak in SSL R15B01 Message-ID: <507C2922.9090207@ninenines.eu> Hello, We are observing a constant increase in the number of ssl_connection processes. We have investigated the problem and are not sure how to go about it further, considering the results. We switched from using ssltunnel to directly speaking to the Erlang nodes in SSL for speaking with our mobile applications. We observed that we have right now around 25000 sockets in ESTABLISHED (increasing over time) and around the same in FIN_WAIT2 (also increasing over time). In total it increases at about 10000 sockets per day. Looking at the applications running it appears only the SSL application has extra processes compared to normal. The other parts of the system appear to be functioning normally. What's more alarming is that there is a lot more SSL connection processes than actual connection processes in Cowboy (between 10-15K and not increasing over time). Investigating the SSL application to find the source, I ended up applying the following snippet to the list of PIDs supervised by ssl_connection_sup: > lists:foldl(fun(X, Sum) -> case erlang:process_info(X) of undefined -> Sum; [{current_function, XXX}|_] -> case lists:keyfind(XXX, 1, Sum) of false -> Curr = 0; {_, Curr} -> ok end, lists:keystore(XXX, 1, Sum, {XXX, Curr + 1}) end end, [], List). [{{prim_inet,recv0,3},25856},{{gen_fsm,loop,7},26574}] Not sure which one is the ESTABLISHED list and which one is the FIN_WAIT2. Of course, I can't use sys:get_status/1 on the PIDs stuck in prim_inet:recv0/3 because the receive there is quite specific. So I can't get the stacktrace. The other case doesn't seem to give anything useful (for my level of knowledge, anyway). The process that owns the SSL socket died. So the ssl_connection process must have received something. Problem is, it didn't close the socket properly, and the ssl_connection process still runs when it shouldn't. Is it a known issue? Is there a patch available? If not, how should I go about fixing it? Thanks. -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From simon@REDACTED Mon Oct 15 17:42:49 2012 From: simon@REDACTED (Simon MacMullen) Date: Mon, 15 Oct 2012 16:42:49 +0100 Subject: [erlang-questions] Subscribing to {inconsistent_database, starting_partitioned_network, Node} Message-ID: <507C2EF9.4080103@rabbitmq.com> So I need to invoke mnesia:subscribe(system) after starting Mnesia: 1> mnesia:subscribe(system). {error,{node_not_running,nonode@REDACTED}} 2> application:start(mnesia). ok 3> mnesia:subscribe(system). {ok,nonode@REDACTED} ...but as far as I can see, starting Mnesia can cause {inconsistent_database, starting_partitioned_network, Node} to be emitted. So what am I supposed to do if I am interested in subscribing to this event? Cheers, Simon -- Simon MacMullen RabbitMQ, VMware From pan@REDACTED Mon Oct 15 17:42:49 2012 From: pan@REDACTED (Patrik Nyblom) Date: Mon, 15 Oct 2012 17:42:49 +0200 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: References: , , <507BDC1A.4080309@erix.ericsson.se>, , , <507C069A.9090301@erix.ericsson.se>, Message-ID: <507C2EF9.6010407@erlang.org> Hi! On 10/15/2012 05:09 PM, Matthew Evans wrote: > I'll rephrase that. This is my concern from the ets documentation for > safe_fixtable/2: > > Note that no deleted objects are actually removed from a fixed table > until it has been released. If a process fixes a table but never > releases it, the memory used by the deleted objects will never be > freed. The performance of operations on the table will also degrade > significantly. > > Is there a potential problem on having a safe_fixtable "active" for > long periods of time on a table having lots of inserts and deletes? It all depends... Memory-wise the deletes will actually take place after the fixation is released. Fixtable also means longer link chains in the buckets, so if you for example will replace all the objects in the table once during the timeslot, lookup will need to traverse twice as long lists for every element. If you do heavy insertion (without deletion), you will also get much longer chains, as the number of buckets will not increase while the table is 'fixed'. So it depends on how much 'lots' is in your case... Have you considered using ordered_set instead? Then you wouldn't need fixation and would have no problem traversing the "moving" table. Performance on ordered set is not as bad as you would think and you can do nice optimizations when selecting by using partially bound keys. /Patrik > > Thanks > > Matt > ________________________________ > > From: mattevans23@@hotmail.com > > To: sverker.eriksson@REDACTED > > Date: Mon, 5 Oct 012 0::4::1 -400 > > CC: erlang-questions@REDACTED > > Subject: Re: [erlang-questions] ets:select badarg question (R5BB) > > > > Many thanks, > > > > I'll give that a go. I am however a little worried about having a > > safe_fixtable "active" on a table for 00 seconds so might need to be a > > little smarter. > > > > Cheers > > > > Matt > > > > > Date: Mon, 5 Oct 012 4::0::4 +200 > > > From: sverker.eriksson@REDACTED > > > To: mattevans23@@hotmail.com > > > CC: erlang-questions@REDACTED > > > Subject: Re: [erlang-questions] ets:select badarg question (R5BB) > > > > > > You need to use ets:safe_fixtable/ to get a "sensible" result from > > > select/match with continuation on unordered tables. Otherwise you may > > > miss records, get double hits and (as in your case) get badarg if the > > > table has shrunken due to deleted record causing the continuation to > > > point out of table bounds. One could argue that '$end_of_table' > would be > > > a more suitable result than badarg for this case. > > > > > > I see that the documentation is not clear about this. It describes > usage > > > of safe_fixtable together with first/ and next/ but it does not state > > > that it also is needed for select/match with continuation. > > > Single call select/match does not need safe_fixtable. > > > > > > /Sverker, Erlang/OTP > > > > > > Matthew Evans wrote: > > > > Hi > > > > These are the options > > > > > > > > > > > > ets:new(StationIdTableName,[{keypos,}},named_table,public]), > > > > The only odd things I can think of is that during the select > > records are getting deleted and added. Also, the select operation is > > run slowly (over 00 seconds) > > > > Again, this isn't every time I get the crash. > > > > Matt > > > > > > > > > > > >> Date: Mon, 5 Oct 012 1::9::4 +200 > > > >> From: sverker.eriksson@REDACTED > > > >> To: mattevans23@@hotmail.com > > > >> CC: erlang-questions@REDACTED > > > >> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) > > > >> > > > >> What options are you using when the table is created? > > > >> > > > >> /Sverker, Erlang/OTP > > > >> > > > >> Matthew Evans wrote: > > > >> > > > >>> Hi, > > > >>> I'm running a select with a continuation on a public ets table > > with approximately 0,,00 records. > > > >>> The original call to the select is: > > ets:select(TableName,[{'$'',[],['$_']}],RecordsToGet), then an > > ets:select/ on the continuation. > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> The process that is spawned to walk the table is sitting in a > > recursive loop doing a select (grabbing about 50 records each time), > > processing the matching records (which includes possibly deleting some > > of these records), then sleeping for a few seconds. It takes about 00 > > seconds to walk through the table. As well as records getting deleted, > > new records are added to the table during this time. > > > >>> Every now and then I get an exception in the process doing the > > select/:: > > > >>> Error in process <..6218..>> on node 'xxxxx@REDACTED'' with > > exit value: {badarg,[{ets,select,[{'table_E::9::D::0::1::0'',01,,,,<< > > bytes>>,[{p_MacEntry....... (truncated here) > > > >>> This is the code (all running in the same process that started > > the select) obviously the last ets:select is causing the crash: > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> start_aging((AgeInterval,MaxAge,SelectData) -> > > > >>> start_aging((AgeInterval,MaxAge,SelectData,)). > > > >>> start_aging((_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> > > > >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); > > > >>> > start_aging((_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> > > > >>> RecordsAgedCount = > > age_records(Matches,[],MaxAge*000000,,os:timestamp()), > > > >>> > > > gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); > > > > >>> > start_aging((AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> > > > >>> RecordsAgedCount = > > age_records(Matches,[],MaxAge*000000,,os:timestamp()), > > > >>> timer:sleep(AgeInterval), > > > >>> > > > start_aging((AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). > > > > >>> > > > >>> I'm not sure what could be the cause of the select getting a > bad arg. > > > >>> Any ideas? > > > >>> Thanks > > > >>> Matt > > > >>> > > > >>> > > > ------------------------------------------------------------------------ > > > >>> > > > >>> _______________________________________________ > > > >>> erlang-questions mailing list > > > >>> erlang-questions@REDACTED > > > >>> http://erlang.org/mailman/listinfo/erlang-questions > > > >>> > > > >>> > > > > > > > > > > > > > > > _______________________________________________ erlang-questions > > mailing list erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > > > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mononcqc@REDACTED Mon Oct 15 17:46:46 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 15 Oct 2012 11:46:46 -0400 Subject: [erlang-questions] Crypto and ensure_loaded (maybe NIFs?) Message-ID: <507C2FE6.4090704@ferd.ca> While running some benchmarks and simulations on code that tends to use crypto's sha functions quite a lot (in R15B02), I've noticed that code that used them a lot tended to have bottlenecks within code_server. After a bit of tracing, I've found that code_server was receiving a bunch of messages of the form: (<0.24.0>) in {code_server,loop,1} (Timestamp: {1350,315353,243397}) (<0.24.0>) <0.227.0> ! {code_server,{module,crypto}} (Timestamp: {1350,315353, 243432}) (<0.24.0>) out {code_server,loop,1} (Timestamp: {1350,315353,243438}) (<0.24.0>) << {code_call,<0.227.0>,{ensure_loaded,crypto}} (Timestamp: {1350, 315353, 243451}) Which happened thousands of times per second. The crypto application was indeed started, and a bunch of fresh processes would do many sha calls per second. I'm wondering what's the underlying need for the code to do so many calls to the code server to make sure it's loaded -- is it related to NIFs in any way? If there's a way to work around that, it would help me greatly. Regards, Fred. From sverker.eriksson@REDACTED Mon Oct 15 17:58:36 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 15 Oct 2012 17:58:36 +0200 Subject: [erlang-questions] ets:select badarg question (R15B) In-Reply-To: References: , , <507BDC1A.4080309@erix.ericsson.se>, , , <507C069A.9090301@erix.ericsson.se>, Message-ID: <507C32AC.8010502@erix.ericsson.se> Deleting a lot of objects in a fixed table may cause memory problems as the deleted object are not deallocated. They are instead marked as deleted and will be kept around until the table is unfixed. Increasing the number of objects in a fixed table may cause performance problems. The hash table will not grow, leading to long linear search through chains of objects hashing to the same bucket. /Sverker Matthew Evans wrote: > I'll rephrase that. This is my concern from the ets documentation for safe_fixtable/2: > Note that no deleted objects are actually removed from a fixed table until it has been released. If a process fixes a table but never releases it, the memory used by the deleted objects will never be freed. The performance of operations on the table will also degrade significantly. > > Is there a potential problem on having a safe_fixtable "active" for long periods of time on a table having lots of inserts and deletes? > Thanks > Matt________________________________ > >> From: mattevans23@@hotmail.com >> To: sverker.eriksson@REDACTED >> Date: Mon, 5 Oct 012 0::4::1 -400 >> CC: erlang-questions@REDACTED >> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) >> >> Many thanks, >> >> I'll give that a go. I am however a little worried about having a >> safe_fixtable "active" on a table for 00 seconds so might need to be a >> little smarter. >> >> Cheers >> >> Matt >> >> >>> Date: Mon, 5 Oct 012 4::0::4 +200 >>> From: sverker.eriksson@REDACTED >>> To: mattevans23@@hotmail.com >>> CC: erlang-questions@REDACTED >>> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) >>> >>> You need to use ets:safe_fixtable/ to get a "sensible" result from >>> select/match with continuation on unordered tables. Otherwise you may >>> miss records, get double hits and (as in your case) get badarg if the >>> table has shrunken due to deleted record causing the continuation to >>> point out of table bounds. One could argue that '$end_of_table' would be >>> a more suitable result than badarg for this case. >>> >>> I see that the documentation is not clear about this. It describes usage >>> of safe_fixtable together with first/ and next/ but it does not state >>> that it also is needed for select/match with continuation. >>> Single call select/match does not need safe_fixtable. >>> >>> /Sverker, Erlang/OTP >>> >>> Matthew Evans wrote: >>> >>>> Hi >>>> These are the options >>>> >>>> >>>> ets:new(StationIdTableName,[{keypos,}},named_table,public]), >>>> The only odd things I can think of is that during the select >>>> >> records are getting deleted and added. Also, the select operation is >> run slowly (over 00 seconds) >> >>>> Again, this isn't every time I get the crash. >>>> Matt >>>> >>>> >>>> >>>>> Date: Mon, 5 Oct 012 1::9::4 +200 >>>>> From: sverker.eriksson@REDACTED >>>>> To: mattevans23@@hotmail.com >>>>> CC: erlang-questions@REDACTED >>>>> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) >>>>> >>>>> What options are you using when the table is created? >>>>> >>>>> /Sverker, Erlang/OTP >>>>> >>>>> Matthew Evans wrote: >>>>> >>>>> >>>>>> Hi, >>>>>> I'm running a select with a continuation on a public ets table >>>>>> >> with approximately 0,,00 records. >> >>>>>> The original call to the select is: >>>>>> >> ets:select(TableName,[{'$'',[],['$_']}],RecordsToGet), then an >> ets:select/ on the continuation. >> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> The process that is spawned to walk the table is sitting in a >>>>>> >> recursive loop doing a select (grabbing about 50 records each time), >> processing the matching records (which includes possibly deleting some >> of these records), then sleeping for a few seconds. It takes about 00 >> seconds to walk through the table. As well as records getting deleted, >> new records are added to the table during this time. >> >>>>>> Every now and then I get an exception in the process doing the >>>>>> >> select/:: >> >>>>>> Error in process <..6218..>> on node 'xxxxx@REDACTED'' with >>>>>> >> exit value: {badarg,[{ets,select,[{'table_E::9::D::0::1::0'',01,,,,<< >> bytes>>,[{p_MacEntry....... (truncated here) >> >>>>>> This is the code (all running in the same process that started >>>>>> >> the select) obviously the last ets:select is causing the crash: >> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> start_aging((AgeInterval,MaxAge,SelectData) -> >>>>>> start_aging((AgeInterval,MaxAge,SelectData,)). >>>>>> start_aging((_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> >>>>>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); >>>>>> start_aging((_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> >>>>>> RecordsAgedCount = >>>>>> >> age_records(Matches,[],MaxAge*000000,,os:timestamp()), >> >> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); >> >>>>>> start_aging((AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> >>>>>> RecordsAgedCount = >>>>>> >> age_records(Matches,[],MaxAge*000000,,os:timestamp()), >> >>>>>> timer:sleep(AgeInterval), >>>>>> >>>>>> >> start_aging((AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). >> >>>>>> I'm not sure what could be the cause of the select getting a bad arg. >>>>>> Any ideas? >>>>>> Thanks >>>>>> Matt >>>>>> >>>>>> >>>>>> >> ------------------------------------------------------------------------ >> >>>>>> _______________________________________________ >>>>>> erlang-questions mailing list >>>>>> erlang-questions@REDACTED >>>>>> http://erlang.org/mailman/listinfo/erlang-questions >>>>>> >>>>>> >>>>>> >>>> >> _______________________________________________ erlang-questions >> mailing list erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> > > From attila.r.nohl@REDACTED Mon Oct 15 18:09:48 2012 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Mon, 15 Oct 2012 18:09:48 +0200 Subject: [erlang-questions] Process/FD leak in SSL R15B01 In-Reply-To: <507C2922.9090207@ninenines.eu> References: <507C2922.9090207@ninenines.eu> Message-ID: 2012/10/15 Lo?c Hoguin : [...] >> lists:foldl(fun(X, Sum) -> case erlang:process_info(X) of undefined -> >> Sum; [{current_function, XXX}|_] -> case lists:keyfind(XXX, 1, Sum) of false >> -> Curr = 0; {_, Curr} -> ok end, lists:keystore(XXX, 1, Sum, {XXX, Curr + >> 1}) end end, [], List). > [{{prim_inet,recv0,3},25856},{{gen_fsm,loop,7},26574}] > > Not sure which one is the ESTABLISHED list and which one is the FIN_WAIT2. > Of course, I can't use sys:get_status/1 on the PIDs stuck in > prim_inet:recv0/3 because the receive there is quite specific. So I can't > get the stacktrace. The other case doesn't seem to give anything useful (for > my level of knowledge, anyway). You can get the stacktrace with erlang:process_info(Pid, backtrace). From essen@REDACTED Mon Oct 15 18:19:41 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Mon, 15 Oct 2012 18:19:41 +0200 Subject: [erlang-questions] Process/FD leak in SSL R15B01 In-Reply-To: References: <507C2922.9090207@ninenines.eu> Message-ID: <507C379D.7010004@ninenines.eu> On 10/15/2012 06:09 PM, Attila Rajmund Nohl wrote: > 2012/10/15 Lo?c Hoguin : > [...] >>> lists:foldl(fun(X, Sum) -> case erlang:process_info(X) of undefined -> >>> Sum; [{current_function, XXX}|_] -> case lists:keyfind(XXX, 1, Sum) of false >>> -> Curr = 0; {_, Curr} -> ok end, lists:keystore(XXX, 1, Sum, {XXX, Curr + >>> 1}) end end, [], List). >> [{{prim_inet,recv0,3},25856},{{gen_fsm,loop,7},26574}] >> >> Not sure which one is the ESTABLISHED list and which one is the FIN_WAIT2. >> Of course, I can't use sys:get_status/1 on the PIDs stuck in >> prim_inet:recv0/3 because the receive there is quite specific. So I can't >> get the stacktrace. The other case doesn't seem to give anything useful (for >> my level of knowledge, anyway). > > You can get the stacktrace with erlang:process_info(Pid, backtrace). Thanks for the tip! So yeah, this one is stuck while trying to terminate. Program counter: 0x00007f05fd6a5608 (prim_inet:recv0/3 + 224) CP: 0x0000000000000000 (invalid) arity = 0 0x00007f052e1b1eb0 Return addr 0x00007f05a3248a98 (ssl_connection:terminate/3 + 800) y(0) 57928 y(1) #Port<0.51824890> 0x00007f052e1b1ec8 Return addr 0x00007f05a3b29670 (gen_fsm:terminate/7 + 168) y(0) [] y(1) [] y(2) [] y(3) [] y(4) #Port<0.51824890> y(5) gen_tcp 0x00007f052e1b1f00 Return addr 0x00007f05a3bb41d0 (proc_lib:init_p_do_apply/3 + 56) y(0) [] y(1) {state,server,{#Ref<0.0.8553.184512>,<0.18913.1670>},gen_tcp,tcp,tcp_closed,tcp_error,"localhost",8443,#Port<0.51824890>,{ssl_options,[],verify_none,{#Fun,[]},false,false,undefined,1,"/home/obfuscated/obfuscatedrlang/lib/obfuscatedunchat-1.6.0/priv/ssl/cert.pem",undefined,"/home/obfuscated/obfuscatedrlang/lib/obfuscatedunchat-1.6.0/priv/ssl/cert.pem",undefined,undefined,undefined,"/home/obfuscated/obfuscatedrlang/lib/obfuscatedunchat-1.6.0/priv/ssl/cacert.pem",undefined,undefined,[<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>,<<2 bytes>>],#Fun,true,268435456,false,[],undefined,false},{socket_options,binary,0,0,0,once},{connection_states,{connection_state,{security_parameters,<<2 bytes>>,0,7,1,16,128,16,unknown,2,20,0,<<48 bytes>>,<<32 bytes>>,<<32 bytes>>,undefined},undefined,{cipher_state,<<16 bytes>>,<<16 bytes>>,undefined},<<20 bytes>>,6,true,<<12 bytes>>,<<12 bytes>>},{connection_state,{security_parameters,undefined,0,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,<<32 bytes>>,undefined},undefined,undefined,undefined,undefined,true,undefined,undefined},{connection_state,{security_parameters,<<2 bytes>>,0,7,1,16,128,16,unknown,2,20,0,<<48 bytes>>,<<32 bytes>>,<<32 bytes>>,undefined},undefined,{cipher_state,<<16 bytes>>,<<16 bytes>>,undefined},<<20 bytes>>,13,true,<<12 bytes>>,<<12 bytes>>},{connection_state,{security_parameters,undefined,0,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,<<32 bytes>>,undefined},undefined,undefined,undefined,undefined,true,undefined,undefined}},[],<<0 bytes>>,<<0 bytes>>,{<<0 bytes>>,<<0 bytes>>},[],12308,{session,<<32 bytes>>,undefined,<<1257 bytes>>,0,<<2 bytes>>,<<48 bytes>>,true,63517514196},24599,ssl_session_cache,{3,1},undefined,false,rsa,undefined,{'RSAPrivateKey','two-prime',26952275589898844250103204854000460899755240864557148991279029405309749386179997816352098133722767607711339559172144166023544257819579600281768301281374192936110073288270279982220728800798557215504319561694659173220853716332492625335748497090542115981135939145850167689577529567336032326581411005966667046488818079418004669621093520594249388264789813277716494693460309931110183497107534349074298533842111855672958994036657571757894555006279600552417098362361837531438833518633912632305124934722467790401548511827982945839067677876435394001531838872958423949934302335970305331259903589271491819721745867851063767101711,65537,72894561061836369429952440001397404435381791098810666626550009339627280447692213756327103684977349565388024121690992163627770872919426951949943259565185707278720577541631553578110444175680062658469881781442213382568569223796242089823480188432467004251893513795 29032795180763714863835283712338222389782464852044908949559099608779063302276133782734662944988246892889439189879440128304200546026414115176004328650285319262737051106741309180479178565669060188052206153201268137707738579437817066853295089724557953207910831295502502266942720391639060038564028714207644340973116764361586980768047522941109290140269958017673,175687943987452481712482925881452602286249755596049611520304319723586191396077353211979651292293422727226888628810410185730059847194115171473543563960899974179642474882228165413574970142069071147678024461620581964093179873537217082000749319715442644574561869121959620883833054096268955094522609955548334417459,153409932282113553454184543331491108535630917003675738115171976005561046830080906444372373873384261619709596605477733172587334100633359814892598660495299494503111498422936185897687222105679379604880102069501875829087474529350923531274442007552106843129760936560325773227109973336264866827986047857553670280629,1653778011280640103960 32533996867303777118782862747708688208092956158440780252498542811490728487320772469810738971008968489145399289747151126453392805772011338882217952599871103786463875892748648418527046734444999723398379211505851743415571090610985953725319559986129544296274474768880307859595812559810466567,99372239273086723091338362047522171285756194037567212940251777246259022537354389739788150444373539745180764989177727522509078951433348961072685633082784597107680994416138776015512122203187528006871997391618377904030112283443023112892909533616156365176077807633229316646127723088806569214057154029767435353361,7323400945433254897172443884831966307350890604864415567101495881249115607826271555845873590083141172347084964771961315371798965570031721570889013199060717814694355367862624287469661994309269724835502396215672690851146331180228281917872372949595188776597872289445053324786304898777105863585382475713592590505,asn1_NOVALUE},{'DHParameter',17976931348623159077083915679378745319786029604875601170644442368 4197180216158519368947833795864925541502180565485980503646440548199239100050792877003355816639229553136239076508735759914822574862575007425302077447712589550957937778424442426617334727629299387668709205606050270810842907692932019128194467627007,2,asn1_NOVALUE},undefined,undefined,#Ref<0.0.0.13264>,{<0.1092.0>,#Ref<0.0.8553.179746>},0,<<0 bytes>>,true,undefined,undefined,{[],[]},false,true} y(2) connection y(3) ssl_connection y(4) {'DOWN',#Ref<0.0.8553.184512>,process,<0.18913.1670>,normal} y(5) <0.18199.1670> y(6) normal y(7) Catch 0x00007f05a3b29670 (gen_fsm:terminate/7 + 168) 0x00007f052e1b1f48 Return addr 0x0000000000883498 () y(0) Catch 0x00007f05a3bb41f0 (proc_lib:init_p_do_apply/3 + 88) -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From rustompmody@REDACTED Mon Oct 15 19:11:56 2012 From: rustompmody@REDACTED (Rustom Mody) Date: Mon, 15 Oct 2012 22:41:56 +0530 Subject: [erlang-questions] suitability of erlang In-Reply-To: <36CF84B4-AAF0-4916-95ED-C1289B90B6BF@erlang-solutions.com> References: <36CF84B4-AAF0-4916-95ED-C1289B90B6BF@erlang-solutions.com> Message-ID: On Mon, Oct 15, 2012 at 2:15 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > > Two, I've found that a very large factor of performance stems from the > ability to try out clever solutions and be clever about what you do. And in > my experience, this is easier to achieve in sane, modern languages like ML, > Erlang or (to a certain extent) Haskell. Thus, if you can trade a slow > solution for a faster one quickly, then you end up with a faster program in > the end. Another factor is time constraint, where a higher level language > lets you express the problem quicker. > > Thanks Jesper for your comments. I was brought up on lisp and have spent a couple of decades on Haskell and such languages. So the functional aspect of Erlang is not (likely to be) an issue for me. Concurrency however is new! So any directions of exercises that I could try to get into 'the Erlang groove' would be helpful. [I am of course reading Joe's book] Rusi -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush@REDACTED Mon Oct 15 19:31:38 2012 From: straightflush@REDACTED (AD) Date: Mon, 15 Oct 2012 13:31:38 -0400 Subject: [erlang-questions] using flush() for batches from mailbox Message-ID: Hello, I have a use case where I would like to flush() the Mailbox in a loop. I have a process that takes about 500ms to execute, and i would like the process to operate on batches of messages from the Mailbox rather than 1 at a time. Since the sender can push messages to the mailbox at tens of messages/sec and I would like to preserve order, the most efficient way right now I think of is to take messages off the mailbox in batches. If there is a process sending to a gen_server:cast at , say, 10 messages/sec , can i use flush to consume all 10 messages from teh mailbox at once? Something like flush() -> receive _ -> flush() after 0 -> ok end. TThanks -AD -------------- next part -------------- An HTML attachment was scrubbed... URL: From sverker.eriksson@REDACTED Mon Oct 15 20:56:20 2012 From: sverker.eriksson@REDACTED (Sverker Eriksson) Date: Mon, 15 Oct 2012 20:56:20 +0200 Subject: [erlang-questions] Crypto and ensure_loaded (maybe NIFs?) In-Reply-To: <507C2FE6.4090704@ferd.ca> References: <507C2FE6.4090704@ferd.ca> Message-ID: <507C5C54.6090003@erix.ericsson.se> That doesn't sound right. Could you supply some code to reproduce that? /Sverker, Erlang/OTP Fred Hebert wrote: > While running some benchmarks and simulations on code that tends to > use crypto's sha functions quite a lot (in R15B02), I've noticed that > code that used them a lot tended to have bottlenecks within code_server. > > After a bit of tracing, I've found that code_server was receiving a > bunch of messages of the form: > > (<0.24.0>) in {code_server,loop,1} (Timestamp: {1350,315353,243397}) > (<0.24.0>) <0.227.0> ! {code_server,{module,crypto}} (Timestamp: > {1350,315353, > > 243432}) > > (<0.24.0>) out {code_server,loop,1} (Timestamp: {1350,315353,243438}) > (<0.24.0>) << {code_call,<0.227.0>,{ensure_loaded,crypto}} (Timestamp: > {1350, > > 315353, > > 243451}) > > Which happened thousands of times per second. The crypto application > was indeed started, and a bunch of fresh processes would do many sha > calls per second. > > I'm wondering what's the underlying need for the code to do so many > calls to the code server to make sure it's loaded -- is it related to > NIFs in any way? If there's a way to work around that, it would help > me greatly. > > Regards, > Fred. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From anthonym@REDACTED Mon Oct 15 21:15:37 2012 From: anthonym@REDACTED (Anthony Molinaro) Date: Mon, 15 Oct 2012 12:15:37 -0700 Subject: [erlang-questions] Getting attributes In-Reply-To: <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> References: <48BF8168-D34E-4261-B2D4-AB44E20218EF@gmail.com> <6C9A98EC-4D83-40C8-8EC0-3AFB07F9E0AA@gmail.com> Message-ID: <20121015191537.GA51847@alumni.caltech.edu> On Sat, Oct 13, 2012 at 06:38:32PM -0500, Steve Davis wrote: > I understated the issue: > > The position of the entity can also change: > {id, {x,y}, attribute} > > So, in fact, the id is the only constant and the pos and attribute are independently mutable. > > > > On Oct 13, 2012, at 6:23 PM, Steve Davis wrote: > > > Hi all, > > > > Suppose I have a 2 dimensional plane {x, y} of 64k by 64k coordinates. > > > > Each coordinate has a mutable attribute {{x, y}, attribute} > > > > If I need to query: What are the attributes of positions "next to" my position, i.e: x +- 1, y +- 1? > > > > What is the correct solution to this without storing everything in "Oracle" and using relational queries? > > > > Can this be sensibly maintained in a KV store? This problem reminds me of one I had to do to map latitude and longitude to location, then look up the closest location based on the latitude and longitude of an input. I used an ordered_set ets table inserting records like {Latitude, Longitude, Record} Then when looking things up given an input latitude and longitude, I would add/subtract a small number of degrees to the input to get a range of latitudes and longitudes. Then query ets first using ets:prev/2 with the input latitude and walking down by repeatedly calling ets:prev/2 until I either hit the start of the table or the lower bound on the latitude. You then do the same with ets:next/2 to the upper bound and finally ets:lookup/2 in case its an exact match. I would pass the upper and lower bound of the longitude along so I could filter out things out of range as I was walking through the latitudes. I've got about 60000 entries in the ets table and queries take about 0.085 milliseconds on average. I found that using ets:prev/2 and ets:next/2 was much faster than say ets:match with a matchspec to do the range checking. I've also got another ordered_set table with 11 million entries which I don't do range checks but just an ets:prev/2 and an ets:lookup/2 and that runs even faster (average about 0.010 millis per pair of calls). Unfortunately the code is not shareable, but its relatively straightforward so shouldn't be hard to recreate. Good Luck, -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro From gleber.p@REDACTED Mon Oct 15 21:36:55 2012 From: gleber.p@REDACTED (Gleb Peregud) Date: Mon, 15 Oct 2012 21:36:55 +0200 Subject: [erlang-questions] using flush() for batches from mailbox In-Reply-To: References: Message-ID: On Mon, Oct 15, 2012 at 7:31 PM, AD wrote: > Hello, > > I have a use case where I would like to flush() the Mailbox in a loop. I > have a process that takes about 500ms to execute, and i would like the > process to operate on batches of messages from the Mailbox rather than 1 at > a time. Since the sender can push messages to the mailbox at tens of > messages/sec and I would like to preserve order, the most efficient way > right now I think of is to take messages off the mailbox in batches. > > If there is a process sending to a gen_server:cast at , say, 10 messages/sec > , can i use flush to consume all 10 messages from teh mailbox at once? > Something like > > flush() -> > receive > _ -> flush() > after > 0 -> ok > end. Of course you can. The code is correct as well. There are two problems here: 1) It will consume gen_server:calls, hence callers will timeout 2) It will consume sys messages, hence e.g. sys:get_status/1 won't work Cheers, Gleb Peregud From tuncer.ayaz@REDACTED Mon Oct 15 22:19:35 2012 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Mon, 15 Oct 2012 22:19:35 +0200 Subject: [erlang-questions] application starting in releases In-Reply-To: References: <20121014214526.GA22270@circlewave.net> Message-ID: On Mon, Oct 15, 2012 at 12:57 AM, Roberto Ostinelli wrote: > This ends up doing "application:start([compiler])" which returns an error, > > > > init(3) probably ignores return value as it couldn't really know > > what to expect. Try with -eval instead: > > > > erl -boot start_sasl -eval 'application:start(compiler).' > > > Oh I see. I wanted to avoid `eval` for some reason, but this works > perfectly. Thank you. > > > > BTW I vaguely recall rebar can do this for you somehow? > > > No idea. Tuncer? :) This is most likely referring to the release support and scripts/templates. > > > Am I actually supposed to use application:start/1 in myapp > > > anyways? > > > > No, it's quite rare to do that, the only use case I've seen is > > bringing protocol stacks up and down manually depending on > > availability of some backend connections (well, lazy man's way to > > do that anyway). > > > Ok this is exactly what I thought. > > Again, thank you, From roberto@REDACTED Mon Oct 15 22:32:56 2012 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 15 Oct 2012 13:32:56 -0700 Subject: [erlang-questions] starting an app with an app.config file In-Reply-To: References: Message-ID: Oh I understand now. Thank you Andrew, will try that. r. On Mon, Oct 15, 2012 at 1:52 AM, Andrew Gopienko wrote: > Just read your config file and set env with application:load before > application:start -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.lapshin@REDACTED Mon Oct 15 22:50:56 2012 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 15 Oct 2012 13:50:56 -0700 (PDT) Subject: [erlang-questions] Gitty: raw git acces in pure erlang In-Reply-To: References: Message-ID: <937f59f9-059e-47a1-99be-90715095ad72@b12g2000vbg.googlegroups.com> Now gitty can commit back to repository. It means, that you can use it with your favorite HTTP framework to store text pages in git repo and serve it from there. For example, I'm using it for http://erlyvideo.org/ website, because it is very convenient for me, when I can checkout whole website, edit it offline and upload back. Also it is possible to edit content online and it will be still compatible with git. From straightflush@REDACTED Tue Oct 16 00:04:44 2012 From: straightflush@REDACTED (AD) Date: Mon, 15 Oct 2012 18:04:44 -0400 Subject: [erlang-questions] using flush() for batches from mailbox In-Reply-To: References: Message-ID: Why would callers timeout (if using cast), is b/c the flush would prevent the gen_server from sending a reply ? On Mon, Oct 15, 2012 at 3:36 PM, Gleb Peregud wrote: > On Mon, Oct 15, 2012 at 7:31 PM, AD wrote: > > Hello, > > > > I have a use case where I would like to flush() the Mailbox in a loop. > I > > have a process that takes about 500ms to execute, and i would like the > > process to operate on batches of messages from the Mailbox rather than 1 > at > > a time. Since the sender can push messages to the mailbox at tens of > > messages/sec and I would like to preserve order, the most efficient way > > right now I think of is to take messages off the mailbox in batches. > > > > If there is a process sending to a gen_server:cast at , say, 10 > messages/sec > > , can i use flush to consume all 10 messages from teh mailbox at once? > > Something like > > > > flush() -> > > receive > > _ -> flush() > > after > > 0 -> ok > > end. > > Of course you can. The code is correct as well. There are two problems > here: > 1) It will consume gen_server:calls, hence callers will timeout > 2) It will consume sys messages, hence e.g. sys:get_status/1 won't work > > Cheers, > Gleb Peregud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush@REDACTED Tue Oct 16 04:43:24 2012 From: straightflush@REDACTED (AD) Date: Mon, 15 Oct 2012 22:43:24 -0400 Subject: [erlang-questions] using flush() for batches from mailbox In-Reply-To: References: Message-ID: But thats a different mailbox no? not sure how another_server is going to be able to access the mailbox of the calling server unless the messages get persisted somehow On Mon, Oct 15, 2012 at 7:54 PM, Yogish Baliga wrote: > You can do the batching of messages in handle_cast. When number of > messages in a batch exceed some number, do the processing in another > process or gen_server? > > handle_cast(Msg, State) when length(State) > 10 -> > gen_server:call(another_server, State), > {noreply, []); > handle_cast(Msg, State) -> > {noreply, [Msg|State]}. > > > > On Mon, Oct 15, 2012 at 3:04 PM, AD wrote: > >> Why would callers timeout (if using cast), is b/c the flush would prevent >> the gen_server from sending a reply ? >> >> >> On Mon, Oct 15, 2012 at 3:36 PM, Gleb Peregud wrote: >> >>> On Mon, Oct 15, 2012 at 7:31 PM, AD wrote: >>> > Hello, >>> > >>> > I have a use case where I would like to flush() the Mailbox in a >>> loop. I >>> > have a process that takes about 500ms to execute, and i would like the >>> > process to operate on batches of messages from the Mailbox rather than >>> 1 at >>> > a time. Since the sender can push messages to the mailbox at tens of >>> > messages/sec and I would like to preserve order, the most efficient way >>> > right now I think of is to take messages off the mailbox in batches. >>> > >>> > If there is a process sending to a gen_server:cast at , say, 10 >>> messages/sec >>> > , can i use flush to consume all 10 messages from teh mailbox at once? >>> > Something like >>> > >>> > flush() -> >>> > receive >>> > _ -> flush() >>> > after >>> > 0 -> ok >>> > end. >>> >>> Of course you can. The code is correct as well. There are two problems >>> here: >>> 1) It will consume gen_server:calls, hence callers will timeout >>> 2) It will consume sys messages, hence e.g. sys:get_status/1 won't work >>> >>> Cheers, >>> Gleb Peregud >>> >> >> >> _______________________________________________ >> erlang-questions mailing list >> erlang-questions@REDACTED >> http://erlang.org/mailman/listinfo/erlang-questions >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Tue Oct 16 06:20:51 2012 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Mon, 15 Oct 2012 23:20:51 -0500 Subject: [erlang-questions] +swt very_low doesn't seem to avoid schedulers getting In-Reply-To: Message of "Wed, 10 Oct 2012 21:12:23 +0200." <5075C897.5070002@erlang.org> Message-ID: <83203.1350361251@snookles.snookles.com> Rickard Green wrote: rg> This is very much expected. Since you have work that can load 2 rg> schedulers full time and shuts down all but one, the run-queue will rg> grow. When you later release all schedulers, there will be lots of rg> work to pick from. Hi, Rickard. Sorry I didn't reply earlier ... Basho kept me busy with an all-hands meeting and conference out in San Francisco. Perhaps I wasn't all that clear about the problem that I saw and that several other customers have witnessed. 1. One node in a Riak cluster is consuming significantly lower CPU than the other nodes in the cluster. The imbalance is not due to application layer workload imbalance, as far as we can tell. (In the case that I personally witnessed, it was a lab environment with an artificial & deterministic load generator talking to all Riak nodes equally (or trying very hard to)). 2. As soon as we do one of two things, the CPU imbalance disappears: a. Restart the Riak app on the slow node. b. Use the erlang:system_flag(schedulers_online, 1) hack and then back to 8 using the same BIF. In situations described by customers, this seems to happen after a day or more of load, where the peak workload is substantially higher than off-peak workload. In the lab environment that I witnessed, the load generators were cycling through 100%-off and 100%-on states. rg> This compaction of load onto fewer schedulers is there in order to rg> reduce communication overhead when there aren't enough work to fully rg> utilize all schedulers. The performance gain of this compaction rg> depends on the hardware. What you describe seeems to be exactly what's happening ... except that when input workload rises again, the idled schedulers aren't waking up, ever. Or we force them to wake up with the system_flag() BIF. rg> We have gotten reports about problems with this functionality, but rg> we have not found any bugs in this functionality. We have only found rg> that it behaves as expected. That is, if more schedulers aren't rg> woken this is due to not enough accumulated overload. The +swt rg> switch was introduced in order to give the user the possibility do rg> define what is enough overload for his or her taste. Hrm, well, we've seen it both with "+swt very_low" and without any +swt flag at all. And it's extremely irritating. :-) What info would you need gathered from the field when this bugaboo strikes next time? -Scott From dgud@REDACTED Tue Oct 16 09:20:29 2012 From: dgud@REDACTED (Dan Gudmundsson) Date: Tue, 16 Oct 2012 09:20:29 +0200 Subject: [erlang-questions] Subscribing to {inconsistent_database, starting_partitioned_network, Node} In-Reply-To: <507C2EF9.4080103@rabbitmq.com> References: <507C2EF9.4080103@rabbitmq.com> Message-ID: You can replace mnesia_event with your own event handler. /Dan On Mon, Oct 15, 2012 at 5:42 PM, Simon MacMullen wrote: > So I need to invoke mnesia:subscribe(system) after starting Mnesia: > > 1> mnesia:subscribe(system). > {error,{node_not_running,nonode@REDACTED}} > 2> application:start(mnesia). > ok > 3> mnesia:subscribe(system). > {ok,nonode@REDACTED} > > ...but as far as I can see, starting Mnesia can cause > {inconsistent_database, starting_partitioned_network, Node} to be emitted. > > So what am I supposed to do if I am interested in subscribing to this event? > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, VMware > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mail@REDACTED Tue Oct 16 10:31:51 2012 From: mail@REDACTED (Dominic Williams) Date: Tue, 16 Oct 2012 10:31:51 +0200 Subject: [erlang-questions] Coding dojo and Erlounge in Paris tonight Message-ID: Hello, The Paris Erlang user group is holding a coding dojo and an Erlounge tonight. The dojo will be at 7pm in UT7 offices, 10 rue d'Uz?s, 75002 Paris. The Erlounge will be around 9pm chez Papa, 153 rue Montmartre, 75002 Paris. Locals and visitors are welcome to either or both! Best regards, Dominic Williams http://dominicwilliams.net ---- From vances@REDACTED Tue Oct 16 13:08:12 2012 From: vances@REDACTED (Vance Shipley) Date: Tue, 16 Oct 2012 16:38:12 +0530 Subject: [erlang-questions] +swt very_low doesn't seem to avoid schedulers getting In-Reply-To: <5075C897.5070002@erlang.org> References: <5075C897.5070002@erlang.org> Message-ID: <20121016110812.GA234@aluminum.motivity.ca> Rickard, Maybe you can offer some insight into a problem I'm having. I have a linked in driver which uses port level blocking with many ports. The driver uses driver_set_timer() to wake up in so many milliseconds (e.g. 10ms) to do a small amount of work in it's timeout() callback. What we see is that despite a low CPU usage and multiple schedulers often the timeout will be called quite late (e.g. 10-20ms late). I have spent quite a bit of time looking at the implementation of the schedulers but am not sure what's going on. I am beginning to think the shceduler is going off to do aux work. We've tried changing +swt without effect. What do you think? On Wed, Oct 10, 2012 at 09:12:23PM +0200, Rickard Green wrote: } This compaction of load onto fewer schedulers is there in order to } reduce communication overhead when there aren't enough work to } fully utilize all schedulers. The performance gain of this } compaction depends on the hardware. -- -Vance From pan@REDACTED Tue Oct 16 15:44:50 2012 From: pan@REDACTED (Patrik Nyblom) Date: Tue, 16 Oct 2012 15:44:50 +0200 Subject: [erlang-questions] +swt very_low doesn't seem to avoid schedulers getting In-Reply-To: <20121016110812.GA234@aluminum.motivity.ca> References: <5075C897.5070002@erlang.org> <20121016110812.GA234@aluminum.motivity.ca> Message-ID: <507D64D2.9030106@erlang.org> Hi! On 10/16/2012 01:08 PM, Vance Shipley wrote: > Rickard, > > Maybe you can offer some insight into a problem I'm having. I have > a linked in driver which uses port level blocking with many ports. > The driver uses driver_set_timer() to wake up in so many milliseconds > (e.g. 10ms) to do a small amount of work in it's timeout() callback. > What we see is that despite a low CPU usage and multiple schedulers > often the timeout will be called quite late (e.g. 10-20ms late). What does the other callbacks in your driver do? Anything that could take 10-20 ms? In that case it's normal, your port is locked for what would be considered a very long time in a driver. > I have spent quite a bit of time looking at the implementation of the > schedulers but am not sure what's going on. I am beginning to think > the shceduler is going off to do aux work. > > We've tried changing +swt without effect. What do you think? > > On Wed, Oct 10, 2012 at 09:12:23PM +0200, Rickard Green wrote: > } This compaction of load onto fewer schedulers is there in order to > } reduce communication overhead when there aren't enough work to > } fully utilize all schedulers. The performance gain of this > } compaction depends on the hardware. Cheers, /Patrik From pan@REDACTED Tue Oct 16 15:51:01 2012 From: pan@REDACTED (Patrik Nyblom) Date: Tue, 16 Oct 2012 15:51:01 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? Message-ID: <507D6645.3070001@erlang.org> Hi all! The OTP Technical board decisions from last Thursday are now published on the erlang.org website, which means that the answers to some questions about changes in R16 are finally officially answered. Cheers, /Patrik From essen@REDACTED Tue Oct 16 16:06:27 2012 From: essen@REDACTED (=?ISO-8859-1?Q?Lo=EFc_Hoguin?=) Date: Tue, 16 Oct 2012 16:06:27 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: <507D69E3.6030401@ninenines.eu> Awesome news! Happy with everything! On 10/16/2012 03:51 PM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are now published > on the erlang.org website, which means that the answers to some > questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- Lo?c Hoguin Erlang Cowboy Nine Nines http://ninenines.eu From mononcqc@REDACTED Tue Oct 16 16:13:26 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 16 Oct 2012 10:13:26 -0400 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: <507D6B86.1060805@ferd.ca> Happy with all measures. I think the balance you strike for parametrized modules is also pretty good. Quick question -- I remember older posts relative to tuple calls about how their implementation somehow limited future optimizations that could have been possible. Does it mean their support for Pmod utilization means further optimizations will be cancelled? Otherwise, it's good to be stripping stuff from Erlang. The less complex, the better. Congrats on the good decisions (unless someone proves otherwise! ;) ) Regards, Fred. On 12-10-16 9:51 AM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are now published > on the erlang.org website, which means that the answers to some > questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mahesh@REDACTED Tue Oct 16 16:13:56 2012 From: mahesh@REDACTED (Mahesh Paolini-Subramanya) Date: Tue, 16 Oct 2012 10:13:56 -0400 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: <8604CF97-699D-4D47-97DA-C56C9FB17E71@dieswaytoofast.com> pmods and packages - very nicely (and alliteratively) done :-) cheers Mahesh Paolini-Subramanya That Tall Bald Indian Guy... Google+ | Blog | Twitter On Oct 16, 2012, at 9:51 AM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are now published on the erlang.org website, which means that the answers to some questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From tuncer.ayaz@REDACTED Tue Oct 16 16:15:18 2012 From: tuncer.ayaz@REDACTED (Tuncer Ayaz) Date: Tue, 16 Oct 2012 16:15:18 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: On Tue, Oct 16, 2012 at 3:51 PM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are > now published on the erlang.org website, which means that > the answers to some questions about changes in R16 are > finally officially answered. I am not certain how to interpret the following: "The default file encoding will be ISO-Latin-1 in R16, but will be changed to UTF-8 in R17. Source code will need no change in R16, but adding a comment denoting ISO-Latin-1 encoding will ensure that the code can be compiled with the R17 compiler." Will we have to add an encoding comment for unmodified source code in R17 (that is pure 7-bit ASCII)? From raimo+erlang-questions@REDACTED Tue Oct 16 16:25:35 2012 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Tue, 16 Oct 2012 16:25:35 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: References: <507D6645.3070001@erlang.org> Message-ID: <20121016142535.GA22236@erix.ericsson.se> On Tue, Oct 16, 2012 at 04:15:18PM +0200, Tuncer Ayaz wrote: > On Tue, Oct 16, 2012 at 3:51 PM, Patrik Nyblom wrote: > > Hi all! > > > > The OTP Technical board decisions from last Thursday are > > now published on the erlang.org website, which means that > > the answers to some questions about changes in R16 are > > finally officially answered. > > I am not certain how to interpret the following: > "The default file encoding will be ISO-Latin-1 in R16, > but will be changed to UTF-8 in R17. > > Source code will need no change in R16, but adding a > comment denoting ISO-Latin-1 encoding will ensure that > the code can be compiled with the R17 compiler." > > Will we have to add an encoding comment for unmodified > source code in R17 (that is pure 7-bit ASCII)? 7-bit ASCII is valid UTF-8 and therefore will need no encoding comment. If you have Latin-1 (8-bit ASCII) source code with codes > 255 it is safest to add an encoding comment. > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From hobson42@REDACTED Tue Oct 16 16:33:54 2012 From: hobson42@REDACTED (Ian) Date: Tue, 16 Oct 2012 15:33:54 +0100 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <20121016142535.GA22236@erix.ericsson.se> References: <507D6645.3070001@erlang.org> <20121016142535.GA22236@erix.ericsson.se> Message-ID: <507D7052.7080909@gmail.com> On 16/10/2012 15:25, Raimo Niskanen wrote: > If you have Latin-1 (8-bit ASCII) source code with codes > 255 it is safest to > add an encoding comment. I think you meant > 127 :) Getting a byte > 255 would be a fine trick if you could do it. Ian From mononcqc@REDACTED Tue Oct 16 16:36:05 2012 From: mononcqc@REDACTED (Fred Hebert) Date: Tue, 16 Oct 2012 10:36:05 -0400 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: <507D70D5.2040708@ferd.ca> Additional question regarding column numbers and future UTF8 support: will the column number be based on byte length, grapheme clusters (what I would usually use when typing text in a module), include combining characters? I think we've had many discussions regarding how difficult the idea of a length or position in a string can be to figure out -- and I know the unicode consortium makes its own recommendation there. I'm just curious which is going to be followed. On 12-10-16 9:51 AM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are now published > on the erlang.org website, which means that the answers to some > questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions From mattevans123@REDACTED Tue Oct 16 17:43:29 2012 From: mattevans123@REDACTED (Matthew Evans) Date: Tue, 16 Oct 2012 11:43:29 -0400 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: Thank you.... I was going to use a parameterized module in a upcoming project....I guess I won't do that now :) When will we get ihints as to other features going into the R16 release? Frames? New NIF features? JIT etc? Thanks Matt > Date: Tue, 16 Oct 2012 15:51:01 +0200 > From: pan@REDACTED > To: erlang-questions@REDACTED > Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? > > Hi all! > > The OTP Technical board decisions from last Thursday are now published > on the erlang.org website, which means that the answers to some > questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjtruog@REDACTED Tue Oct 16 20:30:51 2012 From: mjtruog@REDACTED (Michael Truog) Date: Tue, 16 Oct 2012 11:30:51 -0700 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: <507DA7DB.7040908@gmail.com> On 10/16/2012 06:51 AM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are now published on the erlang.org website, which means that the answers to some questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > Very cool! These changes should help developers avoid problems during development. Could you please consider making UDP multicast support official, so it is no-longer undocumented/unsupported? From robert.virding@REDACTED Tue Oct 16 23:29:16 2012 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 16 Oct 2012 22:29:16 +0100 (BST) Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> Message-ID: <9fd3f9d0-fe5d-4fbb-912b-c93aca264fc8@knuth> Doesn't this mean that now the syntax for parametrised modules is still there but becomes meaningless? Or rather it will mean whatever the writer of a module chooses it to mean. That really won't encourage clarity. Or what am I missing? Robert ----- Original Message ----- > From: "Patrik Nyblom" > To: erlang-questions@REDACTED > Sent: Tuesday, 16 October, 2012 3:51:01 PM > Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in > R16? > > Hi all! > > The OTP Technical board decisions from last Thursday are now > published > on the erlang.org website, which means that the answers to some > questions about changes in R16 are finally officially answered. > > Cheers, > /Patrik > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From robert.virding@REDACTED Tue Oct 16 23:39:29 2012 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 16 Oct 2012 22:39:29 +0100 (BST) Subject: [erlang-questions] binary, ets and memory... In-Reply-To: Message-ID: <1f2e1b84-7aec-42ad-b37d-3a65977c63ac@knuth> Yes, that is why binary:copy was added. For just the case where you are accessing a sm all section of a large binary and the whoe binary is kept. Another problem is with large binaries which are sent between processes. Yes, they are not copied and only a reference is passed, but it also means that it will take a longer time before the system can detect that the binary is no longer live and can be reclaimed. All the processes through which the binary has passed must be garbage collected first . The binary is a global object. Robert ----- Original Message ----- > From: "Chris Hicks" > To: dmkolesnikov@REDACTED, erlang-questions@REDACTED > Sent: Friday, 12 October, 2012 9:27:03 PM > Subject: Re: [erlang-questions] binary, ets and memory... > I could be wrong but I'm going to take a guess and say that in the > first implementation the whole binary is being kept around and never > destroyed. I think what's happening is you are getting a reference > to part of a larger binary and passing that around, but the larger > binary is sticking around since part of it is still being used. > Copying the part you need, and thus creating an entirely new binary, > is probably allowing all references to that large binary to > disappear so that it can be GC'd. > That's a rather naive guess based on what I know about how binaries > work. Can anyone else back that up or tell me I'm wrong? > Chris > > From: dmkolesnikov@REDACTED > > Date: Fri, 12 Oct 2012 22:15:21 +0300 > > To: erlang-questions@REDACTED > > Subject: [erlang-questions] binary, ets and memory... > > > > Hello, > > > > Recently, my system starts to swap. The investigation has indicated > > that memory consumption was almost twice more then I've expected > > and to be honest, I've confused why it so... > > > > I am talking about Erlang R15B (erts-5.9) [source] [64-bit] > > [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] > > > > So, I do have two processes > > * first process handles tcp/ip socket I/O. received data is pushed > > to second process > > * second process splits binaries binary:split(Buf, [<<$,>>], > > [global]), parses data and makes list of tuple. When list of > > tuples is ready, it folds tuples into ets table > > ets:new(cache, [named_table, ordered_set, public]), > > lists:foldl( > > fun({A, B}, Acc) -> ets:insert(cache, {A, B}) end, > > true, > > List > > ). > > > > I do have about 6M tuples, where first element is SHA1 signature, > > second element is integer. Receiver process pushes 100 tuples per > > time. I hope you got rough idea. > > > > When cache is populated, I do have following memory usage, it looks > > suspicious for me: > > {total,1179235080}, > > {processes,2373638}, > > {processes_used,2373570}, > > {system,1176861442}, > > {atom,264505}, > > {atom_used,253241}, > > {binary,434761000}, <-- this looks strange for me. Why binaries are > > left in heap and in ets? > > {code,6521469}, > > {ets,732409416} > > > > If I change my implementation to > > lists:foldl( > > fun({A, B}, Acc) -> ets:insert(cache, {binary:copy(A), B}) end, > > true, > > List > > ). > > then memory utilization is on the par with my estimates > > {total,701448856}, > > {processes,2405251}, > > {processes_used,2405170}, > > {system,699043605}, > > {atom,264505}, > > {atom_used,253241}, > > {binary,2686280}, > > {code,6521493}, > > {ets,686663080} > > > > Best Regards, > > Dmitry > > > > _______________________________________________ > > erlang-questions mailing list > > erlang-questions@REDACTED > > http://erlang.org/mailman/listinfo/erlang-questions > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlsson.richard@REDACTED Tue Oct 16 23:45:21 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Tue, 16 Oct 2012 23:45:21 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <9fd3f9d0-fe5d-4fbb-912b-c93aca264fc8@knuth> References: <9fd3f9d0-fe5d-4fbb-912b-c93aca264fc8@knuth> Message-ID: <507DD571.7070001@gmail.com> On 2012-10-16 23:29 , Robert Virding wrote: > Doesn't this mean that now the syntax for parametrised modules is > still there but becomes meaningless? Or rather it will mean whatever > the writer of a module chooses it to mean. That really won't > encourage clarity. Or what am I missing? It also seems ass-backwards to me that the syntax will be dropped, but the hack that we used for the proof-of-concept implementation will be immortalized as "tuple modules". Drop the parameterized modules if you like, but don't make the apply-hack a documented feature; the old {M,F} was bad enough. /Richard From robert.virding@REDACTED Wed Oct 17 00:09:06 2012 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 16 Oct 2012 23:09:06 +0100 (BST) Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507DD571.7070001@gmail.com> Message-ID: <0b30c7f1-7ac7-485a-9023-424f5f777212@knuth> I agree. Do it properly or get rid of it. Robert ----- Original Message ----- > From: "Richard Carlsson" > To: erlang-questions@REDACTED > Sent: Tuesday, 16 October, 2012 11:45:21 PM > Subject: Re: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will > happen in R16? > > On 2012-10-16 23:29 , Robert Virding wrote: > > Doesn't this mean that now the syntax for parametrised modules is > > still there but becomes meaningless? Or rather it will mean > > whatever > > the writer of a module chooses it to mean. That really won't > > encourage clarity. Or what am I missing? > > It also seems ass-backwards to me that the syntax will be dropped, > but > the hack that we used for the proof-of-concept implementation will be > immortalized as "tuple modules". Drop the parameterized modules if > you > like, but don't make the apply-hack a documented feature; the old > {M,F} > was bad enough. > > /Richard > _______________________________________________ > erlang-questions mailing list > erlang-questions@REDACTED > http://erlang.org/mailman/listinfo/erlang-questions > From carlsson.richard@REDACTED Wed Oct 17 00:29:40 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 17 Oct 2012 00:29:40 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <0b30c7f1-7ac7-485a-9023-424f5f777212@knuth> References: <0b30c7f1-7ac7-485a-9023-424f5f777212@knuth> Message-ID: <507DDFD4.6090104@gmail.com> On 2012-10-17 00:09 , Robert Virding wrote: > I agree. Do it properly or get rid of it. On second thought, there is one advantage to documenting this representation: there can be no doubt that when you make a call through a tuple {m,...}:f(...), you'll call the latest version of the module. We had originally planned to make pmods an opaque datatype, like funs, but unlike funs we didn't want to be bound to the module version used to create the instance. Keeping the representation open should make it clear what it is you're getting. But then don't remove the language support - make it work throughout the toolchain instead. /Richard From carlsson.richard@REDACTED Wed Oct 17 01:00:00 2012 From: carlsson.richard@REDACTED (Richard Carlsson) Date: Wed, 17 Oct 2012 01:00:00 +0200 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507DDFD4.6090104@gmail.com> References: <0b30c7f1-7ac7-485a-9023-424f5f777212@knuth> <507DDFD4.6090104@gmail.com> Message-ID: <507DE6F0.6020908@gmail.com> On 2012-10-17 00:29 , Richard Carlsson wrote: > On 2012-10-17 00:09 , Robert Virding wrote: >> I agree. Do it properly or get rid of it. > > On second thought, there is one advantage to documenting this > representation: there can be no doubt that when you make a call through > a tuple {m,...}:f(...), you'll call the latest version of the module. We > had originally planned to make pmods an opaque datatype, like funs, but > unlike funs we didn't want to be bound to the module version used to > create the instance. Keeping the representation open should make it > clear what it is you're getting. But then don't remove the language > support - make it work throughout the toolchain instead. For the record, the bad part about making a representation like this public is that it takes implementation details that happened to make sense for the underlying platform at the time (BEAM), and makes it impossible to change them later on. In this case, such implementation details are that the tuple contains exactly the module name followed by the hidden parameters, in order. No additional fields can be added in future versions. Also, the calling convention becomes fixed so that the module tuple itself must always be passed as the last parameter. /Richard From ok@REDACTED Wed Oct 17 01:22:18 2012 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 17 Oct 2012 12:22:18 +1300 Subject: [erlang-questions] Pmods, packages, Unicode source code and column numbers in compiler - what will happen in R16? In-Reply-To: <507D6645.3070001@erlang.org> References: <507D6645.3070001@erlang.org> Message-ID: <573BED79-B744-48FA-9548-1D4BD29F22FF@cs.otago.ac.nz> On 17/10/2012, at 2:51 AM, Patrik Nyblom wrote: > Hi all! > > The OTP Technical board decisions from last Thursday are now published on the erlang.org website, which means that the answers to some questions about changes in R16 are finally officially answered. "UTF-8 BOM's will not be handled due to their limited use." As I understand it, byte order marks are not *supposed* to be used with UTF-8, because it has no byte order issues. "Variable names will continue to be limited to Latin characters." I hope that means "for this release." An end to Java envy (otherwise known as removal of 'packages'): Good news. But let's keep thinking about ways to solve the problem that packages were meant to solve but didn't. From mozhenghua19811109@REDACTED Wed Oct 17 04:02:41 2012 From: mozhenghua19811109@REDACTED (=?UTF-8?B?6I6r?=) Date: Wed, 17 Oct 2012 10:02:41 +0800 (CST) Subject: [erlang-questions] one question about BIF link Message-ID: <66398a26.4918.13a6c755b61.Coremail.mozhenghua19811109@126.com> Hi everybody? ? ? I have a question for the erlang BIF link. ? When I use the method link to an local process ,the result will be what I wish to see. ? ??? Like this ??start()-> ?????Pid1=spawn(fun()-> sleep(5000),exit("auto exit") end), Lib_misc:on_exit(Pid1 , fun(Msg)-> io:format("hello ~p~n",[Msg]) end). ? ? sleep(T) -> receive after T -> ?? true end. ??? ? After execute the method start for 5 sec , the main process will receive exit sign from the child process. ? ? ? But when I create an new erlang node on an another server, like below: ?? [baisui@REDACTED code]$ erl -name remote -setcookie abc ? And execute the method linktest: startreceivemsg() wait to receive message from network ? ??? -module(linktest). -export([startreceivemsg/0]). ? ready2receivemsg() -> ? receive ?? Msg -> ??? io:format("receive ~p~n",[Msg]), ??? ready2receivemsg() ? end. ? startreceivemsg() -> register(remote, spawn(fun ready2receivemsg/0)). ??? ? Then i launch an another node on my pc: ? erl -name bbbb -setcookie abc and execute the BIF link method link to remote process what I have just created, (bbbb@REDACTED)11> linktest:link_remote(). and then throw an error: ??? <0.57.0> =ERROR REPORT==== 15-Oct-2012::17:16:22 === Error in process <0.57.0> on node 'bbbb@REDACTED' with exit value: {badarg,[{erlang,link,[{remote,'remote@REDACTED qa.cm6'}]},{lib_misc,'-on_exit/2-fun-0-',2}]} ??? ? My quest is how can I remote to an remote process located in anothre server? Any body can help me ,thanks. ? ??? -module(linktest). -export([start/0, link_remote/0]). ? start()-> Pid1=spawn(fun()-> sleep(5000),exit("auto exit") end), lib_misc:on_exit(Pid1 , fun(Msg)-> io:format("hello ~p~n",[Msg]) end). ? link_remote() -> lib_misc:on_exit( {remote,'remote@REDACTED'} , fun(Msg)-> io:format("hello ~p~n",[Msg]) end). ??? ? The method lib_misc:on_exit() was define as below: ??? on_exit(Pid, Fun) -> ??? spawn(fun() -> ??????? ??process_flag(trap_exit, true), ??????? ? link(Pid),?????????????????? ??????? ??receive ??????? ????? {'EXIT', Pid, Why} ->???? ??????????? ??Fun(Why)?? %%