From alessandro.sivieri@REDACTED Mon Nov 1 00:16:41 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Mon, 1 Nov 2010 00:16:41 +0100 Subject: [erlang-questions] debian & ubuntu R14B In-Reply-To: <1288532917.6645.0.camel@blackrain> References: <1288274732.2975.54.camel@cyclone> <20101028184852.GA33595@alumni.caltech.edu> <1288470480.15939.8.camel@blackrain> <1288473799.15939.10.camel@blackrain> <1288474817.15939.16.camel@blackrain> <1288532917.6645.0.camel@blackrain> Message-ID: I have published Erlang R14 in my PPA ( http://launchpad.net/~scattino/+archive/ppa), in a tentative of direct backport of the packages in Debian testing; if you want, you can try and use it, it works for me(TM) but it may not for you, so please don't blame me for that. In summary: use it at your own risk; if you find it useful, good for you :) -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From ok@REDACTED Mon Nov 1 04:08:12 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 1 Nov 2010 16:08:12 +1300 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: References: Message-ID: On 30/10/2010, at 9:03 PM, Dmitry Demeshchuk wrote: > Greetings. > > I'm writing an article comparing Erlang and Node.js and I stumbled > upon the performance question. Jest: it looks more like you tripped over it. > > My initial goal was to compare some basic arithmetics speed, like the > total distance between randomly distributed points. It seems strange to see floating point called *basic*. I see that you did not include random number generation in your measurement. Wise of you. To give you some figures I happen to have for a recent simulation in C: "Time with checking and -g : 16.5 seconds. Time sans checking and -O3 : 9.4 seconds using drand48(). 8.9 seconds using my_drand48(). 7.5 seconds using Mersenne Twister." drand48() is the classic System V 48-bit linear congruential generator, done using 16-bit chunks. my_drand48() is the same algorithm done using 64-bit arithmetic and with no locking. The Mersenne Twister is the integer one, not the more recent version. So within *ONE* language with *ONE* compiler on *ONE* platform there was a factor of 1.75 from compiler options and a factor of 1.25 from random number generator. The Erlang random number generator is AS183 (about the speed of drand48() or somewhat slower), and it's written in Erlang. In OSSP js, Version 1.6 Math.random() is implemented in C. (It looks pretty much like my_drand48().) However, the fact that Erlang and JavaScript don't use the same generator means you're not measuring the same calculations. > So, I have written > the following code for Erlang: > > ===================================================== > > -module(arith_speed). > -export([ > test/1 > ]). > > test(N) -> > L = lists:seq(1, N), > [{X0, Y0} | Points] = [{random:uniform(1000), > random:uniform(1000)} || _ <- L], > Now = now(), > lists:foldl(fun move_to/2, {0, {X0, Y0}}, Points), > timer:now_diff(now(), Now). > > move_to({X, Y}, {Sum, {X0, Y0}}) -> > {Sum + math:sqrt((X - X0) * (X - X0) + (Y - Y0) * (Y - Y0)), {X, Y}}. This would not be the fastest way to do it. test(N) -> Data = [{random:uniform(1000),random:uniform(1000)} || _ <- lists:seq(1, N)], Before = now(), _ = path_length(Data), timer:now_diff(now(), Before). path_length([{X0,Y0}|Path]) -> path_length(Path, X0, Y0, 0.0). path_length([{X,Y}|Path], X0, Y0, Length) -> Dx = float(X - X0), Dy = float(Y - Y0), path_length(Path, X, Y, Length + math:sqrt(Dx*Dx + Dy*Dy)); path_length([], _, _, Length) -> Length. would probably be faster. In fact, native-compiled on an elderly 500MHz UltraSPARC II, this version is 3/7 the time of the original code. (Using Dx = float(X - X0) does actually improve the results a fair bit.) Note that the original Erlang code and the original Javascript code do rather different things. (1) The Erlang code uses a higher-order function, the JavaScript code does not, even though higher order functions are available in JavaScript. (Good for JavaScript, bad for Erlang.) (2) The Erlang code represents a point by an 2-element array with elements extracted by pattern matching; the JavaScript version uses objects with fields at least notionally found by searching a hash table. (Good for Erlang, bad for JavaScript.) Note also that there are *huge* performance differences between JavaScript implementations. I suspect that a much more relevant benchmark for many potential users of Node.js would be something like "how many clients can a hello world server like the one on the nodejs.org home page serve at once?" (Hey, if it _isn't_ relevant, what's it doing on the home page?) From ok@REDACTED Mon Nov 1 04:16:21 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 1 Nov 2010 16:16:21 +1300 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <4CCBEB85.8050209@amberbio.com> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> Message-ID: <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> On 30/10/2010, at 10:55 PM, Morten Krogh wrote: > Sorry, I just realized that you are only timing the inner loop, not the random number generation. > > But why bother with the random numbers then? Shouldn't multiplication and addition run with the same speed for all values of numbers of a given type? No. For example, according to http://pdos.csail.mit.edu/6.828/2007/readings/i386/IMUL.htm "The 80386 uses an early-out multiply algorithm. The actual number of clocks depends on the position of the most significant bit in the ... multiplier." Look for "early-out multiply". Since these numbers were the results of random:uniform(1000) they should be particularly nice cases for an early-out multiplier. Had they been random:uniform(1000000) the products would have been bignums on a 32-bit machine and thus *much* slower; bignum operations *definitely* depend on the size of the number. Nor can you expect the cost of + and * to be uniform for floats. Operations that overflow to +/-infinity may well be *very* expensive on some machines. One popular implementation strategy for IEEE arithmetic was to do the easy cases in hardware and let the operating system finish the tough jobs. From max.lapshin@REDACTED Mon Nov 1 06:08:51 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Nov 2010 08:08:51 +0300 Subject: [erlang-questions] debian & ubuntu R14B In-Reply-To: References: <1288274732.2975.54.camel@cyclone> <20101028184852.GA33595@alumni.caltech.edu> <1288470480.15939.8.camel@blackrain> <1288473799.15939.10.camel@blackrain> <1288474817.15939.16.camel@blackrain> <1288532917.6645.0.camel@blackrain> Message-ID: Thank you a lot! From max.lapshin@REDACTED Mon Nov 1 06:10:30 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 1 Nov 2010 08:10:30 +0300 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> Message-ID: I'm still interested in the message that using HiPE and protected with guards function, I can unbox integers and generate very fast arithmetics. Is it planning feature or it is already implemented? From mk@REDACTED Mon Nov 1 08:50:35 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 01 Nov 2010 08:50:35 +0100 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> Message-ID: <4CCE714B.3090402@amberbio.com> Richard, my comment was for arithmetic on same type integers between 1 and 1000. Clearly for bignums, the performance depends on the values. By type I mean a fixed memory size and machine type. The OP was generating random integers between 0 and 1000, so I would assume that both js and erlang would run with the same speed inpedendently of whether he multiplied 500 * 500, say, a million times or random1 * random2 a million times. There could be a difference in the accumulator however, when the accumulator reached a critical size. This statement is interesting. The actual number of clocks depends on the position of the most significant bit in the ... multiplier." I tried testing on a C program that would multiply and sum (and subtract to avoid overflow of the accumulator). The type was int. It runs with exact same speed whether it is 0 * 0 or 1000 * 1000. I also tried float multiplication and couldn't see the slightest difference between 0*0 and 1000000 * 1000000 in a loop of billions of iterations. The test was on a intel core duo chip. But I would be interested in seeing an example of such a difference. I guess your link shows that such examples exist. Cheers Morten. On 11/1/10 4:16 AM, Richard O'Keefe wrote: > On 30/10/2010, at 10:55 PM, Morten Krogh wrote: > >> Sorry, I just realized that you are only timing the inner loop, not the random number generation. >> >> But why bother with the random numbers then? Shouldn't multiplication and addition run with the same speed for all values of numbers of a given type? > No. > > For example, according to > http://pdos.csail.mit.edu/6.828/2007/readings/i386/IMUL.htm > "The 80386 uses an early-out multiply algorithm. > The actual number of clocks depends on the position of the most > significant bit in the ... multiplier." > > Look for "early-out multiply". > > Since these numbers were the results of random:uniform(1000) > they should be particularly nice cases for an early-out multiplier. > Had they been random:uniform(1000000) the products would have > been bignums on a 32-bit machine and thus *much* slower; > bignum operations *definitely* depend on the size of the number. > > Nor can you expect the cost of + and * to be uniform for > floats. Operations that overflow to +/-infinity may well be > *very* expensive on some machines. One popular implementation > strategy for IEEE arithmetic was to do the easy cases in hardware > and let the operating system finish the tough jobs. > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From demeshchuk@REDACTED Mon Nov 1 11:21:15 2010 From: demeshchuk@REDACTED (Dmitry Demeshchuk) Date: Mon, 1 Nov 2010 13:21:15 +0300 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: References: Message-ID: Thanks for the explanations, guys. I'll need to fill that empty hole in my computer science knowledge, along with the other ones :) At the beginning, I was too stupid to understand the fact that the whole "arithmetics" question concerns a lot of different conditions, like what we calculate and which algorithm we use. As well as that different operations like function calls, arrays or lists walking have different cost for different platforms. So, in the "arithmetics" section I'll put this explanation with a note that some comparison can be seen at the links provided by Isaac Gouy, but most likely they are also very synthetic and not reliable. Of course, the comparison of web servers will take place as well. It won't be fair as well (because the implementations will differ a lot) but will somehow show how both the platforms can handle this task. Thank you all again. On Mon, Nov 1, 2010 at 6:08 AM, Richard O'Keefe wrote: > > On 30/10/2010, at 9:03 PM, Dmitry Demeshchuk wrote: > >> Greetings. >> >> I'm writing an article comparing Erlang and Node.js and I stumbled >> upon the performance question. > > Jest: ?it looks more like you tripped over it. >> >> My initial goal was to compare some basic arithmetics speed, like the >> total distance between randomly distributed points. > > It seems strange to see floating point called *basic*. > > I see that you did not include random number generation in your > measurement. ?Wise of you. ?To give you > some figures I happen to have for a recent simulation in C: > > ? ? ?"Time with checking and -g ?: 16.5 seconds. > ? ? ? Time sans checking and -O3 : ?9.4 seconds using drand48(). > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 8.9 seconds using my_drand48(). > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 7.5 seconds using Mersenne Twister." > > drand48() is the classic System V 48-bit linear congruential generator, > done using 16-bit chunks. ?my_drand48() is the same algorithm done using > 64-bit arithmetic and with no locking. ?The Mersenne Twister is the > integer one, not the more recent version. > > So within *ONE* language with *ONE* compiler on *ONE* platform there was > a factor of 1.75 from compiler options and a factor of 1.25 from random > number generator. > > The Erlang random number generator is AS183 (about the speed of drand48() > or somewhat slower), and it's written in Erlang. ?In OSSP js, Version 1.6 > Math.random() is implemented in C. ?(It looks pretty much like my_drand48().) > > However, the fact that Erlang and JavaScript don't use the same generator > means you're not measuring the same calculations. > > >> So, I have written >> the following code for Erlang: >> >> ===================================================== >> >> -module(arith_speed). >> -export([ >> ? ?test/1 >> ]). >> >> test(N) -> >> ? ?L = lists:seq(1, N), >> ? ?[{X0, Y0} | Points] = [{random:uniform(1000), >> random:uniform(1000)} || _ <- L], >> ? ?Now = now(), >> ? ?lists:foldl(fun move_to/2, {0, {X0, Y0}}, Points), >> ? ?timer:now_diff(now(), Now). >> >> move_to({X, Y}, {Sum, {X0, Y0}}) -> >> ? ?{Sum + math:sqrt((X - X0) * (X - X0) + (Y - Y0) * (Y - Y0)), {X, Y}}. > > This would not be the fastest way to do it. > > ? ? ? ?test(N) -> > ? ? ? ? ? ?Data = [{random:uniform(1000),random:uniform(1000)} > ? ? ? ? ? ? ? ? ? || _ <- lists:seq(1, N)], > ? ? ? ? ? ?Before = now(), > ? ? ? ? ? ?_ = path_length(Data), > ? ? ? ? ? ?timer:now_diff(now(), Before). > > ? ? ? ?path_length([{X0,Y0}|Path]) -> > ? ? ? ? ? ?path_length(Path, X0, Y0, 0.0). > > ? ? ? ?path_length([{X,Y}|Path], X0, Y0, Length) -> > ? ? ? ? ? ?Dx = float(X - X0), > ? ? ? ? ? ?Dy = float(Y - Y0), > ? ? ? ? ? ?path_length(Path, X, Y, Length + math:sqrt(Dx*Dx + Dy*Dy)); > ? ? ? ?path_length([], _, _, Length) -> > ? ? ? ? ? ?Length. > > would probably be faster. ?In fact, native-compiled on an elderly > 500MHz UltraSPARC II, this version is 3/7 the time of the original code. > (Using Dx = float(X - X0) does actually improve the results a fair bit.) > > Note that the original Erlang code and the original Javascript code do > rather different things. > (1) The Erlang code uses a higher-order function, > ? ?the JavaScript code does not, even though higher order functions > ? ?are available in JavaScript. ?(Good for JavaScript, bad for Erlang.) > (2) The Erlang code represents a point by an 2-element array with > ? ?elements extracted by pattern matching; the JavaScript version uses > ? ?objects with fields at least notionally found by searching a hash > ? ?table. ?(Good for Erlang, bad for JavaScript.) > > Note also that there are *huge* performance differences between JavaScript > implementations. > > I suspect that a much more relevant benchmark for many potential users > of Node.js would be something like "how many clients can a hello world > server like the one on the nodejs.org home page serve at once?" > > (Hey, if it _isn't_ relevant, what's it doing on the home page?) > > > -- Best regards, Dmitry Demeshchuk From dang@REDACTED Mon Nov 1 15:08:42 2010 From: dang@REDACTED (Daniel Goertzen) Date: Mon, 1 Nov 2010 09:08:42 -0500 Subject: [erlang-questions] ei: decoding negative values In-Reply-To: References: Message-ID: See ei_s_print_term() to directly render the number to text, then you don't have to worry about the sign. Alternatively, try a larger signed decode like ei_decode_longlong() or ei_decode_bignum(). Cheers, Dan. On Thu, Oct 28, 2010 at 10:13 AM, Joel Reymont wrote: > When using ei, how do I know whether to decode signed or unsigned values? > > I'm not sure if I should use ei_decode_long or ei_decode_ulong and > ei_get_type is not very helpful here. > > My goal is to display the value that was encoded. > > Thanks, Joel > > --- > http://twitter.com/wagerlabs > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Daniel Goertzen ----------------- dang@REDACTED (work) daniel.goertzen@REDACTED (home) ----------------- 1 204 272 6149 (home/office) 1 204 470 8360 (mobile) ----------------- From bengt.kleberg@REDACTED Mon Nov 1 17:38:39 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Mon, 01 Nov 2010 17:38:39 +0100 Subject: [erlang-questions] limitations of erlang:open_port() and os:cmd() In-Reply-To: <20101029104041.GA7590@corelatus.se> References: <20101029104041.GA7590@corelatus.se> Message-ID: <1288629519.5119.24.camel@seasc1137> Greetings, This is probably not what you want, but the shell rc (http://doc.cat-v.org/plan_9/4th_edition/papers/rc) has multiple exit status from a pipe. Like this: "Rc captures command exit status in the variable $status. For a simple command the value of $status is just as described above. For a pipeline $status is set to the concatenation of the statuses of the pipeline components with | characters for separators." bengt On Fri, 2010-10-29 at 12:40 +0200, Matthias Lang wrote: > Hi, > > I know of a few limitations when running external programs from > Erlang, and I know a few workarounds. > > Does anyone know of limitations and/or workarounds I don't know about? > Would be interesting to hear about them. > > The limitations I know of are: > > 1. There's no way to do flow control. > 2. There's no way to kill a misbehaving process > 3. There's no way to send eof _and_ get the process' exit status > > Expanding on those: > > -------------------- > 1. There's no way to do flow control > > Example: > 1> os:cmd("cat /dev/zero"). > Crash dump was written to: erl_crash.dump > eheap_alloc: Cannot allocate 1048656 bytes of memory (of type "heap_frag"). > > Workaround 1.1: > 1> os:cmd("cat /dev/zero | netcat -l -p 49152"). > User switch command --> s --> c > 1> {ok, S} = gen_tcp:connect("localhost", 49152, [{active, once}]). > {ok,#Port<0.607>} > 2> flush(). > Shell got {tcp,#Port<0.607>, [0,0,0,0,0,0,0,0,0,0 > > Suckage: depends on netcat, I'm not sure which process' exit status you get > > -------------------- > 2. There's no way to kill a misbehaving process > > Example: > 1> os:cmd("wc /dev/zero"). > > Workaround 2.1: > Write a wrapper program which echoes the unix pid before exec()ing > the actual command. > > Workaround 2.2: > Write a wrapper program which forks() and copies IO to the child, > terminating the child on eof. > > Suckage: you have to write a wrapper > > -------------------- > 3. There's no way to send eof _and_ get the process' exit status > > Example: > 1> {ok, Bin} = file:read_file("/tmp/mml.tgz"). > {ok,<<31,139,8,0,100,129,202,76,0,3,236,93,9,120,20,85, > 2> P = open_port({spawn, "tar -xzf -"}, [exit_status, binary]). > #Port<0.535> > 3> port_command(P, Bin). > true > 4> flush(). > ok > 5> port_close(P). > true > 6> flush(). > ok > > Workaround 3.1: > Abuse strace, using knowledge of the unix pid gained somehow: > > os:cmd("strace -p 7974 -e trace=process"). > > Workaround 3.2: > Go via netcat again, e.g. > > 2> P = open_port({spawn, "netcat -l -p 49152 | tar -xzf -"}, [exit_status, binary]). > 3> {ok, S} = gen_tcp:connect("localhost", 49152, []). > 4> gen_tcp:send(S, Bin), gen_tcp:close(S), flush(). > > Suckage: probably only works on unix, requires netcat > > > > Matt > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From zabrane3@REDACTED Mon Nov 1 23:09:47 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Mon, 1 Nov 2010 23:09:47 +0100 Subject: Starting an echo web server from (bash) shell! Message-ID: Hi list, I'm facing a strange problem when trying to start my very basic echo web server from (bash) shell. It doesn't work when I try to run it from (bash) shell with an error in "gen_tcp:accept/1" call: $ erl -s bews But it works fine when I run it from the Erlang shell: $ erl 1> bews:start(). The server is simply listening on socket 8888 and echo back what it gets: ... case catch gen_tcp:accept(ListenSocket) of {ok, Sock} -> ... _ -> exit({error,accept_failed}) end. Any advices please? N.B: I'm under OSX Snow Leopard, Erlang R14B (erts-5.8.1) -- Regards Zabrane From spawn.think@REDACTED Tue Nov 2 00:03:34 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 2 Nov 2010 00:03:34 +0100 Subject: [erlang-questions] Starting an echo web server from (bash) shell! In-Reply-To: References: Message-ID: I would suggest you change the code to reveal the original error that you got so for Example could you raise the error that you get? or remove the catch and change the other case clause to {error, Error} and print out that Error? {error, Error}-> io:format("~p~n", [Error]) end. On Mon, Nov 1, 2010 at 11:09 PM, zabrane Mikael wrote: > Hi list, > > I'm facing a strange problem when trying to start my very basic echo > web server from (bash) shell. > > It doesn't work when I try to run it from (bash) shell with an error > in "gen_tcp:accept/1" call: > $ erl -s bews > > > But it works fine when I run it from the Erlang shell: > $ erl > 1> bews:start(). > > The server is simply listening on socket 8888 and echo back what it gets: > ... > case catch gen_tcp:accept(ListenSocket) of > {ok, Sock} -> > ... > _ -> > exit({error,accept_failed}) > end. > > Any advices please? > > N.B: I'm under OSX Snow Leopard, Erlang R14B (erts-5.8.1) > > -- > Regards > Zabrane > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From ok@REDACTED Tue Nov 2 00:11:47 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 2 Nov 2010 12:11:47 +1300 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <4CCE714B.3090402@amberbio.com> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> <4CCE714B.3090402@amberbio.com> Message-ID: On 1/11/2010, at 8:50 PM, Morten Krogh wrote: > Richard, > > my comment was for arithmetic on same type integers between 1 and 1000. Ah. I didn't understand that. > > The actual number of clocks depends on the position of the most > significant bit in the ... multiplier." > > > I tried testing on a C program ... > The test was on a intel core duo chip. The hardware matters. For example, SPARC V7 didn't have a hardware multiply; the software multiply had a single threshold early-out check. SPARC V8 had 32x32->64 bit multiply in hardware; SPARC V9 has 64x64->64. Current generation desktop chips are likely to have a single-cycle multiplier. These require lots of chip area, but then, current generation desktop chips *have* lots of chip area to spare. Low power devices are likely to use different methods, and so are seriously multicore chips like the Sun T2. It would be interesting to know what the Tilera chips do. (Did the 512-core Tilera servers announced in June ever see the light of day?) From zabrane3@REDACTED Tue Nov 2 00:17:45 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 2 Nov 2010 00:17:45 +0100 Subject: [erlang-questions] Starting an echo web server from (bash) shell! In-Reply-To: References: Message-ID: Hi Ahmed, Following your suggestion, I removed the "catch" and print out the error with: io:format("Accept failed with error: ~p~n", [Error]). I got: Accept failed with error: closed These are my socket options: -define(SOCK_TCP_PORT, 8888). -define(SOCK_TCP_OPTS, [binary, {nodelay, true}, {active, false}, {packet, http_bin}, {reuseaddr, true}]). Hope this help. -- Regards Zabrane 2010/11/2 Ahmed Omar : > I would suggest you change the code to reveal the original error that you > got > so for Example could you raise the error that you get? > or remove the catch and change the other case clause to {error, Error} and > print out that Error? > {error, Error}-> > ?? ? ? io:format("~p~n", [Error]) > end. > > > On Mon, Nov 1, 2010 at 11:09 PM, zabrane Mikael wrote: >> >> Hi list, >> >> I'm facing a strange problem when trying to start my very basic echo >> web server from (bash) shell. >> >> It doesn't work when I try to run it from (bash) shell with an error >> in "gen_tcp:accept/1" call: >> $ erl -s bews >> >> >> But it works fine when I run it from the Erlang shell: >> $ erl >> 1> bews:start(). >> >> The server is simply listening on socket 8888 and echo back what it gets: >> ... >> case catch gen_tcp:accept(ListenSocket) of >> ? ? ? ?{ok, Sock} -> >> ? ? ? ? ? ? ? ... >> ? ? ? ?_ -> >> ? ? ? ? ? exit({error,accept_failed}) >> end. >> >> Any advices please? >> >> N.B: I'm under OSX Snow Leopard, Erlang R14B (erts-5.8.1) >> >> -- >> Regards >> Zabrane >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > > -- > Best Regards, > - Ahmed Omar > http://nl.linkedin.com/in/adiaa > Follow me on twitter > @spawn_think > From spawn.think@REDACTED Tue Nov 2 10:07:10 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 2 Nov 2010 10:07:10 +0100 Subject: [erlang-questions] Starting an echo web server from (bash) shell! In-Reply-To: References: Message-ID: Hi zabrane, I tried with the options you supplied and it works fine. Can you please post the code maybe there's a problem there? Also how are you testing it? On Tue, Nov 2, 2010 at 12:17 AM, zabrane Mikael wrote: > Hi Ahmed, > > Following your suggestion, I removed the "catch" and print out the error > with: > io:format("Accept failed with error: ~p~n", [Error]). > > I got: > Accept failed with error: closed > > These are my socket options: > -define(SOCK_TCP_PORT, 8888). > -define(SOCK_TCP_OPTS, [binary, > {nodelay, true}, > {active, false}, > {packet, http_bin}, > {reuseaddr, true}]). > > Hope this help. > > -- > Regards > Zabrane > > 2010/11/2 Ahmed Omar : > > I would suggest you change the code to reveal the original error that you > > got > > so for Example could you raise the error that you get? > > or remove the catch and change the other case clause to {error, Error} > and > > print out that Error? > > {error, Error}-> > > io:format("~p~n", [Error]) > > end. > > > > > > On Mon, Nov 1, 2010 at 11:09 PM, zabrane Mikael > wrote: > >> > >> Hi list, > >> > >> I'm facing a strange problem when trying to start my very basic echo > >> web server from (bash) shell. > >> > >> It doesn't work when I try to run it from (bash) shell with an error > >> in "gen_tcp:accept/1" call: > >> $ erl -s bews > >> > >> > >> But it works fine when I run it from the Erlang shell: > >> $ erl > >> 1> bews:start(). > >> > >> The server is simply listening on socket 8888 and echo back what it > gets: > >> ... > >> case catch gen_tcp:accept(ListenSocket) of > >> {ok, Sock} -> > >> ... > >> _ -> > >> exit({error,accept_failed}) > >> end. > >> > >> Any advices please? > >> > >> N.B: I'm under OSX Snow Leopard, Erlang R14B (erts-5.8.1) > >> > >> -- > >> Regards > >> Zabrane > >> > >> ________________________________________________________________ > >> erlang-questions (at) erlang.org mailing list. > >> See http://www.erlang.org/faq.html > >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > >> > > > > > > > > -- > > Best Regards, > > - Ahmed Omar > > http://nl.linkedin.com/in/adiaa > > Follow me on twitter > > @spawn_think > > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From matthias@REDACTED Tue Nov 2 12:31:16 2010 From: matthias@REDACTED (Matthias Lang) Date: Tue, 2 Nov 2010 12:31:16 +0100 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> <4CCE714B.3090402@amberbio.com> Message-ID: <20101102113116.GA2886@corelatus.se> On Tuesday, November 02, Richard O'Keefe wrote: > Current generation desktop chips are likely to have a single-cycle multiplier. > These require lots of chip area, but then, current generation desktop chips > *have* lots of chip area to spare. Low power devices are likely to use > different methods The development of low-power devices has been relentless. 32x32bit single cycle multiplies (assuming "can issue one of those operations every cycle" qualifies for "single cycle") multiplies are not uncommon even in devices that draw less than 1W. Some actual, shipping devices: * 600MHz TI DM6443 DSP, draws less than 1W, can issue _two_ 32x32 bit multiplies every cycle. Variants go beyond 1GHz. * 500MHz Au1000 CPU, draws about 500mW, does one 32x32bit multiply every second cycle (and that chip is more than five years old) * 1GHz ARM A8, not sure what it draws, probably < 1W, can issue one 32x32 multiply every cycle if I read the datasheet right. I have no idea what Tilera does. Matt From aaron.broad@REDACTED Tue Nov 2 13:26:52 2010 From: aaron.broad@REDACTED (Aaron Broad) Date: Tue, 2 Nov 2010 09:26:52 -0300 Subject: SMP in emacs? In-Reply-To: <26305_1288567064_4CCDF917_26305_176175_1_AANLkTik9WEdL7HYyHvKBTbW4tL3TuuLJzgasM2zjsWGL@mail.gmail.com> References: <1288274732.2975.54.camel@cyclone> <20101028184852.GA33595@alumni.caltech.edu> <1288470480.15939.8.camel@blackrain> <1288473799.15939.10.camel@blackrain> <1288474817.15939.16.camel@blackrain> <1288532917.6645.0.camel@blackrain> <26305_1288567064_4CCDF917_26305_176175_1_AANLkTik9WEdL7HYyHvKBTbW4tL3TuuLJzgasM2zjsWGL@mail.gmail.com> Message-ID: Hi all, I'm trying to develop an wx application which requires SMP. But by default the emacs tools are leaving me in non-smp mode. How do I switch my default shell in emacs to an smp enabled one? NB: erl -smp works fine on the command prompt. In ubuntu with latest version (tools-2.6.6.1) thanks, Aaron From masklinn@REDACTED Tue Nov 2 13:28:05 2010 From: masklinn@REDACTED (Masklinn) Date: Tue, 2 Nov 2010 13:28:05 +0100 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <20101102113116.GA2886@corelatus.se> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> <4CCE714B.3090402@amberbio.com> <20101102113116.GA2886@corelatus.se> Message-ID: <3466733F-57D0-4214-B83C-69C15CB6F48D@masklinn.net> On 2010-11-02, at 12:31 , Matthias Lang wrote: > > * 1GHz ARM A8, not sure what it draws, probably < 1W That's an understatement. The A8 core is specced for under 0.59mW/MHz. That yields ~600mW for a 1GHz core. And that's without any manufacturer-specific power optimizations and with a core running full tilt, all units enabled. From aaron.broad@REDACTED Tue Nov 2 17:38:00 2010 From: aaron.broad@REDACTED (Aaron Broad) Date: Tue, 2 Nov 2010 13:38:00 -0300 Subject: [erlang-questions] SMP in emacs? In-Reply-To: References: <1288274732.2975.54.camel@cyclone> <20101028184852.GA33595@alumni.caltech.edu> <1288470480.15939.8.camel@blackrain> <1288473799.15939.10.camel@blackrain> <1288474817.15939.16.camel@blackrain> <1288532917.6645.0.camel@blackrain> <26305_1288567064_4CCDF917_26305_176175_1_AANLkTik9WEdL7HYyHvKBTbW4tL3TuuLJzgasM2zjsWGL@mail.gmail.com> Message-ID: <030CADE9-8103-40FE-A450-A042611CB940@unb.ca> Hi Aaron, in /lib/tools-2.6.6.1/emacs/erlang.el I switched the variable erlang-machine-opts to '("-smp") from '() Now I can compile wx and run wx programs in emacs Thanks:-), Aaron On 2010-11-02, at 9:26 AM, Aaron Broad wrote: > Hi all, > > I'm trying to develop an wx application which requires SMP. But by default the emacs tools are leaving me in non-smp mode. How do I switch my default shell in emacs to an smp enabled one? NB: erl -smp works fine on the command prompt. > > In ubuntu with latest version (tools-2.6.6.1) > > thanks, > Aaron > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From dgud@REDACTED Tue Nov 2 17:50:34 2010 From: dgud@REDACTED (Dan Gudmundsson) Date: Tue, 2 Nov 2010 17:50:34 +0100 Subject: [erlang-questions] SMP in emacs? In-Reply-To: <030CADE9-8103-40FE-A450-A042611CB940@unb.ca> References: <1288274732.2975.54.camel@cyclone> <20101028184852.GA33595@alumni.caltech.edu> <1288470480.15939.8.camel@blackrain> <1288473799.15939.10.camel@blackrain> <1288474817.15939.16.camel@blackrain> <1288532917.6645.0.camel@blackrain> <26305_1288567064_4CCDF917_26305_176175_1_AANLkTik9WEdL7HYyHvKBTbW4tL3TuuLJzgasM2zjsWGL@mail.gmail.com> <030CADE9-8103-40FE-A450-A042611CB940@unb.ca> Message-ID: C-u M-x erlang-shell erl -smp +S12 > erlang:system_info(schedulers). 12 /Dan PS: If you read/modified the code you could have seen that :-) On Tue, Nov 2, 2010 at 5:38 PM, Aaron Broad wrote: > Hi Aaron, > > in /lib/tools-2.6.6.1/emacs/erlang.el > > I switched the variable erlang-machine-opts to '("-smp") from '() > > Now I can compile wx and run wx programs in emacs > > Thanks:-), > Aaron > > On 2010-11-02, at 9:26 AM, Aaron Broad wrote: > >> Hi all, >> >> I'm trying to develop an wx application which requires SMP. ?But by default the emacs tools are leaving me in non-smp mode. ?How do I switch my default shell in emacs to an smp enabled one? ?NB: ?erl -smp works fine on the command prompt. >> >> In ubuntu with latest version (tools-2.6.6.1) >> >> thanks, >> Aaron >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From thomasl_erlang@REDACTED Tue Nov 2 17:18:07 2010 From: thomasl_erlang@REDACTED (Thomas Lindgren) Date: Tue, 2 Nov 2010 09:18:07 -0700 (PDT) Subject: [erlang-questions] non-trivial supervisor idioms? -- An open-source plea! In-Reply-To: References: <4CA0AAD3.3030209@erlang-solutions.com> <249668.71494.qm@web111413.mail.gq1.yahoo.com> <716334.78231.qm@web111415.mail.gq1.yahoo.com> Message-ID: <2830.5139.qm@web111403.mail.gq1.yahoo.com> ----- Original Message ---- > From: Attila Rajmund Nohl > > 2010/10/28, Thomas Lindgren : > [...] > > Well, Erlang was intended to be easy to learn and use, and these products > > have > > had dozens or even hundreds of programmers working on them over many years. > > It's > > pretty natural, isn't it? > > On the other hand I think this is true for nearly all programming > languages, the C/Java projects have similar turnabouts. By the way, I > had a colleague who has working on AXD, but he said he didn't wrote a > single line of new code, he just copied and pasted some earlier > written code - and was genuinely surprised that it worked. Hi Attila, Yes indeed -- it's not a language issue but the reality of running a big project. Best, Thomas From mevans@REDACTED Tue Nov 2 18:47:46 2010 From: mevans@REDACTED (Evans, Matthew) Date: Tue, 2 Nov 2010 13:47:46 -0400 Subject: DETS Problems Message-ID: List, I'm using a dets table to save reasonably large records (1K or there abouts). The following sequence is applied to a table: dets:open_file/2/ dets:insert/2 (a few times). ets:to_dets/2 dets:close/1 Sometime later I reopen the table with dets:open_file/1 The table is then corrupted. Reads produce the following error: {error,{{bad_object,eval_work_list}, "/content/c2idx/5/95/10b12417-4629-404a-8361-68bd3de77d90.dat"}} What causes that error? Thanks From mevans@REDACTED Tue Nov 2 19:10:20 2010 From: mevans@REDACTED (Evans, Matthew) Date: Tue, 2 Nov 2010 14:10:20 -0400 Subject: DETS Problems In-Reply-To: References: Message-ID: I should add, that I do ets:insert/2 followed by a ets:to_dets/2. Some of the records are about 4K in size. -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Evans, Matthew Sent: Tuesday, November 02, 2010 1:48 PM To: erlang-questions Subject: [erlang-questions] DETS Problems Importance: High List, I'm using a dets table to save reasonably large records (1K or there abouts). The following sequence is applied to a table: dets:open_file/2/ dets:insert/2 (a few times). ets:to_dets/2 dets:close/1 Sometime later I reopen the table with dets:open_file/1 The table is then corrupted. Reads produce the following error: {error,{{bad_object,eval_work_list}, "/content/c2idx/5/95/10b12417-4629-404a-8361-68bd3de77d90.dat"}} What causes that error? Thanks From rgowka1@REDACTED Tue Nov 2 19:25:02 2010 From: rgowka1@REDACTED (rgowka1) Date: Tue, 2 Nov 2010 14:25:02 -0400 Subject: [erlang-questions] RE: DETS Problems In-Reply-To: References: Message-ID: "Dets tables must be opened before they can be updated or read, and when finished they must be properly closed. If a table has not been properly closed, Dets will automatically repair the table. This can take a substantial time if the table is large. A Dets table is closed when the process which opened the table terminates. If several Erlang processes (users) open the same Dets table, they will share the table. The table is properly closed when all users have either terminated or closed the table. Dets tables are not properly closed if the Erlang runtime system is terminated abnormally." I did play around with dets with the same sized records as you. According to the above blurb and my observations, the corruption is usually not closing the dets in all the processes.. On Tue, Nov 2, 2010 at 2:10 PM, Evans, Matthew wrote: > I should add, that I do ets:insert/2 followed by a ets:to_dets/2. > > Some of the records are about 4K in size. > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Evans, Matthew > Sent: Tuesday, November 02, 2010 1:48 PM > To: erlang-questions > Subject: [erlang-questions] DETS Problems > Importance: High > > List, > > I'm using a dets table to save reasonably large records (1K or there abouts). > > The following sequence is applied to a table: > > dets:open_file/2/ > dets:insert/2 (a few times). > ets:to_dets/2 > dets:close/1 > > > Sometime later I reopen the table with dets:open_file/1 > > The table is then corrupted. Reads produce the following error: > > {error,{{bad_object,eval_work_list}, > ? ? ? ?"/content/c2idx/5/95/10b12417-4629-404a-8361-68bd3de77d90.dat"}} > > What causes that error? > > Thanks > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From spawn.think@REDACTED Tue Nov 2 20:45:36 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Tue, 2 Nov 2010 20:45:36 +0100 Subject: [erlang-questions] RE: DETS Problems In-Reply-To: References: Message-ID: Sorry to understand more, can you provide the options you are using for the table? what do u do with ets:to_dets/2 exactly? please also note that : "The Dets table is emptied before the objects are inserted" http://www.erlang.org/doc/man/ets.html#to_dets-2 On Tue, Nov 2, 2010 at 7:25 PM, rgowka1 wrote: > "Dets tables must be opened before they can be updated or read, and > when finished they must be properly closed. If a table has not been > properly closed, Dets will automatically repair the table. This can > take a substantial time if the table is large. A Dets table is closed > when the process which opened the table terminates. If several Erlang > processes (users) open the same Dets table, they will share the table. > The table is properly closed when all users have either terminated or > closed the table. Dets tables are not properly closed if the Erlang > runtime system is terminated abnormally." > > I did play around with dets with the same sized records as you. > According to the above blurb and my observations, the corruption is > usually not closing the dets in all the processes.. > > > On Tue, Nov 2, 2010 at 2:10 PM, Evans, Matthew wrote: > > I should add, that I do ets:insert/2 followed by a ets:to_dets/2. > > > > Some of the records are about 4K in size. > > > > -----Original Message----- > > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] > On Behalf Of Evans, Matthew > > Sent: Tuesday, November 02, 2010 1:48 PM > > To: erlang-questions > > Subject: [erlang-questions] DETS Problems > > Importance: High > > > > List, > > > > I'm using a dets table to save reasonably large records (1K or there > abouts). > > > > The following sequence is applied to a table: > > > > dets:open_file/2/ > > dets:insert/2 (a few times). > > ets:to_dets/2 > > dets:close/1 > > > > > > Sometime later I reopen the table with dets:open_file/1 > > > > The table is then corrupted. Reads produce the following error: > > > > {error,{{bad_object,eval_work_list}, > > "/content/c2idx/5/95/10b12417-4629-404a-8361-68bd3de77d90.dat"}} > > > > What causes that error? > > > > Thanks > > > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From mevans@REDACTED Tue Nov 2 20:53:07 2010 From: mevans@REDACTED (Evans, Matthew) Date: Tue, 2 Nov 2010 15:53:07 -0400 Subject: [erlang-questions] RE: DETS Problems In-Reply-To: References: Message-ID: It was actually worse than I thought. It seems (initial glance) that DETS doesn't like the large elements I was inserting. The table had 3 records of 3-5K in size, and about 700 of 300 bytes in size. It was constantly failing to find the large elements, if I looked up a small element first. Moving the large elements to a separate table seems to fix the problem. I'm going to do more analysis of this, we also use SSD/flash as the storage device, maybe that is causing problems. ________________________________ From: Ahmed Omar [mailto:spawn.think@REDACTED] Sent: Tuesday, November 02, 2010 3:46 PM To: Evans, Matthew Cc: erlang-questions Subject: Re: [erlang-questions] RE: DETS Problems Sorry to understand more, can you provide the options you are using for the table? what do u do with ets:to_dets/2 exactly? please also note that : " The Dets table is emptied before the objects are inserted" http://www.erlang.org/doc/man/ets.html#to_dets-2 On Tue, Nov 2, 2010 at 7:25 PM, rgowka1 > wrote: "Dets tables must be opened before they can be updated or read, and when finished they must be properly closed. If a table has not been properly closed, Dets will automatically repair the table. This can take a substantial time if the table is large. A Dets table is closed when the process which opened the table terminates. If several Erlang processes (users) open the same Dets table, they will share the table. The table is properly closed when all users have either terminated or closed the table. Dets tables are not properly closed if the Erlang runtime system is terminated abnormally." I did play around with dets with the same sized records as you. According to the above blurb and my observations, the corruption is usually not closing the dets in all the processes.. On Tue, Nov 2, 2010 at 2:10 PM, Evans, Matthew > wrote: > I should add, that I do ets:insert/2 followed by a ets:to_dets/2. > > Some of the records are about 4K in size. > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Evans, Matthew > Sent: Tuesday, November 02, 2010 1:48 PM > To: erlang-questions > Subject: [erlang-questions] DETS Problems > Importance: High > > List, > > I'm using a dets table to save reasonably large records (1K or there abouts). > > The following sequence is applied to a table: > > dets:open_file/2/ > dets:insert/2 (a few times). > ets:to_dets/2 > dets:close/1 > > > Sometime later I reopen the table with dets:open_file/1 > > The table is then corrupted. Reads produce the following error: > > {error,{{bad_object,eval_work_list}, > "/content/c2idx/5/95/10b12417-4629-404a-8361-68bd3de77d90.dat"}} > > What causes that error? > > Thanks > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From silasdb@REDACTED Tue Nov 2 21:14:42 2010 From: silasdb@REDACTED (Silas Silva) Date: Tue, 2 Nov 2010 18:14:42 -0200 Subject: Conceptual questions on key-value databases for RDBMs users Message-ID: <20101102201437.GC626@hope.tifa.renegado> This is a message I sent to the nosql-discussion@REDACTED discussion group. I thought it might be interesting to send it erlang-questions, so here we go... Hi all! I have used SQL RDBMSs for some time. I've never used any very advanced feature, but I know enough of it to make database applications. Nowadays, I decided it would be interesting to learn some NoSQL databases concepts. So I decided to pick up some Erlang and Mnesia, its native key-value database. More than scalability itself, the most valuable feature for me is the possibility of replication and synchronization between nodes. But all pros have cons as companion. The lack of a relationship model is difficult for who is used to RDBMSs. So, my question is: * Is there any guide, tutorial, book, whatever, that tries to introduce NoSQL databases to SQL users? * Key-value databases are surprising simple. I know you solve relationship by denormalizing data. What data should be normalized? What shouldn't? How do you update denormalized data? Sorry for such simple and general questions. Things were simple up to the moment that I realized that it would be easily solved with a JOIN SQL statement. :-) Thank you very much! -- Silas Silva From kennethstone@REDACTED Tue Nov 2 21:34:46 2010 From: kennethstone@REDACTED (Kenny Stone) Date: Tue, 2 Nov 2010 15:34:46 -0500 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <20101102201437.GC626@hope.tifa.renegado> References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: qlc is an erlang query interface you can use with mnesia to do things like joins. http://www.erlang.org/doc/man/qlc.html Joe Armstrong's book explains qlc with equivalent SQL statements. http://www.pragprog.com/titles/jaerlang/programming-erlang Kenny On Tue, Nov 2, 2010 at 3:14 PM, Silas Silva wrote: > This is a message I sent to the nosql-discussion@REDACTED > discussion group. I thought it might be interesting to send it > erlang-questions, so here we go... > > > Hi all! > > I have used SQL RDBMSs for some time. I've never used any very advanced > feature, but I know enough of it to make database applications. > > Nowadays, I decided it would be interesting to learn some NoSQL > databases concepts. So I decided to pick up some Erlang and Mnesia, its > native key-value database. More than scalability itself, the most > valuable feature for me is the possibility of replication and > synchronization between nodes. > > But all pros have cons as companion. The lack of a relationship model > is difficult for who is used to RDBMSs. So, my question is: > > * Is there any guide, tutorial, book, whatever, that tries to introduce > NoSQL databases to SQL users? > > * Key-value databases are surprising simple. I know you solve > relationship by denormalizing data. What data should be normalized? > What shouldn't? How do you update denormalized data? > > Sorry for such simple and general questions. Things were simple up to > the moment that I realized that it would be easily solved with a JOIN > SQL statement. :-) > > Thank you very much! > > -- > Silas Silva > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From silasdb@REDACTED Tue Nov 2 21:57:36 2010 From: silasdb@REDACTED (Silas Silva) Date: Tue, 2 Nov 2010 18:57:36 -0200 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: <20101102205731.GA3005@hope.tifa.renegado> Hi there. On Tue, Nov 02, 2010 at 03:34:46PM -0500, Kenny Stone wrote: > qlc is an erlang query interface you can use with mnesia to do things like > joins. > > http://www.erlang.org/doc/man/qlc.html > > Joe Armstrong's book explains qlc with equivalent SQL statements. > > http://www.pragprog.com/titles/jaerlang/programming-erlang Well, I've read first chapters of Armstrong's book and also the chapter about Mnesia. Some could use relationship with Mnesia, but it would rather have to do some job that a RDBMS do behind the scenes, like setting a secondary index (which could be done adding another "table" in Mnesia), right? Anyway, in Mnesia applications, is there any case wher denormalization is recommended rather than using qlc's joins-like? Thanks. -- Silas Silva From jesper.louis.andersen@REDACTED Tue Nov 2 22:52:06 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 2 Nov 2010 22:52:06 +0100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <20101102201437.GC626@hope.tifa.renegado> References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: > I have used SQL RDBMSs for some time. ?I've never used any very advanced > feature, but I know enough of it to make database applications. > > Nowadays, I decided it would be interesting to learn some NoSQL > databases concepts. ?So I decided to pick up some Erlang and Mnesia, its > native key-value database. ?More than scalability itself, the most > valuable feature for me is the possibility of replication and > synchronization between nodes. I'll rant. Beware. SQL is a declarative language, like QLC. It is quite domain-specific, but it has one distinct advantage: You can query-optimize on it. Most databases that use SQL relies on a heavily relational model in which you idealize normalized data. The advantage is that this support ad-hoc queries very well. If you can dream up the appropriate SQL-query, you have the answer, though it may take some time before it comes to you, depending on the power of optimization and the complexity of the query. Enter the web. The basic premise is that we do not want users to do ad-hoc queries on data. Partly due to security and partly to (inadvertently) denying other user service. So many systems backed by RDBMS systems artificially lock down the allowed queries to a few blessed. Now enter Google. Google has one main thing they need to serve, which is inverted indexes for words. This is a very specific problem with an interesting property: you can shard the keyspace of words into multiple machines for good distribution and parallelism. The limitation is that you just locked down ad-hoc query to specific queries of specific data, but you can get really good query speed on those. You also got the sharding capability and the system is not hard to implement. Data mining can be achieved by batch-runs of map/reduce over all data. It is in some sense slow, but if your query fits the M/R scheme, it parallelizes easily. In a certain sense, the M/R gave you some of the ad-hoc query capabilities back. The final key concept is that triggers easily work on sharding models. Upon insertion, you run hooks which can in turn update specific query indexes, drive full-text-search engines and so on. Now, most web services out there have modest data storage needs. The amount of services using the RDBMS as a glorified file system is abundant and pervasive to an extent which feels pervertedly sick. Such systems never really had the need for all the niceties of an SQL system and their queries are simple. Also, the mapping from SQL into the language of choice is not easy, especially if said language is object oriented. Next, the war drums begin to play. The CAP theorem is proven and this changes the game. Now whenever you do a database system, there is a tradeoff which must be made. Much misunderstanding of the CAP theorem is out there. We don't care about he halting theorem that much either, even though it is very real and there. But it does pose some limitations to what one can achieve. Personally, I think the K/V stores grew out of the Google train. Google did them because no database system was ready for that kind of scalability. But that should not be taken as if you can never make SQL, QLC or the like support sharded queries. We have more stores of the "NoSQL" kind popping up every month which is great for the innovation. But time will show which of these will actually fly and which will thud to the ground. -- J. From ok@REDACTED Wed Nov 3 02:39:54 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 3 Nov 2010 14:39:54 +1300 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <20101102113116.GA2886@corelatus.se> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> <4CCE714B.3090402@amberbio.com> <20101102113116.GA2886@corelatus.se> Message-ID: <4E9558D8-8EB8-4C2D-8BA2-DA0E68D9A780@cs.otago.ac.nz> On 3/11/2010, at 12:31 AM, Matthias Lang wrote: > The development of low-power devices has been relentless. 32x32bit > single cycle multiplies (assuming "can issue one of those operations > every cycle" qualifies for "single cycle") I wouldn't have said so. From ebegumisa@REDACTED Wed Nov 3 07:30:31 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 03 Nov 2010 17:30:31 +1100 Subject: [erlang-questions] non-trivial supervisor idioms? -- An open-source plea! In-Reply-To: <2830.5139.qm@web111403.mail.gq1.yahoo.com> References: <4CA0AAD3.3030209@erlang-solutions.com> <249668.71494.qm@web111413.mail.gq1.yahoo.com> <716334.78231.qm@web111415.mail.gq1.yahoo.com> <2830.5139.qm@web111403.mail.gq1.yahoo.com> Message-ID: Hi Thomas, Attila, It's true that every language suffers when running larger projects, or rather every language environment suffers (e.g Erlang and it's environment). But they tend to suffer differently. Each language environ develops it's own unique problems and proposed solutions as projects grow in size. Ulf's included applications background story is a case in point. When a language environment is more wide spread, say Java, it's easier for programmers who are migrating from a different environment, say C++/boost, to do research on how to deal with/avoid those problems simply because there is so much open source code (good and bad) and literature out there (good and bad). As someone who has migrated from other environments to Erlang, I've found it harder to do the research, even on something as basic as breaking a program up into smaller chunks. This can get frustrating. More experienced Erlang programmers obviously have their solutions to this, but for some reason it is extremely difficult for relative newcomers to discover tried and tested good practices that won't lead to trouble down the road. Another plea I'd like to make is for more experienced Erlang programmers to blog. Erlang blogasphere seems to have been hijacked by newcomers who are overconfident of their abilities and tend to give questionable advice. The problem is, to other newcomers, they are the voice of Elrang. - Edmond - PS: I've looked at the book preview "OTP in Action", it seems to be addressing this sort of thing. On Wed, 03 Nov 2010 03:18:07 +1100, Thomas Lindgren wrote: > > > ----- Original Message ---- >> From: Attila Rajmund Nohl >> >> 2010/10/28, Thomas Lindgren : >> [...] >> > Well, Erlang was intended to be easy to learn and use, and these >> products >> > have >> > had dozens or even hundreds of programmers working on them over many >> years. >> > It's >> > pretty natural, isn't it? >> >> On the other hand I think this is true for nearly all programming >> languages, the C/Java projects have similar turnabouts. By the way, I >> had a colleague who has working on AXD, but he said he didn't wrote a >> single line of new code, he just copied and pasted some earlier >> written code - and was genuinely surprised that it worked. > > Hi Attila, > > Yes indeed -- it's not a language issue but the reality of running a big > project. > > Best, > Thomas > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ulf.wiger@REDACTED Wed Nov 3 08:00:50 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Wed, 3 Nov 2010 08:00:50 +0100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <20101102201437.GC626@hope.tifa.renegado> References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: <7E29EEB3-F835-4693-8DC7-A55357E148D4@erlang-solutions.com> On 2 Nov 2010, at 21:14, Silas Silva wrote: > * Is there any guide, tutorial, book, whatever, that tries to introduce > NoSQL databases to SQL users? Not really, although the Mnesia User's Guide does try to use some RDBMS terminology. http://www.erlang.org/doc/apps/mnesia/Mnesia_chap2.html#id58308 > * Key-value databases are surprising simple. I know you solve > relationship by denormalizing data. What data should be normalized? > What shouldn't? How do you update denormalized data? The thing to keep in mind is that mnesia was not designed to provide a better way to handle relational data sets. It was originally designed to be a real-time, in-memory database for telecom apps, and most of the data there is well suited to simple primary-key access and few enough relations that they can be hard-coded. The actual updating of data is perhaps not so terribly different, but mnesia will not do much checking for you, so you have to keep track of relations and dependent updates in the code. If this gets to be a great burden in your application, it is probably a sign that you need an RDBMS after all. They really excel in certain application domains. > Sorry for such simple and general questions. Things were simple up to > the moment that I realized that it would be easily solved with a JOIN > SQL statement. :-) QLC offers some support for basic joins. You need to be a bit more careful about how you word the queries than with SQL, since QLC only does some rudimentary optimizations. BR, Ulf W Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From ebegumisa@REDACTED Wed Nov 3 09:30:49 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 03 Nov 2010 19:30:49 +1100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <20101102201437.GC626@hope.tifa.renegado> References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: > * Is there any guide, tutorial, book, whatever, that tries to introduce > NoSQL databases to SQL users? The CouchDB book is very good for making the mental transition from SQL to NoSQL in general.. http://guide.couchdb.org/draft/index.html I think a lot of the data organisation concepts described therein are transferable in principle to Mnesia (JSON --> Records, JavaScript --> Erlang). > * Key-value databases are surprising simple. I know you solve > relationship by denormalizing data. What data should be normalized? > What shouldn't? Ahh, yes, the relationship question! There is no one-size fits all answer to this. There are various NoSQL approaches to relationship modeling with different levels of denormalisation and having different advantages and disadvantages. You might find this article helpful... http://wiki.apache.org/couchdb/EntityRelationship Again. I believe in principle, the techniques described are transferable and actively used by Mnesia developers (I'm not an Mnesia user myself.) > How do you update denormalized data? This normally involves a trade-off between having a separate record that represents related data and updating that once, or completely denormalising and updating all related records (see wiki link above.) I usually only denormalise data for which I'm unlikely to need to update several copies of. For example, for a receipt, I'd have an embedded copy of a customer record. When I update the customer record, I won't update related receipts coz I don't DON'T want the receipts to be updated anyway since a receipt should be an exact unmodified snapshot of history. In contrast, for an order, I'd have a embedded only a reference to a customer record, because if someone changes the customers address, I'd want all related orders to reflect that without needing to map through and change them (though for some other cases I might do just that.)4 It's very case-by-case. Other factors like replication and query performance come in to play too. From my experience, it's something that you kinda figure out as you go changing your mind here and there. - Edmond - On Wed, 03 Nov 2010 07:14:42 +1100, Silas Silva wrote: > This is a message I sent to the nosql-discussion@REDACTED > discussion group. I thought it might be interesting to send it > erlang-questions, so here we go... > > > Hi all! > > I have used SQL RDBMSs for some time. I've never used any very advanced > feature, but I know enough of it to make database applications. > > Nowadays, I decided it would be interesting to learn some NoSQL > databases concepts. So I decided to pick up some Erlang and Mnesia, its > native key-value database. More than scalability itself, the most > valuable feature for me is the possibility of replication and > synchronization between nodes. > > But all pros have cons as companion. The lack of a relationship model > is difficult for who is used to RDBMSs. So, my question is: > > * Is there any guide, tutorial, book, whatever, that tries to introduce > NoSQL databases to SQL users? > > * Key-value databases are surprising simple. I know you solve > relationship by denormalizing data. What data should be normalized? > What shouldn't? How do you update denormalized data? > > Sorry for such simple and general questions. Things were simple up to > the moment that I realized that it would be easily solved with a JOIN > SQL statement. :-) > > Thank you very much! > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From torben.lehoff@REDACTED Wed Nov 3 10:35:13 2010 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Wed, 3 Nov 2010 10:35:13 +0100 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: Hi Edmond, I am a bit confused by your statements since my problem was about documentation of an _existing_ gen_fsm. It might be my ignorance about what UBF can offer that is causing this (read Joe's payer a while ago). As I understand UBF it allows you to specify what a component does so that others can interact with it in a sensible manner and you can figure out if a given component implements a given contract - or am I missing the point here? I actually thought about using UBF, but when I looked at it it seemed a bit to immature for a mission critical system, maybe that have changed by now. Do not get me wrong, I firmly believe in specifying protocols and UBF seems like a very interesting thing to use, but in a real life situation you sometimes have code that you want to extract information about and this was the problem I was faced with. If I understand you right you are no longer using gen_fsm, right? How do you piece together your software then? I use the OTP supervisor structure all the time due to the obvious benefits of a battle tested framework, but I would not mind adding another tool to my tool box!! Cheers, Torben On Wed, Nov 3, 2010 at 10:20, Edmond Begumisa wrote: > Hi, > > I don't know how immediately useful this is (perhaps something to keep in > mind), but one of the attractive understated things about UBF is that you > define all this information in one place for both easy > configuration/protocol-definition and reading/self-documentation... > > http://www.sics.se/~joe/ubf/site/quick.html > https://github.com/norton/ubf > > So far, I haven't regretted switching from gen_fsm to UBF(C) > > - Edmond - > > > > On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson < > tomas.abrahamsson@REDACTED> wrote: > > On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann >> wrote: >> >>> Hi, >>> >>> I want to extract the following information from a gen_fsm: >>> >>> - all the states >>> - the incoming events >>> - possible next states >>> >>> but I want to do it per function clause and not only on the function >>> level >>> as TypEr and Dialyzer does it. >>> >> >> One way to do that is to copy each function clause into a >> separate function. Then one can run TypEr on the expanded >> code. Attached is an escript that does such an expansion >> of exported functions of arity 2 and 3, and also of >> handle_event, handle_sync_event and handle_info. The script >> accepts -I and -D options for include paths and macro >> definitions, just like erlc,and prints the expanded program >> on stdout. >> >> I've tried plotting state machines from the output of TypEr >> when run on the expanded results, and it seems to do reasonably >> well. I haven't done any thorough verification. >> >> It doesn't solve all of your problems: it doesn't find any >> outgoing messages, and it has the same limitations you >> already indicated when returning from a function clause in >> different ways, so I guess the results are roughly what >> you already seems to have, or what Vance's drawing tool >> already produces. >> >> Somehow being able to make more use of TypEr's >> knowledge about a file would be really interesting, >> for example to be able to query it for type information >> given various execution paths through the program, >> if it would be possible. >> >> BRs >> Tomas >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > -- http://www.linkedin.com/in/torbenhoffmann From ebegumisa@REDACTED Wed Nov 3 10:20:35 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 03 Nov 2010 20:20:35 +1100 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: Hi, I don't know how immediately useful this is (perhaps something to keep in mind), but one of the attractive understated things about UBF is that you define all this information in one place for both easy configuration/protocol-definition and reading/self-documentation... http://www.sics.se/~joe/ubf/site/quick.html https://github.com/norton/ubf So far, I haven't regretted switching from gen_fsm to UBF(C) - Edmond - On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson wrote: > On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann > wrote: >> Hi, >> >> I want to extract the following information from a gen_fsm: >> >> - all the states >> - the incoming events >> - possible next states >> >> but I want to do it per function clause and not only on the function >> level >> as TypEr and Dialyzer does it. > > One way to do that is to copy each function clause into a > separate function. Then one can run TypEr on the expanded > code. Attached is an escript that does such an expansion > of exported functions of arity 2 and 3, and also of > handle_event, handle_sync_event and handle_info. The script > accepts -I and -D options for include paths and macro > definitions, just like erlc,and prints the expanded program > on stdout. > > I've tried plotting state machines from the output of TypEr > when run on the expanded results, and it seems to do reasonably > well. I haven't done any thorough verification. > > It doesn't solve all of your problems: it doesn't find any > outgoing messages, and it has the same limitations you > already indicated when returning from a function clause in > different ways, so I guess the results are roughly what > you already seems to have, or what Vance's drawing tool > already produces. > > Somehow being able to make more use of TypEr's > knowledge about a file would be really interesting, > for example to be able to query it for type information > given various execution paths through the program, > if it would be possible. > > BRs > Tomas -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Wed Nov 3 12:32:57 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 03 Nov 2010 22:32:57 +1100 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: On Wed, 03 Nov 2010 20:35:13 +1100, Torben Hoffmann wrote: > Hi Edmond, > > I am a bit confused by your statements since my problem was about > documentation of an _existing_ gen_fsm. > It might be my ignorance about what UBF can offer that is causing this > (read > Joe's payer a while ago). > Sorry, I probably should have first asked what you're using gen_fsm for before commenting :) My comment would only helpful *if* you are using gen_fsm for socket IPC between two Erlang nodes or between an Erlang node and a non-Erlang client. In which case I was suggesting, as a point of information, that you consider using UBF for this in the *future* because the Erlang UBF contract checker's state-machine provides the exact information you are looking for (there's no need to go digging for it.) > As I understand UBF it allows you to specify what a component does so > that > others can interact with it in a sensible manner and you can figure out > if a > given component implements a given contract - or am I missing the point > here? > Yes. But along with it comes a state machine that you can use in situations where you'd normally use gen_fsm to "regulate" socket-based IPC (ie: the UBF contract *is* a set of states and what events/transitions can occur). I asked about this on this list recently... http://groups.google.com/group/erlang-programming/browse_thread/thread/017db73ba5fd15d9/169c3a3afe855049?#169c3a3afe855049 > I actually thought about using UBF, but when I looked at it it seemed a > bit > to immature for a mission critical system, maybe that have changed by > now. > Joseph Norton/Gemini Mobile have very robust UBF implementations (2nd link in previous e-mail is one of many great open-source implementations they've released.) These are well tested and (I understand) used in mission critical systems. Joseph would be in a better position to describe to you how they are using them. I'm using two of them -- the standard UBF for Erlang-to-Erlang IPC and the UBF-JSON for Erlang-to-JavaScript IPC -- I haven't had any problems so far. > Do not get me wrong, I firmly believe in specifying protocols and UBF > seems > like a very interesting thing to use, but in a real life situation you > sometimes have code that you want to extract information about and this > was > the problem I was faced with. > This doesn't solve your current problem, but for future reference, I thought it might be good to take note that the UBF(C) state machine handles the information you're looking to extract very nicely so you don't have the problem to begin with. > If I understand you right you are no longer using gen_fsm, right? > How do you piece together your software then? I was using gen_fsm only for socket IPC, but faced the exact problem you describe -- it was too difficult to keep track/document possible state transitions, events, etc. Though my initial attraction to UBF was mainly for other reasons (compactness and debuggability on the wire being the major ones as my apps need to be able to go mobile), I was pleasantly surprised how the UBF(C) contract checker managed state transitions and events more declaratively and thus more clearly than gen_fsm. I was initially tempted to use the UBF wire protocol (UBF-A) together with gen_fsm and discard the state-machine provided by the UBF contract checker (UBF-C) under the mistaken belief that gen_fsm was somehow more robust, but Joe Armstrong quickly pointed out that there's nothing the gen_fsm can do that a well-implemented UBF(C) contract checker can't in this regard. > I use the OTP supervisor structure all the time due to the obvious > benefits > of a battle tested framework, but I would not mind adding another tool > to my > tool box!! > Norton/Gemini UBF implementations are pretty solid and you can plug things into your supervision tree. For socket-based IPC, the UBF(C) contract checker gives you far far more that gen_fsm can simply because, as Joe pointed out to me, it is specialised for that sort of thing while gen_fsm is generic. You won't be missing anything in terms of robustness and reliability, if anything you increase these because states transitions and events are defined nicely in one place and everything is easier to debug. Regards. - Edmond - > Cheers, > Torben > > On Wed, Nov 3, 2010 at 10:20, Edmond Begumisa > wrote: > >> Hi, >> >> I don't know how immediately useful this is (perhaps something to keep >> in >> mind), but one of the attractive understated things about UBF is that >> you >> define all this information in one place for both easy >> configuration/protocol-definition and reading/self-documentation... >> >> http://www.sics.se/~joe/ubf/site/quick.html >> https://github.com/norton/ubf >> >> So far, I haven't regretted switching from gen_fsm to UBF(C) >> >> - Edmond - >> >> >> >> On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson < >> tomas.abrahamsson@REDACTED> wrote: >> >> On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann >> >>> wrote: >>> >>>> Hi, >>>> >>>> I want to extract the following information from a gen_fsm: >>>> >>>> - all the states >>>> - the incoming events >>>> - possible next states >>>> >>>> but I want to do it per function clause and not only on the function >>>> level >>>> as TypEr and Dialyzer does it. >>>> >>> >>> One way to do that is to copy each function clause into a >>> separate function. Then one can run TypEr on the expanded >>> code. Attached is an escript that does such an expansion >>> of exported functions of arity 2 and 3, and also of >>> handle_event, handle_sync_event and handle_info. The script >>> accepts -I and -D options for include paths and macro >>> definitions, just like erlc,and prints the expanded program >>> on stdout. >>> >>> I've tried plotting state machines from the output of TypEr >>> when run on the expanded results, and it seems to do reasonably >>> well. I haven't done any thorough verification. >>> >>> It doesn't solve all of your problems: it doesn't find any >>> outgoing messages, and it has the same limitations you >>> already indicated when returning from a function clause in >>> different ways, so I guess the results are roughly what >>> you already seems to have, or what Vance's drawing tool >>> already produces. >>> >>> Somehow being able to make more use of TypEr's >>> knowledge about a file would be really interesting, >>> for example to be able to query it for type information >>> given various execution paths through the program, >>> if it would be possible. >>> >>> BRs >>> Tomas >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> > > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From mevans@REDACTED Wed Nov 3 12:47:12 2010 From: mevans@REDACTED (Evans, Matthew) Date: Wed, 3 Nov 2010 07:47:12 -0400 Subject: [erlang-questions] RE: DETS Problems In-Reply-To: References: , Message-ID: Somehow adding {repair,force} as an option fixes this problem. I used dets:all() and it does correctly close all dets tables too. Weird ________________________________________ From: erlang-questions@REDACTED [erlang-questions@REDACTED] On Behalf Of Evans, Matthew [mevans@REDACTED] Sent: Tuesday, November 02, 2010 3:53 PM To: Ahmed Omar Cc: erlang-questions Subject: RE: [erlang-questions] RE: DETS Problems It was actually worse than I thought. It seems (initial glance) that DETS doesn't like the large elements I was inserting. The table had 3 records of 3-5K in size, and about 700 of 300 bytes in size. It was constantly failing to find the large elements, if I looked up a small element first. Moving the large elements to a separate table seems to fix the problem. I'm going to do more analysis of this, we also use SSD/flash as the storage device, maybe that is causing problems. ________________________________ From: Ahmed Omar [mailto:spawn.think@REDACTED] Sent: Tuesday, November 02, 2010 3:46 PM To: Evans, Matthew Cc: erlang-questions Subject: Re: [erlang-questions] RE: DETS Problems Sorry to understand more, can you provide the options you are using for the table? what do u do with ets:to_dets/2 exactly? please also note that : " The Dets table is emptied before the objects are inserted" http://www.erlang.org/doc/man/ets.html#to_dets-2 On Tue, Nov 2, 2010 at 7:25 PM, rgowka1 > wrote: "Dets tables must be opened before they can be updated or read, and when finished they must be properly closed. If a table has not been properly closed, Dets will automatically repair the table. This can take a substantial time if the table is large. A Dets table is closed when the process which opened the table terminates. If several Erlang processes (users) open the same Dets table, they will share the table. The table is properly closed when all users have either terminated or closed the table. Dets tables are not properly closed if the Erlang runtime system is terminated abnormally." I did play around with dets with the same sized records as you. According to the above blurb and my observations, the corruption is usually not closing the dets in all the processes.. On Tue, Nov 2, 2010 at 2:10 PM, Evans, Matthew > wrote: > I should add, that I do ets:insert/2 followed by a ets:to_dets/2. > > Some of the records are about 4K in size. > > -----Original Message----- > From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Evans, Matthew > Sent: Tuesday, November 02, 2010 1:48 PM > To: erlang-questions > Subject: [erlang-questions] DETS Problems > Importance: High > > List, > > I'm using a dets table to save reasonably large records (1K or there abouts). > > The following sequence is applied to a table: > > dets:open_file/2/ > dets:insert/2 (a few times). > ets:to_dets/2 > dets:close/1 > > > Sometime later I reopen the table with dets:open_file/1 > > The table is then corrupted. Reads produce the following error: > > {error,{{bad_object,eval_work_list}, > "/content/c2idx/5/95/10b12417-4629-404a-8361-68bd3de77d90.dat"}} > > What causes that error? > > Thanks > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From zabrane3@REDACTED Wed Nov 3 14:00:16 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Wed, 3 Nov 2010 14:00:16 +0100 Subject: [erlang-questions] SMP in emacs? In-Reply-To: References: <1288274732.2975.54.camel@cyclone> <20101028184852.GA33595@alumni.caltech.edu> <1288470480.15939.8.camel@blackrain> <1288473799.15939.10.camel@blackrain> <1288474817.15939.16.camel@blackrain> <1288532917.6645.0.camel@blackrain> <26305_1288567064_4CCDF917_26305_176175_1_AANLkTik9WEdL7HYyHvKBTbW4tL3TuuLJzgasM2zjsWGL@mail.gmail.com> <030CADE9-8103-40FE-A450-A042611CB940@unb.ca> Message-ID: Hi Dan & all, Off topic question. Could someone please explain me what's the benefits of this "+S" option (and the "+A" option by the way). How to choose their values? -- Regards Zabrane 2010/11/2 Dan Gudmundsson : > C-u M-x erlang-shell > erl -smp +S12 > >> erlang:system_info(schedulers). > 12 From andrewmmc@REDACTED Wed Nov 3 14:49:58 2010 From: andrewmmc@REDACTED (andrew mmc) Date: Wed, 3 Nov 2010 14:49:58 +0100 Subject: R14B compile failure on Ubuntu 9.04 Message-ID: Hi there, I'm still having problems identifying what is the problem with compilation on Ubuntu Jaunty: pthread/ethr_event.c:92: Fatal error in wait__(): Function not implemented (38) make[3]: *** [../ebin/hipe_rtl.beam] Aborted make[3]: Leaving directory `/usr/src/otp_src_R14B/lib/hipe/rtl' make[2]: *** [opt] Error 2 make[2]: Leaving directory `/usr/src/otp_src_R14B/lib/hipe' make[1]: *** [opt] Error 2 make[1]: Leaving directory `/usr/src/otp_src_R14B/lib' make: *** [secondary_bootstrap_build] Error 2 the function not implemented is ETHR_FATAL_ERROR__() What could be causing this error, does anyone know how I may fix it? Thankyou, Andrew From alessandro.sivieri@REDACTED Wed Nov 3 15:35:28 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Wed, 3 Nov 2010 15:35:28 +0100 Subject: [erlang-questions] R14B compile failure on Ubuntu 9.04 In-Reply-To: References: Message-ID: 2010/11/3 andrew mmc > I'm still having problems identifying what is the problem with compilation > on Ubuntu Jaunty: > > Mmh, maybe R14B requires some new version of one of the dependencies; is there somewhere an updated list of the dependencies with their versions? -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From mikpe@REDACTED Wed Nov 3 15:38:27 2010 From: mikpe@REDACTED (Mikael Pettersson) Date: Wed, 3 Nov 2010 15:38:27 +0100 Subject: [erlang-questions] R14B compile failure on Ubuntu 9.04 In-Reply-To: References: Message-ID: <19665.29667.899068.581153@pilspetsen.it.uu.se> andrew mmc writes: > Hi there, > > I'm still having problems identifying what is the problem with compilation > on Ubuntu Jaunty: > > pthread/ethr_event.c:92: Fatal error in wait__(): Function not implemented > (38) ... > What could be causing this error, does anyone know how I may fix it? The futex syscall at erts/lib_src/pthread/ethr_event.c:92 failed with ENOSYS. That's not expected, and may be a kernel configuration error or an error in the kernel headers (esp. unistd.h). If it has been a case of the Erlang VM giving it bad parameters it would have failed with EINVAL or EFAULT instead. Bug your distro about it, or try Fedora instead. From matthias@REDACTED Wed Nov 3 15:49:36 2010 From: matthias@REDACTED (Matthias Lang) Date: Wed, 3 Nov 2010 15:49:36 +0100 Subject: [erlang-questions] Erlang arithmetics In-Reply-To: <4E9558D8-8EB8-4C2D-8BA2-DA0E68D9A780@cs.otago.ac.nz> References: <4CCBDCBA.5030602@amberbio.com> <4CCBEB85.8050209@amberbio.com> <4EC3181B-C2E7-4CDD-9273-181CF0DB2F2B@cs.otago.ac.nz> <4CCE714B.3090402@amberbio.com> <20101102113116.GA2886@corelatus.se> <4E9558D8-8EB8-4C2D-8BA2-DA0E68D9A780@cs.otago.ac.nz> Message-ID: <20101103144936.GA12776@corelatus.se> On Wednesday, November 03, Richard O'Keefe wrote: > On 3/11/2010, at 12:31 AM, Matthias Lang wrote: > > The development of low-power devices has been relentless. 32x32bit > > single cycle multiplies (assuming "can issue one of those operations > > every cycle" qualifies for "single cycle") > I wouldn't have said so. Can you be more specific? What would qualify as "single cycle"? I don't know my way around Intel/AMD documentation. An example of such an instruction on a desktop CPU would be interesting. Matt From pascalchapier@REDACTED Wed Nov 3 16:09:10 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Wed, 3 Nov 2010 16:09:10 +0100 Subject: How to use CSS with mochiweb? Message-ID: Hello, I am using mochiweb for a simple "home" web server, and I want to create nice pages using css. The only way I succeed to do this is including the css code between style markers into the html response. I am not able to create a link to a local file on my server machine. I thought that I have understood why when I saw the response to a click on the css file link in the source code (firefox feature): -> unknown request /mycss.css. So I add a match to file access in the dispatch loop of the mochiweb serve. the link test is OK: if I try to follow the link to the css file, I get the right content. But it still does not work. The page is displayed with default style values. Do you know where I can get some information on how this is supposed to work? Thank you, Pascal. From alessandro.sivieri@REDACTED Wed Nov 3 16:17:12 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Wed, 3 Nov 2010 16:17:12 +0100 Subject: [erlang-questions] How to use CSS with mochiweb? In-Reply-To: References: Message-ID: 2010/11/3 Pascal Chapier > Do you know where I can get some information on how this is supposed to > work? > > I'm already using both css and javascript with mochiweb, and without modifications to the web server module: if you place all the files under priv/www, where you should have the html pages, and then link them in your html as usual, i.e. then mochiweb will serve them as requested; if it doesn't, then I may assume that there is something wrong in your web server module, but the autogenerated version works this way... -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From pascalchapier@REDACTED Wed Nov 3 17:25:39 2010 From: pascalchapier@REDACTED (Pascal Chapier) Date: Wed, 3 Nov 2010 17:25:39 +0100 Subject: [erlang-questions] How to use CSS with mochiweb? Message-ID: Well Alessandro, I must have done something very weird. I have no html files and no place where to put them. I buid them "on the fly", when I receive the request, depending on the fields values in each request. I launch mochiweb with the following code (found in some forgotten forum): init([Port]) -> mochiweb_http:start([{port, Port}, {loop, fun(Req) -> dispatch(Req) end}]), {ok, []}. and I decode the requests (get or post) in the dispatch function and its helpers, for example: answer("/test", Req) -> Req:respond({200, [{"Content-Type", "text/html"}], page_build:test()}); :o) While I write this I found my mistake: if i am requested a css file the content-type is text/css. In the mean time I tried different places where to put priv/www, but none of them works, I guess that this come from the init function... Pascal From alexander.harju@REDACTED Wed Nov 3 17:57:52 2010 From: alexander.harju@REDACTED (Alexander Harju) Date: Wed, 3 Nov 2010 17:57:52 +0100 Subject: [erlang-questions] Browser based bittorrent application in Erlang? In-Reply-To: References: Message-ID: No I wouldn't think so, unless you write the browser in Erlang. // Alex On Wed, Oct 27, 2010 at 5:40 PM, rgowka1 wrote: > Hi - > > What would you use to develop a Browser based bittorrent application > in Erlang. Something along the lines of http://www.bitlet.org. > > I am guessing that, Bitlet uses a Java Applet running in the browser > to send and receive data. What would be a more suited technology to > use with Erlang? Is there something in Erlang that could be used to > run a small VM in the browser? > > thanks, > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ebegumisa@REDACTED Wed Nov 3 17:58:12 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 04 Nov 2010 03:58:12 +1100 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: Here's an example of what I mean. Here is a UBF(B) contract copied from Joe's UBF page (http://www.sics.se/~joe/ubf/site/quick.html)... +NAME("file_server"). +VSN("ubf1.0"). +TYPES info() = info; description() = description; services() = services; contract() = contract; file() = string(); ls() = ls; files() = {files, [file()]}; getFile() = {get, file()}; noSuchFile() = noSuchFile. +STATE start ls() => files() & start; getFile() => binary() & start | noSuchFile() & stop. +ANYSTATE info() => string(); description() => string(); contract() => term(). (you can find the corresponding state machine implementation is http://www.sics.se/~joe/ubf/site/eserve.html) ... you could achieve the exact same thing with gen_fsm combined with TypEr/Dialyzer (I was doing just that), but the advantage to using the UBF(B) contract instead is this info is declared in one place, as you design and code away, then verified at runtime by the UBF(C) contract checker. I find this approach more declarative and self-documenting. IMO, state machines written against a UBF contract are much easier to write, maintain, debug, document and modify than those written with gen_fsm. Of course, to make use of this, you have to migrate to UBF(A) encoding for the actual messages between processes. This usually only makes sense if you're crossing OS process boundaries, and probably too costly for communication between Erlang processes in the same VM instance. Actually, there's an idea: maybe someone should write a contract checker for gen_fsm that uses ordinary Erlang-terms rather than UBF(A) encoding. This could then be used in the same VM instance without much cost and make working with gen_fsm more dependable by allowing you to declare how the state-machine is supposed to behave and cross check that against how it's actually behaving -- a major frustration I've had when using gen_fsm. - Edmond - On Wed, 03 Nov 2010 20:35:13 +1100, Torben Hoffmann wrote: > Hi Edmond, > > I am a bit confused by your statements since my problem was about > documentation of an _existing_ gen_fsm. > It might be my ignorance about what UBF can offer that is causing this > (read > Joe's payer a while ago). > > As I understand UBF it allows you to specify what a component does so > that > others can interact with it in a sensible manner and you can figure out > if a > given component implements a given contract - or am I missing the point > here? > > I actually thought about using UBF, but when I looked at it it seemed a > bit > to immature for a mission critical system, maybe that have changed by > now. > > Do not get me wrong, I firmly believe in specifying protocols and UBF > seems > like a very interesting thing to use, but in a real life situation you > sometimes have code that you want to extract information about and this > was > the problem I was faced with. > > If I understand you right you are no longer using gen_fsm, right? > How do you piece together your software then? > I use the OTP supervisor structure all the time due to the obvious > benefits > of a battle tested framework, but I would not mind adding another tool > to my > tool box!! > > Cheers, > Torben > > On Wed, Nov 3, 2010 at 10:20, Edmond Begumisa > wrote: > >> Hi, >> >> I don't know how immediately useful this is (perhaps something to keep >> in >> mind), but one of the attractive understated things about UBF is that >> you >> define all this information in one place for both easy >> configuration/protocol-definition and reading/self-documentation... >> >> http://www.sics.se/~joe/ubf/site/quick.html >> https://github.com/norton/ubf >> >> So far, I haven't regretted switching from gen_fsm to UBF(C) >> >> - Edmond - >> >> >> >> On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson < >> tomas.abrahamsson@REDACTED> wrote: >> >> On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann >> >>> wrote: >>> >>>> Hi, >>>> >>>> I want to extract the following information from a gen_fsm: >>>> >>>> - all the states >>>> - the incoming events >>>> - possible next states >>>> >>>> but I want to do it per function clause and not only on the function >>>> level >>>> as TypEr and Dialyzer does it. >>>> >>> >>> One way to do that is to copy each function clause into a >>> separate function. Then one can run TypEr on the expanded >>> code. Attached is an escript that does such an expansion >>> of exported functions of arity 2 and 3, and also of >>> handle_event, handle_sync_event and handle_info. The script >>> accepts -I and -D options for include paths and macro >>> definitions, just like erlc,and prints the expanded program >>> on stdout. >>> >>> I've tried plotting state machines from the output of TypEr >>> when run on the expanded results, and it seems to do reasonably >>> well. I haven't done any thorough verification. >>> >>> It doesn't solve all of your problems: it doesn't find any >>> outgoing messages, and it has the same limitations you >>> already indicated when returning from a function clause in >>> different ways, so I guess the results are roughly what >>> you already seems to have, or what Vance's drawing tool >>> already produces. >>> >>> Somehow being able to make more use of TypEr's >>> knowledge about a file would be really interesting, >>> for example to be able to query it for type information >>> given various execution paths through the program, >>> if it would be possible. >>> >>> BRs >>> Tomas >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> > > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From michal.dobrogost@REDACTED Wed Nov 3 18:24:23 2010 From: michal.dobrogost@REDACTED (Michal D.) Date: Wed, 3 Nov 2010 13:24:23 -0400 Subject: Passing functions as arguments requires export? Message-ID: Hi All, I'm having some problems working with functions in Erlang. Specifically, passing functions as arguments. Given the following module: -module(test). -export([res/0]). res() -> lists:map({test,incr}, [0,1,2]). incr(X) -> X + 1. I stumbled upon the syntax {test,incr} to pass a (not anonymous) function. When I go to run this code in the emulator, I get an error that doesn't get resolved unless I add "incr/1" to the export list. However, there's no reason to export the function because I don't ever want to call it directly. What am I missing? > c(test). > test:res(). ** exception error: undefined function test:incr/1 in function lists:map/2 > Cheers, Michal From kostis@REDACTED Wed Nov 3 19:25:52 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Wed, 03 Nov 2010 20:25:52 +0200 Subject: [erlang-questions] Passing functions as arguments requires export? In-Reply-To: References: Message-ID: <4CD1A930.8010505@cs.ntua.gr> Michal D. wrote: > Hi All, > > I'm having some problems working with functions in Erlang. Specifically, > passing functions as arguments. Given the following module: > > -module(test). > -export([res/0]). > > res() -> lists:map({test,incr}, [0,1,2]). > > incr(X) -> X + 1. > > I stumbled upon the syntax {test,incr} to pass a (not anonymous) function. > When I go to run this code in the emulator, I get an error that doesn't get > resolved unless I add "incr/1" to the export list. However, there's no > reason to export the function because I don't ever want to call it > directly. What am I missing? You are using a really obsolete language feature that should have been completely eliminated by now. There is absolutely no reason for it to still exist in the language. Use the more modern (and more fun ;-) construct instead: res() -> lists:map(fun incr/1, [0,1,2]). Kostis From kennethstone@REDACTED Wed Nov 3 18:35:31 2010 From: kennethstone@REDACTED (Kenny Stone) Date: Wed, 3 Nov 2010 12:35:31 -0500 Subject: [erlang-questions] Passing functions as arguments requires export? In-Reply-To: References: Message-ID: 'incr' will be called from the lists module, not the test module, so it cannot access it without being exported. This is one way to do what you want: res() -> lists:map(fun incr/1, [0,1,2]). Kenny On Wed, Nov 3, 2010 at 12:24 PM, Michal D. wrote: > Hi All, > > I'm having some problems working with functions in Erlang. Specifically, > passing functions as arguments. Given the following module: > > -module(test). > -export([res/0]). > > res() -> lists:map({test,incr}, [0,1,2]). > > incr(X) -> X + 1. > > I stumbled upon the syntax {test,incr} to pass a (not anonymous) function. > When I go to run this code in the emulator, I get an error that doesn't get > resolved unless I add "incr/1" to the export list. However, there's no > reason to export the function because I don't ever want to call it > directly. What am I missing? > > > c(test). > > test:res(). > ** exception error: undefined function test:incr/1 > in function lists:map/2 > > > > Cheers, > > Michal > From michal.dobrogost@REDACTED Wed Nov 3 18:47:44 2010 From: michal.dobrogost@REDACTED (Michal D.) Date: Wed, 3 Nov 2010 13:47:44 -0400 Subject: [erlang-questions] Passing functions as arguments requires export? In-Reply-To: <4CD1A930.8010505@cs.ntua.gr> References: <4CD1A930.8010505@cs.ntua.gr> Message-ID: Ah, thank you! This way doesn't require the incr/1 to be exported, as expected (at least by me =)). Michal On Wed, Nov 3, 2010 at 2:25 PM, Kostis Sagonas wrote: > Michal D. wrote: > >> Hi All, >> >> I'm having some problems working with functions in Erlang. Specifically, >> passing functions as arguments. Given the following module: >> >> -module(test). >> -export([res/0]). >> >> res() -> lists:map({test,incr}, [0,1,2]). >> >> incr(X) -> X + 1. >> >> I stumbled upon the syntax {test,incr} to pass a (not anonymous) function. >> When I go to run this code in the emulator, I get an error that doesn't >> get >> resolved unless I add "incr/1" to the export list. However, there's no >> reason to export the function because I don't ever want to call it >> directly. What am I missing? >> > > You are using a really obsolete language feature that should have been > completely eliminated by now. There is absolutely no reason for it to still > exist in the language. > > Use the more modern (and more fun ;-) construct instead: > > res() -> lists:map(fun incr/1, [0,1,2]). > > Kostis > From ebegumisa@REDACTED Wed Nov 3 18:50:15 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 04 Nov 2010 04:50:15 +1100 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: This is looking good, and useful. Thanks. - Edmond - On Thu, 04 Nov 2010 04:09:59 +1100, Joseph Wayne Norton wrote: > > Edmond - > > FYI. There is a new UBF user's guide currently being drafted on github. > > http://norton.github.com/ubf/ubf-user-guide.en.html > > There are still (well mostly all) quite a few empty sections that > haven't been started yet. Nevertheless, an ABNF definition of UBF(A), > UBF(B), and UBF(C) is in the appendix. The ABNF definition is complete > and accurate with the github repository. > > thanks, > > > On Thu, 04 Nov 2010 01:58:12 +0900, Edmond Begumisa > wrote: > >> Here's an example of what I mean. Here is a UBF(B) contract copied from >> Joe's UBF page (http://www.sics.se/~joe/ubf/site/quick.html)... >> >> +NAME("file_server"). >> >> +VSN("ubf1.0"). >> >> +TYPES >> >> info() = info; >> description() = description; >> services() = services; >> contract() = contract; >> >> file() = string(); >> ls() = ls; >> files() = {files, [file()]}; >> getFile() = {get, file()}; >> noSuchFile() = noSuchFile. >> >> >> +STATE start >> ls() => files() & start; >> getFile() => binary() & start >> | noSuchFile() & stop. >> >> >> +ANYSTATE >> info() => string(); >> description() => string(); >> contract() => term(). >> >> (you can find the corresponding state machine implementation is >> http://www.sics.se/~joe/ubf/site/eserve.html) >> >> ... you could achieve the exact same thing with gen_fsm combined with >> TypEr/Dialyzer (I was doing just that), but the advantage to using the >> UBF(B) contract instead is this info is declared in one place, as you >> design and code away, then verified at runtime by the UBF(C) contract >> checker. I find this approach more declarative and self-documenting. >> IMO, state machines written against a UBF contract are much easier to >> write, maintain, debug, document and modify than those written with >> gen_fsm. >> >> Of course, to make use of this, you have to migrate to UBF(A) encoding >> for the actual messages between processes. This usually only makes >> sense if you're crossing OS process boundaries, and probably too costly >> for communication between Erlang processes in the same VM instance. >> >> Actually, there's an idea: maybe someone should write a contract >> checker for gen_fsm that uses ordinary Erlang-terms rather than UBF(A) >> encoding. This could then be used in the same VM instance without much >> cost and make working with gen_fsm more dependable by allowing you to >> declare how the state-machine is supposed to behave and cross check >> that against how it's actually behaving -- a major frustration I've had >> when using gen_fsm. >> >> - Edmond - >> >> >> On Wed, 03 Nov 2010 20:35:13 +1100, Torben Hoffmann >> wrote: >> >>> Hi Edmond, >>> >>> I am a bit confused by your statements since my problem was about >>> documentation of an _existing_ gen_fsm. >>> It might be my ignorance about what UBF can offer that is causing this >>> (read >>> Joe's payer a while ago). >>> >>> As I understand UBF it allows you to specify what a component does so >>> that >>> others can interact with it in a sensible manner and you can figure >>> out if a >>> given component implements a given contract - or am I missing the point >>> here? >>> >>> I actually thought about using UBF, but when I looked at it it seemed >>> a bit >>> to immature for a mission critical system, maybe that have changed by >>> now. >>> >>> Do not get me wrong, I firmly believe in specifying protocols and UBF >>> seems >>> like a very interesting thing to use, but in a real life situation you >>> sometimes have code that you want to extract information about and >>> this was >>> the problem I was faced with. >>> >>> If I understand you right you are no longer using gen_fsm, right? >>> How do you piece together your software then? >>> I use the OTP supervisor structure all the time due to the obvious >>> benefits >>> of a battle tested framework, but I would not mind adding another tool >>> to my >>> tool box!! >>> >>> Cheers, >>> Torben >>> >>> On Wed, Nov 3, 2010 at 10:20, Edmond Begumisa >>> wrote: >>> >>>> Hi, >>>> >>>> I don't know how immediately useful this is (perhaps something to >>>> keep in >>>> mind), but one of the attractive understated things about UBF is that >>>> you >>>> define all this information in one place for both easy >>>> configuration/protocol-definition and reading/self-documentation... >>>> >>>> http://www.sics.se/~joe/ubf/site/quick.html >>>> https://github.com/norton/ubf >>>> >>>> So far, I haven't regretted switching from gen_fsm to UBF(C) >>>> >>>> - Edmond - >>>> >>>> >>>> >>>> On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson < >>>> tomas.abrahamsson@REDACTED> wrote: >>>> >>>> On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann >>>> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I want to extract the following information from a gen_fsm: >>>>>> >>>>>> - all the states >>>>>> - the incoming events >>>>>> - possible next states >>>>>> >>>>>> but I want to do it per function clause and not only on the function >>>>>> level >>>>>> as TypEr and Dialyzer does it. >>>>>> >>>>> >>>>> One way to do that is to copy each function clause into a >>>>> separate function. Then one can run TypEr on the expanded >>>>> code. Attached is an escript that does such an expansion >>>>> of exported functions of arity 2 and 3, and also of >>>>> handle_event, handle_sync_event and handle_info. The script >>>>> accepts -I and -D options for include paths and macro >>>>> definitions, just like erlc,and prints the expanded program >>>>> on stdout. >>>>> >>>>> I've tried plotting state machines from the output of TypEr >>>>> when run on the expanded results, and it seems to do reasonably >>>>> well. I haven't done any thorough verification. >>>>> >>>>> It doesn't solve all of your problems: it doesn't find any >>>>> outgoing messages, and it has the same limitations you >>>>> already indicated when returning from a function clause in >>>>> different ways, so I guess the results are roughly what >>>>> you already seems to have, or what Vance's drawing tool >>>>> already produces. >>>>> >>>>> Somehow being able to make more use of TypEr's >>>>> knowledge about a file would be really interesting, >>>>> for example to be able to query it for type information >>>>> given various execution paths through the program, >>>>> if it would be possible. >>>>> >>>>> BRs >>>>> Tomas >>>>> >>>> >>>> >>>> -- >>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >>>> >>> >>> >>> >> >> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From norton@REDACTED Wed Nov 3 18:09:59 2010 From: norton@REDACTED (Joseph Wayne Norton) Date: Thu, 04 Nov 2010 02:09:59 +0900 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: Edmond - FYI. There is a new UBF user's guide currently being drafted on github. http://norton.github.com/ubf/ubf-user-guide.en.html There are still (well mostly all) quite a few empty sections that haven't been started yet. Nevertheless, an ABNF definition of UBF(A), UBF(B), and UBF(C) is in the appendix. The ABNF definition is complete and accurate with the github repository. thanks, On Thu, 04 Nov 2010 01:58:12 +0900, Edmond Begumisa wrote: > Here's an example of what I mean. Here is a UBF(B) contract copied from > Joe's UBF page (http://www.sics.se/~joe/ubf/site/quick.html)... > > +NAME("file_server"). > > +VSN("ubf1.0"). > > +TYPES > > info() = info; > description() = description; > services() = services; > contract() = contract; > > file() = string(); > ls() = ls; > files() = {files, [file()]}; > getFile() = {get, file()}; > noSuchFile() = noSuchFile. > > > +STATE start > ls() => files() & start; > getFile() => binary() & start > | noSuchFile() & stop. > > > +ANYSTATE > info() => string(); > description() => string(); > contract() => term(). > > (you can find the corresponding state machine implementation is > http://www.sics.se/~joe/ubf/site/eserve.html) > > ... you could achieve the exact same thing with gen_fsm combined with > TypEr/Dialyzer (I was doing just that), but the advantage to using the > UBF(B) contract instead is this info is declared in one place, as you > design and code away, then verified at runtime by the UBF(C) contract > checker. I find this approach more declarative and self-documenting. > IMO, state machines written against a UBF contract are much easier to > write, maintain, debug, document and modify than those written with > gen_fsm. > > Of course, to make use of this, you have to migrate to UBF(A) encoding > for the actual messages between processes. This usually only makes sense > if you're crossing OS process boundaries, and probably too costly for > communication between Erlang processes in the same VM instance. > > Actually, there's an idea: maybe someone should write a contract checker > for gen_fsm that uses ordinary Erlang-terms rather than UBF(A) encoding. > This could then be used in the same VM instance without much cost and > make working with gen_fsm more dependable by allowing you to declare how > the state-machine is supposed to behave and cross check that against how > it's actually behaving -- a major frustration I've had when using > gen_fsm. > > - Edmond - > > > On Wed, 03 Nov 2010 20:35:13 +1100, Torben Hoffmann > wrote: > >> Hi Edmond, >> >> I am a bit confused by your statements since my problem was about >> documentation of an _existing_ gen_fsm. >> It might be my ignorance about what UBF can offer that is causing this >> (read >> Joe's payer a while ago). >> >> As I understand UBF it allows you to specify what a component does so >> that >> others can interact with it in a sensible manner and you can figure out >> if a >> given component implements a given contract - or am I missing the point >> here? >> >> I actually thought about using UBF, but when I looked at it it seemed a >> bit >> to immature for a mission critical system, maybe that have changed by >> now. >> >> Do not get me wrong, I firmly believe in specifying protocols and UBF >> seems >> like a very interesting thing to use, but in a real life situation you >> sometimes have code that you want to extract information about and this >> was >> the problem I was faced with. >> >> If I understand you right you are no longer using gen_fsm, right? >> How do you piece together your software then? >> I use the OTP supervisor structure all the time due to the obvious >> benefits >> of a battle tested framework, but I would not mind adding another tool >> to my >> tool box!! >> >> Cheers, >> Torben >> >> On Wed, Nov 3, 2010 at 10:20, Edmond Begumisa >> wrote: >> >>> Hi, >>> >>> I don't know how immediately useful this is (perhaps something to keep >>> in >>> mind), but one of the attractive understated things about UBF is that >>> you >>> define all this information in one place for both easy >>> configuration/protocol-definition and reading/self-documentation... >>> >>> http://www.sics.se/~joe/ubf/site/quick.html >>> https://github.com/norton/ubf >>> >>> So far, I haven't regretted switching from gen_fsm to UBF(C) >>> >>> - Edmond - >>> >>> >>> >>> On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson < >>> tomas.abrahamsson@REDACTED> wrote: >>> >>> On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann >>> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I want to extract the following information from a gen_fsm: >>>>> >>>>> - all the states >>>>> - the incoming events >>>>> - possible next states >>>>> >>>>> but I want to do it per function clause and not only on the function >>>>> level >>>>> as TypEr and Dialyzer does it. >>>>> >>>> >>>> One way to do that is to copy each function clause into a >>>> separate function. Then one can run TypEr on the expanded >>>> code. Attached is an escript that does such an expansion >>>> of exported functions of arity 2 and 3, and also of >>>> handle_event, handle_sync_event and handle_info. The script >>>> accepts -I and -D options for include paths and macro >>>> definitions, just like erlc,and prints the expanded program >>>> on stdout. >>>> >>>> I've tried plotting state machines from the output of TypEr >>>> when run on the expanded results, and it seems to do reasonably >>>> well. I haven't done any thorough verification. >>>> >>>> It doesn't solve all of your problems: it doesn't find any >>>> outgoing messages, and it has the same limitations you >>>> already indicated when returning from a function clause in >>>> different ways, so I guess the results are roughly what >>>> you already seems to have, or what Vance's drawing tool >>>> already produces. >>>> >>>> Somehow being able to make more use of TypEr's >>>> knowledge about a file would be really interesting, >>>> for example to be able to query it for type information >>>> given various execution paths through the program, >>>> if it would be possible. >>>> >>>> BRs >>>> Tomas >>>> >>> >>> >>> -- >>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >>> >> >> >> > > -- norton@REDACTED From ebegumisa@REDACTED Wed Nov 3 20:54:30 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 04 Nov 2010 06:54:30 +1100 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: References: Message-ID: A better way of putting it... While you can look at UBF(A) and UBF(C) as a formal approach to protocols, what's relevant here, is that from Erlang's perspective, you can look at UBF(B) as a formal approach to state machines providing type checking and declarative state transitions. This can be very helpful at all levels of development of a state machine from design right though to coding, debugging and documentation. That's more what I was trying to say. - Edmond - On Thu, 04 Nov 2010 03:58:12 +1100, Edmond Begumisa wrote: > Here's an example of what I mean. Here is a UBF(B) contract copied from > Joe's UBF page (http://www.sics.se/~joe/ubf/site/quick.html)... > > +NAME("file_server"). > > +VSN("ubf1.0"). > > +TYPES > > info() = info; > description() = description; > services() = services; > contract() = contract; > > file() = string(); > ls() = ls; > files() = {files, [file()]}; > getFile() = {get, file()}; > noSuchFile() = noSuchFile. > > > +STATE start > ls() => files() & start; > getFile() => binary() & start > | noSuchFile() & stop. > > > +ANYSTATE > info() => string(); > description() => string(); > contract() => term(). > > (you can find the corresponding state machine implementation is > http://www.sics.se/~joe/ubf/site/eserve.html) > > ... you could achieve the exact same thing with gen_fsm combined with > TypEr/Dialyzer (I was doing just that), but the advantage to using the > UBF(B) contract instead is this info is declared in one place, as you > design and code away, then verified at runtime by the UBF(C) contract > checker. I find this approach more declarative and self-documenting. > IMO, state machines written against a UBF contract are much easier to > write, maintain, debug, document and modify than those written with > gen_fsm. > > Of course, to make use of this, you have to migrate to UBF(A) encoding > for the actual messages between processes. This usually only makes sense > if you're crossing OS process boundaries, and probably too costly for > communication between Erlang processes in the same VM instance. > > Actually, there's an idea: maybe someone should write a contract checker > for gen_fsm that uses ordinary Erlang-terms rather than UBF(A) encoding. > This could then be used in the same VM instance without much cost and > make working with gen_fsm more dependable by allowing you to declare how > the state-machine is supposed to behave and cross check that against how > it's actually behaving -- a major frustration I've had when using > gen_fsm. > > - Edmond - > > > On Wed, 03 Nov 2010 20:35:13 +1100, Torben Hoffmann > wrote: > >> Hi Edmond, >> >> I am a bit confused by your statements since my problem was about >> documentation of an _existing_ gen_fsm. >> It might be my ignorance about what UBF can offer that is causing this >> (read >> Joe's payer a while ago). >> >> As I understand UBF it allows you to specify what a component does so >> that >> others can interact with it in a sensible manner and you can figure out >> if a >> given component implements a given contract - or am I missing the point >> here? >> >> I actually thought about using UBF, but when I looked at it it seemed a >> bit >> to immature for a mission critical system, maybe that have changed by >> now. >> >> Do not get me wrong, I firmly believe in specifying protocols and UBF >> seems >> like a very interesting thing to use, but in a real life situation you >> sometimes have code that you want to extract information about and this >> was >> the problem I was faced with. >> >> If I understand you right you are no longer using gen_fsm, right? >> How do you piece together your software then? >> I use the OTP supervisor structure all the time due to the obvious >> benefits >> of a battle tested framework, but I would not mind adding another tool >> to my >> tool box!! >> >> Cheers, >> Torben >> >> On Wed, Nov 3, 2010 at 10:20, Edmond Begumisa >> wrote: >> >>> Hi, >>> >>> I don't know how immediately useful this is (perhaps something to keep >>> in >>> mind), but one of the attractive understated things about UBF is that >>> you >>> define all this information in one place for both easy >>> configuration/protocol-definition and reading/self-documentation... >>> >>> http://www.sics.se/~joe/ubf/site/quick.html >>> https://github.com/norton/ubf >>> >>> So far, I haven't regretted switching from gen_fsm to UBF(C) >>> >>> - Edmond - >>> >>> >>> >>> On Mon, 01 Nov 2010 05:58:35 +1100, Tomas Abrahamsson < >>> tomas.abrahamsson@REDACTED> wrote: >>> >>> On Tue, Oct 12, 2010 at 09:00, Torben Hoffmann >>> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I want to extract the following information from a gen_fsm: >>>>> >>>>> - all the states >>>>> - the incoming events >>>>> - possible next states >>>>> >>>>> but I want to do it per function clause and not only on the function >>>>> level >>>>> as TypEr and Dialyzer does it. >>>>> >>>> >>>> One way to do that is to copy each function clause into a >>>> separate function. Then one can run TypEr on the expanded >>>> code. Attached is an escript that does such an expansion >>>> of exported functions of arity 2 and 3, and also of >>>> handle_event, handle_sync_event and handle_info. The script >>>> accepts -I and -D options for include paths and macro >>>> definitions, just like erlc,and prints the expanded program >>>> on stdout. >>>> >>>> I've tried plotting state machines from the output of TypEr >>>> when run on the expanded results, and it seems to do reasonably >>>> well. I haven't done any thorough verification. >>>> >>>> It doesn't solve all of your problems: it doesn't find any >>>> outgoing messages, and it has the same limitations you >>>> already indicated when returning from a function clause in >>>> different ways, so I guess the results are roughly what >>>> you already seems to have, or what Vance's drawing tool >>>> already produces. >>>> >>>> Somehow being able to make more use of TypEr's >>>> knowledge about a file would be really interesting, >>>> for example to be able to query it for type information >>>> given various execution paths through the program, >>>> if it would be possible. >>>> >>>> BRs >>>> Tomas >>>> >>> >>> >>> -- >>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >>> >> >> >> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From rzezeski@REDACTED Thu Nov 4 00:35:08 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Wed, 3 Nov 2010 19:35:08 -0400 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <4CBC2F6D.1060509@amberbio.com> References: <4CB57B2A.7070703@gmail.com> <4CB588DF.8090002@amberbio.com> <28167138-3C02-4D75-9A22-2B74BF6C3321@cs.otago.ac.nz> <4CB6A985.4090701@amberbio.com> <4CB6B589.9050003@cs.ntua.gr> <4CB6BDDC.3060900@amberbio.com> <4CBADE65.7010909@amberbio.com> <14a8a90f-2425-4e42-9519-849f0158ff22@h7g2000yqn.googlegroups.com> <4CBB49F3.8060809@amberbio.com> <20101017214635.GA1570@hanele.lan> <4CBC05C4.1090701@amberbio.com> <7455541B-BBDF-41C9-8924-EF63ED1DAE37@erlang-solutions.com> <4CBC2F6D.1060509@amberbio.com> Message-ID: On Mon, Oct 18, 2010 at 7:28 AM, Morten Krogh wrote: > > > On practical terms, I feel that Erlang some times gives up performance for > unnecessary reasons. I am especially thinking about mutability. > There is no reason to give up mutation within a process which is already a > serializer. > > code like this > > loop(Dict) > receive > {put, K, V} -> > Dict2 = dict:store(K, V, Dict), > loop(Dict2) > end. > > > is just a performance wise inefficient way of doing > > loop(Mut) -> > receive > {put, K, V} -> > put(K,V, Mut), > loop(Mut) > end. > > where Mut is a mutable data structure. > In the dict exampe, there is a unncecessary work and garbage collection for > the functional data structure Dict. > And you gain nothing, since it is sequential code. > > I think this is a good point. If you are simply creating a data structure in _one_ process it should be mutable. Then when you publish it it becomes immutable. I have Clojure's transients in mind. http://clojure.org/transients *If a tree falls in the woods, does it make a sound?**If a pure function mutates some local data in order to produce an immutable return value, is that ok?* * * * * From tony.arcieri@REDACTED Thu Nov 4 00:50:47 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Wed, 3 Nov 2010 17:50:47 -0600 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <4CB57B2A.7070703@gmail.com> <4CB588DF.8090002@amberbio.com> <28167138-3C02-4D75-9A22-2B74BF6C3321@cs.otago.ac.nz> <4CB6A985.4090701@amberbio.com> <4CB6B589.9050003@cs.ntua.gr> <4CB6BDDC.3060900@amberbio.com> <4CBADE65.7010909@amberbio.com> <14a8a90f-2425-4e42-9519-849f0158ff22@h7g2000yqn.googlegroups.com> <4CBB49F3.8060809@amberbio.com> <20101017214635.GA1570@hanele.lan> <4CBC05C4.1090701@amberbio.com> <7455541B-BBDF-41C9-8924-EF63ED1DAE37@erlang-solutions.com> <4CBC2F6D.1060509@amberbio.com> Message-ID: I agree with this sentiment as well. A pure immutable system gives more optimization options to VM implementers, but as is without the shared/hybrid heaps Erlang doesn't do a good job of leveraging the potential of a purely immutable state system. On Wed, Nov 3, 2010 at 5:35 PM, Ryan Zezeski wrote: > On Mon, Oct 18, 2010 at 7:28 AM, Morten Krogh wrote: > > > > > > On practical terms, I feel that Erlang some times gives up performance > for > > unnecessary reasons. I am especially thinking about mutability. > > There is no reason to give up mutation within a process which is already > a > > serializer. > > > > code like this > > > > loop(Dict) > > receive > > {put, K, V} -> > > Dict2 = dict:store(K, V, Dict), > > loop(Dict2) > > end. > > > > > > is just a performance wise inefficient way of doing > > > > loop(Mut) -> > > receive > > {put, K, V} -> > > put(K,V, Mut), > > loop(Mut) > > end. > > > > where Mut is a mutable data structure. > > In the dict exampe, there is a unncecessary work and garbage collection > for > > the functional data structure Dict. > > And you gain nothing, since it is sequential code. > > > > > I think this is a good point. If you are simply creating a data structure > in _one_ process it should be mutable. Then when you publish it it becomes > immutable. I have Clojure's transients in mind. > > http://clojure.org/transients > > *If a tree falls in the woods, does it make a sound?**If a pure function > mutates some local data in order to produce an immutable return value, is > that ok?* > * > * > * > * > -- Tony Arcieri Medioh! A Kudelski Brand From robert.virding@REDACTED Thu Nov 4 01:50:26 2010 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 4 Nov 2010 00:50:26 +0000 (GMT) Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <420938007.35431288831679157.JavaMail.root@zimbra> Message-ID: <842437213.35451288831826841.JavaMail.root@zimbra> This question is much more complex that you would at first realise. Some thoughts (in no specific order): 1. Shared/hybrid heaps are wonderful BUT: - They are more difficult to implement in a real-time environment. Yes, this is a known problem with solutions but the benefits are less than you would expect. Real-time gc costs. - They don't port easily to multi-core systems as you suddenly start needing locks and stuff everywhere to control access or have complex schemes to get around this. Which copying doesn't. For some problems shared heaps are much better, for others much worse. 2. I would assert that the operations on immutable data structures cost less then you imagine. 3. From the way you describe it having data structures which are both mutable and immutable would very confusing. You would need to have two sets of operations on them and every application would have to keep track of when to use them. 4. Mutable data structures give you all the things you want to avoid with global data, especially that they can change under your feet without you knowing about it. Yes, you may have to pass a reference around but it still means that I don't know who can access my bit of data and can change it. Yes, you can stipulate that the program should be well-behaved, but we all know how well that works in real-life in a big system. So while I have said that you don't *need* immutable data to get process isolation, instead you can specify data copying between processes, this does not mean that I recommend having mutable data. Immutable data is so much easier to work with and keep track of. Robert ----- "Tony Arcieri" wrote: > I agree with this sentiment as well. A pure immutable system gives > more > optimization options to VM implementers, but as is without the > shared/hybrid > heaps Erlang doesn't do a good job of leveraging the potential of a > purely > immutable state system. > > On Wed, Nov 3, 2010 at 5:35 PM, Ryan Zezeski > wrote: > > > On Mon, Oct 18, 2010 at 7:28 AM, Morten Krogh > wrote: > > > > > > > > > On practical terms, I feel that Erlang some times gives up > performance > > for > > > unnecessary reasons. I am especially thinking about mutability. > > > There is no reason to give up mutation within a process which is > already > > a > > > serializer. > > > > > > code like this > > > > > > loop(Dict) > > > receive > > > {put, K, V} -> > > > Dict2 = dict:store(K, V, Dict), > > > loop(Dict2) > > > end. > > > > > > > > > is just a performance wise inefficient way of doing > > > > > > loop(Mut) -> > > > receive > > > {put, K, V} -> > > > put(K,V, Mut), > > > loop(Mut) > > > end. > > > > > > where Mut is a mutable data structure. > > > In the dict exampe, there is a unncecessary work and garbage > collection > > for > > > the functional data structure Dict. > > > And you gain nothing, since it is sequential code. > > > > > > > > I think this is a good point. If you are simply creating a data > structure > > in _one_ process it should be mutable. Then when you publish it it > becomes > > immutable. I have Clojure's transients in mind. > > > > http://clojure.org/transients > > > > *If a tree falls in the woods, does it make a sound?**If a pure > function > > mutates some local data in order to produce an immutable return > value, is > > that ok?* > > * > > * > > * > > * > > > > > > -- > Tony Arcieri > Medioh! A Kudelski Brand From silasdb@REDACTED Thu Nov 4 02:23:25 2010 From: silasdb@REDACTED (Silas Silva) Date: Wed, 3 Nov 2010 23:23:25 -0200 Subject: Conceptual questions on architecture of functional programs Message-ID: <20101104012322.GA375@hope.tifa.renegado> Hi all! After asking conceptual questions about distributed NoSQL databases, it is time for a more newbie question, about functional programming. There is some years I do fine OO and imperative programming. I needed to learn some functional programming. So I decided to start a basic commercial application (customer, orders, etc.) in Erlang to learn something about it. After some time, I realized my program had the following structure: - model.erl -> basic functional to access Mnesia database - layer.erl -> an abstraction layer between model and front-end. Controller? - frontend.erl -> show a front-end for the user, for now HTTP in Mochiweb. Latter, a GUI. So, from fron-end I use layer functions, that use model functions. So I realized I wasn't actually doing any functional programming. This could be an architecture of a C program using high-level code. I did sequential programming split in dozens of mini-functions and I believed that this was functional programming (a C-like program with high-level constructors)... In OO, I would easily return an object from layer, call its methods to get what I wanted and make heavy use of encapsulation in objects. I could also pass objects in and out. In the program I just wrote, I pass in and out records, that are not that different from passing C structures in and out. So, my question is: where is my mistake? What about a reference (book, tutorial whatever) about functional programming architecture for OO programmers (or imperative programmers)? "Programming Erlang Software for a Concurrent World", by Joe Armstrong is great, but it is more about the language itself than functional programming techniques. "Practical Common Lisp", by Peter Seibel is more worth to read. Other references are found in http://stackoverflow.com/questions/89212/functional-programming-architecture Any tips about that? Thanks! P.S: The above program don't have any concurrent code yet. I know it is really important for Erlang, but would it be really that necessary to get function programming feeling? -- Silas Silva From g9414002.pccu.edu.tw@REDACTED Thu Nov 4 02:56:42 2010 From: g9414002.pccu.edu.tw@REDACTED (=?UTF-8?B?6buD6ICA6LOiIChZYXUtSHNpZW4gSHVhbmcp?=) Date: Thu, 4 Nov 2010 09:56:42 +0800 Subject: [erlang-questions] Conceptual questions on architecture of functional programs In-Reply-To: <20101104012322.GA375@hope.tifa.renegado> References: <20101104012322.GA375@hope.tifa.renegado> Message-ID: Hi, Silas. There's an introductory paper "Why Functional Programming Matters" for you. In functional fashion, you can make second-order, third-order and n-th-order functions when you pass a name of a first-order function into some function. -- Best Regards. --- Y-H. H. On Thu, Nov 4, 2010 at 9:23 AM, Silas Silva wrote: > Hi all! > > After asking conceptual questions about distributed NoSQL databases, it > is time for a more newbie question, about functional programming. > > There is some years I do fine OO and imperative programming. I needed > to learn some functional programming. So I decided to start a basic > commercial application (customer, orders, etc.) in Erlang to learn > something about it. After some time, I realized my program had the > following structure: > > > - model.erl -> basic functional to access Mnesia database > - layer.erl -> an abstraction layer between model and front-end. > Controller? > - frontend.erl -> show a front-end for the user, for now HTTP in > Mochiweb. Latter, a GUI. > > > So, from fron-end I use layer functions, that use model functions. > > So I realized I wasn't actually doing any functional programming. This > could be an architecture of a C program using high-level code. I did > sequential programming split in dozens of mini-functions and I believed > that this was functional programming (a C-like program with high-level > constructors)... > > In OO, I would easily return an object from layer, call its methods to > get what I wanted and make heavy use of encapsulation in objects. I > could also pass objects in and out. > > In the program I just wrote, I pass in and out records, that are not > that different from passing C structures in and out. > > So, my question is: where is my mistake? What about a reference (book, > tutorial whatever) about functional programming architecture for OO > programmers (or imperative programmers)? "Programming Erlang Software > for a Concurrent World", by Joe Armstrong is great, but it is more about > the language itself than functional programming techniques. "Practical > Common Lisp", by Peter Seibel is more worth to read. > > Other references are found in > > http://stackoverflow.com/questions/89212/functional-programming-architecture > > Any tips about that? > > Thanks! > > P.S: The above program don't have any concurrent code yet. I know it is > really important for Erlang, but would it be really that necessary to > get function programming feeling? > > -- > Silas Silva > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From rzezeski@REDACTED Thu Nov 4 03:47:05 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Wed, 3 Nov 2010 22:47:05 -0400 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <842437213.35451288831826841.JavaMail.root@zimbra> References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> Message-ID: On Wed, Nov 3, 2010 at 8:50 PM, Robert Virding < robert.virding@REDACTED> wrote: > This question is much more complex that you would at first realise. Some > thoughts (in no specific order): > > 1. Shared/hybrid heaps are wonderful BUT: > - They are more difficult to implement in a real-time environment. Yes, > this is a known problem with solutions but the benefits are less than you > would expect. Real-time gc costs. > - They don't port easily to multi-core systems as you suddenly start > needing locks and stuff everywhere to control access or have complex schemes > to get around this. Which copying doesn't. > > For some problems shared heaps are much better, for others much worse. > > 2. I would assert that the operations on immutable data structures cost > less then you imagine. > > 3. From the way you describe it having data structures which are both > mutable and immutable would very confusing. You would need to have two sets > of operations on them and every application would have to keep track of when > to use them. > > 4. Mutable data structures give you all the things you want to avoid with > global data, especially that they can change under your feet without you > knowing about it. Yes, you may have to pass a reference around but it still > means that I don't know who can access my bit of data and can change it. > Yes, you can stipulate that the program should be well-behaved, but we all > know how well that works in real-life in a big system. > > So while I have said that you don't *need* immutable data to get process > isolation, instead you can specify data copying between processes, this does > not mean that I recommend having mutable data. Immutable data is so much > easier to work with and keep track of. > > Robert > > > ----- "Tony Arcieri" wrote: > > > I agree with this sentiment as well. A pure immutable system gives > > more > > optimization options to VM implementers, but as is without the > > shared/hybrid > > heaps Erlang doesn't do a good job of leveraging the potential of a > > purely > > immutable state system. > > > > On Wed, Nov 3, 2010 at 5:35 PM, Ryan Zezeski > > wrote: > > > > > On Mon, Oct 18, 2010 at 7:28 AM, Morten Krogh > > wrote: > > > > > > > > > > > > On practical terms, I feel that Erlang some times gives up > > performance > > > for > > > > unnecessary reasons. I am especially thinking about mutability. > > > > There is no reason to give up mutation within a process which is > > already > > > a > > > > serializer. > > > > > > > > code like this > > > > > > > > loop(Dict) > > > > receive > > > > {put, K, V} -> > > > > Dict2 = dict:store(K, V, Dict), > > > > loop(Dict2) > > > > end. > > > > > > > > > > > > is just a performance wise inefficient way of doing > > > > > > > > loop(Mut) -> > > > > receive > > > > {put, K, V} -> > > > > put(K,V, Mut), > > > > loop(Mut) > > > > end. > > > > > > > > where Mut is a mutable data structure. > > > > In the dict exampe, there is a unncecessary work and garbage > > collection > > > for > > > > the functional data structure Dict. > > > > And you gain nothing, since it is sequential code. > > > > > > > > > > > I think this is a good point. If you are simply creating a data > > structure > > > in _one_ process it should be mutable. Then when you publish it it > > becomes > > > immutable. I have Clojure's transients in mind. > > > > > > http://clojure.org/transients > > > > > > *If a tree falls in the woods, does it make a sound?**If a pure > > function > > > mutates some local data in order to produce an immutable return > > value, is > > > that ok?* > > > * > > > * > > > * > > > * > > > > > > > > > > > -- > > Tony Arcieri > > Medioh! A Kudelski Brand > Robert, I'm not sure if this was directed towards me or the thread in general, but I want to clarify my point. I completely agree immutable data structures are the way to go. One reading of Java Concurrency In Practice was enough to make me shiver at shared mutable structures. I don't agree that operations on immutable data structures cost less than *I* imagine. If that was the case why would Rich Hickey go through great lengths to find _performant_ immutable data structures that he implemented in Clojure (Bagwell Trees)? Furthermore, why would he also create a separate set of functions (transients) for building up a data structure in a mutable way and then publishing it to the world as an immutable data structure? Plus, the cost of anything is relative. Yes, in most cases immutable data structures are more than fast enough. But in a few cases maybe they aren't. What is wrong about using a mutable data structure that can only ever be touched by _one_ and only _one_ process? Then, if this process wants to share it it must be converted to an immutable data structure. In Clojure this works by using a set of dedicated functions to work with the mutable data structure. This data structure will not work with the rest of Clojure which expects an immutable thing (or maybe it implicitly converts it, I forget). Then when you are ready to share this thing with other threads you simply change it to be immutable which is a cheap operation. So, in the specific case where only _one_ thread creates the data structure but then is shared by many you get the speed of in-place mutation with the safety of immutability. I'm no programming language expert, but this seems like an obvious win. To clarify even more, the data structure is not "both mutable and immutable." It's mutable by one thread/process and then immutable the second it's published (shared) and can no longer be mutated. -Ryan From kurumbahmp@REDACTED Thu Nov 4 04:00:22 2010 From: kurumbahmp@REDACTED (Kurumba hmp) Date: Thu, 4 Nov 2010 08:30:22 +0530 Subject: Job Opportunity Message-ID: Looking for Architects who is Good in Erlang and has SS7 experience to be based in Colombo Sri Lanka. Work in a relaxed atmosphere in a idyllic location with attractive benefits. Required skills include; Erlang/OTP Mnesia MySQL C/C++ Real time concurrent programming in Erlang Experience with Erlang port drivers Understanding on telecom networks GSM, CDMA, WCDMA, LTE etc HLR, VLR, MSC, SGSN, STP etc Experience in TDM-SS7 and Sigtran Experience in working with SS7 protocols such as, - MTP2, MTP3, SCCP, TCAP, MAP, INAP, CAMEL, WIN, etc Network programming Experience in architecting/developing high performance/high throughput telco grade Value Added Services platforms Stress testing/load testing for signaling based platforms We are developing products and solutions used by over hundreds of People across Asia, Africa, Australia and Middle East. Our expertise is network based multimedia enhanced communication to wireless and wireline carriers world wide has rooted the company firmly among the world?s leaders in software solutions. Armed with the latest enabling technology and equipped with reliable and scalable solutions designed for Teleco?s the most demanding traffic requirements are catered to. Embracing a standards-based open software architectural platform in the telecommunications network , mobile, messaging and VoIP, the business is poised to echo subscriber needs and enrich user experience. Send in your CV to info@REDACTED or contact us on +61417327676 From ok@REDACTED Thu Nov 4 04:43:52 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 4 Nov 2010 16:43:52 +1300 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <4CB57B2A.7070703@gmail.com> <4CB588DF.8090002@amberbio.com> <28167138-3C02-4D75-9A22-2B74BF6C3321@cs.otago.ac.nz> <4CB6A985.4090701@amberbio.com> <4CB6B589.9050003@cs.ntua.gr> <4CB6BDDC.3060900@amberbio.com> <4CBADE65.7010909@amberbio.com> <14a8a90f-2425-4e42-9519-849f0158ff22@h7g2000yqn.googlegroups.com> <4CBB49F3.8060809@amberbio.com> <20101017214635.GA1570@hanele.lan> <4CBC05C4.1090701@amberbio.com> <7455541B-BBDF-41C9-8924-EF63ED1DAE37@erlang-solutions.com> <4CBC2F6D.1060509@amberbio.com> Message-ID: On 4/11/2010, at 12:35 PM, Ryan Zezeski wrote: > *If a tree falls in the woods, does it make a sound?**If a pure function > mutates some local data in order to produce an immutable return value, is > that ok?* Provided literally *nobody*, including the process in question, can *tell*, yes, it's fine. Witness: monads in Haskell, uniqueness types in Clean and Mercury. From jws@REDACTED Thu Nov 4 07:34:46 2010 From: jws@REDACTED (Jeff Schultz) Date: Thu, 4 Nov 2010 17:34:46 +1100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> Message-ID: <20101104063446.GA15679@mulga.csse.unimelb.edu.au> On Wed, Nov 03, 2010 at 10:47:05PM -0400, Ryan Zezeski wrote: > What is wrong about using a mutable data structure that can only ever be > touched by _one_ and only _one_ process? Then, if this process wants to Well, mutability may impose a significant cost on all but the simplest garbage collectors, and part of that cost will be paid even if nothing mutable is being mutated. Whether this is a net performance gain or loss will vary with the circumstances, but it's not an inevitable win. The extra complexity may also lead to more, usually nasty, garbage collection bugs. Jeff Schultz From spawn.think@REDACTED Thu Nov 4 09:29:58 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 4 Nov 2010 09:29:58 +0100 Subject: [erlang-questions] Conceptual questions on architecture of functional programs In-Reply-To: References: <20101104012322.GA375@hope.tifa.renegado> Message-ID: I would recommend this one: http://www.defmacro.org/ramblings/fp.html 2010/11/4 ??? (Yau-Hsien Huang) > Hi, Silas. > > There's an introductory paper "Why Functional Programming Matters" for you. > > In functional fashion, you can make second-order, third-order and > n-th-order > functions > when you pass a name of a first-order function into some function. > > > -- > > Best Regards. > > --- Y-H. H. > > > On Thu, Nov 4, 2010 at 9:23 AM, Silas Silva wrote: > > > Hi all! > > > > After asking conceptual questions about distributed NoSQL databases, it > > is time for a more newbie question, about functional programming. > > > > There is some years I do fine OO and imperative programming. I needed > > to learn some functional programming. So I decided to start a basic > > commercial application (customer, orders, etc.) in Erlang to learn > > something about it. After some time, I realized my program had the > > following structure: > > > > > > - model.erl -> basic functional to access Mnesia database > > - layer.erl -> an abstraction layer between model and front-end. > > Controller? > > - frontend.erl -> show a front-end for the user, for now HTTP in > > Mochiweb. Latter, a GUI. > > > > > > So, from fron-end I use layer functions, that use model functions. > > > > So I realized I wasn't actually doing any functional programming. This > > could be an architecture of a C program using high-level code. I did > > sequential programming split in dozens of mini-functions and I believed > > that this was functional programming (a C-like program with high-level > > constructors)... > > > > In OO, I would easily return an object from layer, call its methods to > > get what I wanted and make heavy use of encapsulation in objects. I > > could also pass objects in and out. > > > > In the program I just wrote, I pass in and out records, that are not > > that different from passing C structures in and out. > > > > So, my question is: where is my mistake? What about a reference (book, > > tutorial whatever) about functional programming architecture for OO > > programmers (or imperative programmers)? "Programming Erlang Software > > for a Concurrent World", by Joe Armstrong is great, but it is more about > > the language itself than functional programming techniques. "Practical > > Common Lisp", by Peter Seibel is more worth to read. > > > > Other references are found in > > > > > http://stackoverflow.com/questions/89212/functional-programming-architecture > > > > Any tips about that? > > > > Thanks! > > > > P.S: The above program don't have any concurrent code yet. I know it is > > really important for Erlang, but would it be really that necessary to > > get function programming feeling? > > > > -- > > Silas Silva > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From hynek@REDACTED Thu Nov 4 10:40:13 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Thu, 4 Nov 2010 10:40:13 +0100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <842437213.35451288831826841.JavaMail.root@zimbra> References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> Message-ID: I honestly will put one more point to your list. On Thu, Nov 4, 2010 at 1:50 AM, Robert Virding wrote: > This question is much more complex that you would at first realise. Some thoughts (in no specific order): > > 1. Shared/hybrid heaps are wonderful BUT: > - They are more difficult to implement in a real-time environment. Yes, this is a known problem with solutions but the benefits are less than you would expect. Real-time gc costs. > - They don't port easily to multi-core systems as you suddenly start needing locks and stuff everywhere to control access or have complex schemes to get around this. Which copying doesn't. > > For some problems shared heaps are much better, for others much worse. > > 2. I would assert that the operations on immutable data structures cost less then you imagine. > > 3. From the way you describe it having data structures which are both mutable and immutable would very confusing. You would need to have two sets of operations on them and every application would have to keep track of when to use them. > > 4. Mutable data structures give you all the things you want to avoid with global data, especially that they can change under your feet without you knowing about it. Yes, you may have to pass a reference around but it still means that I don't know who can access my bit of data and can change it. Yes, you can stipulate that the program should be well-behaved, but we all know how well that works in real-life in a big system. > 5. More dragons lies in exceptions and error handling. With immutable data structures you have transaction memory for "free". To keep immutable semantic but mutable implementation you have to keep mutability only in functions which can't fail which add much more complexity and enables mutability to much less place than is worth for. > So while I have said that you don't *need* immutable data to get process isolation, instead you can specify data copying between processes, this does not mean that I recommend having mutable data. Immutable data is so much easier to work with and keep track of. > > Robert > > > ----- "Tony Arcieri" wrote: > >> I agree with this sentiment as well. A pure immutable system gives >> more >> optimization options to VM implementers, but as is without the >> shared/hybrid >> heaps Erlang doesn't do a good job of leveraging the potential of a >> purely >> immutable state system. >> >> On Wed, Nov 3, 2010 at 5:35 PM, Ryan Zezeski >> wrote: >> >> > On Mon, Oct 18, 2010 at 7:28 AM, Morten Krogh >> wrote: >> > > >> > > >> > > On practical terms, I feel that Erlang some times gives up >> performance >> > for >> > > unnecessary reasons. I am especially thinking about mutability. >> > > There is no reason to give up mutation within a process which is >> already >> > a >> > > serializer. >> > > >> > > code like this >> > > >> > > loop(Dict) >> > > ? ?receive >> > > ? ?{put, K, V} -> >> > > ? ? ? ?Dict2 = dict:store(K, V, Dict), >> > > ? ? ? ?loop(Dict2) >> > > ? ?end. >> > > >> > > >> > > is just a performance wise inefficient way of doing >> > > >> > > loop(Mut) -> >> > > ? ?receive >> > > ? ?{put, K, V} -> >> > > ? ? ? ?put(K,V, Mut), >> > > ? ? ? ?loop(Mut) >> > > ? ?end. >> > > >> > > where Mut is a mutable data structure. >> > > In the dict exampe, there is a unncecessary work and garbage >> collection >> > for >> > > the functional data structure Dict. >> > > And you gain nothing, since it is sequential code. >> > > >> > > >> > I think this is a good point. ?If you are simply creating a data >> structure >> > in _one_ process it should be mutable. ?Then when you publish it it >> becomes >> > immutable. ?I have Clojure's transients in mind. >> > >> > http://clojure.org/transients >> > >> > *If a tree falls in the woods, does it make a sound?**If a pure >> function >> > mutates some local data in order to produce an immutable return >> value, is >> > that ok?* >> > * >> > * >> > * >> > * >> > >> >> >> >> -- >> Tony Arcieri >> Medioh! A Kudelski Brand > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From robert.virding@REDACTED Thu Nov 4 11:46:08 2010 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 4 Nov 2010 10:46:08 +0000 (GMT) Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <14071962.37291288867509666.JavaMail.root@zimbra> Message-ID: <415901635.37311288867568017.JavaMail.root@zimbra> Hi Ryan, My comment was directed to the thread in general, but partially as a comment to your comments. So when I was using "you" it was not specifically you Ryan, but you the reader. "2. I would assert that the operations on immutable data structures cost less then many would imagine." I was really trying to make three points: - Implementing a good shared/hybrid real-time gc is not trivial, and doing it in a multi-core system is even less trivial. Mutable data makes it even less trivial. Yes, it a known problem, but still difficult. - I think that having immutable data even within only one process is a big win. Having mutable data gives you all the problems of global data. Immutable data makes it much easier to keep track of what is happening with your data, which is always a Big Win. - Having the data as both mutable and immutable, even if not at the same time, means I have two different ways of working with it and need to know which it is just now. Not being a Clojure person I could very well be wrong in this point, but at least *I* would find it difficult to keep track of which it is. I don't know if this helped any. Robert ----- "Ryan Zezeski" wrote: > On Wed, Nov 3, 2010 at 8:50 PM, Robert Virding < > robert.virding@REDACTED > wrote: > > > This question is much more complex that you would at first realise. > Some thoughts (in no specific order): > > 1. Shared/hybrid heaps are wonderful BUT: > - They are more difficult to implement in a real-time environment. > Yes, this is a known problem with solutions but the benefits are less > than you would expect. Real-time gc costs. > - They don't port easily to multi-core systems as you suddenly start > needing locks and stuff everywhere to control access or have complex > schemes to get around this. Which copying doesn't. > > For some problems shared heaps are much better, for others much worse. > > 2. I would assert that the operations on immutable data structures > cost less then you imagine. > > 3. From the way you describe it having data structures which are both > mutable and immutable would very confusing. You would need to have two > sets of operations on them and every application would have to keep > track of when to use them. > > 4. Mutable data structures give you all the things you want to avoid > with global data, especially that they can change under your feet > without you knowing about it. Yes, you may have to pass a reference > around but it still means that I don't know who can access my bit of > data and can change it. Yes, you can stipulate that the program should > be well-behaved, but we all know how well that works in real-life in a > big system. > > So while I have said that you don't *need* immutable data to get > process isolation, instead you can specify data copying between > processes, this does not mean that I recommend having mutable data. > Immutable data is so much easier to work with and keep track of. > > Robert > > > > > > ----- "Tony Arcieri" < tony.arcieri@REDACTED > wrote: > > > I agree with this sentiment as well. A pure immutable system gives > > more > > optimization options to VM implementers, but as is without the > > shared/hybrid > > heaps Erlang doesn't do a good job of leveraging the potential of a > > purely > > immutable state system. > > > > On Wed, Nov 3, 2010 at 5:35 PM, Ryan Zezeski < rzezeski@REDACTED > > > wrote: > > > > > On Mon, Oct 18, 2010 at 7:28 AM, Morten Krogh < mk@REDACTED > > > wrote: > > > > > > > > > > > > On practical terms, I feel that Erlang some times gives up > > performance > > > for > > > > unnecessary reasons. I am especially thinking about mutability. > > > > There is no reason to give up mutation within a process which is > > already > > > a > > > > serializer. > > > > > > > > code like this > > > > > > > > loop(Dict) > > > > receive > > > > {put, K, V} -> > > > > Dict2 = dict:store(K, V, Dict), > > > > loop(Dict2) > > > > end. > > > > > > > > > > > > is just a performance wise inefficient way of doing > > > > > > > > loop(Mut) -> > > > > receive > > > > {put, K, V} -> > > > > put(K,V, Mut), > > > > loop(Mut) > > > > end. > > > > > > > > where Mut is a mutable data structure. > > > > In the dict exampe, there is a unncecessary work and garbage > > collection > > > for > > > > the functional data structure Dict. > > > > And you gain nothing, since it is sequential code. > > > > > > > > > > > I think this is a good point. If you are simply creating a data > > structure > > > in _one_ process it should be mutable. Then when you publish it it > > becomes > > > immutable. I have Clojure's transients in mind. > > > > > > http://clojure.org/transients > > > > > > *If a tree falls in the woods, does it make a sound?**If a pure > > function > > > mutates some local data in order to produce an immutable return > > value, is > > > that ok?* > > > * > > > * > > > * > > > * > > > > > > > > > > > -- > > Tony Arcieri > > Medioh! A Kudelski Brand > > > Robert, I'm not sure if this was directed towards me or the thread in > general, but I want to clarify my point. > > > I completely agree immutable data structures are the way to go. One > reading of Java Concurrency In Practice was enough to make me shiver > at shared mutable structures. > > > I don't agree that operations on immutable data structures cost less > than *I* imagine. If that was the case why would Rich Hickey go > through great lengths to find _performant_ immutable data structures > that he implemented in Clojure (Bagwell Trees)? Furthermore, why would > he also create a separate set of functions (transients) for building > up a data structure in a mutable way and then publishing it to the > world as an immutable data structure? Plus, the cost of anything is > relative. Yes, in most cases immutable data structures are more than > fast enough. But in a few cases maybe they aren't. > > > What is wrong about using a mutable data structure that can only ever > be touched by _one_ and only _one_ process? Then, if this process > wants to share it it must be converted to an immutable data structure. > In Clojure this works by using a set of dedicated functions to work > with the mutable data structure. This data structure will not work > with the rest of Clojure which expects an immutable thing (or maybe it > implicitly converts it, I forget). Then when you are ready to share > this thing with other threads you simply change it to be immutable > which is a cheap operation. So, in the specific case where only _one_ > thread creates the data structure but then is shared by many you get > the speed of in-place mutation with the safety of immutability. I'm no > programming language expert, but this seems like an obvious win. > > > To clarify even more, the data structure is not "both mutable and > immutable." It's mutable by one thread/process and then immutable the > second it's published (shared) and can no longer be mutated. > > > -Ryan From psa@REDACTED Thu Nov 4 12:52:07 2010 From: psa@REDACTED (=?UTF-8?B?UGF1bG8gU8OpcmdpbyBBbG1laWRh?=) Date: Thu, 04 Nov 2010 11:52:07 +0000 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <415901635.37311288867568017.JavaMail.root@zimbra> References: <415901635.37311288867568017.JavaMail.root@zimbra> Message-ID: <4CD29E67.3030609@di.uminho.pt> Hi Robert, there is a point on which I do not agree; when you say: On 11/4/10 10:46 AM, Robert Virding wrote: > - I think that having immutable data even within only one process is a big win. Having mutable data gives you all the problems of global data. Immutable data makes it much easier to keep track of what is happening with your data, which is always a Big Win. I would say that having mutable data gives us SOME problems, but much LESS than when we have global mutable data. You are mixing the problems due to: - global data - mutable data In most languages around we have the nightmare of mutable global data, and the nasty problem of aliasing in object references. On the other hand in Erlang we have processes that give us locality, but only immutable data in each process. I (and others) think there is a place for a middle-ground where processes give us locality, allowing a divid-and-conquer of the global problem into simple parts, where the data in a given process is manageable, but where if the algorithm calls for mutable data then we should be able to use it. All the nasty problems of shared memory concurrency (from the programmer's point of view) have already disappeared when we have only per-process mutable data as opposed to global mutable data. If the data is immutable and only pure functions are used, we have extra benefits, but not always can we afford it, nor is it necessary even for proving correctness. Lot's of classic formal methods stuff with pre-conditions, post-conditions, invariants, Hoare logic, etc, were made for the mutable data world. They become difficult to apply when we have global data and aliasing, but may well be applied when we have just a little local data. The idea would be to be able to use the many proven algorithms that were created for the imperative sequential world inside each process, and let the process concept a la Erlang deal with the concurrency aspects. I am not saying it is now practical to add it to Erlang, but that some language in this design spot (actors + local mutable data) makes much sense. Regards, Paulo From tonyhannan2@REDACTED Thu Nov 4 14:32:35 2010 From: tonyhannan2@REDACTED (Tony Hannan) Date: Thu, 4 Nov 2010 09:32:35 -0400 Subject: Actor garbage collection Message-ID: Hello, Does an orphaned waiting process get garbage collected? By orphaned I mean no one references it (no one has its pid) so it will never receive anymore messages. By waiting I mean it is waiting for a message inside a receive. For example, Wait = fun() -> receive _ -> 0 end end. spawn(Wait). Will that spawned process live and wait forever even though we did not capture its pid and thus can't send messages to it, or will it be garbage collected eventually? Thanks, Tony From dang@REDACTED Thu Nov 4 14:33:49 2010 From: dang@REDACTED (Daniel Goertzen) Date: Thu, 4 Nov 2010 08:33:49 -0500 Subject: [erlang-questions] Conceptual questions on architecture of functional programs In-Reply-To: <20101104012322.GA375@hope.tifa.renegado> References: <20101104012322.GA375@hope.tifa.renegado> Message-ID: For learning FP, "Yet Another Haskell Tutorial" is excellent. It goes over the fundamental concepts of FP and has a lot of exercises. It is in Haskell, not Erlang, but for basic FP the languages are very similar. It sounds like you've dabbled in Erlang a bit already, so you should know when you get into Haskell specific territory (ie, monads). http://www.cs.utah.edu/~hal/docs/daume02yaht.pdf Dan. From mk@REDACTED Thu Nov 4 14:49:53 2010 From: mk@REDACTED (Morten Krogh) Date: Thu, 04 Nov 2010 14:49:53 +0100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <4CD29E67.3030609@di.uminho.pt> References: <415901635.37311288867568017.JavaMail.root@zimbra> <4CD29E67.3030609@di.uminho.pt> Message-ID: <4CD2BA01.3000806@amberbio.com> Hi I agree with Paulo and others with the same view. It is like Erlang wants to solve the concurrency problems twice. First by splitting things up into processes and then by having the processes use immutable data. The message passing processes are enough to secure consistency, if the messages themselves are fully copied. The rationale for mutable data is of course mostly performance in space and time. But actually in some cases the code is even nicer with mutable data. Here is a simple test between the process dictionary and a dict. -module(hash). -compile(export_all). insert(0) -> ok; insert(N) -> put(N,N), insert(N-1). dinsert(D, 0) -> ok; dinsert(D, N) -> D2 = dict:store(N, N, D), dinsert(D2, N-1). 1> timer:tc(hash, dinsert, [dict:new(), 3000000]). {481300522,ok} 2> timer:tc(hash, insert, [3000000]). {658345,ok} Exceptions should of course not occur with mutable data, but mutavble data is mostly suited for "databases", where all operations should be well tested "stored procedures". The clojure transients seem interesting. I just don't follow how they can have high performance without copying the data to start with. Suppose we start with map M1 then M2 = M1 M2 transient, do stuff, persistent. M3 = M2. M3 transient, do stuff, persistent. M4 = M1. M4 transient.... M1 transient .... How can the implemntation keep track of this without disturbing any old data, and not have a very high overhead. It seems like all pieces of data within the structure must be reference counted or something like that. I suspect performance of this construct might have a very nasty worst case behaviour. Does anyone know how this is implemented? Also, I will claim that the by far most common pattern is this M1 M2 = M1 + small change. discard M1. Mutable data structures are made for that. Cheers, Morten. On 11/4/10 12:52 PM, Paulo S?rgio Almeida wrote: > Hi Robert, > > there is a point on which I do not agree; when you say: > > On 11/4/10 10:46 AM, Robert Virding wrote: > >> - I think that having immutable data even within only one process is >> a big win. Having mutable data gives you all the problems of global >> data. Immutable data makes it much easier to keep track of what is >> happening with your data, which is always a Big Win. > > I would say that having mutable data gives us SOME problems, but much > LESS than when we have global mutable data. > > You are mixing the problems due to: > > - global data > - mutable data > > In most languages around we have the nightmare of mutable global data, > and the nasty problem of aliasing in object references. > > On the other hand in Erlang we have processes that give us locality, > but only immutable data in each process. > > I (and others) think there is a place for a middle-ground where > processes give us locality, allowing a divid-and-conquer of the global > problem into simple parts, where the data in a given process is > manageable, but where if the algorithm calls for mutable data then we > should be able to use it. All the nasty problems of shared memory > concurrency (from the programmer's point of view) have already > disappeared when we have only per-process mutable data as opposed to > global mutable data. > > If the data is immutable and only pure functions are used, we have > extra benefits, but not always can we afford it, nor is it necessary > even for proving correctness. Lot's of classic formal methods stuff > with pre-conditions, post-conditions, invariants, Hoare logic, etc, > were made for the mutable data world. They become difficult to apply > when we have global data and aliasing, but may well be applied when we > have just a little local data. > > The idea would be to be able to use the many proven algorithms that > were created for the imperative sequential world inside each process, > and let the process concept a la Erlang deal with the concurrency > aspects. > > I am not saying it is now practical to add it to Erlang, but that some > language in this design spot (actors + local mutable data) makes much > sense. > > Regards, > Paulo > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From gleber.p@REDACTED Thu Nov 4 14:52:41 2010 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 4 Nov 2010 14:52:41 +0100 Subject: [erlang-questions] Actor garbage collection In-Reply-To: References: Message-ID: No, AFAIK On Thu, Nov 4, 2010 at 14:32, Tony Hannan wrote: > Hello, > > Does an orphaned waiting process get garbage collected? By orphaned I mean > no one references it (no one has its pid) so it will never receive anymore > messages. By waiting I mean it is waiting for a message inside a receive. > For example, > > ? ?Wait = fun() -> receive _ -> 0 end end. > ? ?spawn(Wait). > > Will that spawned process live and wait forever even though we did not > capture its pid and thus can't send messages to it, or will it be garbage > collected eventually? > > Thanks, > Tony > From jiansenhe@REDACTED Thu Nov 4 15:09:43 2010 From: jiansenhe@REDACTED (Jiansen He) Date: Thu, 4 Nov 2010 14:09:43 +0000 Subject: [erlang-questions] MPI based BST and AVL Message-ID: Hi all, Have anyone tried to implement binary search tree (BST) and AVL based on MPI? I intuitively feel that operations (insertion, deletion and traversal) on left subtree will not affect the operations on right subtree. I'm working on a program which I though would be benefit from concurrent insertions to a binary tree. Before I spend times to implement myself, I would be appreciate if anyone could offer me a sophisticate implementation. Beast Regards Jiansen From spawn.think@REDACTED Thu Nov 4 15:13:15 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 4 Nov 2010 15:13:15 +0100 Subject: [erlang-questions] Actor garbage collection In-Reply-To: References: Message-ID: This process will be waiting forever, until it gets a message (or get an exit signal). It doesn't matter if "we" have the pid or not. It's still a living process. (btw, all living pids still can be listed using erlang:processes/0) A process will terminate when it has no more code to execute or when a run-time error occurs. http://www.erlang.org/doc/reference_manual/processes.html#id78133 Garbage collection has to do with the memory used by the process, not with the process life. On Thu, Nov 4, 2010 at 2:52 PM, Gleb Peregud wrote: > No, AFAIK > > On Thu, Nov 4, 2010 at 14:32, Tony Hannan wrote: > > Hello, > > > > Does an orphaned waiting process get garbage collected? By orphaned I > mean > > no one references it (no one has its pid) so it will never receive > anymore > > messages. By waiting I mean it is waiting for a message inside a receive. > > For example, > > > > Wait = fun() -> receive _ -> 0 end end. > > spawn(Wait). > > > > Will that spawned process live and wait forever even though we did not > > capture its pid and thus can't send messages to it, or will it be garbage > > collected eventually? > > > > Thanks, > > Tony > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From silasdb@REDACTED Thu Nov 4 15:15:10 2010 From: silasdb@REDACTED (Silas Silva) Date: Thu, 4 Nov 2010 12:15:10 -0200 Subject: [erlang-questions] Conceptual questions on architecture of functional programs In-Reply-To: References: <20101104012322.GA375@hope.tifa.renegado> Message-ID: <20101104141508.GA13998@bsoares.omnisys.int.com.br> On Thu, Nov 04, 2010 at 09:56:42AM +0800, ????????? (Yau-Hsien Huang) wrote: > Hi, Silas. > > There's an introductory paper "Why Functional Programming Matters" for you. > > In functional fashion, you can make second-order, third-order and n-th-order > functions > when you pass a name of a first-order function into some function. Thank you very much. I just read a half of this paper and I noted it is a very interesting introductory way to get in touch with FP. Although it helped me with some concepts (nth-order functions, lazy evaluation), I'm still looking for some example about how to architecture big programs in a functional programming language. Anyway, I'll read more (not only this paper, but other famous references as well) and try to get some examples from the open source world. Thank you very much again! -- Silas Silva From silasdb@REDACTED Thu Nov 4 15:15:56 2010 From: silasdb@REDACTED (Silas Silva) Date: Thu, 4 Nov 2010 12:15:56 -0200 Subject: [erlang-questions] Conceptual questions on architecture of functional programs In-Reply-To: References: <20101104012322.GA375@hope.tifa.renegado> Message-ID: <20101104141554.GB13998@bsoares.omnisys.int.com.br> On Thu, Nov 04, 2010 at 09:29:58AM +0100, Ahmed Omar wrote: > I would recommend this one: > http://www.defmacro.org/ramblings/fp.html Nice! It gives examples in Java, which is more familiar to the rest of us, mortals :-) Thanks. -- Silas Silva From silasdb@REDACTED Thu Nov 4 15:17:29 2010 From: silasdb@REDACTED (Silas Silva) Date: Thu, 4 Nov 2010 12:17:29 -0200 Subject: [erlang-questions] Conceptual questions on architecture of functional programs In-Reply-To: References: <20101104012322.GA375@hope.tifa.renegado> Message-ID: <20101104141725.GC13998@bsoares.omnisys.int.com.br> On Thu, Nov 04, 2010 at 08:33:49AM -0500, Daniel Goertzen wrote: > For learning FP, "Yet Another Haskell Tutorial" is excellent. It goes over > the fundamental concepts of FP and has a lot of exercises. It is in > Haskell, not Erlang, but for basic FP the languages are very similar. It > sounds like you've dabbled in Erlang a bit already, so you should know when > you get into Haskell specific territory (ie, monads). > > http://www.cs.utah.edu/~hal/docs/daume02yaht.pdf Thank you very much. It is interesting, but a bit theoretical, not? Also, The Real World Haskell book seems very interesting too: http://book.realworldhaskell.org/read/ Thank you again. -- Silas Silva From robert.virding@REDACTED Thu Nov 4 15:19:31 2010 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 4 Nov 2010 14:19:31 +0000 (GMT) Subject: [erlang-questions] Actor garbage collection In-Reply-To: <700813230.38711288880270326.JavaMail.root@zimbra> Message-ID: <578412877.38761288880371329.JavaMail.root@zimbra> Hi, No, processes are never removed until they are explicitly killed. Even if there is no reference to it anywhere there are still ways to get hold of it, for example by calling processes/0 or list_to_pid/1 (very naughty). Also in a distributed node then other nodes may contain references to a process even if there are no local references. There are some things in Erlang which are not collected, for example pids, ets tables and modules. Robert ----- "Tony Hannan" wrote: > Hello, > > Does an orphaned waiting process get garbage collected? By orphaned I > mean > no one references it (no one has its pid) so it will never receive > anymore > messages. By waiting I mean it is waiting for a message inside a > receive. > For example, > > Wait = fun() -> receive _ -> 0 end end. > spawn(Wait). > > Will that spawned process live and wait forever even though we did > not > capture its pid and thus can't send messages to it, or will it be > garbage > collected eventually? > > Thanks, > Tony From robert.virding@REDACTED Thu Nov 4 15:31:50 2010 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 4 Nov 2010 14:31:50 +0000 (GMT) Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <1004528757.38871288880851500.JavaMail.root@zimbra> Message-ID: <1639101360.38891288881110030.JavaMail.root@zimbra> Hi Paulo, I was referring more here to global *within* a process, not between processes. Sorry for being unclear. If you allow mutable data within a process then it will necessarily be global within that process. Even if I have to pass a reference to it I will still have no control over who or when it mutated. Things can happen under my feet without my knowing about it. This is the same problem as for sequential languages which have mutable data. Allowing non-sharing between processes would lessen some of the problems, but not wholly remove them. Also smart garbage collection is easier if you have immutable data. I know this is an implementation detail, but immutable data makes the garbage collector simpler and more efficient, you can do smart things if you know data does not change. Robert ----- "Paulo S?rgio Almeida" wrote: > Hi Robert, > > there is a point on which I do not agree; when you say: > > On 11/4/10 10:46 AM, Robert Virding wrote: > > > - I think that having immutable data even within only one process is > a big win. Having mutable data gives you all the problems of global > data. Immutable data makes it much easier to keep track of what is > happening with your data, which is always a Big Win. > > I would say that having mutable data gives us SOME problems, but much > > LESS than when we have global mutable data. > > You are mixing the problems due to: > > - global data > - mutable data > > In most languages around we have the nightmare of mutable global data, > > and the nasty problem of aliasing in object references. > > On the other hand in Erlang we have processes that give us locality, > but > only immutable data in each process. > > I (and others) think there is a place for a middle-ground where > processes give us locality, allowing a divid-and-conquer of the global > > problem into simple parts, where the data in a given process is > manageable, but where if the algorithm calls for mutable data then we > > should be able to use it. All the nasty problems of shared memory > concurrency (from the programmer's point of view) have already > disappeared when we have only per-process mutable data as opposed to > global mutable data. > > If the data is immutable and only pure functions are used, we have > extra > benefits, but not always can we afford it, nor is it necessary even > for > proving correctness. Lot's of classic formal methods stuff with > pre-conditions, post-conditions, invariants, Hoare logic, etc, were > made > for the mutable data world. They become difficult to apply when we > have > global data and aliasing, but may well be applied when we have just a > > little local data. > > The idea would be to be able to use the many proven algorithms that > were > created for the imperative sequential world inside each process, and > let > the process concept a la Erlang deal with the concurrency aspects. > > I am not saying it is now practical to add it to Erlang, but that some > > language in this design spot (actors + local mutable data) makes much > sense. > > Regards, > Paulo From tonyhannan2@REDACTED Thu Nov 4 15:46:43 2010 From: tonyhannan2@REDACTED (Tony Hannan) Date: Thu, 4 Nov 2010 10:46:43 -0400 Subject: [erlang-questions] Actor garbage collection In-Reply-To: <578412877.38761288880371329.JavaMail.root@zimbra> References: <700813230.38711288880270326.JavaMail.root@zimbra> <578412877.38761288880371329.JavaMail.root@zimbra> Message-ID: Thanks guys. It would be nice, although I know its hard in a distributed setting, but there is research on it. See http://scholar.google.com/scholar?q=distributed+actor+garbage+collection Cheers, Tony On Thu, Nov 4, 2010 at 10:19 AM, Robert Virding < robert.virding@REDACTED> wrote: > Hi, > > No, processes are never removed until they are explicitly killed. Even if > there is no reference to it anywhere there are still ways to get hold of it, > for example by calling processes/0 or list_to_pid/1 (very naughty). Also in > a distributed node then other nodes may contain references to a process even > if there are no local references. > > There are some things in Erlang which are not collected, for example pids, > ets tables and modules. > > Robert > > ----- "Tony Hannan" wrote: > > > Hello, > > > > Does an orphaned waiting process get garbage collected? By orphaned I > > mean > > no one references it (no one has its pid) so it will never receive > > anymore > > messages. By waiting I mean it is waiting for a message inside a > > receive. > > For example, > > > > Wait = fun() -> receive _ -> 0 end end. > > spawn(Wait). > > > > Will that spawned process live and wait forever even though we did > > not > > capture its pid and thus can't send messages to it, or will it be > > garbage > > collected eventually? > > > > Thanks, > > Tony > From mk@REDACTED Thu Nov 4 15:54:28 2010 From: mk@REDACTED (Morten Krogh) Date: Thu, 04 Nov 2010 15:54:28 +0100 Subject: ownership of an ets table Message-ID: <4CD2C924.6090905@amberbio.com> Hi Can an ets table not live without an owner? Can't public ets tables, at least, live witout an owner? How do you do when you want a long lived public table? Do you create a dummy process just because of this? regards, Morten. From jiansenhe@REDACTED Thu Nov 4 16:07:15 2010 From: jiansenhe@REDACTED (Jiansen He) Date: Thu, 4 Nov 2010 15:07:15 +0000 Subject: [erlang-questions] MPI based BST and AVL In-Reply-To: References: Message-ID: Perhaps my question is a bit misleading. The implementation I am interested in is building a BST and AVL based on actor models which employs message passing style concurrent programming. Jiansen From spawn.think@REDACTED Thu Nov 4 16:47:38 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 4 Nov 2010 16:47:38 +0100 Subject: [erlang-questions] ownership of an ets table In-Reply-To: <4CD2C924.6090905@amberbio.com> References: <4CD2C924.6090905@amberbio.com> Message-ID: Hi No. Public has nothing to do with ownership, it has to do with permissions. You can just make sure the process owns the table won't crash. Also you can use the option *heir* to transfer ownership at that process termination. http://www.erlang.org/doc/man/ets.html#heir On Thu, Nov 4, 2010 at 3:54 PM, Morten Krogh wrote: > Hi > > Can an ets table not live without an owner? > Can't public ets tables, at least, live witout an owner? > > How do you do when you want a long lived public table? Do you create a > dummy process just because of this? > > regards, > > Morten. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From tony.arcieri@REDACTED Thu Nov 4 17:38:02 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Thu, 4 Nov 2010 10:38:02 -0600 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <842437213.35451288831826841.JavaMail.root@zimbra> References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> Message-ID: On Wed, Nov 3, 2010 at 6:50 PM, Robert Virding < robert.virding@REDACTED> wrote: > 1. Shared/hybrid heaps are wonderful BUT: > - They are more difficult to implement in a real-time environment. Yes, > this is a known problem with solutions but the benefits are less than you > would expect. Real-time gc costs. > - They don't port easily to multi-core systems as you suddenly start > needing locks and stuff everywhere to control access or have complex schemes > to get around this. Which copying doesn't. > There's been a considerable amount of research on this matter on the Java platform. The Azul Java platform addresses both of these concerns, with a pauseless GC designed for massively multi-core systems (54 cores per CPU): http://www.azulsystems.com/products/vega/overview I'd be curious how well Erjang would work on such a system, particularly on the types of shared state concurrency problems that don't work well on BEAM, like Chameneos. -- Tony Arcieri Medioh! A Kudelski Brand From gleber.p@REDACTED Thu Nov 4 18:31:06 2010 From: gleber.p@REDACTED (Gleb Peregud) Date: Thu, 4 Nov 2010 18:31:06 +0100 Subject: [erlang-questions] ownership of an ets table In-Reply-To: References: <4CD2C924.6090905@amberbio.com> Message-ID: I haven't tried it myself, but you may try to give ownership to whereis(init) process, which runs forever while node lives. On Thu, Nov 4, 2010 at 16:47, Ahmed Omar wrote: > Hi > No. Public has nothing to do with ownership, it has to do with permissions. > You can just make sure the process owns the table won't crash. Also you can > use the option *heir* to transfer ownership at that process termination. > http://www.erlang.org/doc/man/ets.html#heir > On Thu, Nov 4, 2010 at 3:54 PM, Morten Krogh wrote: > >> Hi >> >> Can an ets table not live without an owner? >> Can't public ets tables, at least, live witout an owner? >> >> How do you do when you want a long lived public table? Do you create a >> dummy process just because of this? >> >> regards, >> >> Morten. >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > -- > Best Regards, > - Ahmed Omar > http://nl.linkedin.com/in/adiaa > Follow me on twitter > @spawn_think > From mk@REDACTED Thu Nov 4 18:56:57 2010 From: mk@REDACTED (Morten Krogh) Date: Thu, 04 Nov 2010 18:56:57 +0100 Subject: [erlang-questions] ownership of an ets table In-Reply-To: References: <4CD2C924.6090905@amberbio.com> Message-ID: <4CD2F3E9.6080804@amberbio.com> Hi Thanks for your answers. Ahmed, the reason I singled out public tables was because tables with public access might naturally live without an owner. Private tables are pretty useless when their owner dies, but public tables could be on their own, I would say. One could have an option in the ets:new to create a table without an owner. Why not? Morten. On 11/4/10 6:31 PM, Gleb Peregud wrote: > I haven't tried it myself, but you may try to give ownership to > whereis(init) process, which runs forever while node lives. > > On Thu, Nov 4, 2010 at 16:47, Ahmed Omar wrote: >> Hi >> No. Public has nothing to do with ownership, it has to do with permissions. >> You can just make sure the process owns the table won't crash. Also you can >> use the option *heir* to transfer ownership at that process termination. >> http://www.erlang.org/doc/man/ets.html#heir >> On Thu, Nov 4, 2010 at 3:54 PM, Morten Krogh wrote: >> >>> Hi >>> >>> Can an ets table not live without an owner? >>> Can't public ets tables, at least, live witout an owner? >>> >>> How do you do when you want a long lived public table? Do you create a >>> dummy process just because of this? >>> >>> regards, >>> >>> Morten. >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >>> >> >> -- >> Best Regards, >> - Ahmed Omar >> http://nl.linkedin.com/in/adiaa >> Follow me on twitter >> @spawn_think >> > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From alessandro.sivieri@REDACTED Thu Nov 4 19:17:38 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Thu, 4 Nov 2010 19:17:38 +0100 Subject: Curiosity about MochiWeb (no questions this time) Message-ID: Hi all, I was looking to an appmon instance launched from a node with MochiWeb active, and I saw a few processes linked to the _web process, so I was wondering if MochiWeb generates something like a pool of processes for client requests, which may explain why there were all those processes... -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From krab@REDACTED Thu Nov 4 19:29:41 2010 From: krab@REDACTED (Kresten Krab Thorup) Date: Thu, 4 Nov 2010 19:29:41 +0100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> Message-ID: <0933BBC4-4F98-4752-B90A-9407C821F98C@trifork.com> Yes, it would be interesting to know how erjang behaves in such an environment. Have been talking to Gil Tene (Azul CTO) here at QCon about doing that; but we need some system to test that is sufficiently complex, runs on erjang, and who's behavior is known so we can make relevant evaluation. Any takers? I can help connect people and make this happen. Kresten On 04/11/2010, at 09.39, "Tony Arcieri" wrote: > On Wed, Nov 3, 2010 at 6:50 PM, Robert Virding < > robert.virding@REDACTED> wrote: > >> 1. Shared/hybrid heaps are wonderful BUT: >> - They are more difficult to implement in a real-time environment. Yes, >> this is a known problem with solutions but the benefits are less than you >> would expect. Real-time gc costs. >> - They don't port easily to multi-core systems as you suddenly start >> needing locks and stuff everywhere to control access or have complex schemes >> to get around this. Which copying doesn't. >> > > There's been a considerable amount of research on this matter on the Java > platform. The Azul Java platform addresses both of these concerns, with a > pauseless GC designed for massively multi-core systems (54 cores per CPU): > > http://www.azulsystems.com/products/vega/overview > > I'd be curious how well Erjang would work on such a system, particularly on > the types of shared state concurrency problems that don't work well on BEAM, > like Chameneos. > > -- > Tony Arcieri > Medioh! A Kudelski Brand From aj@REDACTED Thu Nov 4 22:17:05 2010 From: aj@REDACTED (AJ Heller) Date: Thu, 4 Nov 2010 14:17:05 -0700 Subject: [erlang-questions] Curiosity about MochiWeb (no questions this time) In-Reply-To: References: Message-ID: Mochiweb spawns a pool of short-lived processes (from the `mochiweb_acceptor` module), and re-spawns new acceptor processes when each finishes/times-out. The default settings are: 16 processes are alive at a time, and they timeout after 2 seconds. I did some analysis of the Mochiweb code a while back. There's a diagram at the bottom of the page that lays out the program flow. http://drfloob.com/wiki/erlang_mochiweb_walkthrough.html --aj On Thu, Nov 4, 2010 at 11:17 AM, Alessandro Sivieri wrote: > Hi all, > > I was looking to an appmon instance launched from a node with MochiWeb > active, and I saw a few processes linked to the _web process, so I was > wondering if MochiWeb generates something like a pool of processes for > client requests, which may explain why there were all those processes... > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://www.chimera-bellerofonte.eu/ > http://www.poul.org/ > From erlang@REDACTED Thu Nov 4 22:45:17 2010 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 4 Nov 2010 22:45:17 +0100 Subject: [erlang-questions] Actor garbage collection In-Reply-To: References: <700813230.38711288880270326.JavaMail.root@zimbra> <578412877.38761288880371329.JavaMail.root@zimbra> Message-ID: On Thu, Nov 4, 2010 at 3:46 PM, Tony Hannan wrote: > Thanks guys. It would be nice, although I know its hard in a distributed > setting, but there is research on it. See > http://scholar.google.com/scholar?q=distributed+actor+garbage+collection The main problem in a distributed setting is node failure - if you have dangling references (ie references to nodes that have failed) things become *incredibly* difficult - we realised this in 1985 :-) - that's why erlang processes copy messages between processes - so that remote processes have no dangling data references. Now the problem is no longer hard but reduces to local node collections. /Joe > > Cheers, > Tony > > On Thu, Nov 4, 2010 at 10:19 AM, Robert Virding < > robert.virding@REDACTED> wrote: > >> Hi, >> >> No, processes are never removed until they are explicitly killed. Even if >> there is no reference to it anywhere there are still ways to get hold of it, >> for example by calling processes/0 or list_to_pid/1 (very naughty). Also in >> a distributed node then other nodes may contain references to a process even >> if there are no local references. >> >> There are some things in Erlang which are not collected, for example pids, >> ets tables and modules. >> >> Robert >> >> ----- "Tony Hannan" wrote: >> >> > Hello, >> > >> > Does an orphaned waiting process get garbage collected? By orphaned I >> > mean >> > no one references it (no one has its pid) so it will never receive >> > anymore >> > messages. By waiting I mean it is waiting for a message inside a >> > receive. >> > For example, >> > >> > ? ? Wait = fun() -> receive _ -> 0 end end. >> > ? ? spawn(Wait). >> > >> > Will that spawned process live and wait forever even though we did >> > not >> > capture its pid and thus can't send messages to it, or will it be >> > garbage >> > collected eventually? >> > >> > Thanks, >> > Tony >> > From erlang@REDACTED Thu Nov 4 23:01:30 2010 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 4 Nov 2010 23:01:30 +0100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <20101102201437.GC626@hope.tifa.renegado> References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: > This is a message I sent to the nosql-discussion@REDACTED > discussion group. ?I thought it might be interesting to send it > erlang-questions, so here we go... > > > Hi all! > > I have used SQL RDBMSs for some time. ?I've never used any very advanced > feature, but I know enough of it to make database applications. > > Nowadays, I decided it would be interesting to learn some NoSQL > databases concepts. ?So I decided to pick up some Erlang and Mnesia, its > native key-value database. ?More than scalability itself, the most > valuable feature for me is the possibility of replication and > synchronization between nodes. > > But all pros have cons as companion. ?The lack of a relationship model > is difficult for who is used to RDBMSs. ?So, my question is: > > * Is there any guide, tutorial, book, whatever, that tries to introduce > ?NoSQL databases to SQL users? > > * Key-value databases are surprising simple. ?I know you solve > ?relationship by denormalizing data. ?What data should be normalized? > ?What shouldn't? ?How do you update denormalized data? I'm no database expert so don't quote me here ... As far as I am concerned traditional databases like SQL suffer from the fact that the data stored in an individual column is incredible simple - I can store an integer/string/... in a column but these are incredibly simple data structures. I want to store and retrieve incredibly complicated things - how do I store an XML parse tree in a single cell of a database? - How do I store a database in a database ... In a decent K-V database the value of the key can be *anything* - an entire database for example, a compiler, ... no worries Then when I analyse my problem I start thinking "I can store and retrieve any complex object that keys do I need to solve my problem?" I'm not thinking in terms of joins and normalizing things - the thought process is different - so far I haven't met any problems that don't map onto key-values queries. It seems to my that SQL provides you with the ability do to complex queries on simple things. K-V dbs can do simple queries on complex things. /Joe > > Sorry for such simple and general questions. ?Things were simple up to > the moment that I realized that it would be easily solved with a JOIN > SQL statement. ?:-) > > Thank you very much! > > -- > Silas Silva > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From spawn.think@REDACTED Thu Nov 4 23:11:39 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 4 Nov 2010 23:11:39 +0100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: Hi Joe, All I'm far from being a database expert too, so feel free to correct me , but i believe the assumption of storing simple thing is not a general rule and should not be associated with SQL. For example, postgresql provides more complex data types and even user defined ones. On Thu, Nov 4, 2010 at 11:01 PM, Joe Armstrong wrote: > On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: > > This is a message I sent to the nosql-discussion@REDACTED > > discussion group. I thought it might be interesting to send it > > erlang-questions, so here we go... > > > > > > Hi all! > > > > I have used SQL RDBMSs for some time. I've never used any very advanced > > feature, but I know enough of it to make database applications. > > > > Nowadays, I decided it would be interesting to learn some NoSQL > > databases concepts. So I decided to pick up some Erlang and Mnesia, its > > native key-value database. More than scalability itself, the most > > valuable feature for me is the possibility of replication and > > synchronization between nodes. > > > > But all pros have cons as companion. The lack of a relationship model > > is difficult for who is used to RDBMSs. So, my question is: > > > > * Is there any guide, tutorial, book, whatever, that tries to introduce > > NoSQL databases to SQL users? > > > > * Key-value databases are surprising simple. I know you solve > > relationship by denormalizing data. What data should be normalized? > > What shouldn't? How do you update denormalized data? > > I'm no database expert so don't quote me here ... > > As far as I am concerned traditional databases like SQL suffer > from the fact that the data stored in an individual column is incredible > simple - I can store an integer/string/... in a column but these are > incredibly simple data structures. I want to store and retrieve incredibly > complicated things - how do I store an XML parse tree in a single cell > of a database? - How do I store a database in a database ... > > In a decent K-V database the value of the key can be *anything* - an > entire database for example, a compiler, ... no worries > > Then when I analyse my problem I start thinking "I can store and retrieve > any > complex object that keys do I need to solve my problem?" > > I'm not thinking in terms of joins and normalizing things - the thought > process > is different - so far I haven't met any problems that don't map onto > key-values > queries. > > It seems to my that SQL provides you with the ability do to complex > queries on simple things. K-V dbs can do simple queries on complex > things. > > /Joe > > > > > > > Sorry for such simple and general questions. Things were simple up to > > the moment that I realized that it would be easily solved with a JOIN > > SQL statement. :-) > > > > Thank you very much! > > > > -- > > Silas Silva > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From tony.arcieri@REDACTED Thu Nov 4 23:17:10 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Thu, 4 Nov 2010 16:17:10 -0600 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <0933BBC4-4F98-4752-B90A-9407C821F98C@trifork.com> References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> <0933BBC4-4F98-4752-B90A-9407C821F98C@trifork.com> Message-ID: Hi Kresten, I'd be very interested to know how Chameneos performs under Erjang on an Azul system, although a comparison with BEAM might prove difficult: http://shootout.alioth.debian.org/u32q/performance.php?test=chameneosredux It seems like the type of problem that could really benefit from Erjang and its shared heap. On Thu, Nov 4, 2010 at 12:29 PM, Kresten Krab Thorup wrote: > Yes, it would be interesting to know how erjang behaves in such an > environment. Have been talking to Gil Tene (Azul CTO) here at QCon about > doing that; but we need some system to test that is sufficiently complex, > runs on erjang, and who's behavior is known so we can make relevant > evaluation. Any takers? I can help connect people and make this happen. > > Kresten > > > On 04/11/2010, at 09.39, "Tony Arcieri" wrote: > > > On Wed, Nov 3, 2010 at 6:50 PM, Robert Virding < > > robert.virding@REDACTED> wrote: > > > >> 1. Shared/hybrid heaps are wonderful BUT: > >> - They are more difficult to implement in a real-time environment. Yes, > >> this is a known problem with solutions but the benefits are less than > you > >> would expect. Real-time gc costs. > >> - They don't port easily to multi-core systems as you suddenly start > >> needing locks and stuff everywhere to control access or have complex > schemes > >> to get around this. Which copying doesn't. > >> > > > > There's been a considerable amount of research on this matter on the Java > > platform. The Azul Java platform addresses both of these concerns, with a > > pauseless GC designed for massively multi-core systems (54 cores per > CPU): > > > > http://www.azulsystems.com/products/vega/overview > > > > I'd be curious how well Erjang would work on such a system, particularly > on > > the types of shared state concurrency problems that don't work well on > BEAM, > > like Chameneos. > > > > -- > > Tony Arcieri > > Medioh! A Kudelski Brand > -- Tony Arcieri Medioh! A Kudelski Brand From tony.arcieri@REDACTED Thu Nov 4 23:17:43 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Thu, 4 Nov 2010 16:17:43 -0600 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> <0933BBC4-4F98-4752-B90A-9407C821F98C@trifork.com> Message-ID: Specifically here is the Erlang version: http://shootout.alioth.debian.org/u32q/program.php?test=chameneosredux&lang=hipe&id=1 On Thu, Nov 4, 2010 at 4:17 PM, Tony Arcieri wrote: > Hi Kresten, > > I'd be very interested to know how Chameneos performs under Erjang on an > Azul system, although a comparison with BEAM might prove difficult: > > http://shootout.alioth.debian.org/u32q/performance.php?test=chameneosredux > > > It > seems like the type of problem that could really benefit from Erjang and its > shared heap. > > > On Thu, Nov 4, 2010 at 12:29 PM, Kresten Krab Thorup wrote: > >> Yes, it would be interesting to know how erjang behaves in such an >> environment. Have been talking to Gil Tene (Azul CTO) here at QCon about >> doing that; but we need some system to test that is sufficiently complex, >> runs on erjang, and who's behavior is known so we can make relevant >> evaluation. Any takers? I can help connect people and make this happen. >> >> Kresten >> >> >> On 04/11/2010, at 09.39, "Tony Arcieri" wrote: >> >> > On Wed, Nov 3, 2010 at 6:50 PM, Robert Virding < >> > robert.virding@REDACTED> wrote: >> > >> >> 1. Shared/hybrid heaps are wonderful BUT: >> >> - They are more difficult to implement in a real-time environment. Yes, >> >> this is a known problem with solutions but the benefits are less than >> you >> >> would expect. Real-time gc costs. >> >> - They don't port easily to multi-core systems as you suddenly start >> >> needing locks and stuff everywhere to control access or have complex >> schemes >> >> to get around this. Which copying doesn't. >> >> >> > >> > There's been a considerable amount of research on this matter on the >> Java >> > platform. The Azul Java platform addresses both of these concerns, with >> a >> > pauseless GC designed for massively multi-core systems (54 cores per >> CPU): >> > >> > http://www.azulsystems.com/products/vega/overview >> > >> > I'd be curious how well Erjang would work on such a system, particularly >> on >> > the types of shared state concurrency problems that don't work well on >> BEAM, >> > like Chameneos. >> > >> > -- >> > Tony Arcieri >> > Medioh! A Kudelski Brand >> > > > > -- > Tony Arcieri > Medioh! A Kudelski Brand > -- Tony Arcieri Medioh! A Kudelski Brand From ok@REDACTED Thu Nov 4 23:41:58 2010 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 5 Nov 2010 11:41:58 +1300 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <4CD29E67.3030609@di.uminho.pt> References: <415901635.37311288867568017.JavaMail.root@zimbra> <4CD29E67.3030609@di.uminho.pt> Message-ID: On 5/11/2010, at 12:52 AM, Paulo S?rgio Almeida wrote: > Hi Robert, > > there is a point on which I do not agree; when you say: > > On 11/4/10 10:46 AM, Robert Virding wrote: > >> - I think that having immutable data even within only one process is a big win. Having mutable data gives you all the problems of global data. Immutable data makes it much easier to keep track of what is happening with your data, which is always a Big Win. > > I would say that having mutable data gives us SOME problems, but much LESS than when we have global mutable data. The difficulty is with aliasing. Data don't have to be global to be aliased. It *is* possible to have a language that looks and feels like an imperative language but which is largely free of such problems. The original design of Euclid springs to mind. Quoting "Early Experiences with Euclid": In Euclid it is illegal to have two identifiers that both name the same variable in a scope. This means, for example, that it is illegal to call a procedure with a variable actual parameter which is also imported by the procedure, or for any two variable actual parameters to a procedure to specify the same variable. The imports clause makes it easy for the compiler to check for non-aliasing. The non-aliasing rule effectively eliminates a large class of particularly insidious programming errors. A common example is the accidental passing of an element of an array to a procedure which changes the same array directly. In this situation the value of the actual parameter changes unexpectedly as a result of an assignment to an element of the global array. This kind of bug can be very costly in debugging time, as we discovered early in the compiler project. In the bootstrap and testing phase of the project, a particularly puzzling bug was encountered. An entry in the compiler's symbol table was being changed without ever being assigned to (as was verified by a trace). This problem, which was due to an accidental parameter alias, took almost a week of programmer effort to find. The preliminary version of the compiler which we were then using did not implement non-alias checking. Had it been implemented, the compiler would have detected the potential problem when the program was first compiled and thereby saved us a full week of programmer time. Note that this was nothing to do with *globality* of data, just the combination of mutability and sharing. Now here's the cute thing. There was a Xerox paper that showed that the constraints on aliasing that made Euclid safe meant that it could be read as syntactic sugar for a functional language! > I (and others) think there is a place for a middle-ground where processes give us locality, allowing a divid-and-conquer of the global problem into simple parts, where the data in a given process is manageable, but where if the algorithm calls for mutable data then we should be able to use it. All the nasty problems of shared memory concurrency (from the programmer's point of view) have already disappeared when we have only per-process mutable data as opposed to global mutable data. The nasty problems of shared memory *concurrency* disappear, true; the nasty problems of shared memory *mutability* do not. If you want to add mutable data to an Erlang-like language, you had better have type checking and no-alias enforcement as good as Euclid had. You might well end up with an excellent language, superb for its job, but it would not (and should not) be Erlang any more. > The idea would be to be able to use the many proven algorithms that were created for the imperative sequential world inside each process, and let the process concept a la Erlang deal with the concurrency aspects. Come to think of it, this is not entirely unlike what Occam was (and thanks to free implementations like KROC, still is). Why not start from Occam and see what you can add in the way of trees and pattern matching? From raould@REDACTED Thu Nov 4 23:47:40 2010 From: raould@REDACTED (Raoul Duke) Date: Thu, 4 Nov 2010 15:47:40 -0700 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <415901635.37311288867568017.JavaMail.root@zimbra> <4CD29E67.3030609@di.uminho.pt> Message-ID: 2010/11/4 Richard O'Keefe : > The difficulty is with aliasing. like what Bertrand Meyer is looking into? http://lambda-the-ultimate.org/node/3780 http://bertrandmeyer.com/2010/01/21/the-theory-and-calculus-of-aliasing/ sincerely. From toby@REDACTED Thu Nov 4 23:57:22 2010 From: toby@REDACTED (Toby Thain) Date: Thu, 04 Nov 2010 18:57:22 -0400 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: <4CD33A52.4050402@telegraphics.com.au> On 04/11/10 6:11 PM, Ahmed Omar wrote: > Hi Joe, All > I'm far from being a database expert too, so feel free to correct me , but > i believe the assumption of storing simple thing is not a general rule and > should not be associated with SQL. For example, postgresql provides more > complex data types and even user defined ones. As Joe must be aware, it is a tenet of the relational model that column values be 'atomic' in the sense that they do not encode structure that would be more profitably decomposed into relations. --Toby > > >> On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >>> This is a message I sent to the nosql-discussion@REDACTED >>> discussion group. I thought it might be interesting to send it >>> erlang-questions, so here we go... >>> >>> >>> Hi all! >>> >>> I have used SQL RDBMSs for some time. ... >>> * Key-value databases are surprising simple. I know you solve >>> relationship by denormalizing data. What data should be normalized? >>> What shouldn't? How do you update denormalized data? >> > On Thu, Nov 4, 2010 at 11:01 PM, Joe Armstrong wrote: >> I'm no database expert so don't quote me here ... >> >> As far as I am concerned traditional databases like SQL suffer >> from the fact that the data stored in an individual column is incredible >> simple - I can store an integer/string/... in a column but these are >> incredibly simple data structures. I want to store and retrieve incredibly >> complicated things - how do I store an XML parse tree in a single cell >> of a database? - How do I store a database in a database ... >>... >> It seems to my that SQL provides you with the ability do to complex >> queries on simple things. K-V dbs can do simple queries on complex >> things. >> >> /Joe >> >> >> >>> >>> Sorry for such simple and general questions. Things were simple up to >>> the moment that I realized that it would be easily solved with a JOIN >>> SQL statement. :-) >>> >>> Thank you very much! >>> >>> -- >>> Silas Silva >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >>> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > From ok@REDACTED Fri Nov 5 00:26:31 2010 From: ok@REDACTED (Richard O'Keefe) Date: Fri, 5 Nov 2010 12:26:31 +1300 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: <0C3A5031-F504-4371-8560-9C47D76A8A31@cs.otago.ac.nz> On 5/11/2010, at 11:01 AM, Joe Armstrong wrote: > I'm no database expert so don't quote me here ... > > As far as I am concerned traditional databases like SQL suffer > from the fact that the data stored in an individual column is incredible > simple - I can store an integer/string/... in a column but these are > incredibly simple data structures. See Date & Darwen "The Third Manifesto". They are nearly as incandescently cross about SQL as E. F. Codd was. Indeed, one of their criteria for a good relational data base is "not SQL". The *relational model* makes sense with arbitrarily complex data values in attributes. (Date & Darwen explain how this statement is compatible with 1NF.) > I want to store and retrieve incredibly > complicated things - how do I store an XML parse tree in a single cell > of a database? - How do I store a database in a database ... There was a lot of work done on "nested relational databases" at RMIT. At least three (Atlas, Titan, Titan+) were developed. I have no idea what happened to them. For that matter, I don't know what Alan Kent is up to these days. One of the systems was used to hold and efficiently search large amounts of SGML data. I note that http://xml.apache.org/xindice/ describes "Apache Xindice ... a database designed from the ground up to store XML data". So SQL and K-V are not the only alternatives around. (I'm *sure* I remember another free XML database...) From robert.virding@REDACTED Fri Nov 5 01:14:49 2010 From: robert.virding@REDACTED (Robert Virding) Date: Fri, 5 Nov 2010 00:14:49 +0000 (GMT) Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: Message-ID: <18753784.40161288916089555.JavaMail.root@zimbra> ----- "Tony Arcieri" wrote: > There's been a considerable amount of research on this matter on the > Java platform. The Azul Java platform addresses both of these > concerns, with a pauseless GC designed for massively multi-core > systems (54 cores per CPU): > > > http://www.azulsystems.com/products/vega/overview But they do cheat a bit, it's hardware assisted so not directly comparable to running on general purpose hardware. > I'd be curious how well Erjang would work on such a system, > particularly on the types of shared state concurrency problems that > don't work well on BEAM, like Chameneos. But the results would be interesting. Robert From luismarianoguerra@REDACTED Fri Nov 5 02:18:34 2010 From: luismarianoguerra@REDACTED (Mariano Guerra) Date: Thu, 4 Nov 2010 22:18:34 -0300 Subject: [erlang-questions] Abstract Form for -type and -spec In-Reply-To: References: <08B68DBF-DB94-4292-A907-4A05B12CEE4D@erlang-solutions.com> Message-ID: On Tue, Oct 26, 2010 at 12:50 AM, Alisdair Sullivan wrote: > Is the abstract form for type and spec attributes (as returned by, for example, epp:parse_file/3) documented anywhere? What meaning does the list in the third position in -type specs have? Is it ever not an empty list? this type: -type ordset(X) :: [X]. generates this ast: {attribute,3,type,{ordset,{type,3,list,[{var,3,'X'}]},[{var,3,'X'}]}} that was the only type I could find in more than 600 type declarations taken from erlang otp: https://github.com/marianoguerra/efene/blob/master/test/files/type/all_types.erl hope that helps From krab@REDACTED Fri Nov 5 08:05:29 2010 From: krab@REDACTED (Kresten Krab Thorup) Date: Fri, 5 Nov 2010 08:05:29 +0100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> <0933BBC4-4F98-4752-B90A-9407C821F98C@trifork.com> Message-ID: If you're interested ... on my MacBook pro 2.53GHz Intel Core 2 Duo, I get the following numbers: [HiPE, erts-5.8] $ time erl -noshell -run chameneosredux main 6000000 real 1m16.766s user 1m34.237s sys 0m44.391s CPU load is ~ [95%, 95%] top reporting ~6MB real memory For some reason, BEAM (sans HiPE) is slightly faster real 1m9.116s user 1m25.769s sys 0m35.669s Now erjang, ... [Erjang, erts-5.8] $ time ej +S 1 -noshell -run chameneosredux main 6000000 real 0m46.366s user 0m47.458s sys 0m0.606s CPU load is ~[50%, 50%] top reporting ~75MB real memory java invoked using -server -Xmx15mb [15 mb heap] Increasing the heap size does not make Erjang go faster/slower, but ~10MB seems to be the working set size of the heap. I.e. setting the total heap size below that will put significant pressure on the GC. Erjang on 64-bit Java and a double the heap runs marginally faster. For reference, the plain "java -server version" http://shootout.alioth.debian.org/u32q/benchmark.php?test=chameneosredux&lang=java runs like this on my machine real 0m4.900s user 0m8.460s sys 0m0.098s So my machine runs ~ 1.6 x faster than the one used in the shootout. This should be very easy to try on an Azul box; but right now I think the bottleneck will be that my schedulers will end up competing for work in an dumb way. Running erjang +S 2 (i.e. one scheduler per core) makes erjang run at-par with BEAM/HiPE. Interestingly, the profile for the running Erjang looks like this, and for some reason the critical function EProc.execute is never JIT'ed. Will have to figure out why that is and there should be a potential huge win. [enabling the Java profiler slows it down a bit, which is why this is a profile of ~52 seconds :-] Flat profile of 51.83 secs (4452 total ticks): Thread-1 Interpreted + native Method 78.4% 3399 + 0 erjang.EProc.execute 1.4% 7 + 52 java.lang.Class.getDeclaredConstructors0 0.5% 0 + 23 java.util.zip.Inflater.inflateBytes 0.4% 18 + 0 java.lang.ClassLoader.defineClass1 0.3% 0 + 13 java.lang.Class.forName0 0.1% 0 + 5 java.util.zip.ZipFile.read 0.1% 5 + 0 erjang.m.erlang.ErlBif.apply 0.1% 2 + 3 java.security.AccessController.doPrivileged 0.1% 5 + 0 kilim.Fiber.togglePause Compiled + native Method 2.9% 51 + 74 erjang.m.chameneosredux.chameneosredux.chameneos__4 2.3% 83 + 17 kilim.Mailbox.untilHasMessages 2.0% 67 + 19 erjang.m.chameneosredux.chameneosredux.broker__1 1.6% 71 + 0 erjang.EHandle.send 1.6% 44 + 25 erjang.ERT.wait 1.6% 68 + 0 kilim.WorkerThread.run 0.8% 36 + 0 erjang.ERT.send 0.4% 17 + 0 erjang.m.erlang.ErlBif.apply 0.2% 10 + 0 erjang.m.chameneosredux.chameneosredux$FN_chameneos__4.go 0.2% 9 + 0 erjang.m.chameneosredux.chameneosredux.main__1 Kresten On Nov 4, 2010, at 23:17 , Tony Arcieri wrote: Hi Kresten, I'd be very interested to know how Chameneos performs under Erjang on an Azul system, although a comparison with BEAM might prove difficult: http://shootout.alioth.debian.org/u32q/performance.php?test=chameneosredux It seems like the type of problem that could really benefit from Erjang and its shared heap. On Thu, Nov 4, 2010 at 12:29 PM, Kresten Krab Thorup > wrote: Yes, it would be interesting to know how erjang behaves in such an environment. Have been talking to Gil Tene (Azul CTO) here at QCon about doing that; but we need some system to test that is sufficiently complex, runs on erjang, and who's behavior is known so we can make relevant evaluation. Any takers? I can help connect people and make this happen. Kresten On 04/11/2010, at 09.39, "Tony Arcieri" > wrote: > On Wed, Nov 3, 2010 at 6:50 PM, Robert Virding < > robert.virding@REDACTED> wrote: > >> 1. Shared/hybrid heaps are wonderful BUT: >> - They are more difficult to implement in a real-time environment. Yes, >> this is a known problem with solutions but the benefits are less than you >> would expect. Real-time gc costs. >> - They don't port easily to multi-core systems as you suddenly start >> needing locks and stuff everywhere to control access or have complex schemes >> to get around this. Which copying doesn't. >> > > There's been a considerable amount of research on this matter on the Java > platform. The Azul Java platform addresses both of these concerns, with a > pauseless GC designed for massively multi-core systems (54 cores per CPU): > > http://www.azulsystems.com/products/vega/overview > > I'd be curious how well Erjang would work on such a system, particularly on > the types of shared state concurrency problems that don't work well on BEAM, > like Chameneos. > > -- > Tony Arcieri > Medioh! A Kudelski Brand -- Tony Arcieri Medioh! A Kudelski Brand Kresten Krab Thorup, CTO, Trifork From pablo.platt@REDACTED Fri Nov 5 10:17:12 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Fri, 5 Nov 2010 02:17:12 -0700 (PDT) Subject: [erlang-questions] Curiosity about MochiWeb (no questions this time) In-Reply-To: References: Message-ID: <710633.85213.qm@web112605.mail.gq1.yahoo.com> I'm using Mochiweb for long-polling. Each request can take 60 seconds. I didn't change the default settings so how come I didn't have issues with the 2 seconds limit? Does it means that I won't be able to have more than 16 concurrent long-polling requests? ________________________________ From: AJ Heller To: Alessandro Sivieri Cc: Erlang Questions Sent: Thu, November 4, 2010 11:17:05 PM Subject: Re: [erlang-questions] Curiosity about MochiWeb (no questions this time) Mochiweb spawns a pool of short-lived processes (from the `mochiweb_acceptor` module), and re-spawns new acceptor processes when each finishes/times-out. The default settings are: 16 processes are alive at a time, and they timeout after 2 seconds. I did some analysis of the Mochiweb code a while back. There's a diagram at the bottom of the page that lays out the program flow. http://drfloob.com/wiki/erlang_mochiweb_walkthrough.html --aj On Thu, Nov 4, 2010 at 11:17 AM, Alessandro Sivieri wrote: > Hi all, > > I was looking to an appmon instance launched from a node with MochiWeb > active, and I saw a few processes linked to the _web process, so I was > wondering if MochiWeb generates something like a pool of processes for > client requests, which may explain why there were all those processes... > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://www.chimera-bellerofonte.eu/ > http://www.poul.org/ > ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From bob@REDACTED Fri Nov 5 10:32:27 2010 From: bob@REDACTED (Bob Ippolito) Date: Fri, 5 Nov 2010 17:32:27 +0800 Subject: [erlang-questions] Curiosity about MochiWeb (no questions this time) In-Reply-To: <710633.85213.qm@web112605.mail.gq1.yahoo.com> References: <710633.85213.qm@web112605.mail.gq1.yahoo.com> Message-ID: No, the behavior described is only relevant to how accepting sockets works. This does not limit the number of active requests you have and it is not at all relevant to long polling. It exists for the sole purpose of trying to minimize variance for how long accept will block by doing it from a parallel pool of processes. Once the socket has been accepted the process starts handling the request and a new acceptor process is created to take its place. You have 16+N processes looking at sockets, 16 of them blocking on accept and N of them working on active requests. The timeout of 2 seconds is just a sledgehammer to make sure that the whole accept pool don't get tied up waiting on some series of problematic sockets. This whole part of the system might be a little pessimistic but the point of this work was to try and hunt down some boogeyman in our stack that would make some very small percentage of requests take several seconds instead of several msec. It didn't ultimately fix the whole problem but we did notice lower variance so we kept it. On Fri, Nov 5, 2010 at 5:17 PM, Pablo Platt wrote: > I'm using Mochiweb for long-polling. > Each request can take 60 seconds. > I didn't change the default settings so how come I didn't have issues with the 2 > seconds limit? > Does it means that I won't be able to have more than 16 concurrent long-polling > requests? > > > > > ________________________________ > From: AJ Heller > To: Alessandro Sivieri > Cc: Erlang Questions > Sent: Thu, November 4, 2010 11:17:05 PM > Subject: Re: [erlang-questions] Curiosity about MochiWeb (no questions this > time) > > Mochiweb spawns a pool of short-lived processes (from the > `mochiweb_acceptor` module), and re-spawns new acceptor processes when > each finishes/times-out. The default settings are: 16 processes are > alive at a time, and they timeout after 2 seconds. > > I did some analysis of the Mochiweb code a while back. There's a > diagram at the bottom of the page that lays out the program flow. > http://drfloob.com/wiki/erlang_mochiweb_walkthrough.html > > --aj > > On Thu, Nov 4, 2010 at 11:17 AM, Alessandro Sivieri > wrote: >> Hi all, >> >> I was looking to an appmon instance launched from a node with MochiWeb >> active, and I saw a few processes linked to the _web process, so I was >> wondering if MochiWeb generates something like a pool of processes for >> client requests, which may explain why there were all those processes... >> >> -- >> Sivieri Alessandro >> alessandro.sivieri@REDACTED >> http://www.chimera-bellerofonte.eu/ >> http://www.poul.org/ >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > From als@REDACTED Fri Nov 5 10:09:19 2010 From: als@REDACTED (Anthony Shipman) Date: Fri, 5 Nov 2010 20:09:19 +1100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <20101104063446.GA15679@mulga.csse.unimelb.edu.au> References: <420938007.35431288831679157.JavaMail.root@zimbra> <20101104063446.GA15679@mulga.csse.unimelb.edu.au> Message-ID: <201011052009.19566.als@iinet.net.au> On Thu, 4 Nov 2010 05:34:46 pm Jeff Schultz wrote: > On Wed, Nov 03, 2010 at 10:47:05PM -0400, Ryan Zezeski wrote: > > What is wrong about using a mutable data structure that can only ever be > > touched by _one_ and only _one_ process? Then, if this process wants to > > Well, mutability may impose a significant cost on all but the simplest > garbage collectors, and part of that cost will be paid even if nothing > mutable is being mutated. Whether this is a net performance gain or > loss will vary with the circumstances, but it's not an inevitable win. > > The extra complexity may also lead to more, usually nasty, garbage > collection bugs. > > > Jeff Schultz If I were designing an erlangish language I would enrich the binary pattern system to look more like C data definitions. Then I could have a mutable binary that is accessed through the C-like pattern for reading and writing. I could then have a reasonably efficient image buffer. There would be no possibility of pointers to other data structures inside such a binary so there would be no GC issues. It would be possible to send the binary to another process but the sender would lose access to it. -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From olivier.boudeville@REDACTED Fri Nov 5 11:15:37 2010 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Fri, 5 Nov 2010 11:15:37 +0100 Subject: Efficient Insertions in Mnesia tables Message-ID: Hi, We are trying to write (with mnesia:dirty_write) in a disc_only_copies Mnesia table (type: set, not fragmented, not replicated) records (ex: 60 000 of them) and we observe that the insertion time is increasing as the table is increasingly crowded. This is not really a surprise but something we need to avoid. What we would like is to have constant (and preferably low) insertion times, like we had when writing directly to a file. We tried to get as close as possible with the following settings and use: % We want tables to be dumped less frequently from memory to disc, % in order to buffer writings (default value is 4): ok = application:set_env( mnesia, dc_dump_limit, 1 ), % Increases a lot (default value is 100) the maximum number of % writes to the transaction log before a new dump is performed: ok = application:set_env( mnesia, dump_log_write_threshold, 50000 ), Over time we see the CPU load decrease steadily, the computer seems to spend most of its time fighting for locks. We happen to be in a pretty favorable situation (only writes, no concurrent access to a given table). We chose disc_only_copies as there might be a large number of such tables and if they filled over time they could exhaust the RAM. Is there anything we missed that would allow us (roughly) constant insertion times with Mnesia? Thanks in advance for any hint, Best regards, Olivier. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. From dangud@REDACTED Fri Nov 5 11:52:08 2010 From: dangud@REDACTED (Dan Gudmundsson) Date: Fri, 5 Nov 2010 11:52:08 +0100 Subject: [erlang-questions] Efficient Insertions in Mnesia tables In-Reply-To: References: Message-ID: On Fri, Nov 5, 2010 at 11:15 AM, Olivier BOUDEVILLE wrote: > Hi, > > We are trying to write (with mnesia:dirty_write) in a disc_only_copies > Mnesia table (type: set, not fragmented, not replicated) records (ex: 60 > 000 of them) and we observe that the insertion time is increasing as the > table is increasingly crowded. This is not really a surprise but something > we need to avoid. What we would like is to have constant (and preferably > low) insertion times, like we had when writing directly to a file. > > We tried to get as close as possible with the following settings and use: > > ? ? ? ? ? ? ? ? ? ? ? ?% We want tables to be dumped less frequently from > memory to disc, > ? ? ? ? ? ? ? ? ? ? ? ?% in order to buffer writings (default value is > 4): > ? ? ? ? ? ? ? ? ? ? ? ?ok = application:set_env( mnesia, dc_dump_limit, 1 > ), You are using disc_only_copies, there is no memory to dump here, this does nothing for you. > > ? ? ? ? ? ? ? ? ? ? ? ?% Increases a lot (default value is 100) the > maximum number of > ? ? ? ? ? ? ? ? ? ? ? ?% writes to the transaction log before a new dump > is performed: > ? ? ? ? ? ? ? ? ? ? ? ?ok = application:set_env( mnesia, > dump_log_write_threshold, 50000 ), > > Over time we see the CPU load decrease steadily, the computer seems to > spend most of its time fighting for locks. > Locks ? You are using dirty i.e. no locks are taken, or are you talking about mutex's in the emulator). > We happen to be in a pretty favorable situation (only writes, no > concurrent access to a given table). We chose disc_only_copies as there > might be a large number of such tables and if they filled over time they > could exhaust the RAM. > > Is there anything we missed that would allow us (roughly) constant > insertion times with Mnesia? > Mnesia should mostly behave as dets does when using it directly, does it? i.e. switch the mnesia calls to direct dets calls and see if it does. You are not doing something stupid, like have an index on the table and all test entries have the same value on the index field? (which would cause a O(n) insertion time, and have happend more than once when someone is measuring performance). /Dan > Thanks in advance for any hint, > Best regards, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 > 65 27 13 > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > From psa@REDACTED Fri Nov 5 12:21:58 2010 From: psa@REDACTED (=?ISO-8859-1?Q?Paulo_S=E9rgio_Almeida?=) Date: Fri, 05 Nov 2010 11:21:58 +0000 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <415901635.37311288867568017.JavaMail.root@zimbra> <4CD29E67.3030609@di.uminho.pt> Message-ID: <4CD3E8D6.3040206@di.uminho.pt> On 11/4/10 10:41 PM, Richard O'Keefe wrote: > > On 5/11/2010, at 12:52 AM, Paulo S?rgio Almeida wrote: > >> Hi Robert, >> >> there is a point on which I do not agree; when you say: >> >> On 11/4/10 10:46 AM, Robert Virding wrote: >> >>> - I think that having immutable data even within only one process is a big win. Having mutable data gives you all the problems of global data. Immutable data makes it much easier to keep track of what is happening with your data, which is always a Big Win. >> >> I would say that having mutable data gives us SOME problems, but much LESS than when we have global mutable data. > > The difficulty is with aliasing. Data don't have to be global to be aliased. Yes, aliasing is the bane of imperative languages. But ... (see below) > > It *is* possible to have a language that looks and feels like an imperative > language but which is largely free of such problems. The original design of > Euclid springs to mind. Quoting "Early Experiences with Euclid": If only it was that easy. In those times, the worry was about aliasing involving variables (e.g. a parameter of a function and a global variable). Dynamic allocation was not pervasive as now. With the present use of objects and passing references, things become much worse. Many people (including myself) have tried to do something about it. There are experimental proposals, like ownership types, confined types, etc. But we have not won the battle yet. Mainstream languages lack proper support to deal with aliasing. > > In Euclid it is illegal to have two identifiers that both name the > same variable in a scope. This means, for example, that it is > illegal to call a procedure with a variable actual parameter > which is also imported by the procedure, or for any two variable > actual parameters to a procedure to specify the same variable. > The imports clause makes it easy for the compiler to check for non-aliasing. > > The non-aliasing rule effectively eliminates a large class of > particularly insidious programming errors. A common example is the > accidental passing of an element of an array to a procedure which > changes the same array directly. In this situation the value of > the actual parameter changes unexpectedly as a result of an > assignment to an element of the global array. > > This kind of bug can be very costly in debugging time, as we > discovered early in the compiler project. In the bootstrap and > testing phase of the project, a particularly puzzling bug was > encountered. An entry in the compiler's symbol table was being > changed without ever being assigned to (as was verified by a > trace). This problem, which was due to an accidental parameter > alias, took almost a week of programmer effort to find. The > preliminary version of the compiler which we were then using > did not implement non-alias checking. Had it been implemented, > the compiler would have detected the potential problem when the > program was first compiled and thereby saved us a full week of > programmer time. > > Note that this was nothing to do with *globality* of data, just the > combination of mutability and sharing. Yes ... > > Now here's the cute thing. There was a Xerox paper that showed that > the constraints on aliasing that made Euclid safe meant that it could > be read as syntactic sugar for a functional language! > >> I (and others) think there is a place for a middle-ground where processes give us locality, allowing a divid-and-conquer of the global problem into simple parts, where the data in a given process is manageable, but where if the algorithm calls for mutable data then we should be able to use it. All the nasty problems of shared memory concurrency (from the programmer's point of view) have already disappeared when we have only per-process mutable data as opposed to global mutable data. > > The nasty problems of shared memory *concurrency* disappear, true; This is what I meant. > the nasty problems of shared memory *mutability* do not. They do not disappear. But they are more confined. With no global sharing, each process acts like a course-grained active object that TRULY encapsulates its state from the rest of the system, unlike what we have now with global mutable data sharing in OO languages. We can then reason about those mutability problems looking at each process separately. Even if we do not need to explore multi-core capabilities, the structuring of a system as a bunch of processes will already be helpful from a software engineering view point, to tackle complexity. > > If you want to add mutable data to an Erlang-like language, you had > better have type checking and no-alias enforcement as good as Euclid had. That would be overly restrictive. What I meant would be to be able to program like, e.g., in Python, within each process, and being able to have aliasing, for good or for worse. But, as I said, I was not trying to advocate adding mutability to Erlang now; just to say that actors + local mutability is a useful combination in the design space, between the extremes in one direction (Erlang) or the other (the typical OO language). > You might well end up with an excellent language, superb for its job, > but it would not (and should not) be Erlang any more. > >> The idea would be to be able to use the many proven algorithms that were created for the imperative sequential world inside each process, and let the process concept a la Erlang deal with the concurrency aspects. > > Come to think of it, this is not entirely unlike what Occam was (and > thanks to free implementations like KROC, still is). > > Why not start from Occam and see what you can add in the way of trees > and pattern matching? > Don?t know how Occam is nowadays. When I used it 20 years ago to build a ray-tracer, almost all the pain was due to the lack of dynamic memory allocation. Regards, Paulo From s.j.thompson@REDACTED Fri Nov 5 11:28:57 2010 From: s.j.thompson@REDACTED (Simon Thompson) Date: Fri, 5 Nov 2010 10:28:57 +0000 Subject: SBLP 2011 - call for papers Message-ID: <2ADED951-61FF-4FF7-85A3-5927CB80B5EB@kent.ac.uk> SBLP 2011: Call For Papers 15th Brazilian Symposium on Programming Languages Sao Paulo, Brazil September 26-30, 2010 http://www.each.usp.br/cbsoft2011/ IMPORTANT DATES Paper abstract submission (15 lines): April 22nd, 2011 Full paper submission: April 29th, 2011 Notification of acceptance: May 30th, 2011 Final papers due: July 29th, 2011 INVITED SPEAKERS Gary T. Leavens, Univ. of Central Florida Jose Luis Fiadeiro, Univ. of Leicester INTRODUCTION The 15th Brazilian Symposium on Programming Languages, SBLP 2011, will be held in Sao Paulo, Brazil, between September 26th and 30th, 2011. SBLP provides a venue for researchers and practitioners interested in the fundamental principles and innovations in the design and implementation of programming languages and systems. The symposium will be part of the 2nd Brazilian Conference on Software: Theory and Practice, CBSoft 2011, http://www.each.usp.br/cbsoft2011/, which will host four well-established Brazilian symposia: * XXV Brazilian Symposium on Software Engineering (SBES) * XV Brazilian Symposium on Programming Languages (SBLP) * XIV Brazilian Symposium on Formal Methods (SBMF) * V Brazilian Symposium on Components, Software Architecture and Software Reuse (SBCARS) SBLP 2011 invites authors to contribute with technical papers related (but not limited) to: * Program generation and transformation, including domain-specific languages and model-driven development in the context of programming languages. * Programming paradigms and styles, including functional, object-oriented, aspect-oriented, scripting languages, real-time, service-oriented, multithreaded, parallel, and distributed programming. * Formal semantics and theoretical foundations, including denotational, operational, algebraic and categorical approaches. * Program analysis and verification, including type systems, static analysis and abstract interpretation. * Programming language design and implementation, including new programming models, programming language environments, compilation and interpretation techniques. SUBMISSIONS Submissions should be done using SBLP 2011 installation of the EasyChair conference management system at http://www.easychair.org/conferences/?conf=sblp2011. Contributions should be written in Portuguese or English. We solicit papers that should fall into one of two different categories: full papers, with at most 15 pages, or short papers, with at most 5 pages. All papers should be prepared using the Easychair template. (http://www.easychair.org/easychair.zip) In particular, we encourage the submission of short papers reporting on master dissertations or doctoral theses at early stages of their development. All accepted papers will be published in the conference proceedings. As in previous editions, a journal special issue, with selected papers from accepted contributions, is anticipated. From 2003 to 2008 there were special issues of the Journal of Universal Computer Science, published by Springer, with the post-proceedings of SBLP. The post-proceedings of SBLP 2009 and 2010 are being edited as special issues of Science of Computer Programming, published by Elsevier. GENERAL CO-CHAIRS Denise Stringhini FCI, Mackenzie Alfredo Goldman IME, USP PROGRAMME CHAIRS Christiano Braga, UFF Jose Luiz Fiadeiro, Univ. of Leicester PROGRAMME COMMITTEE * Alberto Pardo, Univ. de La Republica * Alex Garcia, IME * Alvaro Freitas Moreira, UFRGS (TBC) * Andre Santos, UFPE * Artur Boronat, Univ. of Leicester * Carlos Camarao, UFMG * Christiano Braga, UFF (co-chair) * Edward Hermann Haeusler, PUC-Rio (TBC) * Fernando Castor Filho, UFPE * Fernando Pereira, UFMG * Francisco Heron de Carvalho Junior, UFC (TBC) * Giuseppe Castagna, Paris 7 (TBC) * Jens Palsberg, UCLA * Joao Saraiva, Universidade do Minho * Johan Jeuring, Utrecht Univ. * Jonathan Aldrich, Carnegie Mellon Univ. * Jose Luiz Fiadeiro, Univ. of Leicester (co-chair) * Lucilia Figueiredo, UFOP * Luis Soares Barbosa, Univ. do Minho * Marcelo A. Maia, UFU * Marcelo d'Amorim, UFPE * Marco Tulio Valente, UFMG * Mariza A. S. Bigonha, UFMG * Martin A. Musicante, UFRN * Noemi Rodriguez, PUC-Rio * Paulo Borba, UFPE (TBC) * Peter Mosses, Swansea University * Renato Cerqueira, PUC-Rio * Ricardo Massa, UFPE * Roberto S. Bigonha, UFMG (TBC) * Roberto Ierusalimschy, PUC-Rio * Sandro Rigo, UNICAMP * Sergio Soares, UFPE (TBC) * Sergiu Dascalu, Univ. of Nevada * Simon Thompson, Univ. of Kent * Sophia Drossopoulou, Imperial College (TBC) * Varmo Vene, Univ. de Tartu (TBC) * Vladimir Di Iorio, UFV (TBC) From Mike.French@REDACTED Fri Nov 5 12:37:30 2010 From: Mike.French@REDACTED (French, Mike) Date: Fri, 5 Nov 2010 11:37:30 -0000 Subject: [erlang-questions] Extracting detailed information from TypEr /Dialyzer Message-ID: <3F8EEA01CF53D74DB4A9EC314D82B4F3224841@wells154924> > On Thu, 04 Nov 2010 03:58:12 +1100, Edmond Begumisa > wrote: > ... > > Actually, there's an idea: maybe someone should write a contract checker > > for gen_fsm that uses ordinary Erlang-terms rather than UBF(A) encoding. > > This could then be used in the same VM instance without much cost and > > make working with gen_fsm more dependable by allowing you to declare how > > the state-machine is supposed to behave and cross check that against how > > it's actually behaving -- a major frustration I've had when using > > gen_fsm. > -----Original Message----- > From: erlang-questions@REDACTED > [mailto:erlang-questions@REDACTED]On > Behalf Of Edmond Begumisa > Sent: 03 November 2010 19:55 > To: Torben Hoffmann > Subject: Re: [erlang-questions] Extracting detailed information from TypEr/Dialyzer > > A better way of putting it... > > While you can look at UBF(A) and UBF(C) as a formal approach to protocols, > what's relevant here, is that from Erlang's perspective, you can look at > UBF(B) as a formal approach to state machines providing type checking and > declarative state transitions. This can be very helpful at all levels of > development of a state machine from design right though to coding, > debugging and documentation. > > That's more what I was trying to say. A good point, nicely put. It seems a pity to restrict UBF to external communication when we also need specification of internal messsaging. Can we go further and say that UBF(B), or close variant, could be incorporated into the Erlang type system, as a declarative signature for message protocol contracts on processes. We already have -type and -spec, maybe there are -send and -recv statements to be made about what messages can be sent or received by a process. Some -spec'ed methods are also process loops, with process state as their arguments. In some cases, the continuous space of arguments could be pattern-matched into a finite set of states, which might still be useful even if the mapping is partial (rather than the total mapping built in to gen_fsm by design). Perhaps there is are -proc statements based on UBF(B) that correlate the states and the send/recv specifications. Mike Thales UK Ltd (Wells) DISCLAIMER: The information contained in this e-mail is confidential. It may also be legally privileged. It is intended only for the stated addressee(s) and access to it by any other person is unauthorised. If you are not an addressee, you must not disclose, copy, circulate or in any other way use or rely on the information contained in this e-mail. Such unauthorised use may be unlawful. We may monitor all e-mail communications through our networks. If you have received this e-mail in error, please inform us immediately on sender's telephone number above and delete it and all copies from your system. We accept no responsibility for changes to any e-mail which occur after it has been sent. Attachments to this e-mail may contain software viruses which could damage your system. We therefore recommend you virus-check all attachments before opening. Thales UK Ltd. Registered Office: 2 Dashwood Lang Road, The Bourne Business Park, Addlestone, Weybridge, Surrey KT15 2NX Registered in England No. 868273 From olivier.boudeville@REDACTED Fri Nov 5 13:35:38 2010 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Fri, 5 Nov 2010 13:35:38 +0100 Subject: [erlang-questions] Efficient Insertions in Mnesia tables In-Reply-To: Message-ID: Hello Dan, Thanks for your answer! I was thinking that Mnesia, even with disc_only_copies, could maintain a in-RAM buffer to batch even dirty writes (a bit like delayed_writes for files). Maybe I should do the buffering by myself, not sure it would help. Indeed the RAM-only options I specified had then no effect. Actually, wouldn't it be nice to have an intermediate step between disc_only_copies and disc_copies, i.e. basically a batch writing to files with a partial RAM caching for pending writes? This would be useful in all the cases where the whole table could not fit in RAM, and trade a bit of robustness for a performance gain. As for locks, I could see with vmstat that beam.smp was mostly spending its time waiting (not idle), whereas the number of ongoing I/O operations stayed very low, so I was supposing the time spent was mostly linked with some synchronisation happening within the VMs. As I do not think it was due to our application (as it is basically stalled because of the synchronous writes we have to perform), I suppose that locks had to do with Mnesia. As for the default index, unless I am mistaken, it is made on the primary key, in our case there cannot be any collision (as it is basically an increasing time-step). I suppose we cannot temporarily disable the index during this phase and rebuild it when it is over? Thanks, Best regards, Olivier. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 dangud@REDACTED 05/11/2010 11:52 A olivier.boudeville@REDACTED cc erlang-questions@REDACTED Objet Re: [erlang-questions] Efficient Insertions in Mnesia tables On Fri, Nov 5, 2010 at 11:15 AM, Olivier BOUDEVILLE wrote: > Hi, > > We are trying to write (with mnesia:dirty_write) in a disc_only_copies > Mnesia table (type: set, not fragmented, not replicated) records (ex: 60 > 000 of them) and we observe that the insertion time is increasing as the > table is increasingly crowded. This is not really a surprise but something > we need to avoid. What we would like is to have constant (and preferably > low) insertion times, like we had when writing directly to a file. > > We tried to get as close as possible with the following settings and use: > > % We want tables to be dumped less frequently from > memory to disc, > % in order to buffer writings (default value is > 4): > ok = application:set_env( mnesia, dc_dump_limit, 1 > ), You are using disc_only_copies, there is no memory to dump here, this does nothing for you. > > % Increases a lot (default value is 100) the > maximum number of > % writes to the transaction log before a new dump > is performed: > ok = application:set_env( mnesia, > dump_log_write_threshold, 50000 ), > > Over time we see the CPU load decrease steadily, the computer seems to > spend most of its time fighting for locks. > Locks ? You are using dirty i.e. no locks are taken, or are you talking about mutex's in the emulator). > We happen to be in a pretty favorable situation (only writes, no > concurrent access to a given table). We chose disc_only_copies as there > might be a large number of such tables and if they filled over time they > could exhaust the RAM. > > Is there anything we missed that would allow us (roughly) constant > insertion times with Mnesia? > Mnesia should mostly behave as dets does when using it directly, does it? i.e. switch the mnesia calls to direct dets calls and see if it does. You are not doing something stupid, like have an index on the table and all test entries have the same value on the index field? (which would cause a O(n) insertion time, and have happend more than once when someone is measuring performance). /Dan > Thanks in advance for any hint, > Best regards, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 > 65 27 13 > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. From norton@REDACTED Fri Nov 5 15:16:48 2010 From: norton@REDACTED (Joseph Wayne Norton) Date: Fri, 05 Nov 2010 23:16:48 +0900 Subject: [erlang-questions] Extracting detailed information from TypEr /Dialyzer In-Reply-To: <3F8EEA01CF53D74DB4A9EC314D82B4F3224841@wells154924> References: <3F8EEA01CF53D74DB4A9EC314D82B4F3224841@wells154924> Message-ID: With regards to this thread, I'd like to mention a few points. On Fri, 05 Nov 2010 20:37:30 +0900, French, Mike wrote: > > A good point, nicely put. > > It seems a pity to restrict UBF to external communication > when we also need specification of internal messsaging. > Can we go further and say that UBF(B), or close variant, > could be incorporated into the Erlang type system, as a > declarative signature for message protocol contracts on processes. > We added the concept of "lpc" (Local Procedure Call) to the UBF framework. This feature is not as built-in as you may expect (or want) but it permits one to re-use a UBF(b) contract and to call the contract's implementation directly without a tcp/ip socket. There is an example of it's usage in this module - look for "ubf_client:lpc". "Plugin" is an alias for a UBF(b) contract since it implies both a contract and it's corresponding implementation module. https://github.com/norton/ubf-jsonrpc/blob/master/src/ubf_jsonrpc_inets_httpd_simple_auth.erl The above module glues an json-rpc http listener to a UBF(b) contract. The same UBF(b) contract could be used with a tcp/ip socket as well. > We already have -type and -spec, > maybe there are -send and -recv statements to be made > about what messages can be sent or received by a process. > > Some -spec'ed methods are also process loops, > with process state as their arguments. > In some cases, the continuous space of arguments > could be pattern-matched into a finite set of states, > which might still be useful even if the mapping is partial > (rather than the total mapping built in to gen_fsm by design). > Perhaps there is are -proc statements based on UBF(B) > that correlate the states and the send/recv specifications. In the following two modules, there is also a simple prototype that demonstrates how to import eep8-style -type definitions into a UBF(b) contract. https://github.com/norton/ubf-eep8/tree/master/src/Unit-EUnit-Files/ This prototype is incomplete (and not yet fully useful) but it does illustrate the possibility to use -type definitions as part of the UBF framework. regards, Joe N. > Mike > > > Thales UK Ltd (Wells) DISCLAIMER: The information contained in this > e-mail > is confidential. It may also be legally privileged. It is intended only > for > the stated addressee(s) and access to it by any other person is > unauthorised. If you are not an addressee, you must not disclose, copy, > circulate or in any other way use or rely on the information contained in > this e-mail. Such unauthorised use may be unlawful. We may monitor all > e-mail communications through our networks. If you have received this > e-mail > in error, please inform us immediately on sender's telephone number above > and delete it and all copies from your system. We accept no > responsibility > for changes to any e-mail which occur after it has been sent. > Attachments > to this e-mail may contain software viruses which could damage your > system. > We therefore recommend you virus-check all attachments before opening. > Thales UK Ltd. Registered Office: 2 Dashwood Lang Road, The Bourne > Business > Park, Addlestone, Weybridge, Surrey KT15 2NX Registered in England No. > 868273 > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- norton@REDACTED From dionne@REDACTED Fri Nov 5 15:23:40 2010 From: dionne@REDACTED (Robert Dionne) Date: Fri, 5 Nov 2010 10:23:40 -0400 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: Great comment Joe, I couldn't help taking the liberty to quote you on twitter. Best regards, Bob > It seems to my that SQL provides you with the ability do to complex > queries on simple things. K-V dbs can do simple queries on complex > things. > > /Joe From ivan060111ad@REDACTED Fri Nov 5 17:24:58 2010 From: ivan060111ad@REDACTED (=?ISO-8859-1?Q?Ivan_Carmenates_Garc=EDa?=) Date: Fri, 05 Nov 2010 11:24:58 -0500 Subject: Here I am again with EVO (ExtendedVisualOtp) Message-ID: <4CD42FDA.1070300@cmg.jovenclub.cu> *ExtendedVisualOtp* *(EVO)* *The User's Guide* *Version 1.4* Ing. Ivan Carmenates Garc?a Ivanco Software Company in association with SPI Team November 3 2010 Index Abstract 5 Introduction. 6 Content 13 Functionalities of the ExtendedVisualOtp.dll library. 13 ConvertToCSharp: 13 ? FromErlangObject(Otp.Erlang.Object _object) 13 ? FromErlangReply(Otp.Erlang.Object _replyObject) 13 ConvertToErlang: 13 ? ToErlangObject(object _object) 13 ? ToErlangObjectFromString(string _object) 13 ErlangServerInterface: 13 ? AutoTrace. 13 ? Connect() 13 ? ErlangCookie. 13 ? Disconnect() 13 ? IsConnected. 13 ? OnDisconnected(IServerReply _reason) 14 ? OnReceive(IServerReply _reply) 14 ? PingTimeOut 14 ? RemoteAddress. 14 ? RemoteNodeName. 14 ? RemoteProcessName. 14 ? SetOwnerForm(Form _ownerForm) 14 ? SentToErlangServerRequest(ErlangServerRequest _erlangServerRequest, IServerReply _reply) 14 ? ServerPort 14 ? StartEVOTestApplication() 14 ? UseShortNames. 14 ? AllowReceiveClientImage. 14 ErlangServerRequest: 15 ? AbortAllRequests() 15 ? AbortAsyncRequests() 15 ? AbortNormalRequests() 15 ? AbortRequest(IRequestInfo _reqInfo) 15 ? Description. 15 ? Message. 15 ? ErlangServerInterface. 15 ? OnReceive(IServerReply _reply) 15 ? Request(string _description, object _message) 15 ? RequestAsync(string _description, object _message) 15 ? Request(object _message) 15 ? RequestAsync(object _message) 15 ? SetOwnerForm(Form _ownerForm) 16 ? PublishClientImage() 16 IRequestInfo: 16 ? AbortRequest() 16 ? Description. 16 ? Kind. 16 ? WaitForReply() 16 ? WaitForReply(int _timeout) 16 ? WasAnswered. 16 ? WasMade. 16 IServerReply: 16 ? CSharpReply. 16 ? Description. 16 ? ErlangReply. 16 ? RequestInfo. 16 RequestKind. 17 ? Async. 17 ? Normal 17 ? ServerSent 17 The evo_template template. 18 The evo_template has many files listed at following: 18 ? 1.install.cmd. 18 ? 1.start.cmd. 18 ? 2.service_install-start.cmd. 18 ? 2.service_stop.cmd. 18 ? 2.service_uninstall.cmd. 18 ? compile.cmd. 18 ? config.cmd. 18 ? Folder [beam] 18 ? appname.app. 18 ? appname.beam.. 18 ? appname_debug_module.beam.. 18 ? appname_main_interface.beam.. 18 ? appname_main_supervisor.beam.. 18 ? Folder [sources] 18 ? appname_db_module.erl 19 ? appname_db_server.erl 19 ? appname_request_handler.erl 19 The Big Example. 22 Annexes. 26 Conclusions. 29 Abstract The ExtendedVisualOtp (EVO) is a component specialized in client-server communication, implemented in two programming languages, C# of Microsoft and Erlang of Ericsson, oriented to the development of real time applications, where the server applications are written in Erlang and the client applications in .Net. The main concept of this component is the development of client-server applications where the productivity factor is highest and the cost of the production is quite low. Ideal for small companies with few experience in that kind of applications, which want to insert into this so wide world that is the interaction and exchange of information through the network, in a fast and powerful manner. Introduction The ExtendedVisualOtp (EVO) is a component specialized in client-server communication, implemented in two programming languages, C# of Microsoft and Erlang of Ericsson, oriented to the development of real time applications, where the server applications are written in Erlang and the client applications in .Net. The main concept of this component is the development of client-server applications where the productivity factor is highest and the cost of the production is quite low. Ideal for small companies with few experience in that kind of applications, which want to insert into this so wide world that is the interaction and exchange of information through the network, in a fast and powerful manner. It can be illustrated through a simple example: Example #1: Imagine you want to develop a small client application, so you want to make a request "know my payment" to a server application and to be notified about the answer generated by that request. Develop this simple example in any programming language without the usage of a helper component, could represent a lot of work. However, using the ExtendedVisualOtp.dll library for the development of client applications and the evo_template template to write server applications becomes from a large and restless work to a simple and comfortable one. Answer for the example #1 using EVO At the client application, after a few and simple visual configurations for the ErlangServerInterface and ErlangServerRequest components of the ExtendedVisualOtp.dll library, you should write: // to connect to the server erlangServerInterface1.Connect(); // to make a request erlangServerRequest1.Request("Description of the request", object _message); or erlangServerRequest1.Request(object _message); NOTE: * Request-OnReceive means the OnReceive event of the ErlangServerRequest component. * Interface-OnReceive means the OnReceive event of the ErlangServerInterface component. // to receive the answers The answers of the made requests could be caught of Async way through the Request-OnReceive event, or could be caught of Sync way using the function WaitForReply() immediately after the request, example for the Sync way: IRequestInfo reqinfo = erlangServerRequest1.Request("knows the salary", new object[] { " 'know_my_payment' ", "Ivan Carmenates Garc?a"}); IServerReply reply = reqinfo.WaitForReply(); MessageBox.Show(reply.CSharpReply.ToString()); reqinfo is an object of the class or interface IRequestInfo. It contains the information of the made request, allowing to do some stuffs with it, such as, block the program, to wait for the answer, reqinfo.WaitForReply(), or abort the made request, reqinfo.AbortRequest(). reply is an object of the class or interface IServerReply that contains the information to treat the answer sent by the server. In case that you wish to receive the answers using the Async way, you must use the Request-OnReceive event without the usage of the function WaitForReply(), example for the Async way: IRequestInfo reqinfo = erlangServerRequest1.Request("knows the salary", new object[] { " 'know_my_payment' ", "Ivan Carmenates Garc?a"}); And to receive the answer, you should write something like this: OnReceiveErlangServerRequestMethodNameHere(IServerReply _reply) { switch(_reply.Description) { case "knows the salary": MessageBox.Show(_reply.CSharpReply.ToString()); break; } } The same implementation, works for receive all the answers of all requests that you make, you just have to add a new clause to the switch sentence with a new description, a description per request you make. So, to write server applications, you must edit the server template, evo_template and in the file called evo_request_handler.erl, located in the sources folder, about the line 28, part: [USER FUNCTIONS TO MODIFY] write your code. For the case of example #1, could be: {{know_my_payment, Name}, Pid, RequestInfo}-> Reply = evo_db_module:get_payment (Name), %% this function is implemented in other module. Pid ! {reply, Reply, RequestInfo}; As it seems, is easy to write client-server applications in EVO. If you're wandering, how it is possible? The answer is as simple as it sound, that's all! The EVO component, in its total integrity, both, the template evo_template to develop server applications and the ExtendedVisualOtp.dll library to develop client applications, it handles for you all the dirty work. You just have to abstract your mind to the existence of a fast and efficient postman, which you call to deliver your letter or messages to a given destination, in this case the server. The server can send if you wish, the message to all other people in the "world", or just to one person, if it is programmed of that way, acting as intermediary. An example, if you wish to send "directly" a message to another client, the EVO component has a functionality, where once your client is connected to the server, each time that other clients get connected or disconnected, the server sends to your client a notification message of new client connected, {new_member, Pid}, or client disconnected, {crashed_member, Pid}, and the ID or address Pid. With this ID, you can send to those clients through the server a message, simple as it may seem! Making a request like this: IRequestInfo reqinfo = erlangServerRequest1.Request(new object[] { " 'send_to' ", ClientPid, Message}); Notice here the usage of the other way to make requests to the server, where is not necessary to specify a description for the request, it is the case of the call to the function Request(object _message). The description for this request is obtained by the first element inside the message, if the message is an array or a list of elements and if its first element is an Erlang atom, or if the message is just one element and it is an Erlang atom. For the case before, note that the first element of the array that you send as message is the Erlang atom " 'send_to' ". This atom is the description for the request and the message at the same time. In case that the first or unique element of the message isn't an Erlang atom, the description for the request will be set to the Erlang atom " 'no_description' ". How to catch the message {new_member, Pid} / {crashed_member, Pid}? Through the Interface-OnReceive event, example: OnReceiveErlangServerInterfaceMethod(IServerReply _reply) { If (_reply.Description == "new_member") { erlangServerRequest1.Request(new object[] { " 'send_to' ", _reply.ErlangReply as Otp.Erlang.Pid, "Are you new?"}); } } With this, your client sends to each client that got connected to the server, the message: "Are you new?" _reply.ErlangReply, it's the message, that is, the ID of the client recently connected to the server. On the server application, in the template evo_template to develop server applications, in the file, evo_request_handler.erl, you should write something like this: %% The 'send_to' message. {{send_to, WhoPid, Message}, Pid, RequestInfo = {Description, _, _, _, _}}-> send_to(WhoPid, Description, Message), Pid ! {reply, "The message was successfully sent.", RequestInfo}; The function send_to(ClientPid, Description, Message) is a function that comes within the component EVO, in the template for server applications, evo_template, with which you can send messages to any client that got connected to the server, specifying its id (Pid) in the connection. The Interface-OnReceive event is fired each time that any message sent by the server, is received on the client. And the Request-OnReceive event is fired only when a message received, is the answer of a request made by it. So if you want to get low level messages, you must use the Interface-OnReceive event. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fritchie@REDACTED Fri Nov 5 17:40:00 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 05 Nov 2010 11:40:00 -0500 Subject: [erlang-questions] SMP in emacs? In-Reply-To: Message of "Wed, 03 Nov 2010 14:00:16 BST." Message-ID: <27417.1288975200@snookles.snookles.com> zabrane Mikael wrote: mz> Off topic question. Could someone please explain me what's the mz> benefits of this "+S" option (and the "+A" option by the way). The +A option allows you to specify the size of the I/O worker thread pool. For moderate and high workloads that involve the local file system(*), you'll likely see a performance improvement by using it. -Scott (*) All UNIX variants that I'm aware of will ignore any request to perform asynchronous I/O on the local file system unless you explicitly use the async API. And even the async API doesn't have async versions of some system calls like unlink(2). From michal.dobrogost@REDACTED Fri Nov 5 17:47:08 2010 From: michal.dobrogost@REDACTED (Michal D.) Date: Fri, 5 Nov 2010 12:47:08 -0400 Subject: gb_trees, strange behaviour Message-ID: Hi again, I'm having a lot of problems starting out with Erlang. It appears to me that I can't update a specific key of a gb_tree, while the other ones work fine. Can anyone explain the following behaviour? > X = gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). {3,{limit,-1,{trace,false,nil,nil},{timeout,-1,nil,nil}}} > > gb_trees:update(limit,true,X). {3,{limit,true,{trace,false,nil,nil},{timeout,-1,nil,nil}}} > > gb_trees:update(timeout,true,X). {3,{limit,-1,{trace,false,nil,nil},{timeout,true,nil,nil}}} > > gb_trees:update(trace,true,X). ** exception error: no function clause matching gb_trees:update_1(trace,true,nil) in function gb_trees:update_1/3 in call from gb_trees:update/3 > Thanks! From fritchie@REDACTED Fri Nov 5 17:54:03 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 05 Nov 2010 11:54:03 -0500 Subject: [erlang-questions] Extracting detailed information from TypEr/Dialyzer In-Reply-To: Message of "Thu, 04 Nov 2010 03:58:12 +1100." Message-ID: <28328.1288976043@snookles.snookles.com> Edmond Begumisa wrote: eb> Of course, to make use of this, you have to migrate to UBF(A) eb> encoding for the actual messages between processes. That isn't true today. You can use UBF(A), JSON-RPC (JSON over HTTP), JSON over TCP, EBF (Erlang's native term_to_binary() serialization format), and (work still in progress, I'm not quite certain) Thrift. eb> Actually, there's an idea: maybe someone should write a contract eb> checker for gen_fsm that uses ordinary Erlang-terms rather than eb> UBF(A) encoding. I believe that's already done: that protocol is called 'etf', IIRC. -Scott P.S. While Joe Armstrong's original implementation & docs are useful, the work that Joe Norton has done with the fork at github.com/norton/ubf (and the ubf-thrift and ubf-jsonrpc and ubf-eep8 and ubf-bertrpc and ubf-abnf) is under active development & maintenance. P.P.S. Joe Norton will be giving a hands-on tutorial on UBF on 15 November, the day before the EUC in Stockholm. From kostis@REDACTED Fri Nov 5 18:52:45 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Fri, 05 Nov 2010 19:52:45 +0200 Subject: [erlang-questions] gb_trees, strange behaviour In-Reply-To: References: Message-ID: <4CD4446D.1030401@cs.ntua.gr> Michal D. wrote: > Hi again, > > I'm having a lot of problems starting out with Erlang. It appears to me that > I can't update a specific key of a gb_tree, while the other ones work fine. > Can anyone explain the following behaviour? > >> X = gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). > {3,{limit,-1,{trace,false,nil,nil},{timeout,-1,nil,nil}}} >> gb_trees:update(limit,true,X). > {3,{limit,true,{trace,false,nil,nil},{timeout,-1,nil,nil}}} >> gb_trees:update(timeout,true,X). > {3,{limit,-1,{trace,false,nil,nil},{timeout,true,nil,nil}}} >> gb_trees:update(trace,true,X). > ** exception error: no function clause matching > gb_trees:update_1(trace,true,nil) > in function gb_trees:update_1/3 > in call from gb_trees:update/3 The update calls that you think they work, work by fluke. The problem is that even though you may think you have created a gb tree, but in fact you have not. The problem lies in your first call: X = gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). You promised you will supply an orddict, but in fact you have not. Use: X = gb_trees:from_orddict(orddict:from_list([{trace,false},{limit,-1},{timeout,-1}])). instead and things will work as expected. Kostis From fritchie@REDACTED Fri Nov 5 18:10:46 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 05 Nov 2010 12:10:46 -0500 Subject: [erlang-questions] ownership of an ets table In-Reply-To: Message of "Thu, 04 Nov 2010 18:56:57 BST." <4CD2F3E9.6080804@amberbio.com> Message-ID: <29614.1288977046@snookles.snookles.com> Morten Krogh wrote: mk> Ahmed, the reason I singled out public tables was because tables mk> with public access might naturally live without an owner. Private mk> tables are pretty useless when their owner dies, but public tables mk> could be on their own, I would say. "Naturally" is in the eye of the beholder. I'd want to have such a public ETS table go away the moment that I stopped the OTP application that was using it. As a thought experiment: pretend that your virtual machine (and all applications & processes that run within it) will be running for 10 years. mk> One could have an option in the ets:new to create a table without an mk> owner. Why not? It's pretty common to create a public table in a supervisor's init() function. The supervisor won't die unless its application (or at least its parent supervisor) is stopped. -Scott From michal.dobrogost@REDACTED Fri Nov 5 18:35:26 2010 From: michal.dobrogost@REDACTED (Michal D.) Date: Fri, 5 Nov 2010 12:35:26 -0500 Subject: [erlang-questions] gb_trees, strange behaviour In-Reply-To: <4CD4446D.1030401@cs.ntua.gr> References: <4CD4446D.1030401@cs.ntua.gr> Message-ID: > Michal D. wrote: >> >> Hi again, >> >> I'm having a lot of problems starting out with Erlang. It appears to me >> that >> I can't update a specific key of a gb_tree, while the other ones work >> fine. >> Can anyone explain the following behaviour? >> >>> X = gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). >> >> {3,{limit,-1,{trace,false,nil,nil},{timeout,-1,nil,nil}}} >>> >>> gb_trees:update(limit,true,X). >> >> {3,{limit,true,{trace,false,nil,nil},{timeout,-1,nil,nil}}} >>> >>> gb_trees:update(timeout,true,X). >> >> {3,{limit,-1,{trace,false,nil,nil},{timeout,true,nil,nil}}} >>> >>> gb_trees:update(trace,true,X). >> >> ** exception error: no function clause matching >> ? ? ? ? ? ? ? ? ? ?gb_trees:update_1(trace,true,nil) >> ? ? in function ?gb_trees:update_1/3 >> ? ? in call from gb_trees:update/3 > > The update calls that you think they work, work by fluke. The problem is > that even though you may think you have created a gb tree, but in fact you > have not. > > The problem lies in your first call: > > ?X = gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). > > You promised you will supply an orddict, but in fact you have not. > Use: > > ?X = > gb_trees:from_orddict(orddict:from_list([{trace,false},{limit,-1},{timeout,-1}])). > > instead and things will work as expected. > > Kostis > Thank you for the analysis! Maybe we should update the documentation? It's pretty common to use a list of tuples as a dictionary in other languages and that's what appears to be specified in the Erlang documentation: ---------- from_orddict(List) -> Tree Types: List = [{Key, Val}] Key = Val = term() Tree = gb_tree() Turns an ordered list List of key-value tuples into a tree. The list must not contain duplicate keys. ---------- Although on a second reading, it seems that the word "orderered" in "ordered list" is key. So is an orddict just an ordered list of tuples? But then again, why would it have to_list/1 if it was just a list? Cheers, Michal From kiszl@REDACTED Fri Nov 5 19:14:11 2010 From: kiszl@REDACTED (Zoltan Lajos Kis) Date: Fri, 05 Nov 2010 19:14:11 +0100 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <842437213.35451288831826841.JavaMail.root@zimbra> References: <842437213.35451288831826841.JavaMail.root@zimbra> Message-ID: <4CD44973.2080400@tmit.bme.hu> While a shared/hybrid heap on a global VM level does involve locking, pauses, etc., wouldn't a schedluer-level shared heap be free of these problems? I have no much experience with the VM internals, but I suppose that a running process can manipulate its scheduler's heap for free (i.e. without locking). For example when sending a message it could simply move the data structure from its own heap to the scheduler's, and only pass on the reference. Also, the garbage collector would be "just another process" in this case, so I don't see why it would affect the real-timeness of the VM. I would also imagine that the global shared heap is easier to implement in a two-level fashion, where processes share their scheduler's heap, and schedulers share the global shared heap. Unfortunately, I have even less experience in shared heaps, and garbage collectors than in the VM itself. Anyway, I think this would fit well the current spawning and migration procedures; and hope this message was more, than just noise :o). Regards, Zoltan. On 11/4/2010 1:50 AM, Robert Virding wrote: > This question is much more complex that you would at first realise. Some thoughts (in no specific order): > > 1. Shared/hybrid heaps are wonderful BUT: > - They are more difficult to implement in a real-time environment. Yes, this is a known problem with solutions but the benefits are less than you would expect. Real-time gc costs. > - They don't port easily to multi-core systems as you suddenly start needing locks and stuff everywhere to control access or have complex schemes to get around this. Which copying doesn't. > > For some problems shared heaps are much better, for others much worse. > > 2. I would assert that the operations on immutable data structures cost less then you imagine. > > 3. From the way you describe it having data structures which are both mutable and immutable would very confusing. You would need to have two sets of operations on them and every application would have to keep track of when to use them. > > 4. Mutable data structures give you all the things you want to avoid with global data, especially that they can change under your feet without you knowing about it. Yes, you may have to pass a reference around but it still means that I don't know who can access my bit of data and can change it. Yes, you can stipulate that the program should be well-behaved, but we all know how well that works in real-life in a big system. > > So while I have said that you don't *need* immutable data to get process isolation, instead you can specify data copying between processes, this does not mean that I recommend having mutable data. Immutable data is so much easier to work with and keep track of. > > Robert From tony.arcieri@REDACTED Fri Nov 5 19:18:43 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Fri, 5 Nov 2010 12:18:43 -0600 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: <18753784.40161288916089555.JavaMail.root@zimbra> References: <18753784.40161288916089555.JavaMail.root@zimbra> Message-ID: On Thu, Nov 4, 2010 at 6:14 PM, Robert Virding < robert.virding@REDACTED> wrote: > But they do cheat a bit, it's hardware assisted so not directly comparable > to running on general purpose hardware. > Actually, Azul recently released the Zing JVM, which runs a virtualized Azul environment on general purpose hardware: http://www.azulsystems.com/products/zing/virtual-machine Offhand I have no idea how it performs, though -- Tony Arcieri Medioh! A Kudelski Brand From timo.lindemann@REDACTED Fri Nov 5 21:36:17 2010 From: timo.lindemann@REDACTED (Timo Lindemann) Date: Fri, 5 Nov 2010 21:36:17 +0100 Subject: EOF to external program Message-ID: Hello erlangers, I like to use external programs, so I thought I could use ports a lot. I fail to understand one detail about ports: Why can one not close the write end (such that the connected external program registers an EOF on its stdin descriptor)? I dug around the web some, and found that this has stumped other people in the past, with suggestions about using netcat to work around that problem, to writing a wrapper that parses the data for some marker the process would send to the external program signalling EOF and have the wrapper close the write end. Why is there NO way in erlang to let a program know the input's data has ended *and* fetching its response *and* getting its (normal) exit status? port_close/1 will normally instantly kill the external program. Omitting this feature severely limits the things one can do with ports. awk will not process its END clause, wc will never exit, ... if there's no EOF to be found. There's a lot of programs waiting for stdin to close. So please, what's the reason for not allowing this? Will this be added sometime later on? Or have I just not found a way to do it? All the best -T. -- Timo Lindemann Software Systems Engineer From zabrane3@REDACTED Fri Nov 5 21:40:50 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Fri, 5 Nov 2010 21:40:50 +0100 Subject: [erlang-questions] SMP in emacs? In-Reply-To: <27417.1288975200@snookles.snookles.com> References: <27417.1288975200@snookles.snookles.com> Message-ID: Thanks a lot Scott. -- Regards Zabrane 2010/11/5 Scott Lystig Fritchie : > zabrane Mikael wrote: > > mz> Off topic question. Could someone please explain me what's the > mz> benefits of this "+S" option (and the "+A" option by the way). > > The +A option allows you to specify the size of the I/O worker thread > pool. ?For moderate and high workloads that involve the local file > system(*), you'll likely see a performance improvement by using it. > > -Scott > > (*) All UNIX variants that I'm aware of will ignore any request to > perform asynchronous I/O on the local file system unless you explicitly > use the async API. ?And even the async API doesn't have async versions > of some system calls like unlink(2). > From robert.virding@REDACTED Fri Nov 5 21:44:59 2010 From: robert.virding@REDACTED (Robert Virding) Date: Fri, 5 Nov 2010 20:44:59 +0000 (GMT) Subject: [erlang-questions] gb_trees, strange behaviour In-Reply-To: Message-ID: <277175450.47991288989899268.JavaMail.root@zimbra> The reason is that orddict is defined to have exactly the same interface as dict. So it has a to_list/1. In dict this actually builds a new list while orddict just returns its list, of course. Robert ----- "Michal D." wrote: > > Michal D. wrote: > >> > >> Hi again, > >> > >> I'm having a lot of problems starting out with Erlang. It appears > to me > >> that > >> I can't update a specific key of a gb_tree, while the other ones > work > >> fine. > >> Can anyone explain the following behaviour? > >> > >>> X = > gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). > >> > >> {3,{limit,-1,{trace,false,nil,nil},{timeout,-1,nil,nil}}} > >>> > >>> gb_trees:update(limit,true,X). > >> > >> {3,{limit,true,{trace,false,nil,nil},{timeout,-1,nil,nil}}} > >>> > >>> gb_trees:update(timeout,true,X). > >> > >> {3,{limit,-1,{trace,false,nil,nil},{timeout,true,nil,nil}}} > >>> > >>> gb_trees:update(trace,true,X). > >> > >> ** exception error: no function clause matching > >> ? ? ? ? ? ? ? ? ? ?gb_trees:update_1(trace,true,nil) > >> ? ? in function ?gb_trees:update_1/3 > >> ? ? in call from gb_trees:update/3 > > > > The update calls that you think they work, work by fluke. The > problem is > > that even though you may think you have created a gb tree, but in > fact you > > have not. > > > > The problem lies in your first call: > > > > ?X = > gb_trees:from_orddict([{trace,false},{limit,-1},{timeout,-1}]). > > > > You promised you will supply an orddict, but in fact you have not. > > Use: > > > > ?X = > > > gb_trees:from_orddict(orddict:from_list([{trace,false},{limit,-1},{timeout,-1}])). > > > > instead and things will work as expected. > > > > Kostis > > > > Thank you for the analysis! > > Maybe we should update the documentation? It's pretty common to use a > list of tuples as a dictionary in other languages and that's what > appears to be specified in the Erlang documentation: > > ---------- > from_orddict(List) -> Tree > > Types: > > List = [{Key, Val}] > Key = Val = term() > Tree = gb_tree() > > Turns an ordered list List of key-value tuples into a tree. The list > must not contain duplicate keys. > ---------- > > Although on a second reading, it seems that the word "orderered" in > "ordered list" is key. So is an orddict just an ordered list of > tuples? But then again, why would it have to_list/1 if it was just a > list? > > Cheers, > > Michal > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From tony.arcieri@REDACTED Fri Nov 5 21:45:57 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Fri, 5 Nov 2010 14:45:57 -0600 Subject: [erlang-questions] Re: Shared/Hybrid Heap In-Reply-To: References: <420938007.35431288831679157.JavaMail.root@zimbra> <842437213.35451288831826841.JavaMail.root@zimbra> <0933BBC4-4F98-4752-B90A-9407C821F98C@trifork.com> Message-ID: Interesting, it is indeed faster, although not as much as I was expecting. Ulf Wiger has often cited Chameneos as the type of problem to which Erlang is ill-suited. It's certainly interesting to see that the benchmarks show Erjang doing better, although I'm uncertain as to whether the shared heap is the reason or not. On Fri, Nov 5, 2010 at 1:05 AM, Kresten Krab Thorup wrote: > If you're interested ... on my MacBook pro 2.53GHz Intel Core 2 Duo, I get > the following numbers: > > [HiPE, erts-5.8] > $ time erl -noshell -run chameneosredux main 6000000 > > real 1m16.766s > user 1m34.237s > sys 0m44.391s > > CPU load is ~ [95%, 95%] > top reporting ~6MB real memory > > For some reason, BEAM (sans HiPE) is slightly faster > > real 1m9.116s > user 1m25.769s > sys 0m35.669s > > > Now erjang, ... > > [Erjang, erts-5.8] > $ time ej +S 1 -noshell -run chameneosredux main 6000000 > > real 0m46.366s > user 0m47.458s > sys 0m0.606s > > CPU load is ~[50%, 50%] > top reporting ~75MB real memory > java invoked using -server -Xmx15mb [15 mb heap] > > Increasing the heap size does not make Erjang go faster/slower, but ~10MB > seems to be the working set size of the heap. I.e. setting the total heap > size below that will put significant pressure on the GC. > > > Erjang on 64-bit Java and a double the heap runs marginally faster. > > > For reference, the plain "java -server version" > http://shootout.alioth.debian.org/u32q/benchmark.php?test=chameneosredux&lang=java > runs like this on my machine > > real 0m4.900s > user 0m8.460s > sys 0m0.098s > > So my machine runs ~ 1.6 x faster than the one used in the shootout. > > > This should be very easy to try on an Azul box; but right now I think the > bottleneck will be that my schedulers will end up competing for work in an > dumb way. Running erjang +S 2 (i.e. one scheduler per core) makes erjang > run at-par with BEAM/HiPE. > > > Interestingly, the profile for the running Erjang looks like this, and for > some reason the critical function EProc.execute is never JIT'ed. Will have > to figure out why that is and there should be a potential huge win. > [enabling the Java profiler slows it down a bit, which is why this is a > profile of ~52 seconds :-] > > Flat profile of 51.83 secs (4452 total ticks): Thread-1 > > Interpreted + native Method > 78.4% 3399 + 0 erjang.EProc.execute > 1.4% 7 + 52 java.lang.Class.getDeclaredConstructors0 > 0.5% 0 + 23 java.util.zip.Inflater.inflateBytes > 0.4% 18 + 0 java.lang.ClassLoader.defineClass1 > 0.3% 0 + 13 java.lang.Class.forName0 > 0.1% 0 + 5 java.util.zip.ZipFile.read > 0.1% 5 + 0 erjang.m.erlang.ErlBif.apply > 0.1% 2 + 3 java.security.AccessController.doPrivileged > 0.1% 5 + 0 kilim.Fiber.togglePause > > Compiled + native Method > 2.9% 51 + 74 erjang.m.chameneosredux.chameneosredux.chameneos__4 > 2.3% 83 + 17 kilim.Mailbox.untilHasMessages > 2.0% 67 + 19 erjang.m.chameneosredux.chameneosredux.broker__1 > 1.6% 71 + 0 erjang.EHandle.send > 1.6% 44 + 25 erjang.ERT.wait > 1.6% 68 + 0 kilim.WorkerThread.run > 0.8% 36 + 0 erjang.ERT.send > 0.4% 17 + 0 erjang.m.erlang.ErlBif.apply > 0.2% 10 + 0 > erjang.m.chameneosredux.chameneosredux$FN_chameneos__4.go > 0.2% 9 + 0 erjang.m.chameneosredux.chameneosredux.main__1 > > > > Kresten > > > On Nov 4, 2010, at 23:17 , Tony Arcieri wrote: > > Hi Kresten, > > I'd be very interested to know how Chameneos performs under Erjang on an > Azul system, although a comparison with BEAM might prove difficult: > > http://shootout.alioth.debian.org/u32q/performance.php?test=chameneosredux > > < > http://shootout.alioth.debian.org/u32q/performance.php?test=chameneosredux>It > seems like the type of problem that could really benefit from Erjang and its > shared heap. > > On Thu, Nov 4, 2010 at 12:29 PM, Kresten Krab Thorup > wrote: > Yes, it would be interesting to know how erjang behaves in such an > environment. Have been talking to Gil Tene (Azul CTO) here at QCon about > doing that; but we need some system to test that is sufficiently complex, > runs on erjang, and who's behavior is known so we can make relevant > evaluation. Any takers? I can help connect people and make this happen. > > Kresten > > > On 04/11/2010, at 09.39, "Tony Arcieri" tony.arcieri@REDACTED>> wrote: > > > On Wed, Nov 3, 2010 at 6:50 PM, Robert Virding < > > robert.virding@REDACTED robert.virding@REDACTED>> wrote: > > > >> 1. Shared/hybrid heaps are wonderful BUT: > >> - They are more difficult to implement in a real-time environment. Yes, > >> this is a known problem with solutions but the benefits are less than > you > >> would expect. Real-time gc costs. > >> - They don't port easily to multi-core systems as you suddenly start > >> needing locks and stuff everywhere to control access or have complex > schemes > >> to get around this. Which copying doesn't. > >> > > > > There's been a considerable amount of research on this matter on the Java > > platform. The Azul Java platform addresses both of these concerns, with a > > pauseless GC designed for massively multi-core systems (54 cores per > CPU): > > > > http://www.azulsystems.com/products/vega/overview > > > > I'd be curious how well Erjang would work on such a system, particularly > on > > the types of shared state concurrency problems that don't work well on > BEAM, > > like Chameneos. > > > > -- > > Tony Arcieri > > Medioh! A Kudelski Brand > > > > -- > Tony Arcieri > Medioh! A Kudelski Brand > > Kresten Krab Thorup, CTO, Trifork > > -- Tony Arcieri Medioh! A Kudelski Brand From co7eb@REDACTED Fri Nov 5 22:45:42 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Fri, 05 Nov 2010 16:45:42 -0500 Subject: ExtendedVisualOtp (EVO) Release 1.4 Message-ID: Hi all!! here a Release 1.4 of EVO, I hope some feedbacks from you. ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: EVO 1.4 (distribute ready, 03-Nov-2010).rar Type: application/octet-stream Size: 67079 bytes Desc: not available URL: From francisstephens@REDACTED Sat Nov 6 00:24:39 2010 From: francisstephens@REDACTED (Francis Stephens) Date: Fri, 5 Nov 2010 23:24:39 +0000 Subject: MPI based BST and AV Message-ID: Jiansen, I am interested in your problem. However, I don't think I fully understand what you are asking. Are you able to spell out more clearly what problem you are trying to solve? Francis From co7eb@REDACTED Sat Nov 6 03:35:03 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Fri, 05 Nov 2010 21:35:03 -0500 Subject: ExtendedVisualOtp EVO Documentation 1/4 Message-ID: Hi all, here the documentation for EVO. part 1 ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. -------------- next part -------------- A non-text attachment was scrubbed... Name: ExtendedVisualOtp (English Version).part1.rar Type: application/x-rar-compressed Size: 80000 bytes Desc: not available URL: From co7eb@REDACTED Sat Nov 6 03:37:07 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Fri, 05 Nov 2010 21:37:07 -0500 Subject: ExtendedVisualOtp EVO Documentation 2/4 Message-ID: part 2 ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. -------------- next part -------------- A non-text attachment was scrubbed... Name: ExtendedVisualOtp (English Version).part2.rar Type: application/x-rar-compressed Size: 80000 bytes Desc: not available URL: From co7eb@REDACTED Sat Nov 6 03:39:25 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Fri, 05 Nov 2010 21:39:25 -0500 Subject: ExtendedVisualOtp EVO Documentation 3/4 Message-ID: part 3 ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. -------------- next part -------------- A non-text attachment was scrubbed... Name: ExtendedVisualOtp (English Version).part3.rar Type: application/x-rar-compressed Size: 80000 bytes Desc: not available URL: From co7eb@REDACTED Sat Nov 6 03:39:54 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Fri, 05 Nov 2010 21:39:54 -0500 Subject: ExtendedVisualOtp EVO Documentation 4/4 Message-ID: part 4 ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. -------------- next part -------------- A non-text attachment was scrubbed... Name: ExtendedVisualOtp (English Version).part4.rar Type: application/x-rar-compressed Size: 60043 bytes Desc: not available URL: From uaforum1@REDACTED Sat Nov 6 06:27:30 2010 From: uaforum1@REDACTED (u a) Date: Fri, 5 Nov 2010 22:27:30 -0700 (PDT) Subject: Wurfl and erlang Message-ID: Hello, is anybody using http://wurfl.sourceforge.net/ and erlang together. I mean, has anybody implemented an erlang module to query with the capabilities and get a list of devices. Thanks, Ulf From jiansenhe@REDACTED Sat Nov 6 16:45:49 2010 From: jiansenhe@REDACTED (Jiansen He) Date: Sat, 6 Nov 2010 15:45:49 +0000 Subject: [erlang-questions] MPI based BST and AVL In-Reply-To: References: Message-ID: BST is a common data structure. In some cases, the speed of producing data might be faster than the speed of inserting data. Or, many users might want to do operations(inserting, deleting, etc.) at the same time, if the tree is shared by many users. It is practical to do tree (both BST and AVL) operations concurrently based on locks and (software transactional memory) STM. I wonder if it is practical to build concurrent BST/AVL in message passing style. In a locks/STM based concurrent BST/AVL, it is parent node's job to check the accessibility of child nodes before performing an operation. This increase the complexity of programming. A beauty of message passing style is that the sender does not need to worry about the status of receivers. If a concurrent BST/AVL is build in message passing style, the solution will be simpler, even if it will not give a better performance. Please let me know if the problem is not clear. I plan to build such a tree if I have time. A design for concurrent BST has in my mind for a while, but it might be difficulty to build a lock free concurrent AVL. In the mean time, I wounder if someone else has implemented this idea, or it might be in library. Jiansen On Fri, Nov 5, 2010 at 11:24 PM, Francis Stephens wrote: > Jiansen, > > I am interested in your problem. However, I don't think I fully understand > what you are asking. Are you able to spell out more clearly what problem > you are trying to solve? > > Francis > From kostis@REDACTED Sat Nov 6 17:53:21 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Sat, 06 Nov 2010 18:53:21 +0200 Subject: [erlang-questions] gb_trees, strange behaviour In-Reply-To: References: <4CD4446D.1030401@cs.ntua.gr> Message-ID: <4CD58801.9000001@cs.ntua.gr> Michal D. wrote: > > Maybe we should update the documentation? It's pretty common to use a > list of tuples as a dictionary in other languages and that's what > appears to be specified in the Erlang documentation: > > ---------- > from_orddict(List) -> Tree > > Types: > > List = [{Key, Val}] > Key = Val = term() > Tree = gb_tree() > > Turns an ordered list List of key-value tuples into a tree. The list > must not contain duplicate keys. > ---------- > > Although on a second reading, it seems that the word "ordered" in > "ordered list" is key. So is an orddict just an ordered list of tuples? In my opinion, the documentation should definitely be updated. It should instead read: from_orddict(OrdDict) -> Tree Types: OrdDict = orddict(Key, Val) (see ) Also, in an ideal world, the orddict() type should have been declared opaque instead so that tools such as dialyzer complain when finding code that creates random lists (like yours) and passes them to a function expecting an orddict() instead. I think this change is still doable and should happen. I volunteer to do this if there is enough consensus in the community. Kostis From mjtruog@REDACTED Sat Nov 6 18:27:49 2010 From: mjtruog@REDACTED (Michael Truog) Date: Sat, 06 Nov 2010 10:27:49 -0700 Subject: [erlang-questions] MPI based BST and AVL In-Reply-To: References: Message-ID: <4CD59015.8050609@gmail.com> A library that might be useful for parallel algorithms in a functional language is here: http://www.cs.cmu.edu/~scandal/nesl.html On 11/06/2010 08:45 AM, Jiansen He wrote: > BST is a common data structure. In some cases, the speed of producing data > might be faster than the speed of inserting data. Or, many users might want > to do operations(inserting, deleting, etc.) at the same time, if the tree is > shared by many users. > > It is practical to do tree (both BST and AVL) operations concurrently based > on locks and (software transactional memory) STM. I wonder if it is > practical to build concurrent BST/AVL in message passing style. > > In a locks/STM based concurrent BST/AVL, it is parent node's job to check > the accessibility of child nodes before performing an operation. This > increase the complexity of programming. A beauty of message passing style > is that the sender does not need to worry about the status of receivers. > If a concurrent BST/AVL is build in message passing style, the solution will > be simpler, even if it will not give a better performance. > > Please let me know if the problem is not clear. > > I plan to build such a tree if I have time. A design for concurrent BST has > in my mind for a while, but it might be difficulty to build a lock free > concurrent AVL. > > In the mean time, I wounder if someone else has implemented this idea, or it > might be in library. > > > Jiansen > > > On Fri, Nov 5, 2010 at 11:24 PM, Francis Stephens >> wrote: >> > >> Jiansen, >> >> I am interested in your problem. However, I don't think I fully understand >> what you are asking. Are you able to spell out more clearly what problem >> you are trying to solve? >> >> Francis >> >> > From zvi.avraham@REDACTED Sat Nov 6 23:51:54 2010 From: zvi.avraham@REDACTED (Zvi) Date: Sat, 6 Nov 2010 15:51:54 -0700 (PDT) Subject: Is gen_server behavior in R14B hibernate friendly? Message-ID: <95fcaa07-6a18-4681-804c-619481310077@e26g2000vbz.googlegroups.com> Hi, I re-reading Erlang at Facebook slides. And found this. Is this still an issue? Zvi "Hibernation ? Drastically shrink memory usage with erlang:hibernate/3 ? Throws away the call stack ? Minimizes the heap ? Enters a wait state for new messages ? ?Jumps? into a passed-in function for a received message ? Perfect for a long-running, idling HTTP request handler ? But ... not compatible with gen_server:call (and gen_server:reply) ? gen_server:call has its own receive() loop ? hibernate() doesn?t support have an explicit timeout ? Fixed with a few hours and a look at gen.erl" From tonyhannan2@REDACTED Sun Nov 7 00:48:49 2010 From: tonyhannan2@REDACTED (Tony Hannan) Date: Sat, 6 Nov 2010 19:48:49 -0400 Subject: Built-in type var(_) Message-ID: Hello, I was writing an MVar module (a' la Haskell) and defining the type -type var(A) :: {var, pid(), A}. % initial value kept in reference for type only, current value held in process But the compiler complained that "type var(_) is a builtin type; it cannot be redefined" I don't see this built-in var mentioned in any of the documentation and Google didn't find it either. So where is this var type defined and can I use it instead of creating my own var type. Thanks, Tony From kostis@REDACTED Sun Nov 7 02:05:35 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Sun, 07 Nov 2010 03:05:35 +0200 Subject: [erlang-questions] Built-in type var(_) In-Reply-To: References: Message-ID: <4CD5FB5F.9070208@cs.ntua.gr> Tony Hannan wrote: > Hello, > > I was writing an MVar module (a' la Haskell) and defining the type > > -type var(A) :: {var, pid(), A}. > % initial value kept in reference for type only, current value held in > process > > But the compiler complained that > > "type var(_) is a builtin type; it cannot be redefined" > > I don't see this built-in var mentioned in any of the documentation and > Google didn't find it either. So where is this var type defined and can I > use it instead of creating my own var type. The underlying type representation needs to have a way of representing type variables (the A in your declaration) and it so happens that it has chosen something involving var/1. I guess you are unlucky. Simply choose some other name like mvar(A), myvar(A), my_var(A), ... But you are right that this should probably be mentioned somewhere. Kostis From tonyhannan2@REDACTED Sun Nov 7 01:16:59 2010 From: tonyhannan2@REDACTED (Tony Hannan) Date: Sat, 6 Nov 2010 20:16:59 -0400 Subject: [erlang-questions] Built-in type var(_) In-Reply-To: <4CD5FB5F.9070208@cs.ntua.gr> References: <4CD5FB5F.9070208@cs.ntua.gr> Message-ID: Thanks, but I more interested in the built-in type var that I am conflicting with. Where is it? How do I use it? Thanks, Tony On Sat, Nov 6, 2010 at 9:05 PM, Kostis Sagonas wrote: > Tony Hannan wrote: > >> Hello, >> >> I was writing an MVar module (a' la Haskell) and defining the type >> >> -type var(A) :: {var, pid(), A}. >> % initial value kept in reference for type only, current value held in >> process >> >> But the compiler complained that >> >> "type var(_) is a builtin type; it cannot be redefined" >> >> I don't see this built-in var mentioned in any of the documentation and >> Google didn't find it either. So where is this var type defined and can I >> use it instead of creating my own var type. >> > > The underlying type representation needs to have a way of representing type > variables (the A in your declaration) and it so happens that it has chosen > something involving var/1. I guess you are unlucky. > > Simply choose some other name like mvar(A), myvar(A), my_var(A), ... > > But you are right that this should probably be mentioned somewhere. > > Kostis > From rujia.liu@REDACTED Sun Nov 7 02:31:44 2010 From: rujia.liu@REDACTED (Rujia Liu) Date: Sun, 7 Nov 2010 09:31:44 +0800 Subject: [erlang-questions] MPI based BST and AVL In-Reply-To: References: Message-ID: What is your main concern? The simplicity or the performance? you mentioned "If a concurrent BST/AVL is build in message passing style, the solution will be simpler, even if it will not give a better performance." so I assume simplicity is more important. Then what's wrong with the following straight-forward solution that comes to my mind? 1. employ a BST-controller process which receives commands, then execute them one at a time, just like normal serial case. 2. clients (process that needs to do insertions and deletions) send messages to the controller process. Maybe I misunderstood you, so could you please mention some details of the use case of your concurrent BST? The size of the BST, the amount/speed of tree operations, how well you want to scale. For example, if your BST is too big, you need to do something like: 1. dynamically create and destroy erlang nodes (don't do this if the size of your tree does not change drastically during execution) 2. operations happen at different parts of the tree goes to different erlang nodes 3. schedule some rebalances of the tree, by moving subtrees between erlang nodes (ps: I don't think keeping strict balance, like AVL, is a good idea to build concurrent BSTs) I have never implemented these idea, but if your BST is really huge, even the simpliest use of these ideas will be a lot better than the previous straight-forward solution I mentioned above. BTW: You may also consider external balanced trees like B trees/B+ trees. The tree nodes are easier to distribute between erlang nodes. Good luck! - Rujia On Sat, Nov 6, 2010 at 11:45 PM, Jiansen He wrote: > BST is a common data structure. ?In some cases, the speed of producing data > might be faster than the speed of inserting data. ?Or, many users might want > to do operations(inserting, deleting, etc.) at the same time, if the tree is > shared by many users. > > It is practical to do tree (both BST and AVL) operations concurrently based > on locks and (software transactional memory) STM. ?I wonder if it is > practical ?to build concurrent BST/AVL in message passing style. > > In a locks/STM based concurrent BST/AVL, it is parent node's job to check > the accessibility of child nodes before performing an operation. ?This > increase the complexity ?of programming. ?A beauty of message passing style > is that the sender does not need to worry about the status of receivers. > If a concurrent BST/AVL is build in message passing style, the solution will > be simpler, even if it will not give a better performance. > > Please let me know if the problem is not clear. > > I plan to build such a tree if I have time. ?A design for concurrent BST has > in my mind for a while, but it might be difficulty to build a lock free > concurrent AVL. > > In the mean time, I wounder if someone else has implemented this idea, or it > might be in library. > > > Jiansen > > > On Fri, Nov 5, 2010 at 11:24 PM, Francis Stephens > wrote: > >> Jiansen, >> >> I am interested in your problem. ?However, I don't think I fully understand >> what you are asking. ?Are you able to spell out more clearly what problem >> you are trying to solve? >> >> Francis >> > From mononcqc@REDACTED Sun Nov 7 03:49:55 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Sat, 6 Nov 2010 22:49:55 -0400 Subject: [erlang-questions] gb_trees, strange behaviour In-Reply-To: <4CD58801.9000001@cs.ntua.gr> References: <4CD4446D.1030401@cs.ntua.gr> <4CD58801.9000001@cs.ntua.gr> Message-ID: I definitely agree with that. The same should also be done with ordsets. On Sat, Nov 6, 2010 at 12:53 PM, Kostis Sagonas wrote: > > > Also, in an ideal world, the orddict() type should have been declared > opaque instead so that tools such as dialyzer complain when finding code > that creates random lists (like yours) and passes them to a function > expecting an orddict() instead. > > Kostis > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From uaforum1@REDACTED Sun Nov 7 06:39:27 2010 From: uaforum1@REDACTED (u a) Date: Sat, 6 Nov 2010 22:39:27 -0700 (PDT) Subject: Symbian SIS Message-ID: <6dc4dac4-0a34-46c2-bac6-bd1e1911c0a7@t13g2000yqm.googlegroups.com> Hello, does anybody has experience with reading Symbian sis files with erlang? I want to read the content of an sis file like the supported devices, OS and so on. Thanks for the help, Ulf From boris.muehmer@REDACTED Sun Nov 7 07:41:03 2010 From: boris.muehmer@REDACTED (=?UTF-8?Q?Boris_M=C3=BChmer?=) Date: Sun, 7 Nov 2010 07:41:03 +0100 Subject: Problems with distributed Erlang on Ubuntu (9.x and 10.x) because of avahi-daemon! Message-ID: I wanted to do some distributed Erlang tests on my systems at home. I already had success doing some tests earlier this year, but this time I couldn't get them working. Only two of my four systems could net_adm/ping each other. Than I realized, that the other two (not connecting) systems were (almost) fresh Ubuntu 10.10 installations, where I didn't remove/disable the avahi-daemon. After just stopping the avahi-daemon all four systems could successfully net_adm:ping each other again... I can reproduce this behaviour on Ubuntu 10.04 and 10.10 systems with Erlang R13B4 and R14B, but I think I also had this problem on Ubuntu 9.x systems. Did anyone else experience this kind of behaviour? - boris From john.hughes@REDACTED Sun Nov 7 10:36:13 2010 From: john.hughes@REDACTED (John Hughes) Date: Sun, 7 Nov 2010 10:36:13 +0100 Subject: [erlang-questions] gb_trees, strange behaviour In-Reply-To: References: <4CD4446D.1030401@cs.ntua.gr> <4CD58801.9000001@cs.ntua.gr> Message-ID: Wouldn't it be quite cheap to actually check the invariant in from_ord_dict? It also requires that there be no duplicate keys, which an orddict doesn't guarantee. Ran into this recently in a QuickCheck spec! John Sent from my iPhone On 7 Nov 2010, at 03:49, Fred Hebert wrote: > I definitely agree with that. The same should also be done with ordsets. > > On Sat, Nov 6, 2010 at 12:53 PM, Kostis Sagonas wrote: > >> >> >> Also, in an ideal world, the orddict() type should have been declared >> opaque instead so that tools such as dialyzer complain when finding code >> that creates random lists (like yours) and passes them to a function >> expecting an orddict() instead. >> >> Kostis >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> From rvg@REDACTED Sun Nov 7 12:26:48 2010 From: rvg@REDACTED (Rudolph van Graan) Date: Sun, 7 Nov 2010 11:26:48 +0000 Subject: [erlang-questions] Efficient Insertions in Mnesia tables In-Reply-To: References: Message-ID: Hi, This sounds like a disk subsystem issue. DETS (disc_only mnesia tables) uses buckets to store objects and will allocate (and reallocate) objects within buckets as you add more objects to it. If a bucket does not have space for a new object, the bucket must be split. This means the DETS file grows and some of the data is moved. Depending on your operating system, file system, record size on the file system, this will result in a lot of IO. In my opinion, what you see is to be expected - DETS selects a bucket based on the object's key's MD5 hash, so a specific insert can hit any bucket essentially at random. DETS is not a good choice if you want to constantly append to a table, but it works reasonably well if you have a finite set of keys. Rudolph van Graan www.patternmatched.com On Nov 5, 2010, at 10:15 AM, Olivier BOUDEVILLE wrote: > Hi, > > We are trying to write (with mnesia:dirty_write) in a disc_only_copies > Mnesia table (type: set, not fragmented, not replicated) records (ex: 60 > 000 of them) and we observe that the insertion time is increasing as the > table is increasingly crowded. This is not really a surprise but something > we need to avoid. What we would like is to have constant (and preferably > low) insertion times, like we had when writing directly to a file. > > We tried to get as close as possible with the following settings and use: > > % We want tables to be dumped less frequently from > memory to disc, > % in order to buffer writings (default value is > 4): > ok = application:set_env( mnesia, dc_dump_limit, 1 > ), > > % Increases a lot (default value is 100) the > maximum number of > % writes to the transaction log before a new dump > is performed: > ok = application:set_env( mnesia, > dump_log_write_threshold, 50000 ), > > Over time we see the CPU load decrease steadily, the computer seems to > spend most of its time fighting for locks. > > We happen to be in a pretty favorable situation (only writes, no > concurrent access to a given table). We chose disc_only_copies as there > might be a large number of such tables and if they filled over time they > could exhaust the RAM. > > Is there anything we missed that would allow us (roughly) constant > insertion times with Mnesia? > > Thanks in advance for any hint, > Best regards, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 > 65 27 13 > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3822 bytes Desc: not available URL: From rvg@REDACTED Sun Nov 7 13:09:04 2010 From: rvg@REDACTED (Rudolph van Graan) Date: Sun, 7 Nov 2010 12:09:04 +0000 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> Hi Joe, I am not sure I agree with your statement that traditional databases "suffer" from the fact that columns have simple types. You can easily (within reason) make them store complex types. They are at the most basic level also KV stores. The most basic abstraction for a "Table" is a set of tuples stored in a certain way. There is no reason why you cannot define a table with an arbitrary key and an arbitrary "blob" as value column and then storing your parse tree in that blob field. (Obviously you need to serialise/deserialise it - as you would need to do for any storage abstraction). The issue is that the information in the "blob" is not useful. Neither stored in the RDBMS, nor in a plain KV-store. You have to make it useful before you can do something with it or extract information from it. I guess that is why you said thus: > I'm not thinking in terms of joins and normalizing things - the thought process > is different - so far I haven't met any problems that don't map onto key-values > queries. I agree with you. If you throw distribution, map-reduce and lots of indices into the equation you can solve most issues this way. Except for one thing - invariance between multiple records and record types. There are very few examples where it is useful to live without invariance - searching being one example. Invariance is a first order concept in any RDBMS system and essential for many types of systems. However, in my experience, you can achieve application-level invariance in other ways. But it requires substantially more work to approximate first order invariance using KV-stores. It is sometimes simpler to have your data properly normalised I think. Rudolph van Graan On Nov 4, 2010, at 10:01 PM, Joe Armstrong wrote: > On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >> This is a message I sent to the nosql-discussion@REDACTED >> discussion group. I thought it might be interesting to send it >> erlang-questions, so here we go... >> >> >> Hi all! >> >> I have used SQL RDBMSs for some time. I've never used any very advanced >> feature, but I know enough of it to make database applications. >> >> Nowadays, I decided it would be interesting to learn some NoSQL >> databases concepts. So I decided to pick up some Erlang and Mnesia, its >> native key-value database. More than scalability itself, the most >> valuable feature for me is the possibility of replication and >> synchronization between nodes. >> >> But all pros have cons as companion. The lack of a relationship model >> is difficult for who is used to RDBMSs. So, my question is: >> >> * Is there any guide, tutorial, book, whatever, that tries to introduce >> NoSQL databases to SQL users? >> >> * Key-value databases are surprising simple. I know you solve >> relationship by denormalizing data. What data should be normalized? >> What shouldn't? How do you update denormalized data? > > I'm no database expert so don't quote me here ... > > As far as I am concerned traditional databases like SQL suffer > from the fact that the data stored in an individual column is incredible > simple - I can store an integer/string/... in a column but these are > incredibly simple data structures. I want to store and retrieve incredibly > complicated things - how do I store an XML parse tree in a single cell > of a database? - How do I store a database in a database ... > > In a decent K-V database the value of the key can be *anything* - an > entire database for example, a compiler, ... no worries > > Then when I analyse my problem I start thinking "I can store and retrieve any > complex object that keys do I need to solve my problem?" > > I'm not thinking in terms of joins and normalizing things - the thought process > is different - so far I haven't met any problems that don't map onto key-values > queries. > > It seems to my that SQL provides you with the ability do to complex > queries on simple things. K-V dbs can do simple queries on complex > things. > > /Joe > > > >> >> Sorry for such simple and general questions. Things were simple up to >> the moment that I realized that it would be easily solved with a JOIN >> SQL statement. :-) >> >> Thank you very much! >> >> -- >> Silas Silva >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3822 bytes Desc: not available URL: From petter.egesund@REDACTED Sun Nov 7 15:49:57 2010 From: petter.egesund@REDACTED (Petter Egesund) Date: Sun, 7 Nov 2010 15:49:57 +0100 Subject: Unstable erlang compared to java or perl Message-ID: Hi, I have a small program with lots of memory-updates which I try to run in Erlang. The same algorithm works fine in both Java and Perl, but fails in Erlang because the program runs out of memory - and I can not figure out why. Frustrating, as my Erlang-versjon seems to be the easiest to scale as well as being the most readable. The program is threaded and each thread writes to a ets-table which is created at the beginning of the thread. When the thread dies I try to do a ets:delete(Table), like described in the manual, but the memory used by the thread never seems to be released. Some facts: - The memory usage of each thread is rather constant. This is confirmed when I use ets:i() to show info about memory usage. - The number of threads are constant - confirmed by both running top and writing out the number of threads regularly. When a thread dies, I create a new one. - I have tried to end the thread by sending a exit-signal as the last statement. This helps some, but does not solve the leak. - I put small lists of size 3-4 integers into the ets as values, the keys are list of same size as well. - I garbage-collect each thread before it dies, as well as doing regular global garbage-collects. No help. - Information from ets:i() about memory when I sum usage by each thread, is much lower than stated by memory() when i run erlang:memory(). This might indicate something? Does not seem logical to me, at least. - Info from erlang:memory is about half of what top/the os tells. - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well. Any clues? Dump from ets:i() and erlang:memory() is like below. Cheers, Petter --- dump --- eNumber of processes: 27 ets:i(): id name type size mem owner ---------------------------------------------------------------------------- 13 code set 261 10692 code_server 4110 code_names set 58 7804 code_server 6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0> 7022018584 the_synapses ordered_set 15143493 310909950 <0.48.0> 7774416922 the_synapses ordered_set 8794649 182005810 <0.49.0> ac_tab ac_tab set 6 848 application_controller file_io_servers file_io_servers set 0 302 file_server_2 global_locks global_locks set 0 302 global_name_server global_names global_names set 0 302 global_name_server global_names_ext global_names_ext set 0 302 global_name_server global_pid_ids global_pid_ids bag 0 302 global_name_server global_pid_names global_pid_names bag 0 302 global_name_server inet_cache inet_cache bag 0 302 inet_db inet_db inet_db set 29 571 inet_db inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db inet_hosts_byname inet_hosts_byname bag 0 302 inet_db inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302 inet_db inet_hosts_file_byname inet_hosts_file_byname bag 0 302 inet_db neurone_counter neurone_counter set 258394 1846182 entity_server neurone_group_counter neurone_group_counter set 6 344 entity_group_server neurone_group_name neurone_group_name set 6 426 entity_group_server neurone_group_name_reverse neurone_group_name_reverse set 6 426 entity_group_server neurone_name neurone_name set 258394 11824602 entity_server neurone_name_reverse neurone_name_reverse set 258394 11824602 entity_server memory(): [{total,5568669792}, {processes,1138936}, {processes_used,1128120}, {system,5567530856}, {atom,349769}, {atom_used,336605}, {binary,82704}, {code,3046365}, {ets,5562163256}] From ulf.wiger@REDACTED Sun Nov 7 16:03:44 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Sun, 7 Nov 2010 07:03:44 -0800 Subject: [erlang-questions] Is gen_server behavior in R14B hibernate friendly? In-Reply-To: <95fcaa07-6a18-4681-804c-619481310077@e26g2000vbz.googlegroups.com> References: <95fcaa07-6a18-4681-804c-619481310077@e26g2000vbz.googlegroups.com> Message-ID: <5785C41E-9A80-41A4-8A95-17F46D65A7DA@erlang-solutions.com> Hi Zvi, Actually, nowadays, gen_server supports 'hibernate' as a special value of the internal timeout, so your server callbacks can instruct the main loop to go into hibernation between messages. It is true that hibernate/3 doesn't have a timeout, but OTOH, it is easy enough to start a timer that will send a message, bringing you out of hibernation, after a specified time. BR, Ulf W On 6 Nov 2010, at 15:51, Zvi wrote: > Hi, > I re-reading Erlang at Facebook slides. And found this. Is this still > an issue? > > Zvi > > "Hibernation > ? > Drastically shrink memory usage with erlang:hibernate/3 > ? Throws away the call stack > ? Minimizes the heap > ? Enters a wait state for new messages > ? ?Jumps? into a passed-in function for a received message > ? Perfect for a long-running, idling HTTP request handler > ? But ... not compatible with gen_server:call (and gen_server:reply) > ? gen_server:call has its own receive() loop > ? hibernate() doesn?t support have an explicit timeout > ? Fixed with a few hours and a look at gen.erl" > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From steven.charles.davis@REDACTED Sun Nov 7 16:06:03 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Sun, 7 Nov 2010 07:06:03 -0800 (PST) Subject: Conceptual questions on key-value databases for RDBMs users In-Reply-To: <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> Message-ID: <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> On Nov 7, 6:09?am, Rudolph van Graan wrote: > Invariance is a first order concept in any RDBMS system and essential for many types of systems. However, in my experience, you can achieve application-level invariance in other ways. But it requires substantially more work to approximate first order invariance using KV-stores. ?It is sometimes simpler to have your data properly normalised I think. "not an expert" caveat applies to me also... It appears to me that this discussion is another expression of the 'strong vs weak/dynamic vs static type' discussion. Some observations/connections that seem relevant to me: - The underlying nature of SQL DBs is revealed by the fact that queries by anything except the primary key usually need additional work (sequences/indexes) to be respectably efficient. - C.J.Date uses the term tuple to refer to a "row in a table" (cf also Mnesia). - Date also notes that "objects" can always be mapped directly on to a relational model. - Mnesia is interesting in that it encourages a table to be confined to a single record type, and allows indexing of record tuple members (cf column indexing in SQL databases). - The process of normalization suggests very strong typing of the data. ...it makes me suspect that an imperative and strongly-typed language paradigm has been a very strong motivator in the evolution of SQL databases; and perhaps the popularity of NoSQL/NotSQL is an expression/ outcome of the rise of recent trends in programming language uptake. /s From bengt.kleberg@REDACTED Sun Nov 7 16:27:35 2010 From: bengt.kleberg@REDACTED (Bengt Kleberg) Date: Sun, 07 Nov 2010 16:27:35 +0100 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: <1289143655.14188.1.camel@seasc1137> Greetings, Could you include the program, or preferably a small subset of it, that runs out of memory? bengt On Sun, 2010-11-07 at 15:49 +0100, Petter Egesund wrote: > Hi, I have a small program with lots of memory-updates which I try to > run in Erlang. > > The same algorithm works fine in both Java and Perl, but fails in > Erlang because the program runs out of memory - and I can not figure > out why. Frustrating, as my Erlang-versjon seems to be the easiest to > scale as well as being the most readable. > > The program is threaded and each thread writes to a ets-table which is > created at the beginning of the thread. When the thread dies I try to > do a ets:delete(Table), like described in the manual, but the memory > used by the thread never seems to be released. > > Some facts: > > - The memory usage of each thread is rather constant. This is > confirmed when I use ets:i() to show info about memory usage. > - The number of threads are constant - confirmed by both running top > and writing out the number of threads regularly. When a thread dies, I > create a new one. > - I have tried to end the thread by sending a exit-signal as the last > statement. This helps some, but does not solve the leak. > - I put small lists of size 3-4 integers into the ets as values, the > keys are list of same size as well. > - I garbage-collect each thread before it dies, as well as doing > regular global garbage-collects. No help. > - Information from ets:i() about memory when I sum usage by each > thread, is much lower than stated by memory() when i run > erlang:memory(). This might indicate something? Does not seem logical > to me, at least. > - Info from erlang:memory is about half of what top/the os tells. > - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well. > > Any clues? Dump from ets:i() and erlang:memory() is like below. > > Cheers, > > Petter > > --- dump --- > > eNumber of processes: 27 > ets:i(): > id name type size mem owner > ---------------------------------------------------------------------------- > 13 code set 261 10692 code_server > 4110 code_names set 58 7804 code_server > 6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0> > 7022018584 the_synapses ordered_set 15143493 310909950 <0.48.0> > 7774416922 the_synapses ordered_set 8794649 182005810 <0.49.0> > ac_tab ac_tab set 6 848 application_controller > file_io_servers file_io_servers set 0 302 file_server_2 > global_locks global_locks set 0 302 global_name_server > global_names global_names set 0 302 global_name_server > global_names_ext global_names_ext set 0 302 global_name_server > global_pid_ids global_pid_ids bag 0 302 global_name_server > global_pid_names global_pid_names bag 0 302 global_name_server > inet_cache inet_cache bag 0 302 inet_db > inet_db inet_db set 29 571 inet_db > inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db > inet_hosts_byname inet_hosts_byname bag 0 302 inet_db > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302 inet_db > inet_hosts_file_byname inet_hosts_file_byname bag 0 302 inet_db > neurone_counter neurone_counter set 258394 1846182 entity_server > neurone_group_counter neurone_group_counter set 6 344 > entity_group_server > neurone_group_name neurone_group_name set 6 426 entity_group_server > neurone_group_name_reverse neurone_group_name_reverse set 6 > 426 entity_group_server > neurone_name neurone_name set 258394 11824602 entity_server > neurone_name_reverse neurone_name_reverse set 258394 11824602 entity_server > memory(): [{total,5568669792}, > {processes,1138936}, > {processes_used,1128120}, > {system,5567530856}, > {atom,349769}, > {atom_used,336605}, > {binary,82704}, > {code,3046365}, > {ets,5562163256}] > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From petter.egesund@REDACTED Sun Nov 7 16:39:43 2010 From: petter.egesund@REDACTED (Petter Egesund) Date: Sun, 7 Nov 2010 16:39:43 +0100 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: <1289143655.14188.1.camel@seasc1137> References: <1289143655.14188.1.camel@seasc1137> Message-ID: Sure :-) The most important part of my main loop works like this: Spawn up to a given number of threads. When a thread dies it sends a msg back to the main loop, so I can spawn a new one. And so on... The problem, as it seems to me, is that ets-garbage from dead threads never are collected? I have also tried to do this from my initial call: erlang:system_flag(fullsweep_after, 0), but without any change. Code: parse_dir_loop(ThreadCounter, MaxThreads, Dir, OldSpawnFiles) -> io:fwrite("Number of processes: ~p ~n", [length(processes())]), ets:i(), ThreadCounterNew = case ThreadCounter < MaxThreads of true -> {ok, RemainingFiles} = file:list_dir(Dir), case RemainingFiles of [] -> SpawnFiles = [], ThreadCounter; Files -> SpawnFilesNum = min(MaxThreads - ThreadCounter, length(Files -- OldSpawnFiles)), SpawnFiles = lists:sublist(lists:sort(Files -- OldSpawnFiles), 1, SpawnFilesNum), Receiver = self(), lists:foreach(fun(File) -> spawn_link(fun() -> parse_file(Dir ++ File), %% Receiver ! {done, File}, exit({kill_me, File}) end) end, SpawnFiles), ThreadCounter + SpawnFilesNum end; false -> SpawnFiles = [], ThreadCounter end, NewSpawnFiles = OldSpawnFiles ++ SpawnFiles, receive {done, File} -> io:fwrite("Done with a file ~n"), parse_dir_loop(ThreadCounterNew - 1, MaxThreads, Dir, NewSpawnFiles -- File); {'EXIT', Pid, {kill_me, File}} -> io:fwrite("Should be dead by now: ~p ~n", [Pid]), parse_dir_loop(ThreadCounterNew - 1, MaxThreads, Dir, NewSpawnFiles -- File); {'EXIT', Pid, normal} -> io:fwrite("Should be dead by now: ~p ~n", [Pid]), garbage_collect(Pid), parse_dir_loop(ThreadCounter, MaxThreads, Dir, NewSpawnFiles); Signal -> io:fwrite("Got unknown signal in parse_dir_loop ~p~n", [Signal]) after 5000 -> case file:list_dir(Dir) of {ok, []} -> io:fwrite("Parsing directory is empty - job done: ~p~n", [Dir]); {ok, _} -> parse_dir_loop(ThreadCounterNew, MaxThreads, Dir, NewSpawnFiles) end end. parse_dir(ThreadCounter, MaxThreads, Dir) -> %% probably fix path for Windows?! parse_dir_loop(ThreadCounter, MaxThreads, Dir, []). On Sun, Nov 7, 2010 at 4:27 PM, Bengt Kleberg wrote: > Greetings, > > Could you include the program, or preferably a small subset of it, that > runs out of memory? > > > bengt > > On Sun, 2010-11-07 at 15:49 +0100, Petter Egesund wrote: >> Hi, I have a small program with lots of memory-updates which I try to >> run in Erlang. >> >> The same algorithm works fine in both Java and Perl, but fails in >> Erlang because the program runs out of memory - and I can not figure >> out why. Frustrating, as my Erlang-versjon seems to be the easiest to >> scale as well as being the most readable. >> >> The program is threaded and each thread writes to a ets-table which is >> created at the beginning of the thread. When the thread dies I try to >> do a ets:delete(Table), like described in the manual, but the memory >> used by the thread never seems to be released. >> >> Some facts: >> >> - The memory usage of each thread is rather constant. This is >> confirmed when I use ets:i() to show info about memory usage. >> - The number of threads are constant - confirmed by both running top >> and writing out the number of threads regularly. When a thread dies, I >> create a new one. >> - I have tried to end the thread by sending a exit-signal as the last >> statement. This helps some, but does not solve the leak. >> - I put small lists of size 3-4 integers into the ets as values, the >> keys are list of same size as well. >> - I garbage-collect each thread before it dies, as well as doing >> regular global garbage-collects. No help. >> - Information from ets:i() about memory when I sum usage by each >> thread, is much lower than stated by memory() when i run >> erlang:memory(). This might indicate something? Does not seem logical >> to me, at least. >> - Info from erlang:memory is about half of what top/the os tells. >> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well. >> >> Any clues? Dump from ets:i() and erlang:memory() is like below. >> >> Cheers, >> >> Petter >> >> --- dump --- >> >> eNumber of processes: 27 >> ets:i(): >> ?id ? ? ? ? ? ? ?name ? ? ? ? ? ? ?type ?size ? mem ? ? ?owner >> ?---------------------------------------------------------------------------- >> ?13 ? ? ? ? ? ? ?code ? ? ? ? ? ? ?set ? 261 ? ?10692 ? ?code_server >> ?4110 ? ? ? ? ? ?code_names ? ? ? ?set ? 58 ? ? 7804 ? ? code_server >> ?6746271765 ? ? ?the_synapses ? ? ?ordered_set 5425194 113336012 <0.47.0> >> ?7022018584 ? ? ?the_synapses ? ? ?ordered_set 15143493 310909950 <0.48.0> >> ?7774416922 ? ? ?the_synapses ? ? ?ordered_set 8794649 182005810 <0.49.0> >> ?ac_tab ? ? ? ? ?ac_tab ? ? ? ? ? ?set ? 6 ? ? ?848 ? ? ?application_controller >> ?file_io_servers file_io_servers ? set ? 0 ? ? ?302 ? ? ?file_server_2 >> ?global_locks ? ?global_locks ? ? ?set ? 0 ? ? ?302 ? ? ?global_name_server >> ?global_names ? ?global_names ? ? ?set ? 0 ? ? ?302 ? ? ?global_name_server >> ?global_names_ext global_names_ext ?set ? 0 ? ? ?302 ? ? ?global_name_server >> ?global_pid_ids ?global_pid_ids ? ?bag ? 0 ? ? ?302 ? ? ?global_name_server >> ?global_pid_names global_pid_names ?bag ? 0 ? ? ?302 ? ? ?global_name_server >> ?inet_cache ? ? ?inet_cache ? ? ? ?bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_db ? ? ? ? inet_db ? ? ? ? ? set ? 29 ? ? 571 ? ? ?inet_db >> ?inet_hosts_byaddr inet_hosts_byaddr bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_hosts_byname inet_hosts_byname bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_hosts_file_byaddr inet_hosts_file_byaddr bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_hosts_file_byname inet_hosts_file_byname bag ? 0 ? ? ?302 ? ? ?inet_db >> ?neurone_counter neurone_counter ? set ? 258394 1846182 ?entity_server >> ?neurone_group_counter neurone_group_counter set ? 6 ? ? ?344 >> entity_group_server >> ?neurone_group_name neurone_group_name set ? 6 ? ? ?426 ? ? ?entity_group_server >> ?neurone_group_name_reverse neurone_group_name_reverse set ? 6 >> 426 ? ? ?entity_group_server >> ?neurone_name ? ?neurone_name ? ? ?set ? 258394 11824602 entity_server >> ?neurone_name_reverse neurone_name_reverse set ? 258394 11824602 entity_server >> memory(): ? ? ? [{total,5568669792}, >> ? ? ? ? ? ? ? ? ? ?{processes,1138936}, >> ? ? ? ? ? ? ? ? ? ?{processes_used,1128120}, >> ? ? ? ? ? ? ? ? ? ?{system,5567530856}, >> ? ? ? ? ? ? ? ? ? ?{atom,349769}, >> ? ? ? ? ? ? ? ? ? ?{atom_used,336605}, >> ? ? ? ? ? ? ? ? ? ?{binary,82704}, >> ? ? ? ? ? ? ? ? ? ?{code,3046365}, >> ? ? ? ? ? ? ? ? ? ?{ets,5562163256}] >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ebegumisa@REDACTED Sun Nov 7 18:16:43 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 08 Nov 2010 04:16:43 +1100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: Java's James Gosling seems to share your views... "... I've never got it when it comes to SQL databases. It's like, why? Just give me a hash table and a sh*tload of RAM and I'm happy. And then you do something to deal with failures. And you look at the way things like the NoSQL movement is. It's various flavors of large scale distributed hash tables and trying to deal with massive scale and massive replication, and you can't back up the database because no tape farm is big enough. And you find scale and reliability can fit together at the same time..." http://www.basementcoders.com/transcripts/James_Gosling_Transcript.html Funny, never thought two co-inventors of two different languages would actually agree on something :) The main reason I've switched to NoSQL (CouchDB) after many years of using SQL-RDBMSs is for modeling hierarchical data. RDBMSs force you to do this with relationships across different tables even when the data belongs to the same domain/object -- this feels unnatural and results in the DB equivalent of spaghetti code with hours staring at relationship diagrams. Take the simple example of orders that have items. In a typical SQL-RDBMS, this would normally be modeled using two tables related to each other in 1:many orders->items. A kv-store like Couch allows me to store this in a more sane manner, just a bunch of orders, each with items embedded inside them. But where kv-stores really shine is when your data has uneven levels of hierarchy within the same types of objects. I've always struggled with this in RDBMSs. I think a big reason kv-stores are winning over a lot of us long-time RDBMSs users is they allow us to model things-that-have-things-inside-them in the database much closer to how they are modeled in our applications. Orders/receipts/invoices with their items, users with their permissions, all these map nicely in kv-stores from db to application. This allows us to model relationships only when WE REALLY WANT RELATIONSHIPS (this receipt belongs to that invoice). That fact alone won me over and I've never looked back. - Edmond - On Fri, 05 Nov 2010 09:01:30 +1100, Joe Armstrong wrote: > On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >> This is a message I sent to the nosql-discussion@REDACTED >> discussion group. I thought it might be interesting to send it >> erlang-questions, so here we go... >> >> >> Hi all! >> >> I have used SQL RDBMSs for some time. I've never used any very advanced >> feature, but I know enough of it to make database applications. >> >> Nowadays, I decided it would be interesting to learn some NoSQL >> databases concepts. So I decided to pick up some Erlang and Mnesia, its >> native key-value database. More than scalability itself, the most >> valuable feature for me is the possibility of replication and >> synchronization between nodes. >> >> But all pros have cons as companion. The lack of a relationship model >> is difficult for who is used to RDBMSs. So, my question is: >> >> * Is there any guide, tutorial, book, whatever, that tries to introduce >> NoSQL databases to SQL users? >> >> * Key-value databases are surprising simple. I know you solve >> relationship by denormalizing data. What data should be normalized? >> What shouldn't? How do you update denormalized data? > > I'm no database expert so don't quote me here ... > > As far as I am concerned traditional databases like SQL suffer > from the fact that the data stored in an individual column is incredible > simple - I can store an integer/string/... in a column but these are > incredibly simple data structures. I want to store and retrieve > incredibly > complicated things - how do I store an XML parse tree in a single cell > of a database? - How do I store a database in a database ... > > In a decent K-V database the value of the key can be *anything* - an > entire database for example, a compiler, ... no worries > > Then when I analyse my problem I start thinking "I can store and > retrieve any > complex object that keys do I need to solve my problem?" > > I'm not thinking in terms of joins and normalizing things - the thought > process > is different - so far I haven't met any problems that don't map onto > key-values > queries. > > It seems to my that SQL provides you with the ability do to complex > queries on simple things. K-V dbs can do simple queries on complex > things. > > /Joe > > > >> >> Sorry for such simple and general questions. Things were simple up to >> the moment that I realized that it would be easily solved with a JOIN >> SQL statement. :-) >> >> Thank you very much! >> >> -- >> Silas Silva >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From bpedro@REDACTED Sun Nov 7 18:33:58 2010 From: bpedro@REDACTED (Bruno Pedro) Date: Sun, 7 Nov 2010 09:33:58 -0800 (PST) Subject: mnesia web interface In-Reply-To: References: Message-ID: <7290459c-7ccd-46fa-aa8e-15a7bd0a124d@a37g2000yqi.googlegroups.com> You might be interested in webnesia, a REST interface to mnesia and Web UI I'm writing: https://github.com/tarpipe/webnesia On Sep 19, 5:44?pm, Joe Armstrong wrote: > Is there a web interface to mnesia? > > /Joe > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > Seehttp://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED From bpedro@REDACTED Sun Nov 7 18:36:05 2010 From: bpedro@REDACTED (Bruno Pedro) Date: Sun, 7 Nov 2010 09:36:05 -0800 (PST) Subject: webnesia: a REST interface to mnesia and Web UI Message-ID: <69c62a29-7df3-434b-be0b-3312ad823dae@32g2000yqz.googlegroups.com> Hi all, I'm writing a REST interface to mnesia and also a Web UI to access any mnesia database. The code is available at github: https://github.com/tarpipe/webnesia Please let me know what you think. Cheers, Bruno Pedro From icarus.alive@REDACTED Sun Nov 7 19:00:16 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Sun, 7 Nov 2010 23:30:16 +0530 Subject: Why Erlang/OTP binaries are available only for Win32 ? Message-ID: wondering as to what is the reason that Erlang/OTP binaries are available only for win32 and not for Linux (and it's variants), BSD etc. ? given the nature of Erlang, it seems a bit unusual that it's binaries are available for a platform that is predominantly a desktop OS, but none for more server-oriented OSs. Of course, making it available on windows, may help a part (possibly large) of the developer community, but when it comes to deploying Erlang applications, user runs into lot of trouble. With the Ubuntu 9.10, after bit struggle I did manage to install Erlang R13, but then on CentOS, the best I can do with EPEL-5.3 is R12, while R14 is already out since Sept-15th. Searching through sugestions or how-to's, on installing Erlang on CentOS, it's amazing to find how much trouble it can be, even building it from source. So, is it only because there hasn't been a maintainer who is available for packages on these other OSs, which BTW is quite understandable, or there is something more to it ? Awareness about Erlang is growing, and it's popularity is rising, however the speed of adoption may grow much more, if getting Erlang on Linux. cheers, Icarus. From ebegumisa@REDACTED Sun Nov 7 19:15:05 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 08 Nov 2010 05:15:05 +1100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> Message-ID: Hello Rudolph, Small comment. I think one has to be careful not to suggest that blobs in typical RDBMSs and values in typical kv-stores are similar. I've heard that comparison being made before and forgive me, but I just don't see it (if that is indeed what what you were suggesting.) In SQL-RDBMSs, blob fields are normally opaque to the db. Storing blobs is the exception rather than the norm so querying against the content of these is usually poorly catered for (if catered for at all) by the query engine. The result is you normally can't use blobs to store anything complex because querying against them is a pain (i.e you'd only store things in a blob that you don't intend on using in the criteria of an SQL statement.) Where as for (I think most) kv-stores, the 'values' are not opaque. Storying complex data in the values is the norm, so most kv-stores are aware of the structural content of those values (e.g. JSON in CouchDB, Erlang-Terms in Mnesia.) The result is that kv-store query-engines have a mechanism for querying against the complex values being stored (views for Couch, qlc for Mnesia). I would think this renders the use-cases for blobs in RDBM's and values in kv-stores in-comparable. For couch's case, the equivalent to a blob would be more like an attachment. For Mnesia, a blob would be closer to a binary member of a tuple/record. > The issue is that the information in the "blob" is not useful. Neither > stored in the RDBMS, nor in a plain KV-store. That said, I don't really a agree with this statement. In a typical kv-store the 'value' is useful because the kv-store query engine would understand how to parse it. A typical SQL query-engine would not. I'm no Mnesia user, but I would think that you could easily query against parts of an Erlang-term parse tree stored in the database. To my knowledge, the only way to get the equivalent in an SQL-database would be to either... a) Spread the parse tree across different tables/records (hierarchical data modeling difficulty with tables non-withstanding) b) Use an atypical SQL database like Postgres/Oracle that supports complex/user-defined types c) Cheat -- write some sort of plugin that you can from a stored-procedure - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ On Sun, 07 Nov 2010 23:09:04 +1100, Rudolph van Graan wrote: > Hi Joe, > > I am not sure I agree with your statement that traditional databases > "suffer" from the fact that columns have simple types. You can easily > (within reason) make them store complex types. They are at the most > basic level also KV stores. The most basic abstraction for a "Table" is > a set of tuples stored in a certain way. There is no reason why you > cannot define a table with an arbitrary key and an arbitrary "blob" as > value column and then storing your parse tree in that blob field. > (Obviously you need to serialise/deserialise it - as you would need to > do for any storage abstraction). > > The issue is that the information in the "blob" is not useful. Neither > stored in the RDBMS, nor in a plain KV-store. You have to make it useful > before you can do something with it or extract information from it. I > guess that is why you said thus: > >> I'm not thinking in terms of joins and normalizing things - the thought >> process >> is different - so far I haven't met any problems that don't map onto >> key-values >> queries. > > > I agree with you. If you throw distribution, map-reduce and lots of > indices into the equation you can solve most issues this way. Except for > one thing - invariance between multiple records and record types. There > are very few examples where it is useful to live without invariance - > searching being one example. > > Invariance is a first order concept in any RDBMS system and essential > for many types of systems. However, in my experience, you can achieve > application-level invariance in other ways. But it requires > substantially more work to approximate first order invariance using > KV-stores. It is sometimes simpler to have your data properly > normalised I think. > > Rudolph van Graan > > > On Nov 4, 2010, at 10:01 PM, Joe Armstrong wrote: > >> On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >>> This is a message I sent to the nosql-discussion@REDACTED >>> discussion group. I thought it might be interesting to send it >>> erlang-questions, so here we go... >>> >>> >>> Hi all! >>> >>> I have used SQL RDBMSs for some time. I've never used any very >>> advanced >>> feature, but I know enough of it to make database applications. >>> >>> Nowadays, I decided it would be interesting to learn some NoSQL >>> databases concepts. So I decided to pick up some Erlang and Mnesia, >>> its >>> native key-value database. More than scalability itself, the most >>> valuable feature for me is the possibility of replication and >>> synchronization between nodes. >>> >>> But all pros have cons as companion. The lack of a relationship model >>> is difficult for who is used to RDBMSs. So, my question is: >>> >>> * Is there any guide, tutorial, book, whatever, that tries to introduce >>> NoSQL databases to SQL users? >>> >>> * Key-value databases are surprising simple. I know you solve >>> relationship by denormalizing data. What data should be normalized? >>> What shouldn't? How do you update denormalized data? >> >> I'm no database expert so don't quote me here ... >> >> As far as I am concerned traditional databases like SQL suffer >> from the fact that the data stored in an individual column is incredible >> simple - I can store an integer/string/... in a column but these are >> incredibly simple data structures. I want to store and retrieve >> incredibly >> complicated things - how do I store an XML parse tree in a single cell >> of a database? - How do I store a database in a database ... >> >> In a decent K-V database the value of the key can be *anything* - an >> entire database for example, a compiler, ... no worries >> >> Then when I analyse my problem I start thinking "I can store and >> retrieve any >> complex object that keys do I need to solve my problem?" >> >> I'm not thinking in terms of joins and normalizing things - the thought >> process >> is different - so far I haven't met any problems that don't map onto >> key-values >> queries. >> >> It seems to my that SQL provides you with the ability do to complex >> queries on simple things. K-V dbs can do simple queries on complex >> things. >> >> /Joe >> >> >> >>> >>> Sorry for such simple and general questions. Things were simple up to >>> the moment that I realized that it would be easily solved with a JOIN >>> SQL statement. :-) >>> >>> Thank you very much! >>> >>> -- >>> Silas Silva >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >>> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From vladdu55@REDACTED Sun Nov 7 21:13:09 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Sun, 7 Nov 2010 21:13:09 +0100 Subject: utf8 in source files Message-ID: Hi! I've got lately more and more requests (some even with foul language) about why erlide (the erlang ide for eclipse) doesn't support utf8 encoding of erlang source files. My answer has always been to point out to the erlang docs where it is clearly stated that source files are still to be Latin1 encoded, regardless of the recent unicode-supporting libraries. The documentation (http://www.erlang.org/doc/apps/stdlib/unicode_usage.html) that says: "Also the source code is (for now) still expected to be written using the ISO-latin-1 character set, why Unicode characters beyond that range cannot be entered in string literals." The trouble is that the scanner actually accepts utf8 encoding in literal strings and comments, possibly even in quoted atoms. See the attached file for an example, it compiles fine even on my Swedish system (the output from running it is garbled for me, I assume it would look fine on a properly setup terminal). So users from countries where Latin1 is useless but utf8 is not, may use the latter instead without getting any problems. Eclipse requires the encoding of a file to be specified and of course, I can't set that to something else than the official Latin1. But people that used utf8 before and it worked, now get frustrated because it's no longer accepted when they use erlide. It's easy to blame erlide for that and ask for a fix because it looks like it works outside of eclipse... The simple solution would be to update the docs above to say something like "Please note that UTF-8 encoded files can be accepted by the compiler and may work as expected in some environments, but this usage is not recommended and not supported." Also, comments should be mentioned alongside literal strings. Would this be acceptable? The complex solution would be to make sources encoding-aware (maybe as suggested here http://www.erlang.org/cgi-bin/ezmlm-cgi?4:msp:46892), but I know this is by no means a simple task. Does the OTP team have anything scheduled in this area? R15, R16? best regards, Vlad -------------- next part -------------- A non-text attachment was scrubbed... Name: main.erl Type: application/octet-stream Size: 306 bytes Desc: not available URL: From rzezeski@REDACTED Sun Nov 7 21:37:50 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Sun, 7 Nov 2010 15:37:50 -0500 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: On Sun, Nov 7, 2010 at 9:49 AM, Petter Egesund wrote: > Hi, I have a small program with lots of memory-updates which I try to > run in Erlang. > > The same algorithm works fine in both Java and Perl, but fails in > Erlang because the program runs out of memory - and I can not figure > out why. Frustrating, as my Erlang-versjon seems to be the easiest to > scale as well as being the most readable. > > The program is threaded and each thread writes to a ets-table which is > created at the beginning of the thread. When the thread dies I try to > do a ets:delete(Table), like described in the manual, but the memory > used by the thread never seems to be released. > > Some facts: > > - The memory usage of each thread is rather constant. This is > confirmed when I use ets:i() to show info about memory usage. > - The number of threads are constant - confirmed by both running top > and writing out the number of threads regularly. When a thread dies, I > create a new one. > - I have tried to end the thread by sending a exit-signal as the last > statement. This helps some, but does not solve the leak. > - I put small lists of size 3-4 integers into the ets as values, the > keys are list of same size as well. > - I garbage-collect each thread before it dies, as well as doing > regular global garbage-collects. No help. > - Information from ets:i() about memory when I sum usage by each > thread, is much lower than stated by memory() when i run > erlang:memory(). This might indicate something? Does not seem logical > to me, at least. > - Info from erlang:memory is about half of what top/the os tells. > - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well. > > Any clues? Dump from ets:i() and erlang:memory() is like below. > > Cheers, > > Petter > > --- dump --- > > eNumber of processes: 27 > ets:i(): > id name type size mem owner > > ---------------------------------------------------------------------------- > 13 code set 261 10692 code_server > 4110 code_names set 58 7804 code_server > 6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0> > 7022018584 the_synapses ordered_set 15143493 310909950 <0.48.0> > 7774416922 the_synapses ordered_set 8794649 182005810 <0.49.0> > ac_tab ac_tab set 6 848 > application_controller > file_io_servers file_io_servers set 0 302 file_server_2 > global_locks global_locks set 0 302 global_name_server > global_names global_names set 0 302 global_name_server > global_names_ext global_names_ext set 0 302 > global_name_server > global_pid_ids global_pid_ids bag 0 302 global_name_server > global_pid_names global_pid_names bag 0 302 > global_name_server > inet_cache inet_cache bag 0 302 inet_db > inet_db inet_db set 29 571 inet_db > inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db > inet_hosts_byname inet_hosts_byname bag 0 302 inet_db > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302 > inet_db > inet_hosts_file_byname inet_hosts_file_byname bag 0 302 > inet_db > neurone_counter neurone_counter set 258394 1846182 entity_server > neurone_group_counter neurone_group_counter set 6 344 > entity_group_server > neurone_group_name neurone_group_name set 6 426 > entity_group_server > neurone_group_name_reverse neurone_group_name_reverse set 6 > 426 entity_group_server > neurone_name neurone_name set 258394 11824602 entity_server > neurone_name_reverse neurone_name_reverse set 258394 11824602 > entity_server > memory(): [{total,5568669792}, > {processes,1138936}, > {processes_used,1128120}, > {system,5567530856}, > {atom,349769}, > {atom_used,336605}, > {binary,82704}, > {code,3046365}, > {ets,5562163256}] > > > Hi Peter, ETS tables are not garbage collected. Each ETS table has _one_ owner (a process). When that owner dies the table is deleted and it's memory is reclaimed. You can also delete a table (and reclaim the memory) by calling ets:delete/1. Looking at your memory result, your ETS tables are taking up ~5.2GB of data. However, you binary usage is very low so I'm going to take a guess that you are sotring a list of strings? If so you should note that on a 64-bit system *each character* in a string will use 16 bytes of memory! I highly recommend using binaries where possible when dealing with a large amount of data; your program will not only be more space efficient but also faster. I've written a non-trivial Erlang application for work and I deal with CSV files that get up to 18 million rows. I make heavy use of binaries and the binary module to parse these files and write entries to ETS--you'd be surprised how fast it is! If you'd like I could provide an example. When you say "thread" do you mean "process?" You do realize that an OS thread and Erlang process are two completely different things. IIRC, the VM spawn's an OS thread per scheduler (along w/ some other threads for I/O and such). Erlang processes are extremely cheap...don't be afraid to make thousands or even tens-of-thousands of them. You should not have to perform manual garbage collection, that seems like a code smell to me. When a process dies it's heap will be reclaimed. Each process has it's own isolated heap. Do you have multiple processes all writing to the same ETS table? If so there are some improvements that were made to ETS (and Erlang in general) for concurrent writing/reading of an ETS table in 14B that you might want to look at. Finally, it would be helpful to see the full source code. There is a good chance your solution is not optimal for Erlang. By that, I mean that if your translation follows closely from your Java and Perl solutions than chances are it's not an optimal Erlang program as the paradigms are vastly different. -Ryan From mk@REDACTED Sun Nov 7 22:09:44 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 07 Nov 2010 22:09:44 +0100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> Message-ID: <4CD71598.9000109@amberbio.com> Hi, Edmond, you talk about datastores that can peek inside the values instead of regarding them as opaque binaries. But are those datastores really kv-stores? I would claim that, by definition, kv-stores can only do get, put, delete and maybe some batch insertions and retrievals, but not more. Couchdb and mnesia should probably not be named kv-stores, but rather something more descriptive of their capabilities. Also, no-sql is not the same as key-value. Part of the problem is that no-sql is too loose and broad a category. Morten. inside th eI agree with your points about data stores that can look inside the values On 11/7/10 7:15 PM, Edmond Begumisa wrote: > Hello Rudolph, > > Small comment. I think one has to be careful not to suggest that blobs > in typical RDBMSs and values in typical kv-stores are similar. I've > heard that comparison being made before and forgive me, but I just > don't see it (if that is indeed what what you were suggesting.) > > In SQL-RDBMSs, blob fields are normally opaque to the db. Storing > blobs is the exception rather than the norm so querying against the > content of these is usually poorly catered for (if catered for at all) > by the query engine. The result is you normally can't use blobs to > store anything complex because querying against them is a pain (i.e > you'd only store things in a blob that you don't intend on using in > the criteria of an SQL statement.) > > Where as for (I think most) kv-stores, the 'values' are not opaque. > Storying complex data in the values is the norm, so most kv-stores are > aware of the structural content of those values (e.g. JSON in CouchDB, > Erlang-Terms in Mnesia.) The result is that kv-store query-engines > have a mechanism for querying against the complex values being stored > (views for Couch, qlc for Mnesia). > > I would think this renders the use-cases for blobs in RDBM's and > values in kv-stores in-comparable. For couch's case, the equivalent to > a blob would be more like an attachment. For Mnesia, a blob would be > closer to a binary member of a tuple/record. > >> The issue is that the information in the "blob" is not useful. >> Neither stored in the RDBMS, nor in a plain KV-store. > > That said, I don't really a agree with this statement. In a typical > kv-store the 'value' is useful because the kv-store query engine would > understand how to parse it. A typical SQL query-engine would not. > > I'm no Mnesia user, but I would think that you could easily query > against parts of an Erlang-term parse tree stored in the database. To > my knowledge, the only way to get the equivalent in an SQL-database > would be to either... > > a) Spread the parse tree across different tables/records (hierarchical > data modeling difficulty with tables non-withstanding) > b) Use an atypical SQL database like Postgres/Oracle that supports > complex/user-defined types > c) Cheat -- write some sort of plugin that you can from a > stored-procedure > > - Edmond - > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > > On Sun, 07 Nov 2010 23:09:04 +1100, Rudolph van Graan > wrote: > >> Hi Joe, >> >> I am not sure I agree with your statement that traditional databases >> "suffer" from the fact that columns have simple types. You can easily >> (within reason) make them store complex types. They are at the most >> basic level also KV stores. The most basic abstraction for a "Table" >> is a set of tuples stored in a certain way. There is no reason why >> you cannot define a table with an arbitrary key and an arbitrary >> "blob" as value column and then storing your parse tree in that blob >> field. (Obviously you need to serialise/deserialise it - as you would >> need to do for any storage abstraction). >> >> The issue is that the information in the "blob" is not useful. >> Neither stored in the RDBMS, nor in a plain KV-store. You have to >> make it useful before you can do something with it or extract >> information from it. I guess that is why you said thus: >> >>> I'm not thinking in terms of joins and normalizing things - the >>> thought process >>> is different - so far I haven't met any problems that don't map onto >>> key-values >>> queries. >> >> >> I agree with you. If you throw distribution, map-reduce and lots of >> indices into the equation you can solve most issues this way. Except >> for one thing - invariance between multiple records and record types. >> There are very few examples where it is useful to live without >> invariance - searching being one example. >> >> Invariance is a first order concept in any RDBMS system and essential >> for many types of systems. However, in my experience, you can achieve >> application-level invariance in other ways. But it requires >> substantially more work to approximate first order invariance using >> KV-stores. It is sometimes simpler to have your data properly >> normalised I think. >> >> Rudolph van Graan >> >> >> On Nov 4, 2010, at 10:01 PM, Joe Armstrong wrote: >> >>> On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >>>> This is a message I sent to the nosql-discussion@REDACTED >>>> discussion group. I thought it might be interesting to send it >>>> erlang-questions, so here we go... >>>> >>>> >>>> Hi all! >>>> >>>> I have used SQL RDBMSs for some time. I've never used any very >>>> advanced >>>> feature, but I know enough of it to make database applications. >>>> >>>> Nowadays, I decided it would be interesting to learn some NoSQL >>>> databases concepts. So I decided to pick up some Erlang and >>>> Mnesia, its >>>> native key-value database. More than scalability itself, the most >>>> valuable feature for me is the possibility of replication and >>>> synchronization between nodes. >>>> >>>> But all pros have cons as companion. The lack of a relationship model >>>> is difficult for who is used to RDBMSs. So, my question is: >>>> >>>> * Is there any guide, tutorial, book, whatever, that tries to >>>> introduce >>>> NoSQL databases to SQL users? >>>> >>>> * Key-value databases are surprising simple. I know you solve >>>> relationship by denormalizing data. What data should be normalized? >>>> What shouldn't? How do you update denormalized data? >>> >>> I'm no database expert so don't quote me here ... >>> >>> As far as I am concerned traditional databases like SQL suffer >>> from the fact that the data stored in an individual column is >>> incredible >>> simple - I can store an integer/string/... in a column but these are >>> incredibly simple data structures. I want to store and retrieve >>> incredibly >>> complicated things - how do I store an XML parse tree in a single cell >>> of a database? - How do I store a database in a database ... >>> >>> In a decent K-V database the value of the key can be *anything* - an >>> entire database for example, a compiler, ... no worries >>> >>> Then when I analyse my problem I start thinking "I can store and >>> retrieve any >>> complex object that keys do I need to solve my problem?" >>> >>> I'm not thinking in terms of joins and normalizing things - the >>> thought process >>> is different - so far I haven't met any problems that don't map onto >>> key-values >>> queries. >>> >>> It seems to my that SQL provides you with the ability do to complex >>> queries on simple things. K-V dbs can do simple queries on complex >>> things. >>> >>> /Joe >>> >>> >>> >>>> >>>> Sorry for such simple and general questions. Things were simple up to >>>> the moment that I realized that it would be easily solved with a JOIN >>>> SQL statement. :-) >>>> >>>> Thank you very much! >>>> >>>> -- >>>> Silas Silva >>>> >>>> ________________________________________________________________ >>>> erlang-questions (at) erlang.org mailing list. >>>> See http://www.erlang.org/faq.html >>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>> >>>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> > > From andrew@REDACTED Sun Nov 7 22:11:26 2010 From: andrew@REDACTED (Andrew Thompson) Date: Sun, 7 Nov 2010 16:11:26 -0500 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: Message-ID: <20101107211126.GA2468@hijacked.us> Erlang is already in the package management systems for most Linux distributions, in OSX macports and in the ports/pkgsrc trees for the *BSDs. Granted sometimes this is an outdated version, but its almost always available in some form. The reason that binaries are only available for windows is probably twofold: 1) Compiling erlang for windows is a pain in the ass. 2) There's no standard package manager for windows (unlike pretty much every other major OS). Even if the package manager for your non windows OS doesn't ship a recent erlang release, you can usually build it from source by just installing a few dependancies. Andrew From petter.egesund@REDACTED Sun Nov 7 22:15:12 2010 From: petter.egesund@REDACTED (Petter Egesund) Date: Sun, 7 Nov 2010 22:15:12 +0100 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: Hi, and thanks for answering! Yes, the memory should be reclaimed - I think the problem might be that the garbage collector is failing?! Summing up ets-memory-usage for each thread gives reasonable numbers, but erlang:memory() shows ets is using much more ram - calling ets:delete before the process finishes does not help. No, I am only keeping integers in the ets, no strings :-) The program is using binaries a lot, but these seems to be reclaimed from the gb without problems. Yes, I know the difference between processes/threads, only a couple small ets-tables are global and shared between processes. The large ones, which I think are the cause of the problem, is only written to/read by one process. Full source code? It is unfortunately to long and to data-bound to make sense. Still hoping for a clue - ets:i() tells me that total ets should be less than to 1 gb total, but memory() says it is using more than 5 gb for ets. Cheers, Petter On Sun, Nov 7, 2010 at 9:37 PM, Ryan Zezeski wrote: > > > On Sun, Nov 7, 2010 at 9:49 AM, Petter Egesund > wrote: >> >> Hi, I have a small program with lots of memory-updates which I try to >> run in Erlang. >> >> The same algorithm works fine in both Java and Perl, but fails in >> Erlang because the program runs out of memory - and I can not figure >> out why. Frustrating, as my Erlang-versjon seems to be the easiest to >> scale as well as being the most readable. >> >> The program is threaded and each thread writes to a ets-table which is >> created at the beginning of the thread. When the thread dies I try to >> do a ets:delete(Table), like described in the manual, but the memory >> used by the thread never seems to be released. >> >> Some facts: >> >> - The memory usage of each thread is rather constant. This is >> confirmed when I use ets:i() to show info about memory usage. >> - The number of threads are constant - confirmed by both running top >> and writing out the number of threads regularly. When a thread dies, I >> create a new one. >> - I have tried to end the thread by sending a exit-signal as the last >> statement. This helps some, but does not solve the leak. >> - I put small lists of size 3-4 integers into the ets as values, the >> keys are list of same size as well. >> - I garbage-collect each thread before it dies, as well as doing >> regular global garbage-collects. No help. >> - Information from ets:i() about memory when I sum usage by each >> thread, is much lower than stated by memory() when i run >> erlang:memory(). This might indicate something? Does not seem logical >> to me, at least. >> - Info from erlang:memory is about half of what top/the os tells. >> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well. >> >> Any clues? Dump from ets:i() and erlang:memory() is like below. >> >> Cheers, >> >> Petter >> >> --- dump --- >> >> eNumber of processes: 27 >> ets:i(): >> ?id ? ? ? ? ? ? ?name ? ? ? ? ? ? ?type ?size ? mem ? ? ?owner >> >> ?---------------------------------------------------------------------------- >> ?13 ? ? ? ? ? ? ?code ? ? ? ? ? ? ?set ? 261 ? ?10692 ? ?code_server >> ?4110 ? ? ? ? ? ?code_names ? ? ? ?set ? 58 ? ? 7804 ? ? code_server >> ?6746271765 ? ? ?the_synapses ? ? ?ordered_set 5425194 113336012 <0.47.0> >> ?7022018584 ? ? ?the_synapses ? ? ?ordered_set 15143493 310909950 <0.48.0> >> ?7774416922 ? ? ?the_synapses ? ? ?ordered_set 8794649 182005810 <0.49.0> >> ?ac_tab ? ? ? ? ?ac_tab ? ? ? ? ? ?set ? 6 ? ? ?848 >> ?application_controller >> ?file_io_servers file_io_servers ? set ? 0 ? ? ?302 ? ? ?file_server_2 >> ?global_locks ? ?global_locks ? ? ?set ? 0 ? ? ?302 >> ?global_name_server >> ?global_names ? ?global_names ? ? ?set ? 0 ? ? ?302 >> ?global_name_server >> ?global_names_ext global_names_ext ?set ? 0 ? ? ?302 >> ?global_name_server >> ?global_pid_ids ?global_pid_ids ? ?bag ? 0 ? ? ?302 >> ?global_name_server >> ?global_pid_names global_pid_names ?bag ? 0 ? ? ?302 >> ?global_name_server >> ?inet_cache ? ? ?inet_cache ? ? ? ?bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_db ? ? ? ? inet_db ? ? ? ? ? set ? 29 ? ? 571 ? ? ?inet_db >> ?inet_hosts_byaddr inet_hosts_byaddr bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_hosts_byname inet_hosts_byname bag ? 0 ? ? ?302 ? ? ?inet_db >> ?inet_hosts_file_byaddr inet_hosts_file_byaddr bag ? 0 ? ? ?302 >> ?inet_db >> ?inet_hosts_file_byname inet_hosts_file_byname bag ? 0 ? ? ?302 >> ?inet_db >> ?neurone_counter neurone_counter ? set ? 258394 1846182 ?entity_server >> ?neurone_group_counter neurone_group_counter set ? 6 ? ? ?344 >> entity_group_server >> ?neurone_group_name neurone_group_name set ? 6 ? ? ?426 >> ?entity_group_server >> ?neurone_group_name_reverse neurone_group_name_reverse set ? 6 >> 426 ? ? ?entity_group_server >> ?neurone_name ? ?neurone_name ? ? ?set ? 258394 11824602 entity_server >> ?neurone_name_reverse neurone_name_reverse set ? 258394 11824602 >> entity_server >> memory(): ? ? ? ? [{total,5568669792}, >> ? ? ? ? ? ? ? ? ? {processes,1138936}, >> ? ? ? ? ? ? ? ? ? {processes_used,1128120}, >> ? ? ? ? ? ? ? ? ? {system,5567530856}, >> ? ? ? ? ? ? ? ? ? {atom,349769}, >> ? ? ? ? ? ? ? ? ? {atom_used,336605}, >> ? ? ? ? ? ? ? ? ? {binary,82704}, >> ? ? ? ? ? ? ? ? ? {code,3046365}, >> ? ? ? ? ? ? ? ? ? {ets,5562163256}] >> >> > > Hi Peter, ETS tables are not garbage collected. ?Each ETS table has _one_ > owner (a process). ?When that owner dies the table is deleted and it's > memory is reclaimed. ?You can also delete a table (and reclaim the memory) > by calling ets:delete/1. ?Looking at your memory result, your ETS tables are > taking up ~5.2GB of data. ?However, you binary usage is very low so I'm > going to take a guess that you are sotring a list of strings? ?If so you > should note that on a 64-bit system *each character* in a string will use 16 > bytes of memory! ?I?highly?recommend using binaries where possible when > dealing with a large amount of data; your program will not only be more > space efficient but also faster. ?I've written a non-trivial Erlang > application for work and I deal with CSV files that get up to 18 million > rows. ?I make heavy use of binaries and the binary module to parse these > files and write entries to ETS--you'd be surprised how fast it is! ?If you'd > like I could provide an example. > When you say "thread" do you mean "process?" ?You do realize that an OS > thread and Erlang process are two completely different things. ?IIRC, the VM > spawn's an OS thread per scheduler (along w/ some other threads for I/O and > such). ?Erlang processes are extremely cheap...don't be afraid to make > thousands or even tens-of-thousands of them. > You should not have to perform manual garbage collection, that seems like a > code smell to me. ?When a process dies it's heap will be reclaimed. ?Each > process has it's own isolated heap. > Do you have multiple processes all writing to the same ETS table? ?If so > there are some improvements that were made to ETS (and Erlang in general) > for concurrent writing/reading of an ETS table in 14B that you might want to > look at. > Finally, it would be helpful to see the full source code. ?There is a good > chance your solution is not optimal for Erlang. ?By that, I mean that if > your translation follows closely from your Java and Perl solutions than > chances are it's not an optimal Erlang program as the paradigms are vastly > different. > -Ryan From jesper.louis.andersen@REDACTED Sun Nov 7 22:21:58 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sun, 7 Nov 2010 22:21:58 +0100 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: On Sun, Nov 7, 2010 at 10:15 PM, Petter Egesund wrote: > Full source code? It is unfortunately to long and to data-bound to > make sense. Still hoping for a clue - ets:i() tells me that total ets > should be less than to 1 gb total, but memory() says it is using more > than 5 gb for ets. ets:i/0 reports words. erlang:memory/0 reports bytes. So that should not be surprising that they differ by a factor of 4. -- J. From mk@REDACTED Sun Nov 7 22:24:15 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 07 Nov 2010 22:24:15 +0100 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: <4CD718FF.1080001@amberbio.com> Hi Are you sure that your ets memory information is in bytes, and not in words? Morten. On 11/7/10 10:15 PM, Petter Egesund wrote: > Hi, and thanks for answering! > > Yes, the memory should be reclaimed - I think the problem might be > that the garbage collector is failing?! > > Summing up ets-memory-usage for each thread gives reasonable numbers, > but erlang:memory() shows ets is using much more ram - calling > ets:delete before the process finishes does not help. > > No, I am only keeping integers in the ets, no strings :-) The program > is using binaries a lot, but these seems to be reclaimed from the gb > without problems. > > Yes, I know the difference between processes/threads, only a couple > small ets-tables are global and shared between processes. The large > ones, which I think are the cause of the problem, is only written > to/read by one process. > > Full source code? It is unfortunately to long and to data-bound to > make sense. Still hoping for a clue - ets:i() tells me that total ets > should be less than to 1 gb total, but memory() says it is using more > than 5 gb for ets. > > Cheers, > > Petter > > > > > > On Sun, Nov 7, 2010 at 9:37 PM, Ryan Zezeski wrote: >> >> On Sun, Nov 7, 2010 at 9:49 AM, Petter Egesund >> wrote: >>> Hi, I have a small program with lots of memory-updates which I try to >>> run in Erlang. >>> >>> The same algorithm works fine in both Java and Perl, but fails in >>> Erlang because the program runs out of memory - and I can not figure >>> out why. Frustrating, as my Erlang-versjon seems to be the easiest to >>> scale as well as being the most readable. >>> >>> The program is threaded and each thread writes to a ets-table which is >>> created at the beginning of the thread. When the thread dies I try to >>> do a ets:delete(Table), like described in the manual, but the memory >>> used by the thread never seems to be released. >>> >>> Some facts: >>> >>> - The memory usage of each thread is rather constant. This is >>> confirmed when I use ets:i() to show info about memory usage. >>> - The number of threads are constant - confirmed by both running top >>> and writing out the number of threads regularly. When a thread dies, I >>> create a new one. >>> - I have tried to end the thread by sending a exit-signal as the last >>> statement. This helps some, but does not solve the leak. >>> - I put small lists of size 3-4 integers into the ets as values, the >>> keys are list of same size as well. >>> - I garbage-collect each thread before it dies, as well as doing >>> regular global garbage-collects. No help. >>> - Information from ets:i() about memory when I sum usage by each >>> thread, is much lower than stated by memory() when i run >>> erlang:memory(). This might indicate something? Does not seem logical >>> to me, at least. >>> - Info from erlang:memory is about half of what top/the os tells. >>> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well. >>> >>> Any clues? Dump from ets:i() and erlang:memory() is like below. >>> >>> Cheers, >>> >>> Petter >>> >>> --- dump --- >>> >>> eNumber of processes: 27 >>> ets:i(): >>> id name type size mem owner >>> >>> ---------------------------------------------------------------------------- >>> 13 code set 261 10692 code_server >>> 4110 code_names set 58 7804 code_server >>> 6746271765 the_synapses ordered_set 5425194 113336012<0.47.0> >>> 7022018584 the_synapses ordered_set 15143493 310909950<0.48.0> >>> 7774416922 the_synapses ordered_set 8794649 182005810<0.49.0> >>> ac_tab ac_tab set 6 848 >>> application_controller >>> file_io_servers file_io_servers set 0 302 file_server_2 >>> global_locks global_locks set 0 302 >>> global_name_server >>> global_names global_names set 0 302 >>> global_name_server >>> global_names_ext global_names_ext set 0 302 >>> global_name_server >>> global_pid_ids global_pid_ids bag 0 302 >>> global_name_server >>> global_pid_names global_pid_names bag 0 302 >>> global_name_server >>> inet_cache inet_cache bag 0 302 inet_db >>> inet_db inet_db set 29 571 inet_db >>> inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db >>> inet_hosts_byname inet_hosts_byname bag 0 302 inet_db >>> inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302 >>> inet_db >>> inet_hosts_file_byname inet_hosts_file_byname bag 0 302 >>> inet_db >>> neurone_counter neurone_counter set 258394 1846182 entity_server >>> neurone_group_counter neurone_group_counter set 6 344 >>> entity_group_server >>> neurone_group_name neurone_group_name set 6 426 >>> entity_group_server >>> neurone_group_name_reverse neurone_group_name_reverse set 6 >>> 426 entity_group_server >>> neurone_name neurone_name set 258394 11824602 entity_server >>> neurone_name_reverse neurone_name_reverse set 258394 11824602 >>> entity_server >>> memory(): [{total,5568669792}, >>> {processes,1138936}, >>> {processes_used,1128120}, >>> {system,5567530856}, >>> {atom,349769}, >>> {atom_used,336605}, >>> {binary,82704}, >>> {code,3046365}, >>> {ets,5562163256}] >>> >>> >> Hi Peter, ETS tables are not garbage collected. Each ETS table has _one_ >> owner (a process). When that owner dies the table is deleted and it's >> memory is reclaimed. You can also delete a table (and reclaim the memory) >> by calling ets:delete/1. Looking at your memory result, your ETS tables are >> taking up ~5.2GB of data. However, you binary usage is very low so I'm >> going to take a guess that you are sotring a list of strings? If so you >> should note that on a 64-bit system *each character* in a string will use 16 >> bytes of memory! I highly recommend using binaries where possible when >> dealing with a large amount of data; your program will not only be more >> space efficient but also faster. I've written a non-trivial Erlang >> application for work and I deal with CSV files that get up to 18 million >> rows. I make heavy use of binaries and the binary module to parse these >> files and write entries to ETS--you'd be surprised how fast it is! If you'd >> like I could provide an example. >> When you say "thread" do you mean "process?" You do realize that an OS >> thread and Erlang process are two completely different things. IIRC, the VM >> spawn's an OS thread per scheduler (along w/ some other threads for I/O and >> such). Erlang processes are extremely cheap...don't be afraid to make >> thousands or even tens-of-thousands of them. >> You should not have to perform manual garbage collection, that seems like a >> code smell to me. When a process dies it's heap will be reclaimed. Each >> process has it's own isolated heap. >> Do you have multiple processes all writing to the same ETS table? If so >> there are some improvements that were made to ETS (and Erlang in general) >> for concurrent writing/reading of an ETS table in 14B that you might want to >> look at. >> Finally, it would be helpful to see the full source code. There is a good >> chance your solution is not optimal for Erlang. By that, I mean that if >> your translation follows closely from your Java and Perl solutions than >> chances are it's not an optimal Erlang program as the paradigms are vastly >> different. >> -Ryan > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From rzezeski@REDACTED Sun Nov 7 22:28:01 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Sun, 7 Nov 2010 16:28:01 -0500 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: On Sun, Nov 7, 2010 at 4:21 PM, Jesper Louis Andersen < jesper.louis.andersen@REDACTED> wrote: > On Sun, Nov 7, 2010 at 10:15 PM, Petter Egesund > wrote: > > > Full source code? It is unfortunately to long and to data-bound to > > make sense. Still hoping for a clue - ets:i() tells me that total ets > > should be less than to 1 gb total, but memory() says it is using more > > than 5 gb for ets. > > ets:i/0 reports words. > erlang:memory/0 reports bytes. > > So that should not be surprising that they differ by a factor of 4. > > -- > J. > Jesper beat me to it...the memory reported by ets:info and ets:i is in words...not bytes. Petter, I don't think you're actually deleting these tables, which is proven by the fact that they show up in ets:i/0. I'll say it again, they are not garbage collected...they don't live on the process heap. THere are two ways to delete an ETS table: 1) call ets:delete or 2) the owner dies and there is no heir. When they are deleted the memory is reclaimed. Here is a erl session showing what I mean... 1> ets:i(). id name type size mem owner ---------------------------------------------------------------------------- 13 code set 257 10711 code_server 4110 code_names set 57 7754 code_server 8207 shell_records ordered_set 0 112 <0.25.0> ac_tab ac_tab set 6 878 application_controller file_io_servers file_io_servers set 0 322 file_server_2 global_locks global_locks set 0 322 global_name_server global_names global_names set 0 322 global_name_server global_names_ext global_names_ext set 0 322 global_name_server global_pid_ids global_pid_ids bag 0 322 global_name_server global_pid_names global_pid_names bag 0 322 global_name_server inet_cache inet_cache bag 0 322 inet_db inet_db inet_db set 29 706 inet_db inet_hosts_byaddr inet_hosts_byaddr bag 0 322 inet_db inet_hosts_byname inet_hosts_byname bag 0 322 inet_db inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 322 inet_db inet_hosts_file_byname inet_hosts_file_byname bag 0 322 inet_db ok 2> ets:new(test, [public, named_table]). test 3> ets:i(). id name type size mem owner ---------------------------------------------------------------------------- 13 code set 257 10711 code_server 4110 code_names set 57 7754 code_server 8207 shell_records ordered_set 0 112 <0.25.0> ac_tab ac_tab set 6 878 application_controller file_io_servers file_io_servers set 0 322 file_server_2 global_locks global_locks set 0 322 global_name_server global_names global_names set 0 322 global_name_server global_names_ext global_names_ext set 0 322 global_name_server global_pid_ids global_pid_ids bag 0 322 global_name_server global_pid_names global_pid_names bag 0 322 global_name_server inet_cache inet_cache bag 0 322 inet_db inet_db inet_db set 29 706 inet_db inet_hosts_byaddr inet_hosts_byaddr bag 0 322 inet_db inet_hosts_byname inet_hosts_byname bag 0 322 inet_db inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 322 inet_db inet_hosts_file_byname inet_hosts_file_byname bag 0 322 inet_db test test set 0 322 <0.31.0> ok 4> ets:delete(test). true 5> ets:i(). id name type size mem owner ---------------------------------------------------------------------------- 13 code set 257 10711 code_server 4110 code_names set 57 7754 code_server 8207 shell_records ordered_set 0 112 <0.25.0> ac_tab ac_tab set 6 878 application_controller file_io_servers file_io_servers set 0 322 file_server_2 global_locks global_locks set 0 322 global_name_server global_names global_names set 0 322 global_name_server global_names_ext global_names_ext set 0 322 global_name_server global_pid_ids global_pid_ids bag 0 322 global_name_server global_pid_names global_pid_names bag 0 322 global_name_server inet_cache inet_cache bag 0 322 inet_db inet_db inet_db set 29 706 inet_db inet_hosts_byaddr inet_hosts_byaddr bag 0 322 inet_db inet_hosts_byname inet_hosts_byname bag 0 322 inet_db inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 322 inet_db inet_hosts_file_byname inet_hosts_file_byname bag 0 322 inet_db ok From petter.egesund@REDACTED Sun Nov 7 22:50:39 2010 From: petter.egesund@REDACTED (Petter Egesund) Date: Sun, 7 Nov 2010 22:50:39 +0100 Subject: [erlang-questions] Unstable erlang compared to java or perl In-Reply-To: References: Message-ID: Yes, the words vs bytes theory might explain something :-) I know about delete - I allready do that, the ets-tables which appear in the list are expected. They belong to not-finished processes. Some math then: Summing the ets:i() info gives about 631747158 words which corresponds to 5562163256 bytes, meaning that each word is using 8 bytes, which makes sense on a 64-bit operating system to me. Using this line as an example shows that each entry in ets (entry is key/value, both tuples with three integers), show that each integer needs 113336012 * 8 / 5425194 = 167 bytes (or 27 bytes pr. integer): 6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0> This is what tricked me - erlang is using much space serializing data than the other languages - I will try to store my data as binary and see if that helps. Thanks! Petter On Sun, Nov 7, 2010 at 10:28 PM, Ryan Zezeski wrote: > > > On Sun, Nov 7, 2010 at 4:21 PM, Jesper Louis Andersen > wrote: >> >> On Sun, Nov 7, 2010 at 10:15 PM, Petter Egesund >> wrote: >> >> > Full source code? It is unfortunately to long and to data-bound to >> > make sense. Still hoping for a clue - ets:i() tells me that total ets >> > should be less than to 1 gb total, but memory() says it is using more >> > than 5 gb for ets. >> >> ets:i/0 reports words. >> erlang:memory/0 reports bytes. >> >> So that should not be surprising that they differ by a factor of 4. >> >> -- >> J. > > Jesper beat me to it...the memory reported by ets:info and ets:i is in > words...not bytes. > Petter, I don't think you're actually deleting these tables, which is proven > by the fact that they show up in ets:i/0. ?I'll say it again, they are not > garbage collected...they don't live on the process heap. ?THere are two ways > to delete an ETS table: 1) call ets:delete or 2) the owner dies and there is > no heir. ?When they are deleted ?the memory is reclaimed. > Here is a erl session showing what I mean... > 1> ets:i(). > ?id ? ? ? ? ? ? ?name ? ? ? ? ? ? ?type ?size ? mem ? ? ?owner > ?---------------------------------------------------------------------------- > ?13 ? ? ? ? ? ? ?code ? ? ? ? ? ? ?set ? 257 ? ?10711 ? ?code_server > ?4110 ? ? ? ? ? ?code_names ? ? ? ?set ? 57 ? ? 7754 ? ? code_server > ?8207 ? ? ? ? ? ?shell_records ? ? ordered_set 0 ? ? ?112 ? ? ?<0.25.0> > ?ac_tab ? ? ? ? ?ac_tab ? ? ? ? ? ?set ? 6 ? ? ?878 > ?application_controller > ?file_io_servers file_io_servers ? set ? 0 ? ? ?322 ? ? ?file_server_2 > ?global_locks ? ?global_locks ? ? ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_names ? ?global_names ? ? ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_names_ext global_names_ext ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_pid_ids ?global_pid_ids ? ?bag ? 0 ? ? ?322 ? ? ?global_name_server > ?global_pid_names global_pid_names ?bag ? 0 ? ? ?322 ? ? ?global_name_server > ?inet_cache ? ? ?inet_cache ? ? ? ?bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_db ? ? ? ? inet_db ? ? ? ? ? set ? 29 ? ? 706 ? ? ?inet_db > ?inet_hosts_byaddr inet_hosts_byaddr bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_byname inet_hosts_byname bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_file_byaddr inet_hosts_file_byaddr bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_file_byname inet_hosts_file_byname bag ? 0 ? ? ?322 ? ? ?inet_db > ok > 2> ets:new(test, [public, named_table]). > test > 3> ets:i(). > ?id ? ? ? ? ? ? ?name ? ? ? ? ? ? ?type ?size ? mem ? ? ?owner > ?---------------------------------------------------------------------------- > ?13 ? ? ? ? ? ? ?code ? ? ? ? ? ? ?set ? 257 ? ?10711 ? ?code_server > ?4110 ? ? ? ? ? ?code_names ? ? ? ?set ? 57 ? ? 7754 ? ? code_server > ?8207 ? ? ? ? ? ?shell_records ? ? ordered_set 0 ? ? ?112 ? ? ?<0.25.0> > ?ac_tab ? ? ? ? ?ac_tab ? ? ? ? ? ?set ? 6 ? ? ?878 > ?application_controller > ?file_io_servers file_io_servers ? set ? 0 ? ? ?322 ? ? ?file_server_2 > ?global_locks ? ?global_locks ? ? ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_names ? ?global_names ? ? ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_names_ext global_names_ext ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_pid_ids ?global_pid_ids ? ?bag ? 0 ? ? ?322 ? ? ?global_name_server > ?global_pid_names global_pid_names ?bag ? 0 ? ? ?322 ? ? ?global_name_server > ?inet_cache ? ? ?inet_cache ? ? ? ?bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_db ? ? ? ? inet_db ? ? ? ? ? set ? 29 ? ? 706 ? ? ?inet_db > ?inet_hosts_byaddr inet_hosts_byaddr bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_byname inet_hosts_byname bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_file_byaddr inet_hosts_file_byaddr bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_file_byname inet_hosts_file_byname bag ? 0 ? ? ?322 ? ? ?inet_db > ?test ? ? ? ? ? ?test ? ? ? ? ? ? ?set ? 0 ? ? ?322 ? ? ?<0.31.0> > ok > 4> ets:delete(test). > true > 5> ets:i(). > ?id ? ? ? ? ? ? ?name ? ? ? ? ? ? ?type ?size ? mem ? ? ?owner > ?---------------------------------------------------------------------------- > ?13 ? ? ? ? ? ? ?code ? ? ? ? ? ? ?set ? 257 ? ?10711 ? ?code_server > ?4110 ? ? ? ? ? ?code_names ? ? ? ?set ? 57 ? ? 7754 ? ? code_server > ?8207 ? ? ? ? ? ?shell_records ? ? ordered_set 0 ? ? ?112 ? ? ?<0.25.0> > ?ac_tab ? ? ? ? ?ac_tab ? ? ? ? ? ?set ? 6 ? ? ?878 > ?application_controller > ?file_io_servers file_io_servers ? set ? 0 ? ? ?322 ? ? ?file_server_2 > ?global_locks ? ?global_locks ? ? ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_names ? ?global_names ? ? ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_names_ext global_names_ext ?set ? 0 ? ? ?322 ? ? ?global_name_server > ?global_pid_ids ?global_pid_ids ? ?bag ? 0 ? ? ?322 ? ? ?global_name_server > ?global_pid_names global_pid_names ?bag ? 0 ? ? ?322 ? ? ?global_name_server > ?inet_cache ? ? ?inet_cache ? ? ? ?bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_db ? ? ? ? inet_db ? ? ? ? ? set ? 29 ? ? 706 ? ? ?inet_db > ?inet_hosts_byaddr inet_hosts_byaddr bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_byname inet_hosts_byname bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_file_byaddr inet_hosts_file_byaddr bag ? 0 ? ? ?322 ? ? ?inet_db > ?inet_hosts_file_byname inet_hosts_file_byname bag ? 0 ? ? ?322 ? ? ?inet_db > ok > From ebegumisa@REDACTED Mon Nov 8 08:13:04 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 08 Nov 2010 18:13:04 +1100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <4CD71598.9000109@amberbio.com> References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4CD71598.9000109@amberbio.com> Message-ID: I should have probably used the term "key-value database" as the original poster did to better home in on the category. Specifically, that class of key-based NoSQL databases that you would list when starting a new project and looking for an alternative to RDBMSs which is what I assumed the thread was about. Mnesia, CouchDB, MongoDB, Cassandra, etc, all have query engines that are aware of how to access the structures being stored. When querying, you can go far beyond just "give me the value with this key". This would be too tedious for the kind of datasets used in RDBMSs that you're looking to store in NoSQL databases instead. So typically, these kv-databases allow you to construct queries that can access the contents of the structures being stored, in-fact this is the norm. You then hand the query over to the query engine and say "fetch me this data with this criteria". - Edmond - On Mon, 08 Nov 2010 08:09:44 +1100, Morten Krogh wrote: > Hi, > > Edmond, you talk about datastores that can peek inside the values > instead of regarding them as opaque binaries. > But are those datastores really kv-stores? > I would claim that, by definition, kv-stores can only do get, put, > delete and maybe some batch insertions and retrievals, but not more. > Couchdb and mnesia should probably not be named kv-stores, but rather > something more descriptive of their capabilities. > > > Also, no-sql is not the same as key-value. > Part of the problem is that no-sql is too loose and broad a category. > > Morten. > > > inside th eI agree with your points about data stores that can look > inside the values > > > On 11/7/10 7:15 PM, Edmond Begumisa wrote: >> Hello Rudolph, >> >> Small comment. I think one has to be careful not to suggest that blobs >> in typical RDBMSs and values in typical kv-stores are similar. I've >> heard that comparison being made before and forgive me, but I just >> don't see it (if that is indeed what what you were suggesting.) >> >> In SQL-RDBMSs, blob fields are normally opaque to the db. Storing blobs >> is the exception rather than the norm so querying against the content >> of these is usually poorly catered for (if catered for at all) by the >> query engine. The result is you normally can't use blobs to store >> anything complex because querying against them is a pain (i.e you'd >> only store things in a blob that you don't intend on using in the >> criteria of an SQL statement.) >> >> Where as for (I think most) kv-stores, the 'values' are not opaque. >> Storying complex data in the values is the norm, so most kv-stores are >> aware of the structural content of those values (e.g. JSON in CouchDB, >> Erlang-Terms in Mnesia.) The result is that kv-store query-engines have >> a mechanism for querying against the complex values being stored (views >> for Couch, qlc for Mnesia). >> >> I would think this renders the use-cases for blobs in RDBM's and values >> in kv-stores in-comparable. For couch's case, the equivalent to a blob >> would be more like an attachment. For Mnesia, a blob would be closer to >> a binary member of a tuple/record. >> >>> The issue is that the information in the "blob" is not useful. Neither >>> stored in the RDBMS, nor in a plain KV-store. >> >> That said, I don't really a agree with this statement. In a typical >> kv-store the 'value' is useful because the kv-store query engine would >> understand how to parse it. A typical SQL query-engine would not. >> >> I'm no Mnesia user, but I would think that you could easily query >> against parts of an Erlang-term parse tree stored in the database. To >> my knowledge, the only way to get the equivalent in an SQL-database >> would be to either... >> >> a) Spread the parse tree across different tables/records (hierarchical >> data modeling difficulty with tables non-withstanding) >> b) Use an atypical SQL database like Postgres/Oracle that supports >> complex/user-defined types >> c) Cheat -- write some sort of plugin that you can from a >> stored-procedure >> >> - Edmond - >> >> -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> >> >> On Sun, 07 Nov 2010 23:09:04 +1100, Rudolph van Graan >> wrote: >> >>> Hi Joe, >>> >>> I am not sure I agree with your statement that traditional databases >>> "suffer" from the fact that columns have simple types. You can easily >>> (within reason) make them store complex types. They are at the most >>> basic level also KV stores. The most basic abstraction for a "Table" >>> is a set of tuples stored in a certain way. There is no reason why you >>> cannot define a table with an arbitrary key and an arbitrary "blob" as >>> value column and then storing your parse tree in that blob field. >>> (Obviously you need to serialise/deserialise it - as you would need to >>> do for any storage abstraction). >>> >>> The issue is that the information in the "blob" is not useful. Neither >>> stored in the RDBMS, nor in a plain KV-store. You have to make it >>> useful before you can do something with it or extract information from >>> it. I guess that is why you said thus: >>> >>>> I'm not thinking in terms of joins and normalizing things - the >>>> thought process >>>> is different - so far I haven't met any problems that don't map onto >>>> key-values >>>> queries. >>> >>> >>> I agree with you. If you throw distribution, map-reduce and lots of >>> indices into the equation you can solve most issues this way. Except >>> for one thing - invariance between multiple records and record types. >>> There are very few examples where it is useful to live without >>> invariance - searching being one example. >>> >>> Invariance is a first order concept in any RDBMS system and essential >>> for many types of systems. However, in my experience, you can achieve >>> application-level invariance in other ways. But it requires >>> substantially more work to approximate first order invariance using >>> KV-stores. It is sometimes simpler to have your data properly >>> normalised I think. >>> >>> Rudolph van Graan >>> >>> >>> On Nov 4, 2010, at 10:01 PM, Joe Armstrong wrote: >>> >>>> On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >>>>> This is a message I sent to the nosql-discussion@REDACTED >>>>> discussion group. I thought it might be interesting to send it >>>>> erlang-questions, so here we go... >>>>> >>>>> >>>>> Hi all! >>>>> >>>>> I have used SQL RDBMSs for some time. I've never used any very >>>>> advanced >>>>> feature, but I know enough of it to make database applications. >>>>> >>>>> Nowadays, I decided it would be interesting to learn some NoSQL >>>>> databases concepts. So I decided to pick up some Erlang and Mnesia, >>>>> its >>>>> native key-value database. More than scalability itself, the most >>>>> valuable feature for me is the possibility of replication and >>>>> synchronization between nodes. >>>>> >>>>> But all pros have cons as companion. The lack of a relationship >>>>> model >>>>> is difficult for who is used to RDBMSs. So, my question is: >>>>> >>>>> * Is there any guide, tutorial, book, whatever, that tries to >>>>> introduce >>>>> NoSQL databases to SQL users? >>>>> >>>>> * Key-value databases are surprising simple. I know you solve >>>>> relationship by denormalizing data. What data should be normalized? >>>>> What shouldn't? How do you update denormalized data? >>>> >>>> I'm no database expert so don't quote me here ... >>>> >>>> As far as I am concerned traditional databases like SQL suffer >>>> from the fact that the data stored in an individual column is >>>> incredible >>>> simple - I can store an integer/string/... in a column but these are >>>> incredibly simple data structures. I want to store and retrieve >>>> incredibly >>>> complicated things - how do I store an XML parse tree in a single cell >>>> of a database? - How do I store a database in a database ... >>>> >>>> In a decent K-V database the value of the key can be *anything* - an >>>> entire database for example, a compiler, ... no worries >>>> >>>> Then when I analyse my problem I start thinking "I can store and >>>> retrieve any >>>> complex object that keys do I need to solve my problem?" >>>> >>>> I'm not thinking in terms of joins and normalizing things - the >>>> thought process >>>> is different - so far I haven't met any problems that don't map onto >>>> key-values >>>> queries. >>>> >>>> It seems to my that SQL provides you with the ability do to complex >>>> queries on simple things. K-V dbs can do simple queries on complex >>>> things. >>>> >>>> /Joe >>>> >>>> >>>> >>>>> >>>>> Sorry for such simple and general questions. Things were simple up >>>>> to >>>>> the moment that I realized that it would be easily solved with a JOIN >>>>> SQL statement. :-) >>>>> >>>>> Thank you very much! >>>>> >>>>> -- Silas Silva >>>>> >>>>> ________________________________________________________________ >>>>> erlang-questions (at) erlang.org mailing list. >>>>> See http://www.erlang.org/faq.html >>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>>> >>>>> >>>> >>>> ________________________________________________________________ >>>> erlang-questions (at) erlang.org mailing list. >>>> See http://www.erlang.org/faq.html >>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>> >>> >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From alceste@REDACTED Mon Nov 8 11:06:19 2010 From: alceste@REDACTED (Alceste Scalas) Date: Mon, 08 Nov 2010 11:06:19 +0100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: On Mon, 08 Nov 2010 04:16:43 +1100, "Edmond Begumisa" wrote: > The main reason I've switched to NoSQL (CouchDB) after many years of > using SQL-RDBMSs is for modeling hierarchical data. RDBMSs force you > to do this with relationships across different tables even when the > data belongs to the same domain/object -- this feels unnatural and > results in the DB equivalent of spaghetti code with hours staring at > relationship diagrams. The relational model can express hierarchical relations. SQL cannot express the whole relational algebra, but it does allow to handle hierarchical relations. You can use self-referencing tables: CREATE TABLE hierarchy ( id INTEGER PRIMARY KEY, parent_id INTEGER REFERENCES hierarchy(id) ON UPDATE CASCADE ON DELETE RESTRICT, ... ); And then you could use common table expressions in queries: http://en.wikipedia.org/wiki/Common_table_expressions Funny note: the relational model was developed 40 years ago in order to provide a declarative and formal method for structuring data, and overcome the limitations of the DBMSs of the time --- which were, in fact, mostly hierarchical and document-oriented, and required a procedural interface in order to perform queries. As a result, the data in hierarchical DBMSs was structured without formal analysis, and only considering the immediate requirements of the application that was going to use it. When other applications needed to access the DB, or the application itself required updates which broke the mapping between DB objects and application objects, then *huge* pains would arise... Regards, -- Alceste Scalas From erlang@REDACTED Mon Nov 8 11:16:40 2010 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 8 Nov 2010 11:16:40 +0100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: I agree - I'd go further. For most of what I want to do the weaker the type the better - even dynamic types are too structured. A lot of what I want can be done with unstructured text and full-text indexing (which has even less structure that a weak type) I find that in all the applications I've programmed so far I want the following (In priority order) 0) Easy of setup I'm lazy - but I don't want to struggle through hour long sessions of Googling for bug-fixes to get stuff working. 1) Key - Value lookup/retrieval What I put in I can get out - the key is simple the Value can be very complex 2) Full text searches I want to search strings for free text 3) Encapsulation I like all the database in a single file - so I can send it in a message move it around etc. 4) Replication I want stuff to be stored forever 5) Scalability I want it to scale Most apps I can do with 1 and 2 - I really like 3 for distributing the result of something. Ets+Dets is pretty ok for 1+3+0. Mnesia for 4. For 5 I guess riak/couchDB are good but I haven't felt the need yet. I've actually used poor-man's replication, this is pure KISS stuff. Put the database in a dets file and replicate using dropbox. I feel that 2) is the big thing that missing at the moment - dets/ets with in-built full text indexing would be *wonderful* - I've made a full-text indexing engine (See https://github.com/joearms/elib1/tree/master/supported/indexer/) But it's not integrated with ets/dets. Does anybody know of any pure-erlang add-ons to ets/dets that do full-text indexing? the tricky but seems to be word extraction and not making/sorting/building the index also building an incremental index might be difficult (I haven't done this). I have never seen a "one-size-fits-all" database - and I haven't even mentioned security availability ... I just don't believe that one paradigm can suit all needs - what I want is a toolkit of pluggable components that do single things and do them well - ets and dets are two such components - missing components are the full-text-indexer, the replication manager, ... A number of components that can be glued together is what I want not a pre-packaged solution. Right now what I want is dets/ets with full-text indexing and search ... /Joe On Sun, Nov 7, 2010 at 4:06 PM, Steve Davis wrote: > On Nov 7, 6:09?am, Rudolph van Graan wrote: >> Invariance is a first order concept in any RDBMS system and essential for many types of systems. However, in my experience, you can achieve application-level invariance in other ways. But it requires substantially more work to approximate first order invariance using KV-stores. ?It is sometimes simpler to have your data properly normalised I think. > > "not an expert" caveat applies to me also... > > It appears to me that this discussion is another expression of the > 'strong vs weak/dynamic vs static type' discussion. > > Some observations/connections that seem relevant to me: > - The underlying nature of SQL DBs is revealed by the fact that > queries by anything except the primary key usually need additional > work (sequences/indexes) to be respectably efficient. > - C.J.Date uses the term tuple to refer to a "row in a table" (cf also > Mnesia). > - Date also notes that "objects" can always be mapped directly on to a > relational model. > - Mnesia is interesting in that it encourages a table to be confined > to a single record type, and allows indexing of record tuple members > (cf column indexing in SQL databases). > - The process of normalization suggests very strong typing of the > data. > > ...it makes me suspect that an imperative and strongly-typed language > paradigm has been a very strong motivator in the evolution of SQL > databases; and perhaps the popularity of NoSQL/NotSQL is an expression/ > outcome of the rise of recent trends in programming language uptake. > > /s > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ebegumisa@REDACTED Mon Nov 8 14:49:35 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 09 Nov 2010 00:49:35 +1100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> Message-ID: Alceste, Very true. Hierarchical data can be expressed even in simple SQL databases that don't support the self-reference/common-table constructs you describe. You can do interesting things with indices and almost-circular SQL-statements. Some DBs even have a whole API devoted to this sort of thing -- I used Microsoft's "Data Shaping" extension to SQL for quite some time. Nevertheless, it never feels as natural and is never as easy as it is in NoSQL/kv-databases. Your historical note is interesting. It's almost as though we are going backwards with NoSQL, like in a lot of other areas of development that are proclaimed as "new" (e.g. I always looked at Ajax/Web 2.0 and thought - "wait a minute, that's just rich/smart client development that's been around since the 90s!") Your observation of data creeping into a database that's not consistent with the application model is one of the arguments used for RDBMs against NoSQL. The idea that relationships and constraints can be used to safeguard bad data. I guess this is one of the things that you weight/prioritise when choosing either to go NoSQL or SQL. My problem with relationships is that they tend to be used to implement too many things and introduce excessive complexity in an attempt to guard against bad data. From my experience, SQL is uses relationships/joins for two different things... 1) Containment: Orders contain items --- create a relationship. 2) Actual Relationships: Orders have corresponding invoices -- create a relationship. The problem is containment is such a common thing that a good chunk of your relationships aren't really relationships. This is where relationship diagrams and SQL queries start to bloat. To the extent that I sometimes question whether the benefits gained from guarding against bad data are negated by the complexity introduced. NoSQL/kv-dbs on the on-the-other-hand tend to give you real containment when you want it (even deep hierarchies if you need them), and relationships when you actually want relationships. The cost is less sophistication in auto-guarding against bad data. The "SQL-way" is: don't trust the programmer, make sure everything fits the per-defined data model (even if it is inconvenient/doesn't map nicely against the application's model.) The "NoSQL-way" is: trust the programmer, make it easier to model data that is convenient for/maps nicely against the application's model. For me, the NoSQL-way makes more sense. The database should work for the programmer, not the other way around. - Edmond - On Mon, 08 Nov 2010 21:06:19 +1100, Alceste Scalas wrote: > > On Mon, 08 Nov 2010 04:16:43 +1100, "Edmond Begumisa" > wrote: >> The main reason I've switched to NoSQL (CouchDB) after many years of >> using SQL-RDBMSs is for modeling hierarchical data. RDBMSs force you >> to do this with relationships across different tables even when the >> data belongs to the same domain/object -- this feels unnatural and >> results in the DB equivalent of spaghetti code with hours staring at >> relationship diagrams. > > The relational model can express hierarchical relations. SQL cannot > express the whole relational algebra, but it does allow to handle > hierarchical relations. You can use self-referencing tables: > > CREATE TABLE hierarchy ( > id INTEGER PRIMARY KEY, > parent_id INTEGER > REFERENCES hierarchy(id) > ON UPDATE CASCADE > ON DELETE RESTRICT, > ... > ); > > And then you could use common table expressions in queries: > > http://en.wikipedia.org/wiki/Common_table_expressions > > Funny note: the relational model was developed 40 years ago in > order to provide a declarative and formal method for structuring > data, and overcome the limitations of the DBMSs of the time --- > which were, in fact, mostly hierarchical and document-oriented, > and required a procedural interface in order to perform queries. > > As a result, the data in hierarchical DBMSs was structured without > formal analysis, and only considering the immediate requirements of > the application that was going to use it. When other applications > needed to access the DB, or the application itself required updates > which broke the mapping between DB objects and application objects, > then *huge* pains would arise... > > Regards, -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Nov 8 15:00:39 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 09 Nov 2010 01:00:39 +1100 Subject: [erlang-questions] Conceptual questions on key-value databases for RDBMs users In-Reply-To: <4CD33A52.4050402@telegraphics.com.au> References: <20101102201437.GC626@hope.tifa.renegado> <4CD33A52.4050402@telegraphics.com.au> Message-ID: Hello Ahmed, Toby... > it is a tenet of the relational model that column values > be 'atomic' in the sense that they do not encode structure that > would be more profitably decomposed into relations. This is very well put. The "SQL-way" of modeling complex data is through multiple tables and relationships/joins. Other techniques are exceptional. >> i believe the assumption of storing simple thing is not a general rule >> and >> should not be associated with SQL. For example, postgresql provides more >> complex data types and even user defined ones. I'd hardly describe Postgres, a database engine that features things like table-inheritance, as a typical SQL-RDBMS. In many ways, Posgres has always been cutting edge and distinctly atypical in it's field. But you're right, using orders->items as an example, with Postgres, you can store the items in a column of an orders table. If you then want to query all orders that contain X-Box, you could then write something similar to... SELECT Orders.* FROM Orders WHERE (Orders.Item).Desc = "X-Box"; This would be somewhat similar to how a NoSQL/kv-db would look at it (though I'm not sure how you would go about tallying the item totals for each order, but knowing Posgres, there must be a way.) However, this is NOT typically how an SQL database would be organised, even with Postgres that has that capability. SQL people tend to think in terms of joins and relationships. Instead, you'd have two tables, one with orders and another with items, then a similar (but not exactly the same) query would look something like... SELECT Orders.*, Items.* FROM Orders, Items INNER JOIN Orders ON (Orders.Id = Items.OrderId) WHERE Items.Desc = "X-Box"; Complex data in the SQL world is normally modeled using relationships/joins. IMO, this is both SQL-RDBMS biggest strength and biggest weakness. Strength -- you can define "rules" for what's valid and invalid in your database. Weakness -- when the complex data you're trying to model is wide and deep, the database gets insanely complex, very quickly, resulting in a lot of head-scratching when querying. Contrast with the NoSQL approach. Strength -- modeling wide and deep complex data is much easier and tends to map more directly to the application's idea of the data. Weakness -- auto-guarding against bad data is more difficult so you must trust the programmers. - Edmond - On Fri, 05 Nov 2010 09:57:22 +1100, Toby Thain wrote: > On 04/11/10 6:11 PM, Ahmed Omar wrote: >> Hi Joe, All >> I'm far from being a database expert too, so feel free to correct me , >> but >> i believe the assumption of storing simple thing is not a general rule >> and >> should not be associated with SQL. For example, postgresql provides more >> complex data types and even user defined ones. > > As Joe must be aware, it is a tenet of the relational model that column > values be 'atomic' in the sense that they do not encode structure that > would be more profitably decomposed into relations. > > --Toby > >> > >> >>> On Tue, Nov 2, 2010 at 9:14 PM, Silas Silva wrote: >>>> This is a message I sent to the nosql-discussion@REDACTED >>>> discussion group. I thought it might be interesting to send it >>>> erlang-questions, so here we go... >>>> >>>> >>>> Hi all! >>>> >>>> I have used SQL RDBMSs for some time. ... >>>> * Key-value databases are surprising simple. I know you solve >>>> relationship by denormalizing data. What data should be normalized? >>>> What shouldn't? How do you update denormalized data? >>> > >> On Thu, Nov 4, 2010 at 11:01 PM, Joe Armstrong wrote: > >>> I'm no database expert so don't quote me here ... >>> >>> As far as I am concerned traditional databases like SQL suffer >>> from the fact that the data stored in an individual column is >>> incredible >>> simple - I can store an integer/string/... in a column but these are >>> incredibly simple data structures. I want to store and retrieve >>> incredibly >>> complicated things - how do I store an XML parse tree in a single cell >>> of a database? - How do I store a database in a database ... >>> ... >>> It seems to my that SQL provides you with the ability do to complex >>> queries on simple things. K-V dbs can do simple queries on complex >>> things. >>> >>> /Joe >>> >>> >>> >>>> >>>> Sorry for such simple and general questions. Things were simple up to >>>> the moment that I realized that it would be easily solved with a JOIN >>>> SQL statement. :-) >>>> >>>> Thank you very much! >>>> >>>> -- >>>> Silas Silva >>>> >>>> ________________________________________________________________ >>>> erlang-questions (at) erlang.org mailing list. >>>> See http://www.erlang.org/faq.html >>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>> >>>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >>> >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From mevans@REDACTED Mon Nov 8 16:15:49 2010 From: mevans@REDACTED (Evans, Matthew) Date: Mon, 8 Nov 2010 10:15:49 -0500 Subject: [erlang-questions] Efficient Insertions in Mnesia tables In-Reply-To: References: Message-ID: I'm wondering if you could edit the mnesia library, find out where the DETS file is created and add the option: {min_no_slots,SomeLargeNumber} http://www.erlang.org/doc/man/dets.html#open_file-2 Matt -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Rudolph van Graan Sent: Sunday, November 07, 2010 6:27 AM To: Olivier BOUDEVILLE Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Efficient Insertions in Mnesia tables Hi, This sounds like a disk subsystem issue. DETS (disc_only mnesia tables) uses buckets to store objects and will allocate (and reallocate) objects within buckets as you add more objects to it. If a bucket does not have space for a new object, the bucket must be split. This means the DETS file grows and some of the data is moved. Depending on your operating system, file system, record size on the file system, this will result in a lot of IO. In my opinion, what you see is to be expected - DETS selects a bucket based on the object's key's MD5 hash, so a specific insert can hit any bucket essentially at random. DETS is not a good choice if you want to constantly append to a table, but it works reasonably well if you have a finite set of keys. Rudolph van Graan www.patternmatched.com On Nov 5, 2010, at 10:15 AM, Olivier BOUDEVILLE wrote: > Hi, > > We are trying to write (with mnesia:dirty_write) in a disc_only_copies > Mnesia table (type: set, not fragmented, not replicated) records (ex: 60 > 000 of them) and we observe that the insertion time is increasing as the > table is increasingly crowded. This is not really a surprise but something > we need to avoid. What we would like is to have constant (and preferably > low) insertion times, like we had when writing directly to a file. > > We tried to get as close as possible with the following settings and use: > > % We want tables to be dumped less frequently from > memory to disc, > % in order to buffer writings (default value is > 4): > ok = application:set_env( mnesia, dc_dump_limit, 1 > ), > > % Increases a lot (default value is 100) the > maximum number of > % writes to the transaction log before a new dump > is performed: > ok = application:set_env( mnesia, > dump_log_write_threshold, 50000 ), > > Over time we see the CPU load decrease steadily, the computer seems to > spend most of its time fighting for locks. > > We happen to be in a pretty favorable situation (only writes, no > concurrent access to a given table). We chose disc_only_copies as there > might be a large number of such tables and if they filled over time they > could exhaust the RAM. > > Is there anything we missed that would allow us (roughly) constant > insertion times with Mnesia? > > Thanks in advance for any hint, > Best regards, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 > 65 27 13 > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. From olivier.boudeville@REDACTED Mon Nov 8 17:30:43 2010 From: olivier.boudeville@REDACTED (Olivier BOUDEVILLE) Date: Mon, 8 Nov 2010 17:30:43 +0100 Subject: [erlang-questions] Efficient Insertions in Mnesia tables In-Reply-To: Message-ID: Hello Matt and Rudolph, Thanks for your advice. Indeed a min_no_slot option could be neat for some kinds of use. I was hoping that I could use Mnesia to store my records very much like my basic file I/O implementation (just appending data to the disc_only_copies), but I had not anticipated that the insertion time could not be a 0(1) and could grow that much. As it is disc_only_copies, why would a large number of buckets be kept in RAM? Unless a default index is used for such a case? This is a bit of a surprise for me as I imagined that for example telco logs could be using Mnesia for similar cases: the use case of dumping a large number of entries to a file-based backend, with no caching in RAM lest it does not fit seemed quite common for me. Anyway, as Rudolph hinted, I guess I was trying to use the wrong tool for the task; I will stick to the writing to files. Thanks again, Best regards, Olivier Boudeville. --------------------------- Olivier Boudeville EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France D?partement SINETICS, groupe ASICS (I2A), bureau B-226 Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 65 27 13 mevans@REDACTED 08/11/2010 16:16 A rvg@REDACTED, olivier.boudeville@REDACTED cc erlang-questions@REDACTED Objet RE: [erlang-questions] Efficient Insertions in Mnesia tables I'm wondering if you could edit the mnesia library, find out where the DETS file is created and add the option: {min_no_slots,SomeLargeNumber} http://www.erlang.org/doc/man/dets.html#open_file-2 Matt -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Rudolph van Graan Sent: Sunday, November 07, 2010 6:27 AM To: Olivier BOUDEVILLE Cc: erlang-questions@REDACTED Subject: Re: [erlang-questions] Efficient Insertions in Mnesia tables Hi, This sounds like a disk subsystem issue. DETS (disc_only mnesia tables) uses buckets to store objects and will allocate (and reallocate) objects within buckets as you add more objects to it. If a bucket does not have space for a new object, the bucket must be split. This means the DETS file grows and some of the data is moved. Depending on your operating system, file system, record size on the file system, this will result in a lot of IO. In my opinion, what you see is to be expected - DETS selects a bucket based on the object's key's MD5 hash, so a specific insert can hit any bucket essentially at random. DETS is not a good choice if you want to constantly append to a table, but it works reasonably well if you have a finite set of keys. Rudolph van Graan www.patternmatched.com On Nov 5, 2010, at 10:15 AM, Olivier BOUDEVILLE wrote: > Hi, > > We are trying to write (with mnesia:dirty_write) in a disc_only_copies > Mnesia table (type: set, not fragmented, not replicated) records (ex: 60 > 000 of them) and we observe that the insertion time is increasing as the > table is increasingly crowded. This is not really a surprise but something > we need to avoid. What we would like is to have constant (and preferably > low) insertion times, like we had when writing directly to a file. > > We tried to get as close as possible with the following settings and use: > > % We want tables to be dumped less frequently from > memory to disc, > % in order to buffer writings (default value is > 4): > ok = application:set_env( mnesia, dc_dump_limit, 1 > ), > > % Increases a lot (default value is 100) the > maximum number of > % writes to the transaction log before a new dump > is performed: > ok = application:set_env( mnesia, > dump_log_write_threshold, 50000 ), > > Over time we see the CPU load decrease steadily, the computer seems to > spend most of its time fighting for locks. > > We happen to be in a pretty favorable situation (only writes, no > concurrent access to a given table). We chose disc_only_copies as there > might be a large number of such tables and if they filled over time they > could exhaust the RAM. > > Is there anything we missed that would allow us (roughly) constant > insertion times with Mnesia? > > Thanks in advance for any hint, > Best regards, > > Olivier. > --------------------------- > Olivier Boudeville > > EDF R&D : 1, avenue du G?n?ral de Gaulle, 92140 Clamart, France > D?partement SINETICS, groupe ASICS (I2A), bureau B-226 > Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 > 65 27 13 > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. From kenneth.lundin@REDACTED Mon Nov 8 17:31:31 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Mon, 8 Nov 2010 17:31:31 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: References: Message-ID: On Sun, Nov 7, 2010 at 9:13 PM, Vlad Dumitrescu wrote: > Hi! > > I've got lately more and more requests (some even with foul language) > about why erlide (the erlang ide for eclipse) doesn't support utf8 > encoding of erlang source files. My answer has always been to point > out to the erlang docs where it is clearly stated that source files > are still to be Latin1 encoded, regardless of the recent > unicode-supporting libraries. The documentation > (http://www.erlang.org/doc/apps/stdlib/unicode_usage.html) that says: > "Also the source code is (for now) still expected to be written using > the ISO-latin-1 character set, why Unicode characters beyond that > range cannot be entered in string literals." > > The trouble is that the scanner actually accepts utf8 encoding in > literal strings and comments, possibly even in quoted atoms. See the > attached file for an example, it compiles fine even on my Swedish > system (the output from running it is garbled for me, I assume it > would look fine on a properly setup terminal). So users from countries > where Latin1 is useless but utf8 is not, may use the latter instead > without getting any problems. > > Eclipse requires the encoding of a file to be specified and of course, > I can't set that to something else than the official Latin1. But > people that used utf8 before and it worked, now get frustrated because > it's no longer accepted when they use erlide. It's easy to blame > erlide for that and ask for a fix because it looks like it works > outside of eclipse... > > The simple solution would be to update the docs above to say something > like "Please note that UTF-8 encoded files can be accepted by the > compiler and may work as expected in some environments, but this usage > is not recommended and not supported." Also, comments should be > mentioned alongside literal strings. Would this be acceptable? > > The complex solution would be to make sources encoding-aware (maybe as > suggested here http://www.erlang.org/cgi-bin/ezmlm-cgi?4:msp:46892), > but I know this is by no means a simple task. Does the OTP team have > anything scheduled in this area? R15, R16? The OTP team does not have anything scheduled regarding support for UTF-8 in sources yet, but I agree on that we ought to support it. The question is "only" how it should work to be as useful as possible and still reasonably enough backward compatible. I think this is a prefect fit for an EEP. Maybe the link http://www.erlang.org/cgi-bin/ezmlm-cgi?4:msp:46892 is a good start. Any volunteers? /Kenneth , Erlang/OTP, Ericsson > > best regards, > Vlad > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From banibrata.dutta@REDACTED Mon Nov 8 18:47:36 2010 From: banibrata.dutta@REDACTED (Banibrata Dutta) Date: Mon, 8 Nov 2010 23:17:36 +0530 Subject: [erlang-questions] Efficient Insertions in Mnesia tables In-Reply-To: References: Message-ID: On Mon, Nov 8, 2010 at 10:00 PM, Olivier BOUDEVILLE < olivier.boudeville@REDACTED> wrote: > This is a bit of a surprise for me as I imagined that for example telco > logs could be using Mnesia for similar cases: > Specifically for this purpose the log4erl module may be used. From allanwegan@REDACTED Mon Nov 8 21:53:22 2010 From: allanwegan@REDACTED (Allan Wegan) Date: Mon, 08 Nov 2010 21:53:22 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: References: Message-ID: <4CD86342.20904@allanwegan.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > The question is "only" how it should work to be as useful as possible > and still reasonably enough backward compatible. It should assume that a BOM is present at the start of the file. Then fall back to UTF-8 if none is found. And fall back to ISO-8859-1 if UTF-8 parsing failed. This procedure is suggested for files with unknown encoding in Chapter "Heuristic identification of UTF-8" in the Erlang documentation at . Most valid character combinations of ISO-8859-1 that also are valid UTF-8 look weird even in non-English texts. Some examples: ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? There are in fact as many valid combinations, as there are valid Unicode code points. But the chance of recognizing a file as valid UTF-8 where it is in fact encoded in ISO-8859-1 is really small. - -- Allan Wegan Jabber: allanwegan@REDACTED ICQ: 209459114 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) iQEcBAEBAgAGBQJM2GNCAAoJENm5axHh7Acxe2EH/A4lmp4dxvButPapWREvuePk wg+PnoRUU9PAue4sspPddYOPhdqlD5lWXVL86hUG84uIDiKNx2PSoBL0WT0XeQPj E6g21100HpjTcyI0qcBoWpj2KKurAGVtc3TP6eFbHn21X9AsP3CAckj1xyiy2BRh 4R16kDUABhgp4SkwP0D51wgRcCjaKmZHJvHPhCZFfogk/vs1SWajZ/JBi/Z8DDkV rlBS4ddu1+AMICFGlybV/99uqh8nc2YTQTl1HqyPf10KMz676ar1m5P75ZUw0wdd khLXyEs+gzOD5vxeEZ3uflFPQm2UwBM6+J9N5X0hiV2McTG5pLp69ccP5375F68= =pNEA -----END PGP SIGNATURE----- From allanwegan@REDACTED Mon Nov 8 21:58:22 2010 From: allanwegan@REDACTED (Allan Wegan) Date: Mon, 08 Nov 2010 21:58:22 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: References: Message-ID: <4CD8646E.4070106@allanwegan.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > So users from countries where Latin1 is useless but utf8 is not, > may use the latter instead without getting any problems. Just let them use UTF-8 at will. As long as they do not use any non-ASCII code points in language constructs except binary string literals, they should be on the safe side anyway. - -- Allan Wegan Jabber: allanwegan@REDACTED ICQ: 209459114 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) iQEcBAEBAgAGBQJM2GRuAAoJENm5axHh7AcxWZAIALMT5BsobLOvAzlbL9Sq6yPK xPBSHNih7YiNwFsyXZ5EQIf2bWMK+uI0ycJv4XRYb4RXcQLXVFCNcHX76mOHBJvR V8+cStiE6fM2sWmep2Os0Kk76WUCy25GyUCJedi6to1kUDv4WiAkKcgZ3533wpVN WL+XQmW5XtcTNd4XQIyWHdsF6Oqmp3CtAuedZK4Dv82Y0tRZ69/wygoZZ4mTZx84 j29zOh71L3qEZecN0MwGGAkWK8pbkqhS3A0LVDLL2EUq77CHTVTSP3bpFRtlEdIU tFzDuFVSXWFenVtvPOUmXWrm4uW6xouyXBvVacM3MrCwV/aoKbnkoohruveUGgY= =bHNm -----END PGP SIGNATURE----- From vladdu55@REDACTED Mon Nov 8 22:29:43 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Mon, 8 Nov 2010 22:29:43 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: <4CD8646E.4070106@allanwegan.de> References: <4CD8646E.4070106@allanwegan.de> Message-ID: On Mon, Nov 8, 2010 at 21:58, Allan Wegan wrote: >> So users from countries where Latin1 is useless but utf8 is not, >> may use the latter instead without getting any problems. > > Just let them use UTF-8 at will. As long as they do not use any > non-ASCII code points in language constructs except binary string > literals, they should be on the safe side anyway. I have no problem with that. Eclipse requires that a default encoding has to be declared and users can override it. It works for me, but not for everybody it seems (otherwise they would not want me to fix it). >From what I saw, the usage is in regular literal strings and in comments. Some of the reports mean that some code (probably the lexical scanner) can get confused, but I couldn't debug it to find out the cause. If a full solution was relatively near at hand from OTP, then I could have waited for it. Since it's not I will have to handle that somehow. regards, Vlad From masklinn@REDACTED Mon Nov 8 22:44:11 2010 From: masklinn@REDACTED (Masklinn) Date: Mon, 8 Nov 2010 22:44:11 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: <4CD86342.20904@allanwegan.de> References: <4CD86342.20904@allanwegan.de> Message-ID: <5F66736A-C44D-4A76-A956-ED956F8B7E91@masklinn.net> On 2010-11-08, at 21:53 , Allan Wegan wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 >> The question is "only" how it should work to be as useful as possible >> and still reasonably enough backward compatible. > It should assume that a BOM is present at the start of the file. Why? BOMs are for UTF-16 and UTF-32 which are generally silly encodings to use in source file From ericsetton@REDACTED Mon Nov 8 23:09:02 2010 From: ericsetton@REDACTED (Eric Setton (PA)) Date: Mon, 8 Nov 2010 17:09:02 -0500 Subject: Tango needs Erlang Engineers Message-ID: <93FAAFD5-619E-4B22-A538-091900038883@tango.me> I am the CTO/Founder of Tango, a mobile2mobile video calling service that allows iPhones, Androids, and other videophones to connect over the 3G network. We launched across the mobile app market at the end of September and we were already over million users in our first 10 days. I am attempting to do some recruiting myself, and looking for Senior Erlang Engineers. I'm hoping you might be interested in helping us build mobile video communication software that could change how millions of people connect around the world. If you would be interested email me at ericsetton@REDACTED I look forward to hearing from you, Eric Setton a link to an article regarding our meteoric growth: http://www.bloomberg.com/news/2010-11-02/tango-challenges-apple-yahoo-in-smartphone-video-chat-market.html From fritchie@REDACTED Mon Nov 8 23:29:34 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Mon, 08 Nov 2010 16:29:34 -0600 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: Message of "Mon, 08 Nov 2010 11:16:40 +0100." Message-ID: <26517.1289255374@snookles.snookles.com> I've a couple of comments, joining the thread late, sorry. Regarding the labels of "key-value database" vs. "document-oriented database", I typically think of KV DBs as considering their value blobs to be opaque and doc-oriented DBs as understanding all (or at least part) of the structure of their values. * Hibari is KV * Riak is both (the lower parts of the KV store treat blobs as opaque, but both the map-reduce KV framework and the new Riak Search app can parse the value blobs as JSON, Erlang terms, or whatever a Search text analyzer decides to parse) * CouchDB and MongoDB are doc-oriented Joe Armstrong wrote: ja> Does anybody know of any pure-erlang add-ons to ets/dets that do ja> full-text indexing? the tricky but seems to be word extraction and ja> not making/sorting/building the index also building an incremental ja> index might be difficult (I haven't done this) Not that I'm aware of. In the Riak Search app, IIRC, text analysis (and stemming, etc.) can be done in Erlang or Java. The inverted indexes managed at each node are pure Erlang (specifically the merge_index app, see apps/merge_index/src subdir). Replication of the merge_index indexes are managed by the riak_core application. https://github.com/basho/riak_search https://github.com/basho/riak_core -Scott From th.lachmann@REDACTED Mon Nov 8 23:37:17 2010 From: th.lachmann@REDACTED (Thomas Lachmann) Date: Mon, 8 Nov 2010 23:37:17 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: <4CD8646E.4070106@allanwegan.de> References: <4CD8646E.4070106@allanwegan.de> Message-ID: <20101108233717.6f5ed265@optd.heidelberg.local> Hi. Am Mon, 08 Nov 2010 21:58:22 +0100 schrieb Allan Wegan : > Just let them use UTF-8 at will. As long as they do not use any > non-ASCII code points in language constructs except binary string > literals, they should be on the safe side anyway. May be I'm wrong in my understanding of binary string literal is wrong. But using non-ASCII letters in binaries has been the place where I first realized, that I did store my erlang source code in utf-8 instead of latin-1. The following code does print "Size: 6" if encoded in latin-1 and "Size: 12" if encoded in utf-8. -module(src_enc). -compile([export_all]). t()-> B = <<"??????">>, io:format("Size: ~w~n", [byte_size(B)]). best regards, Thomas From ok@REDACTED Tue Nov 9 02:59:11 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 9 Nov 2010 14:59:11 +1300 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: On 8/11/2010, at 4:06 AM, Steve Davis wrote: > It appears to me that this discussion is another expression of the > 'strong vs weak/dynamic vs static type' discussion. > > ...it makes me suspect that an imperative and strongly-typed language > paradigm has been a very strong motivator in the evolution of SQL > databases; and perhaps the popularity of NoSQL/NotSQL is an expression/ > outcome of the rise of recent trends in programming language uptake. You *cannot* call the types in classic SQL "strong". Numbers, strings, and byte strings for everything is what Joe is complaining of and he is right to do so. Encoding something as a string or blob basically results in the data base itself having no clue about the structure or meaning of the data. It is important to understand that SQL is not a good example of a relational system. A move away from SQL *could* be a move towards relational! One of the core concepts in relational systems is that of enforced integrity constraints, via primary keys and foreign keys. If you don't have any integrity constraints that you care to tell the data base about, you probably don't need a relational system. From ok@REDACTED Tue Nov 9 03:03:59 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 9 Nov 2010 15:03:59 +1300 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: Message-ID: On 8/11/2010, at 7:00 AM, Icarus Alive wrote: > wondering as to what is the reason that Erlang/OTP binaries are > available only for win32 and not for Linux (and it's variants), BSD > etc. ? Let's see. I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos 64-bit, and Fedora. Just how many sets of binaries do you expect to be provided? Then there are some older machines that would be fun to set up, several PowerPC boxes and a couple of SPARCs. How many sets of binaries? From ulf.wiger@REDACTED Tue Nov 9 06:19:16 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 8 Nov 2010 22:19:16 -0700 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: <77347084-21B9-4D05-9BBB-852BCF464F35@erlang-solutions.com> On 8 Nov 2010, at 03:16, Joe Armstrong wrote: > Does anybody know of any pure-erlang add-ons to ets/dets that do > full-text indexing? http://jungerl.cvs.sourceforge.net/viewvc/jungerl/jungerl/lib/rdbms/src/ The rdbms_wsearch*.erl modules. They were never actually integrated into RDBMS, as they needed a form of higher-order indexing level (aggregated index). By the time I had figured out how to do that, I got side-tracked... The author of the modules is Hans Nilsson, and they implement the Porter Stemming algorithm. BR, Ulf W Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From icarus.alive@REDACTED Tue Nov 9 06:32:14 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Tue, 9 Nov 2010 11:02:14 +0530 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: <20101107195134.GA12011@corelatus.com> References: <20101107195134.GA12011@corelatus.com> Message-ID: Thank you Matthias and others who have responded. Kindly see some comments inline. On Mon, Nov 8, 2010 at 1:21 AM, Matthias Lang wrote: > I think you have two choices. > > Either, you install from a package and accept that the version you get > is not the latest and greatest. But that's pretty usual, e.g. Ubuntu > 10.10 comes with Perl 5.10.1, not 5.12.2. > That is indeed true, but the feature/bugs delta between 5.12.2 and 5.10.1 (probably so, because haven't actually checked, so this could only be a matter of perception) doesn't look as "crucial" as the delta between R10 / R14B, or even R13A / R14B... which was the case for RHEL5.5 (prior) and Ubuntu 9.10 (latter). > Or, you install from source. Of course, but that means I must have the build system setup. This is not a big inconvenience, but would be so nice to have prebuilt binaries. My target installation platform was "minimal" server install in virtualized guests, with a target of keeping disk-space/RAM usage to a minimum. I could build on a different machine and then install on target servers, but it would have been more-convenient if I could avoid the build stage altogether... especially if figuring out the right build environment (given the choices and issues with multiple GCC versions), isn't something I wish to indulge in at the moment. > Packaging support for the major distributions is not too bad. It could > be better, but not _that_ much better. Looking at the machines here: > > ?Debian unstable, testing and stable: R14A, R14A and R12B, respectively > > ?Ubuntu 10.10: R13B03. > > R13B04 would be the best "conservative" choice and R14B the best "must > have the latest" choice, so they're not far off. > > I don't have any experience with the BSDs, but I think R14B is in the > ports, so that looks fine too. Thanks for pointing those out. Looks like I needed to look bit further/deeper... especially for FreeBSD under "ports" and for RHEL/CentOS in EPEL. > Anyway, your question was, sort of, "Why do Windows users get binaries > from erlang.org?". Part of the reason is that they have special needs: > > ?- Windows tradition is that you install binaries from vendors > ? ?(i.e. not from a central repository) > > ?- Windows users/developers are generally less capable and willing to > ? ?compile from source. E.g. most windows systems don't even have a C > ? ?compiler! > > ?- For a long time, building Erlang under Windows was tricky > > There are probably other reasons, possibly related to commercial Erlang. > > What's the problem with installing from source on CentOS? > > ---------------------------------------------------------------------- > > On Sunday, November 07, Icarus Alive wrote: >> wondering as to what is the reason that Erlang/OTP binaries are >> available only for win32 and not for Linux (and it's variants), BSD >> etc. ? >> given the nature of Erlang, it seems a bit unusual that it's binaries >> are available for a platform that is predominantly a desktop OS, but >> none for more server-oriented OSs. Of course, making it available on >> windows, may help a part (possibly large) of the developer community, >> but when it comes to deploying Erlang applications, user runs into lot >> of trouble. With the Ubuntu 9.10, after bit struggle I did manage to >> install Erlang R13, but then on CentOS, the best I can do with >> EPEL-5.3 is R12, while R14 is already out since Sept-15th. Searching >> through sugestions or how-to's, on installing Erlang on CentOS, it's >> amazing to find how much trouble it can be, even building it from >> source. >> >> So, is it only because there hasn't been a maintainer who is available >> for packages on these other OSs, which BTW is quite understandable, or >> there is something more to it ? >> >> Awareness about Erlang is growing, and it's popularity is rising, >> however the speed of adoption may grow much more, if getting Erlang on >> Linux. >> >> cheers, >> Icarus. >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > From icarus.alive@REDACTED Tue Nov 9 06:36:34 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Tue, 9 Nov 2010 11:06:34 +0530 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 7:33 AM, Richard O'Keefe wrote: > > On 8/11/2010, at 7:00 AM, Icarus Alive wrote: > >> wondering as to what is the reason that Erlang/OTP binaries are >> available only for win32 and not for Linux (and it's variants), BSD >> etc. ? > > Let's see. ?I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, > Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos > 64-bit, and Fedora. ?Just how many sets of binaries do you expect to be > provided? ?Then there are some older machines that would be fun to set > up, several PowerPC boxes and a couple of SPARCs. ?How many sets of > binaries? Am new to the whole Erlang ecosystem (language, runtime, community, resources), so pardon me if this the expectation seemed excessive. For one single person building/maintaining so many variants is definitely a herculean task, and I wasn't aware that it was so. My expectations were set by distribution of Python, Java, where the latest/greatest (& older) versions are available for most commonly used OS's... the mainstream Linux distros, FreeBSD, Solaris, Windows etc. From allanwegan@REDACTED Tue Nov 9 06:52:03 2010 From: allanwegan@REDACTED (Allan Wegan) Date: Tue, 09 Nov 2010 06:52:03 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: <5F66736A-C44D-4A76-A956-ED956F8B7E91@masklinn.net> References: <4CD86342.20904@allanwegan.de> <5F66736A-C44D-4A76-A956-ED956F8B7E91@masklinn.net> Message-ID: <4CD8E183.5080708@allanwegan.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> It should assume that a BOM is present at the start of the file. > Why? BOMs are for UTF-16 and UTF-32 which are generally silly encodings to use in source file It does not hurt to support BOMs. They may be used with UTF-8 too and Erlang already has easy to use functions dealing with BOM detection. If there is no BOM, UTF-8 will be assumed anyway. Nothing is lost due to supporting BOMs. - -- Allan Wegan Jabber: allanwegan@REDACTED ICQ: 209459114 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) iQEcBAEBAgAGBQJM2OGDAAoJENm5axHh7Acx2VwH/R0vUnMhtjCSgxT8ncaMOmC9 z4Ewj9ssbT4PyY9csvBtiBLWpQbdavmJ10j07qJMb/WLB2IyP62NhBjHvHC1jBeE HX8lgb1lTonZHpMbY2zZH9ScHCVe9sX+9pIBEaVMMs3AnCTyCqch+ab3P5EsKHOD UUA8CVgDwWyDnInvzrgF49duT7VTRL0i8BEqa4cw6Zn5Qs6fVJLXtATgUluZM7Fn JYed8aXim6a9W7agRDfFNyw1x3Ug5qGIa3c1qEDeot0H+7GoR1VGxmZyGTZ2PgEY bdUC8XyN+31pWTnXWiMPkkGnj22ioQc45GdJbM4lWuyIdtbKzXPEOId1b6lpbUQ= =oX7i -----END PGP SIGNATURE----- From allanwegan@REDACTED Tue Nov 9 07:10:53 2010 From: allanwegan@REDACTED (Allan Wegan) Date: Tue, 09 Nov 2010 07:10:53 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: <20101108233717.6f5ed265@optd.heidelberg.local> References: <4CD8646E.4070106@allanwegan.de> <20101108233717.6f5ed265@optd.heidelberg.local> Message-ID: <4CD8E5ED.4010804@allanwegan.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> Just let them use UTF-8 at will. As long as they do not use any >> non-ASCII code points in language constructs except binary string >> literals, they should be on the safe side anyway. > > May be I'm wrong in my understanding of binary string literal is wrong. > But using non-ASCII letters in binaries has been the place where I > first realized, that I did store my erlang source code in utf-8 instead > of latin-1. > > The following code does print "Size: 6" if encoded in latin-1 and "Size: > 12" if encoded in utf-8. It works as expected: Put UTF-8 in and you get UTF-8 out. Put ISO-8859-1 in and you get exactly that out. If you want to use UTF-8 "string literals", just edit the source code as UTF-8 and use binaries for string storage. Beware, that $? does currently _not_ work even when used in binaries. Only the first byte of the code point in UTF-8 representation makes it into the resulting binary (at least on my system running R14A on windows). - -- Allan Wegan Jabber: allanwegan@REDACTED ICQ: 209459114 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) iQEcBAEBAgAGBQJM2OXtAAoJENm5axHh7AcxaTkH/1vRYAGezEM61TfqlXZyTbjf G8R9Y5NSKnmp5ATAdLlU7tBpB5w+hopFmfCwD8V1RCKSt3NdmkrPPb/JA/prKWRX c6LOXKMIzKrjgEykw3syyj6Gv3IEAbXOc09+SlMOvThT+fO8+SI+ld7Zlv9rmoH2 QcCUwRU1dcbL/eW93L7yQPgb5N7rhp/a7MJsBXVmKDEEBG51nshIgtlBISrzNhW3 nnMKMVVgrwCHFFBWYPVgTyvFB4iDRz7PfHGxl4R7UI2dI6GBmoWLwLfCFJacoGkq cnyI2ZyKH5ULAnEuq6zeKnLHotiJBSLCbhTXtRHVY9shx4fT7+2EIho7D/CikTM= =hCnQ -----END PGP SIGNATURE----- From allanwegan@REDACTED Tue Nov 9 07:34:12 2010 From: allanwegan@REDACTED (Allan Wegan) Date: Tue, 09 Nov 2010 07:34:12 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: References: <4CD8646E.4070106@allanwegan.de> Message-ID: <4CD8EB64.6040107@allanwegan.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > From what I saw, the usage is in regular literal strings and in > comments. Standard list based string literals of course do not work. Each byte of a UTF-8 encoded string becomes a separate integer in the resulting list. Simply using UTF-8 without changing the parsing process does only work with binary string literals. For binaries it does not matter whether the parser treats code points encoded in multiple bytes as a whole or as separate bytes. - -- Allan Wegan Jabber: allanwegan@REDACTED ICQ: 209459114 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (MingW32) iQEcBAEBAgAGBQJM2OtjAAoJENm5axHh7AcxmW8H/jJjfaXaF0xIXDwumG7ahosA BgEtYpGlaXsjTVPWXip0cDxg8c/jPrNHUqSsMx4bq7BVAVYH/QemaxEkS2QKHBzm rerFewlFKfsWVTDdfvTu3g3jDdHk7ICGcZvLtsFEZY2woXc4hSukivCa9fCQo4ND EzScAtRka2tyv37bsv/MtbgpknUASQyL51RCxXs9jIfWg8W8jE9PJEyEYGKUB1an cLI1Lpy/D5R5XN0vA8BvtgW7PV4kJdG8/oYSiWv3mBgmd0Oe6ceXwhtwnO1FGjCu xa6n04nSUTW2rMqvjNMvCE38+PkX5twueURe2Vo2Bx/mFGWH2W3F5pwqoV25ILc= =tPpJ -----END PGP SIGNATURE----- From kuleshovmail@REDACTED Tue Nov 9 08:00:38 2010 From: kuleshovmail@REDACTED (shk) Date: Mon, 8 Nov 2010 23:00:38 -0800 (PST) Subject: Erlang mail server Message-ID: <1289286038500-3033192.post@n4.nabble.com> Hello, I need simple erlang mail server example. I find erlmail, but i don't find how to setting it. Where can i find erlang mail server and instructions how to run, test, bind domain name to it,? Thank you. -- View this message in context: http://erlang.2086793.n4.nabble.com/Erlang-mail-server-tp3033192p3033192.html Sent from the Erlang Questions mailing list archive at Nabble.com. From zabrane3@REDACTED Tue Nov 9 08:09:33 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 9 Nov 2010 08:09:33 +0100 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: Message-ID: http://cean.process-one.net/downloads/ 2010/11/9 Icarus Alive : > On Tue, Nov 9, 2010 at 7:33 AM, Richard O'Keefe wrote: >> >> On 8/11/2010, at 7:00 AM, Icarus Alive wrote: >> >>> wondering as to what is the reason that Erlang/OTP binaries are >>> available only for win32 and not for Linux (and it's variants), BSD >>> etc. ? >> >> Let's see. ?I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, >> Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos >> 64-bit, and Fedora. ?Just how many sets of binaries do you expect to be >> provided? ?Then there are some older machines that would be fun to set >> up, several PowerPC boxes and a couple of SPARCs. ?How many sets of >> binaries? > > Am new to the whole Erlang ecosystem (language, runtime, community, > resources), so pardon me if this the expectation seemed excessive. For > one single person building/maintaining so many variants is definitely > a herculean task, and I wasn't aware that it was so. My expectations > were set by distribution of Python, Java, where the latest/greatest (& > older) versions are available for most commonly used OS's... the > mainstream Linux distros, FreeBSD, Solaris, Windows etc. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Regards Zabrane From vladdu55@REDACTED Tue Nov 9 08:19:46 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Tue, 9 Nov 2010 08:19:46 +0100 Subject: [erlang-questions] utf8 in source files In-Reply-To: <4CD8EB64.6040107@allanwegan.de> References: <4CD8646E.4070106@allanwegan.de> <4CD8EB64.6040107@allanwegan.de> Message-ID: Hi, On Tue, Nov 9, 2010 at 07:34, Allan Wegan wrote: >> From what I saw, the usage is in regular literal strings and in >> comments. > > Standard list based string literals of course do not work. Each byte of > a UTF-8 encoded string becomes a separate integer in the resulting list. > > Simply using UTF-8 without changing the parsing process does only work > with binary string literals. For binaries it does not matter whether the > parser treats code points encoded in multiple bytes as a whole or as > separate bytes. Yes and no. If one only uses the string as a message to be output on the teerminal and the terminal supports utf-8, then it works. Erlang will see a weird-looking string, but doesn't care. The terminal recognizes utf-8 and prints the right character. regards, Vlad From icarus.alive@REDACTED Tue Nov 9 08:28:09 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Tue, 9 Nov 2010 12:58:09 +0530 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: Message-ID: Thanks Zabrane. Very impressive. A link to CEAN should be there on the Erlang.org site, somewhere under the Download section. However, I see that CEAN is still at R12B. On Tue, Nov 9, 2010 at 12:39 PM, zabrane Mikael wrote: > http://cean.process-one.net/downloads/ > > 2010/11/9 Icarus Alive : >> On Tue, Nov 9, 2010 at 7:33 AM, Richard O'Keefe wrote: >>> >>> On 8/11/2010, at 7:00 AM, Icarus Alive wrote: >>> >>>> wondering as to what is the reason that Erlang/OTP binaries are >>>> available only for win32 and not for Linux (and it's variants), BSD >>>> etc. ? >>> >>> Let's see. ?I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, >>> Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos >>> 64-bit, and Fedora. ?Just how many sets of binaries do you expect to be >>> provided? ?Then there are some older machines that would be fun to set >>> up, several PowerPC boxes and a couple of SPARCs. ?How many sets of >>> binaries? >> >> Am new to the whole Erlang ecosystem (language, runtime, community, >> resources), so pardon me if this the expectation seemed excessive. For >> one single person building/maintaining so many variants is definitely >> a herculean task, and I wasn't aware that it was so. My expectations >> were set by distribution of Python, Java, where the latest/greatest (& >> older) versions are available for most commonly used OS's... the >> mainstream Linux distros, FreeBSD, Solaris, Windows etc. >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > > -- > Regards > Zabrane > From rtrlists@REDACTED Tue Nov 9 10:35:00 2010 From: rtrlists@REDACTED (Robert Raschke) Date: Tue, 9 Nov 2010 09:35:00 +0000 Subject: [erlang-questions] Erlang mail server In-Reply-To: <1289286038500-3033192.post@n4.nabble.com> References: <1289286038500-3033192.post@n4.nabble.com> Message-ID: On Tue, Nov 9, 2010 at 7:00 AM, shk wrote: > I need simple erlang mail server example. I find erlmail, but i don't find > how to setting it. > > Where can i find erlang mail server and instructions how to run, test, bind > domain name to it,? > > I don't think you'll find a full blown email server. A good starting point is probably the gen_smtp module. There appear to be several incarnations of that (http://www.google.com/search?q=gen_smtp). Robby From rvg@REDACTED Tue Nov 9 10:43:14 2010 From: rvg@REDACTED (Rudolph van Graan) Date: Tue, 9 Nov 2010 09:43:14 +0000 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: > One of the core concepts in relational systems is that of > enforced integrity constraints, via primary keys and foreign keys. > If you don't have any integrity constraints that you care to tell the > data base about, you probably don't need a relational system. This was exactly what I meant with "invariance" earlier. If you have (strong) requirements for invariance in your model - an RDBMS is almost the only solution as integrity constraints etc are all first class concepts. If you don't have a requirement for invariance (as in Edmund's example): > It is important to understand that SQL is not a good example of a relational system. Yes - SQL is a functional language where you state the solution in terms of relational concepts and ask the RDMBS to solve it and present you with an answer. Some of the Non-SQL systems uses search terms "Name = 'John'" or jscript for the same purpose. QLC is also an example (in the case of mnesia). > I think a big reason kv-stores are winning over a lot of us long-time RDBMSs users is they allow us to model things-that-have-things-inside-them in the database much closer to how they are modeled in our applications. Orders/receipts/invoices with their items, users with their permissions, all these map nicely in kv-stores from db to application. This allows us to model relationships only when WE REALLY WANT RELATIONSHIPS (this receipt belongs to that invoice). That fact alone won me over and I've never looked back. However, it is only simple to store things this way (the example of Orders/Receipts/Invoices with items "inside" them), if your only interest in the data is that the "outer" or container object encapsulates the items. Typically you want to read or write the whole thing in one operation. In real life, (and in my experience), you will pretty soon find that someone wants to know how many orders during the last 60 days included item X with quantities larger than say 6. If your design decision was to store this whole thing (the order and its items) as one big document (my term for it), the only way to retrieve this data is to literally open up every order, filter out the items you want and aggregate them. The only way to make this fast is to avoid reading every single document and processing it over and over. And to do this optimisation, you need an index, or more likely several indices - on dates, on items types, etc. Indices require that you understand what is inside your "document" (in this case line items). By definition, this implies a relationship - orders have among other things - lines. Completely independent of the fact that you are storing the items inside the document/order. So as a summary from my side - all data has some sort of structure, be it words within documents, or line items within orders. You can represent this any way you want. In the distant past we wrote all the items on a single piece of paper called an order. It was all on one physical page. The page contained all the information. For the same reason it is difficult to query pieces of paper (you need to either index them or summarise them in another way), in the same way it is difficult to query data with implied relations stored in a single "thing" (blob/object/values). - It is very difficult to enforce invariance in KV stores - It is very difficult to index KV stores - It is hard work to query KV stores. - It is trivial to read from or write into KV stores - It is hard to read from or write to database (drivers, SQL, ...) - RDMBS systems are hard to scale - KV stores scale easily Rudolph van Graan On Nov 9, 2010, at 1:59 AM, Richard O'Keefe wrote: > > On 8/11/2010, at 4:06 AM, Steve Davis wrote: >> It appears to me that this discussion is another expression of the >> 'strong vs weak/dynamic vs static type' discussion. >> >> ...it makes me suspect that an imperative and strongly-typed language >> paradigm has been a very strong motivator in the evolution of SQL >> databases; and perhaps the popularity of NoSQL/NotSQL is an expression/ >> outcome of the rise of recent trends in programming language uptake. > > You *cannot* call the types in classic SQL "strong". > Numbers, strings, and byte strings for everything is what Joe is complaining > of and he is right to do so. Encoding something as a string or blob basically > results in the data base itself having no clue about the structure or meaning > of the data. > > It is important to understand that SQL is not a good example of a relational > system. A move away from SQL *could* be a move towards relational! > such as > One of the core concepts in relational systems is that of > enforced integrity constraints, via primary keys and foreign keys. > If you don't have any integrity constraints that you care to tell the > data base about, you probably don't need a relational system. > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3822 bytes Desc: not available URL: From ebegumisa@REDACTED Tue Nov 9 11:54:09 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 09 Nov 2010 21:54:09 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: <26517.1289255374@snookles.snookles.com> References: <26517.1289255374@snookles.snookles.com> Message-ID: On Tue, 09 Nov 2010 09:29:34 +1100, Scott Lystig Fritchie wrote: > Regarding the labels of "key-value database" vs. "document-oriented > database", I typically think of KV DBs as considering their value blobs > to be opaque and doc-oriented DBs as understanding all (or at least > part) of the structure of their values. > * Hibari is KV > * Riak is both (the lower parts of the KV store treat blobs as > opaque, but both the map-reduce KV framework and the new Riak > Search app can parse the value blobs as JSON, Erlang terms, or > whatever a Search text analyzer decides to parse) > * CouchDB and MongoDB are doc-oriented Hmmmm, I guess that's one way of interpreting the terminology. I'd argue that within the *context* of the original post "key-value databases for RDBMS users", one would be talking more about stores like CouchDB, MongoDB, Mnesia. I think that the primary concern for a SQL-RDBMS user like Silas (and myself a while back), looking for an alternative datastore would be some reasonably powerful querying so one can ask the kind of complex questions from the data that any decent SQL-RDBMS could answer. This would necessitate a query engine that can facilitate peering into the structures being stored, otherwise I wouldn't see the point in migrating. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Tue Nov 9 12:23:33 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 09 Nov 2010 22:23:33 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: On Tue, 09 Nov 2010 12:59:11 +1100, Richard O'Keefe wrote: > It is important to understand that SQL is not a good example of a > relational > system. A move away from SQL *could* be a move towards relational! I see what you mean. However, most RDBMSs today (if there is such a thing) are accessed via SQL interfaces. So there is a tendency to use the terms SQL and RDBMS (wrongly) interchangeably. I think SQL-RDBMS is the best description of this class of tabular DB that seems to dominate the sector. Optional integrity-checks/primary/foreign-keys with an SQL interface on top. > One of the core concepts in relational systems is that of > enforced integrity constraints, via primary keys and foreign keys. > If you don't have any integrity constraints that you care to tell the > data base about, you probably don't need a relational system. I agree. In fact, this has been one of my main issues with SQL-RDBMSs: many times one is forced to create 1:N, 1:1, N:N relationships (using integrity constraints, primary keys, foreign keys) or simulate them with JOINS, to do things that are not really modeling related data from the eyes of the application (relationship bloat). The concept is stretched too far. The nice thing about this increasingly popular other class of database (Couch/Mnesia/Mongo), is that you have more control over this bloat and can choose when and how you want relationships. This normally comes at the cost of not having enforced constraints. In my view, the trade off is usually worth it. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Tue Nov 9 14:38:37 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 10 Nov 2010 00:38:37 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: Hello Rudolph, Some comments... On Tue, 09 Nov 2010 20:43:14 +1100, Rudolph van Graan wrote: > >> One of the core concepts in relational systems is that of >> enforced integrity constraints, via primary keys and foreign keys. >> If you don't have any integrity constraints that you care to tell the >> data base about, you probably don't need a relational system. > > This was exactly what I meant with "invariance" earlier. If you have > (strong) requirements for invariance in your model - an RDBMS is almost > the only solution as integrity constraints etc are all first class > concepts. If you don't have a requirement for invariance (as in Edmund's > example): > In many cases (at least for a migrating RDBMS user), invariance is *usually* indeed very important. And as you pointed out earlier there are ways of faking this in the emerging class of NoSQL/kv-db, the absence of which means tougher checks on code that's writing data (the db is unlikely to do this for you.) I've found that when one is making the transition (from SQL-RDBMS to NoSQL/Kv-db), the benefits of easier mapping of db-object to application-object normally out-weight the loss of things like constraints. I guess you have to trust more that programmers know what they are doing and won't write bad data! Some people cringe when hearing this, but I've found it's as bad as it sounds! However, I think as this newer class of DB start to attract more and more RDBMS migrants, they will have to figure out a ways of solving these issues. Some have already stared (CouchDB has some concept of data validation.) >> It is important to understand that SQL is not a good example of a >> relational system. > > Yes - SQL is a functional language where you state the solution in terms > of relational concepts and ask the RDMBS to solve it and present you > with an answer. Some of the Non-SQL systems uses search terms "Name = > 'John'" or jscript for the same purpose. QLC is also an example (in the > case of mnesia). > I'd go further and say SQL has evolved into a primary API for accessing todays RDBMSs. You can do much more that just query for data. You can design/redesign the db, add/remove indicies. Many vendors add elaborate extensions like Microsoft's Data Shaping for hierarchical data. In many ways SQL is like the non-standard standard shell language for many databases. >> I think a big reason kv-stores are winning over a lot of us long-time >> RDBMSs users is they allow us to model >> things-that-have-things-inside-them in the database much closer to how >> they are modeled in our applications. Orders/receipts/invoices with >> their items, users with their permissions, all these map nicely in >> kv-stores from db to application. This allows us to model relationships >> only when WE REALLY WANT RELATIONSHIPS (this receipt belongs to that >> invoice). That fact alone won me over and I've never looked back. > > However, it is only simple to store things this way (the example of > Orders/Receipts/Invoices with items "inside" them), I disagree, I store things this way not because it's simple for the database but because that's how my applications understand those things (BTW, those were *REAL* examples I was using orders/receipts/invoices from real code in development. Code previously written against SQL-RDBMSs and being re-writing against CouchDB. The latter is proving to be a much more pleasant experience!) I've seen many SQL-RDBMS developers try to achieve a similar end-result using object-relational mapping. You know, so when they create an instance of a user class in say Java, a new user is automatically added the users table in the db and corresponding entries are made in the permissions table. I've never agreed with that way of doing things but it *does* illustrate that there is a disconnect between how developers want to organise data and how SQL-RDBMSs want them organise data. > if your only interest in the data is that the "outer" or container > object encapsulates the items. This isn't true. At least not with the class of kv-database I'm referring to. CouchDB (and I assume Mnesia) allows you to access the 'inner' objects (to whatever depth) inside your queries. You can very easily access the items inside an invoice. > Typically you want to read or write the whole thing in one operation. In > real life, (and in my experience), you will pretty soon find that > someone wants to know how many orders during the last 60 days included > item X with quantities larger than say 6. > Easy. I do that sort of thing everywhere (no accounting tool would be very helpful if it couldn't do those kind of complex queries). I don't know about query engines for other kv-dbs like Mnesia, but CouchDB handles that sort of thing very well. Just create a view that gathers the inner data according to the criteria you are looking for (map step) then do the necessary tallying (reduce step). > If your design decision was to store this whole thing (the order and its > items) as one big document (my term for it), the only way to retrieve > this data is to literally open up every order, filter out the items you > want and aggregate them. The only way to make this fast is to avoid > reading every single document and processing it over and over. Couch is very smart at ensuring that any lengthy read/map-reduce operations are cached in an internal data structure (B-tree I think) and tries it's best to keep the cache update-to-date. So you normally don't have to worry too much about the whole vs partial read/write operations -- you just get on with business of writing your application using whatever structures make sense to you. But if the query is one that is rarely executed and lengthly, you can run it at startup so the next time it runs it's faster. This is one of the things I mean when I say we have to make clear the class of kv-store that the original poster should look into. An RDBMS user looking for a suitable alternative isn't looking for just a plain kv-store. You're normally looking for something that has a nice well thought-out query engine on top and has figured a lot of these issues out. The Couch team have done an *excellent* job here. > And to do this optimisation, you need an index, or more likely several > indices - on dates, on items types, etc. Correct. For couch's case, when you're creating a query (a view in couch lingo) what you're essentially doing at the map step is asking couch to create an index for you. Couch tries it's best to be very smart at how it handles/optimises that index. > Indices require that you understand what is inside your "document" (in > this case line items). Correct. You can look at couch's map step as providing the query engine an understanding of the documents being stored, including any contained or deeply hierarchical* data. *YAY! I've _ALWAYS_ wanted this and tried desperately to simulate it in SQL-RDBMSs with varying degrees of success. > By definition, this implies a relationship - orders have among other > things - lines. Completely independent of the fact that you are storing > the items inside the document/order. > Okay, I see what you're saying here. Indeed... http://wiki.apache.org/couchdb/EntityRelationship ... you can view embedded data as related data. I hadn't thought of it that way. Let me make my argument clear by referring to what SQL-RDBMS folk tend to think of as a relationship -- that is -- a constraint based on a foreign-key. The kv-db equivalent of this would be a member of the document that is merely a key pointing to another document rather than meaningful data in it's own right. Now, using *that* definition... For Couch's case, the map-reduce paradigm can be effectively used so that you *DON'T HAVE TO CREATE RELATIONSHIPS (of the external key variety) IF YOU DON'T WANT THEM* This is my major argument against SQL-RDMBSs and in favor of NoSQL/kv-dbs like Couch -- give me containers when I want containers and give me relationships when I specifically want relationships. I don't want my orders to *relate* to lines elsewhere, this is not how my application looks at orders. I want orders to *contain* items. On the other-hand, I want invoices to *relate* to orders. Only in the latter case does it make sense for me to have and foreign-key/id reference stored in the orders. I want data in my db to look like data in my application. The db should fit my application, my application shouldn't have to fit the db. I always found I would invest considerable time with SQL-RDBMSs, first trying to force the db to understand how I want my data organised, then giving up and just adapting the code of my app to think like the db -- in terms of tables and foreign keys. I've found NoSQL dbs like Couch treat me more like a grown up. Give me the freedom to define and enforce my own rules in a way that makes sense to me not in some generic way the *supposedly* adapts to every situation. Sure, the RDBMS style may adapt to a wide array of situations, but from my experience, this is never without pain. > So as a summary from my side - all data has some sort of structure, be > it words within documents, or line items within orders. You can > represent this any way you want. > True, but with most SQL-RDBMSs, things can get particularly hard. Aside from my relationship rant, a lot of data just doesn't want to be stored in tables, and when you force it, exponential complexity arises. > In the distant past we wrote all the items on a single piece of paper > called an order. It was all on one physical page. The page contained all > the information. For the same reason it is difficult to query pieces of > paper (you need to either index them or summarise them in another way), > in the same way it is difficult to query data with implied relations > stored in a single "thing" (blob/object/values). > > - It is very difficult to enforce invariance in KV stores I agree. And constraints in general are painful. > - It is very difficult to index KV stores This depends. For couch, map-reduce is a form of indexing -- a very effective and surprisingly flexible one at that. Doors that are normally closed in SQL-RDBMS world are suddenly opened wide in Couch. > - It is hard work to query KV stores. I disagree. I haven't found a query for *my* data yet that I cannot express in a couch view. I'd go further and say I've found it easier than SQL because I can organise data in a way that makes more sense to my application. This somehow always translates to easier query construction for the application domain. Perhaps it's a question of learning curve. SQL-RDBMS users are so used to thinking in terms of SELECTS and JOINS, but that doesn't make SQL easy! It just makes it easy once you know how. I think the same applies to many kv-dbs. > - It is trivial to read from or write into KV stores > - It is hard to read from or write to database (drivers, SQL, ...) > - RDMBS systems are hard to scale I agree with all. > - KV stores scale easily Largely true. But sometimes it's not as easy as they promise. Some things work well locally and break when you replicate. And you only realise this when you try and scale out. There is some element of false advertising when NoSQL folk promise simple scaling, there should be a big red disclaimer attached. - Edmond - > Rudolph van Graan > > > On Nov 9, 2010, at 1:59 AM, Richard O'Keefe wrote: > >> >> On 8/11/2010, at 4:06 AM, Steve Davis wrote: >>> It appears to me that this discussion is another expression of the >>> 'strong vs weak/dynamic vs static type' discussion. >>> >>> ...it makes me suspect that an imperative and strongly-typed language >>> paradigm has been a very strong motivator in the evolution of SQL >>> databases; and perhaps the popularity of NoSQL/NotSQL is an expression/ >>> outcome of the rise of recent trends in programming language uptake. >> >> You *cannot* call the types in classic SQL "strong". >> Numbers, strings, and byte strings for everything is what Joe is >> complaining >> of and he is right to do so. Encoding something as a string or blob >> basically >> results in the data base itself having no clue about the structure or >> meaning >> of the data. >> >> It is important to understand that SQL is not a good example of a >> relational >> system. A move away from SQL *could* be a move towards relational! >> such as >> One of the core concepts in relational systems is that of >> enforced integrity constraints, via primary keys and foreign keys. >> If you don't have any integrity constraints that you care to tell the >> data base about, you probably don't need a relational system. >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From steven.charles.davis@REDACTED Tue Nov 9 15:07:40 2010 From: steven.charles.davis@REDACTED (Steve Davis) Date: Tue, 09 Nov 2010 08:07:40 -0600 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users Message-ID: <4CD955AC.1060309@gmail.com> I'm not sure if you picked up on my point. You seem to have focused on the "native SQL types". The "strong types" I'm talking about are the complex data structures mapped across tables using referential constraints to map relationships (i.e. ORM). regs, Steve On 08/11/2010 19:59, Richard O'Keefe wrote: > > On 8/11/2010, at 4:06 AM, Steve Davis wrote: >> It appears to me that this discussion is another expression of the >> 'strong vs weak/dynamic vs static type' discussion. >> >> ...it makes me suspect that an imperative and strongly-typed language >> paradigm has been a very strong motivator in the evolution of SQL >> databases; and perhaps the popularity of NoSQL/NotSQL is an expression/ >> outcome of the rise of recent trends in programming language uptake. > > You *cannot* call the types in classic SQL "strong". > Numbers, strings, and byte strings for everything is what Joe is complaining > of and he is right to do so. Encoding something as a string or blob basically > results in the data base itself having no clue about the structure or meaning > of the data. > > It is important to understand that SQL is not a good example of a relational > system. A move away from SQL *could* be a move towards relational! > > One of the core concepts in relational systems is that of > enforced integrity constraints, via primary keys and foreign keys. > If you don't have any integrity constraints that you care to tell the > data base about, you probably don't need a relational system. > > From ebegumisa@REDACTED Tue Nov 9 17:01:08 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 10 Nov 2010 03:01:08 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: Another observation... I think a lot of it has to do with conceptual difference between SQL-RDBMS and NoSQL/kv-db/free-structure in terms of how both camps view the *role* of the database. I've found that SQL-RDMBSs tend to think of the database as central to the design+development of an application. So naturally things must follow the db. My issue with this is that as an app developer, I found myself always begging and pleading with the database. The conversation would go something like... Me: "Please, please store this data for me so I can query it later." DB: "Not so fast. First, split that into three tables. Add a constraint here, foreign key there. Replace that with a type I can understand then I'll see what I can do. BTW, you'll have to have another conversation with my bro, the SQL query engine later. Go on... Off you go!" The other camp seems to view the database role as more of a supplementary one in the design+development of an application. A helper. A tool. The conversation for me has gone more like... Me: "Oi, Couch! Come over here. Hang onto this for me. I'll need that back." DB: "Sure thing boss!" - Edmond - On Wed, 10 Nov 2010 00:38:37 +1100, Edmond Begumisa wrote: > Hello Rudolph, > > Some comments... > > On Tue, 09 Nov 2010 20:43:14 +1100, Rudolph van Graan > wrote: > >> >>> One of the core concepts in relational systems is that of >>> enforced integrity constraints, via primary keys and foreign keys. >>> If you don't have any integrity constraints that you care to tell the >>> data base about, you probably don't need a relational system. >> >> This was exactly what I meant with "invariance" earlier. If you have >> (strong) requirements for invariance in your model - an RDBMS is almost >> the only solution as integrity constraints etc are all first class >> concepts. If you don't have a requirement for invariance (as in >> Edmund's example): >> > > In many cases (at least for a migrating RDBMS user), invariance is > *usually* indeed very important. And as you pointed out earlier there > are ways of faking this in the emerging class of NoSQL/kv-db, the > absence of which means tougher checks on code that's writing data (the > db is unlikely to do this for you.) > > I've found that when one is making the transition (from SQL-RDBMS to > NoSQL/Kv-db), the benefits of easier mapping of db-object to > application-object normally out-weight the loss of things like > constraints. I guess you have to trust more that programmers know what > they are doing and won't write bad data! Some people cringe when hearing > this, but I've found it's as bad as it sounds! > > However, I think as this newer class of DB start to attract more and > more RDBMS migrants, they will have to figure out a ways of solving > these issues. Some have already stared (CouchDB has some concept of data > validation.) > >>> It is important to understand that SQL is not a good example of a >>> relational system. >> >> Yes - SQL is a functional language where you state the solution in >> terms of relational concepts and ask the RDMBS to solve it and present >> you with an answer. Some of the Non-SQL systems uses search terms "Name >> = 'John'" or jscript for the same purpose. QLC is also an example (in >> the case of mnesia). >> > > I'd go further and say SQL has evolved into a primary API for accessing > todays RDBMSs. You can do much more that just query for data. You can > design/redesign the db, add/remove indicies. Many vendors add elaborate > extensions like Microsoft's Data Shaping for hierarchical data. In many > ways SQL is like the non-standard standard shell language for many > databases. > >>> I think a big reason kv-stores are winning over a lot of us long-time >>> RDBMSs users is they allow us to model >>> things-that-have-things-inside-them in the database much closer to how >>> they are modeled in our applications. Orders/receipts/invoices with >>> their items, users with their permissions, all these map nicely in >>> kv-stores from db to application. This allows us to model >>> relationships only when WE REALLY WANT RELATIONSHIPS (this receipt >>> belongs to that invoice). That fact alone won me over and I've never >>> looked back. >> >> However, it is only simple to store things this way (the example of >> Orders/Receipts/Invoices with items "inside" them), > > I disagree, I store things this way not because it's simple for the > database but because that's how my applications understand those things > (BTW, those were *REAL* examples I was using orders/receipts/invoices > from real code in development. Code previously written against > SQL-RDBMSs and being re-writing against CouchDB. The latter is proving > to be a much more pleasant experience!) > > I've seen many SQL-RDBMS developers try to achieve a similar end-result > using object-relational mapping. You know, so when they create an > instance of a user class in say Java, a new user is automatically added > the users table in the db and corresponding entries are made in the > permissions table. I've never agreed with that way of doing things but > it *does* illustrate that there is a disconnect between how developers > want to organise data and how SQL-RDBMSs want them organise data. > >> if your only interest in the data is that the "outer" or container >> object encapsulates the items. > > This isn't true. At least not with the class of kv-database I'm > referring to. CouchDB (and I assume Mnesia) allows you to access the > 'inner' objects (to whatever depth) inside your queries. You can very > easily access the items inside an invoice. > >> Typically you want to read or write the whole thing in one operation. >> In real life, (and in my experience), you will pretty soon find that >> someone wants to know how many orders during the last 60 days included >> item X with quantities larger than say 6. >> > > Easy. I do that sort of thing everywhere (no accounting tool would be > very helpful if it couldn't do those kind of complex queries). > > I don't know about query engines for other kv-dbs like Mnesia, but > CouchDB handles that sort of thing very well. Just create a view that > gathers the inner data according to the criteria you are looking for > (map step) then do the necessary tallying (reduce step). > >> If your design decision was to store this whole thing (the order and >> its items) as one big document (my term for it), the only way to >> retrieve this data is to literally open up every order, filter out the >> items you want and aggregate them. The only way to make this fast is to >> avoid reading every single document and processing it over and over. > > Couch is very smart at ensuring that any lengthy read/map-reduce > operations are cached in an internal data structure (B-tree I think) and > tries it's best to keep the cache update-to-date. So you normally don't > have to worry too much about the whole vs partial read/write operations > -- you just get on with business of writing your application using > whatever structures make sense to you. But if the query is one that is > rarely executed and lengthly, you can run it at startup so the next time > it runs it's faster. > > This is one of the things I mean when I say we have to make clear the > class of kv-store that the original poster should look into. An RDBMS > user looking for a suitable alternative isn't looking for just a plain > kv-store. You're normally looking for something that has a nice well > thought-out query engine on top and has figured a lot of these issues > out. The Couch team have done an *excellent* job here. > >> And to do this optimisation, you need an index, or more likely several >> indices - on dates, on items types, etc. > > Correct. For couch's case, when you're creating a query (a view in couch > lingo) what you're essentially doing at the map step is asking couch to > create an index for you. Couch tries it's best to be very smart at how > it handles/optimises that index. > >> Indices require that you understand what is inside your "document" (in >> this case line items). > > Correct. You can look at couch's map step as providing the query engine > an understanding of the documents being stored, including any contained > or deeply hierarchical* data. > > *YAY! I've _ALWAYS_ wanted this and tried desperately to simulate it in > SQL-RDBMSs with varying degrees of success. > >> By definition, this implies a relationship - orders have among other >> things - lines. Completely independent of the fact that you are storing >> the items inside the document/order. >> > > Okay, I see what you're saying here. Indeed... > > http://wiki.apache.org/couchdb/EntityRelationship > > ... you can view embedded data as related data. I hadn't thought of it > that way. > > Let me make my argument clear by referring to what SQL-RDBMS folk tend > to think of as a relationship -- that is -- a constraint based on a > foreign-key. The kv-db equivalent of this would be a member of the > document that is merely a key pointing to another document rather than > meaningful data in it's own right. Now, using *that* definition... > > For Couch's case, the map-reduce paradigm can be effectively used so > that you *DON'T HAVE TO CREATE RELATIONSHIPS (of the external key > variety) IF YOU DON'T WANT THEM* > > This is my major argument against SQL-RDMBSs and in favor of > NoSQL/kv-dbs like Couch -- give me containers when I want containers and > give me relationships when I specifically want relationships. > > I don't want my orders to *relate* to lines elsewhere, this is not how > my application looks at orders. I want orders to *contain* items. On the > other-hand, I want invoices to *relate* to orders. Only in the latter > case does it make sense for me to have and foreign-key/id reference > stored in the orders. I want data in my db to look like data in my > application. > > The db should fit my application, my application shouldn't have to fit > the db. I always found I would invest considerable time with SQL-RDBMSs, > first trying to force the db to understand how I want my data organised, > then giving up and just adapting the code of my app to think like the db > -- in terms of tables and foreign keys. > > I've found NoSQL dbs like Couch treat me more like a grown up. Give me > the freedom to define and enforce my own rules in a way that makes sense > to me not in some generic way the *supposedly* adapts to every > situation. Sure, the RDBMS style may adapt to a wide array of > situations, but from my experience, this is never without pain. > >> So as a summary from my side - all data has some sort of structure, be >> it words within documents, or line items within orders. You can >> represent this any way you want. >> > > True, but with most SQL-RDBMSs, things can get particularly hard. Aside > from my relationship rant, a lot of data just doesn't want to be stored > in tables, and when you force it, exponential complexity arises. > >> In the distant past we wrote all the items on a single piece of paper >> called an order. It was all on one physical page. The page contained >> all the information. For the same reason it is difficult to query >> pieces of paper (you need to either index them or summarise them in >> another way), in the same way it is difficult to query data with >> implied relations stored in a single "thing" (blob/object/values). >> >> - It is very difficult to enforce invariance in KV stores > > I agree. And constraints in general are painful. > >> - It is very difficult to index KV stores > > This depends. For couch, map-reduce is a form of indexing -- a very > effective and surprisingly flexible one at that. Doors that are normally > closed in SQL-RDBMS world are suddenly opened wide in Couch. > >> - It is hard work to query KV stores. > > I disagree. I haven't found a query for *my* data yet that I cannot > express in a couch view. I'd go further and say I've found it easier > than SQL because I can organise data in a way that makes more sense to > my application. This somehow always translates to easier query > construction for the application domain. > > Perhaps it's a question of learning curve. SQL-RDBMS users are so used > to thinking in terms of SELECTS and JOINS, but that doesn't make SQL > easy! It just makes it easy once you know how. I think the same applies > to many kv-dbs. > >> - It is trivial to read from or write into KV stores >> - It is hard to read from or write to database (drivers, SQL, ...) >> - RDMBS systems are hard to scale > > I agree with all. > >> - KV stores scale easily > > Largely true. But sometimes it's not as easy as they promise. Some > things work well locally and break when you replicate. And you only > realise this when you try and scale out. There is some element of false > advertising when NoSQL folk promise simple scaling, there should be a > big red disclaimer attached. > > - Edmond - > > >> Rudolph van Graan >> >> >> On Nov 9, 2010, at 1:59 AM, Richard O'Keefe wrote: >> >>> >>> On 8/11/2010, at 4:06 AM, Steve Davis wrote: >>>> It appears to me that this discussion is another expression of the >>>> 'strong vs weak/dynamic vs static type' discussion. >>>> >>>> ...it makes me suspect that an imperative and strongly-typed language >>>> paradigm has been a very strong motivator in the evolution of SQL >>>> databases; and perhaps the popularity of NoSQL/NotSQL is an >>>> expression/ >>>> outcome of the rise of recent trends in programming language uptake. >>> >>> You *cannot* call the types in classic SQL "strong". >>> Numbers, strings, and byte strings for everything is what Joe is >>> complaining >>> of and he is right to do so. Encoding something as a string or blob >>> basically >>> results in the data base itself having no clue about the structure or >>> meaning >>> of the data. >>> >>> It is important to understand that SQL is not a good example of a >>> relational >>> system. A move away from SQL *could* be a move towards relational! >>> such as >>> One of the core concepts in relational systems is that of >>> enforced integrity constraints, via primary keys and foreign keys. >>> If you don't have any integrity constraints that you care to tell the >>> data base about, you probably don't need a relational system. >>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Tue Nov 9 22:15:02 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 10 Nov 2010 08:15:02 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: Sorry, I missed this bit... On Wed, 10 Nov 2010 01:36:46 +1100, wrote: > In the distant past we wrote all the items on a single piece of paper > called an order. It was all on one physical page. The page contained all > the information. I'm not advocating for moving towards data models that are more physical. I just want data models that are closer to my application's world-view. It just so happens that in this case, the model that make sense to my application is also close to the physical. Someone else with a different coding style might write the very same application completely differently. He should then have more freedom than SQL-RDBMSs typically gives for adapting the database to speak his language. > For the same reason it is difficult to query pieces of paper (you need > to either index them or summarise them in another way), in the same way > it is difficult to query data with implied relations stored in a single > "thing" (blob/object/values). > I really don't see the difference. Whether you query line items that are inside the document or outside the document the human work is the same. The data still has to be pulled out (when it's inside the document you do this later when you need it, when it's outside you do this before-hand EVEN THOUGH YOU MIGHT NOT NEED IT). Either way you have to write that bit only once. And either way it probalby needs to be indexed. My experience has been this: when I'm given the option of implied relations as you put them (I call these containers), I find my database design greatly simplified firstly because a good chunk of relationships devoted to containers are erased, and secondly because this is a concept I find I use a lot within my applications. Using the order example, from *my application view*, it pulls the document from the db and displays an order on the screen, lines items are added/edited are edited. The save button is pressed, I then write this to the db in one big doc as you describe it to the db. This happens to fit very nicely in the way the app is designed. I want it that way in the db *by design* With the SQL-RDBMS two-table solution to this, I'd have to force my code to think of this in two steps (when the address is edited thats one table, when a line is added thats another table, grrrrr!) I've gotten around this in the past using various techniques, hierarchical data modeling being the most advanced -- a structure that is aware that lines belong to the order and can have various levels of depth but that has always been a lot of work. The second stage, querying. My user wants all orders for the X-Box 360 before July 30. No problem, I'd have a couch view for this sort of thing. Yes, sure the view essentially goes through all the orders but... 1) this is exactly what the my app needs (it is what the user has asked for) 2) Couch is smart about how it caches this 3) a SQL-RDBMS would essentially do the same thing (it would just walk though several tables instead and cache/index) 4) Couch would give me more control over the indexing But here's the thing. You *don't* have to do it that way. That's just the way I usually approach it. I might be a terrible way. But it's *my* terrible way -- DB design constraints haven't forced things on me. With SQL-RDBMSs I always felt I was being nudged -- being told what was good for me. I never felt complete freedom. And I might change my mind when I find orders contain thousands of lines and do it some other way, maybe the equivalent to the two-table RDBMS solution. I just find the NoSQL/kv-db more flexible in allowing the application developer to choose how he/she wants to do things. If I want to hang myself give me rope. That's my business. You're just the db. Don't be too smart. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ok@REDACTED Tue Nov 9 23:04:38 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 10 Nov 2010 11:04:38 +1300 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: Message-ID: <1FC0CBB0-55BC-4E39-AC8E-BE9114C1131D@cs.otago.ac.nz> On 9/11/2010, at 6:36 PM, Icarus Alive wrote: >> Let's see. I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, >> Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos >> 64-bit, and Fedora. Just how many sets of binaries do you expect to be >> provided? Then there are some older machines that would be fun to set >> up, several PowerPC boxes and a couple of SPARCs. How many sets of >> binaries? > > Am new to the whole Erlang ecosystem (language, runtime, community, > resources), so pardon me if this the expectation seemed excessive. For > one single person building/maintaining so many variants is definitely > a herculean task, I didn't say that I *maintain* Erlang on such a range of machines, only that this is (a subset of) the range of machines available to me. The software that I am personally developing, I *do* port between those systems (and also Cygwin, but we were talking about Unix distributions). In the third-year software engineering class this year, only 1/4 of the groups managed to develop software that ported from Linux to Linux. One of the *major* issues was 32-bit vs 64-bit. People will tell you that oh, it's no problem, you can run 32-bit programs on 64-bit operating systems; what they _don't_ tell you is that this only works if *all* the required 32-bit .so files are present, and no, sometimes they are not. > and I wasn't aware that it was so. My expectations > were set by distribution of Python, Java, where the latest/greatest (& > older) versions are available for most commonly used OS's... the > mainstream Linux distros, FreeBSD, Solaris, Windows etc. OK, let's try a specific example: JDK 7. http://dlc.sun.com.edgesuite.net/jdk7/binaries/index.html What do we find? Windows intel 32-bit Windows intel 64-bit Solaris SPARC 32-bit (presumably s10) Solaris SPARC 64-bit (presumably s10) Solaris intel 32-bit (will it work under OpenSolaris?) Solaris intel 64-bit (will it work under OpenSolaris?) Linux intel 32-bit Linux intel 64-bit Do we find any version of Mac OS X? Do we find any version of *BSD? Do we find anything other than intel or SPARC (and would we find SPARC if Sun/Oracle didn't make SPARC systems)? No. Let's try another one. Let's make it a functional programming language that has been used to do exciting web-related stuff. http://wiki.clean.cs.ru.nl/Download_Clean What do we find? Windows intel 32-bit Windows intel 64-bit Linux intel 32-bit (no IDE) Linux intel 64-bit (no IDE) MacOS PPC 32-bit Do we find any version for SPARC? No. (A long time ago...) Do we find any version for Solaris? No. Do we find any version for *BSD? No. Do we find any version for Mac OS X? No. ("MacOS" above appears to mean MacOS 9.) When we download the "Windows and Mac complete sources", do we find any instructions about how to build the thing? Not that either. It turns out that instructions for building on Mac OS X *are* available on the Wiki for that project, since last month, but they are not complete, and the process (install a different version of gas, install another program to convert gas output to something the operating system understands, apply a lot of patches, manually visit one directory after another, section "TO DO", ..., building the compiler "TO DO", ... ... installing "TO DO") unpleasant. Erlang is a *dream* to install by contrast. > From andrew@REDACTED Tue Nov 9 23:35:44 2010 From: andrew@REDACTED (Andrew Thompson) Date: Tue, 9 Nov 2010 17:35:44 -0500 Subject: [erlang-questions] Erlang mail server In-Reply-To: References: <1289286038500-3033192.post@n4.nabble.com> Message-ID: <20101109223543.GJ2468@hijacked.us> On Tue, Nov 09, 2010 at 09:35:00AM +0000, Robert Raschke wrote: > On Tue, Nov 9, 2010 at 7:00 AM, shk wrote: > > > I need simple erlang mail server example. I find erlmail, but i don't find > > how to setting it. > > > > Where can i find erlang mail server and instructions how to run, test, bind > > domain name to it,? > > > > > I don't think you'll find a full blown email server. A good starting point > is probably the gen_smtp module. There appear to be several incarnations of > that (http://www.google.com/search?q=gen_smtp). Yeah, gen_smtp is the only thing that even comes close, to my (biased) knowledge, but its a SMTP/MIME toolkit, not a complete mailserver (but you could build one with it). You'd need to write some kind of mail storage and then expose the mail to the users somehow (IMAP/POP/mbox/maildir). Someday I'd like to write a distributed mailserver using gen_smtp that stores mail to riak or mnesia or something and lets you access your mail from any host in the cluster (and send from any of them). No chance of that happening any time soon, though, because I don't need such a thing for anything other than a toy. Andrew From gordon@REDACTED Tue Nov 9 23:42:13 2010 From: gordon@REDACTED (Gordon Guthrie) Date: Tue, 9 Nov 2010 22:42:13 +0000 Subject: Strage Message on q() from shell Message-ID: Folks I have been having intermittent problems where a new shell can't joint a cluster - although member of the cluster can all see each other. I have been brute forcing this by closing down the errant VM and restarting it. I need to force this from an attached shell and I do with with using q(). I just did it and got a message: write_ctrl: Bad write Anyone has a clue what this is? Cheers Gordon -- Gordon Guthrie CEO hypernumbers http://hypernumbers.com t: hypernumbers +44 7776 251669 From ok@REDACTED Wed Nov 10 00:23:50 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 10 Nov 2010 12:23:50 +1300 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: <534537B5-B65F-4090-B6FF-073F9B57DCD5@cs.otago.ac.nz> On 10/11/2010, at 5:01 AM, Edmond Begumisa wrote: > Another observation... > > I think a lot of it has to do with conceptual difference between SQL-RDBMS and NoSQL/kv-db/free-structure in terms of how both camps view the *role* of the database. This is precisely how Date and Darwen characterise the difference between the relational and OO data base camps: Relational: Applications come, applications go. But if anything goes wrong with the data, we're out of business. Object: Applications are the centre of the universe. Databases are just for persistent storage, with maybe some indexing. I'd point out that OO databases have been around for much longer than the modern set of NoSQL or K-V systems. Gemstone is still around, and Magma is pretty cool. Magma is really easy to use: objects in the data base materialise in your workspace when you look at them, modified objects get written back when you commit, and objects become known to the data base by being reachable from the root object. If what you need is persistent storage for a single application, and you don't expect the data to outlive the application, a relational database probably is not for you. I've known a student write thousands of lines of Java code talking to an SQL system in order to deal with data that could have been a simple flat file or two with far better efficiency (all the data would have fitted comfortably into memory). I've known another group of students insist on using SQL for an assignment where the explict point of the assignment was to develop an in-memory data structure (which for this application outperformed SQL a thousand-fold). From ebegumisa@REDACTED Wed Nov 10 02:20:38 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 10 Nov 2010 12:20:38 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: <534537B5-B65F-4090-B6FF-073F9B57DCD5@cs.otago.ac.nz> References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> <534537B5-B65F-4090-B6FF-073F9B57DCD5@cs.otago.ac.nz> Message-ID: On Wed, 10 Nov 2010 10:23:50 +1100, Richard O'Keefe wrote: > If what you need is persistent storage for a single application, > and you don't expect the data to outlive the application, a relational > database probably is not for you. Ditto! It has taken me 12 years as an SQL-RDBMS user to realise that the problem was I was using the wrong class of database the whole time! Silly, silly me. Square peg, round hole. (actually I started suspecting this about 5 years ago.) I wonder how many more silly me's there are out there? I think a major reason many assume the relational + sql solution as the default answer is not because they need relational constraints per se, but because they need complex querying for which it is assumed only SQL can provide (and the best SQL dbs seem to be relational). My requirement list was normally like this... 1. Persistent storage for my application that be _might_ later be shared with other applications, but I'm free to dictate how. 2. Complex querying preferably with indexing. 3. Data unlikely to outlive my application (if it needs to, it can be migrated). 4. Data won't fit in RAM (without hogging too much of it). 5. (NEW) Data can be replicated. For a long time I assumed only SQL-RDBMSs could give me these. I've found, like many, that I was wrong. In retrospect, a lot of it was do-as-others-do I guess. As you point out OO dbs could have given me many of these. Some other alternatives were there too. I discovered Btrieve about five years back and really liked it. I think what the NoSQL/kv-dbs like Couch are adding to their older OO counterparts is 1 (HTTP APIs for other apps), 2, & 5. It's interesting too to observe the increasing popularity of object-relational mapping layers for popular SQL-RDBMSs. Essentially trying to turn RDBMSs into something closer to OO dbs. Funny that! Round and round we go! - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From volerdo@REDACTED Wed Nov 10 07:19:54 2010 From: volerdo@REDACTED (=?ISO-8859-9?Q?=DE=2E_Volkan_Erdo=F0an?=) Date: Wed, 10 Nov 2010 08:19:54 +0200 Subject: About Garbage Collection Message-ID: Hi, I have an idea about memory management in Erlang and although the idea is still raw I'd like to share with you. As far as I know: - There are no global variables in Erlang. Memory is allocated in functions. - Function calls in processes and message passing operations to other processes have copying semantics. So I believe that these properties guarantee that any memory location allocated by runtime is exclusive to the function which requested that memory and it is not possible for a reference of a memory location to escape from the function that created it. And as a result it is possible to safely deallocate all memory allocated by the function at function exit since those memory locations can only be accessed by that function. However I think it is possible to do better because Erlang has single assignment property and one can easily find the last location that a variable is used and can safely deallocate memory after that location. Also we can relax copying semantics for some type of function calls and pass a reference to the functions since compiler prevents variable modification (I am not sure if runtime is already doing this). To enable these operations following modifications are needed to be done in Erlang runtime and compiler: - Erlang virtual machine will need a new instruction for memory deallocation operations (This operation will be called DEL) - The compiler should pass references for parameters instead of copying them during function calls After these modifications are done, the analysis algorithm described below can be executed for each function before that function is loaded to the virtual machine For each variable including the formal parameters of the function -Find last usage of the variable -If last usage of the parameter is in a function call -do nothing (since it will be analyzed or already analyzed in that function as formal parameter) -Else -Insert a DEL instruction for that variable just after the usage To sum up I believe the properties of the language lets an analyzer to exactly find the life times of variables created in functions (btw ETS tables might need special care). This is much like working with heap memory in C, only this time an external program is inserting free operations. From chandrashekhar.mullaparthi@REDACTED Wed Nov 10 07:29:23 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 10 Nov 2010 06:29:23 +0000 Subject: ANNOUNCE: ibrowse-2.1.0 Message-ID: ibrowse-2.1.0 has been released. ibrowse is a HTTP client written in erlang as an alternative to the built in HTTP client httpc. Get it from https://github.com/cmullaparthi/ibrowse/archives/v2.1.0 Release notes: * Fixed build on OpenSolaris. Bug report and patch from tholschuh. http://github.com/cmullaparthi/ibrowse/issues/issue/10 * Fixed behaviour of inactivity_timeout option. Reported by Jo?o Lopes. http://github.com/cmullaparthi/ibrowse/issues/issue/11 * Prevent atom table pollution when bogus URLs are input to ibrowse. Bug report by Jo?o Lopes. http://github.com/cmullaparthi/ibrowse/issues/issue/13 * Automatically do Chunked-Transfer encoding of request body when the body is generated by a fun. Patch provided by Filipe David Manana. http://github.com/cmullaparthi/ibrowse/issues/issue/14 * Depending on input options, ibrowse sometimes included multiple Content-Length headers. Bug reported by Paul J. Davis http://github.com/cmullaparthi/ibrowse/issues/issue/15 * Deal with webservers which do not provide a Reason-Phrase on the response Status-Line. Patch provided by Jeroen Koops. http://github.com/cmullaparthi/ibrowse/issues/issue/16 * Fixed http://github.com/cmullaparthi/ibrowse/issues/issue/17 This was reported by Filipe David Manana. * Fixed http://github.com/cmullaparthi/ibrowse/issues/issue/19 This was reported by Dan Kelley and Filipe David Manana. * Added ibrowse:stream_close/1 to close the connection associated with a certain response stream. Patch provided by Jo?o Lopes. * Prevent port number being included in the Host header when port 443 is intended. Bug reported by Andrew Tunnell-Jones Chandru From john.hughes@REDACTED Wed Nov 10 07:53:44 2010 From: john.hughes@REDACTED (John Hughes) Date: Wed, 10 Nov 2010 07:53:44 +0100 Subject: [erlang-questions] About Garbage Collection In-Reply-To: References: Message-ID: Hi Volkan, Message passing copies the message, but function arguments and results are not copied--they live on the local process heap. Hence memory cannot be deallocated on a return. John Sent from my iPhone On 10 Nov 2010, at 07:19, ?. Volkan Erdo?an wrote: > Hi, > I have an idea about memory management in Erlang and although the idea > is still raw I'd like to share with you. > > As far as I know: > - There are no global variables in Erlang. Memory is allocated in functions. > - Function calls in processes and message passing operations to other > processes have copying semantics. > > So I believe that these properties guarantee that any memory location > allocated by runtime is exclusive to the function which requested that > memory and it is not possible for a reference of a memory location to > escape from the function that created it. And as a result it is > possible to safely deallocate all memory allocated by the function at > function exit since those memory locations can only be accessed by > that function. However I think it is possible to do better because > Erlang has single assignment property and one can easily find the last > location that a variable is used and can safely deallocate memory > after that location. Also we can relax copying semantics for some type > of function calls and pass a reference to the functions since compiler > prevents variable modification (I am not sure if runtime is already > doing this). To enable these operations following modifications are > needed to be done in Erlang runtime and compiler: > - Erlang virtual machine will need a new instruction for memory > deallocation operations (This operation will be called DEL) > - The compiler should pass references for parameters instead of > copying them during function calls > > After these modifications are done, the analysis algorithm described > below can be executed for each function before that function is loaded > to the virtual machine > For each variable including the formal parameters of the function > -Find last usage of the variable > -If last usage of the parameter is in a function call > -do nothing (since it will be analyzed or already analyzed in that > function as formal parameter) > -Else > -Insert a DEL instruction for that variable just after the usage > > To sum up I believe the properties of the language lets an analyzer to > exactly find the life times of variables created in functions (btw ETS > tables might need special care). This is much like working with heap > memory in C, only this time an external program is inserting free > operations. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From max.lapshin@REDACTED Wed Nov 10 08:54:48 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 10 Nov 2010 10:54:48 +0300 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: Message-ID: Thank you a lot! Especially for merging stream_close Btw, stream_close was working only if stream was opened like {active, once}, not {active, true}. Have you looked at this issue? From xramtsov@REDACTED Wed Nov 10 09:05:34 2010 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Wed, 10 Nov 2010 17:05:34 +0900 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: Message-ID: <4CDA524E.4010103@gmail.com> 10.11.2010 15:29, Chandru wrote: > ibrowse-2.1.0 has been released. > > ibrowse is a HTTP client written in erlang as an alternative to the > built in HTTP client httpc. > > Get it from https://github.com/cmullaparthi/ibrowse/archives/v2.1.0 > My 5 cents about ibrowse: after switching it from standard httpc library, we found *significant* cpu consumption (almost twice). Our profiler shows that the bottle-neck is HTTP headers parsing. Is it possible to optimize that stuff? Probably switching to erlang:decode_packet makes sense. -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From max.lapshin@REDACTED Wed Nov 10 09:16:35 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 10 Nov 2010 11:16:35 +0300 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: <4CDA524E.4010103@gmail.com> References: <4CDA524E.4010103@gmail.com> Message-ID: > > My 5 cents about ibrowse: after switching it from standard httpc library, we > found *significant* cpu consumption (almost twice). Our profiler shows that > the bottle-neck is HTTP headers parsing. Is it possible to optimize that > stuff? Probably switching to erlang:decode_packet makes sense. What profiler do you use? From xramtsov@REDACTED Wed Nov 10 09:29:49 2010 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Wed, 10 Nov 2010 17:29:49 +0900 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: <4CDA524E.4010103@gmail.com> Message-ID: <4CDA57FD.3050009@gmail.com> 10.11.2010 17:16, Max Lapshin wrote: >> My 5 cents about ibrowse: after switching it from standard httpc library, we >> found *significant* cpu consumption (almost twice). Our profiler shows that >> the bottle-neck is HTTP headers parsing. Is it possible to optimize that >> stuff? Probably switching to erlang:decode_packet makes sense. >> > What profiler do you use? > > A wrapper for fprof. Is it so important? :) -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From ebegumisa@REDACTED Wed Nov 10 09:31:38 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 10 Nov 2010 19:31:38 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> <534537B5-B65F-4090-B6FF-073F9B57DCD5@cs.otago.ac.nz> Message-ID: Ooops, big, big correction. My 1st requirement should have been... 1. ACIDic transactional concurrent persistent storage for my application that be _might_ later be shared with other applications, but I'm free to dictate how. Very important distinction. - Edmond - On Wed, 10 Nov 2010 12:20:38 +1100, Edmond Begumisa wrote: > On Wed, 10 Nov 2010 10:23:50 +1100, Richard O'Keefe > wrote: > >> If what you need is persistent storage for a single application, >> and you don't expect the data to outlive the application, a relational >> database probably is not for you. > > Ditto! > > It has taken me 12 years as an SQL-RDBMS user to realise that the > problem was I was using the wrong class of database the whole time! > Silly, silly me. Square peg, round hole. (actually I started suspecting > this about 5 years ago.) I wonder how many more silly me's there are out > there? > > I think a major reason many assume the relational + sql solution as the > default answer is not because they need relational constraints per se, > but because they need complex querying for which it is assumed only SQL > can provide (and the best SQL dbs seem to be relational). My requirement > list was normally like this... > > 1. Persistent storage for my application that be _might_ later be shared > with other applications, but I'm free to dictate how. > 2. Complex querying preferably with indexing. > 3. Data unlikely to outlive my application (if it needs to, it can be > migrated). > 4. Data won't fit in RAM (without hogging too much of it). > 5. (NEW) Data can be replicated. > > For a long time I assumed only SQL-RDBMSs could give me these. I've > found, like many, that I was wrong. In retrospect, a lot of it was > do-as-others-do I guess. As you point out OO dbs could have given me > many of these. Some other alternatives were there too. I discovered > Btrieve about five years back and really liked it. > > I think what the NoSQL/kv-dbs like Couch are adding to their older OO > counterparts is 1 (HTTP APIs for other apps), 2, & 5. > > It's interesting too to observe the increasing popularity of > object-relational mapping layers for popular SQL-RDBMSs. Essentially > trying to turn RDBMSs into something closer to OO dbs. Funny that! Round > and round we go! > > - Edmond - > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From max.lapshin@REDACTED Wed Nov 10 09:32:58 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 10 Nov 2010 11:32:58 +0300 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: <4CDA57FD.3050009@gmail.com> References: <4CDA524E.4010103@gmail.com> <4CDA57FD.3050009@gmail.com> Message-ID: > > A wrapper for fprof. Is it so important? :) > Not in this thread, but yes, it is. Seems, that some of profilers are showing not how many time was consumed, but how many times function was called. From xramtsov@REDACTED Wed Nov 10 09:38:22 2010 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Wed, 10 Nov 2010 17:38:22 +0900 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: <4CDA524E.4010103@gmail.com> <4CDA57FD.3050009@gmail.com> Message-ID: <4CDA59FE.4090400@gmail.com> 10.11.2010 17:32, Max Lapshin wrote: >> A wrapper for fprof. Is it so important? :) >> >> > Not in this thread, but yes, it is. Seems, that some of profilers are > showing not how many time was consumed, but how many times function > was called. > > In fact, it doesn't matter, for instanse, why these functions are needed? ibrowse_http_client:parse_headers_1/3 10.60% ibrowse_http_client:get_crlf_crlf_pos/2 10.34% This can be done more efficiently using erlang:decode_packet, IMHO. -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From max.lapshin@REDACTED Wed Nov 10 09:40:47 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 10 Nov 2010 11:40:47 +0300 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: <4CDA59FE.4090400@gmail.com> References: <4CDA524E.4010103@gmail.com> <4CDA57FD.3050009@gmail.com> <4CDA59FE.4090400@gmail.com> Message-ID: > In fact, it doesn't matter, for instanse, why these functions are needed? > ibrowse_http_client:parse_headers_1/3 ? ? ? ? ? ? ? 10.60% > ibrowse_http_client:get_crlf_crlf_pos/2 ? ? ? ? ? ? 10.34% > > This can be done more efficiently using erlang:decode_packet, IMHO. > You are right. I think, I will try to fix it even myself, because erlyvideo is also migrating to ibrowse From chandrashekhar.mullaparthi@REDACTED Wed Nov 10 10:58:38 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 10 Nov 2010 09:58:38 +0000 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: <4CDA524E.4010103@gmail.com> References: <4CDA524E.4010103@gmail.com> Message-ID: On 10 November 2010 08:05, Evgeniy Khramtsov wrote: > 10.11.2010 15:29, Chandru wrote: >> >> ibrowse-2.1.0 has been released. >> >> ibrowse is a HTTP client written in erlang as an alternative to the >> built in HTTP client httpc. >> >> Get it from https://github.com/cmullaparthi/ibrowse/archives/v2.1.0 >> > > My 5 cents about ibrowse: after switching it from standard httpc library, we > found *significant* cpu consumption (almost twice). Our profiler shows that > the bottle-neck is HTTP headers parsing. Is it possible to optimize that > stuff? Probably switching to erlang:decode_packet makes sense. > I'll take a look. Thanks for reporting it. cheers Chandru From chandrashekhar.mullaparthi@REDACTED Wed Nov 10 11:05:07 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 10 Nov 2010 10:05:07 +0000 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: Message-ID: On 10 November 2010 07:54, Max Lapshin wrote: > Thank you a lot! Especially for merging stream_close You're welcome! > Btw, stream_close was working only if stream was opened like {active, > once}, not {active, true}. Have you looked at this issue? > No, I didn't. I'll take a look. ta Chandru From kuleshovmail@REDACTED Wed Nov 10 15:53:19 2010 From: kuleshovmail@REDACTED (shk) Date: Wed, 10 Nov 2010 06:53:19 -0800 (PST) Subject: Erlang mail server In-Reply-To: <20101109223543.GJ2468@hijacked.us> References: <1289286038500-3033192.post@n4.nabble.com> <20101109223543.GJ2468@hijacked.us> Message-ID: <1289400799424-3036214.post@n4.nabble.com> Thank you for reply. Is anywhere mail server based gen_smtp? -- View this message in context: http://erlang.2086793.n4.nabble.com/Erlang-mail-server-tp3033192p3036214.html Sent from the Erlang Questions mailing list archive at Nabble.com. From cappy2112@REDACTED Wed Nov 10 16:44:12 2010 From: cappy2112@REDACTED (Mr C) Date: Wed, 10 Nov 2010 07:44:12 -0800 (PST) Subject: Why does the cd() command make the shell stop processing ? Message-ID: After entering cd ("c:\Users\MyUserName\Documents\Src\Erlang\"). the prompt line number that is displayed is the same line number as before entering the cd() command. The shell doesn't properly process any erlang code after that line is entered. I'm using Windows 7 Professional with Erlang R14B (erts-5.8.1.1) From qoocku@REDACTED Wed Nov 10 16:45:54 2010 From: qoocku@REDACTED (=?UTF-8?B?RGFtaWFuIERvYnJvY3p5xYRza2k=?=) Date: Wed, 10 Nov 2010 16:45:54 +0100 Subject: Release handling with several nodes of the same release using the same target system Message-ID: <4CDABE32.6010705@gmail.com> Hi list! What is the proper scenario of (up/down)grading a release having more than one node running the same release on the same target system? Should it be something like this: 1) on node, say "n1", I unpack/install and make permanent the release version 2) switch to other nodes and do only what? "install_release" ? There is RELEASES file which is updated every time release_handler:install_release/1,2 is called, so each node using the same target system will try to update this file according to the internal state of its release_handler process. I'm not sure what is the proper way of doing this... Thnx in adv., --D. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0x0CAE3AEB.asc Type: application/pgp-keys Size: 4563 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From mononcqc@REDACTED Wed Nov 10 16:47:52 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Wed, 10 Nov 2010 10:47:52 -0500 Subject: [erlang-questions] Why does the cd() command make the shell stop processing ? In-Reply-To: References: Message-ID: You're escaping the last " with the system path. This means the string is never closed. Try forward slashes, they work well on Windows 7 (untested on other Windowses). On Wed, Nov 10, 2010 at 10:44 AM, Mr C wrote: > > After entering cd ("c:\Users\MyUserName\Documents\Src\Erlang\"). > the prompt line number that is displayed is the same line number as > before entering the cd() command. > > The shell doesn't properly process any erlang code after that line is > entered. > > I'm using Windows 7 Professional with Erlang R14B (erts-5.8.1.1) > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From andrey.pampukha@REDACTED Wed Nov 10 16:57:38 2010 From: andrey.pampukha@REDACTED (Andrey Pampukha) Date: Wed, 10 Nov 2010 18:57:38 +0300 Subject: [erlang-questions] Why does the cd() command make the shell stop processing ? In-Reply-To: References: Message-ID: Enter ". (quote and dot) then press Enter and shell will come back to you :) Also, I think that backslashes won't work in Erlang under Windows, only forward ones. 2010/11/10, Mr C : > > After entering cd ("c:\Users\MyUserName\Documents\Src\Erlang\"). > the prompt line number that is displayed is the same line number as > before entering the cd() command. > From ivan@REDACTED Wed Nov 10 17:00:54 2010 From: ivan@REDACTED (Ivan Uemlianin) Date: Wed, 10 Nov 2010 16:00:54 +0000 Subject: [erlang-questions] Why does the cd() command make the shell stop processing ? In-Reply-To: References: Message-ID: <4CDAC1B6.9040606@llaisdy.com> You could try escaping the backslashes: cd ("c:\\Users\\MyUserName\\Documents\\Src\\Erlang\\"). On 10/11/2010 15:47, Fred Hebert wrote: > You're escaping the last " with the system path. This means the string is > never closed. > > Try forward slashes, they work well on Windows 7 (untested on other > Windowses). > > On Wed, Nov 10, 2010 at 10:44 AM, Mr C wrote: > >> After entering cd ("c:\Users\MyUserName\Documents\Src\Erlang\"). >> the prompt line number that is displayed is the same line number as >> before entering the cd() command. >> >> The shell doesn't properly process any erlang code after that line is >> entered. >> >> I'm using Windows 7 Professional with Erlang R14B (erts-5.8.1.1) >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> -- ============================================================ Ivan A. Uemlianin Speech Technology Research and Development ivan@REDACTED www.llaisdy.com llaisdy.wordpress.com www.linkedin.com/in/ivanuemlianin "Froh, froh! Wie seine Sonnen, seine Sonnen fliegen" (Schiller, Beethoven) ============================================================ From bob@REDACTED Wed Nov 10 17:12:23 2010 From: bob@REDACTED (Bob Ippolito) Date: Thu, 11 Nov 2010 00:12:23 +0800 Subject: [erlang-questions] Re: Mochiweb and SSL In-Reply-To: References: <2558e383-4d2c-489a-aa6b-000ce993a367@e14g2000yqe.googlegroups.com> Message-ID: I think the problem is that you aren't starting all of the prerequisite OTP applications to make SSL work. If you post a complete runnable example I can try and take a look. On Wed, Oct 27, 2010 at 4:44 PM, Alessandro Sivieri wrote: > 2010/10/27 Steve Davis > >> What happens when you change the port to 8443 (i.e. above 1024)? >> >> > Yes, I had to use a port above 1024... anyway, I tried with 8443, and here > it is the output: > http://pastebin.com/6H67CUhP > > It looks like something is running, but then > when I try to access https://IP:8443 it crashes... I have not changed > anything else in the Mochiweb generated application, except the parameters > cited above, so I don't know if anything else has to be changed. > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://www.chimera-bellerofonte.eu/ > http://www.poul.org/ > From alessandro.sivieri@REDACTED Wed Nov 10 17:18:18 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Wed, 10 Nov 2010 17:18:18 +0100 Subject: [erlang-questions] Re: Mochiweb and SSL In-Reply-To: References: <2558e383-4d2c-489a-aa6b-000ce993a367@e14g2000yqe.googlegroups.com> Message-ID: Oh, I have solved the problem (and forgot to write about it): I think the problem was that I was using R13; on R14, it just works with the following configuration structure: WebConfigSSL = [ {ip, Ip}, {port, 8443}, {docroot, crest_deps:local_path(["priv", "www"])}, {ssl, true}, {ssl_opts, [ {certfile, crest_deps:local_path(["ca", "certs", "01.pem"])}, {keyfile, crest_deps:local_path(["ca", "private", "crest.key"])}, {verify, verify_peer}, {cacertfile, crest_deps:local_path(["ca", "cacert.pem"])}, {fail_if_no_peer_cert, true}, {verify_fun, fun(_) -> false end} ]}], of course, the three local paths point to the directory with certificates and keys, and that's all it needed; now the _web module answers correctly to SSL requests (with mutual authentication, because of the fail_if_no_peer_cert). -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From dmercer@REDACTED Wed Nov 10 17:29:11 2010 From: dmercer@REDACTED (David Mercer) Date: Wed, 10 Nov 2010 10:29:11 -0600 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: Message-ID: <00e701cb80f4$6bc0e3d0$4342ab70$@com> On Wednesday, November 10, 2010, Chandru wrote: > ibrowse is a HTTP client written in erlang as an alternative to the > built in HTTP client httpc. What is the difference between the two? That is, how do I decide whether to use ibrowse or httpc? Cheers, DBM From icarus.alive@REDACTED Wed Nov 10 17:57:54 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Wed, 10 Nov 2010 22:27:54 +0530 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: <1FC0CBB0-55BC-4E39-AC8E-BE9114C1131D@cs.otago.ac.nz> References: <1FC0CBB0-55BC-4E39-AC8E-BE9114C1131D@cs.otago.ac.nz> Message-ID: On Wed, Nov 10, 2010 at 3:34 AM, Richard O'Keefe wrote: > > On 9/11/2010, at 6:36 PM, Icarus Alive wrote: >>> Let's see. ?I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, >>> Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos >>> 64-bit, and Fedora. ?Just how many sets of binaries do you expect to be >>> provided? ?Then there are some older machines that would be fun to set >>> up, several PowerPC boxes and a couple of SPARCs. ?How many sets of >>> binaries? >> >> Am new to the whole Erlang ecosystem (language, runtime, community, >> resources), so pardon me if this the expectation seemed excessive. For >> one single person building/maintaining so many variants is definitely >> a herculean task, > > I didn't say that I *maintain* Erlang on such a range of machines, > only that this is (a subset of) the range of machines available to > me. ?The software that I am personally developing, I *do* port between > those systems (and also Cygwin, but we were talking about Unix > distributions). > > In the third-year software engineering class this year, only 1/4 of > the groups managed to develop software that ported from Linux to Linux. > One of the *major* issues was 32-bit vs 64-bit. ?People will tell you > that oh, it's no problem, you can run 32-bit programs on 64-bit > operating systems; what they _don't_ tell you is that this only works > if *all* the required 32-bit .so files are present, and no, sometimes > they are not. > >> and I wasn't aware that it was so. My expectations >> were set by distribution of Python, Java, where the latest/greatest (& >> older) versions are available for most commonly used OS's... the >> mainstream Linux distros, FreeBSD, Solaris, Windows etc. > > OK, let's try a specific example: JDK 7. > http://dlc.sun.com.edgesuite.net/jdk7/binaries/index.html > > What do we find? > ? ? ? ?Windows intel 32-bit > ? ? ? ?Windows intel 64-bit > ? ? ? ?Solaris SPARC 32-bit (presumably s10) > ? ? ? ?Solaris SPARC 64-bit (presumably s10) > ? ? ? ?Solaris intel 32-bit (will it work under OpenSolaris?) > ? ? ? ?Solaris intel 64-bit (will it work under OpenSolaris?) > ? ? ? ?Linux ? intel 32-bit > ? ? ? ?Linux ? intel 64-bit > Do we find any version of Mac OS X? > Do we find any version of *BSD? > Do we find anything other than intel or SPARC (and would we find > SPARC if Sun/Oracle didn't make SPARC systems)? > No. > > Let's try another one. ?Let's make it a functional programming > language that has been used to do exciting web-related stuff. > http://wiki.clean.cs.ru.nl/Download_Clean > > What do we find? > ? ? ? ?Windows intel 32-bit > ? ? ? ?Windows intel 64-bit > ? ? ? ?Linux ? intel 32-bit ?(no IDE) > ? ? ? ?Linux ? intel 64-bit ?(no IDE) > ? ? ? ?MacOS ? PPC ? 32-bit > Do we find any version for SPARC? ?No. ?(A long time ago...) > Do we find any version for Solaris? ?No. > Do we find any version for *BSD? ?No. > Do we find any version for Mac OS X? ?No. > ("MacOS" above appears to mean MacOS 9.) > When we download the "Windows and Mac complete sources", > do we find any instructions about how to build the thing? > Not that either. > > It turns out that instructions for building on Mac OS X > *are* available on the Wiki for that project, since last > month, but they are not complete, and the process > (install a different version of gas, install another > program to convert gas output to something the operating > system understands, apply a lot of patches, manually > visit one directory after another, section "TO DO", > ..., building the compiler "TO DO", ... ... > installing "TO DO") unpleasant. > > Erlang is a *dream* to install by contrast. Point taken. I rest my case. From icarus.alive@REDACTED Wed Nov 10 17:58:13 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Wed, 10 Nov 2010 22:28:13 +0530 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: <1FC0CBB0-55BC-4E39-AC8E-BE9114C1131D@cs.otago.ac.nz> References: <1FC0CBB0-55BC-4E39-AC8E-BE9114C1131D@cs.otago.ac.nz> Message-ID: On Wed, Nov 10, 2010 at 3:34 AM, Richard O'Keefe wrote: > > On 9/11/2010, at 6:36 PM, Icarus Alive wrote: >>> Let's see. ?I have to worry about Mac OS X 10.5, Mac OS X 10.6, OpenBSD, >>> Solaris, OpenSolaris, Ubuntu 32-bit, Ubuntu 64-bit, Centos 32-bit, Centos >>> 64-bit, and Fedora. ?Just how many sets of binaries do you expect to be >>> provided? ?Then there are some older machines that would be fun to set >>> up, several PowerPC boxes and a couple of SPARCs. ?How many sets of >>> binaries? >> >> Am new to the whole Erlang ecosystem (language, runtime, community, >> resources), so pardon me if this the expectation seemed excessive. For >> one single person building/maintaining so many variants is definitely >> a herculean task, > > I didn't say that I *maintain* Erlang on such a range of machines, > only that this is (a subset of) the range of machines available to > me. ?The software that I am personally developing, I *do* port between > those systems (and also Cygwin, but we were talking about Unix > distributions). > > In the third-year software engineering class this year, only 1/4 of > the groups managed to develop software that ported from Linux to Linux. > One of the *major* issues was 32-bit vs 64-bit. ?People will tell you > that oh, it's no problem, you can run 32-bit programs on 64-bit > operating systems; what they _don't_ tell you is that this only works > if *all* the required 32-bit .so files are present, and no, sometimes > they are not. > >> and I wasn't aware that it was so. My expectations >> were set by distribution of Python, Java, where the latest/greatest (& >> older) versions are available for most commonly used OS's... the >> mainstream Linux distros, FreeBSD, Solaris, Windows etc. > > OK, let's try a specific example: JDK 7. > http://dlc.sun.com.edgesuite.net/jdk7/binaries/index.html > > What do we find? > ? ? ? ?Windows intel 32-bit > ? ? ? ?Windows intel 64-bit > ? ? ? ?Solaris SPARC 32-bit (presumably s10) > ? ? ? ?Solaris SPARC 64-bit (presumably s10) > ? ? ? ?Solaris intel 32-bit (will it work under OpenSolaris?) > ? ? ? ?Solaris intel 64-bit (will it work under OpenSolaris?) > ? ? ? ?Linux ? intel 32-bit > ? ? ? ?Linux ? intel 64-bit > Do we find any version of Mac OS X? > Do we find any version of *BSD? > Do we find anything other than intel or SPARC (and would we find > SPARC if Sun/Oracle didn't make SPARC systems)? > No. > > Let's try another one. ?Let's make it a functional programming > language that has been used to do exciting web-related stuff. > http://wiki.clean.cs.ru.nl/Download_Clean > > What do we find? > ? ? ? ?Windows intel 32-bit > ? ? ? ?Windows intel 64-bit > ? ? ? ?Linux ? intel 32-bit ?(no IDE) > ? ? ? ?Linux ? intel 64-bit ?(no IDE) > ? ? ? ?MacOS ? PPC ? 32-bit > Do we find any version for SPARC? ?No. ?(A long time ago...) > Do we find any version for Solaris? ?No. > Do we find any version for *BSD? ?No. > Do we find any version for Mac OS X? ?No. > ("MacOS" above appears to mean MacOS 9.) > When we download the "Windows and Mac complete sources", > do we find any instructions about how to build the thing? > Not that either. > > It turns out that instructions for building on Mac OS X > *are* available on the Wiki for that project, since last > month, but they are not complete, and the process > (install a different version of gas, install another > program to convert gas output to something the operating > system understands, apply a lot of patches, manually > visit one directory after another, section "TO DO", > ..., building the compiler "TO DO", ... ... > installing "TO DO") unpleasant. > > Erlang is a *dream* to install by contrast. Point taken. I rest my case. From chandrashekhar.mullaparthi@REDACTED Wed Nov 10 18:01:37 2010 From: chandrashekhar.mullaparthi@REDACTED (Chandru) Date: Wed, 10 Nov 2010 17:01:37 +0000 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: <00e701cb80f4$6bc0e3d0$4342ab70$@com> References: <00e701cb80f4$6bc0e3d0$4342ab70$@com> Message-ID: On 10 November 2010 16:29, David Mercer wrote: > On Wednesday, November 10, 2010, Chandru wrote: > >> ibrowse is a HTTP client written in erlang as an alternative to the >> built in HTTP client httpc. > > What is the difference between the two? ?That is, how do I decide whether to > use ibrowse or httpc? > Here is a totally biased view! ibrowse was developed a long time ago when the built in HTTP client sucked big time. It wasn't even called httpc then. httpc has come a long way, but you still hear about problems people are having with it. I've never used it as ibrowse was good enough for me. A long time ago the OTP team looked at ibrowse to see if they could integrate it into OTP, but then decided against it. I would suggest you try both and use what works for you. The only advantage the builtin client offers is that you don't need to install a 3rd party package, whereas ibrowse benefits from an active community which is using it a lot. cheers Chandru From jack@REDACTED Wed Nov 10 18:13:11 2010 From: jack@REDACTED (Jack Moffitt) Date: Wed, 10 Nov 2010 10:13:11 -0700 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: <00e701cb80f4$6bc0e3d0$4342ab70$@com> Message-ID: > I would suggest you try both and use what works for you. The only > advantage the builtin client offers is that you don't need to install > a 3rd party package, whereas ibrowse benefits from an active community > which is using it a lot. I have not used httpc at all, but I have been extremely happy with ibrowse over the last two years. I've used a lot of HTTP client libraries in other languages, and ibrowse is by far my favorite. jack. From spawn.think@REDACTED Wed Nov 10 18:24:16 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Wed, 10 Nov 2010 18:24:16 +0100 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: <00e701cb80f4$6bc0e3d0$4342ab70$@com> Message-ID: what's about lhttpc? http://bitbucket.org/etc/lhttpc/wiki/Home It has a list of limitations, but what about performance measures? any benchmarks or comparisons? On Wed, Nov 10, 2010 at 6:13 PM, Jack Moffitt wrote: > > I would suggest you try both and use what works for you. The only > > advantage the builtin client offers is that you don't need to install > > a 3rd party package, whereas ibrowse benefits from an active community > > which is using it a lot. > > I have not used httpc at all, but I have been extremely happy with > ibrowse over the last two years. I've used a lot of HTTP client > libraries in other languages, and ibrowse is by far my favorite. > > jack. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From pguyot@REDACTED Wed Nov 10 18:25:42 2010 From: pguyot@REDACTED (Paul Guyot) Date: Wed, 10 Nov 2010 18:25:42 +0100 Subject: Release handling with several nodes of the same release using the same target system In-Reply-To: <1289408519.19243.ezmlm@erlang.org> References: <1289408519.19243.ezmlm@erlang.org> Message-ID: > Hi list! > > What is the proper scenario of (up/down)grading a release having more > than one node running the same release on the same target system? > > Should it be something like this: > > 1) on node, say "n1", I unpack/install and make permanent the release > version > 2) switch to other nodes and do only what? "install_release" ? > > There is RELEASES file which is updated every time > release_handler:install_release/1,2 is called, so each node using the > same target system will try to update this file according to the > internal state of its release_handler process. I'm not sure what is the > proper way of doing this... > > Thnx in adv., > > --D. If all nodes share the same OTP files, this is what you can do: On one node, call release_handler:unpack_release/1. On the other nodes, call release_handler:set_unpacked/2. On each node, call release_handler:install_release/1,2. A node that would fail to upgrade/downgrade might restart to the previous release, since so far the RELEASES file is untouched. Once all nodes are running the new release, you can make it permanent. You actually only need to do this on one node, since each node will write the RELEASES file, but you can do it on all nodes to update the state of the release_handler server. If you want to remove a release, perform likewise. Remove it on one node, call release_handler:set_removed/1 on the others. Paul -- Semiocast http://semiocast.com/ +33.175000290 - 62 bis rue Gay-Lussac, 75005 Paris From xramtsov@REDACTED Wed Nov 10 18:47:38 2010 From: xramtsov@REDACTED (Evgeniy Khramtsov) Date: Thu, 11 Nov 2010 02:47:38 +0900 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: <00e701cb80f4$6bc0e3d0$4342ab70$@com> Message-ID: <4CDADABA.1030900@gmail.com> 11.11.2010 02:24, Ahmed Omar wrote: > what's about lhttpc? > http://bitbucket.org/etc/lhttpc/wiki/Home > It has a list of limitations, but what about performance measures? any > benchmarks or comparisons? > We have no benchmarks, but we have some experience with both in production environment. In some projects where we need statefull http connections we use ibrowse, but in projects where we need stateless (Connection: close) connections with very high rate (thousands connections per second) we use lhttpc. Also, lhttpc code is pretty compact and, thus, easily to understand :) -- Regards, Evgeniy Khramtsov, ProcessOne. xmpp:xram@REDACTED From per.melin@REDACTED Wed Nov 10 20:23:54 2010 From: per.melin@REDACTED (Per Melin) Date: Wed, 10 Nov 2010 20:23:54 +0100 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: <00e701cb80f4$6bc0e3d0$4342ab70$@com> References: <00e701cb80f4$6bc0e3d0$4342ab70$@com> Message-ID: On Wed, Nov 10, 2010 at 5:29 PM, David Mercer wrote: > On Wednesday, November 10, 2010, Chandru wrote: >> ibrowse is a HTTP client written in erlang as an alternative to the >> built in HTTP client httpc. > > What is the difference between the two? ?That is, how do I decide whether to > use ibrowse or httpc? I don't currently use ibrowse but I can say something about httpc; I don't think there's been a release of OTP since 2008 where it hasn't had at least one serious bug. Someone please correct me if I'm wrong. Two times I've had to roll back to older versions of OTP until I could get/make a patch. Last time was with the httpc memory leak in R13B04. From jeffm@REDACTED Thu Nov 11 00:10:56 2010 From: jeffm@REDACTED (jeff) Date: Thu, 11 Nov 2010 10:10:56 +1100 Subject: [erlang-questions] Erlang mail server In-Reply-To: <20101109223543.GJ2468@hijacked.us> References: <1289286038500-3033192.post@n4.nabble.com> <20101109223543.GJ2468@hijacked.us> Message-ID: <4CDB2680.5090701@ghostgun.com> I've been thinking along the same lines but was thinking of a distributed file system, eg lustre, instead of a distributed DB. Jeff. On 10/11/10 9:35 AM, Andrew Thompson wrote: > > Yeah, gen_smtp is the only thing that even comes close, to my (biased) > knowledge, but its a SMTP/MIME toolkit, not a complete mailserver (but > you could build one with it). You'd need to write some kind of mail > storage and then expose the mail to the users somehow > (IMAP/POP/mbox/maildir). > > Someday I'd like to write a distributed mailserver using gen_smtp that > stores mail to riak or mnesia or something and lets you access your mail > from any host in the cluster (and send from any of them). No chance of > that happening any time soon, though, because I don't need such a thing > for anything other than a toy. > From nesrait@REDACTED Thu Nov 11 02:32:45 2010 From: nesrait@REDACTED (=?ISO-8859-1?Q?Davide_Marqu=EAs?=) Date: Thu, 11 Nov 2010 01:32:45 +0000 Subject: [erlang-questions] Is gen_server behavior in R14B hibernate friendly? In-Reply-To: <5785C41E-9A80-41A4-8A95-17F46D65A7DA@erlang-solutions.com> References: <95fcaa07-6a18-4681-804c-619481310077@e26g2000vbz.googlegroups.com> <5785C41E-9A80-41A4-8A95-17F46D65A7DA@erlang-solutions.com> Message-ID: Hi Zvi, Check out the attached gen_server_light module. :) IIRC it differs from gen_server in that the handle_call/3 supports a new return value: {noreply, NewState, {call_and_hibernate, {Server, Req, ReqTimeout}}}; which instructs it to issue a call to another server and then hibernate while waiting for the reply. When that reply comes (or the ReqTimeout is reached) handle_reply/2 is called enabling the caller to proceed with what it was doing. Hope the code still works. :) Cheers, :Davide On Sun, Nov 7, 2010 at 3:03 PM, Ulf Wiger wrote: > > Hi Zvi, > > Actually, nowadays, gen_server supports 'hibernate' as a special > value of the internal timeout, so your server callbacks can instruct > the main loop to go into hibernation between messages. > > It is true that hibernate/3 doesn't have a timeout, but OTOH, it is > easy enough to start a timer that will send a message, bringing > you out of hibernation, after a specified time. > > BR, > Ulf W > > On 6 Nov 2010, at 15:51, Zvi wrote: > > > Hi, > > I re-reading Erlang at Facebook slides. And found this. Is this still > > an issue? > > > > Zvi > > > > "Hibernation > > ? > > Drastically shrink memory usage with erlang:hibernate/3 > > ? Throws away the call stack > > ? Minimizes the heap > > ? Enters a wait state for new messages > > ? ?Jumps? into a passed-in function for a received message > > ? Perfect for a long-running, idling HTTP request handler > > ? But ... not compatible with gen_server:call (and gen_server:reply) > > ? gen_server:call has its own receive() loop > > ? hibernate() doesn?t support have an explicit timeout > > ? Fixed with a few hours and a look at gen.erl" > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gen_server hibernate.zip Type: application/zip Size: 11801 bytes Desc: not available URL: From ok@REDACTED Thu Nov 11 02:50:14 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 11 Nov 2010 14:50:14 +1300 Subject: [erlang-questions] About Garbage Collection In-Reply-To: References: Message-ID: <291F7A28-E9B6-43EF-A858-2DE50EF09CF1@cs.otago.ac.nz> On 10/11/2010, at 7:19 PM, ?. Volkan Erdo?an wrote: > Hi, > I have an idea about memory management in Erlang and although the idea > is still raw I'd like to share with you. > > As far as I know: > - There are no global variables in Erlang. Memory is allocated in functions. > - Function calls in processes and message passing operations to other > processes have copying semantics. More precisely, you can't *tell* whether things are copied or not. But in practice, function arguments are NOT copied. Consider for example len(List) -> len_loop(List, 0). len_loop([], L) -> L; len_loop([_|Tail], L) -> len_loop(Tail, L+1). Suppose you have a list L = [1,...,N]. If arguments were copied, len(L) would take O(N**2) time. Since arguments are *not* copied, len(L) takes O(N) time. > So I believe that these properties guarantee that any memory location > allocated by runtime is exclusive to the function which requested that > memory > and it is not possible for a reference of a memory location to > escape from the function that created it. Consider f(X) -> Y = g(X), {Y,Y}. g(X) -> [X,X,X]. The memory allocated by g/1 for the three-element list [X,X,X] is *not* exclusive to g/1 but outlives the call that created it. The value of Y in f/1 is the very same list allocated in g/1. So this is wrong. > And as a result it is > possible to safely deallocate all memory allocated by the function at > function exit since those memory locations can only be accessed by > that function. And that's wrong too. However, there's something that _could_ be done. There was a system called I think "Formula Algol" developed at CWI many years ago; a computer algebra system. I reimplemented it in Burroughs B6700 Algol in the late 70s and had a lot of fun with it. The key idea there was that when a function returned, the new cells it had allocated were copied back to its caller, and everything not returned was freed. However, that has problems too. Consider make_list(N) when is_integer(N), N >= 0 -> make_list_loop(N, []). make_list_loop(0, L) -> L; make_list_loop(N, L) -> make_list_loop(N-1, [N|L]). For example, make_list(3) ==> [1,2,3]. Copying the results back would make this take O(N**2) time whereas the real non-copying Erlang system takes O(N) time. However, if we push on that idea a little harder, we come close to the idea of REGION allocation. In effect, the heap is broken into a lot of little heaps, and a function can allocate its result in (one of) its caller's heap(s). A good quick reference to that is http://mlton.org/Regions which explains why the MLTon compiler for ML does NOT use regions. I don't *think* regions are compatible with hot loading, but it would be nice to be wrong about that. From rzezeski@REDACTED Thu Nov 11 04:10:26 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Wed, 10 Nov 2010 22:10:26 -0500 Subject: license question Message-ID: I'm sorry if this is an obvious question but I am very ignorant of how licenses work. I want to put an Erlang application on github. However, this particular application would include direct copies of error_logger_file_h.erl, error_logger_tty_h.erl, sasl_report_file_h.erl and sasl_report_tty_h.erl. Furthermore, I would add my own modifications to these modules as well as rename them by adding a prefix of my app name. If I do this I have a few questions: 1. Am I allowed to do this? 2. If I am, do I leave the Ericsson copyright in place? If not, what do I put there? 3. Does it have to remain under the Erlang Public License? I honestly don't really care what license I put it under but I'm curious. -Ryan From matthias@REDACTED Thu Nov 11 06:54:27 2010 From: matthias@REDACTED (Matthias Lang) Date: Thu, 11 Nov 2010 06:54:27 +0100 Subject: [erlang-questions] license question In-Reply-To: References: Message-ID: <20101111055427.GA3272@corelatus.se> On Wednesday, November 10, Ryan Zezeski wrote: > I'm sorry if this is an obvious question but I am very ignorant of how > licenses work. It's not likely that you'll get an authoritative answer on the list, so here's a guess (with reasoning). > I want to put an Erlang application on github. However, > this particular application would include direct copies of > error_logger_file_h.erl, error_logger_tty_h.erl, sasl_report_file_h.erl and > sasl_report_tty_h.erl. Furthermore, I would add my own modifications to > these modules as well as rename them by adding a prefix of my app name. > > If I do this I have a few questions: > > 1. Am I allowed to do this? Yes. I think section 2.1.a of the "Erlang Public License" (included in the source distribution as "EPLICENCE") gives you the right to do the above. > 2. If I am, do I leave the Ericsson copyright in place? If not, what do I > put there? Yes, you must leave the Ericsson copyright in place. "EXHIBIT A" is the three paragraphs at the bottom of the EPL, which includes the "Ericsson copyright". Section 3.5 requires you to include Exhibit A in each source file. > 3. Does it have to remain under the Erlang Public License? I honestly don't > really care what license I put it under but I'm curious. Yes, the code remains under the EPL. You are obliged to put (leave) "EXHIBIT A" in the file, and Exhibit A says "The contents of this file are subject to the Erlang Public Licence..." You have some other obligations, take a look at section 3 of the licence file. Matt From volerdo@REDACTED Thu Nov 11 09:44:51 2010 From: volerdo@REDACTED (=?ISO-8859-9?Q?=DE=2E_Volkan_Erdo=F0an?=) Date: Thu, 11 Nov 2010 10:44:51 +0200 Subject: [erlang-questions] About Garbage Collection In-Reply-To: <291F7A28-E9B6-43EF-A858-2DE50EF09CF1@cs.otago.ac.nz> References: <291F7A28-E9B6-43EF-A858-2DE50EF09CF1@cs.otago.ac.nz> Message-ID: It seemed like a good idea at first :) I just thought that , since there is no memory is shared between processes one can determine when a memory location is not used in a process conservatively by static analysis and tried to present a scheme for that but it turns out to be wrong. Thanks a lot for the explanations. 2010/11/11 Richard O'Keefe : > > On 10/11/2010, at 7:19 PM, ?. Volkan Erdo?an wrote: > >> Hi, >> I have an idea about memory management in Erlang and although the idea >> is still raw I'd like to share with you. >> >> As far as I know: >> - There are no global variables in Erlang. Memory is allocated in functions. >> - Function calls in processes and message passing operations to other >> processes have copying semantics. > > More precisely, you can't *tell* whether things are copied or not. > > But in practice, function arguments are NOT copied. > Consider for example > > ? ? ? ?len(List) -> len_loop(List, 0). > > ? ? ? ?len_loop([], L) -> L; > ? ? ? ?len_loop([_|Tail], L) -> len_loop(Tail, L+1). > > Suppose you have a list L = [1,...,N]. > If arguments were copied, len(L) would take O(N**2) time. > Since arguments are *not* copied, len(L) takes O(N) time. > >> So I believe that these properties guarantee that any memory location >> allocated by runtime is exclusive to the function which requested that >> memory >> and it is not possible for a reference of a memory location to >> escape from the function that created it. > > Consider > ? ? ? ?f(X) -> Y = g(X), {Y,Y}. > ? ? ? ?g(X) -> [X,X,X]. > > The memory allocated by g/1 for the three-element list [X,X,X] > is *not* exclusive to g/1 but outlives the call that created it. > The value of Y in f/1 is the very same list allocated in g/1. > > So this is wrong. > >> ?And as a result it is >> possible to safely deallocate all memory allocated by the function at >> function exit since those memory locations can only be accessed by >> that function. > > And that's wrong too. > > However, there's something that _could_ be done. > There was a system called I think "Formula Algol" developed at CWI > many years ago; a computer algebra system. ?I reimplemented it in > Burroughs B6700 Algol in the late 70s and had a lot of fun with it. > The key idea there was that when a function returned, > the new cells it had allocated were copied back to its caller, > and everything not returned was freed. > > However, that has problems too. > > Consider > > ? ? ? ?make_list(N) > ? ? ? ? ?when is_integer(N), N >= 0 -> > ? ? ? ? ? ?make_list_loop(N, []). > > ? ? ? ?make_list_loop(0, L) -> > ? ? ? ? ? ?L; > ? ? ? ?make_list_loop(N, L) -> > ? ? ? ? ? ?make_list_loop(N-1, [N|L]). > > For example, make_list(3) ==> [1,2,3]. > > Copying the results back would make this take O(N**2) time > whereas the real non-copying Erlang system takes O(N) time. > > However, if we push on that idea a little harder, we come > close to the idea of REGION allocation. ?In effect, the heap > is broken into a lot of little heaps, and a function can > allocate its result in (one of) its caller's heap(s). > A good quick reference to that is > http://mlton.org/Regions > which explains why the MLTon compiler for ML does NOT use > regions. > > I don't *think* regions are compatible with hot loading, but > it would be nice to be wrong about that. > > > From lemenkov@REDACTED Thu Nov 11 12:48:13 2010 From: lemenkov@REDACTED (Peter Lemenkov) Date: Thu, 11 Nov 2010 14:48:13 +0300 Subject: Lots of pre-built stuff in otp_src_R14B.tar.gz Message-ID: Hello All! I just found that tarball with sources for R14B contains pre-built *.beam files. Is it intentional? http://erlang.org/download/otp_src_R14B.tar.gz -- With best regards, Peter Lemenkov. From raimo+erlang-questions@REDACTED Thu Nov 11 13:21:30 2010 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 11 Nov 2010 13:21:30 +0100 Subject: [erlang-questions] Lots of pre-built stuff in otp_src_R14B.tar.gz In-Reply-To: References: Message-ID: <20101111122130.GA364@erix.ericsson.se> On Thu, Nov 11, 2010 at 02:48:13PM +0300, Peter Lemenkov wrote: > Hello All! > > I just found that tarball with sources for R14B contains pre-built > *.beam files. Is it intentional? Oh yes. It is needed for the bootstrap process; roughly; first the Virtual Machine is built, then it is used together with the pre-built *.beam files to form a small Erlang system capable of compiling the essential parts of the release e.g the stdlib, kernel and the compiler. Then this newly built system is used to compile the rest of the release. > > http://erlang.org/download/otp_src_R14B.tar.gz > > -- > With best regards, Peter Lemenkov. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From sgolovan@REDACTED Thu Nov 11 13:33:08 2010 From: sgolovan@REDACTED (Sergei Golovan) Date: Thu, 11 Nov 2010 15:33:08 +0300 Subject: [erlang-questions] Lots of pre-built stuff in otp_src_R14B.tar.gz In-Reply-To: <20101111122130.GA364@erix.ericsson.se> References: <20101111122130.GA364@erix.ericsson.se> Message-ID: On Thu, Nov 11, 2010 at 3:21 PM, Raimo Niskanen wrote: > On Thu, Nov 11, 2010 at 02:48:13PM +0300, Peter Lemenkov wrote: >> Hello All! >> >> I just found that tarball with sources for R14B contains pre-built >> *.beam files. Is it intentional? > > Oh yes. > > It is needed for the bootstrap process; In fact, starting from R11B-5 Erlang/OTP source tarball contains all beam files prebuilt. AFAIK it's to conserve build time as now it's sufficient to build only virtual machine (and a few binaries needed by some applications). Cheers! -- Sergei Golovan From vladdu55@REDACTED Thu Nov 11 13:45:03 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Thu, 11 Nov 2010 13:45:03 +0100 Subject: internal pubsub framework Message-ID: Hi! Is there any implementation/work/paper on a pubsub-like framework in Erlang, to be used internally between processes (as opposed to a generic and much heavier one like rabbitmq)? Google didn't show anything, maybe I'm using the wrong keywords. If nothing is there, does it look like an interesting idea to try to create one? I have an application where I use direct connections between processes, but now want to support a dynamic settings where processes can come and go and it feels like a pubsub solution would make everything much easier. best regards, Vlad From lemenkov@REDACTED Thu Nov 11 13:51:27 2010 From: lemenkov@REDACTED (Peter Lemenkov) Date: Thu, 11 Nov 2010 15:51:27 +0300 Subject: [erlang-questions] Lots of pre-built stuff in otp_src_R14B.tar.gz In-Reply-To: <20101111122130.GA364@erix.ericsson.se> References: <20101111122130.GA364@erix.ericsson.se> Message-ID: 2010/11/11 Raimo Niskanen : >> I just found that tarball with sources for R14B contains pre-built >> *.beam files. Is it intentional? > It is needed for the bootstrap process; It seems that the number of prebuilt files greatly surpasses the number of ones actually required for bootstrapping. According to README.bootstrap the only required binaries are lib/{compiler,kernel,stdlib}/ebin/*.beam. -- With best regards, Peter Lemenkov. From spawn.think@REDACTED Thu Nov 11 13:55:02 2010 From: spawn.think@REDACTED (Ahmed Omar) Date: Thu, 11 Nov 2010 13:55:02 +0100 Subject: [erlang-questions] internal pubsub framework In-Reply-To: References: Message-ID: http://groups.google.com/group/erlang-programming/browse_frm/thread/b0f17706a58b4c66?tvc=1 On Thu, Nov 11, 2010 at 1:45 PM, Vlad Dumitrescu wrote: > Hi! > > Is there any implementation/work/paper on a pubsub-like framework in > Erlang, to be used internally between processes (as opposed to a > generic and much heavier one like rabbitmq)? Google didn't show > anything, maybe I'm using the wrong keywords. > > If nothing is there, does it look like an interesting idea to try to > create one? I have an application where I use direct connections > between processes, but now want to support a dynamic settings where > processes can come and go and it feels like a pubsub solution would > make everything much easier. > > best regards, > Vlad > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- Best Regards, - Ahmed Omar http://nl.linkedin.com/in/adiaa Follow me on twitter @spawn_think From max.lapshin@REDACTED Thu Nov 11 13:56:08 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 11 Nov 2010 15:56:08 +0300 Subject: [erlang-questions] Lots of pre-built stuff in otp_src_R14B.tar.gz In-Reply-To: References: <20101111122130.GA364@erix.ericsson.se> Message-ID: > It ?seems that the number of prebuilt files greatly surpasses the > number of ones actually required for bootstrapping. According to > README.bootstrap the only required binaries are > lib/{compiler,kernel,stdlib}/ebin/*.beam. > Sergey told you, that it is prebuilt beams make build of erlang faster. From vladdu55@REDACTED Thu Nov 11 14:07:08 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Thu, 11 Nov 2010 14:07:08 +0100 Subject: [erlang-questions] internal pubsub framework In-Reply-To: References: Message-ID: On Thu, Nov 11, 2010 at 13:55, Ahmed Omar wrote: > http://groups.google.com/group/erlang-programming/browse_frm/thread/b0f17706a58b4c66?tvc=1 Thanks! It wasn't even too long ago that question was asked! :-) My excuse is that I was on vacation that week :-) /Vlad > On Thu, Nov 11, 2010 at 1:45 PM, Vlad Dumitrescu wrote: >> >> Hi! >> >> Is there any implementation/work/paper on a pubsub-like framework in >> Erlang, to be used internally between processes (as opposed to a >> generic and much heavier one like rabbitmq)? Google didn't show >> anything, maybe I'm using the wrong keywords. From rtrlists@REDACTED Thu Nov 11 14:27:46 2010 From: rtrlists@REDACTED (Robert Raschke) Date: Thu, 11 Nov 2010 13:27:46 +0000 Subject: [erlang-questions] Lots of pre-built stuff in otp_src_R14B.tar.gz In-Reply-To: References: <20101111122130.GA364@erix.ericsson.se> Message-ID: On Thu, Nov 11, 2010 at 12:33 PM, Sergei Golovan wrote: > On Thu, Nov 11, 2010 at 3:21 PM, Raimo Niskanen > > > wrote: > > On Thu, Nov 11, 2010 at 02:48:13PM +0300, Peter Lemenkov wrote: > >> Hello All! > >> > >> I just found that tarball with sources for R14B contains pre-built > >> *.beam files. Is it intentional? > > > > Oh yes. > > > > It is needed for the bootstrap process; > > In fact, starting from R11B-5 Erlang/OTP source tarball contains all > beam files prebuilt. AFAIK it's to conserve build time as now it's > sufficient to build only virtual machine (and a few binaries needed > by some applications). > > Cheers! > -- > Sergei Golovan > > I wonder ... could that be the cause for the warnings "xmerl: Object code (xmerl_*) out of date" when building a release that includes xmerl against a vanilla install (at least R12 and R13, haven't tried R14 yet)? Not sure why the other modules don't have this tho'. Robby From jesper.louis.andersen@REDACTED Thu Nov 11 14:45:02 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 11 Nov 2010 14:45:02 +0100 Subject: [erlang-questions] About Garbage Collection In-Reply-To: References: <291F7A28-E9B6-43EF-A858-2DE50EF09CF1@cs.otago.ac.nz> Message-ID: 2010/11/11 ?. Volkan Erdo?an : > It seemed like a good idea at first :) > I just thought that , since there is no memory is shared between > processes one can determine when a memory location is not used in a > process conservatively by static analysis and tried to present a > scheme for that but it turns out to be wrong. Essentially, what region based memory management does is to make a conservative static analysis of liveness of variables. You lace your original language with a region language that has operations like alloc_new_heap, alloc_in_heap, reset_heap, destroy_heap. The region language can either be lexically scoped and functional (The Tofte-Talpin interpretation), so imperative (the Henglein-Makholm-Niss interpretation), so you have some freedom in choosing the language. Static analysis now inserts operations from the region language into the original one and then compilation will compile the region operations as well, usually by calling a runtime function to handle it. The problem is the "conservative" part, which is the reason MLton shunted the idea. When you can't determine liveness you default to the top-level region, which you then have to garbage collect. In other words, you have done a lot of very hard work (lacing a region-langauge into your system, region inference analysis, runtime modifications etc) and the benefit is probably negligible. In MLtons case there is so SSA-PRE pass yet, so time would be much better spent on adding that as it could be much more beneficial. (Obvious bias: I was somewhat loosely affiliated with the MLton project years ago). -- J. From raimo+erlang-questions@REDACTED Thu Nov 11 15:03:55 2010 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Thu, 11 Nov 2010 15:03:55 +0100 Subject: [erlang-questions] Lots of pre-built stuff in otp_src_R14B.tar.gz In-Reply-To: References: <20101111122130.GA364@erix.ericsson.se> Message-ID: <20101111140355.GA4540@erix.ericsson.se> On Thu, Nov 11, 2010 at 03:56:08PM +0300, Max Lapshin wrote: > > It ?seems that the number of prebuilt files greatly surpasses the > > number of ones actually required for bootstrapping. According to > > README.bootstrap the only required binaries are > > lib/{compiler,kernel,stdlib}/ebin/*.beam. > > > > Sergey told you, that it is prebuilt beams make build of erlang faster. Yes. Sergey is right. I had our internal builds from scratch in mind. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From kenneth.lundin@REDACTED Thu Nov 11 16:31:53 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Thu, 11 Nov 2010 16:31:53 +0100 Subject: [eeps] EEP 35 "Binary string modules" Message-ID: We have a quite new EEP (EEP 35) with the title "Binary string module(s) that we want your feedback on so we can start implement it. We would like to get your feedback before November 25. If you are attending the Erlang User Conference next week , take the opportunity to discuss the EEP with your fellow Erlangers and with us in the OTP team at Ericsson. /Kenneth, Erlang/OTP Ericsson ---------- Forwarded message ---------- From: Raimo Niskanen Date: Mon, Oct 4, 2010 at 5:36 PM Subject: [eeps] EEP 35 To: eeps@REDACTED A new EEP ? ?EEP 35: Binary string module(s) has been added. See ? ?http://demo.erlang.org/eeps/ andalso ? ?http://github.com/erlang/eep/tree/master//eeps/ -- / Raimo Niskanen, Erlang/OTP, Ericsson AB ________________________________________________________________ eeps (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:eeps-unsubscribe@REDACTED From dmercer@REDACTED Thu Nov 11 17:01:26 2010 From: dmercer@REDACTED (David Mercer) Date: Thu, 11 Nov 2010 10:01:26 -0600 Subject: [erlang-questions] internal pubsub framework In-Reply-To: References: Message-ID: <012201cb81b9$b2d3f800$187be800$@com> On Thursday, November 11, 2010, Vlad Dumitrescu wrote: > Is there any implementation/work/paper on a pubsub-like framework in > Erlang, to be used internally between processes (as opposed to a > generic and much heavier one like rabbitmq)? Google didn't show > anything, maybe I'm using the wrong keywords. Isn't gen_event a pub-sub framework? DBM From alain.odea@REDACTED Thu Nov 11 18:32:25 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Thu, 11 Nov 2010 14:02:25 -0330 Subject: Erlang/OTP debugger on Mac OS X 10.6 Snow Leopard crashes regularly Message-ID: I have installed OTP R14B using http://github.com/mxcl/homebrew/. It installs 64-bit ERTS and applications, but it doesn't have wxErlang support because wxMac 2.8 does not build in 64-bit. When I use the debugger it launches a Wish application that appears to use native OS X widgets. However, the debugger is extremely unstable in this environment. It worked much better in the X11 wish in 10.5 and earlier. I can rarely make it through a debugging session without the Wish application freezing. I have successfully restarted debugger by restarting the whole node, but it rapidly crashes again. I have also managed to force the monitor application to restart with exit(im(), 'EXIT'). It seems that it doesn't notice that wish has crashed. Is there a reliable way to debug Erlang applications on Mac OS X 10.6 Snow Leopard? I am in the process of building a 32-bit OTP dev linked against a 32-bit wxWidgets 2.8.11. Hopefully this will work, but I still wonder if ErlIDE would serve me better for this use case. IMHO, wxErlang should be a service provided by a C node rather than a linked-in driver. To the best of my knowledge that would make 32/64-bit a non-issue. This would be similar to the WebKit plug-in solution which allows 32-bit plug-ins (like Flash) in a 64-bit browser. From kiszl@REDACTED Thu Nov 11 19:27:56 2010 From: kiszl@REDACTED (Zoltan Lajos Kis) Date: Thu, 11 Nov 2010 19:27:56 +0100 Subject: [erlang-questions] [eeps] EEP 35 "Binary string modules" In-Reply-To: References: Message-ID: <4CDC35AC.8030900@tmit.bme.hu> My opinion is that it would be more convenient to have a (single) module that provides string handling on the iodata() data type. Right now we have some functionality available only for list strings, some for binary strings, and some for iodata strings; some with unicode support, some without. This new module would definitely broaden the available functionality, but it feels like we will still have to convert b/w iodata, lists and binaries. Regards, Zoltan. On 11/11/2010 4:31 PM, Kenneth Lundin wrote: > We have a quite new EEP (EEP 35) with the title "Binary string > module(s) that we want your feedback on so we can > start implement it. > > We would like to get your feedback before November 25. > > If you are attending the Erlang User Conference next week , take the > opportunity to discuss the EEP with your fellow Erlangers and > with us in the OTP team at Ericsson. > > /Kenneth, Erlang/OTP Ericsson > > > ---------- Forwarded message ---------- > From: Raimo Niskanen > Date: Mon, Oct 4, 2010 at 5:36 PM > Subject: [eeps] EEP 35 > To: eeps@REDACTED > > > A new EEP > EEP 35: Binary string module(s) > has been added. See > http://demo.erlang.org/eeps/ > andalso > http://github.com/erlang/eep/tree/master//eeps/ > > -- > > / Raimo Niskanen, Erlang/OTP, Ericsson AB > > ________________________________________________________________ > eeps (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:eeps-unsubscribe@REDACTED > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From fritchie@REDACTED Thu Nov 11 19:59:02 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Thu, 11 Nov 2010 12:59:02 -0600 Subject: New OTP app available at GitHub: riak_err Message-ID: <24880.1289501942@snookles.snookles.com> Hi, all, there's a new OTP app available over at GitHub that other users may find useful: riak_err. See https://github.com/basho/riak_err/ If you've ever been bitten by having a 5MB binary in an error message explode (on a 64-bit platform) into over 300MB of short-term memory usage(*) when the error_logger tries to format the message before sending it to the console/TTY or writing it to a file, then you *really* want some kind of solution. Answering the question, "Why did the VM crash with a memory allocation error?" for a grumpy customer isn't fun. If an info/error/warning message (or report) is too big(**), then Matthias Lang's nifty trunc_io.erl library is used to truncate the formatted data. The riak_err app is designed to be a mostly-painless way to limit the amount of RAM that error messages consume while being formatted. It's just a matter of adding the riak_err application to your app's app-dependency list. That list is probably already starting the 'sasl' app, so just add 'riak_err' immediately after 'sasl', and all should be well. :-) I'd be really interested to know if there are any messages that are overlooked when measuring & truncating, e.g. a SASL message or something from the kernel or a user-generated message (using error_logger:info_msg() and friends). -Scott (*) This has bitten me and my colleagues several times over the last several years. (**) As defined by two config knobs, see the README or EDoc for details. From max.lapshin@REDACTED Thu Nov 11 20:36:46 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 11 Nov 2010 22:36:46 +0300 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: <24880.1289501942@snookles.snookles.com> References: <24880.1289501942@snookles.snookles.com> Message-ID: > I'd be really interested to know if there are any messages that are > overlooked when measuring & truncating, e.g. a SASL message or > something from the kernel or a user-generated message (using > error_logger:info_msg() and friends). > You have to modify gen_server to make error-proof message logging: https://github.com/erlyvideo/erlyvideo/blob/master/src/core/gen_server_ems.erl#L747 It is a pity, but current state of erlang error logging is error proof only until large errors happen. I don't know if my way of error limiting is ok for everyone: https://github.com/erlyvideo/erlyvideo/blob/master/src/core/io_lib_pretty_limited.erl or yours one is better, but I want OTP team to communicate about it. gen_server:format_state is not enough because of dumping message queue and error reason (it includes dumping state). Problem is larger and it requires looking into all gen_ code and proc_lib infrastructure. OTP is excelent software, but not ready for errors in large state processes =( I'm willing to help to fix it. Your patches are here, my patches are here. RabbitMQ patches are available (they are so deep, that R14 got even new instruction). Lets work out some best solution. From max.lapshin@REDACTED Thu Nov 11 20:46:59 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 11 Nov 2010 22:46:59 +0300 Subject: OOM killer inside erlang VM Message-ID: I've asked this question one year ago: how to behave if it is impossible to allocate memory. mem_sup knows which process is the largest. In 99% of my cases the largest process is faulty one, which must be killed. Kenneth told about per-process quota: http://groups.google.com/group/erlang-programming/browse_thread/thread/185e0a0deb4233f0/7a1d59884409bd3e?lnk=gst&q=oom+killer#7a1d59884409bd3e Maybe someone has got any new ideas about how not to fail beause of one faulty process? Some time ago I've discussed it with author of Nginx: Igor Sysoev and hi told about two features as of most important nginx ideas: survive failed malloc and survive failed log appending. From woody.peterson@REDACTED Thu Nov 11 21:16:06 2010 From: woody.peterson@REDACTED (Woody Peterson) Date: Thu, 11 Nov 2010 12:16:06 -0800 Subject: mnesia performance data? Message-ID: <71789DF7-B841-4988-A8AB-9121084F6A47@gmail.com> I'd like to see misc mnesia stats like transactions/sec per table on a graph, and it's totally do-able via some stats process and mnesia:subscribe. I feel like I can't be the first one to have wanted this, but a googling hasn't been helpful. Anyone know of something like this already? -Woody From rzezeski@REDACTED Thu Nov 11 22:19:26 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Thu, 11 Nov 2010 16:19:26 -0500 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: <24880.1289501942@snookles.snookles.com> References: <24880.1289501942@snookles.snookles.com> Message-ID: On Thu, Nov 11, 2010 at 1:59 PM, Scott Lystig Fritchie < fritchie@REDACTED> wrote: > Hi, all, there's a new OTP app available over at GitHub that other users > may find useful: riak_err. See https://github.com/basho/riak_err/ > > If you've ever been bitten by having a 5MB binary in an error message > explode (on a 64-bit platform) into over 300MB of short-term memory > usage(*) when the error_logger tries to format the message before > sending it to the console/TTY or writing it to a file, then you *really* > want some kind of solution. Answering the question, "Why did the VM > crash with a memory allocation error?" for a grumpy customer isn't fun. > > If an info/error/warning message (or report) is too big(**), then > Matthias Lang's nifty trunc_io.erl library is used to truncate the > formatted data. > > The riak_err app is designed to be a mostly-painless way to limit the > amount of RAM that error messages consume while being formatted. It's > just a matter of adding the riak_err application to your app's > app-dependency list. That list is probably already starting the 'sasl' > app, so just add 'riak_err' immediately after 'sasl', and all should be > well. :-) > > I'd be really interested to know if there are any messages that are > overlooked when measuring & truncating, e.g. a SASL message or > something from the kernel or a user-generated message (using > error_logger:info_msg() and friends). > > -Scott > > (*) This has bitten me and my colleagues several times over the last > several years. > > (**) As defined by two config knobs, see the README or EDoc for details. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > This is funny because I was _just_ about to release an application on github that does the exact same thing you are talking about. I've already been bitten a few times by this. The method I took was to copy/paste the standard handlers and modify them to use a function I called 'make_safe' which would take any Erlang term and make it safe (in recursive manner) to be formatted by io_lib. My app then removes the standard handlers, reads their configuration, and then adds my modified versions to the error_logger mgr. I'll certainly be taking a look at this ASAP. -Ryan From max.lapshin@REDACTED Thu Nov 11 22:57:59 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 12 Nov 2010 00:57:59 +0300 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: References: <24880.1289501942@snookles.snookles.com> Message-ID: > be formatted by io_lib. ?My app then removes the standard handlers, reads > their configuration, and then adds my modified versions to the error_logger > mgr. It is not enough. Before sending several gigabytes of string data to error logger, gen_server first formats it inside. From ebegumisa@REDACTED Thu Nov 11 23:21:43 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Fri, 12 Nov 2010 09:21:43 +1100 Subject: Komodo Edit + Erlang Message-ID: Hello all, Are there an Komodo Edit/IDE users out there that have written/know of a code intelligence extension for Erlang? I've made myself a nice little Erlang/OTP Komodo toolbox* already (menus for compiling, dialyzing, edoc, etc) -- code intel is what I'm missing and this requires a fully-fledged extension. To preempt any answers pointing me to 'how-to' links on adding code intel to Komodo: Yes, I know of the koext command but my python sucks and I'd prefer something working. - Edmond - * Send me an e-mail if you want this. -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From fritchie@REDACTED Fri Nov 12 01:06:53 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Thu, 11 Nov 2010 18:06:53 -0600 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: Message of "Thu, 11 Nov 2010 22:36:46 +0300." Message-ID: <40904.1289520413@snookles.snookles.com> Max Lapshin wrote: ml> You have to modify gen_server to make error-proof message logging: ml> https://github.com/erlyvideo/erlyvideo/blob/master/src/core/gen_server_ems.erl#L747 Max, my reading of the code says that the formatting is not done by the gen_server process ... instead, the gen_server sends the format string & args to error_logger, and error_logger's event handlers do the actual formatting. Included at the end of this message is a bunch of testing that I did with a very intentionally buggy piece of code called foo.erl. I use three different variations to try to get it to mis-behave. I can't get riak_err to format something that's "too "big". If someone can find a counter-example (preferably an executable one :-), I'd love to hear it. If I use gen_server:start_link() instead, then the {'EXIT', ...} being caught by the shell will cause lib:format() to be executed, and *that* can cause huge messages to be formatted. But the (slightly sad) moral of the story is to be careful of what's linked to the shell ... and that's a lesson that most Erlang folks end up learning anyway. ml> [...] or yours one is better, but I want OTP team to communicate ml> about it. gen_server:format_state is not enough because of dumping ml> message queue and error reason (it includes dumping state). Problem ml> is larger and it requires looking into all gen_ code and proc_lib ml> infrastructure. Agreed. The default should be to have some kind of size limits (and therefore RAM-consuming limit) on any OTP-generated event/message/etc. If that default also helps allow my/your/others' code (and their events) to also observe size limits when formatting events(*), that'd be even better. -Scott (*) Apologies to Arthur C. Clarke: "I'm sorry, Dave, I can't allow you to point that gun at your foot. Here's a smaller thing you can use instead." --- snip --- snip --- snip --- snip --- snip --- snip --- Common steps: riak_err_handler:set_fmt_max_bytes(100). riak_err_handler:set_term_max_size(100). f(Pid). {ok, Pid} = foo:start(). Variation #1: foo:crash(Pid). Output: (riak@REDACTED)164> foo:crash(Pid). byebye (riak@REDACTED)165> =ERROR REPORT==== 11-Nov-2010::17:48:04 === Oversize args for format "** Generic server ~p terminating ** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n": [<0.3676.1>,{crash,"mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm..."}...] =CRASH REPORT==== 11-Nov-2010::17:48:04 === crasher: initial call: foo:init/1 pid: <0.3676.1> registered_name: [] exception exit: {call_was,{crash,"mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm..."}...} in function gen_server:handle_msg/5 in call from proc_lib:init_p_do_apply/3 ancestors: [<0.3605.1>] messages: [] links: [] dictionary: [] trap_exit: false status: running heap_size: 17711 stack_size: 24 reductions: 6774 neighbours: Variation #2: gen_server:cast(Pid, please_crash). Output: (riak@REDACTED)169> gen_server:cast(Pid, please_crash). ok (riak@REDACTED)170> =ERROR REPORT==== 11-Nov-2010::17:49:18 === Oversize args for format "** Generic server ~p terminating ** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n": [<0.4236.1>,{$gen_cast,please_crash},{state,"sssssssssssssssssssssssssssssssssssssssssssssssssss..."}...] =CRASH REPORT==== 11-Nov-2010::17:49:18 === crasher: initial call: foo:init/1 pid: <0.4236.1> registered_name: [] exception exit: {bummer,please_crash,<<"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc...">>} in function gen_server:terminate/6 in call from proc_lib:init_p_do_apply/3 ancestors: [<0.3605.1>] messages: [] links: [] dictionary: [] trap_exit: false status: running heap_size: 10946 stack_size: 24 reductions: 6373 neighbours: Variation #3: Pid ! {arbitrary_message, lists:duplicate(2*1024, $a)}, atom_here_to_avoid_confusion_with_bangs_return_value_at_the_shell. Output: (riak@REDACTED)174> Pid ! {arbitrary_message, lists:duplicate(2*1024, $a)}, atom_here_to_avoid_confusion_with_bangs_return_value_at_the_shell. atom_here_to_avoid_confusion_with_bangs_return_value_at_the_shell =ERROR REPORT==== 11-Nov-2010::17:50:19 === Oversize args for format "~w: ~s:handle_info got ~w ": [<0.4648.1>,foo,{arbitrary_message,"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..."}] (riak@REDACTED)175> Code for foo.erl follows. -module(foo). -author('fritchie@REDACTED'). -behaviour(gen_server). -define(NAME, ?MODULE). -define(Timeout, infinity). %% External exports -export([start/0]). -export([crash/1]). %% gen_server callbacks -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). -record(state, {big}). start() -> gen_server:start(?MODULE, [], []). crash(Pid) -> gen_server:call(Pid, {crash, lists:duplicate(2*1024, $m)}). init([]) -> {ok, #state{big = lists:duplicate(2*1024, $s)}}. handle_call(Request, _From, State) -> {stop, {call_was, Request, lists:duplicate(2*1024, $z)}, byebye, State}. handle_cast(Msg, _State) -> exit({bummer, Msg, list_to_binary(lists:duplicate(2*1024, $c))}). handle_info(Info, State) -> error_logger:error_msg("~w: ~s:handle_info got ~w\n", [self(), ?MODULE, Info]), {noreply, State}. terminate(_Reason, _State) -> ok. code_change(_X, _Y, Z) -> {ok, Z}. From max.lapshin@REDACTED Fri Nov 12 01:10:22 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 12 Nov 2010 03:10:22 +0300 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: <40904.1289520413@snookles.snookles.com> References: <40904.1289520413@snookles.snookles.com> Message-ID: Don't forget to add 5000000 of messages into message queue when making these tests =) From fritchie@REDACTED Fri Nov 12 02:27:33 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Thu, 11 Nov 2010 19:27:33 -0600 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: Message of "Fri, 12 Nov 2010 03:10:22 +0300." Message-ID: <44938.1289525253@snookles.snookles.com> Max Lapshin wrote: ml> Don't forget to add 5000000 of messages into message queue when ml> making these tests =) Yes, there is that little problem ... but that isn't a problem that riak_err is trying to solve(*). As has been pointed out by Ulf Wiger and many others, if there isn't a way to apply backpressure on the client (i.e. error/whatever event generators) to get them to slow down, then the server (i.e. a gen_event server like error_logger) doesn't have many options for defense. However, on an N-CPU-core system, it's even less likely that one of those options will work: throw away work. Even if the event handler could keep state about how many events/sec it could handle and then throw events that exceed that limit, there's only 1 CPU core (maximum) that can run the gen_event server. There are N-1 cores still available to run obnoxious clients to overflow the server's mailbox anyway. I'm not above putting a less-vulnerable-to-mailbox-overruns gen_event.beam file into another directory, un-sticky the kernel's code directory, and load the patched gen_event.beam into a running system ... but then again, I'm also known(**) for dirtier-than-my-colleagues'- tastes-permit hacks. -Scott (*) At least, not yet. (**) If those individuals don't have better solutions, they are welcome to remain silent. :-) From rzezeski@REDACTED Fri Nov 12 03:15:52 2010 From: rzezeski@REDACTED (Ryan Zezeski) Date: Thu, 11 Nov 2010 21:15:52 -0500 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: <44938.1289525253@snookles.snookles.com> References: <44938.1289525253@snookles.snookles.com> Message-ID: On Thu, Nov 11, 2010 at 8:27 PM, Scott Lystig Fritchie < fritchie@REDACTED> wrote: > Max Lapshin wrote: > > ml> Don't forget to add 5000000 of messages into message queue when > ml> making these tests =) > > Yes, there is that little problem ... but that isn't a problem that > riak_err is trying to solve(*). As has been pointed out by Ulf Wiger > and many others, if there isn't a way to apply backpressure on the > client (i.e. error/whatever event generators) to get them to slow down, > then the server (i.e. a gen_event server like error_logger) doesn't have > many options for defense. > > However, on an N-CPU-core system, it's even less likely that one of > those options will work: throw away work. Even if the event handler > could keep state about how many events/sec it could handle and then > throw events that exceed that limit, there's only 1 CPU core (maximum) > that can run the gen_event server. There are N-1 cores still available > to run obnoxious clients to overflow the server's mailbox anyway. > > I'm not above putting a less-vulnerable-to-mailbox-overruns > gen_event.beam file into another directory, un-sticky the kernel's code > directory, and load the patched gen_event.beam into a running system > ... but then again, I'm also known(**) for dirtier-than-my-colleagues'- > tastes-permit hacks. > > -Scott > > (*) At least, not yet. > > (**) If those individuals don't have better solutions, they are welcome > to remain silent. :-) > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > In would think 5,000,000 msgs on a mailbox would be a sign of a larger problem and at that point maybe the VM crashing is a good thing? Then again, I'm not sure. I know in my particular case I needed to guard against things like getting a 'badmatch' against a 300MB binary (the binary being a user input). It's something that only occurred on bad user input and it shouldn't be something that causes my app to crash. Sure, there were things I could have done in the code to defend against this malformed data, but in some cases it might make the code more complex and harder to understand. Furthermore, I can't possibly anticipate all the potential areas where this could happen without a _lot_ of discipline. I know help from some automated tool would go a long way. But at the end of the day, patching the error_logger handlers seemed like the "simplest thing that could possibly work." While patching the handlers I actually started thinking that maybe this work should actually go into the io functions themselves. For example, in Clojure if you try to print a large data structure in the REPL it will automatically truncate it unless you dynamically bind a special variable. This is something implemented all the way down at the very primitive I/O level, IIRC. Remember, you can still shoot yourself in the foot if you're doing your own I/O. I know, because I did it myself--ever try to write a 300MB binary as a value in a CouchDB document...I have :) Scott, I did find it quite the coincidence that you posted this shortly after my license post :) I look forward to seeing your solution and comparing with mine. -Ryan From icarus.alive@REDACTED Fri Nov 12 04:21:22 2010 From: icarus.alive@REDACTED (Icarus Alive) Date: Fri, 12 Nov 2010 08:51:22 +0530 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: On Tue, Nov 9, 2010 at 7:08 PM, Edmond Begumisa wrote: > >> - KV stores scale easily > > Largely true. But sometimes it's not as easy as they promise. Some things > work well locally and break when you replicate. And you only realise this > when you try and scale out. There is some element of false advertising when > NoSQL folk promise simple scaling, there should be a big red disclaimer > attached. That's quite a revelation, for someone (me) whose exposure to NoSQL (KV stores) is purely theoretical so far. To me that promise of "scaling easily" was the single biggest motivation, i.e. scaling out in a large distributed cluster environment. Could you elaborate on that point, and share some instances, scenarios, examples of what may be the scaling pitfalls ? From max.lapshin@REDACTED Fri Nov 12 07:03:33 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 12 Nov 2010 09:03:33 +0300 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: References: <44938.1289525253@snookles.snookles.com> Message-ID: > > In would think 5,000,000 msgs on a mailbox would be a sign of a larger > problem and at that point maybe the VM crashing is a good thing? Then > again, I'm not sure. You are wrong. VM can survive such failures, but when failed state is dumping, gen_server tries to send HUGE amount of data (yes, it doesn't format, but it sends all data to error_logger) to error logger and then everything fails. 50000000 of messages is a problem in dumping state: format("** Generic server ~p terminating \n" "** Last message in was ~p~n" "** When Server state == ~p~n" "** Reason for termination == ~n** ~p~n", [Name, Msg, State, Reason1]), Here Msg, State and Reason1 can be horribly big, that is why we _must_ limit them right here and I suppose, that best way is to print into binary inside failing gen_server because there is one error_logger per all cores and it may become a bottleneck. From dgud@REDACTED Fri Nov 12 08:41:40 2010 From: dgud@REDACTED (Dan Gudmundsson) Date: Fri, 12 Nov 2010 08:41:40 +0100 Subject: [erlang-questions] Erlang/OTP debugger on Mac OS X 10.6 Snow Leopard crashes regularly In-Reply-To: References: Message-ID: On Thu, Nov 11, 2010 at 6:32 PM, Alain O'Dea wrote: > I have installed OTP R14B using http://github.com/mxcl/homebrew/. ?It > installs 64-bit ERTS and applications, but it doesn't have wxErlang support > because wxMac 2.8 does not build in 64-bit. > The wxWidgets team is working on that for 3.0. > When I use the debugger it launches a Wish application that appears to use > native OS X widgets. ?However, the debugger is extremely unstable in this > environment. ?It worked much better in the X11 wish in 10.5 and earlier. ?I > can rarely make it through a debugging session without the Wish application > freezing. > > I have successfully restarted debugger by restarting the whole node, but it > rapidly crashes again. ?I have also managed to force the monitor application > to restart with exit(im(), 'EXIT'). ?It seems that it doesn't notice that > wish has crashed. > GS uses the wish from the path, if you put X11 variant first in path that should work. > Is there a reliable way to debug Erlang applications on Mac OS X 10.6 Snow > Leopard? > > I am in the process of building a 32-bit OTP dev linked against a 32-bit > wxWidgets 2.8.11. ?Hopefully this will work, but I still wonder if ErlIDE > would serve me better for this use case. > 2.8.11 is needed on Snow leopard, so it's a good choice, but wx on the mac is still the least tested variant.. If ErlIDE is working for you, it might be a good choice. > IMHO, wxErlang should be a service provided by a C node rather than a > linked-in driver. ?To the best of my knowledge that would make 32/64-bit a > non-issue. ?This would be similar to the WebKit plug-in solution which > allows 32-bit plug-ins (like Flash) in a 64-bit browser. Agree, that wouldn't bring down the node if it crashes either, it is doable but I beleive there is a large time investment before that happens, which I don't have time to do. But that would be great for linux and windows as well. /Dan From ingela.andin@REDACTED Fri Nov 12 10:34:34 2010 From: ingela.andin@REDACTED (Ingela Andin) Date: Fri, 12 Nov 2010 10:34:34 +0100 Subject: [erlang-questions] ANNOUNCE: ibrowse-2.1.0 In-Reply-To: References: <00e701cb80f4$6bc0e3d0$4342ab70$@com> Message-ID: Hi! I suppose I will need to give you some what of the OTP take on this subject. I think it is very hard to give any general advise here. The thing I can say in general is that we are pretty quick at fixing reported bugs but if no one uses it the bugs will not be reported. One thing that we are planning (not making any promising on when though) for the httpc is a simpler "wget-like-API function" as the current client offers more than the need is in many cases, and maybe then ibrowser will feel simpler. Admittingly it was a bad idea to put in some contributed badly tested http-client in the distribution even though it was unsupported at first, just because it was a big demand for having one and we did not have the time make one. However when we change the underlaying implementation we try not to take away any functionality and when looking at ibrowser and the client we had tried to improve they had quite different approaches and replacing what we had then with ibrowse was not trivial and would have required a lot of additions and refactorizations of ibrowse and it did not feel eaiser then to to continue on the improvments we where already doing. (Both clients have evolved since) . Legacy is a bitch and I think we learned the lesson all too well ;) Regards Ingela Erlang/OTP-team, Ericsson AB 2010/11/10 Per Melin : > On Wed, Nov 10, 2010 at 5:29 PM, David Mercer wrote: >> On Wednesday, November 10, 2010, Chandru wrote: >>> ibrowse is a HTTP client written in erlang as an alternative to the >>> built in HTTP client httpc. >> >> What is the difference between the two? ?That is, how do I decide whether to >> use ibrowse or httpc? > > I don't currently use ibrowse but I can say something about httpc; I > don't think there's been a release of OTP since 2008 where it hasn't > had at least one serious bug. Someone please correct me if I'm wrong. > Two times I've had to roll back to older versions of OTP until I could > get/make a patch. Last time was with the httpc memory leak in R13B04. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From roques@REDACTED Fri Nov 12 15:11:44 2010 From: roques@REDACTED (Christian von Roques) Date: Fri, 12 Nov 2010 10:11:44 -0400 Subject: [erlang-questions] [eeps] EEP 35 "Binary string modules" -- locales In-Reply-To: (Kenneth Lundin's message of "Thu, 11 Nov 2010 16:31:53 +0100") References: Message-ID: <87y68yljhb.fsf@pepper.mti.ag> Not all text is meant for human consumption. I'd even venture so far as to say that the overwhelming mass of program generated text is not for human consumption, its intended consumers are other programs. The most common locale programs "speak" is the default "C" (also called "POSIX") locale. It is complicated to solve the general problem of supporting all human locales. It is much simpler to just support a default locale. Even programs intended to create/consume text for humans often have to create/consume text in the C locale as well. I've been told the anecdote that in the 70s a delegation of IBM compiler engineers flew to Germany to proudly demonstrate their new optimizing Fortran compiler and all it did was spew gibberish and crash because it used the standard routines for reading/writing numbers, which in Germany used commas for dots and dots for commas due to the then new locale awareness of the OS. Since then I've been convinced that it is a good thing to have two separate sets of functions, one small, simple, and fast handling only the default locale and another one huge, complicated, and not so fast trying to handle all the intricacies of as many locales as feasible. Therefore I'd like to see to_integer and to_float in bstring, grokking numbers in the C locale. to_lower and to_upper too as long as it's documented on which characters they are working on. They wouldn't even need to know if the bstring was iso8859-1 or utf-8 encoded as long as they only touch ASCII characters. I don't think it's practical to see bstring as locale independent. Rather bstring should be seen as operating in the default locale. One being able to imagine a locale dependent variant of a function should not be ground for omitting the function from bstring. I can even imagine concat(<<"Fu?">>, <<"Ball">>) being expected to result in <<"Fussball">> in the DE_de locale. Christian. From jodie.burch@REDACTED Fri Nov 12 15:44:07 2010 From: jodie.burch@REDACTED (Jodie Burch) Date: Fri, 12 Nov 2010 14:44:07 +0000 Subject: 2010 Erlang User Conference Sold Out! Message-ID: Hi The Erlang User Conference, despite us adding additional places, is now sold out! We have 310 delegates, up from 250 last year. If you missed out on registering, you can email conferences@REDACTED and be added to the standby list. We will contact you in the event of any cancellations. We will also be recording the talks and making them available on the conference site so while you will be missing out on the socialising and networking you can still catch on the latest goings on. You can also follow the conference on twitter #euc2010 and Facebook: http://www.facebook.com/home.php?#!/pages/Erlang-User-Conference/12134671124 7900 For those of you who are attending, we look forward to seeing you there! Thanks Jodie Erlang Solutions Ltd - London www.erlang-solutions.com From ebegumisa@REDACTED Fri Nov 12 17:29:32 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Sat, 13 Nov 2010 03:29:32 +1100 Subject: [erlang-questions] Re: Conceptual questions on key-value databases for RDBMs users In-Reply-To: References: <20101102201437.GC626@hope.tifa.renegado> <820862C6-9E89-49A6-B8A6-5C2548DA64C3@patternmatched.com> <4c5fb16b-23a8-4392-8dae-28e40790453c@k14g2000pre.googlegroups.com> Message-ID: > That's quite a revelation, for someone (me) whose exposure to NoSQL > (KV stores) is purely theoretical so far. To me that promise of > "scaling easily" was the single biggest motivation, i.e. scaling out > in a large distributed cluster environment. A more accurate description would be "scale easIER" than other camps as opposed to "scale easILY". This is not just semantics. Scaling databases is an inherently difficult problem. The NoSQL camp haven't just discovered some new miracle cure. Scalability can can be made easier or harder by the design decisions of _BOTH_ the implementor and user of the db (and there are always trade offs.) It cannot be made easy (with no trade offs.) > Could you elaborate on > that point, and share some instances, scenarios, examples of what may > be the scaling pitfalls ? My issue with the promise of easy scaling is that it is usually worded to imply that you can take an application that is *designed* for local data access to a NoSQL db, sprinkle some NoSQL scalability pixie-dust on it, and suddenly it will run on google's world-wide network. There's usually no consideration given to the fact that scalability needs to be designed into the application *itself* from the word go, not just be made available by the db and there are trade offs involved. To elaborate... ---- Caveat: The following is based on *my* experience as an SQL-RDBMS -> NoSQL/kv-db migrant who has tried a number of dbs in both camps. I'll deliberately not name names here to avoid flames. You can easily investigate further and imply which stores fall where (according to lay me). I speak for myself as a db user and make the rather large assumption that I *may* be speaking for many others like me. I am NOT an expert in distrusted systems. ---- My understanding of scaling in kv-stores/dbs/document-stores (I don't even know what to call these anymore) is rooted in three common requirements/wishes that an RDBMS migrant goes to bed dreaming about and decides to make the NoSQL boat-trip to the promised land. Usually, you look into scaling out when you want either... (A) DISTRIBUTION (don't know if this is the correct term, table-splitting perhaps) What you want: The same class of documents/values stored but not duplicated across several databases (usually on different machines in the same location). e.g I've got a gazillion orders and they won't fit on one machine or render the database very ineffective when querying them, so I want them distributed round-robin or some other way using a cluster of dbs/machines. What you expect: When you do this, your application doesn't need to know. When updating, the db will decide where things should go (say, based on some configuration setting). Queries just get executed across the different databases and the result is given to you. Your code doesn't have to change (I think Mnesia can do this with fragmented tables?) What you get: Very few give you the full Mnesia style. Some give you none. Some give you more. Many CANNOT provide this 'table-splitting' feature AT ALL and instead offer clone replication (below) which is different. Worst case, you have to do extra work to expressly decide where to save data and run the queries across the different databases then manually combine the results. Your code normally has to change. Your code normally gets more complicated. (B) REPLICATION What you want: the same class of documents/values stored and duplicated+synced across several databases (usually on different machines in different locations) e.g I've got several branches of my business and I want a receipt that appears at my Melbourne branch to appear in my Sydney head office so I can export it to the central general ledger. What you don't expect: That you are ignorant on how the CAP Theorem and strong vs eventual consistency will impact on how you write your code. What you get: Code that's written in absence of the informed design decisions required to make eventual consistency work for you. This code has to be re-thought and re-written when you finally decide to replicate. This could be very large chunks of your application especially for someone coming from the SQL-RDBMS world who is used to ACID/release consistency and awakens to the fact that several assumptions in your application suddenly break when one moves to BASE/eventual consistency. ACID(ic) local + BASE(ic) remote = big an impact on your application design+coding. You can't just paste this into your code later. (C) MORE BEEF What you want: queries take forever, your server is way too stressed. What you expect: Throw more hardware at the problem (more cores, memory, etc). The more you throw the better things get. What you get: This seems to be the one area where NoSQL dbs consistently deliver. Some exhibit almost linear improvement. I haven't scientifically benchmarked this myself, but I have noticed with rudimentary tests using different machines I have at my disposal. I understand though that you do arrive to a saturation point where the law of diminishing returns sets in (so I've read), someone else would be better at explaining why. That is something people neglect to mention though. IN THEIR DEFENSE: Expectations are probably not the fault of the NoSQL database. It's just something that one is led into believing by the NoSQL chatter on the internet. That things just magically scale with little effort. That all the well-documented and well-researched problems with distrusted programming/storage are just "somehow taken care of" by the brilliance of NoSQL. The interpretation is that the application programmer doesn't need to do anything. Doesn't need rethink local vs distributed/replicated data for his application. I've found this to be misleading. It's certainly less effort than the alternatives, and certainly works *far* *far* better than alternatives once you get it working, BUT effort and planning *is* required. These things are probably obvious for someone used to working in that world so are rarely mentioned. There's no scalability check-box that you just flip. Things have to be well thought out and consequences have to be planned for and code written in awareness and many times even to implement some aspects. SQL-RDBMS migrants are rarely prepared for this. They are used to the db making all the decisions for them. "Dig in then just scale up later when you need to" is unlikely work. Light reading up on elementary CAP theorem, eventual consistency and the associated trade-offs is essential for an SQL-RDBMS -> NoSQL migrant who is unlikely to have even considered/needed to consider these things in the past. These will greatly affect how you do your part and harmonise it with how the db in question does it's part to get the best (but _never_ ideal) situation. THE DISCLAIMER: Should read: "EasIER scaling BUT the onus is on you to ensure you understand the impact of the CAP theorem and eventual consistency many have and investigate any extra work that may be required for the database in question BEFORE you design let alone code your application". For the NoSQL camp it might be be like "Duh!" But IMO, these things are not obvious for the RDBMS migrant who has been promised easy scaling from a bullet list of features. This renders scalability not easy, just more achievable, with trade-offs. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ On Fri, 12 Nov 2010 14:21:22 +1100, Icarus Alive wrote: > On Tue, Nov 9, 2010 at 7:08 PM, Edmond Begumisa > wrote: >> >>> - KV stores scale easily >> >> Largely true. But sometimes it's not as easy as they promise. Some >> things >> work well locally and break when you replicate. And you only realise >> this >> when you try and scale out. There is some element of false advertising >> when >> NoSQL folk promise simple scaling, there should be a big red disclaimer >> attached. > > That's quite a revelation, for someone (me) whose exposure to NoSQL > (KV stores) is purely theoretical so far. To me that promise of > "scaling easily" was the single biggest motivation, i.e. scaling out > in a large distributed cluster environment. Could you elaborate on > that point, and share some instances, scenarios, examples of what may > be the scaling pitfalls ? > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From hynek@REDACTED Fri Nov 12 17:52:21 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Fri, 12 Nov 2010 17:52:21 +0100 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: <40904.1289520413@snookles.snookles.com> References: <40904.1289520413@snookles.snookles.com> Message-ID: I think that init([]) -> {ok, #state{big = lists:foldl(fun(_, X) -> [X|X] end, $s, lists:seq(1,50))}. should be enough to kill it in easy, but not tested yet. Yes, it is little bit unfair. On Fri, Nov 12, 2010 at 1:06 AM, Scott Lystig Fritchie wrote: > Max Lapshin wrote: > > ml> You have to modify gen_server to make error-proof message logging: > ml> https://github.com/erlyvideo/erlyvideo/blob/master/src/core/gen_server_ems.erl#L747 > > Max, my reading of the code says that the formatting is not done by the > gen_server process ... instead, the gen_server sends the format string & > args to error_logger, and error_logger's event handlers do the actual > formatting. > > Included at the end of this message is a bunch of testing that I did > with a very intentionally buggy piece of code called foo.erl. ?I use > three different variations to try to get it to mis-behave. ?I can't get > riak_err to format something that's "too "big". ?If someone can find a > counter-example (preferably an executable one :-), I'd love to hear it. > > If I use gen_server:start_link() instead, then the {'EXIT', ...} being > caught by the shell will cause lib:format() to be executed, and *that* > can cause huge messages to be formatted. ?But the (slightly sad) moral > of the story is to be careful of what's linked to the shell ... and > that's a lesson that most Erlang folks end up learning anyway. > > ml> [...] or yours one is better, but I want OTP team to communicate > ml> about it. ?gen_server:format_state is not enough because of dumping > ml> message queue and error reason (it includes dumping state). Problem > ml> is larger and it requires looking into all gen_ code and proc_lib > ml> infrastructure. > > Agreed. ?The default should be to have some kind of size limits (and > therefore RAM-consuming limit) on any OTP-generated event/message/etc. > If that default also helps allow my/your/others' code (and their events) > to also observe size limits when formatting events(*), that'd be even > better. > > -Scott > > (*) Apologies to Arthur C. Clarke: "I'm sorry, Dave, I can't allow you > to point that gun at your foot. ?Here's a smaller thing you can use > instead." > > --- snip --- snip --- snip --- snip --- snip --- snip --- > > Common steps: > > ? ?riak_err_handler:set_fmt_max_bytes(100). > ? ?riak_err_handler:set_term_max_size(100). > ? ?f(Pid). > ? ?{ok, Pid} = foo:start(). > > Variation #1: > > ? ?foo:crash(Pid). > > Output: > > ? ?(riak@REDACTED)164> ? ? foo:crash(Pid). > ? ?byebye > ? ?(riak@REDACTED)165> > ? ?=ERROR REPORT==== 11-Nov-2010::17:48:04 === > ? ?Oversize args for format "** Generic server ~p terminating > ? ?** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n": [<0.3676.1>,{crash,"mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm..."}...] > ? ?=CRASH REPORT==== 11-Nov-2010::17:48:04 === > ? ? ?crasher: > ? ? ? ?initial call: foo:init/1 > ? ? ? ?pid: <0.3676.1> > ? ? ? ?registered_name: [] > ? ? ? ?exception exit: {call_was,{crash,"mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm..."}...} > ? ? ? ? ?in function ?gen_server:handle_msg/5 > ? ? ? ? ?in call from proc_lib:init_p_do_apply/3 > ? ? ? ?ancestors: [<0.3605.1>] > ? ? ? ?messages: [] > ? ? ? ?links: [] > ? ? ? ?dictionary: [] > ? ? ? ?trap_exit: false > ? ? ? ?status: running > ? ? ? ?heap_size: 17711 > ? ? ? ?stack_size: 24 > ? ? ? ?reductions: 6774 > ? ? ?neighbours: > > Variation #2: > > ? ?gen_server:cast(Pid, please_crash). > > Output: > > ? ?(riak@REDACTED)169> ? ? gen_server:cast(Pid, please_crash). > ? ?ok > ? ?(riak@REDACTED)170> > ? ?=ERROR REPORT==== 11-Nov-2010::17:49:18 === > ? ?Oversize args for format "** Generic server ~p terminating > ? ?** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n": [<0.4236.1>,{$gen_cast,please_crash},{state,"sssssssssssssssssssssssssssssssssssssssssssssssssss..."}...] > ? ?=CRASH REPORT==== 11-Nov-2010::17:49:18 === > ? ? ?crasher: > ? ? ? ?initial call: foo:init/1 > ? ? ? ?pid: <0.4236.1> > ? ? ? ?registered_name: [] > ? ? ? ?exception exit: {bummer,please_crash,<<"cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc...">>} > ? ? ? ? ?in function ?gen_server:terminate/6 > ? ? ? ? ?in call from proc_lib:init_p_do_apply/3 > ? ? ? ?ancestors: [<0.3605.1>] > ? ? ? ?messages: [] > ? ? ? ?links: [] > ? ? ? ?dictionary: [] > ? ? ? ?trap_exit: false > ? ? ? ?status: running > ? ? ? ?heap_size: 10946 > ? ? ? ?stack_size: 24 > ? ? ? ?reductions: 6373 > ? ? ?neighbours: > > Variation #3: > > ? ?Pid ! {arbitrary_message, lists:duplicate(2*1024, $a)}, atom_here_to_avoid_confusion_with_bangs_return_value_at_the_shell. > > Output: > > ? ?(riak@REDACTED)174> ? ? Pid ! {arbitrary_message, lists:duplicate(2*1024, $a)}, atom_here_to_avoid_confusion_with_bangs_return_value_at_the_shell. > ? ?atom_here_to_avoid_confusion_with_bangs_return_value_at_the_shell > > ? ?=ERROR REPORT==== 11-Nov-2010::17:50:19 === > ? ?Oversize args for format "~w: ~s:handle_info got ~w > ? ?": [<0.4648.1>,foo,{arbitrary_message,"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..."}] > ? ?(riak@REDACTED)175> > > Code for foo.erl follows. > > -module(foo). > -author('fritchie@REDACTED'). > > -behaviour(gen_server). > > -define(NAME, ?MODULE). > -define(Timeout, infinity). > > %% External exports > -export([start/0]). > -export([crash/1]). > > %% gen_server callbacks > -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, > ? ? ? ? code_change/3]). > > -record(state, {big}). > > start() -> > ? ?gen_server:start(?MODULE, [], []). > > crash(Pid) -> > ? ?gen_server:call(Pid, {crash, lists:duplicate(2*1024, $m)}). > > init([]) -> > ? ?{ok, #state{big = lists:duplicate(2*1024, $s)}}. > > handle_call(Request, _From, State) -> > ? ?{stop, {call_was, Request, lists:duplicate(2*1024, $z)}, byebye, State}. > > handle_cast(Msg, _State) -> > ? ?exit({bummer, Msg, list_to_binary(lists:duplicate(2*1024, $c))}). > > handle_info(Info, State) -> > ? ?error_logger:error_msg("~w: ~s:handle_info got ~w\n", [self(), ?MODULE, Info]), > ? ?{noreply, State}. > > terminate(_Reason, _State) -> > ? ?ok. > > code_change(_X, _Y, Z) -> > ? ?{ok, Z}. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From fritchie@REDACTED Fri Nov 12 18:57:02 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Fri, 12 Nov 2010 11:57:02 -0600 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: Message of "Fri, 12 Nov 2010 09:03:33 +0300." Message-ID: <95224.1289584622@snookles.snookles.com> Max Lapshin wrote: ml> format("** Generic server ~p terminating \n" ... ml> Here Msg, State and Reason1 can be horribly big, that is why we ml> _must_ limit them right here and ... "Must"? Hmmm. That depends. I agree that it would be a good idea. But looking at the R13B04 code, that format() call is actually a call to lib:format(), which sends a message to error_logger with the format string & args, and then error_logger does the actual formatting. (I hope the R14B code doesn't change this pattern....) If the arguments are big, say X megabytes of heap space, then yes it's bad that you'll copy X megabytes of stuff onto error_logger's heap via sending a message. But X is a much smaller number than the X*ReallyBadFactor number that we get when io_lib:format() does its naive thing when formatting the args. ml> ... and I suppose, that best way is to print into binary inside ml> failing gen_server because there is one error_logger per all cores ml> and it may become a bottleneck. The Hibari project's logging code attempts to do that, since (ignoring memory for a moment) the formatting is much more CPU-intensive (and can be done in parallel) than the writing-to-TTY or writing-to-disk stuff (serialized by gen_event). I don't know if it would be a positive thing to have the client (i.e., the "failing gen_server") use a iolist_to_binary() to flatten the formatted message into a single binary (which, if larger than 64 bytes) will be stored in the shared binary heap and thus make the message sent to error_logger much cheaper. In theory. Haven't measured it. Then there's also the matter of log-level handling, which gets tricky if the client is making the "should I send this message to error_logger?" decision. Assuming a syslog-style logging scheme, if the minimum logging level is LOG_INFO, then ... the client needs to discover that logging level before it can decide what to do with a severity LOG_CRIT or LOG_DEBUG message. What if The Administrator changes the logging level? What if the gen_event server has both syslog-style and non-syslog-style event handler? {sigh} Complexity is, er, um, tough. -Scott From max.lapshin@REDACTED Fri Nov 12 19:00:26 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Fri, 12 Nov 2010 21:00:26 +0300 Subject: [erlang-questions] New OTP app available at GitHub: riak_err In-Reply-To: <95224.1289584622@snookles.snookles.com> References: <95224.1289584622@snookles.snookles.com> Message-ID: > > Then there's also the matter of log-level handling, which gets tricky if > the client is making the "should I send this message to error_logger?" > decision. ?Assuming a syslog-style logging scheme, if the minimum Take a look at log4erl. It dynamically rebuilds module that will throw away too verbose log according to config From co7eb@REDACTED Sat Nov 13 08:37:08 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Sat, 13 Nov 2010 02:37:08 -0500 Subject: Erlang shows its slow face! Message-ID: Hi all! I have been doing tests to Erlang I found this funny stuff that makes Pythagorean Triplets pythag(N) -> [ {A,B,C} || A <- lists:seq(1,N), B <- lists:seq(1,N), C <- lists:seq(1,N), A+B+C =< N, A*A+B*B =:= C*C]. I tested it agains an implementation I made in C# so, and takes 14 secounds in my pc to do with 300 numbers in Erlang however in c# is just a secound, even when C# runs under VM too. So I did all possible ways for me to implement differents manners in Erlang looking for speed and all is the same, listed as follows: So my question is, there are any way to do that even more fast, why 3 nestes fors structs in C# are more effients that lists:foldr or lists:foreach in Erlang. %% FORMA 1 py1(Max)-> L = lists:seq(1, Max), lists:foldr( fun(A, Acc3)-> lists:foldr( fun(B, Acc2)-> lists:foldr( fun(C, Acc)-> case ((A*A + B*B =:= C*C) andalso (A+B+C =< Max)) of true-> [{A,B,C}|Acc]; false-> Acc end end , Acc2, L) end , Acc3, L) end , [], L). %% FORMA 2 py2(Max)-> fora(1, [], Max). fora(A, Acc, Max)-> Acc1 = forb(A,1, Acc, Max), case A < Max of true-> fora(A+1, Acc1, Max); false-> Acc1 end. forb(A,B, Acc, Max)-> Acc1 = forc(A,B,1, Acc, Max), case B < Max of true-> forb(A,B+1, Acc1, Max); false-> Acc1 end. forc(A,B,C, Acc, Max)-> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true-> [{A,B,C}|Acc]; _-> Acc end, case C < Max of true-> forc(A,B,C+1, Acc1, Max); false-> Acc1 end. %% FORMA 3. py3(Max)-> [{A,B,C} || A <-lists:seq(1, Max), B <-lists:seq(1, Max), C <-lists:seq(1, Max), A*A + B*B =:= C*C, A+B+C =< Max]. ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From als@REDACTED Sat Nov 13 09:10:28 2010 From: als@REDACTED (Anthony Shipman) Date: Sat, 13 Nov 2010 19:10:28 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: <201011131910.28926.als@iinet.net.au> On Sat, 13 Nov 2010 06:37:08 pm Gilberto Carmenate Garc?a wrote: > So my question is, there are any way to do that even more fast, why 3 > nestes fors structs in C# are more effients that lists:foldr or > lists:foreach in Erlang. It goes 10 times faster when compiled to native code. The following takes 0.8s on my machine. pythag2(N) -> L = lists:seq(1,N), [ {A,B,C} || A <- L, B <- L, C <- L, A+B+C =< N, A*A+B*B =:= C*C]. -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From bernie@REDACTED Sat Nov 13 09:39:59 2010 From: bernie@REDACTED (Bernard Duggan) Date: Sat, 13 Nov 2010 19:39:59 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: <4CDE4EDF.6060500@m5net.com> On 13/11/10 18:37, Gilberto Carmenate Garc?a wrote: > Hi all! > I have been doing tests to Erlang I found this funny stuff that makes > Pythagorean Triplets > > pythag(N) -> > [ {A,B,C} || > A<- lists:seq(1,N), > B<- lists:seq(1,N), > C<- lists:seq(1,N), > A+B+C =< N, > A*A+B*B =:= C*C]. > > I tested it agains an implementation I made in C# so, and takes 14 > secounds in my pc to do with 300 numbers in Erlang however in c# is just > a secound, even when C# runs under VM too. > So I did all possible ways for me to implement differents manners in > Erlang looking for speed and all is the same, listed as follows: > > So my question is, there are any way to do that even more fast, why 3 > nestes fors structs in C# are more effients that lists:foldr or > lists:foreach in Erlang. > Okay, the first and most obvious improvement to that code above is to change it slightly to: pythag(N) -> L = lists:seq(1,N), [ {A,B,C} || A <- L, B <- L, C <- L, A+B+C =< N, A*A+B*B =:= C*C]. That sped it up on my laptop from 4 seconds to 3.5. If you're really after speed in erlang, though, a good thing to try is HiPE. Add "+native" to your erlc command options. That sped it up (for me) to 1 second for your code and .5 of a second for my version. Bear in mind that HiPE carries tradeoffs in terms of tracing/debugging etc. As far as your question, though, the first thing I'd say is that if you're using Erlang for high-performance numerical analysis, you've chosen the wrong language. I love Erlang to bits, but that's not its strength. Part of this stems from the fact that when you have a number (A, B, C), it's always treated as an having an arbitrary size (people more familiar with the guts of the VM please feel free to correct me here...). That means that if you want to store more than 32 or 64 bits in it, you can do so with zero modification to your code. That's brilliant and can be really useful, but the tradeoff is the performance hit you take on any arithmetic operation - and it's that hit, I suspect, that you're seeing here. Cheers, B From alexey.v.romanov@REDACTED Sat Nov 13 12:19:46 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Sat, 13 Nov 2010 14:19:46 +0300 Subject: Separate messages from stdout and stderr of a port Message-ID: I can get the output of a spawned port on its stderr stream by using stderr_to_stdout. However, is there a way to get messages from the port's stdout and stderr separately? I don't see one in http://www.erlang.org/doc/man/erlang.html In fact, it would be enough for my purposes to only get stderr and ignore stdout. Yours, Alexey Romanov From ebegumisa@REDACTED Sat Nov 13 15:36:07 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Sun, 14 Nov 2010 01:36:07 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: Hello Gilberto, Just an observation... When one approaches a language, programming environment or development tool, it's important to establish it's promises. Then judge it on how well it delivers on those promises. One shouldn't be surprised when it fails to deliver on something it has not promised. One should be suspicious if it promises everything. For Erlang's case, I think the official FAQ sums it up pretty well... "...The most common class of 'less suitable' problems is characterised by performance being a prime requirement and constant-factors having a large effect on performance..." http://www.erlang.org/faq/introduction.html#id50247 http://www.erlang.org/faq/introduction.html#id49850 It would be odd if you found Erlang was slow at creating processes, or sending messages between processes, or the emulator kept crashing. That would indicate that it's failing to deliver on what it has promised. Numerical algorithms? I don't think even C# makes any promises there. You should probably be looking for a math library like AMD's ACML with gcc/gfortran... http://developer.amd.com/cpu/Libraries/acml/Pages/default.aspx I'm no mathematician, but I experimented with ACML once when I was toying around with programming audio processor plugins for music production software. It made a world of difference (even though my plugins sucked). If for some reason want to do high performance math from Erlang, you could look into writing a port, a linked-in driver or a NIF with a more appropriate set of tools then call that from Erlang. - Edmond - On Sat, 13 Nov 2010 18:37:08 +1100, Gilberto Carmenate Garc?a wrote: > Hi all! > I have been doing tests to Erlang I found this funny stuff that makes > Pythagorean Triplets > > pythag(N) -> > [ {A,B,C} || > A <- lists:seq(1,N), > B <- lists:seq(1,N), > C <- lists:seq(1,N), > A+B+C =< N, > A*A+B*B =:= C*C]. > > I tested it agains an implementation I made in C# so, and takes 14 > secounds in my pc to do with 300 numbers in Erlang however in c# is just > a secound, even when C# runs under VM too. > So I did all possible ways for me to implement differents manners in > Erlang looking for speed and all is the same, listed as follows: > > So my question is, there are any way to do that even more fast, why 3 > nestes fors structs in C# are more effients that lists:foldr or > lists:foreach in Erlang. > > %% FORMA 1 > py1(Max)-> > L = lists:seq(1, Max), > lists:foldr( > fun(A, Acc3)-> > lists:foldr( > fun(B, Acc2)-> > lists:foldr( > fun(C, Acc)-> > case ((A*A + B*B =:= C*C) andalso (A+B+C =< > Max)) of > true-> > [{A,B,C}|Acc]; > false-> > Acc > end > end > , Acc2, L) > end > , Acc3, L) > end > , [], L). > > > > %% FORMA 2 > py2(Max)-> > fora(1, [], Max). > > fora(A, Acc, Max)-> > Acc1 = forb(A,1, Acc, Max), > case A < Max of > true-> > fora(A+1, Acc1, Max); > false-> > Acc1 > end. > > forb(A,B, Acc, Max)-> > Acc1 = forc(A,B,1, Acc, Max), > case B < Max of > true-> > forb(A,B+1, Acc1, Max); > false-> > Acc1 > end. > > forc(A,B,C, Acc, Max)-> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true-> > [{A,B,C}|Acc]; > _-> > Acc > end, > case C < Max of > true-> > forc(A,B,C+1, Acc1, Max); > false-> > Acc1 > end. > > %% FORMA 3. > py3(Max)-> > [{A,B,C} || > A <-lists:seq(1, Max), > B <-lists:seq(1, Max), > C <-lists:seq(1, Max), > A*A + B*B =:= C*C, > A+B+C =< Max]. > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electronico > que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para > respaldar el cumplimiento de los objetivos de la organizacion y su > politica informativa. La persona que envia este correo asume el > compromiso de usar el servicio a tales fines y cumplir con las > regulaciones establecidas. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From jameschurchman@REDACTED Sat Nov 13 17:31:25 2010 From: jameschurchman@REDACTED (James Churchman) Date: Sat, 13 Nov 2010 16:31:25 +0000 Subject: [erlang-questions] Erlang/OTP debugger on Mac OS X 10.6 Snow Leopard crashes regularly In-Reply-To: References: Message-ID: <49F90974-F5EF-4751-B14E-35BC751A1E8C@gmail.com> I am using r13 and the debugger is just as unstable. Not only that its a horrid piece of software even if it did work correctly!! I can 100 % vouch for ErlIDE, it works great on os x,is simple to setup and takes a instant to add breakpoints. You can get the most up-to-date version by adding http://erlide.org/update_nightly to the "add new software" menu option. I dont like Eclipse one bit for actual code editing, bit big and bloated, so i use TextMate with a number of keyboard shortcuts to erlc to compile source & check for errors and to do a "jump to definition in project" and then ErlIDE to do debugging. This setup works perfectly for me James On 12 Nov 2010, at 07:41, Dan Gudmundsson wrote: > On Thu, Nov 11, 2010 at 6:32 PM, Alain O'Dea wrote: >> I have installed OTP R14B using http://github.com/mxcl/homebrew/. It >> installs 64-bit ERTS and applications, but it doesn't have wxErlang support >> because wxMac 2.8 does not build in 64-bit. >> > > The wxWidgets team is working on that for 3.0. > >> When I use the debugger it launches a Wish application that appears to use >> native OS X widgets. However, the debugger is extremely unstable in this >> environment. It worked much better in the X11 wish in 10.5 and earlier. I >> can rarely make it through a debugging session without the Wish application >> freezing. >> >> I have successfully restarted debugger by restarting the whole node, but it >> rapidly crashes again. I have also managed to force the monitor application >> to restart with exit(im(), 'EXIT'). It seems that it doesn't notice that >> wish has crashed. >> > > GS uses the wish from the path, if you put X11 variant first in path > that should work. > >> Is there a reliable way to debug Erlang applications on Mac OS X 10.6 Snow >> Leopard? >> >> I am in the process of building a 32-bit OTP dev linked against a 32-bit >> wxWidgets 2.8.11. Hopefully this will work, but I still wonder if ErlIDE >> would serve me better for this use case. >> > > 2.8.11 is needed on Snow leopard, so it's a good choice, but wx on the mac is > still the least tested variant.. > > If ErlIDE is working for you, it might be a good choice. > >> IMHO, wxErlang should be a service provided by a C node rather than a >> linked-in driver. To the best of my knowledge that would make 32/64-bit a >> non-issue. This would be similar to the WebKit plug-in solution which >> allows 32-bit plug-ins (like Flash) in a 64-bit browser. > > Agree, that wouldn't bring down the node if it crashes either, > it is doable but I beleive there is a large time investment before > that happens, > which I don't have time to do. > But that would be great for linux and windows as well. > > /Dan > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From tony@REDACTED Sat Nov 13 16:59:18 2010 From: tony@REDACTED (Tony Rogvall) Date: Sat, 13 Nov 2010 16:59:18 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: Hi Gilberto! I did some rewrite of the FORM 2, that I think is more compatible with the condition that A+B+C =< N, and the result is of course much better. How did you implement the C# version ? My results for N = 300, times are measured with timer:tc. pythag0: 4837528 (4.8s) pythag1: 4289769 (4.3s) pythag2: 703632 (0.7s) And the result with +native pythag0: 1006865 (1.0s) pythag1: 441254 (0.4s) pythag2: 107387 (0.1s) /Tony My versions (original + the version that only generates the list once) -module(pythag). -compile(export_all). pythag0(N) -> [ {A,B,C} || A <- lists:seq(1,N), B <- lists:seq(1,N), C <- lists:seq(1,N), A+B+C =< N, A*A+B*B =:= C*C]. pythag1(N) -> L = lists:seq(1,N), [ {A,B,C} || A <- L, B <- L, C <- L, A+B+C =< N, A*A+B*B =:= C*C]. pythag2(N) -> lists:reverse(pythan2_A(1, N, [])). pythan2_A(A, N, Acc) when A > N -> Acc; pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). pythan2_B(A, B, N, Acc) when A+B > N -> Acc; pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). pythan2_C(A, B, C, N, Acc) when A+B+C > N -> Acc; pythan2_C(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_C(A, B, C+1, N, Acc) end. On 13 nov 2010, at 08.37, Gilberto Carmenate Garc?a wrote: > Hi all! > I have been doing tests to Erlang I found this funny stuff that makes > Pythagorean Triplets > > pythag(N) -> > [ {A,B,C} || > A <- lists:seq(1,N), > B <- lists:seq(1,N), > C <- lists:seq(1,N), > A+B+C =< N, > A*A+B*B =:= C*C]. > > I tested it agains an implementation I made in C# so, and takes 14 > secounds in my pc to do with 300 numbers in Erlang however in c# is just > a secound, even when C# runs under VM too. > So I did all possible ways for me to implement differents manners in > Erlang looking for speed and all is the same, listed as follows: > > So my question is, there are any way to do that even more fast, why 3 > nestes fors structs in C# are more effients that lists:foldr or > lists:foreach in Erlang. > > %% FORMA 1 > py1(Max)-> > L = lists:seq(1, Max), > lists:foldr( > fun(A, Acc3)-> > lists:foldr( > fun(B, Acc2)-> > lists:foldr( > fun(C, Acc)-> > case ((A*A + B*B =:= C*C) andalso (A+B+C =< > Max)) of > true-> > [{A,B,C}|Acc]; > false-> > Acc > end > end > , Acc2, L) > end > , Acc3, L) > end > , [], L). > > > > %% FORMA 2 > py2(Max)-> > fora(1, [], Max). > > fora(A, Acc, Max)-> > Acc1 = forb(A,1, Acc, Max), > case A < Max of > true-> > fora(A+1, Acc1, Max); > false-> > Acc1 > end. > > forb(A,B, Acc, Max)-> > Acc1 = forc(A,B,1, Acc, Max), > case B < Max of > true-> > forb(A,B+1, Acc1, Max); > false-> > Acc1 > end. > > forc(A,B,C, Acc, Max)-> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true-> > [{A,B,C}|Acc]; > _-> > Acc > end, > case C < Max of > true-> > forc(A,B,C+1, Acc1, Max); > false-> > Acc1 > end. > > %% FORMA 3. > py3(Max)-> > [{A,B,C} || > A <-lists:seq(1, Max), > B <-lists:seq(1, Max), > C <-lists:seq(1, Max), > A*A + B*B =:= C*C, > A+B+C =< Max]. > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > "Have run Make so many times I dunno what's installed anymore" From hynek@REDACTED Sat Nov 13 20:30:18 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Sat, 13 Nov 2010 20:30:18 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: On Sat, Nov 13, 2010 at 4:59 PM, Tony Rogvall wrote: > Hi Gilberto! > > I did some rewrite of the FORM 2, that I think is more compatible with the condition that A+B+C =< N, > and the result is of course much better. How did you implement the C# version ? > > My results for N = 300, times are measured with timer:tc. > > pythag0: ? ? ? ? ? ? ? ?4837528 ? ? ? ? (4.8s) > pythag1: ? ? ? ? ? ? ? ?4289769 ? ? ? ? (4.3s) > pythag2: ? ? ? ? ? ? ? ?703632 ? ? ? ? ? ? ? ? ?(0.7s) > > And the result with +native > > pythag0: ? ? ? ? ? ? ? ?1006865 ? ? ? ? (1.0s) > pythag1: ? ? ? ? ? ? ? ?441254 ? ? ? ? ? ? ? ? ?(0.4s) > pythag2: ? ? ? ? ? ? ? ?107387 ? ? ? ? ? ? ? ? ?(0.1s) > > /Tony > > My versions (original + the version that only generates the list once) > > -module(pythag). > > -compile(export_all). > > pythag0(N) -> > ? ?[ {A,B,C} || > ? ? ? ?A <- lists:seq(1,N), > ? ? ? ?B <- lists:seq(1,N), > ? ? ? ?C <- lists:seq(1,N), > ? ? ? ?A+B+C =< N, > ? ? ? ?A*A+B*B =:= C*C]. > > pythag1(N) -> > ? ?L = lists:seq(1,N), > ? ?[ {A,B,C} || > ? ? ? ?A <- L, > ? ? ? ?B <- L, > ? ? ? ?C <- L, > ? ? ? ?A+B+C =< N, > ? ? ? ?A*A+B*B =:= C*C]. > > > pythag2(N) -> > ? ?lists:reverse(pythan2_A(1, N, [])). > > pythan2_A(A, N, Acc) when A > N -> Acc; > pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). > > pythan2_B(A, B, N, Acc) when A+B > N -> Acc; > pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). > > pythan2_C(A, B, C, N, Acc) when A+B+C > N -> Acc; > pythan2_C(A, B, C, N, Acc) -> > ? ?if A*A+B*B =:= C*C -> > ? ? ? ? ? ?pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); > ? ? ? true -> > ? ? ? ? ? ?pythan2_C(A, B, C+1, N, Acc) > ? ?end. > > Simpler and about 5% faster version: pythag3(N) when is_integer(N) -> pythag3(N,1). pythag3(N, A) when A+2 > N -> []; pythag3(N, A) -> pythag3(N, A, 1). pythag3(N, A, B) when A+B+1 > N -> pythag3(N, A+1); pythag3(N, A, B) -> pythag3(N, A, B, 1). pythag3(N, A, B, C) when A+B+C > N -> pythag3(N, A, B+1); pythag3(N, A, B, C) when A*A + B*B =:= C*C -> [{A, B, C}|pythag3(N, A, B, C+1)]; pythag3(N, A, B, C) -> pythag3(N, A, B, C+1). > > > > On 13 nov 2010, at 08.37, Gilberto Carmenate Garc?a wrote: > >> Hi all! >> I have been doing tests to Erlang I found this funny stuff that makes >> Pythagorean Triplets >> >> pythag(N) -> >> ? ?[ {A,B,C} || >> ? ? ? ?A <- lists:seq(1,N), >> ? ? ? ?B <- lists:seq(1,N), >> ? ? ? ?C <- lists:seq(1,N), >> ? ? ? ?A+B+C =< N, >> ? ? ? ?A*A+B*B =:= C*C]. >> >> I tested it agains an implementation I made in C# so, and takes 14 >> secounds in my pc to do with 300 numbers in Erlang however in c# is just >> a secound, even when C# runs under VM too. >> So I did all possible ways for me to implement differents manners in >> Erlang looking for speed and all is the same, listed as ?follows: >> >> So my question is, there are any way to do that even more fast, why 3 >> nestes fors structs in C# are more effients that lists:foldr or >> lists:foreach in Erlang. >> >> %% FORMA 1 >> py1(Max)-> >> ? ?L = lists:seq(1, Max), >> ? ?lists:foldr( >> ? ? ? ?fun(A, Acc3)-> >> ? ? ? ? ? ?lists:foldr( >> ? ? ? ? ? ? ? ?fun(B, Acc2)-> >> ? ? ? ? ? ? ? ? ? ?lists:foldr( >> ? ? ? ? ? ? ? ? ? ? ? ?fun(C, Acc)-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ?case ((A*A + B*B =:= C*C) andalso (A+B+C =< >> Max)) of >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{A,B,C}|Acc]; >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?false-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Acc >> ? ? ? ? ? ? ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ? ? ? ? ?, Acc2, L) >> ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ?, Acc3, L) >> ? ? ? ?end >> ? ?, [], L). >> >> >> >> %% FORMA 2 >> py2(Max)-> >> ? ? ? fora(1, [], Max). >> >> fora(A, Acc, Max)-> >> ? ? ? Acc1 = forb(A,1, Acc, Max), >> ? ? ? case A < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?fora(A+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> forb(A,B, Acc, Max)-> >> ? ?Acc1 = forc(A,B,1, Acc, Max), >> ? ?case B < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?forb(A,B+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> forc(A,B,C, Acc, Max)-> >> ? ?Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of >> ? ? ? ?true-> >> ? ? ? ? ? ?[{A,B,C}|Acc]; >> ? ? ? ?_-> >> ? ? ? ? ? ?Acc >> ? ?end, >> ? ?case C < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?forc(A,B,C+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> %% FORMA 3. >> py3(Max)-> >> ? ? ? [{A,B,C} || >> ? ? ? ? ? ? ? A <-lists:seq(1, Max), >> ? ? ? ? ? ? ? B <-lists:seq(1, Max), >> ? ? ? ? ? ? ? C <-lists:seq(1, Max), >> ? ? ? ? ? ? ? A*A + B*B =:= C*C, >> ? ? ? ? ? ? ? A+B+C =< Max]. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de ?usar el servicio a tales fines y cumplir con las regulaciones establecidas. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > "Have run Make so many times I dunno what's installed anymore" > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From ebegumisa@REDACTED Sun Nov 14 01:33:25 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Sun, 14 Nov 2010 11:33:25 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: PARALLELISE, PARALLELISE, PARALLELISE!!! I took your second algo and Tony's improved version (both pretty much unchanged) and parallelised them below with little effort. Permutations for B and C are calculated on a separate concurrent* process for every A then the result is combined (e.g when N = 300, 300 concurrent worker processes are spawned.) I consistently obtained a 40-45% improvement in execution time regardless of algo. I also noticed that non-parallel versions were using only ONE of my cores whereas parallel versions were using BOTH my cores. Here are the results for N = 300 (debug_info BEAM complied)... Sequential Concurrent Your original 8.44s N/A Your py2 9.86s 5.51s Tony's pythag2 1.95s 1.17s** * This is just to illustrate that with any language, one should focus on it's promises/strengths. Sometimes this can even de-emphasise it's weaknesses. Though I still think it's best to use a tool that makes pledges towards the problem area in the first place. ** This is probably roughly as fast the C# implementation but that's besides the point! I agree with Bernard. IMO, you're barking up the wrong tree for doing this sort of thing quickly using either Erlang or C#. When you look closer, the bottleneck with all the solutions so far isn't the calculation itself (I was wrong about that earlier) -- it's actually the permutation (the part done with accumulators/generators). Possibly a language with pointers would be better at doing this fast since you'd be able to allocate a fixed-size destructive memory structure to walk through and modify (that's just an educated guess). I'd put my money on the horse named C NIF. Anyways, here's the parallelised code... NB: lpmap is a library function best stuck in a separate utility module. It's my modified + edocumented version of Joe Armstrong's pmap from his book "Programming Erlang: Software for a Concurrent World" Section 20.2 "Parallelizing Sequential Code" (go buy it!) The original source is at http://www.pragprog.com/titles/jaerlang/source_code/ ---- START CODE ---- py2(Max)-> lists:flatten(lpmap(fun(A) -> forb(A, 1, [], Max) end, lists:seq(1, Max), ordered)). forb(A, B, Acc, Max) -> Acc1 = forc(A, B, 1, Acc, Max), case B < Max of true -> forb(A, B+1, Acc1, Max); false -> Acc1 end. forc(A, B, C, Acc, Max) -> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true -> [{A,B,C}|Acc]; _ -> Acc end, case C < Max of true-> forc(A, B, C+1, Acc1, Max); false-> Acc1 end. pythag2(N)-> lists:flatten(lpmap(fun(A) -> pythan2_B(A, 1, N, []) end, lists:seq(1, N), ordered)). pythan2_A(A, N, Acc) when A > N -> Acc; pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). pythan2_B(A, B, N, Acc) when A+B > N -> Acc; pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). pythan2_C(A, B, C, N, Acc) when A+B+C > N -> Acc; pythan2_C(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_C(A, B, C+1, N, Acc) end. %% @spec lpmap(fun(), list(), (atom() = ordered|unordered)) -> list() %% @doc Spawns a process for each element in list L, performs specified %% function F against each in parallel and then returns results either %% same order as L (ordered) or in any order (unordered). %% NB: See also lpmap/4. lpmap(F, L, ordered) -> Ref = erlang:make_ref(), Pids = [lpmap_spawn_link(self(), Ref, F, I) || I <- L], lpmap_gather_ordered(Pids, Ref, [], 0, void); lpmap(F, L, unordered) -> Ref = erlang:make_ref(), lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, L), lpmap_gather_unordered(length(L), Ref, [], 0, void). %% @spec lpmap(fun(), integer(), list(), (atom() = ordered|unordered)) -> list() %% @doc Same as lpmap/3 except ensures only a maximum of MaxPs parallel %% processes execute function F at any one time (i.e. first takes MaxPs %% items from list, executes F in parallel against each, then as each %% process returns, spawns another process on next item in L as long as %% active processes are less than MaxPs). %% NB: See also lpmap/4. lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L) > MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, Pids = [lpmap_spawn_link(self(), Ref, F, I) || I <- HPids], lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L) > MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, HPids), lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). %% lpmap internal functions lpmap_spawn_link(Parent, Ref, F, I) -> spawn_link(fun() -> Parent ! {self(), Ref, F(I)} end). lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> []; lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> receive {HPid, Ref, Ret} when length(TPids) [H | T] = L, [Ret | lpmap_gather_ordered( lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, H)]), Ref, T, MaxPs, F)]; {HPid, Ref, Ret} -> [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] end. lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> []; lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> receive {_Pid, Ref, Ret} when NPs-1 [H | T] = L, lpmap_spawn_link(self(), Ref, F, H), [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; {_Pid, Ref, Ret} -> [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] end. ---- END CODE ----- - Edmond - On Sun, 14 Nov 2010 02:59:18 +1100, Tony Rogvall wrote: > Hi Gilberto! > > I did some rewrite of the FORM 2, that I think is more compatible with > the condition that A+B+C =< N, > and the result is of course much better. How did you implement the C# > version ? > > My results for N = 300, times are measured with timer:tc. > > pythag0: 4837528 (4.8s) > pythag1: 4289769 (4.3s) > pythag2: 703632 (0.7s) > > And the result with +native > > pythag0: 1006865 (1.0s) > pythag1: 441254 (0.4s) > pythag2: 107387 (0.1s) > > /Tony > > My versions (original + the version that only generates the list once) > > -module(pythag). > > -compile(export_all). > > pythag0(N) -> > [ {A,B,C} || > A <- lists:seq(1,N), > B <- lists:seq(1,N), > C <- lists:seq(1,N), > A+B+C =< N, > A*A+B*B =:= C*C]. > > pythag1(N) -> > L = lists:seq(1,N), > [ {A,B,C} || > A <- L, > B <- L, > C <- L, > A+B+C =< N, > A*A+B*B =:= C*C]. > > > pythag2(N) -> > lists:reverse(pythan2_A(1, N, [])). > > pythan2_A(A, N, Acc) when A > N -> Acc; > pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). > > pythan2_B(A, B, N, Acc) when A+B > N -> Acc; > pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). > > pythan2_C(A, B, C, N, Acc) when A+B+C > N -> Acc; > pythan2_C(A, B, C, N, Acc) -> > if A*A+B*B =:= C*C -> > pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); > true -> > pythan2_C(A, B, C+1, N, Acc) > end. > > > > > > On 13 nov 2010, at 08.37, Gilberto Carmenate Garc?a wrote: > >> Hi all! >> I have been doing tests to Erlang I found this funny stuff that makes >> Pythagorean Triplets >> >> pythag(N) -> >> [ {A,B,C} || >> A <- lists:seq(1,N), >> B <- lists:seq(1,N), >> C <- lists:seq(1,N), >> A+B+C =< N, >> A*A+B*B =:= C*C]. >> >> I tested it agains an implementation I made in C# so, and takes 14 >> secounds in my pc to do with 300 numbers in Erlang however in c# is just >> a secound, even when C# runs under VM too. >> So I did all possible ways for me to implement differents manners in >> Erlang looking for speed and all is the same, listed as follows: >> >> So my question is, there are any way to do that even more fast, why 3 >> nestes fors structs in C# are more effients that lists:foldr or >> lists:foreach in Erlang. >> >> %% FORMA 1 >> py1(Max)-> >> L = lists:seq(1, Max), >> lists:foldr( >> fun(A, Acc3)-> >> lists:foldr( >> fun(B, Acc2)-> >> lists:foldr( >> fun(C, Acc)-> >> case ((A*A + B*B =:= C*C) andalso (A+B+C =< >> Max)) of >> true-> >> [{A,B,C}|Acc]; >> false-> >> Acc >> end >> end >> , Acc2, L) >> end >> , Acc3, L) >> end >> , [], L). >> >> >> >> %% FORMA 2 >> py2(Max)-> >> fora(1, [], Max). >> >> fora(A, Acc, Max)-> >> Acc1 = forb(A,1, Acc, Max), >> case A < Max of >> true-> >> fora(A+1, Acc1, Max); >> false-> >> Acc1 >> end. >> >> forb(A,B, Acc, Max)-> >> Acc1 = forc(A,B,1, Acc, Max), >> case B < Max of >> true-> >> forb(A,B+1, Acc1, Max); >> false-> >> Acc1 >> end. >> >> forc(A,B,C, Acc, Max)-> >> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of >> true-> >> [{A,B,C}|Acc]; >> _-> >> Acc >> end, >> case C < Max of >> true-> >> forc(A,B,C+1, Acc1, Max); >> false-> >> Acc1 >> end. >> >> %% FORMA 3. >> py3(Max)-> >> [{A,B,C} || >> A <-lists:seq(1, Max), >> B <-lists:seq(1, Max), >> C <-lists:seq(1, Max), >> A*A + B*B =:= C*C, >> A+B+C =< Max]. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico >> que ofrece la Federacion de Radioaficionados de Cuba a sus miembros >> para respaldar el cumplimiento de los objetivos de la organizacion y su >> politica informativa. La persona que envia este correo asume el >> compromiso de usar el servicio a tales fines y cumplir con las >> regulaciones establecidas. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > "Have run Make so many times I dunno what's installed anymore" > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From alessandro.sivieri@REDACTED Sun Nov 14 02:22:08 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Sun, 14 Nov 2010 02:22:08 +0100 Subject: SSL and client authentication Message-ID: Hi all, I'm using the SSL verify_peer option in a Web server with fail_if_no_peer_cert set to true; I was wondering if there is a way to force a client to send a valid certificate for some Web server paths and ignore it for others: for example, if a client tries to make a request to, say, https://host/url1, then I want it to send also a valid certificate, but if it makes a request to https://host/url2, then it may not send any certificate and the request will be accepted anyway. Basically I want mutual auth only for a couple of URLs... A friend who works with Apache said to me that it should be possible, but I don't know if this works in Erlang (and how to make it work with the SSL module options). Thanks, Alessandro -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From cappy2112@REDACTED Sun Nov 14 04:36:51 2010 From: cappy2112@REDACTED (Mr C) Date: Sat, 13 Nov 2010 19:36:51 -0800 (PST) Subject: Why does the cd() command make the shell stop processing ? In-Reply-To: References: Message-ID: <065692bb-8222-416c-9ea5-9baf1398a72f@j29g2000prf.googlegroups.com> Ok, thanks to all who replied. Does ^G work on Windows? When I use ^G, it doesn't abort a running program. On Nov 10, 7:47?am, Fred Hebert wrote: > You're escaping the last " with the system path. This means the string is > never closed. > > Try forward slashes, they work well on Windows 7 (untested on other > Windowses). > > On Wed, Nov 10, 2010 at 10:44 AM, Mr C wrote: > > > After entering cd ("c:\Users\MyUserName\Documents\Src\Erlang\"). > > the prompt line number that is displayed is the same line number as > > before entering the cd() command. > > > The shell doesn't properly process any erlang code after that line is > > entered. > > > I'm using Windows 7 Professional with Erlang R14B (erts-5.8.1.1) > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > Seehttp://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED > > From mononcqc@REDACTED Sun Nov 14 04:55:06 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Sat, 13 Nov 2010 22:55:06 -0500 Subject: [erlang-questions] Re: Why does the cd() command make the shell stop processing ? In-Reply-To: <065692bb-8222-416c-9ea5-9baf1398a72f@j29g2000prf.googlegroups.com> References: <065692bb-8222-416c-9ea5-9baf1398a72f@j29g2000prf.googlegroups.com> Message-ID: It certainly does for me with werl.exe, but not erl.exe. On Sat, Nov 13, 2010 at 10:36 PM, Mr C wrote: > Ok, thanks to all who replied. > > Does ^G work on Windows? > > When I use ^G, it doesn't abort a running program. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From w.a.de.jong@REDACTED Sun Nov 14 09:52:47 2010 From: w.a.de.jong@REDACTED (Willem de Jong) Date: Sun, 14 Nov 2010 09:52:47 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: Hi, I think the key is to minimize the number of calculations and comparisons. In your version you do a lot of multiplications that can easily be avoided. The version below is about 10 x as fast on my PC: pythag2(N) -> L = [{A, A*A} || A <- lists:seq(1,N)], lists:flatten([forAllBs(A, A2, L, N) || {A, A2} <- L]). forAllBs(A, A2, L, N) -> [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2} <- L, A + B < N]. forAllCs(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2} <- L, A2B2 =:= C2, AB + C =< N]. Regards, Willem On Sat, Nov 13, 2010 at 8:37 AM, Gilberto Carmenate Garc?a < co7eb@REDACTED> wrote: > Hi all! > I have been doing tests to Erlang I found this funny stuff that makes > Pythagorean Triplets > > pythag(N) -> > [ {A,B,C} || > A <- lists:seq(1,N), > B <- lists:seq(1,N), > C <- lists:seq(1,N), > A+B+C =< N, > A*A+B*B =:= C*C]. > > I tested it agains an implementation I made in C# so, and takes 14 > secounds in my pc to do with 300 numbers in Erlang however in c# is just > a secound, even when C# runs under VM too. > So I did all possible ways for me to implement differents manners in > Erlang looking for speed and all is the same, listed as follows: > > So my question is, there are any way to do that even more fast, why 3 > nestes fors structs in C# are more effients that lists:foldr or > lists:foreach in Erlang. > > %% FORMA 1 > py1(Max)-> > L = lists:seq(1, Max), > lists:foldr( > fun(A, Acc3)-> > lists:foldr( > fun(B, Acc2)-> > lists:foldr( > fun(C, Acc)-> > case ((A*A + B*B =:= C*C) andalso (A+B+C =< > Max)) of > true-> > [{A,B,C}|Acc]; > false-> > Acc > end > end > , Acc2, L) > end > , Acc3, L) > end > , [], L). > > > > %% FORMA 2 > py2(Max)-> > fora(1, [], Max). > > fora(A, Acc, Max)-> > Acc1 = forb(A,1, Acc, Max), > case A < Max of > true-> > fora(A+1, Acc1, Max); > false-> > Acc1 > end. > > forb(A,B, Acc, Max)-> > Acc1 = forc(A,B,1, Acc, Max), > case B < Max of > true-> > forb(A,B+1, Acc1, Max); > false-> > Acc1 > end. > > forc(A,B,C, Acc, Max)-> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true-> > [{A,B,C}|Acc]; > _-> > Acc > end, > case C < Max of > true-> > forc(A,B,C+1, Acc1, Max); > false-> > Acc1 > end. > > %% FORMA 3. > py3(Max)-> > [{A,B,C} || > A <-lists:seq(1, Max), > B <-lists:seq(1, Max), > C <-lists:seq(1, Max), > A*A + B*B =:= C*C, > A+B+C =< Max]. > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electronico que > ofrece la Federacion de Radioaficionados de Cuba a sus miembros para > respaldar el cumplimiento de los objetivos de la organizacion y su politica > informativa. La persona que envia este correo asume el compromiso de usar > el servicio a tales fines y cumplir con las regulaciones establecidas. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From hynek@REDACTED Sun Nov 14 10:17:58 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Sun, 14 Nov 2010 10:17:58 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: On Sun, Nov 14, 2010 at 9:52 AM, Willem de Jong wrote: > Hi, > > I think the key is to minimize the number of calculations and comparisons. > In your version you do a lot of multiplications that can easily be avoided. > > The version below is about 10 x as fast on my PC: > > pythag2(N) -> > ? L = [{A, A*A} || A <- lists:seq(1,N)], > ? lists:flatten([forAllBs(A, A2, L, N) || {A, A2} <- L]). > > forAllBs(A, A2, L, N) -> > ?[forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2} <- L, A + B < N]. > > forAllCs(A, B, AB, A2B2, L, N) -> > ?[{A, B, C} || {C, C2} <- L, A2B2 =:= C2, AB + C =< N]. > > Regards, > Willem > Good idea but it seems that native (HiPE) compiler does this optimization for you when you keep staying in one module. (I also keeps exporting only main function. It can possibly take effect here also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not right tool for this job. You should use NIF if performance matter. pythag4(N) when is_integer(N) -> pythag4(N,1). pythag4(N, A) when A+2 > N -> []; pythag4(N, A) -> pythag4(N, A, A*A, 1). pythag4(N, A, _A2, B) when A+B+1 > N -> pythag4(N, A+1); pythag4(N, A, A2, B) -> pythag4(N, A, A2, B, B*B, 1). pythag4(N, A, A2, B, _B2, C) when A+B+C > N -> pythag4(N, A, A2, B+1); pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; pythag4(N, A, A2, B, B2, C) -> pythag4(N, A, A2, B, B2, C+1). > On Sat, Nov 13, 2010 at 8:37 AM, Gilberto Carmenate Garc?a < > co7eb@REDACTED> wrote: > >> Hi all! >> I have been doing tests to Erlang I found this funny stuff that makes >> Pythagorean Triplets >> >> pythag(N) -> >> ? ?[ {A,B,C} || >> ? ? ? ?A <- lists:seq(1,N), >> ? ? ? ?B <- lists:seq(1,N), >> ? ? ? ?C <- lists:seq(1,N), >> ? ? ? ?A+B+C =< N, >> ? ? ? ?A*A+B*B =:= C*C]. >> >> I tested it agains an implementation I made in C# so, and takes 14 >> secounds in my pc to do with 300 numbers in Erlang however in c# is just >> a secound, even when C# runs under VM too. >> So I did all possible ways for me to implement differents manners in >> Erlang looking for speed and all is the same, listed as ?follows: >> >> So my question is, there are any way to do that even more fast, why 3 >> nestes fors structs in C# are more effients that lists:foldr or >> lists:foreach in Erlang. >> >> %% FORMA 1 >> py1(Max)-> >> ? ?L = lists:seq(1, Max), >> ? ?lists:foldr( >> ? ? ? ?fun(A, Acc3)-> >> ? ? ? ? ? ?lists:foldr( >> ? ? ? ? ? ? ? ?fun(B, Acc2)-> >> ? ? ? ? ? ? ? ? ? ?lists:foldr( >> ? ? ? ? ? ? ? ? ? ? ? ?fun(C, Acc)-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ?case ((A*A + B*B =:= C*C) andalso (A+B+C =< >> Max)) of >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?true-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{A,B,C}|Acc]; >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?false-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Acc >> ? ? ? ? ? ? ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ? ? ? ? ?, Acc2, L) >> ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ?, Acc3, L) >> ? ? ? ?end >> ? ?, [], L). >> >> >> >> %% FORMA 2 >> py2(Max)-> >> ? ? ? ?fora(1, [], Max). >> >> fora(A, Acc, Max)-> >> ? ? ? ?Acc1 = forb(A,1, Acc, Max), >> ? ? ? ?case A < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?fora(A+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> forb(A,B, Acc, Max)-> >> ? ?Acc1 = forc(A,B,1, Acc, Max), >> ? ?case B < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?forb(A,B+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> forc(A,B,C, Acc, Max)-> >> ? ?Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of >> ? ? ? ?true-> >> ? ? ? ? ? ?[{A,B,C}|Acc]; >> ? ? ? ?_-> >> ? ? ? ? ? ?Acc >> ? ?end, >> ? ?case C < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?forc(A,B,C+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> %% FORMA 3. >> py3(Max)-> >> ? ? ? ?[{A,B,C} || >> ? ? ? ? ? ? ? ?A <-lists:seq(1, Max), >> ? ? ? ? ? ? ? ?B <-lists:seq(1, Max), >> ? ? ? ? ? ? ? ?C <-lists:seq(1, Max), >> ? ? ? ? ? ? ? ?A*A + B*B =:= C*C, >> ? ? ? ? ? ? ? ?A+B+C =< Max]. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico que >> ofrece la Federacion de Radioaficionados de Cuba a sus miembros para >> respaldar el cumplimiento de los objetivos de la organizacion y su politica >> informativa. La persona que envia este correo asume el compromiso de ?usar >> el servicio a tales fines y cumplir con las regulaciones establecidas. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From tony@REDACTED Sun Nov 14 11:55:28 2010 From: tony@REDACTED (Tony Rogvall) Date: Sun, 14 Nov 2010 11:55:28 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> On 14 nov 2010, at 10.17, Hynek Vychodil wrote: >> >> > > Good idea but it seems that native (HiPE) compiler does this > optimization for you when you keep staying in one module. (I also > keeps exporting only main function. It can possibly take effect here > also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not > right tool for this job. You should use NIF if performance matter. > > pythag4(N) when is_integer(N) -> pythag4(N,1). I have implemented several small Erlang programs that beat "native" C code. Most of the time the C programs where badly/hastily written, but that is the point ;-) You must select the algorithm and the implementation wisely, and of course use the golden rule "Make it work, then make it fast". I would add, if it is needed. This does not imply that you should write your program unnecessary slow to start with! Any how. A couple of months ago I implemented the AKS algorithm in Erlang. The AKS algorithm is a deterministic primality test algorithm (http://en.wikipedia.org/wiki/AKS_primality_test) I benchmarked this implementation with an implementation in C++. I was shocked: The Erlang version was about 5 times faster, NOT using obvious parallelism. In this case I would suspect that garbage collection is the major contributor! The implementation use a lot of temporary polynomials intermediate results. A copy collector does the trick. /Tony "Have run Make so many times I dunno what's installed anymore" From hynek@REDACTED Sun Nov 14 12:32:42 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Sun, 14 Nov 2010 12:32:42 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: On Sun, Nov 14, 2010 at 11:55 AM, Tony Rogvall wrote: > > On 14 nov 2010, at 10.17, Hynek Vychodil wrote: > > > > Good idea but it seems that native (HiPE) compiler does this > optimization for you when you keep staying in one module. (I also > keeps exporting only main function. It can possibly take effect here > also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not > right tool for this job. You should use NIF if performance matter. > > pythag4(N) when is_integer(N) -> pythag4(N,1). > > I have implemented several small Erlang programs that beat "native" C code. > Most of the time the C programs where badly/hastily written, but that is the > point ;-) > You must select the algorithm and the implementation wisely, and of course > use the golden rule "Make it work, then make it fast". I would add, if it is > needed. > This does not imply that you should write your program unnecessary slow to > start with! > Any how. A couple of months ago I implemented the AKS algorithm in Erlang. > The AKS algorithm is a deterministic primality test algorithm > (http://en.wikipedia.org/wiki/AKS_primality_test) > I benchmarked this implementation with an implementation in C++. > I was shocked: The Erlang version was about 5 times faster, NOT using > obvious parallelism. > In this case I would suspect that garbage collection is the major > contributor! The implementation use > a lot of temporary polynomials intermediate results. A copy collector does > the trick. > /Tony > > > > "Have run Make so many times I dunno what's installed anymore" > When Erlang implementation is faster than C implementation *is* badly written even it can be for maintainability or haste reason. C++ is another beast. Pun intended. I have similar experience to you but when I found that Erlang implementation is faster then I would look how is it possible and you are right, usually it is memory issue. Anyway every time you can do same tricks as Erlang does but you will end up with "half and error prone implementation of Erlang". You can also do tricks in C which you can't in Erlang. Can you write k/v store which is able do 200 millions of look up operations per second on one 2.4GHz i5 core? Anyway HiPE compiler does very good work here. If I count it correctly pythag3 and pythag4 does about hundred millions checks per seconds. Very good result I think. -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From ebegumisa@REDACTED Sun Nov 14 17:02:50 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 15 Nov 2010 03:02:50 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: Willem's version in parallel below using a maximum of 2000 concurrent processes (find lpmap function is in previous e-mail). This is approx. 16 x faster on my dual-core. Probably surpasses C# already (but again, not really a good point to try to be making.) pythag2(N) -> L = [{A, A*A} || A <- lists:seq(1,N)], % For all A's lists:flatten(lpmap(fun({A, A2}) -> % For all B's in parallel [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2} <- L, A + B < N] end, L, 2000, ordered)). forAllCs(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2} <- L, A2B2 =:= C2, AB + C =< N]. - Edmond - On Sun, 14 Nov 2010 19:52:47 +1100, Willem de Jong wrote: > Hi, > > I think the key is to minimize the number of calculations and > comparisons. > In your version you do a lot of multiplications that can easily be > avoided. > > The version below is about 10 x as fast on my PC: > > pythag2(N) -> > L = [{A, A*A} || A <- lists:seq(1,N)], > lists:flatten([forAllBs(A, A2, L, N) || {A, A2} <- L]). > > forAllBs(A, A2, L, N) -> > [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2} <- L, A + B < N]. > > forAllCs(A, B, AB, A2B2, L, N) -> > [{A, B, C} || {C, C2} <- L, A2B2 =:= C2, AB + C =< N]. > > Regards, > Willem > > On Sat, Nov 13, 2010 at 8:37 AM, Gilberto Carmenate Garc?a < > co7eb@REDACTED> wrote: > >> Hi all! >> I have been doing tests to Erlang I found this funny stuff that makes >> Pythagorean Triplets >> >> pythag(N) -> >> [ {A,B,C} || >> A <- lists:seq(1,N), >> B <- lists:seq(1,N), >> C <- lists:seq(1,N), >> A+B+C =< N, >> A*A+B*B =:= C*C]. >> >> I tested it agains an implementation I made in C# so, and takes 14 >> secounds in my pc to do with 300 numbers in Erlang however in c# is just >> a secound, even when C# runs under VM too. >> So I did all possible ways for me to implement differents manners in >> Erlang looking for speed and all is the same, listed as follows: >> >> So my question is, there are any way to do that even more fast, why 3 >> nestes fors structs in C# are more effients that lists:foldr or >> lists:foreach in Erlang. >> >> %% FORMA 1 >> py1(Max)-> >> L = lists:seq(1, Max), >> lists:foldr( >> fun(A, Acc3)-> >> lists:foldr( >> fun(B, Acc2)-> >> lists:foldr( >> fun(C, Acc)-> >> case ((A*A + B*B =:= C*C) andalso (A+B+C =< >> Max)) of >> true-> >> [{A,B,C}|Acc]; >> false-> >> Acc >> end >> end >> , Acc2, L) >> end >> , Acc3, L) >> end >> , [], L). >> >> >> >> %% FORMA 2 >> py2(Max)-> >> fora(1, [], Max). >> >> fora(A, Acc, Max)-> >> Acc1 = forb(A,1, Acc, Max), >> case A < Max of >> true-> >> fora(A+1, Acc1, Max); >> false-> >> Acc1 >> end. >> >> forb(A,B, Acc, Max)-> >> Acc1 = forc(A,B,1, Acc, Max), >> case B < Max of >> true-> >> forb(A,B+1, Acc1, Max); >> false-> >> Acc1 >> end. >> >> forc(A,B,C, Acc, Max)-> >> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of >> true-> >> [{A,B,C}|Acc]; >> _-> >> Acc >> end, >> case C < Max of >> true-> >> forc(A,B,C+1, Acc1, Max); >> false-> >> Acc1 >> end. >> >> %% FORMA 3. >> py3(Max)-> >> [{A,B,C} || >> A <-lists:seq(1, Max), >> B <-lists:seq(1, Max), >> C <-lists:seq(1, Max), >> A*A + B*B =:= C*C, >> A+B+C =< Max]. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico >> que >> ofrece la Federacion de Radioaficionados de Cuba a sus miembros para >> respaldar el cumplimiento de los objetivos de la organizacion y su >> politica >> informativa. La persona que envia este correo asume el compromiso de >> usar >> el servicio a tales fines y cumplir con las regulaciones establecidas. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Sun Nov 14 17:11:05 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 15 Nov 2010 03:11:05 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: On Sun, 14 Nov 2010 21:55:28 +1100, Tony Rogvall wrote: > In this case I would suspect that garbage collection is the major > contributor! The implementation use > a lot of temporary polynomials intermediate results. Hmmmm, I wonder if you could use a private ETS table to somehow pre-allocate space for the permutations? ETS is more-or-less mutable memory for Erlang is it not? A more experienced Erlang programmer than myself would know if this is worth pursuing. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From mk@REDACTED Sun Nov 14 17:26:32 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 14 Nov 2010 17:26:32 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: <72B7FCEB-8BD0-4677-9584-E30EF4A7C7A3@amberbio.com> Hi Here is another version of the same program using a different approach. if A^2 + B^2 = C^2, then A and B cannot both be odd, so at least one of the must be even, say, B. Write B^2 = (C+A)*(C-A) Now A and C are both even or both odd, so we can write, (B/2)^2 = M1 * M2, where M1 = (C+A)/2, M2 = (C-A)/2 where M1 and M2 are integers. Let p be a prime number and let alpha1 be the highest number such that p^alpha1 divides M1, and similarly for alpha2, then alpha1 and alpha2 must both be even or both be odd, since the left hand side (B/2)^2 only has even powers of any prime factor. On the other hand any M1, M2 built in this way will lead to a pythagorean triple, as can be seen by just plugging in. If A+B+C <= N, then no prime number greater than N/2 can occur. This lead to the following algorithm. 1. Generate all prime numbers up to N/2. I used Eratosthenes sieve for that. 2. Generate all M1 and M2, by distributing even or odd powers as explained above. Only keep numbers that do not already break A+B+C <= N, and M1 > M2. 3. Put them all in a list, permute A and B, e.g. {3,4,5} and {4,3,5}. 4. unique sort it. Unique because of the A and B permutation, and sort to make the outpout the same as all the other implementations. The code is pasted below. The runtime for N=300 is 5 ms on a computer where pythag3 is 80 ms. But it is for larger values of N that this algorithm will really pull away. For N=3000, it uses 680 ms compared to 76 s for pythag3. The earlier programs were all N^3, this one seems, by testing, to be N^2. I haven't thought theoretically about the complexity of this algotithm. I haven't optimised this program. Cheers, Morten. pythag5(N) when is_integer(N) -> Primes = sieve(N div 2), M1M2s = incorporate_primes([{1,1}], N, Primes), lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2} <- M1M2s, M1 > M2, A <- [M1-M2], B <- [2*round(math:sqrt(M1*M2))], C <- [M1+M2], A+B+C =< N])). sieve(N) when is_integer(N) -> erase(), sieve(N,2). sieve(N, K) when K >= N -> [X || X <- lists:seq(2, N), erase(X) == undefined]; sieve(N, K) -> cross_off(K, K, N div K - 1), sieve(N, find_next_in_sieve(K + 1)). cross_off(_K, _Current, 0) -> ok; cross_off(K, Current, Left) -> Next = Current + K, put(Next, out), cross_off(K, Next, Left - 1). find_next_in_sieve(K) -> case get(K) of undefined -> K; _ -> find_next_in_sieve(K+1) end. incorporate_prime(M1M2s, N, P) -> lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2} <- M1M2s]). incorporate_prime_single({M1,M2}, N, P) -> Evens = [{X, Y} || X <- incorporate_prime_even(M1, N, P), Y <- incorporate_prime_even(M2, N, P)], Odds = [{X, Y} || X <- incorporate_prime_odd(M1, N, P), Y <- incorporate_prime_odd(M2, N, P)], Evens ++ Odds. incorporate_prime_even(M, N, P) -> incorporate_prime(M, N, P, []). incorporate_prime_odd(M, N, P) -> incorporate_prime(M * P, N, P, []). incorporate_prime(M, N, _P, Acc) when M > N/2 -> Acc; incorporate_prime(M, N, P, Acc) -> incorporate_prime(M * P * P, N, P, [M|Acc]). incorporate_primes(M1M2s, _N, []) -> M1M2s; incorporate_primes(M1M2s, N, [P|Rest]) -> M1M2s_new = incorporate_prime(M1M2s, N, P), incorporate_primes(M1M2s_new, N, Rest). From pguyot@REDACTED Sun Nov 14 17:27:58 2010 From: pguyot@REDACTED (Paul Guyot) Date: Sun, 14 Nov 2010 17:27:58 +0100 Subject: SSL and client authentication In-Reply-To: <1289750585.2064.ezmlm@erlang.org> References: <1289750585.2064.ezmlm@erlang.org> Message-ID: > I'm using the SSL verify_peer option in a Web server with > fail_if_no_peer_cert set to true; I was wondering if there is a way to force > a client to send a valid certificate for some Web server paths and ignore it > for others: for example, if a client tries to make a request to, say, > https://host/url1, then I want it to send also a valid certificate, but if > it makes a request to https://host/url2, then it may not send any > certificate and the request will be accepted anyway. Basically I want mutual > auth only for a couple of URLs... SSL handshake happens before any HTTP data is exchanged. Therefore, the server cannot know in advance, when the handshake occurs, if the client wants to access url1 or url2. For such a situation, you can have two hostnames, e.g. https://verified.host.com/ and https://regular.host.com/ Paul -- Semiocast http://semiocast.com/ +33.175000290 - 62 bis rue Gay-Lussac, 75005 Paris From alessandro.sivieri@REDACTED Sun Nov 14 17:36:14 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Sun, 14 Nov 2010 17:36:14 +0100 Subject: SSL and client authentication In-Reply-To: References: <1289750585.2064.ezmlm@erlang.org> Message-ID: I think Nginx will be my solution; thanks for the answers! -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From ebegumisa@REDACTED Sun Nov 14 17:50:30 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Mon, 15 Nov 2010 03:50:30 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: Hynek, A few comments... > When Erlang implementation is faster than C implementation *is* badly > written even it can be for maintainability or haste reason. C++ is > another beast I disagree. Quick example... 1. Write a "well-written" web-server in Erlang, the "Erlang way" (lightweight-weight concurrency, message passing). 2. Write a "well-written" web-server in C/C++, the "C/C++ way" (mulit-threading, shared-memory). See which is faster. Or, you could save yourself some time and look at Apache + Yaws. I wouldn't describe either of these as badly written. But the performance difference is undeniable. > You can also do tricks in C which you can't in Erlang. And you can do tricks in Erlang that you can't do in typical C (e.g easy parallelism with little change to code as evidenced on this thread.) > Can you write k/v store which > is able do 200 millions of look up operations per second on one 2.4GHz > i5 core? Something is getting lost here. I go back to the promises a language makes. In delivering on those promises, priorities have to be set and compromises made. The language user has to weight these. C/C++ gives us pointers (amongst other things). The language makes great sacrifices to give us pointers (like garbage collection and debugability) so we can create insanely efficient data structures. When the benefits of pointers far outweigh the trade-offs, the choice is clear. Erlang gives us high concurrency (amongst other things). The Erlang team made great sacrifices to give us insanely efficient and scalable concurrency. When the benefits of this far outweigh the trade-offs the choice is clear. I see few areas where these benefits/trade-offs overlap. The choice is usually very clear. In cases where overlapping does occur, there are various ways of using both. - Edmond - On Sun, 14 Nov 2010 22:32:42 +1100, Hynek Vychodil wrote: > On Sun, Nov 14, 2010 at 11:55 AM, Tony Rogvall wrote: >> >> On 14 nov 2010, at 10.17, Hynek Vychodil wrote: >> >> >> >> Good idea but it seems that native (HiPE) compiler does this >> optimization for you when you keep staying in one module. (I also >> keeps exporting only main function. It can possibly take effect here >> also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not >> right tool for this job. You should use NIF if performance matter. >> >> pythag4(N) when is_integer(N) -> pythag4(N,1). >> >> I have implemented several small Erlang programs that beat "native" C >> code. >> Most of the time the C programs where badly/hastily written, but that >> is the >> point ;-) >> You must select the algorithm and the implementation wisely, and of >> course >> use the golden rule "Make it work, then make it fast". I would add, if >> it is >> needed. >> This does not imply that you should write your program unnecessary slow >> to >> start with! >> Any how. A couple of months ago I implemented the AKS algorithm in >> Erlang. >> The AKS algorithm is a deterministic primality test algorithm >> (http://en.wikipedia.org/wiki/AKS_primality_test) >> I benchmarked this implementation with an implementation in C++. >> I was shocked: The Erlang version was about 5 times faster, NOT using >> obvious parallelism. >> In this case I would suspect that garbage collection is the major >> contributor! The implementation use >> a lot of temporary polynomials intermediate results. A copy collector >> does >> the trick. >> /Tony >> >> >> >> "Have run Make so many times I dunno what's installed anymore" >> > > When Erlang implementation is faster than C implementation *is* badly > written even it can be for maintainability or haste reason. C++ is > another beast. Pun intended. I have similar experience to you but when > I found that Erlang implementation is faster then I would look how is > it possible and you are right, usually it is memory issue. Anyway > every time you can do same tricks as Erlang does but you will end up > with "half and error prone implementation of Erlang". You can also do > tricks in C which you can't in Erlang. Can you write k/v store which > is able do 200 millions of look up operations per second on one 2.4GHz > i5 core? Anyway HiPE compiler does very good work here. If I count it > correctly pythag3 and pythag4 does about hundred millions checks per > seconds. Very good result I think. > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From roberto@REDACTED Sun Nov 14 18:56:19 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 14 Nov 2010 18:56:19 +0100 Subject: fastcgi backend of http servers Message-ID: dear list, i was going through some thoughts today regarding http and fastcgi with language interpreters in the backend. for instance, re php it is quite common to use a combination of nginx & a fastcgi process manager [such as php-fpm] to achieve interesting results in terms of speed and performance. i was wandering if any solid attempt has been done so far to see whether an http server in erlang providing fastcgi bridges to a fastcgi process manager could be something worthy to pursue, or if nginx's C-way of dealing things simply is unbeatable afa speed, CPU, RAM usage, and concurrent open tcp connections are concerned. i am completely aware that the benefits of having such a server in erlang would mean being able to integrate it into an OTP application, with all the resulting benefits, but i'm more interested in evaluating this effort in terms of performances [for instance, number of concurrent open connections]. ..any inputs/thoughts on this? cheers, r. From info@REDACTED Sun Nov 14 18:45:01 2010 From: info@REDACTED (info) Date: Sun, 14 Nov 2010 18:45:01 +0100 Subject: odbc error Message-ID: <201011141844548282526@its3.ch> Hello, I received this message in a gen_server: "odbc MySQL server has gone away SQLSTATE IS: 08S01 " I suppose that the defaut idle time of 8 hours has expired. What is the best action to do with erlang+odbc: - restart the server ? - catch the error and restart odbc ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From fredrik@REDACTED Sun Nov 14 22:21:40 2010 From: fredrik@REDACTED (Fredrik Thulin) Date: Sun, 14 Nov 2010 22:21:40 +0100 Subject: ANNOUNCE: erlang-yubico client Message-ID: <1289769700.4626.59.camel@ft-laptop.thulin.net> Hi I've written a simple Erlang client for validating Yubico OTPs (one time passwords) against online verification servers. It can be found (BSD license) at https://github.com/fredrikt/erlang-yubico Yubico is a company that makes Yubikeys - hardware OTP tokens (USB HID, acts as a keyboard). With erlang-yubico, you can easily get the security of OTPs in your Erlang application if you acquire one or more of these tokens. The validation server is open source, so you can set up your own instead of trusting the default YubiCloud servers. If you want to see one of the USB thingys and are at the EUC on Tuesday, come see me. /Fredrik From andrew@REDACTED Sun Nov 14 22:33:34 2010 From: andrew@REDACTED (Andrew Thompson) Date: Sun, 14 Nov 2010 16:33:34 -0500 Subject: [erlang-questions] odbc error In-Reply-To: <201011141844548282526@its3.ch> References: <201011141844548282526@its3.ch> Message-ID: <20101114213334.GE32129@hijacked.us> On Sun, Nov 14, 2010 at 06:45:01PM +0100, info wrote: > Hello, > I received this message in a gen_server: > "odbc MySQL server has gone away SQLSTATE IS: 08S01 " > I suppose that the defaut idle time of 8 hours has expired. > > What is the best action to do with erlang+odbc: > - restart the server ? > - catch the error and restart odbc ? > I had a seperate process dealing with ODBC which just crashed in this case and got restarted by a supervisor. You should be able to just restart ODBC, though (or rather just reconnect, since it should already be started). Andrew From erlangpriest@REDACTED Sun Nov 14 23:03:24 2010 From: erlangpriest@REDACTED (ErlangPriest) Date: Sun, 14 Nov 2010 23:03:24 +0100 Subject: A tribute to Erlang/OTP Message-ID: Hi, With the upcoming EUC 2010 and Thanksgiving lurking around the corner I felt that it some sort of tribute/thanks to Erlang/OTP and the great team behind it was long overdue, so I wrote a song that I would like to share with you all. It does not compile yet (i.e., the lyrics does not quite fit the tune), but I hope that someone with greater artistic talents than me will run it through dialyzer and QuickCheck to make everything correct ;-) Happy EUC2010 and happy Thanksgiving! The ErlangPriest ============================= Deadlock free holiday ============== Original title: Dreadlock Holiday Original authors: Graham Keith Gouldman; Eric Michael Stewart Adapted to Erlang/OTP by the ErlangPriest. I was workin? on the API, concentratin? on codin? right I heard a dark voice beside of me And I look ?round in a state of fright I saw four faces, one mad; a dude from the management They looked me up and down and turned to each other I say, I don?t like Erlang, oh no, I love it I don?t like Erlang, oh no, I love it. Don?t you walk through my code You got to show some insight Don?t you walk though my code Cause you ain?t an Erlanger yet Well, he looked down on my silver bullet He said: ?I?ll give you one dollar? I said: ?You?ve got to be jokin?, man It was a present from Ericsson? He said: ?I like it, I want it, I?ll take it off your hands And you?ll be sorry you crossed me You better understand That you?re alone (many nodes away from home)?. And I say, I don?t like spawn, oh no, I love it I don?t like spawn, oh no, I love it Don?t you cramp me style, don?t you queer me pitch Don?t you walk through my code ?Cause you ain?t an Erlanger yet I hurried back to the Erlang prompt, passin? atoms around I heard a dark voice inside of me say ?Would you like something harder?? She said: I?ve got it, you want it, my callbacks are the best And if you try it You?ll like it and whollow in deadlock free holiday. And I say, don?t like OTP, oh no, I love her Don?t like OTP, oh no, I love her, oh yeah Don?t you walk through her code You go to show some insight Don?t you walk throug her code ?Cause you ain?t an Erlanger yet I don?t like Erlang, oh no, I love it (Deadlock free holiday) I don?t like spawn, oh no, I love it (Deadlock free holiday) I don?t like OTP, oh no, I love her (Deadlock free holiday) =============================== From co7eb@REDACTED Sun Nov 14 21:58:22 2010 From: co7eb@REDACTED (=?ISO-8859-1?Q?Gilberio_Carmenates_Garc=EDa?=) Date: Sun, 14 Nov 2010 17:58:22 -0300 Subject: feedbacks from Erlang shows its slow face message Message-ID: <4CE04D6E.4080508@frcuba.co.cu> Hi all, first of all, thanks to all for your feedbacks, I really impressed since I knew what Erlang can do, so I know that Erlang isn't the best choice for maths calculations and those stuffs. I knew that using Erlang and spawing multiple process I can optimeze it, It's just that I did not knew how to do it for that case, I am not so good in algotims so, what I really wanted was to comprovate that Erlang, even in that!, making maths calculations, is so good than C# with all its supermacy of be a good and fast language that runs on "VM"!!!. (less in threads, threads sucks) of course that Erlang processes are the best!!!, well almost the best, Mozart/Oz languaje hum.. it is good spawning process too. So, now I have to take a time to analize all your feedbacks to learn more. Thanks again. Cheers Ivan ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From corticalcomputer@REDACTED Mon Nov 15 00:31:09 2010 From: corticalcomputer@REDACTED (G.S.) Date: Sun, 14 Nov 2010 18:31:09 -0500 Subject: What's a solid way to have erlang automatically send email. Message-ID: Hello, I have a website where the administrator needs to receive email alert (to login and fix a problem) when certain events occur. Is there an Erlang module that allows one to easily send out an email? Or something that can be used in the same manner? Thanks, -Gene From ok@REDACTED Mon Nov 15 02:06:36 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 15 Nov 2010 14:06:36 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: On 13/11/2010, at 8:37 PM, Gilberto Carmenate Garc?a wrote: > Hi all! > I have been doing tests to Erlang I found this funny stuff that makes > Pythagorean Triplets > > pythag(N) -> > [ {A,B,C} || > A <- lists:seq(1,N), > B <- lists:seq(1,N), > C <- lists:seq(1,N), > A+B+C =< N, > A*A+B*B =:= C*C]. That's not as efficient as it could be. C <- lists:seq(1, N), A <- lists:seq(1, N-C-1), B <- lists:seq(1, N-C-A), does a factor of 6 or better fewer iterations (tested up to N = 500). We can do better still. Since A^2 + B^2 = C^2, C >= max(A,B), so it follows that A, B are both <= (N-1) div 2, and given A and B there is only one possible C. So there are only about (N*N)/4 cases to check instead of N*N*N, Here's a table. Fun N Iters Filtered Answers old 10 1000 120 0 new 10 120 120 0 sqt 10 16 13 0 old 20 8000 1140 2 new 20 1140 1140 2 sqt 20 81 58 2 old 50 125000 19600 12 new 50 19600 19600 12 sqt 50 576 372 12 old 100 1000000 161700 34 new 100 161700 161700 34 sqt 100 2401 1513 34 old 200 8000000 1313400 86 new 200 1313400 1313400 86 sqt 200 9801 6094 86 old 500 125000000 20708500 274 new 500 20708500 20708500 274 sqt 500 62001 38258 274 Fun is old (your loop nest), new (my CAB loop nest above), or sqt (AB loop nest with C = hypot(A,B) truncated to integer); N is N; Iters is the number of iterations of the loop nest; Filtered is the number of cases that pass the A+B+C =< N filter, and Answers is the number of triples found this way. !!!! Looking at the last line, we see a reduction in iterations !!!! by a factor of 2016.1, which dwarfs any difference between !!!! Erlang and C#. (And it's possible to do even better.) One issue of course is that C# is presumably not building and iterating over any lists at all here, so if you want a *fair* comparison between C# and Erlang, your iterations should not iterate over any lists either. p > > I tested it agains an implementation I made in C# so, and takes 14 > secounds in my pc to do with 300 numbers in Erlang however in c# is just > a secound, even when C# runs under VM too. > So I did all possible ways for me to implement differents manners in > Erlang looking for speed and all is the same, listed as follows: > > So my question is, there are any way to do that even more fast, why 3 > nestes fors structs in C# are more effients that lists:foldr or > lists:foreach in Erlang. > > %% FORMA 1 > py1(Max)-> > L = lists:seq(1, Max), > lists:foldr( > fun(A, Acc3)-> > lists:foldr( > fun(B, Acc2)-> > lists:foldr( > fun(C, Acc)-> > case ((A*A + B*B =:= C*C) andalso (A+B+C =< > Max)) of > true-> > [{A,B,C}|Acc]; > false-> > Acc > end > end > , Acc2, L) > end > , Acc3, L) > end > , [], L). > > > > %% FORMA 2 > py2(Max)-> > fora(1, [], Max). > > fora(A, Acc, Max)-> > Acc1 = forb(A,1, Acc, Max), > case A < Max of > true-> > fora(A+1, Acc1, Max); > false-> > Acc1 > end. > > forb(A,B, Acc, Max)-> > Acc1 = forc(A,B,1, Acc, Max), > case B < Max of > true-> > forb(A,B+1, Acc1, Max); > false-> > Acc1 > end. > > forc(A,B,C, Acc, Max)-> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true-> > [{A,B,C}|Acc]; > _-> > Acc > end, > case C < Max of > true-> > forc(A,B,C+1, Acc1, Max); > false-> > Acc1 > end. > > %% FORMA 3. > py3(Max)-> > [{A,B,C} || > A <-lists:seq(1, Max), > B <-lists:seq(1, Max), > C <-lists:seq(1, Max), > A*A + B*B =:= C*C, > A+B+C =< Max]. > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ok@REDACTED Mon Nov 15 02:24:59 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 15 Nov 2010 14:24:59 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: <03BC861E-8141-4F78-82F2-F473A3DAD291@cs.otago.ac.nz> On 14/11/2010, at 3:36 AM, Edmond Begumisa wrote: > Numerical algorithms? Enumerating pythagorean triples is not a "numerical" algorithm. It's integer-only, and it doesn't even need large integers. As far as I know there's nothing in ACML or GSL that would help. You could call it combinatorics or number theory. > > From ok@REDACTED Mon Nov 15 02:29:49 2010 From: ok@REDACTED (Richard O'Keefe) Date: Mon, 15 Nov 2010 14:29:49 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> On 14/11/2010, at 10:17 PM, Hynek Vychodil wrote: Anyway Erlang is not > right tool for this job. You should use NIF if performance matter. Erlang is a fine tool for jobs like this one. In fact, given its support for large integers, it is about as good for combinatorial algorithms as Smalltalk, though perhaps not as good as Lisp. (But see LFE...) There is little point in optimising a bad algorithm. A fairly naive rewrite turns it from O(N**3) into O(N**2). What really counts here is how easy it is to spot the algorithmic problem and switch to a better algorithm. From alain.odea@REDACTED Mon Nov 15 04:14:18 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Sun, 14 Nov 2010 23:44:18 -0330 Subject: [erlang-questions] What's a solid way to have erlang automatically send email. In-Reply-To: References: Message-ID: On Sunday, November 14, 2010, G.S. wrote: > Hello, > > I have a website where the administrator needs to receive email alert (to > login and fix a problem) when certain events occur. Is there an Erlang > module that allows one to easily send out an email? Or something that can be > used in the same manner? > > Thanks, > -Gene > Zotonic's mod_emailer could be used to achieve this: http://zotonic.com/mod-emailer I'm fairly certain it can be used pretty well standalone, but zotonic-users@REDACTED is the place to field such questions if you go that route :) From g9414002.pccu.edu.tw@REDACTED Mon Nov 15 04:41:03 2010 From: g9414002.pccu.edu.tw@REDACTED (=?UTF-8?B?6buD6ICA6LOiIChZYXUtSHNpZW4gSHVhbmcp?=) Date: Mon, 15 Nov 2010 11:41:03 +0800 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: I appreciate that this answer is very insight. Some discussions before showed that rewriting and rewriting the Erlang code may beat the speed of the C# program. Some did specific programming techniques on aspects of this problem. However, it ought to be that keeping C# program written in operational semantics the same as the Erlang one. Thus, your answer is a better one, simple and fair. On Mon, Nov 15, 2010 at 9:06 AM, Richard O'Keefe wrote: > > On 13/11/2010, at 8:37 PM, Gilberto Carmenate Garc?a wrote: > > > Hi all! > > I have been doing tests to Erlang I found this funny stuff that makes > > Pythagorean Triplets > > > > pythag(N) -> > > [ {A,B,C} || > > A <- lists:seq(1,N), > > B <- lists:seq(1,N), > > C <- lists:seq(1,N), > > A+B+C =< N, > > A*A+B*B =:= C*C]. > > That's not as efficient as it could be. > > C <- lists:seq(1, N), > A <- lists:seq(1, N-C-1), > B <- lists:seq(1, N-C-A), > > does a factor of 6 or better fewer iterations (tested up to N = 500). > > -- Best Regards. --- Y-H. H. From mpalmer@REDACTED Mon Nov 15 06:47:23 2010 From: mpalmer@REDACTED (mpalmer@REDACTED) Date: Mon, 15 Nov 2010 16:47:23 +1100 (EST) Subject: SSL and client authentication In-Reply-To: References: <1289750585.2064.ezmlm@erlang.org> Message-ID: >> I'm using the SSL verify_peer option in a Web server with >> fail_if_no_peer_cert set to true; I was wondering if there is a way to >> force >> a client to send a valid certificate for some Web server paths and >> ignore it >> for others: for example, if a client tries to make a request to, say, >> https://host/url1, then I want it to send also a valid certificate, but >> if >> it makes a request to https://host/url2, then it may not send any >> certificate and the request will be accepted anyway. Basically I want >> mutual >> auth only for a couple of URLs... > > SSL handshake happens before any HTTP data is exchanged. Therefore, the > server cannot know in advance, when the handshake occurs, if the client > wants to access url1 or url2. However, the SSL handshake *has* occured by the time the response is sent back, so it shouldn't be impossible for the server to check the URL and (whether a/which) client certificate was presented, and serve or deny the request as appropriate. Since the OP has decided to use nginx, there's not much point in going into the mechanics of how to do that in Erlang. - Matt From co7eb@REDACTED Mon Nov 15 05:25:24 2010 From: co7eb@REDACTED (=?ISO-8859-1?Q?Gilberio_Carmenates_Garc=EDa?=) Date: Mon, 15 Nov 2010 01:25:24 -0300 Subject: Hi all here the RESUME for Pytagorians Numbers Algorithm!!! Message-ID: <4CE0B634.9060407@frcuba.co.cu> Made on a pc with Petium 4 CPU 1.6 Ghz one core, 512 Ram. *** Pytagoriam's Numbers REPORT! *** Using N = 300 *** THE ORIGINAL IMPLEMENTATIONS (ME!)**** -py1: 15547000 mcs = 15.547 s -py2: 14514999 mcs = 14.514999 s -py3: 14468999 mcs = 14.468999 s *** Tony's implementations **** -pythag1: 13218999 mcs = 13.218999 s -pythag2: 2296999 mcs = 2.296999 s *** Hynek's implementation **** Simpler and about 5% faster version -pythag3: 2202999 mcs = 2.202999 s *** Edmond's implementation, using parallelism**** -py2E: 14624999 mcs = 14.624999 s *** Willem's implementation**** -wpythag2: 2202999 mcs = 2.202999 s *** Hynek's new implementation**** -pythag4: 1530999 mcs = 1.530999 s *** Willem's new implementation in parallel by Hynek**** -wpythag2P: 2202999 mcs = 2.202999 s *** Morten's implementation**** -pythag5: 62999 mcs = 0.062999 s *** Richard's improvement**** -py3R: 2452999 mcs = 2.452999 s Comparisons in results agains 'pythag1' Tony's function py1 returns the same than 'pythag1' py2 NO returns the same than 'pythag1' %% since my secound implementation 'py2' is wrong!!! py3 returns the same than 'pythag1' pythag1 returns the same than 'pythag1' pythag2 returns the same than 'pythag1' pythag3 returns the same than 'pythag1' py2E NO returns the same than 'pythag1' %% since Edmond use my wrong implementation. wpythag2 returns the same than 'pythag1' pythag4 returns the same than 'pythag1' wpythag2P returns the same than 'pythag1' pythag5 returns the same than 'pythag1' py3R NO returns the same than 'pythag1' %% I just tries to put the Richard explanation in a algorithm but, seems to me I was wrong here too. NOTE: Time was took using timer:tc/3 function. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Here the source code of all algorithm %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -module(py). -compile([export_all]). start(N)-> %% My implementations. {T1,R1} = timer:tc(py,py1, [N]), {T2,R2} = timer:tc(py,py2,[N]), {T3,R3} = timer:tc(py,py3,[N]), %% Tony's improvement of the original form 3. {T4,R4} = timer:tc(py,pythag1,[N]), %% Tony's implementation. {T5,R5} = timer:tc(py,pythag2,[N]), %% Hynek's implementation. %% Simpler and about 5% faster version: {T6,R6} = timer:tc(py,pythag3,[N]), %% Edmond's implementation using parallelism. {T7,R7} = timer:tc(py,py2E,[N]), %% Willem's implementation. {T8,R8} = timer:tc(py,wpythag2,[N]), %% Hynek's new version {T9,R9} = timer:tc(py,pythag4,[N]), %% Willem's implementation in parallel by Hynek {T10,R10} = timer:tc(py,wpythag2P,[N]), %% Morten's implementation. {T11,R11} = timer:tc(py,pythag5,[N]), %% Richard's improvement. {T12,R12} = timer:tc(py,py3R,[N]), io:format("~n *** Pytagoriam's Numbers REPORT! ***~n~n"), io:format("Using N = ~p~n", [N]), io:format("~n*** THE ORIGINAL IMPLEMENTATIONS (ME!)****~n"), io:format("-py1: ~p mcs = ~p s~n", [T1,T1/1000000]), io:format("-py2: ~p mcs = ~p s~n", [T2,T2/1000000]), io:format("-py3: ~p mcs = ~p s~n", [T3,T3/1000000]), io:format("~n*** Tony's implementations ****~n"), io:format("-pythag1: ~p mcs = ~p s~n", [T4,T4/1000000]), io:format("-pythag2: ~p mcs = ~p s~n", [T5,T5/1000000]), io:format("~n*** Hynek's implementation ****~n"), io:format("Simpler and about 5% faster version~n"), io:format("-pythag3: ~p mcs = ~p s~n", [T6,T6/1000000]), io:format("~n*** Edmond's implementation, using parallelism****~n"), io:format("-py2E: ~p mcs = ~p s~n", [T7,T7/1000000]), io:format("~n*** Willem's implementation****~n"), io:format("-wpythag2: ~p mcs = ~p s~n", [T8,T8/1000000]), io:format("~n*** Hynek's new implementation****~n"), io:format("-pythag4: ~p mcs = ~p s~n", [T9,T9/1000000]), io:format("~n*** Willem's new implementation in parallel by Hynek****~n"), io:format("-wpythag2P: ~p mcs = ~p s~n", [T10,T10/1000000]), io:format("~n*** Morten's implementation****~n"), io:format("-pythag5: ~p mcs = ~p s~n", [T11,T11/1000000]), io:format("~n*** Richard's improvement****~n"), io:format("-py3R: ~p mcs = ~p s~n", [T12,T12/1000000]), io:format("~nComparisons in results agains 'pythag1' Tony's function~n"), Rs = [{py1,R1}, {py2,R2}, {py3,R3}, {pythag1,R4}, {pythag2,R5}, {pythag3,R6}, {py2E,R7}, {wpythag2,R8}, {pythag4,R9}, {wpythag2P,R10}, {pythag5,R11}, {py3R,R12}], lists:foreach(fun({Name, R})-> if (R=:=R4)-> io:format("~p returns the same than 'pythag1'~n", [Name]); true-> io:format("~p NO returns the same than 'pythag1'~n", [Name]) end end, Rs), io:format("~nNOTE: Time took using timer:tc/3 function.~n"). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% The original form 1. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py1(Max)-> L = lists:seq(1, Max), lists:foldr( fun(A, Acc3)-> lists:foldr( fun(B, Acc2)-> lists:foldr( fun(C, Acc)-> case ((A*A + B*B =:= C*C) andalso (A+B+C =< Max)) of true-> [{A,B,C}|Acc]; false-> Acc end end , Acc2, L) end , Acc3, L) end , [], L). %% The original form 2. py2(Max)-> fora(1, [], Max). fora(A, Acc, Max)-> Acc1 = forb(A,1, Acc, Max), case A< Max of true-> fora(A+1, Acc1, Max); false-> Acc1 end. forb(A,B, Acc, Max)-> Acc1 = forc(A,B,1, Acc, Max), case B< Max of true-> forb(A,B+1, Acc1, Max); false-> Acc1 end. forc(A,B,C, Acc, Max)-> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true-> [{A,B,C}|Acc]; _-> Acc end, case C< Max of true-> forc(A,B,C+1, Acc1, Max); false-> Acc1 end. %% The original form 3. py3(Max)-> [{A,B,C} || A<-lists:seq(1, Max), B<-lists:seq(1, Max), C<-lists:seq(1, Max), A*A + B*B =:= C*C, A+B+C =< Max]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Tony's improvement of the original form 3. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag1(N) -> L = lists:seq(1,N), [ {A,B,C} || A<- L, B<- L, C<- L, A+B+C =< N, A*A+B*B =:= C*C]. %% Tony's implementation. pythag2(N) -> lists:reverse(pythan2_A(1, N, [])). pythan2_A(A, N, Acc) when A> N -> Acc; pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). pythan2_B(A, B, N, Acc) when A+B> N -> Acc; pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). pythan2_C(A, B, C, N, Acc) when A+B+C> N -> Acc; pythan2_C(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_C(A, B, C+1, N, Acc) end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Hynek's implementation. %% Simpler and about 5% faster version: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag3(N) when is_integer(N) -> pythag3(N,1). pythag3(N, A) when A+2> N -> []; pythag3(N, A) -> pythag3(N, A, 1). pythag3(N, A, B) when A+B+1> N -> pythag3(N, A+1); pythag3(N, A, B) -> pythag3(N, A, B, 1). pythag3(N, A, B, C) when A+B+C> N -> pythag3(N, A, B+1); pythag3(N, A, B, C) when A*A + B*B =:= C*C -> [{A, B, C}|pythag3(N, A, B, C+1)]; pythag3(N, A, B, C) -> pythag3(N, A, B, C+1). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Edmond's implementation using parallelism. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%---- START CODE ---- py2E(Max)-> lists:flatten(lpmap(fun(A) -> forbE(A, 1, [], Max) end, lists:seq(1, Max), ordered)). forbE(A, B, Acc, Max) -> Acc1 = forcE(A, B, 1, Acc, Max), case B< Max of true -> forbE(A, B+1, Acc1, Max); false -> Acc1 end. forcE(A, B, C, Acc, Max) -> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true -> [{A,B,C}|Acc]; _ -> Acc end, case C< Max of true-> forcE(A, B, C+1, Acc1, Max); false-> Acc1 end. pythag2E(N)-> lists:flatten(lpmap(fun(A) -> pythan2_BE(A, 1, N, []) end, lists:seq(1, N), ordered)). pythan2_AE(A, N, Acc) when A> N -> Acc; pythan2_AE(A, N, Acc) -> pythan2_AE(A+1,N,pythan2_BE(A, 1, N, Acc)). pythan2_BE(A, B, N, Acc) when A+B> N -> Acc; pythan2_BE(A, B, N, Acc) -> pythan2_BE(A,B+1,N,pythan2_CE(A, B, 1, N, Acc)). pythan2_CE(A, B, C, N, Acc) when A+B+C> N -> Acc; pythan2_CE(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_CE(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_CE(A, B, C+1, N, Acc) end. %% @spec lpmap(fun(), list(), (atom() = ordered|unordered)) -> list() %% @doc Spawns a process for each element in list L, performs specified %% function F against each in parallel and then returns results either %% same order as L (ordered) or in any order (unordered). %% NB: See also lpmap/4. lpmap(F, L, ordered) -> Ref = erlang:make_ref(), Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- L], lpmap_gather_ordered(Pids, Ref, [], 0, void); lpmap(F, L, unordered) -> Ref = erlang:make_ref(), lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, L), lpmap_gather_unordered(length(L), Ref, [], 0, void). %% @spec lpmap(fun(), integer(), list(), (atom() = ordered|unordered)) -> list() %% @doc Same as lpmap/3 except ensures only a maximum of MaxPs parallel %% processes execute function F at any one time (i.e. first takes MaxPs %% items from list, executes F in parallel against each, then as each %% process returns, spawns another process on next item in L as long as %% active processes are less than MaxPs). %% NB: See also lpmap/4. lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L)> MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- HPids], lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L)> MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, HPids), lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). %% lpmap internal functions lpmap_spawn_link(Parent, Ref, F, I) -> spawn_link(fun() -> Parent ! {self(), Ref, F(I)} end). lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> []; lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> receive {HPid, Ref, Ret} when length(TPids) [H | T] = L, [Ret | lpmap_gather_ordered( lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, H)]), Ref, T, MaxPs, F)]; {HPid, Ref, Ret} -> [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] end. lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> []; lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> receive {_Pid, Ref, Ret} when NPs-1 [H | T] = L, lpmap_spawn_link(self(), Ref, F, H), [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; {_Pid, Ref, Ret} -> [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] end. %%---- END CODE ----- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Willem's implementation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% wpythag2(N) -> L = [{A, A*A} || A<- lists:seq(1,N)], lists:flatten([forAllBs(A, A2, L, N) || {A, A2}<- L]). forAllBs(A, A2, L, N) -> [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N]. forAllCs(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Hynek's new version %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag4(N) when is_integer(N) -> pythag4(N,1). pythag4(N, A) when A+2> N -> []; pythag4(N, A) -> pythag4(N, A, A*A, 1). pythag4(N, A, _A2, B) when A+B+1> N -> pythag4(N, A+1); pythag4(N, A, A2, B) -> pythag4(N, A, A2, B, B*B, 1). pythag4(N, A, A2, B, _B2, C) when A+B+C> N -> pythag4(N, A, A2, B+1); pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; pythag4(N, A, A2, B, B2, C) -> pythag4(N, A, A2, B, B2, C+1). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Willem's implementation in parallel by Hynek %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% wpythag2P(N) -> L = [{A, A*A} || A<- lists:seq(1,N)], % For all A's lists:flatten(lpmap(fun({A, A2}) -> % For all B's in parallel [forAllCsWH(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N] end, L, 2000, ordered)). forAllCsWH(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Morten's implementation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag5(N) when is_integer(N) -> Primes = sieve(N div 2), M1M2s = incorporate_primes([{1,1}], N, Primes), lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2}<- M1M2s, M1> M2, A<- [M1-M2], B<- [2*round(math:sqrt(M1*M2))], C<- [M1+M2], A+B+C =< N])). sieve(N) when is_integer(N) -> erase(), sieve(N,2). sieve(N, K) when K>= N -> [X || X<- lists:seq(2, N), erase(X) == undefined]; sieve(N, K) -> cross_off(K, K, N div K - 1), sieve(N, find_next_in_sieve(K + 1)). cross_off(_K, _Current, 0) -> ok; cross_off(K, Current, Left) -> Next = Current + K, put(Next, out), cross_off(K, Next, Left - 1). find_next_in_sieve(K) -> case get(K) of undefined -> K; _ -> find_next_in_sieve(K+1) end. incorporate_prime(M1M2s, N, P) -> lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2}<- M1M2s]). incorporate_prime_single({M1,M2}, N, P) -> Evens = [{X, Y} || X<- incorporate_prime_even(M1, N, P), Y<- incorporate_prime_even(M2, N, P)], Odds = [{X, Y} || X<- incorporate_prime_odd(M1, N, P), Y<- incorporate_prime_odd(M2, N, P)], Evens ++ Odds. incorporate_prime_even(M, N, P) -> incorporate_prime(M, N, P, []). incorporate_prime_odd(M, N, P) -> incorporate_prime(M * P, N, P, []). incorporate_prime(M, N, _P, Acc) when M> N/2 -> Acc; incorporate_prime(M, N, P, Acc) -> incorporate_prime(M * P * P, N, P, [M|Acc]). incorporate_primes(M1M2s, _N, []) -> M1M2s; incorporate_primes(M1M2s, N, [P|Rest]) -> M1M2s_new = incorporate_prime(M1M2s, N, P), incorporate_primes(M1M2s_new, N, Rest). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Richard's improvement. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py3R(N)-> [{A,B,C} || C<- lists:seq(1, N), A<- lists:seq(1, N-C-1), B<- lists:seq(1, N-C-A), A*A + B*B =:= C*C, A+B+C =< N]. %%%%%%%%%%%%%%%%%%%%% END %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Cheers, Ivan. ---------------------------------------------------------------------- Do you want to get EVO (ExtendedVisualOtp) to develops client-server applicacions using C# / Erlang? Just say it!!!. ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From erlang@REDACTED Mon Nov 15 08:31:30 2010 From: erlang@REDACTED (Joe Armstrong) Date: Mon, 15 Nov 2010 08:31:30 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: Message-ID: I think you can do a wee bit better ... py3a(Max) -> N = Max div 2, [{A,B,C} || A <- lists:seq(1,N+1), B <- lists:seq(1,Max-A), C <- lists:seq(1,Max-A-B), A*A + B*B =:= C*C]. Since the largest side of the triangle (A) cannot be bigger than Max/2 /Joe On Mon, Nov 15, 2010 at 2:06 AM, Richard O'Keefe wrote: > > On 13/11/2010, at 8:37 PM, Gilberto Carmenate Garc?a wrote: > >> Hi all! >> I have been doing tests to Erlang I found this funny stuff that makes >> Pythagorean Triplets >> >> pythag(N) -> >> ? ?[ {A,B,C} || >> ? ? ? ?A <- lists:seq(1,N), >> ? ? ? ?B <- lists:seq(1,N), >> ? ? ? ?C <- lists:seq(1,N), >> ? ? ? ?A+B+C =< N, >> ? ? ? ?A*A+B*B =:= C*C]. > > That's not as efficient as it could be. > > ? ? ? ?C <- lists:seq(1, N), > ? ? ? ?A <- lists:seq(1, N-C-1), > ? ? ? ?B <- lists:seq(1, N-C-A), > > does a factor of 6 or better fewer iterations (tested up to N = 500). > > We can do better still. ?Since A^2 + B^2 = C^2, C >= max(A,B), > so it follows that A, B are both <= (N-1) div 2, and > given A and B there is only one possible C. > So there are only about (N*N)/4 cases to check instead of N*N*N, > > Here's a table. > > Fun ? N ? ? ?Iters ? Filtered ? ?Answers > old ?10 ? ? ? 1000 ? ? ? ?120 ? ? ? ? ?0 > new ?10 ? ? ? ?120 ? ? ? ?120 ? ? ? ? ?0 > sqt ?10 ? ? ? ? 16 ? ? ? ? 13 ? ? ? ? ?0 > old ?20 ? ? ? 8000 ? ? ? 1140 ? ? ? ? ?2 > new ?20 ? ? ? 1140 ? ? ? 1140 ? ? ? ? ?2 > sqt ?20 ? ? ? ? 81 ? ? ? ? 58 ? ? ? ? ?2 > old ?50 ? ? 125000 ? ? ?19600 ? ? ? ? 12 > new ?50 ? ? ?19600 ? ? ?19600 ? ? ? ? 12 > sqt ?50 ? ? ? ?576 ? ? ? ?372 ? ? ? ? 12 > old 100 ? ?1000000 ? ? 161700 ? ? ? ? 34 > new 100 ? ? 161700 ? ? 161700 ? ? ? ? 34 > sqt 100 ? ? ? 2401 ? ? ? 1513 ? ? ? ? 34 > old 200 ? ?8000000 ? ?1313400 ? ? ? ? 86 > new 200 ? ?1313400 ? ?1313400 ? ? ? ? 86 > sqt 200 ? ? ? 9801 ? ? ? 6094 ? ? ? ? 86 > old 500 ?125000000 ? 20708500 ? ? ? ?274 > new 500 ? 20708500 ? 20708500 ? ? ? ?274 > sqt 500 ? ? ?62001 ? ? ?38258 ? ? ? ?274 > > Fun is old (your loop nest), new (my CAB loop nest above), or > sqt (AB loop nest with C = hypot(A,B) truncated to integer); > N is N; Iters is the number of iterations of the loop nest; > Filtered is the number of cases that pass the A+B+C =< N > filter, and Answers is the number of triples found this way. > > !!!! Looking at the last line, we see a reduction in iterations > !!!! by a factor of 2016.1, which dwarfs any difference between > !!!! Erlang and C#. ?(And it's possible to do even better.) > > One issue of course is that C# is presumably not building > and iterating over any lists at all here, so if you want > a *fair* comparison between C# and Erlang, your iterations > should not iterate over any lists either. > > ? ? ? ?p > > > > > >> >> I tested it agains an implementation I made in C# so, and takes 14 >> secounds in my pc to do with 300 numbers in Erlang however in c# is just >> a secound, even when C# runs under VM too. >> So I did all possible ways for me to implement differents manners in >> Erlang looking for speed and all is the same, listed as ?follows: >> >> So my question is, there are any way to do that even more fast, why 3 >> nestes fors structs in C# are more effients that lists:foldr or >> lists:foreach in Erlang. >> >> %% FORMA 1 >> py1(Max)-> >> ? ?L = lists:seq(1, Max), >> ? ?lists:foldr( >> ? ? ? ?fun(A, Acc3)-> >> ? ? ? ? ? ?lists:foldr( >> ? ? ? ? ? ? ? ?fun(B, Acc2)-> >> ? ? ? ? ? ? ? ? ? ?lists:foldr( >> ? ? ? ? ? ? ? ? ? ? ? ?fun(C, Acc)-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ?case ((A*A + B*B =:= C*C) andalso (A+B+C =< >> Max)) of >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{A,B,C}|Acc]; >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?false-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Acc >> ? ? ? ? ? ? ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ? ? ? ? ?, Acc2, L) >> ? ? ? ? ? ? ? ?end >> ? ? ? ? ? ?, Acc3, L) >> ? ? ? ?end >> ? ?, [], L). >> >> >> >> %% FORMA 2 >> py2(Max)-> >> ? ? ? fora(1, [], Max). >> >> fora(A, Acc, Max)-> >> ? ? ? Acc1 = forb(A,1, Acc, Max), >> ? ? ? case A < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?fora(A+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> forb(A,B, Acc, Max)-> >> ? ?Acc1 = forc(A,B,1, Acc, Max), >> ? ?case B < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?forb(A,B+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> forc(A,B,C, Acc, Max)-> >> ? ?Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of >> ? ? ? ?true-> >> ? ? ? ? ? ?[{A,B,C}|Acc]; >> ? ? ? ?_-> >> ? ? ? ? ? ?Acc >> ? ?end, >> ? ?case C < Max of >> ? ? ? ?true-> >> ? ? ? ? ? ?forc(A,B,C+1, Acc1, Max); >> ? ? ? ?false-> >> ? ? ? ? ? ?Acc1 >> ? ?end. >> >> %% FORMA 3. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de ?usar el servicio a tales fines y cumplir con las regulaciones establecidas. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ulf.wiger@REDACTED Mon Nov 15 09:23:42 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 15 Nov 2010 09:23:42 +0100 Subject: Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 Message-ID: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> Just a short note for those of you Down Under. I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec 8-9). http://www.yowconference.com.au/index.html This should be a good opportunity for those interested in Erlang to get together, ask questions and perhaps talk over beer afterwards. I'd love to hear from some locals about how we can make the most of the opportunity. The conference itself will feature talks by me, Justin Sheehy and Kresten Krab Thorup - as well as a load of distinguished speakers from different camps. http://www.yowconference.com.au/melbourne/speakers/index.html It should be a hoot. Please sign up and recruit some friends too. I hope to see you there. BR, Ulf W Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From raimo+erlang-questions@REDACTED Mon Nov 15 09:29:25 2010 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Mon, 15 Nov 2010 09:29:25 +0100 Subject: [erlang-questions] A tribute to Erlang/OTP In-Reply-To: References: Message-ID: <20101115082925.GA6134@erix.ericsson.se> On Sun, Nov 14, 2010 at 11:03:24PM +0100, ErlangPriest wrote: > Hi, > > With the upcoming EUC 2010 and Thanksgiving lurking around the corner I felt > that it some sort of tribute/thanks to Erlang/OTP and the great team behind > it was long overdue, so I wrote a song that I would like to share with you > all. > > It does not compile yet (i.e., the lyrics does not quite fit the tune), but > I hope that someone with greater artistic talents than me will run it > through dialyzer and QuickCheck to make everything correct ;-) > > Happy EUC2010 and happy Thanksgiving! > The ErlangPriest Awsm! I can't wait to hear it. (I can almost hear it in my head already) > > ============================= > Deadlock free holiday > ============== > Original title: Dreadlock Holiday > Original authors: Graham Keith Gouldman; Eric Michael Stewart > Adapted to Erlang/OTP by the ErlangPriest. > > I was workin? on the API, concentratin? on codin? right > I heard a dark voice beside of me > And I look ?round in a state of fright > I saw four faces, one mad; a dude from the management > They looked me up and down and turned to each other > > I say, I don?t like Erlang, oh no, I love it > I don?t like Erlang, oh no, I love it. > > Don?t you walk through my code > You got to show some insight > Don?t you walk though my code > Cause you ain?t an Erlanger yet > > Well, he looked down on my silver bullet > He said: ?I?ll give you one dollar? > I said: ?You?ve got to be jokin?, man > It was a present from Ericsson? > He said: ?I like it, I want it, I?ll take it off your hands > And you?ll be sorry you crossed me > You better understand > That you?re alone (many nodes away from home)?. > > And I say, I don?t like spawn, oh no, I love it > I don?t like spawn, oh no, I love it > > Don?t you cramp me style, don?t you queer me pitch > Don?t you walk through my code > ?Cause you ain?t an Erlanger yet > > I hurried back to the Erlang prompt, passin? atoms around > I heard a dark voice inside of me say > ?Would you like something harder?? > She said: I?ve got it, you want it, my callbacks are the best > And if you try it > You?ll like it and whollow in deadlock free holiday. > > And I say, don?t like OTP, oh no, I love her > Don?t like OTP, oh no, I love her, oh yeah > > Don?t you walk through her code > You go to show some insight > Don?t you walk throug her code > ?Cause you ain?t an Erlanger yet > > I don?t like Erlang, oh no, I love it > (Deadlock free holiday) > I don?t like spawn, oh no, I love it > (Deadlock free holiday) > I don?t like OTP, oh no, I love her > (Deadlock free holiday) > =============================== -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From huss01@REDACTED Mon Nov 15 10:38:41 2010 From: huss01@REDACTED (=?ISO-8859-1?Q?H=E5kan_Huss?=) Date: Mon, 15 Nov 2010 10:38:41 +0100 Subject: [erlang-questions] Hi all here the RESUME for Pytagorians Numbers Algorithm!!! In-Reply-To: <4CE0B634.9060407@frcuba.co.cu> References: <4CE0B634.9060407@frcuba.co.cu> Message-ID: 2010/11/15 Gilberio Carmenates Garc?a : > > py3R NO returns the same than 'pythag1' %% I just tries to put the > Richard explanation in a > algorithm but, seems to me I > was wrong here too. I interpreted Richard's algorithm to be something along the lines of: py3R(N)-> [{A,B,C} || A <- lists:seq(1, N div 2), B <- lists:seq(1, N - A), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C]. This version comes out on top when N is in the hundreds and above. Below this it is omly matched by Morten's algorithm. I would argue that Richard's is slightly less complex... Regards, /H?kan From pan+eq@REDACTED Mon Nov 15 11:11:16 2010 From: pan+eq@REDACTED (pan+eq@REDACTED) Date: Mon, 15 Nov 2010 11:11:16 +0100 (CET) Subject: [erlang-questions] [eeps] EEP 35 "Binary string modules" In-Reply-To: <4CDC35AC.8030900@tmit.bme.hu> References: <4CDC35AC.8030900@tmit.bme.hu> Message-ID: Hi! This is a really good point! I've taken the liberty to forvard your comment to the EEP mailing list, where discussions about the written EEP's are supposed to be held. Please subscribe to that list if you haven't already. Cheers, /Patrik On Thu, 11 Nov 2010, Zoltan Lajos Kis wrote: > My opinion is that it would be more convenient to have a (single) module that > provides string handling on the iodata() data type. > > Right now we have some functionality available only for list strings, some > for binary strings, and some for iodata strings; > some with unicode support, some without. This new module would definitely > broaden the available functionality, but it > feels like we will still have to convert b/w iodata, lists and binaries. > > Regards, > Zoltan. > > > On 11/11/2010 4:31 PM, Kenneth Lundin wrote: >> We have a quite new EEP (EEP 35) with the title "Binary string >> module(s) that we want your feedback on so we can >> start implement it. >> >> We would like to get your feedback before November 25. >> >> If you are attending the Erlang User Conference next week , take the >> opportunity to discuss the EEP with your fellow Erlangers and >> with us in the OTP team at Ericsson. >> >> /Kenneth, Erlang/OTP Ericsson >> >> >> ---------- Forwarded message ---------- >> From: Raimo Niskanen >> Date: Mon, Oct 4, 2010 at 5:36 PM >> Subject: [eeps] EEP 35 >> To: eeps@REDACTED >> >> >> A new EEP >> EEP 35: Binary string module(s) >> has been added. See >> http://demo.erlang.org/eeps/ >> andalso >> http://github.com/erlang/eep/tree/master//eeps/ >> >> -- >> >> / Raimo Niskanen, Erlang/OTP, Ericsson AB >> >> ________________________________________________________________ >> eeps (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:eeps-unsubscribe@REDACTED >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From pan+eq@REDACTED Mon Nov 15 11:13:20 2010 From: pan+eq@REDACTED (pan+eq@REDACTED) Date: Mon, 15 Nov 2010 11:13:20 +0100 (CET) Subject: [erlang-questions] [eeps] EEP 35 "Binary string modules" -- locales In-Reply-To: <87y68yljhb.fsf@pepper.mti.ag> References: <87y68yljhb.fsf@pepper.mti.ag> Message-ID: Hi, I forwarded this comment (as well) to the EEP mailing list and will answer there. Cheers, /Patrik On Fri, 12 Nov 2010, Christian von Roques wrote: > Not all text is meant for human consumption. I'd even venture so far as > to say that the overwhelming mass of program generated text is not for > human consumption, its intended consumers are other programs. The most > common locale programs "speak" is the default "C" (also called "POSIX") > locale. It is complicated to solve the general problem of supporting > all human locales. It is much simpler to just support a default locale. > Even programs intended to create/consume text for humans often have to > create/consume text in the C locale as well. > > I've been told the anecdote that in the 70s a delegation of IBM compiler > engineers flew to Germany to proudly demonstrate their new optimizing > Fortran compiler and all it did was spew gibberish and crash because it > used the standard routines for reading/writing numbers, which in Germany > used commas for dots and dots for commas due to the then new locale > awareness of the OS. Since then I've been convinced that it is a good > thing to have two separate sets of functions, one small, simple, and > fast handling only the default locale and another one huge, complicated, > and not so fast trying to handle all the intricacies of as many locales > as feasible. > > Therefore I'd like to see to_integer and to_float in bstring, grokking > numbers in the C locale. to_lower and to_upper too as long as it's > documented on which characters they are working on. They wouldn't even > need to know if the bstring was iso8859-1 or utf-8 encoded as long as > they only touch ASCII characters. > > I don't think it's practical to see bstring as locale independent. > Rather bstring should be seen as operating in the default locale. One > being able to imagine a locale dependent variant of a function should > not be ground for omitting the function from bstring. I can even > imagine concat(<<"Fu?">>, <<"Ball">>) being expected to result in > <<"Fussball">> in the DE_de locale. > > Christian. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From alessandro.sivieri@REDACTED Mon Nov 15 11:19:45 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Mon, 15 Nov 2010 11:19:45 +0100 Subject: [erlang-questions] Re: SSL and client authentication In-Reply-To: References: <1289750585.2064.ezmlm@erlang.org> Message-ID: 2010/11/15 > However, the SSL handshake *has* occured by the time the response is sent > back, so it shouldn't be impossible for the server to check the URL and > (whether a/which) client certificate was presented, and serve or deny the > request as appropriate. > > This is what I thought, too; Apache and Nginx allow a user to configure which Web paths have to be mutually authenticated and which not, even if both use HTTPS... > Since the OP has decided to use nginx, there's not much point in going > into the mechanics of how to do that in Erlang. > > So there is a way to do this in (pure) Erlang? -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From saager.mhatre@REDACTED Mon Nov 15 12:05:30 2010 From: saager.mhatre@REDACTED (Saager Mhatre) Date: Mon, 15 Nov 2010 16:35:30 +0530 Subject: [erlang-questions] Re: SSL and client authentication In-Reply-To: References: <1289750585.2064.ezmlm@erlang.org> Message-ID: Shouldn't it be trivial to kick off another https process to do the ssl handoff for various urls? Sent from my Motorola Milestone. On Nov 15, 2010 3:51 PM, "Alessandro Sivieri" wrote: 2010/11/15 > However, the SSL handshake *has* occured by the time the response is sent > back, so it shouldn't... This is what I thought, too; Apache and Nginx allow a user to configure which Web paths have to be mutually authenticated and which not, even if both use HTTPS... > Since the OP has decided to use nginx, there's not much point in going > into the mechanics of h... So there is a way to do this in (pure) Erlang? -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.... From ebegumisa@REDACTED Mon Nov 15 14:36:41 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 00:36:41 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <03BC861E-8141-4F78-82F2-F473A3DAD291@cs.otago.ac.nz> References: <03BC861E-8141-4F78-82F2-F473A3DAD291@cs.otago.ac.nz> Message-ID: Yes, I corrected myself later on that. - Edmond - On Mon, 15 Nov 2010 12:24:59 +1100, Richard O'Keefe wrote: > > On 14/11/2010, at 3:36 AM, Edmond Begumisa wrote: >> Numerical algorithms? > > Enumerating pythagorean triples is not a "numerical" algorithm. > It's integer-only, and it doesn't even need large integers. > As far as I know there's nothing in ACML or GSL that would help. > You could call it combinatorics or number theory. >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Nov 15 14:41:49 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 00:41:49 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <03BC861E-8141-4F78-82F2-F473A3DAD291@cs.otago.ac.nz> References: <03BC861E-8141-4F78-82F2-F473A3DAD291@cs.otago.ac.nz> Message-ID: On Sun, 14 Nov 2010 11:33:25 +1100, Edmond Begumisa wrote: > ... When you look closer, the bottleneck with all the solutions so far > isn't the calculation itself (I was wrong about that earlier) -- it's > actually the permutation (the part done with accumulators/generators) ... - Edmond - On Mon, 15 Nov 2010 12:24:59 +1100, Richard O'Keefe wrote: > > On 14/11/2010, at 3:36 AM, Edmond Begumisa wrote: >> Numerical algorithms? > > Enumerating pythagorean triples is not a "numerical" algorithm. > It's integer-only, and it doesn't even need large integers. > As far as I know there's nothing in ACML or GSL that would help. > You could call it combinatorics or number theory. >> >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From pablo.platt@REDACTED Mon Nov 15 15:11:46 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Mon, 15 Nov 2010 06:11:46 -0800 (PST) Subject: gproc global will create a bootleneck in my app? Message-ID: <868787.92083.qm@web112610.mail.gq1.yahoo.com> Hi, I'm using gproc to handle IM sessions on one node and it works great. I want to use it with gen_leader to extend to several nodes. Saving a global name will be slower because that leader has to coordinate uniqueness. Will it affect reads as well? If one gen_server is responsible for everything, does writing a global name might block all reads? What is a reasonable number of unique names and non-unique properties I can save with gproc? Am I only limited by RAM? Thanks From ebegumisa@REDACTED Mon Nov 15 15:15:56 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 01:15:56 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: On Mon, 15 Nov 2010 04:26:12 +1100, Toby Thain wrote: > What were these great sacrifices? Pointers? > > --Toby http://www.erlang.org/faq/introduction.html#id49850 But yeah, "trade-offs" would have been a better term than "sacrifices". - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Nov 15 16:02:50 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 02:02:50 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: Hello Richard, I have a question... > Erlang is a fine tool for jobs like this one. Interesting... I would have never thought this. I would have instinctively reached for a language with mutable memory, thinking that this would make the most sense for combinatorial* work (I don't know this for fact, and I could be very wrong, but that's what instinct would tell me.) But then, I'm relatively new to Erlang. * I've been calling this permutation but I don't know if that's accurate. > There is little point in optimising a bad algorithm. Well put. But say you have an 'ok' algorithm. Not the best, but not blatantly flawed either. I think of optimisation as something you put on a version 2 to-do list. Optimisation to me means staring hard at code and trying to figure out ways to get it to perform better (faster, less memory, less CPU time, etc). This normally means re-writing code that's easy to follow into code that's no-so-easy to follow. Mind you, for Erlang, I don't look at parallelising as optimisation as others seem to. To me, it's just a building block made available and normally it can be applied without changing an algo too much (I'd say it's not used enough). Of all the variations presented so far, IMO, Garcia's first is the easiest to follow (ignoring that obvious flaw with the repeated list:seq call). A non-mathematical mind like myself can see exactly what he's trying to do. The others (including the ones from Willem and Tony that I tried to parallelise), seem harder to follow. Maybe that's coz they are optimised versions. The authors could have stared long and hard. So my question is: if version 1 isn't performing "reasonably" acceptably for Garcia's purpose, and version 1 isn't blatantly flawed. Isn't this a strong indicator that he's using the wrong tool? - Edmond - On Mon, 15 Nov 2010 12:29:49 +1100, Richard O'Keefe wrote: > > On 14/11/2010, at 10:17 PM, Hynek Vychodil wrote: > Anyway Erlang is not >> right tool for this job. You should use NIF if performance matter. > > Erlang is a fine tool for jobs like this one. > In fact, given its support for large integers, > it is about as good for combinatorial algorithms as > Smalltalk, though perhaps not as good as Lisp. > (But see LFE...) > > There is little point in optimising a bad algorithm. > A fairly naive rewrite turns it from O(N**3) into O(N**2). > What really counts here is how easy it is to spot the > algorithmic problem and switch to a better algorithm. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From mononcqc@REDACTED Mon Nov 15 16:21:44 2010 From: mononcqc@REDACTED (Fred Hebert) Date: Mon, 15 Nov 2010 10:21:44 -0500 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: I think the first thing to do before optimizing is understanding how you use the data. In the case of Pythagorean triplets like these, the right question might as well be "how many triplets do I need? What values do I need to work with?" If 90% of your calls fall in the same range, you might as well just cache the results and speed up 90% of the computations. No optimization required. Then to reduce the overhead of the rest and maybe make the whole set of queries more predictable, you could use Morgen's algorithm with a static or a cached sieve table/incorporated primes for the edge cases, or allow the algorithm to resume from the one you stored, etc. There are many practical solutions that can require far less work than rewriting the algorithm, making it harder to read. Understand the problem before finding its solution. On Mon, Nov 15, 2010 at 10:02 AM, Edmond Begumisa < ebegumisa@REDACTED> wrote: > > I think of optimisation as something you put on a version 2 to-do list. > Optimisation to me means staring hard at code and trying to figure out ways > to get it to perform better (faster, less memory, less CPU time, etc). This > normally means re-writing code that's easy to follow into code that's > no-so-easy to follow. Mind you, for Erlang, I don't look at parallelising as > optimisation as others seem to. To me, it's just a building block made > available and normally it can be applied without changing an algo too much > (I'd say it's not used enough). > From ebegumisa@REDACTED Mon Nov 15 16:28:15 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 02:28:15 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: Small Correction... > Of all the variations presented so far, IMO, Garcia's first is the > easiest to follow They've been since some variations of this that are equally easy to follow and try to eliminate small but important flaws (including yours). The one's that try to use functions instead of list combination generators for the combinatorial part are the ones I have trouble reading. But that may just be me. - Edmond - On Tue, 16 Nov 2010 02:02:50 +1100, Edmond Begumisa wrote: > Hello Richard, > > I have a question... > >> Erlang is a fine tool for jobs like this one. > > Interesting... I would have never thought this. I would have > instinctively reached for a language with mutable memory, thinking that > this would make the most sense for combinatorial* work (I don't know > this for fact, and I could be very wrong, but that's what instinct would > tell me.) But then, I'm relatively new to Erlang. > > * I've been calling this permutation but I don't know if that's accurate. > >> There is little point in optimising a bad algorithm. > > Well put. But say you have an 'ok' algorithm. Not the best, but not > blatantly flawed either. > > I think of optimisation as something you put on a version 2 to-do list. > Optimisation to me means staring hard at code and trying to figure out > ways to get it to perform better (faster, less memory, less CPU time, > etc). This normally means re-writing code that's easy to follow into > code that's no-so-easy to follow. Mind you, for Erlang, I don't look at > parallelising as optimisation as others seem to. To me, it's just a > building block made available and normally it can be applied without > changing an algo too much (I'd say it's not used enough). > > Of all the variations presented so far, IMO, Garcia's first is the > easiest to follow (ignoring that obvious flaw with the repeated list:seq > call). A non-mathematical mind like myself can see exactly what he's > trying to do. The others (including the ones from Willem and Tony that I > tried to parallelise), seem harder to follow. Maybe that's coz they are > optimised versions. The authors could have stared long and hard. > > So my question is: if version 1 isn't performing "reasonably" acceptably > for Garcia's purpose, and version 1 isn't blatantly flawed. Isn't this a > strong indicator that he's using the wrong tool? > > - Edmond - > > On Mon, 15 Nov 2010 12:29:49 +1100, Richard O'Keefe > wrote: > >> >> On 14/11/2010, at 10:17 PM, Hynek Vychodil wrote: >> Anyway Erlang is not >>> right tool for this job. You should use NIF if performance matter. >> >> Erlang is a fine tool for jobs like this one. >> In fact, given its support for large integers, >> it is about as good for combinatorial algorithms as >> Smalltalk, though perhaps not as good as Lisp. >> (But see LFE...) >> >> There is little point in optimising a bad algorithm. >> A fairly naive rewrite turns it from O(N**3) into O(N**2). >> What really counts here is how easy it is to spot the >> algorithmic problem and switch to a better algorithm. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Nov 15 16:52:57 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 02:52:57 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: On Tue, 16 Nov 2010 02:21:44 +1100, Fred Hebert wrote: > > If 90% of your calls fall in the same range, you might as well just cache > the results and speed up 90% of the computations. No optimization > required. > Then to reduce the overhead of the rest and maybe make the whole set of > queries more predictable, you could use Morgen's algorithm with a static > or > a cached sieve table/incorporated primes for the edge cases, or allow the > algorithm to resume from the one you stored, etc. I thought about doing something like that. I tried to write some code my side with using an ets table but at the end of the day got no speed ups coz ets:insert had to be called as I populated the cache. I suspect it might be that the table had to keep growing. I haven't used ets much -- is there a way of creating an empty table of certain size in one go? - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Nov 15 17:30:27 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 03:30:27 +1100 Subject: [erlang-questions] Hi all here the RESUME for Pytagorians Numbers Algorithm!!! In-Reply-To: <4CE0B634.9060407@frcuba.co.cu> References: <4CE0B634.9060407@frcuba.co.cu> Message-ID: It's interesting to see that parallelising on a single core comes at 0 cost. Why did I think parallel versions would be slightly slower on a single core machine? I was even tempted to put in a test and only go parallel if there's more than one core. BTW, are you sure your py2 algo is wrong or does it just return the list in a different order? Try sorting the lists before the =:= test. - Edmond - On Mon, 15 Nov 2010 15:25:24 +1100, Gilberio Carmenates Garc?a wrote: > > Made on a pc with Petium 4 CPU 1.6 Ghz one core, 512 Ram. > > *** Pytagoriam's Numbers REPORT! *** > > Using N = 300 > > *** THE ORIGINAL IMPLEMENTATIONS (ME!)**** > -py1: 15547000 mcs = 15.547 s > -py2: 14514999 mcs = 14.514999 s > -py3: 14468999 mcs = 14.468999 s > > *** Tony's implementations **** > -pythag1: 13218999 mcs = 13.218999 s > -pythag2: 2296999 mcs = 2.296999 s > > *** Hynek's implementation **** > Simpler and about 5% faster version > -pythag3: 2202999 mcs = 2.202999 s > > *** Edmond's implementation, using parallelism**** > -py2E: 14624999 mcs = 14.624999 s > > *** Willem's implementation**** > -wpythag2: 2202999 mcs = 2.202999 s > > *** Hynek's new implementation**** > -pythag4: 1530999 mcs = 1.530999 s > > *** Willem's new implementation in parallel by Hynek**** > -wpythag2P: 2202999 mcs = 2.202999 s > > *** Morten's implementation**** > -pythag5: 62999 mcs = 0.062999 s > > *** Richard's improvement**** > -py3R: 2452999 mcs = 2.452999 s > > Comparisons in results agains 'pythag1' Tony's function > py1 returns the same than 'pythag1' > py2 NO returns the same than 'pythag1' %% since my secound > implementation 'py2' is wrong!!! > py3 returns the same than 'pythag1' > pythag1 returns the same than 'pythag1' > pythag2 returns the same than 'pythag1' > pythag3 returns the same than 'pythag1' > py2E NO returns the same than 'pythag1' %% since Edmond use my > wrong implementation. > wpythag2 returns the same than 'pythag1' > pythag4 returns the same than 'pythag1' > wpythag2P returns the same than 'pythag1' > pythag5 returns the same than 'pythag1' > py3R NO returns the same than 'pythag1' %% I just tries to put the > Richard explanation in a > algorithm but, seems to > me I was wrong here too. > > NOTE: Time was took using timer:tc/3 function. > > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Here the source code of all algorithm > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > > > -module(py). > -compile([export_all]). > > > start(N)-> > %% My implementations. > {T1,R1} = timer:tc(py,py1, [N]), > {T2,R2} = timer:tc(py,py2,[N]), > {T3,R3} = timer:tc(py,py3,[N]), > > %% Tony's improvement of the original form 3. > {T4,R4} = timer:tc(py,pythag1,[N]), > %% Tony's implementation. > {T5,R5} = timer:tc(py,pythag2,[N]), > > %% Hynek's implementation. > %% Simpler and about 5% faster version: > {T6,R6} = timer:tc(py,pythag3,[N]), > > %% Edmond's implementation using parallelism. > {T7,R7} = timer:tc(py,py2E,[N]), > > %% Willem's implementation. > {T8,R8} = timer:tc(py,wpythag2,[N]), > > %% Hynek's new version > {T9,R9} = timer:tc(py,pythag4,[N]), > > %% Willem's implementation in parallel by Hynek > {T10,R10} = timer:tc(py,wpythag2P,[N]), > > %% Morten's implementation. > {T11,R11} = timer:tc(py,pythag5,[N]), > > %% Richard's improvement. > {T12,R12} = timer:tc(py,py3R,[N]), > > io:format("~n *** Pytagoriam's Numbers REPORT! ***~n~n"), > io:format("Using N = ~p~n", [N]), > > io:format("~n*** THE ORIGINAL IMPLEMENTATIONS (ME!)****~n"), > io:format("-py1: ~p mcs = ~p s~n", [T1,T1/1000000]), > io:format("-py2: ~p mcs = ~p s~n", [T2,T2/1000000]), > io:format("-py3: ~p mcs = ~p s~n", [T3,T3/1000000]), > > io:format("~n*** Tony's implementations ****~n"), > io:format("-pythag1: ~p mcs = ~p s~n", [T4,T4/1000000]), > io:format("-pythag2: ~p mcs = ~p s~n", [T5,T5/1000000]), > > io:format("~n*** Hynek's implementation ****~n"), > io:format("Simpler and about 5% faster version~n"), > io:format("-pythag3: ~p mcs = ~p s~n", [T6,T6/1000000]), > > io:format("~n*** Edmond's implementation, using parallelism****~n"), > io:format("-py2E: ~p mcs = ~p s~n", [T7,T7/1000000]), > > io:format("~n*** Willem's implementation****~n"), > io:format("-wpythag2: ~p mcs = ~p s~n", [T8,T8/1000000]), > > io:format("~n*** Hynek's new implementation****~n"), > io:format("-pythag4: ~p mcs = ~p s~n", [T9,T9/1000000]), > > io:format("~n*** Willem's new implementation in parallel by > Hynek****~n"), > io:format("-wpythag2P: ~p mcs = ~p s~n", [T10,T10/1000000]), > > io:format("~n*** Morten's implementation****~n"), > io:format("-pythag5: ~p mcs = ~p s~n", [T11,T11/1000000]), > > io:format("~n*** Richard's improvement****~n"), > io:format("-py3R: ~p mcs = ~p s~n", [T12,T12/1000000]), > > io:format("~nComparisons in results agains 'pythag1' Tony's > function~n"), > Rs = [{py1,R1}, {py2,R2}, {py3,R3}, {pythag1,R4}, {pythag2,R5}, > {pythag3,R6}, > {py2E,R7}, {wpythag2,R8}, {pythag4,R9}, {wpythag2P,R10}, > {pythag5,R11}, {py3R,R12}], > lists:foreach(fun({Name, R})-> > if > (R=:=R4)-> > io:format("~p returns the same than 'pythag1'~n", > [Name]); > true-> > io:format("~p NO returns the same than 'pythag1'~n", > [Name]) > end > end, Rs), > io:format("~nNOTE: Time took using timer:tc/3 function.~n"). > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% The original form 1. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > py1(Max)-> > L = lists:seq(1, Max), > lists:foldr( > fun(A, Acc3)-> > lists:foldr( > fun(B, Acc2)-> > lists:foldr( > fun(C, Acc)-> > case ((A*A + B*B =:= C*C) andalso (A+B+C =< > Max)) of > true-> > [{A,B,C}|Acc]; > false-> > Acc > end > end > , Acc2, L) > end > , Acc3, L) > end > , [], L). > > > > %% The original form 2. > py2(Max)-> > fora(1, [], Max). > > fora(A, Acc, Max)-> > Acc1 = forb(A,1, Acc, Max), > case A< Max of > true-> > fora(A+1, Acc1, Max); > false-> > Acc1 > end. > > forb(A,B, Acc, Max)-> > Acc1 = forc(A,B,1, Acc, Max), > case B< Max of > true-> > forb(A,B+1, Acc1, Max); > false-> > Acc1 > end. > > forc(A,B,C, Acc, Max)-> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true-> > [{A,B,C}|Acc]; > _-> > Acc > end, > case C< Max of > true-> > forc(A,B,C+1, Acc1, Max); > false-> > Acc1 > end. > > %% The original form 3. > py3(Max)-> > [{A,B,C} || > A<-lists:seq(1, Max), > B<-lists:seq(1, Max), > C<-lists:seq(1, Max), > A*A + B*B =:= C*C, > A+B+C =< Max]. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Tony's improvement of the original form 3. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > pythag1(N) -> > L = lists:seq(1,N), > [ {A,B,C} || > A<- L, > B<- L, > C<- L, > A+B+C =< N, > A*A+B*B =:= C*C]. > > %% Tony's implementation. > pythag2(N) -> > lists:reverse(pythan2_A(1, N, [])). > > pythan2_A(A, N, Acc) when A> N -> Acc; > pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). > > pythan2_B(A, B, N, Acc) when A+B> N -> Acc; > pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, > Acc)). > > pythan2_C(A, B, C, N, Acc) when A+B+C> N -> Acc; > pythan2_C(A, B, C, N, Acc) -> > if A*A+B*B =:= C*C -> > pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); > true -> > pythan2_C(A, B, C+1, N, Acc) > end. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Hynek's implementation. > %% Simpler and about 5% faster version: > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > pythag3(N) when is_integer(N) -> pythag3(N,1). > > pythag3(N, A) when A+2> N -> []; > pythag3(N, A) -> pythag3(N, A, 1). > > pythag3(N, A, B) when A+B+1> N -> pythag3(N, A+1); > pythag3(N, A, B) -> pythag3(N, A, B, 1). > > pythag3(N, A, B, C) when A+B+C> N -> pythag3(N, A, B+1); > pythag3(N, A, B, C) when A*A + B*B =:= C*C -> [{A, B, C}|pythag3(N, A, > B, C+1)]; > pythag3(N, A, B, C) -> pythag3(N, A, B, C+1). > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Edmond's implementation using parallelism. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > %%---- START CODE ---- > > py2E(Max)-> > lists:flatten(lpmap(fun(A) -> > forbE(A, 1, [], Max) > end, lists:seq(1, Max), ordered)). > > forbE(A, B, Acc, Max) -> > Acc1 = forcE(A, B, 1, Acc, Max), > case B< Max of > true -> forbE(A, B+1, Acc1, Max); > false -> Acc1 > end. > > forcE(A, B, C, Acc, Max) -> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true -> [{A,B,C}|Acc]; > _ -> Acc > end, > case C< Max of > true-> forcE(A, B, C+1, Acc1, Max); > false-> Acc1 > end. > > > pythag2E(N)-> > lists:flatten(lpmap(fun(A) -> > pythan2_BE(A, 1, N, []) > end, lists:seq(1, N), ordered)). > > pythan2_AE(A, N, Acc) when A> N -> Acc; > pythan2_AE(A, N, Acc) -> pythan2_AE(A+1,N,pythan2_BE(A, 1, N, Acc)). > > pythan2_BE(A, B, N, Acc) when A+B> N -> Acc; > pythan2_BE(A, B, N, Acc) -> pythan2_BE(A,B+1,N,pythan2_CE(A, B, 1, N, > Acc)). > > pythan2_CE(A, B, C, N, Acc) when A+B+C> N -> Acc; > pythan2_CE(A, B, C, N, Acc) -> > if A*A+B*B =:= C*C -> > pythan2_CE(A, B, C+1, N, [{A,B,C}|Acc]); > true -> > pythan2_CE(A, B, C+1, N, Acc) > end. > > %% @spec lpmap(fun(), list(), (atom() = ordered|unordered)) -> list() > %% @doc Spawns a process for each element in list L, performs > specified > %% function F against each in parallel and then returns results > either > %% same order as L (ordered) or in any order (unordered). > %% NB: See also lpmap/4. > > lpmap(F, L, ordered) -> > Ref = erlang:make_ref(), > Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- L], > lpmap_gather_ordered(Pids, Ref, [], 0, void); > lpmap(F, L, unordered) -> > Ref = erlang:make_ref(), > lists:foreach(fun(I) -> > lpmap_spawn_link(self(), Ref, F, I) > end, L), > lpmap_gather_unordered(length(L), Ref, [], 0, void). > > %% @spec lpmap(fun(), integer(), list(), (atom() = > ordered|unordered)) -> list() > %% @doc Same as lpmap/3 except ensures only a maximum of MaxPs > parallel > %% processes execute function F at any one time (i.e. first > takes MaxPs > %% items from list, executes F in parallel against each, then > as each > %% process returns, spawns another process on next item in L as > long as > %% active processes are less than MaxPs). > %% NB: See also lpmap/4. > > lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> > Ref = erlang:make_ref(), > {HPids, TPids} = if > length(L)> MaxPs -> lists:split(MaxPs, L); > true -> {L, []} > end, > Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- HPids], > lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); > lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> > Ref = erlang:make_ref(), > {HPids, TPids} = if > length(L)> MaxPs -> lists:split(MaxPs, L); > true -> {L, []} > end, > lists:foreach(fun(I) -> > lpmap_spawn_link(self(), Ref, F, I) > end, HPids), > lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). > > %% lpmap internal functions > > lpmap_spawn_link(Parent, Ref, F, I) -> > spawn_link(fun() -> > Parent ! {self(), Ref, F(I)} > end). > > lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> > []; > lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> > receive > {HPid, Ref, Ret} when length(TPids) > [H | T] = L, > [Ret | lpmap_gather_ordered( > lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, > H)]), > Ref, T, MaxPs, F)]; > {HPid, Ref, Ret} -> > [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] > end. > > lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> > []; > lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> > receive > {_Pid, Ref, Ret} when NPs-1 > [H | T] = L, > lpmap_spawn_link(self(), Ref, F, H), > [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; > {_Pid, Ref, Ret} -> > [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] > end. > > > %%---- END CODE ----- > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Willem's implementation. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > wpythag2(N) -> > L = [{A, A*A} || A<- lists:seq(1,N)], > lists:flatten([forAllBs(A, A2, L, N) || {A, A2}<- L]). > > forAllBs(A, A2, L, N) -> > [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N]. > > forAllCs(A, B, AB, A2B2, L, N) -> > [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Hynek's new version > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > pythag4(N) when is_integer(N) -> pythag4(N,1). > > pythag4(N, A) when A+2> N -> []; > pythag4(N, A) -> pythag4(N, A, A*A, 1). > > pythag4(N, A, _A2, B) when A+B+1> N -> pythag4(N, A+1); > pythag4(N, A, A2, B) -> pythag4(N, A, A2, B, B*B, 1). > > pythag4(N, A, A2, B, _B2, C) when A+B+C> N -> pythag4(N, A, A2, B+1); > pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> > [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; > pythag4(N, A, A2, B, B2, C) -> pythag4(N, A, A2, B, B2, C+1). > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Willem's implementation in parallel by Hynek > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > wpythag2P(N) -> > L = [{A, A*A} || A<- lists:seq(1,N)], % For all A's > lists:flatten(lpmap(fun({A, A2}) -> % For all B's in parallel > [forAllCsWH(A, B, A + B, A2 + B2, L, N) > || {B, B2}<- L, A + B< N] > end, L, 2000, ordered)). > > forAllCsWH(A, B, AB, A2B2, L, N) -> > [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. > > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Morten's implementation. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > pythag5(N) when is_integer(N) -> > Primes = sieve(N div 2), > M1M2s = incorporate_primes([{1,1}], N, Primes), > lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2}<- M1M2s, > M1> M2, A<- [M1-M2], B<- [2*round(math:sqrt(M1*M2))], C<- [M1+M2], > A+B+C =< N])). > > sieve(N) when is_integer(N) -> > erase(), > sieve(N,2). > > sieve(N, K) when K>= N -> > [X || X<- lists:seq(2, N), erase(X) == undefined]; > sieve(N, K) -> > cross_off(K, K, N div K - 1), > sieve(N, find_next_in_sieve(K + 1)). > > cross_off(_K, _Current, 0) -> > ok; > cross_off(K, Current, Left) -> > Next = Current + K, > put(Next, out), > cross_off(K, Next, Left - 1). > > find_next_in_sieve(K) -> > case get(K) of > undefined -> > K; > _ -> > find_next_in_sieve(K+1) > end. > > incorporate_prime(M1M2s, N, P) -> > lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2}<- > M1M2s]). > > incorporate_prime_single({M1,M2}, N, P) -> > Evens = [{X, Y} || X<- incorporate_prime_even(M1, N, P), Y<- > incorporate_prime_even(M2, N, P)], > Odds = [{X, Y} || X<- incorporate_prime_odd(M1, N, P), Y<- > incorporate_prime_odd(M2, N, P)], > Evens ++ Odds. > > incorporate_prime_even(M, N, P) -> > incorporate_prime(M, N, P, []). > > incorporate_prime_odd(M, N, P) -> > incorporate_prime(M * P, N, P, []). > > incorporate_prime(M, N, _P, Acc) when M> N/2 -> > Acc; > incorporate_prime(M, N, P, Acc) -> > incorporate_prime(M * P * P, N, P, [M|Acc]). > > incorporate_primes(M1M2s, _N, []) -> > M1M2s; > incorporate_primes(M1M2s, N, [P|Rest]) -> > M1M2s_new = incorporate_prime(M1M2s, N, P), > incorporate_primes(M1M2s_new, N, Rest). > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Richard's improvement. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > py3R(N)-> > [{A,B,C} || > C<- lists:seq(1, N), > A<- lists:seq(1, N-C-1), > B<- lists:seq(1, N-C-A), > A*A + B*B =:= C*C, > A+B+C =< N]. > > > %%%%%%%%%%%%%%%%%%%%% END > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > > Cheers, > > Ivan. > > ---------------------------------------------------------------------- > Do you want to get EVO (ExtendedVisualOtp) to develops client-server > applicacions using C# / Erlang? Just say it!!!. > > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electr?nico > que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para > respaldar el cumplimiento de los objetivos de la organizaci?n y su > pol?tica informativa. La persona que env?a este correo asume el > compromiso de usar el servicio a tales fines y cumplir con las > regulaciones establecidas. -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From co7eb@REDACTED Mon Nov 15 17:29:45 2010 From: co7eb@REDACTED (Gilberto Carmenate =?iso-8859-1?Q?Garc=EDa?=) Date: Mon, 15 Nov 2010 11:29:45 -0500 Subject: **NEW** Now we are talking about!!! RESUME Pytagoriam Numbers Message-ID: Now we are talking about!!! Tested on Intel(R) Celeron CPU 2.13 GHz, 192 of Ram *** Pytagoriam's Numbers REPORT! *** Using N = 300 *** THE ORIGINAL IMPLEMENTATIONS (ME!)**** -py1: 14860000 mcs = 14.86 s -py2: 14843999 mcs = 14.843999 s -py3: 12749999 mcs = 12.749999 s *** Tony's implementations **** -pythag1: 11702999 mcs = 11.702999 s -pythag2: 1983999 mcs = 1.983999 s *** Hynek's implementation **** Simpler and about 5% faster version -pythag3: 1952999 mcs = 1.952999 s *** Edmond's implementation, using parallelism**** -py2E: 14843999 mcs = 14.843999 s *** Willem's implementation**** -wpythag2: 1749999 mcs = 1.749999 s *** Hynek's new implementation**** -pythag4: 1140999 mcs = 1.140999 s *** Willem's new implementation in parallel by Hynek**** -wpythag2P: 1795999 mcs = 1.795999 s *** Morten's implementation**** -pythag5: 46999 mcs = 0.046999 s *** Richard's improvement**** -py3R: 46999 mcs = 0.046999 s *** NEW *** *** Joe's improvement**** -py3a: 1577999 mcs = 1.577999 s *** NEW *** Comparisons in results agains 'pythag1' Tony's function py1 returns the same than 'pythag1' py2 NO returns the same than 'pythag1' py3 returns the same than 'pythag1' pythag1 returns the same than 'pythag1' pythag2 returns the same than 'pythag1' pythag3 returns the same than 'pythag1' py2E NO returns the same than 'pythag1' wpythag2 returns the same than 'pythag1' pythag4 returns the same than 'pythag1' wpythag2P returns the same than 'pythag1' pythag5 returns the same than 'pythag1' py3R returns the same than 'pythag1' <- now are we talking about!!! (is the best!!!) py3a returns the same than 'pythag1' NOTE: Time took using timer:tc/3 function. %******************* START CODE ********************* -module(py). -compile([export_all]). start(N)-> %% My implementations. {T1,R1} = timer:tc(py,py1, [N]), {T2,R2} = timer:tc(py,py2,[N]), {T3,R3} = timer:tc(py,py3,[N]), %% Tony's improvement of the original form 3. {T4,R4} = timer:tc(py,pythag1,[N]), %% Tony's implementation. {T5,R5} = timer:tc(py,pythag2,[N]), %% Hynek's implementation. %% Simpler and about 5% faster version: {T6,R6} = timer:tc(py,pythag3,[N]), %% Edmond's implementation using parallelism. {T7,R7} = timer:tc(py,py2E,[N]), %% Willem's implementation. {T8,R8} = timer:tc(py,wpythag2,[N]), %% Hynek's new version {T9,R9} = timer:tc(py,pythag4,[N]), %% Willem's implementation in parallel by Hynek {T10,R10} = timer:tc(py,wpythag2P,[N]), %% Morten's implementation. {T11,R11} = timer:tc(py,pythag5,[N]), %% Richard's improvement. {T12,R12} = timer:tc(py,py3R,[N]), %% Joe's improvement. {T13,R13} = timer:tc(py,py3a,[N]), io:format("~n *** Pytagoriam's Numbers REPORT! ***~n~n"), io:format("Using N = ~p~n", [N]), io:format("~n*** THE ORIGINAL IMPLEMENTATIONS (ME!)****~n"), io:format("-py1: ~p mcs = ~p s~n", [T1,T1/1000000]), io:format("-py2: ~p mcs = ~p s~n", [T2,T2/1000000]), io:format("-py3: ~p mcs = ~p s~n", [T3,T3/1000000]), io:format("~n*** Tony's implementations ****~n"), io:format("-pythag1: ~p mcs = ~p s~n", [T4,T4/1000000]), io:format("-pythag2: ~p mcs = ~p s~n", [T5,T5/1000000]), io:format("~n*** Hynek's implementation ****~n"), io:format("Simpler and about 5% faster version~n"), io:format("-pythag3: ~p mcs = ~p s~n", [T6,T6/1000000]), io:format("~n*** Edmond's implementation, using parallelism****~n"), io:format("-py2E: ~p mcs = ~p s~n", [T7,T7/1000000]), io:format("~n*** Willem's implementation****~n"), io:format("-wpythag2: ~p mcs = ~p s~n", [T8,T8/1000000]), io:format("~n*** Hynek's new implementation****~n"), io:format("-pythag4: ~p mcs = ~p s~n", [T9,T9/1000000]), io:format("~n*** Willem's new implementation in parallel by Hynek****~n"), io:format("-wpythag2P: ~p mcs = ~p s~n", [T10,T10/1000000]), io:format("~n*** Morten's implementation****~n"), io:format("-pythag5: ~p mcs = ~p s~n", [T11,T11/1000000]), io:format("~n*** Richard's improvement****~n"), io:format("-py3R: ~p mcs = ~p s~n", [T12,T12/1000000]), io:format("~n*** Joe's improvement****~n"), io:format("-py3a: ~p mcs = ~p s~n", [T13,T13/1000000]), io:format("~nComparisons in results agains 'pythag1' Tony's function~n"), Rs = [{py1,R1}, {py2,R2}, {py3,R3}, {pythag1,R4}, {pythag2,R5}, {pythag3,R6}, {py2E,R7}, {wpythag2,R8}, {pythag4,R9}, {wpythag2P,R10}, {pythag5,R11}, {py3R,R12}, {py3a,R13}], lists:foreach(fun({Name, R})-> if (R=:=R4)-> io:format("~p returns the same than 'pythag1'~n", [Name]); true-> io:format("~p NO returns the same than 'pythag1'~n", [Name]) end end, Rs), io:format("~nNOTE: Time took using timer:tc/3 function.~n"). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% The original form 1. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py1(Max)-> L = lists:seq(1, Max), lists:foldr( fun(A, Acc3)-> lists:foldr( fun(B, Acc2)-> lists:foldr( fun(C, Acc)-> case ((A*A + B*B =:= C*C) andalso (A+B+C =< Max)) of true-> [{A,B,C}|Acc]; false-> Acc end end , Acc2, L) end , Acc3, L) end , [], L). %% The original form 2. py2(Max)-> fora(1, [], Max). fora(A, Acc, Max)-> Acc1 = forb(A,1, Acc, Max), case A< Max of true-> fora(A+1, Acc1, Max); false-> Acc1 end. forb(A,B, Acc, Max)-> Acc1 = forc(A,B,1, Acc, Max), case B< Max of true-> forb(A,B+1, Acc1, Max); false-> Acc1 end. forc(A,B,C, Acc, Max)-> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true-> [{A,B,C}|Acc]; _-> Acc end, case C< Max of true-> forc(A,B,C+1, Acc1, Max); false-> Acc1 end. %% The original form 3. py3(Max)-> [{A,B,C} || A<-lists:seq(1, Max), B<-lists:seq(1, Max), C<-lists:seq(1, Max), A*A + B*B =:= C*C, A+B+C =< Max]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Tony's improvement of the original form 3. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag1(N) -> L = lists:seq(1,N), [ {A,B,C} || A<- L, B<- L, C<- L, A+B+C =< N, A*A+B*B =:= C*C]. %% Tony's implementation. pythag2(N) -> lists:reverse(pythan2_A(1, N, [])). pythan2_A(A, N, Acc) when A> N -> Acc; pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). pythan2_B(A, B, N, Acc) when A+B> N -> Acc; pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). pythan2_C(A, B, C, N, Acc) when A+B+C> N -> Acc; pythan2_C(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_C(A, B, C+1, N, Acc) end. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Hynek's implementation. %% Simpler and about 5% faster version: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag3(N) when is_integer(N) -> pythag3(N,1). pythag3(N, A) when A+2> N -> []; pythag3(N, A) -> pythag3(N, A, 1). pythag3(N, A, B) when A+B+1> N -> pythag3(N, A+1); pythag3(N, A, B) -> pythag3(N, A, B, 1). pythag3(N, A, B, C) when A+B+C> N -> pythag3(N, A, B+1); pythag3(N, A, B, C) when A*A + B*B =:= C*C -> [{A, B, C}|pythag3(N, A, B, C+1)]; pythag3(N, A, B, C) -> pythag3(N, A, B, C+1). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Edmond's implementation using parallelism. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%---- START CODE ---- py2E(Max)-> lists:flatten(lpmap(fun(A) -> forbE(A, 1, [], Max) end, lists:seq(1, Max), ordered)). forbE(A, B, Acc, Max) -> Acc1 = forcE(A, B, 1, Acc, Max), case B< Max of true -> forbE(A, B+1, Acc1, Max); false -> Acc1 end. forcE(A, B, C, Acc, Max) -> Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of true -> [{A,B,C}|Acc]; _ -> Acc end, case C< Max of true-> forcE(A, B, C+1, Acc1, Max); false-> Acc1 end. pythag2E(N)-> lists:flatten(lpmap(fun(A) -> pythan2_BE(A, 1, N, []) end, lists:seq(1, N), ordered)). pythan2_AE(A, N, Acc) when A> N -> Acc; pythan2_AE(A, N, Acc) -> pythan2_AE(A+1,N,pythan2_BE(A, 1, N, Acc)). pythan2_BE(A, B, N, Acc) when A+B> N -> Acc; pythan2_BE(A, B, N, Acc) -> pythan2_BE(A,B+1,N,pythan2_CE(A, B, 1, N, Acc)). pythan2_CE(A, B, C, N, Acc) when A+B+C> N -> Acc; pythan2_CE(A, B, C, N, Acc) -> if A*A+B*B =:= C*C -> pythan2_CE(A, B, C+1, N, [{A,B,C}|Acc]); true -> pythan2_CE(A, B, C+1, N, Acc) end. %% @spec lpmap(fun(), list(), (atom() = ordered|unordered)) -> list() %% @doc Spawns a process for each element in list L, performs specified %% function F against each in parallel and then returns results either %% same order as L (ordered) or in any order (unordered). %% NB: See also lpmap/4. lpmap(F, L, ordered) -> Ref = erlang:make_ref(), Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- L], lpmap_gather_ordered(Pids, Ref, [], 0, void); lpmap(F, L, unordered) -> Ref = erlang:make_ref(), lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, L), lpmap_gather_unordered(length(L), Ref, [], 0, void). %% @spec lpmap(fun(), integer(), list(), (atom() = ordered|unordered)) -> list() %% @doc Same as lpmap/3 except ensures only a maximum of MaxPs parallel %% processes execute function F at any one time (i.e. first takes MaxPs %% items from list, executes F in parallel against each, then as each %% process returns, spawns another process on next item in L as long as %% active processes are less than MaxPs). %% NB: See also lpmap/4. lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L)> MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- HPids], lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> Ref = erlang:make_ref(), {HPids, TPids} = if length(L)> MaxPs -> lists:split(MaxPs, L); true -> {L, []} end, lists:foreach(fun(I) -> lpmap_spawn_link(self(), Ref, F, I) end, HPids), lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). %% lpmap internal functions lpmap_spawn_link(Parent, Ref, F, I) -> spawn_link(fun() -> Parent ! {self(), Ref, F(I)} end). lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> []; lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> receive {HPid, Ref, Ret} when length(TPids) [H | T] = L, [Ret | lpmap_gather_ordered( lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, H)]), Ref, T, MaxPs, F)]; {HPid, Ref, Ret} -> [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] end. lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> []; lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> receive {_Pid, Ref, Ret} when NPs-1 [H | T] = L, lpmap_spawn_link(self(), Ref, F, H), [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; {_Pid, Ref, Ret} -> [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] end. %%---- END CODE ----- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Willem's implementation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% wpythag2(N) -> L = [{A, A*A} || A<- lists:seq(1,N)], lists:flatten([forAllBs(A, A2, L, N) || {A, A2}<- L]). forAllBs(A, A2, L, N) -> [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N]. forAllCs(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Hynek's new version %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag4(N) when is_integer(N) -> pythag4(N,1). pythag4(N, A) when A+2> N -> []; pythag4(N, A) -> pythag4(N, A, A*A, 1). pythag4(N, A, _A2, B) when A+B+1> N -> pythag4(N, A+1); pythag4(N, A, A2, B) -> pythag4(N, A, A2, B, B*B, 1). pythag4(N, A, A2, B, _B2, C) when A+B+C> N -> pythag4(N, A, A2, B+1); pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; pythag4(N, A, A2, B, B2, C) -> pythag4(N, A, A2, B, B2, C+1). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Willem's implementation in parallel by Hynek %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% wpythag2P(N) -> L = [{A, A*A} || A<- lists:seq(1,N)], % For all A's lists:flatten(lpmap(fun({A, A2}) -> % For all B's in parallel [forAllCsWH(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N] end, L, 2000, ordered)). forAllCsWH(A, B, AB, A2B2, L, N) -> [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Morten's implementation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pythag5(N) when is_integer(N) -> Primes = sieve(N div 2), M1M2s = incorporate_primes([{1,1}], N, Primes), lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2}<- M1M2s, M1> M2, A<- [M1-M2], B<- [2*round(math:sqrt(M1*M2))], C<- [M1+M2], A+B+C =< N])). sieve(N) when is_integer(N) -> erase(), sieve(N,2). sieve(N, K) when K>= N -> [X || X<- lists:seq(2, N), erase(X) == undefined]; sieve(N, K) -> cross_off(K, K, N div K - 1), sieve(N, find_next_in_sieve(K + 1)). cross_off(_K, _Current, 0) -> ok; cross_off(K, Current, Left) -> Next = Current + K, put(Next, out), cross_off(K, Next, Left - 1). find_next_in_sieve(K) -> case get(K) of undefined -> K; _ -> find_next_in_sieve(K+1) end. incorporate_prime(M1M2s, N, P) -> lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2}<- M1M2s]). incorporate_prime_single({M1,M2}, N, P) -> Evens = [{X, Y} || X<- incorporate_prime_even(M1, N, P), Y<- incorporate_prime_even(M2, N, P)], Odds = [{X, Y} || X<- incorporate_prime_odd(M1, N, P), Y<- incorporate_prime_odd(M2, N, P)], Evens ++ Odds. incorporate_prime_even(M, N, P) -> incorporate_prime(M, N, P, []). incorporate_prime_odd(M, N, P) -> incorporate_prime(M * P, N, P, []). incorporate_prime(M, N, _P, Acc) when M> N/2 -> Acc; incorporate_prime(M, N, P, Acc) -> incorporate_prime(M * P * P, N, P, [M|Acc]). incorporate_primes(M1M2s, _N, []) -> M1M2s; incorporate_primes(M1M2s, N, [P|Rest]) -> M1M2s_new = incorporate_prime(M1M2s, N, P), incorporate_primes(M1M2s_new, N, Rest). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Richard's improvement. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py3R(N)-> [{A,B,C} || A <- lists:seq(1, N div 2), B <- lists:seq(1, N - A), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Joe's improvement. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% py3a(Max) -> N = Max div 2, [{A,B,C} || A <- lists:seq(1,N+1), B <- lists:seq(1,Max-A), C <- lists:seq(1,Max-A-B), A*A + B*B =:= C*C]. %************************* END CODE************************* Cheers, Ivan. ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electronico que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizacion y su politica informativa. La persona que envia este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From ebegumisa@REDACTED Mon Nov 15 17:35:32 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 03:35:32 +1100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> Message-ID: Hello Ulf, I'm in Melbourne! For those of us unfortunately unable to get time off to attend, is there a possibility for an ErlLounge? I personally would love to hear some war stories from your AXD days! - Edmond - On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger wrote: > > Just a short note for those of you Down Under. > > I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 > Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec 8-9). > > http://www.yowconference.com.au/index.html > > This should be a good opportunity for those interested in Erlang to get > together, ask questions and perhaps talk over beer afterwards. I'd love > to hear > from some locals about how we can make the most of the opportunity. > > The conference itself will feature talks by me, Justin Sheehy and > Kresten Krab Thorup - as well as a load of distinguished speakers from > different camps. > > http://www.yowconference.com.au/melbourne/speakers/index.html > > It should be a hoot. Please sign up and recruit some friends too. > > I hope to see you there. > > BR, > Ulf W > > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ulf.wiger@REDACTED Mon Nov 15 17:58:44 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 15 Nov 2010 17:58:44 +0100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> Message-ID: <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> Sure. My preference for Erlounges would be Melbourne: Dec 2 or 3 Brisbane: Dec 9 BR, Ulf On 15 Nov 2010, at 17:35, Edmond Begumisa wrote: > Hello Ulf, > > I'm in Melbourne! > > For those of us unfortunately unable to get time off to attend, is there a possibility for an ErlLounge? I personally would love to hear some war stories from your AXD days! > > - Edmond - > > > On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger wrote: > >> >> Just a short note for those of you Down Under. >> >> I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 >> Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec 8-9). >> >> http://www.yowconference.com.au/index.html >> >> This should be a good opportunity for those interested in Erlang to get >> together, ask questions and perhaps talk over beer afterwards. I'd love to hear >> from some locals about how we can make the most of the opportunity. >> >> The conference itself will feature talks by me, Justin Sheehy and >> Kresten Krab Thorup - as well as a load of distinguished speakers from >> different camps. >> >> http://www.yowconference.com.au/melbourne/speakers/index.html >> >> It should be a hoot. Please sign up and recruit some friends too. >> >> I hope to see you there. >> >> BR, >> Ulf W >> >> Ulf Wiger, CTO, Erlang Solutions, Ltd. >> http://erlang-solutions.com >> >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From toby@REDACTED Mon Nov 15 18:35:59 2010 From: toby@REDACTED (Toby Thain) Date: Mon, 15 Nov 2010 12:35:59 -0500 Subject: [erlang-questions] **NEW** Now we are talking about!!! RESUME Pytagoriam Numbers In-Reply-To: References: Message-ID: <4CE16F7F.8000500@telegraphics.com.au> On 15/11/10 11:29 AM, Gilberto Carmenate Garc?a wrote: > Now we are talking about!!! > > Tested on Intel(R) Celeron CPU 2.13 GHz, 192 of Ram > > *** Pytagoriam's Numbers REPORT! *** > > Using N = 300 > Benchmarking a single problem size doesn't expose the differing time complexity of the solutions (pace ROK). --Toby From info@REDACTED Mon Nov 15 19:24:47 2010 From: info@REDACTED (info) Date: Mon, 15 Nov 2010 19:24:47 +0100 Subject: [erlang-questions] odbc error References: <201011141844548282526@its3.ch>, <20101114213334.GE32129@hijacked.us> Message-ID: <201011151924467372867@its3.ch> Thank you. My idea was to send every hours the following message: odbc:sql_query(Conn,"mysql_ping()") J-Ph. Constantin ITS3 Gen?ve www.its3.ch From ulf.wiger@REDACTED Mon Nov 15 20:24:21 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 15 Nov 2010 20:24:21 +0100 Subject: [erlang-questions] gproc global will create a bootleneck in my app? In-Reply-To: <868787.92083.qm@web112610.mail.gq1.yahoo.com> References: <868787.92083.qm@web112610.mail.gq1.yahoo.com> Message-ID: <6C0B9FEA-EB18-4F14-A4C4-A56F2E62F49D@erlang-solutions.com> On 15 Nov 2010, at 15:11, Pablo Platt wrote: > Hi, > > I'm using gproc to handle IM sessions on one node and it works great. > I want to use it with gen_leader to extend to several nodes. > Saving a global name will be slower because that leader has to coordinate > uniqueness. > Will it affect reads as well? No, they are performed the same way for local and global lookups. > If one gen_server is responsible for everything, does writing a global name > might block all reads? No, the lookup is done in the client process. This constitutes a race condition, but process death is asynchronous anyway so the race condition is inherent in the problem and cannot be removed. > > What is a reasonable number of unique names and non-unique properties I can save > with gproc? > Am I only limited by RAM? Reasonable... when a process dies, the server will iterate through all its registered names and properties and remove them. A huge number of registered items per process will likely cause problems - perhaps even severe problems. But as long as each process keeps within a reasonable limit (say, no more than a hundred or so items), it should scale quite well, and you will be limited by available RAM or (in 32-bit) addressable memory. If processes come and go at a very high rate and register lots of names and properties, the gproc process is likely to become a bottleneck in your system. I don't know when this would become a problem. It's been used in systems with tens of thousand processes registering a fair number of properties each at a rate of several hundred processes/sec. BR, Ulf W Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From ebegumisa@REDACTED Mon Nov 15 21:03:04 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 07:03:04 +1100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> Message-ID: Any one else in Melbourne up for an Erlounge Dec 2nd or 3rd? - Edmond - On Tue, 16 Nov 2010 03:58:44 +1100, Ulf Wiger wrote: > > Sure. My preference for Erlounges would be > > Melbourne: Dec 2 or 3 > Brisbane: Dec 9 > > BR, > Ulf > > On 15 Nov 2010, at 17:35, Edmond Begumisa wrote: > >> Hello Ulf, >> >> I'm in Melbourne! >> >> For those of us unfortunately unable to get time off to attend, is >> there a possibility for an ErlLounge? I personally would love to hear >> some war stories from your AXD days! >> >> - Edmond - >> >> >> On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger >> wrote: >> >>> >>> Just a short note for those of you Down Under. >>> >>> I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 >>> Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec >>> 8-9). >>> >>> http://www.yowconference.com.au/index.html >>> >>> This should be a good opportunity for those interested in Erlang to get >>> together, ask questions and perhaps talk over beer afterwards. I'd >>> love to hear >>> from some locals about how we can make the most of the opportunity. >>> >>> The conference itself will feature talks by me, Justin Sheehy and >>> Kresten Krab Thorup - as well as a load of distinguished speakers from >>> different camps. >>> >>> http://www.yowconference.com.au/melbourne/speakers/index.html >>> >>> It should be a hoot. Please sign up and recruit some friends too. >>> >>> I hope to see you there. >>> >>> BR, >>> Ulf W >>> >>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>> http://erlang-solutions.com >>> >>> >>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From pablo.platt@REDACTED Mon Nov 15 21:15:08 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Mon, 15 Nov 2010 12:15:08 -0800 (PST) Subject: [erlang-questions] gproc global will create a bootleneck in my app? In-Reply-To: <6C0B9FEA-EB18-4F14-A4C4-A56F2E62F49D@erlang-solutions.com> References: <868787.92083.qm@web112610.mail.gq1.yahoo.com> <6C0B9FEA-EB18-4F14-A4C4-A56F2E62F49D@erlang-solutions.com> Message-ID: <65618.57398.qm@web112602.mail.gq1.yahoo.com> Sounds perfect. Thank you for the explanation. ________________________________ From: Ulf Wiger To: Pablo Platt Cc: erlang-questions@REDACTED Sent: Mon, November 15, 2010 9:24:21 PM Subject: Re: [erlang-questions] gproc global will create a bootleneck in my app? On 15 Nov 2010, at 15:11, Pablo Platt wrote: > Hi, > > I'm using gproc to handle IM sessions on one node and it works great. > I want to use it with gen_leader to extend to several nodes. > Saving a global name will be slower because that leader has to coordinate > uniqueness. > Will it affect reads as well? No, they are performed the same way for local and global lookups. > If one gen_server is responsible for everything, does writing a global name > might block all reads? No, the lookup is done in the client process. This constitutes a race condition, but process death is asynchronous anyway so the race condition is inherent in the problem and cannot be removed. > > What is a reasonable number of unique names and non-unique properties I can >save > > with gproc? > Am I only limited by RAM? Reasonable... when a process dies, the server will iterate through all its registered names and properties and remove them. A huge number of registered items per process will likely cause problems - perhaps even severe problems. But as long as each process keeps within a reasonable limit (say, no more than a hundred or so items), it should scale quite well, and you will be limited by available RAM or (in 32-bit) addressable memory. If processes come and go at a very high rate and register lots of names and properties, the gproc process is likely to become a bottleneck in your system. I don't know when this would become a problem. It's been used in systems with tens of thousand processes registering a fair number of properties each at a rate of several hundred processes/sec. BR, Ulf W Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From ebegumisa@REDACTED Mon Nov 15 21:53:30 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 07:53:30 +1100 Subject: [erlang-questions] LAST RESUME of Pytagoriam Numbers!!! In-Reply-To: <43C9CB7E.7080606@cmg.jovenclub.cu> References: <43C9CB7E.7080606@cmg.jovenclub.cu> Message-ID: Below is a parallelised version of Richard/H?kan's blazing and yet easy-to-read algo using a max of 2000 processes (find lpmap function on previous thread.) On my dual-core it's about 20% faster for when N is small (300) and gets up to 40% faster when N is large (3,000). Evens out from there. I'm getting about 4.5s for N = 5,000! I must say, I'm floored. I need to learn much, much, more about Erlang. I just wasn't convinced it would be any good at this! And their algo retains Garcia's original beauty too! It reads much like the high-school math problems. py3R_p(N)-> lists:flatten(lpmap(fun(A) -> [{A, B, C} || B <- lists:seq(1, N - A), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C] end, lists:seq(1, N div 2), 2000, ordered)). - Edmond - H?kan Huss" On Sun, 15 Jan 2006 15:11:42 +1100, Ivan Carmenates Garc?a wrote: > > Compiled using c(py) normal in the shell. > > Feel free to make a new Resume on a DualCore CPU!!! > > > Now we are re-talking about!!! > > Tested on Intel(R) Celeron CPU 2.13 GHz, 192 of Ram > > *** Pytagoriam's Numbers REPORT! *** > > Using N = 300 > > *** THE ORIGINAL IMPLEMENTATIONS (ME!)**** > -py1: 14860000 mcs = 14.86 s > -py2: 14843999 mcs = 14.843999 s > -py3: 12749999 mcs = 12.749999 s > > *** Tony's implementations **** > -pythag1: 11702999 mcs = 11.702999 s > -pythag2: 1983999 mcs = 1.983999 s > > *** Hynek's implementation **** > Simpler and about 5% faster version > -pythag3: 1952999 mcs = 1.952999 s > > *** Edmond's implementation, using parallelism**** > -py2E: 14843999 mcs = 14.843999 s > > *** Willem's implementation**** > -wpythag2: 1749999 mcs = 1.749999 s > > *** Hynek's new implementation**** > -pythag4: 1140999 mcs = 1.140999 s > > *** Willem's new implementation in parallel by Edmond!!!**** > -wpythag2P: 1795999 mcs = 1.795999 s > > *** Morten's implementation**** > -pythag5: 46999 mcs = 0.046999 s > > *** Richard's improvement**** > -py3R: 46999 mcs = 0.046999 s *** NEW *** > > *** Joe's improvement**** > -py3a: 1577999 mcs = 1.577999 s *** NEW *** > > Comparisons in results agains 'pythag1' Tony's > function > py1 returns the same than 'pythag1' > py2 returns the same than 'pythag1' > py3 returns the same than 'pythag1' > pythag1 returns the same than 'pythag1' > pythag2 returns the same than 'pythag1' > pythag3 returns the same than 'pythag1' > py2E returns the same than 'pythag1' > wpythag2 returns the same than 'pythag1' > pythag4 returns the same than 'pythag1' > wpythag2P returns the same than 'pythag1' > pythag5 returns the same than 'pythag1' > py3R returns the same than 'pythag1' <- now are we talking about!!! (is > the best!!!) > py3a returns the same than 'pythag1' > > NOTE: Time took using timer:tc/3 function. > > > %******************* START CODE ********************* > > -module(py). > -compile([export_all]). > > > start(N)-> > %% My implementations. > {T1,R1} = timer:tc(py,py1, [N]), > {T2,R2} = timer:tc(py,py2,[N]), > {T3,R3} = timer:tc(py,py3,[N]), > > %% Tony's improvement of the original form 3. > {T4,R4} = timer:tc(py,pythag1,[N]), > %% Tony's implementation. > {T5,R5} = timer:tc(py,pythag2,[N]), > > %% Hynek's implementation. > %% Simpler and about 5% faster version: > {T6,R6} = timer:tc(py,pythag3,[N]), > > %% Edmond's implementation using parallelism. > {T7,R7} = timer:tc(py,py2E,[N]), > > %% Willem's implementation. > {T8,R8} = timer:tc(py,wpythag2,[N]), > > %% Hynek's new version > {T9,R9} = timer:tc(py,pythag4,[N]), > > %% Willem's implementation in parallel by Edmond. > {T10,R10} = timer:tc(py,wpythag2P,[N]), > > %% Morten's implementation. > {T11,R11} = timer:tc(py,pythag5,[N]), > > %% Richard's improvement. > {T12,R12} = timer:tc(py,py3R,[N]), > > %% Joe's improvement. > {T13,R13} = timer:tc(py,py3a,[N]), > > io:format("~n *** Pytagoriam's Numbers REPORT! ***~n~n"), > io:format("Using N = ~p~n", [N]), > > io:format("~n*** THE ORIGINAL IMPLEMENTATIONS (ME!)****~n"), > io:format("-py1: ~p mcs = ~p s~n", [T1,T1/1000000]), > io:format("-py2: ~p mcs = ~p s~n", [T2,T2/1000000]), > io:format("-py3: ~p mcs = ~p s~n", [T3,T3/1000000]), > > io:format("~n*** Tony's implementations ****~n"), > io:format("-pythag1: ~p mcs = ~p s~n", [T4,T4/1000000]), > io:format("-pythag2: ~p mcs = ~p s~n", [T5,T5/1000000]), > > io:format("~n*** Hynek's implementation ****~n"), > io:format("Simpler and about 5% faster version~n"), > io:format("-pythag3: ~p mcs = ~p s~n", [T6,T6/1000000]), > > io:format("~n*** Edmond's implementation, using parallelism****~n"), > io:format("-py2E: ~p mcs = ~p s~n", [T7,T7/1000000]), > > io:format("~n*** Willem's implementation****~n"), > io:format("-wpythag2: ~p mcs = ~p s~n", [T8,T8/1000000]), > > io:format("~n*** Hynek's new implementation****~n"), > io:format("-pythag4: ~p mcs = ~p s~n", [T9,T9/1000000]), > > io:format("~n*** Willem's new implementation in parallel by > Edmond!!!****~n"), > io:format("-wpythag2P: ~p mcs = ~p s~n", [T10,T10/1000000]), > > io:format("~n*** Morten's implementation****~n"), > io:format("-pythag5: ~p mcs = ~p s~n", [T11,T11/1000000]), > > io:format("~n*** Richard's improvement****~n"), > io:format("-py3R: ~p mcs = ~p s~n", [T12,T12/1000000]), > > io:format("~n*** Joe's improvement****~n"), > io:format("-py3a: ~p mcs = ~p s~n", [T13,T13/1000000]), > > io:format("~nComparisons in results agains 'pythag1' Tony's > function~n"), > Rs = [{py1,R1}, {py2,R2}, {py3,R3}, {pythag1,R4}, {pythag2,R5}, > {pythag3,R6}, > {py2E,R7}, {wpythag2,R8}, {pythag4,R9}, {wpythag2P,R10}, > {pythag5,R11}, {py3R,R12}, > {py3a,R13}], > lists:foreach(fun({Name, R})-> > if > (R=:=R4)-> > io:format("~p returns the same than 'pythag1'~n", > [Name]); > true-> > io:format("~p NO returns the same than 'pythag1'~n", > [Name]) > end > end, Rs), > io:format("~nNOTE: Time took using timer:tc/3 function.~n"). > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% The original form 1. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > py1(Max)-> > L = lists:seq(1, Max), > lists:foldr( > fun(A, Acc3)-> > lists:foldr( > fun(B, Acc2)-> > lists:foldr( > fun(C, Acc)-> > case ((A*A + B*B =:= C*C) andalso (A+B+C =< > Max)) of > true-> > [{A,B,C}|Acc]; > false-> > Acc > end > end > , Acc2, L) > end > , Acc3, L) > end > , [], L). > > > > %% The original form 2. > py2(Max)-> > lists:reverse(fora(1, [], Max)). > > fora(A, Acc, Max)-> > Acc1 = forb(A,1, Acc, Max), > case A< Max of > true-> > fora(A+1, Acc1, Max); > false-> > Acc1 > end. > > forb(A,B, Acc, Max)-> > Acc1 = forc(A,B,1, Acc, Max), > case B< Max of > true-> > forb(A,B+1, Acc1, Max); > false-> > Acc1 > end. > > forc(A,B,C, Acc, Max)-> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true-> > [{A,B,C}|Acc]; > _-> > Acc > end, > case C< Max of > true-> > forc(A,B,C+1, Acc1, Max); > false-> > Acc1 > end. > > %% The original form 3. > py3(Max)-> > [{A,B,C} || > A<-lists:seq(1, Max), > B<-lists:seq(1, Max), > C<-lists:seq(1, Max), > A*A + B*B =:= C*C, > A+B+C =< Max]. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Tony's improvement of the original form 3. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > pythag1(N) -> > L = lists:seq(1,N), > [ {A,B,C} || > A<- L, > B<- L, > C<- L, > A+B+C =< N, > A*A+B*B =:= C*C]. > > %% Tony's implementation. > pythag2(N) -> > lists:reverse(pythan2_A(1, N, [])). > > pythan2_A(A, N, Acc) when A> N -> Acc; > pythan2_A(A, N, Acc) -> pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). > > pythan2_B(A, B, N, Acc) when A+B> N -> Acc; > pythan2_B(A, B, N, Acc) -> pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, > Acc)). > > pythan2_C(A, B, C, N, Acc) when A+B+C> N -> Acc; > pythan2_C(A, B, C, N, Acc) -> > if A*A+B*B =:= C*C -> > pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); > true -> > pythan2_C(A, B, C+1, N, Acc) > end. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Hynek's implementation. > %% Simpler and about 5% faster version: > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > pythag3(N) when is_integer(N) -> pythag3(N,1). > > pythag3(N, A) when A+2> N -> []; > pythag3(N, A) -> pythag3(N, A, 1). > > pythag3(N, A, B) when A+B+1> N -> pythag3(N, A+1); > pythag3(N, A, B) -> pythag3(N, A, B, 1). > > pythag3(N, A, B, C) when A+B+C> N -> pythag3(N, A, B+1); > pythag3(N, A, B, C) when A*A + B*B =:= C*C -> [{A, B, C}|pythag3(N, A, > B, C+1)]; > pythag3(N, A, B, C) -> pythag3(N, A, B, C+1). > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Edmond's implementation using parallelism. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > %%---- START CODE ---- > > py2E(Max)-> > lists:flatten(lpmap(fun(A) -> > forbE(A, 1, [], Max) > end, lists:seq(1, Max), ordered)). > > forbE(A, B, Acc, Max) -> > Acc1 = forcE(A, B, 1, Acc, Max), > case B< Max of > true -> forbE(A, B+1, Acc1, Max); > false -> Acc1 > end. > > forcE(A, B, C, Acc, Max) -> > Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< Max) of > true -> [{A,B,C}|Acc]; > _ -> Acc > end, > case C< Max of > true-> forcE(A, B, C+1, Acc1, Max); > false-> Acc1 > end. > > > pythag2E(N)-> > lists:flatten(lpmap(fun(A) -> > pythan2_BE(A, 1, N, []) > end, lists:seq(1, N), ordered)). > > pythan2_AE(A, N, Acc) when A> N -> Acc; > pythan2_AE(A, N, Acc) -> pythan2_AE(A+1,N,pythan2_BE(A, 1, N, Acc)). > > pythan2_BE(A, B, N, Acc) when A+B> N -> Acc; > pythan2_BE(A, B, N, Acc) -> pythan2_BE(A,B+1,N,pythan2_CE(A, B, 1, N, > Acc)). > > pythan2_CE(A, B, C, N, Acc) when A+B+C> N -> Acc; > pythan2_CE(A, B, C, N, Acc) -> > if A*A+B*B =:= C*C -> > pythan2_CE(A, B, C+1, N, [{A,B,C}|Acc]); > true -> > pythan2_CE(A, B, C+1, N, Acc) > end. > > %% @spec lpmap(fun(), list(), (atom() = ordered|unordered)) -> list() > %% @doc Spawns a process for each element in list L, performs > specified > %% function F against each in parallel and then returns results > either > %% same order as L (ordered) or in any order (unordered). > %% NB: See also lpmap/4. > > lpmap(F, L, ordered) -> > Ref = erlang:make_ref(), > Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- L], > lpmap_gather_ordered(Pids, Ref, [], 0, void); > lpmap(F, L, unordered) -> > Ref = erlang:make_ref(), > lists:foreach(fun(I) -> > lpmap_spawn_link(self(), Ref, F, I) > end, L), > lpmap_gather_unordered(length(L), Ref, [], 0, void). > > %% @spec lpmap(fun(), integer(), list(), (atom() = > ordered|unordered)) -> > list() > %% @doc Same as lpmap/3 except ensures only a maximum of MaxPs > parallel > %% processes execute function F at any one time (i.e. first > takes > MaxPs > %% items from list, executes F in parallel against each, then as > each > %% process returns, spawns another process on next item in L as > long as > %% active processes are less than MaxPs). > %% NB: See also lpmap/4. > > lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> > Ref = erlang:make_ref(), > {HPids, TPids} = if > length(L)> MaxPs -> lists:split(MaxPs, L); > true -> {L, []} > end, > Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- HPids], > lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); > lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> > Ref = erlang:make_ref(), > {HPids, TPids} = if > length(L)> MaxPs -> lists:split(MaxPs, L); > true -> {L, []} > end, > lists:foreach(fun(I) -> > lpmap_spawn_link(self(), Ref, F, I) > end, HPids), > lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). > > %% lpmap internal functions > > lpmap_spawn_link(Parent, Ref, F, I) -> > spawn_link(fun() -> > Parent ! {self(), Ref, F(I)} > end). > > lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> > []; > lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> > receive > {HPid, Ref, Ret} when length(TPids) > [H | T] = L, > [Ret | lpmap_gather_ordered( > lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, > H)]), > Ref, T, MaxPs, F)]; > {HPid, Ref, Ret} -> > [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] > end. > > lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> > []; > lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> > receive > {_Pid, Ref, Ret} when NPs-1 > [H | T] = L, > lpmap_spawn_link(self(), Ref, F, H), > [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; > {_Pid, Ref, Ret} -> > [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] > end. > > > %%---- END CODE ----- > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Willem's implementation. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > wpythag2(N) -> > L = [{A, A*A} || A<- lists:seq(1,N)], > lists:flatten([forAllBs(A, A2, L, N) || {A, A2}<- L]). > > forAllBs(A, A2, L, N) -> > [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< N]. > > forAllCs(A, B, AB, A2B2, L, N) -> > [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Hynek's new version > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > pythag4(N) when is_integer(N) -> pythag4(N,1). > > pythag4(N, A) when A+2> N -> []; > pythag4(N, A) -> pythag4(N, A, A*A, 1). > > pythag4(N, A, _A2, B) when A+B+1> N -> pythag4(N, A+1); > pythag4(N, A, A2, B) -> pythag4(N, A, A2, B, B*B, 1). > > pythag4(N, A, A2, B, _B2, C) when A+B+C> N -> pythag4(N, A, A2, B+1); > pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> > [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; > pythag4(N, A, A2, B, B2, C) -> pythag4(N, A, A2, B, B2, C+1). > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Willem's implementation in parallel by Hynek > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > wpythag2P(N) -> > L = [{A, A*A} || A<- lists:seq(1,N)], % For all A's > lists:flatten(lpmap(fun({A, A2}) -> % For all B's in parallel > [forAllCsWH(A, B, A + B, A2 + B2, L, N) > || {B, B2}<- L, A + B< N] > end, L, 2000, ordered)). > > forAllCsWH(A, B, AB, A2B2, L, N) -> > [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< N]. > > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Morten's implementation. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > pythag5(N) when is_integer(N) -> > Primes = sieve(N div 2), > M1M2s = incorporate_primes([{1,1}], N, Primes), > lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2}<- M1M2s, > M1> M2, A<- [M1-M2], B<- [2*round(math:sqrt(M1*M2))], C<- [M1+M2], > A+B+C =< N])). > > sieve(N) when is_integer(N) -> > erase(), > sieve(N,2). > > sieve(N, K) when K>= N -> > [X || X<- lists:seq(2, N), erase(X) == undefined]; > sieve(N, K) -> > cross_off(K, K, N div K - 1), > sieve(N, find_next_in_sieve(K + 1)). > > cross_off(_K, _Current, 0) -> > ok; > cross_off(K, Current, Left) -> > Next = Current + K, > put(Next, out), > cross_off(K, Next, Left - 1). > > find_next_in_sieve(K) -> > case get(K) of > undefined -> > K; > _ -> > find_next_in_sieve(K+1) > end. > > incorporate_prime(M1M2s, N, P) -> > lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2}<- > M1M2s]). > > incorporate_prime_single({M1,M2}, N, P) -> > Evens = [{X, Y} || X<- incorporate_prime_even(M1, N, P), Y<- > incorporate_prime_even(M2, N, P)], > Odds = [{X, Y} || X<- incorporate_prime_odd(M1, N, P), Y<- > incorporate_prime_odd(M2, N, P)], > Evens ++ Odds. > > incorporate_prime_even(M, N, P) -> > incorporate_prime(M, N, P, []). > > incorporate_prime_odd(M, N, P) -> > incorporate_prime(M * P, N, P, []). > > incorporate_prime(M, N, _P, Acc) when M> N/2 -> > Acc; > incorporate_prime(M, N, P, Acc) -> > incorporate_prime(M * P * P, N, P, [M|Acc]). > > incorporate_primes(M1M2s, _N, []) -> > M1M2s; > incorporate_primes(M1M2s, N, [P|Rest]) -> > M1M2s_new = incorporate_prime(M1M2s, N, P), > incorporate_primes(M1M2s_new, N, Rest). > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Richard's improvement. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > py3R(N)-> > [{A,B,C} || > A <- lists:seq(1, N div 2), > B <- lists:seq(1, N - A), > C <- [trunc(math:sqrt(A * A + B * B))], > A + B + C =< N, > A*A + B*B =:= C*C]. > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %% Joe's improvement. > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > py3a(Max) -> > N = Max div 2, > [{A,B,C} || > A <- lists:seq(1,N+1), > B <- lists:seq(1,Max-A), > C <- lists:seq(1,Max-A-B), > A*A + B*B =:= C*C]. > > %************************* END CODE************************* > > Cheers, > > Ivan. > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electronico > que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para > respaldar el cumplimiento de los objetivos de la organizacion y su > politica informativa. La persona que envia este correo asume el > compromiso de usar el servicio a tales fines y cumplir con las > regulaciones establecidas. > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electronico > que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para > respaldar el cumplimiento de los objetivos de la organizacion y su > politica informativa. La persona que envia este correo asume el > compromiso de usar el servicio a tales fines y cumplir con las > regulaciones establecidas. > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Mon Nov 15 22:12:17 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Tue, 16 Nov 2010 08:12:17 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: Richard... > So my question is: if version 1 isn't performing "reasonably" acceptably > for Garcia's purpose, and version 1 isn't blatantly flawed. Isn't this a > strong indicator that he's using the wrong tool? > You can probably ignore that last question now. I've answered my own question after taking a closer look at other solutions! - Edmond - On Tue, 16 Nov 2010 02:02:50 +1100, Edmond Begumisa wrote: > Hello Richard, > > I have a question... > >> Erlang is a fine tool for jobs like this one. > > Interesting... I would have never thought this. I would have > instinctively reached for a language with mutable memory, thinking that > this would make the most sense for combinatorial* work (I don't know > this for fact, and I could be very wrong, but that's what instinct would > tell me.) But then, I'm relatively new to Erlang. > > * I've been calling this permutation but I don't know if that's accurate. > >> There is little point in optimising a bad algorithm. > > Well put. But say you have an 'ok' algorithm. Not the best, but not > blatantly flawed either. > > I think of optimisation as something you put on a version 2 to-do list. > Optimisation to me means staring hard at code and trying to figure out > ways to get it to perform better (faster, less memory, less CPU time, > etc). This normally means re-writing code that's easy to follow into > code that's no-so-easy to follow. Mind you, for Erlang, I don't look at > parallelising as optimisation as others seem to. To me, it's just a > building block made available and normally it can be applied without > changing an algo too much (I'd say it's not used enough). > > Of all the variations presented so far, IMO, Garcia's first is the > easiest to follow (ignoring that obvious flaw with the repeated list:seq > call). A non-mathematical mind like myself can see exactly what he's > trying to do. The others (including the ones from Willem and Tony that I > tried to parallelise), seem harder to follow. Maybe that's coz they are > optimised versions. The authors could have stared long and hard. > > So my question is: if version 1 isn't performing "reasonably" acceptably > for Garcia's purpose, and version 1 isn't blatantly flawed. Isn't this a > strong indicator that he's using the wrong tool? > > - Edmond - > > On Mon, 15 Nov 2010 12:29:49 +1100, Richard O'Keefe > wrote: > >> >> On 14/11/2010, at 10:17 PM, Hynek Vychodil wrote: >> Anyway Erlang is not >>> right tool for this job. You should use NIF if performance matter. >> >> Erlang is a fine tool for jobs like this one. >> In fact, given its support for large integers, >> it is about as good for combinatorial algorithms as >> Smalltalk, though perhaps not as good as Lisp. >> (But see LFE...) >> >> There is little point in optimising a bad algorithm. >> A fairly naive rewrite turns it from O(N**3) into O(N**2). >> What really counts here is how easy it is to spot the >> algorithmic problem and switch to a better algorithm. >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From jesper.louis.andersen@REDACTED Mon Nov 15 22:28:22 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 15 Nov 2010 22:28:22 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: On Mon, Nov 15, 2010 at 4:21 PM, Fred Hebert wrote: > I think the first thing to do before optimizing is understanding how you use > the data. In the case of Pythagorean triplets like these, the right question > might as well be "how many triplets do I need? What values do I need to work > with?" Let me add: Optimization is many things. One of the tenets are how easy it is to change your implementation, facing new knowledge. If you look at some of the optimized variants, it should be fairly obvious that many of them provide impressive speedups (for large N) with very few changes to the original code as soon as an observation has been made. My experience is that C, C++ or Java tend to require much more work for these iterations. It turns out there is more to fast code than just having a language isomorphic to assembly :) -- J. From mk@REDACTED Mon Nov 15 22:51:30 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 15 Nov 2010 22:51:30 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: <4CE1AB62.8030403@amberbio.com> Hi again, The py3R can be made to have a much better check on the bounds of the lists. Also, you should use that the triples are symmetric so {A,B,C} can be replaced with {B,A,C}. py5R(N)-> lists:sort(lists:flatten([[{A,B,C}, {B,A,C}] || A <- lists:seq(1, N div 2), B <- lists:seq(A + 1, max(A, N*(N-2*A) div (2*(N-A)))), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C])). the max(..,..) is really stupid, but I learnt something new about erlang, namely that lists:seq(10,8), for example, produces an exception, whereas lists:seq(10,9) = [] if lists:seq(A,B) = [] for A > B, the max above could have been removed. Morten. On 11/15/10 10:28 PM, Jesper Louis Andersen wrote: > On Mon, Nov 15, 2010 at 4:21 PM, Fred Hebert wrote: >> I think the first thing to do before optimizing is understanding how you use >> the data. In the case of Pythagorean triplets like these, the right question >> might as well be "how many triplets do I need? What values do I need to work >> with?" > Let me add: > > Optimization is many things. One of the tenets are how easy it is to > change your implementation, facing new knowledge. If you look at some > of the optimized variants, it should be fairly obvious that many of > them provide impressive speedups (for large N) with very few changes > to the original code as soon as an observation has been made. > > My experience is that C, C++ or Java tend to require much more work > for these iterations. It turns out there is more to fast code than > just having a language isomorphic to assembly :) > From mk@REDACTED Mon Nov 15 23:11:29 2010 From: mk@REDACTED (Morten Krogh) Date: Mon, 15 Nov 2010 23:11:29 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <4CE1AB62.8030403@amberbio.com> References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE1AB62.8030403@amberbio.com> Message-ID: <4CE1B011.9040208@amberbio.com> And a solution that uses the fact that A and B can not both be odd. py6R(N)-> EvenA = lists:flatten([[{A,B,C}, {B,A,C}] || A <- lists:seq(2, N div 2, 2), B <- lists:seq(A + 1, max(A, N*(N-2*A) div (2*(N-A)))), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C]), OddA = lists:flatten([[{A,B,C}, {B,A,C}] || A <- lists:seq(1, N div 2, 2), B <- lists:seq(A + 1, max(A, N*(N-2*A) div (2*(N-A))), 2), C <- [trunc(math:sqrt(A * A + B * B))], A + B + C =< N, A*A + B*B =:= C*C]), lists:sort(EvenA ++ OddA). Morten. On 11/15/10 10:51 PM, Morten Krogh wrote: > Hi again, > > The py3R can be made to have a much better check on the bounds of the > lists. > Also, you should use that the triples are symmetric so {A,B,C} can be > replaced with {B,A,C}. > > py5R(N)-> > lists:sort(lists:flatten([[{A,B,C}, {B,A,C}] || > A <- lists:seq(1, N div 2), > B <- lists:seq(A + 1, max(A, N*(N-2*A) div (2*(N-A)))), > C <- [trunc(math:sqrt(A * A + B * B))], > A + B + C =< N, > A*A + B*B =:= C*C])). > > > > the max(..,..) is really stupid, but I learnt something new about > erlang, namely that > > lists:seq(10,8), for example, produces an exception, whereas > lists:seq(10,9) = [] > > if lists:seq(A,B) = [] for A > B, the max above could have been removed. > > Morten. > > > On 11/15/10 10:28 PM, Jesper Louis Andersen wrote: >> On Mon, Nov 15, 2010 at 4:21 PM, Fred Hebert wrote: >>> I think the first thing to do before optimizing is understanding how >>> you use >>> the data. In the case of Pythagorean triplets like these, the right >>> question >>> might as well be "how many triplets do I need? What values do I need >>> to work >>> with?" >> Let me add: >> >> Optimization is many things. One of the tenets are how easy it is to >> change your implementation, facing new knowledge. If you look at some >> of the optimized variants, it should be fairly obvious that many of >> them provide impressive speedups (for large N) with very few changes >> to the original code as soon as an observation has been made. >> >> My experience is that C, C++ or Java tend to require much more work >> for these iterations. It turns out there is more to fast code than >> just having a language isomorphic to assembly :) >> > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From nbowe@REDACTED Tue Nov 16 02:14:17 2010 From: nbowe@REDACTED (Nikolas Bowe) Date: Tue, 16 Nov 2010 12:14:17 +1100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> Message-ID: <4CE1DAE9.6050500@infinite-interactive.com> Definitely! Id love to have a beer with Ulf and Justin and talk about mnesia and riak. On 11/16/2010 7:03 AM, Edmond Begumisa wrote: > Any one else in Melbourne up for an Erlounge Dec 2nd or 3rd? > > - Edmond - > > On Tue, 16 Nov 2010 03:58:44 +1100, Ulf Wiger > wrote: > >> >> Sure. My preference for Erlounges would be >> >> Melbourne: Dec 2 or 3 >> Brisbane: Dec 9 >> >> BR, >> Ulf >> >> On 15 Nov 2010, at 17:35, Edmond Begumisa wrote: >> >>> Hello Ulf, >>> >>> I'm in Melbourne! >>> >>> For those of us unfortunately unable to get time off to attend, is >>> there a possibility for an ErlLounge? I personally would love to >>> hear some war stories from your AXD days! >>> >>> - Edmond - >>> >>> >>> On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger >>> wrote: >>> >>>> >>>> Just a short note for those of you Down Under. >>>> >>>> I will be giving 1-day tutorials on Erlang and OTP during the YOW! >>>> 2010 >>>> Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec >>>> 8-9). >>>> >>>> http://www.yowconference.com.au/index.html >>>> >>>> This should be a good opportunity for those interested in Erlang to >>>> get >>>> together, ask questions and perhaps talk over beer afterwards. I'd >>>> love to hear >>>> from some locals about how we can make the most of the opportunity. >>>> >>>> The conference itself will feature talks by me, Justin Sheehy and >>>> Kresten Krab Thorup - as well as a load of distinguished speakers from >>>> different camps. >>>> >>>> http://www.yowconference.com.au/melbourne/speakers/index.html >>>> >>>> It should be a hoot. Please sign up and recruit some friends too. >>>> >>>> I hope to see you there. >>>> >>>> BR, >>>> Ulf W >>>> >>>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>>> http://erlang-solutions.com >>>> >>>> >>>> >>>> >>>> ________________________________________________________________ >>>> erlang-questions (at) erlang.org mailing list. >>>> See http://www.erlang.org/faq.html >>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>> >>> >>> >>> -- >>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> >> Ulf Wiger, CTO, Erlang Solutions, Ltd. >> http://erlang-solutions.com >> >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > This message and its attachments may contain legally privileged or confidential information. This message is intended for the use of the individual or entity to which it is addressed. If you are not the addressee indicated in this message, or the employee or agent responsible for delivering the message to the intended recipient, you may not copy or deliver this message or its attachments to anyone. Rather, you should permanently delete this message and its attachments and kindly notify the sender by reply e-mail. Any content of this message and its attachments, which does not relate to the official business of the sending company must be taken not to have been sent or endorsed by the sending company or any of its related entities. No warranty is made that the e-mail or attachment(s) are free from computer virus or other defect. From ok@REDACTED Tue Nov 16 02:20:20 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 16 Nov 2010 14:20:20 +1300 Subject: [erlang-questions] **NEW** Now we are talking about!!! RESUME Pytagoriam Numbers In-Reply-To: <4CE16F7F.8000500@telegraphics.com.au> References: <4CE16F7F.8000500@telegraphics.com.au> Message-ID: <1CAE6784-69CA-4496-BCF0-458B9045F1BB@cs.otago.ac.nz> On 16/11/2010, at 6:35 AM, Toby Thain wrote: > > Benchmarking a single problem size doesn't expose the differing time > complexity of the solutions (pace ROK). Why "pace"? I agree 100%. That's why I was talking about O(N**2) vs O(N**3). From ok@REDACTED Tue Nov 16 02:55:01 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 16 Nov 2010 14:55:01 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: On 16/11/2010, at 4:02 AM, Edmond Begumisa wrote: > Hello Richard, > > I have a question... > >> Erlang is a fine tool for jobs like this one. > > Interesting... I would have never thought this. I would have instinctively reached for a language with mutable memory, thinking that this would make the most sense for combinatorial* work (I don't know this for fact, and I could be very wrong, but that's what instinct would tell me.) > * I've been calling this permutation but I don't know if that's accurate. It isn't. "Permutation" has a very specific meaning, which does not apply here. Let me quibble slightly: *memoisation* can help a lot for some combinatorial algorithms, but mutable memory as such doesn't strike me as obviously useful. (Indeed, since mutable memory makes memoisation invalid, it can seriously get in your way.) Good support for large integers helps a lot. Different algorithms may well have different needs. > >> There is little point in optimising a bad algorithm. > > Well put. But say you have an 'ok' algorithm. Not the best, but not blatantly flawed either. I have a C program to compute various things from a data set which consists of a hundred million lines, each containing four non-negative integers. My program had to read it several times. (The first pass determines the sizes of major areas; the second pass determines the size of minor areas within major areas; the third pass loads the data.) It gets the effect of sorting without actually sorting (or more precisely, by count-sorting). The first version of the program took 68 seconds per pass on the fastest machine I had access to. Considering that wc takes 20 seconds to _not_ process the numbers in the file, that didn't seem too bad. But bypassing stdio and using my own code let me get the time down to 6 seconds per pass, which means that I can afford to run the program a lot more often. That was a case where there was a _good_ algorithm, but a horrible library. (Time is linear in the size of the input; results depend on all of the input; up to a constant factor the algorithm is as good as it gets.) But in the case we are discussing here, the initial algorithm was O(N**3) when an O(N**2) algorithm -- in fact, several of them, there's even a Wikipedia article on enumerating Pythagorean triples -- was not hard to find. > > So my question is: if version 1 isn't performing "reasonably" acceptably for Garcia's purpose, and version 1 isn't blatantly flawed. To me, version 1 *IS* blatantly flawed. In a pythagorean triple (a,b,c), a and b uniquely determine c, so it seems *obvious* that a quadratic algorithm should be sought. (It is not obvious that one *exists*, just that it should be sought.) Make no mistake: I have used brute force algorithms quite happily myself. But I never blame the language for the performance I get when I do so! > Isn't this a strong indicator that he's using the wrong tool? Ah, but what precisely is "the tool" in question? - functional languages as such? - Erlang? - list comprehensions as such? - Erlang's current implementation of list comprehensions? Here are some more times. Erlang R14A, MacBook Pro. 8> pyth:t(p3, 300). {p3,300,{length,146},{msec,10}} 9> pyth:t(p2, 300). {p2,300,{length,146},{msec,210}} 10> pyth:t(p1, 300). {p1,300,{length,146},{msec,970}} 11> pyth:t(p0, 300). {p0,300,{length,146},{msec,420}} What's p0? It's the utterly naive cubic loop nest, with the loops written as function calls working on integers, with the only list in existence being the one that's being built. The time for p0 was 420 milliseconds in Erlang. On the same machine, the same function, the time was 91 milliseconds in SML/NJ. SML/NJ MLton Erlang n=100 L= 34 s=0.004 m=0.004 e=0.020 n=200 L= 86 s=0.028 m=0.027 e=0.130 n=300 L=146 s=0.090 m=0.086 e=0.410 n=400 L=210 s=0.213 m=0.202 e=0.980 n=500 L=274 s=0.415 m=0.394 e=1.900 We see that Erlang is about 5 times slower than SML. Here's the SML code. fun p0 n = let fun fa 0 l = l | fa a l = fb n a l and fb 0 a l = fa (a-1) l | fb b a l = fc n b a l and fc 0 b a l = fb (b-1) a l | fc c b a l = fc (c-1) b a ( if a+b+c <= n andalso a*a+b*b = c*c then (a,b,c)::l else l) in fa n [] end It's completely tail recursive, building the list from right to left. *THIS* is the version that is strictly comparable to any C or C# code one might plausibly write; it allocates no storage not part of the result, all looping is done with "while" loops that compare and increment integers. What's more, SML is strongly typed, and the Dialyzer should be able to handle this, so the compiler *knows* it is dealing with integers. I grant you that this code is not immediately clear. Clear code in SML might look like this: fun cat_tab l u f = (* concatenate + tabulate *) let fun loop i r = if i < l then r else loop (i-1) (f i @ r) in loop u [] end (* _happens_ not to be in List *) fun p1 n = cat_tab 1 n (fn a => cat_tab 1 n (fn b => cat_tab 1 n (fn c => if a+b+c <= n andalso a*a+b*b = c*c then [(a,b,c)] else []))) What happens to the times? (----------direct-----) (------cat_tab------) original! SML/NJ MLton Erlang SML/NJ Mlton Erlang Erlang n=100 L= 34 s=0.004 m=0.004 e=0.020 s=0.015 m=0.006 e=0.040 o=0.040 n=200 L= 86 s=0.028 m=0.027 e=0.130 s=0.120 m=0.039 e=0.300 o=0.290 n=300 L=146 s=0.090 m=0.086 e=0.410 s=0.403 m=0.128 e=0.950 o=0.980 n=400 L=210 s=0.213 m=0.202 e=0.980 s=0.954 m=0.303 e=2.240 o=2.280 n=500 L=274 s=0.415 m=0.394 e=1.900 s=1.857 m=0.591 e=4.300 o=4.400 The MLton compiler is a whole-program compiler, and I've a strong suspicion that it is able to inline cat_tab, so it may be able to avoid building the answer twice. I'm pretty sure that SML/NJ isn't inlining cat_tab, and is building the answer twice. While the Erlang compiler _could_ be taught about cat_tab, it's not generally wonderful at inlining recursive procedures, which is why the rightmost column shows list comprehension not doing so well. There are two basic issues here: (1) The Erlang compiler doesn't really handle list comprehension as native loops; it compiles list comprehensions to recursive functions. lists:seq(1, N) is not special syntax -- unlike Haskell -- and almost surely builds an actual list. This could be changed, and maybe some day it will change. (2) Erlang is compiled incrementally to support hot loading and (again in order to support hot loading) does not do much if any cross-module inlining. MLton is a whole-program optimising compiler, it can SML _are_ allowed to do cross-module inlining by the nature of the language. This *can't* be changed without making Erlang less useful for its job. Ah, but there's one more wrinkle! SML/NJ Erlang SML/NJ Smalltalk n=100 L= 34 s=0.004 e=0.020 s= 0.250 p1=0.016 n=200 L= 86 s=0.028 e=0.130 s= 1.904 p1=0.109 n=300 L=146 s=0.090 e=0.410 s= 6.567 p1=0.369 n=400 L=210 s=0.213 e=0.980 s=15.581 p1=0.873 n=500 L=274 s=0.415 e=1.900 s=30.218 p1=1.705 (3) Erlang integer arithmetic is *not* limited to some ridiculous size like 32 bits, it is _semantically_ unbounded integer arithmetic, with a space-efficient representation and fast calculation path for small integers. To get something comparable in SML, you have to use the type IntInf.int instead of Int.int. And when you do *that*, Erlang really really shines. The difference here is that if you use int in C or Java or C# or SML, then you'll find a positive integer n such that n+1 < 0. That kind of nonsense simply doesn't happen in Erlang. It comes with a price, but the price of *not* using a Lispy (or Smalltalky, or Erlangy) approach to integers is the price of getting wrong answers without knowing. The Smalltalk column was done using my Smalltalk-via-C compiler. It might be possible to go a factor of 2 faster using machine- specific assembly code, but what we seem to be looking at in the Erlang column really does seem to be no more than the price of arithmetic that gives correct results. And by the way, no, I am not advocating writing code like p0 above *UNTIL* it has been determined that the function is a performance problem and that the algorithm is good. Like I said, you should stick with the clean high level code until you understand the problem space better. From toby@REDACTED Tue Nov 16 03:17:56 2010 From: toby@REDACTED (Toby Thain) Date: Mon, 15 Nov 2010 21:17:56 -0500 Subject: [erlang-questions] **NEW** Now we are talking about!!! RESUME Pytagoriam Numbers In-Reply-To: <1CAE6784-69CA-4496-BCF0-458B9045F1BB@cs.otago.ac.nz> References: <4CE16F7F.8000500@telegraphics.com.au> <1CAE6784-69CA-4496-BCF0-458B9045F1BB@cs.otago.ac.nz> Message-ID: <4CE1E9D4.9040901@telegraphics.com.au> On 15/11/10 8:20 PM, Richard O'Keefe wrote: > > On 16/11/2010, at 6:35 AM, Toby Thain wrote: >> >> Benchmarking a single problem size doesn't expose the differing time >> complexity of the solutions (pace ROK). > > Why "pace"? I agree 100%. That's why I was talking about O(N**2) > vs O(N**3). I should have said "per". There isn't a language in which I can't make a malapropism. :) --Toby > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From mk@REDACTED Tue Nov 16 10:30:48 2010 From: mk@REDACTED (Morten Krogh) Date: Tue, 16 Nov 2010 10:30:48 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: <9838A96D-88AC-4532-9D62-753CAC717EE0@amberbio.com> Hi I couldn't wrap my head around why my prime number algorithm was so slow and had complexity N^2 or worse. I realized that it was simply because I forgot to terminate some recursions as early as possible, hence doing an increasing number of irrelevant computations. Theoretically I had expected N*log^2(N) behavior or so, because the relevant powers of the prime numbers falls of like log(P), which should maybe give N*log(N) behavior, and then there is also the sorting in the end because the prime number algorithm will produce the triples in another order. Anyway, I corrected the error, and see something like N*log^alpha(N) behavior. The runtime for N = 300 is 300 microseconds, and for N = 1,000,000 8 seconds. The final sorting of the result is actually a major overhead. Morten. pythag6(N) when is_integer(N) -> Primes = sieve(N div 2), M1M2s = incorporate_primes([{1,1}], [], N, Primes), lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2} <- M1M2s, M1 > M2, A <- [M1-M2], B <- [2*round(math:sqrt(M1*M2))], C <- [M1+M2], A+B+C =< N])). incorporate_primes(Active_all, Passive_all, _N, []) -> Active_all ++ Passive_all; incorporate_primes(Active_all, Passive_all, N, [P|Primes]) -> {Active, Passive} = incorporate_prime(Active_all, [], [], N, P), incorporate_primes(Active, Passive ++ Passive_all, N, Primes). incorporate_prime([], Active, Passive, _N, _P) -> {Active, Passive}; incorporate_prime([{M1, M2}|M1M2s], Active, Passive, N, P) -> {EvensM1, OddsM1} = m_times_powers_of_p(M1, N div 2, P, even, [], []), {EvensM2, OddsM2} = m_times_powers_of_p(M2, N div 2, P, even, [], []), case {EvensM1, OddsM1, EvensM2, OddsM2} of {[_], [], [_], []} -> incorporate_prime(M1M2s, Active, [{M1,M2}| Passive], N, P); {[_], [_], [_], []} -> incorporate_prime(M1M2s, Active, [{M1,M2}| Passive], N, P); {[_], [], [_], [_]} -> incorporate_prime(M1M2s, Active, [{M1,M2}| Passive], N, P); _ -> Evens = [{X, Y} || X <- EvensM1, Y <- EvensM2], Odds = [{X, Y} || X <- OddsM1, Y <- OddsM2], incorporate_prime(M1M2s, Odds ++ Evens ++ Active, Passive, N, P) end. m_times_powers_of_p(M, L, _P, _Parity, Evens, Odds) when M > L -> {Evens, Odds}; m_times_powers_of_p(M, L, P, even, Evens, Odds) -> m_times_powers_of_p(M*P, L, P, odd, [M|Evens], Odds); m_times_powers_of_p(M, L, P, odd, Evens, Odds) -> m_times_powers_of_p(M*P, L, P, even, Evens, [M|Odds]). sieve(N) when is_integer(N) -> erase(), sieve(N,2). sieve(N, K) when K >= N -> [X || X <- lists:seq(2, N), erase(X) == undefined]; sieve(N, K) -> cross_off(K, K, N div K - 1), sieve(N, find_next_in_sieve(K + 1)). cross_off(_K, _Current, 0) -> ok; cross_off(K, Current, Left) -> Next = Current + K, put(Next, out), cross_off(K, Next, Left - 1). find_next_in_sieve(K) -> case get(K) of undefined -> K; _ -> find_next_in_sieve(K+1) end. On Nov 16, 2010, at 2:55 AM, Richard O'Keefe wrote: > > On 16/11/2010, at 4:02 AM, Edmond Begumisa wrote: > >> Hello Richard, >> >> I have a question... >> >>> Erlang is a fine tool for jobs like this one. >> >> Interesting... I would have never thought this. I would have instinctively reached for a language with mutable memory, thinking that this would make the most sense for combinatorial* work (I don't know this for fact, and I could be very wrong, but that's what instinct would tell me.) > >> * I've been calling this permutation but I don't know if that's accurate. > > It isn't. "Permutation" has a very specific meaning, which does not apply here. > > Let me quibble slightly: *memoisation* can help a lot for some combinatorial > algorithms, but mutable memory as such doesn't strike me as obviously useful. > (Indeed, since mutable memory makes memoisation invalid, it can seriously get > in your way.) Good support for large integers helps a lot. > > Different algorithms may well have different needs. >> >>> There is little point in optimising a bad algorithm. >> >> Well put. But say you have an 'ok' algorithm. Not the best, but not blatantly flawed either. > > I have a C program to compute various things from a data set which consists of > a hundred million lines, each containing four non-negative integers. My > program had to read it several times. (The first pass determines the sizes > of major areas; the second pass determines the size of minor areas within > major areas; the third pass loads the data.) It gets the effect of sorting > without actually sorting (or more precisely, by count-sorting). The first > version of the program took 68 seconds per pass on the fastest machine I > had access to. Considering that wc takes 20 seconds to _not_ process the > numbers in the file, that didn't seem too bad. But bypassing stdio and using > my own code let me get the time down to 6 seconds per pass, which means that > I can afford to run the program a lot more often. > > That was a case where there was a _good_ algorithm, but a horrible library. > (Time is linear in the size of the input; results depend on all of the input; > up to a constant factor the algorithm is as good as it gets.) > > But in the case we are discussing here, the initial algorithm was O(N**3) > when an O(N**2) algorithm -- in fact, several of them, there's even a > Wikipedia article on enumerating Pythagorean triples -- was not hard to find. > >> >> So my question is: if version 1 isn't performing "reasonably" acceptably for Garcia's purpose, and version 1 isn't blatantly flawed. > > To me, version 1 *IS* blatantly flawed. > In a pythagorean triple (a,b,c), a and b uniquely determine c, > so it seems *obvious* that a quadratic algorithm should be sought. > (It is not obvious that one *exists*, just that it should be sought.) > > Make no mistake: I have used brute force algorithms quite happily myself. > But I never blame the language for the performance I get when I do so! > > >> Isn't this a strong indicator that he's using the wrong tool? > > Ah, but what precisely is "the tool" in question? > - functional languages as such? > - Erlang? > - list comprehensions as such? > - Erlang's current implementation of list comprehensions? > > Here are some more times. Erlang R14A, MacBook Pro. > > 8> pyth:t(p3, 300). > {p3,300,{length,146},{msec,10}} > 9> pyth:t(p2, 300). > {p2,300,{length,146},{msec,210}} > 10> pyth:t(p1, 300). > {p1,300,{length,146},{msec,970}} > 11> pyth:t(p0, 300). > {p0,300,{length,146},{msec,420}} > > What's p0? > It's the utterly naive cubic loop nest, with the loops written > as function calls working on integers, with the only list in > existence being the one that's being built. > > The time for p0 was 420 milliseconds in Erlang. > On the same machine, the same function, the time was > 91 milliseconds in SML/NJ. > SML/NJ MLton Erlang > n=100 L= 34 s=0.004 m=0.004 e=0.020 > n=200 L= 86 s=0.028 m=0.027 e=0.130 > n=300 L=146 s=0.090 m=0.086 e=0.410 > n=400 L=210 s=0.213 m=0.202 e=0.980 > n=500 L=274 s=0.415 m=0.394 e=1.900 > > We see that Erlang is about 5 times slower than SML. > Here's the SML code. > > fun p0 n = > let fun fa 0 l = l > | fa a l = fb n a l > and fb 0 a l = fa (a-1) l > | fb b a l = fc n b a l > and fc 0 b a l = fb (b-1) a l > | fc c b a l = fc (c-1) b a ( > if a+b+c <= n andalso a*a+b*b = c*c > then (a,b,c)::l else l) > in fa n [] > end > > It's completely tail recursive, building the list from right to > left. *THIS* is the version that is strictly comparable to any > C or C# code one might plausibly write; it allocates no > storage not part of the result, all looping is done with "while" > loops that compare and increment integers. What's more, SML is > strongly typed, and the Dialyzer should be able to handle this, > so the compiler *knows* it is dealing with integers. > > I grant you that this code is not immediately clear. > Clear code in SML might look like this: > > fun cat_tab l u f = (* concatenate + tabulate *) > let fun loop i r = if i < l then r else loop (i-1) (f i @ r) > in loop u [] > end (* _happens_ not to be in List *) > > fun p1 n = > cat_tab 1 n (fn a => > cat_tab 1 n (fn b => > cat_tab 1 n (fn c => > if a+b+c <= n andalso a*a+b*b = c*c > then [(a,b,c)] else []))) > > What happens to the times? > (----------direct-----) (------cat_tab------) original! > SML/NJ MLton Erlang SML/NJ Mlton Erlang Erlang > n=100 L= 34 s=0.004 m=0.004 e=0.020 s=0.015 m=0.006 e=0.040 o=0.040 > n=200 L= 86 s=0.028 m=0.027 e=0.130 s=0.120 m=0.039 e=0.300 o=0.290 > n=300 L=146 s=0.090 m=0.086 e=0.410 s=0.403 m=0.128 e=0.950 o=0.980 > n=400 L=210 s=0.213 m=0.202 e=0.980 s=0.954 m=0.303 e=2.240 o=2.280 > n=500 L=274 s=0.415 m=0.394 e=1.900 s=1.857 m=0.591 e=4.300 o=4.400 > > The MLton compiler is a whole-program compiler, and I've a > strong suspicion that it is able to inline cat_tab, so it > may be able to avoid building the answer twice. I'm pretty > sure that SML/NJ isn't inlining cat_tab, and is building the > answer twice. While the Erlang compiler _could_ be taught > about cat_tab, it's not generally wonderful at inlining recursive > procedures, which is why the rightmost column shows list > comprehension not doing so well. > > There are two basic issues here: > (1) The Erlang compiler doesn't really handle list comprehension as > native loops; it compiles list comprehensions to recursive > functions. lists:seq(1, N) is not special syntax -- unlike Haskell -- > and almost surely builds an actual list. > > This could be changed, and maybe some day it will change. > > (2) Erlang is compiled incrementally to support hot loading and > (again in order to support hot loading) does not do much if any > cross-module inlining. > MLton is a whole-program optimising compiler, it can SML _are_ > allowed to do cross-module inlining by the nature of the language. > > This *can't* be changed without making Erlang less useful for its job. > > Ah, but there's one more wrinkle! > > SML/NJ Erlang SML/NJ Smalltalk > n=100 L= 34 s=0.004 e=0.020 s= 0.250 p1=0.016 > n=200 L= 86 s=0.028 e=0.130 s= 1.904 p1=0.109 > n=300 L=146 s=0.090 e=0.410 s= 6.567 p1=0.369 > n=400 L=210 s=0.213 e=0.980 s=15.581 p1=0.873 > n=500 L=274 s=0.415 e=1.900 s=30.218 p1=1.705 > > (3) Erlang integer arithmetic is *not* limited to some ridiculous > size like 32 bits, it is _semantically_ unbounded integer > arithmetic, with a space-efficient representation and fast > calculation path for small integers. To get something > comparable in SML, you have to use the type IntInf.int > instead of Int.int. And when you do *that*, Erlang really > really shines. > > The difference here is that if you use int in C or Java or C# > or SML, then you'll find a positive integer n such that n+1 < 0. > That kind of nonsense simply doesn't happen in Erlang. > > It comes with a price, but the price of *not* using a Lispy > (or Smalltalky, or Erlangy) approach to integers is the price > of getting wrong answers without knowing. > > The Smalltalk column was done using my Smalltalk-via-C compiler. > It might be possible to go a factor of 2 faster using machine- > specific assembly code, but what we seem to be looking at in > the Erlang column really does seem to be no more than the > price of arithmetic that gives correct results. > > And by the way, no, I am not advocating writing code like p0 above > *UNTIL* it has been determined that the function is a performance > problem and that the algorithm is good. Like I said, you should > stick with the clean high level code until you understand the > problem space better. > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From tom@REDACTED Tue Nov 16 11:14:01 2010 From: tom@REDACTED (tom@REDACTED) Date: Tue, 16 Nov 2010 11:14:01 +0100 Subject: crash dump at ejabberd startup Message-ID: <201011161114.01488.tom@diogunix.com> Hello everybody in the Erlang universe, I got stuck with a problem getting ejabberd to run in a FreeBSD8 Jail. Erlang and ejabberd were built using the Ports Collection (this week). As far as I could learn from googling all this, the issue goes back to the erlang environment and abviously has something to do with "inet", TCP etc.. Well, even I know a bit about Unix I can't help myself with the error messages erlang spits out (no clue what these messages actually / precisely want to tell me): ejabberd Start: # ejabberdctl status {error_logger,{{2010,11,16},{9,56,20}},"Protocol: ~p: register error: ~p~n", ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, {net_kernel,start_protos,4},{net_kernel,start_protos,3}, {net_kernel,init_node,2},{net_kernel,init,1},{gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}]} {error_logger,{{2010,11,16},{9,56,20}},crash_report,[[{initial_call, {net_kernel,init,['Argument__1']}},{pid,<0.19.0>},{registered_name,[]}, {error_info,{exit,{error,badarg},[{gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}},{ancestors,[net_sup,kernel_sup,<0.9.0>]}, {messages,[]},{links,[#Port<0.101>,<0.16.0>]},{dictionary, [{longnames,false}]},{trap_exit,true},{status,running},{heap_size,610}, {stack_size,24},{reductions,489}],[]]} {error_logger,{{2010,11,16},{9,56,20}},supervisor_report,[{supervisor, {local,net_sup}},{errorContext,start_error},{reason, {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, {mfargs,{net_kernel,start_link,[['ctl-17-ejabberd@REDACTED',shortnames]]}}, {restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} {error_logger,{{2010,11,16},{9,56,20}},supervisor_report,[{supervisor, {local,kernel_sup}},{errorContext,start_error},{reason,shutdown},{offender, [{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}}, {restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} {error_logger,{{2010,11,16},{9,56,20}},std_info,[{application,kernel}, {exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]} {"Kernel pid terminated",application_controller,"{application_start_failure,kernel, {shutdown,{kernel,start,[normal,[]]}}}"} Crash dump was written to: /var/log/ejabberd/erl_crash_20101116-095620.dump Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}}) Well, I wouldn't be surprised if all this had to do with the mysterious "inetrc" configuration file. The example from the FreeBSD port contains: # cat inetrc {lookup,["file","native"]}. {host,{127,0,0,1}, ["localhost","hostalias"]}. {file, resolv, "/etc/resolv.conf"}. I already tried some hints from what could be found via Google - but nonetheless didn't succeed or even made progress. The funny thing is, I once had ejabberd running as a test system in another FreeBSD Jail but unfortunately did not note all details before wiping that old host. So, anybody out there capable to give any hints ? Understanding the problem might be 90% of getting the solution ... Many thanks in advance ! kind regards Tom From als@REDACTED Tue Nov 16 11:37:39 2010 From: als@REDACTED (Anthony Shipman) Date: Tue, 16 Nov 2010 21:37:39 +1100 Subject: [erlang-questions] What's a solid way to have erlang automatically send email. In-Reply-To: References: Message-ID: <201011162137.39132.als@iinet.net.au> On Mon, 15 Nov 2010 10:31:09 am G.S. wrote: > Hello, > > I have a website where the administrator needs to receive email alert (to > login and fix a problem) when certain events occur. Is there an Erlang > module that allows one to easily send out an email? Or something that can > be used in the same manner? > > Thanks, > -Gene On UNIX, you could feed the text into stdin of /usr/sbin/sendmail or /usr/bin/mutt. erlang:open_port() will let you do this. -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From hynek@REDACTED Tue Nov 16 13:34:51 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Tue, 16 Nov 2010 13:34:51 +0100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: On Sun, Nov 14, 2010 at 5:50 PM, Edmond Begumisa wrote: > Hynek, > > A few comments... > >> When Erlang implementation is faster than C implementation *is* badly >> written even it can be for maintainability or haste reason. C++ is >> another beast > > I disagree. Quick example... > > 1. Write a "well-written" web-server in Erlang, the "Erlang way" > (lightweight-weight concurrency, message passing). > 2. Write a "well-written" web-server in C/C++, the "C/C++ way" > (mulit-threading, shared-memory). > > See which is faster. Or, you could save yourself some time and look at > Apache + Yaws. I wouldn't describe either of these as badly written. But the > performance difference is undeniable. > Why I should write C/C++ web-server in "C/C++ way" (mulit-threading, shared-memory)? Is dumbness included in using C? I can write C web-server using libevent for example. I can use kernel poll. I can do all this cool stuff as Erlang does and I can overperform Erlang using its own weapons. I can do it better and tune for this particular case and will be better. I can but if I should or would is another question. Keep in mind that Erlang VM itself is written in C. >> You can also do tricks in C which you can't in Erlang. > > And you can do tricks in Erlang that you can't do in typical C (e.g easy > parallelism with little change to code as evidenced on this thread.) Everything what I can do in Erlang I can do in C, principally. Opposite is not true. Keep in mind that Erlang VM itself *is* written in C. > >> Can you write k/v store which >> is able do 200 millions of look up operations per second on one 2.4GHz >> i5 core? > > Something is getting lost here. I go back to the promises a language makes. > In delivering on those promises, priorities have to be set and compromises > made. The language user has to weight these. > > C/C++ gives us pointers (amongst other things). The language makes great > sacrifices to give us pointers (like garbage collection and debugability) so > we can create insanely efficient data structures. When the benefits of > pointers far outweigh the trade-offs, the choice is clear. > > Erlang gives us high concurrency (amongst other things). The Erlang team > made great sacrifices to give us insanely efficient and scalable > concurrency. When the benefits of this far outweigh the trade-offs the > choice is clear. > > I see few areas where these benefits/trade-offs overlap. The choice is > usually very clear. In cases where overlapping does occur, there are various > ways of using both. > > - Edmond - > > I would not argue using only one language for everything. Did I? As I have written slight down in mine reply, I'm surprised by Erlang HiPE compiler work. It performs surprisingly well and I would not expect many orders of magnitude improvement by rewriting to C. I would not say it would be even ten times unless I would try, but I bet it would, if I pay enough effort. I would argue that Erlang with NIFs gives you ultimate tool. Erlang and C are not enemies, they don't fight themselves, they makes prefect pair for almost all purposes. > > On Sun, 14 Nov 2010 22:32:42 +1100, Hynek Vychodil > wrote: > >> On Sun, Nov 14, 2010 at 11:55 AM, Tony Rogvall wrote: >>> >>> On 14 nov 2010, at 10.17, Hynek Vychodil wrote: >>> >>> >>> >>> Good idea but it seems that native (HiPE) compiler does this >>> optimization for you when you keep staying in one module. (I also >>> keeps exporting only main function. It can possibly take effect here >>> also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not >>> right tool for this job. You should use NIF if performance matter. >>> >>> pythag4(N) when is_integer(N) -> pythag4(N,1). >>> >>> I have implemented several small Erlang programs that beat "native" C >>> code. >>> Most of the time the C programs where badly/hastily written, but that is >>> the >>> point ;-) >>> You must select the algorithm and the implementation wisely, and of >>> course >>> use the golden rule "Make it work, then make it fast". I would add, if it >>> is >>> needed. >>> This does not imply that you should write your program unnecessary slow >>> to >>> start with! >>> Any how. A couple of months ago I implemented the AKS algorithm in >>> Erlang. >>> The AKS algorithm is a deterministic primality test algorithm >>> (http://en.wikipedia.org/wiki/AKS_primality_test) >>> I benchmarked this implementation with an implementation in C++. >>> I was shocked: The Erlang version was about 5 times faster, NOT using >>> obvious parallelism. >>> In this case I would suspect that garbage collection is the major >>> contributor! The implementation use >>> a lot of temporary polynomials intermediate results. A copy collector >>> does >>> the trick. >>> /Tony >>> >>> >>> >>> "Have run Make so many times I dunno what's installed anymore" >>> >> >> When Erlang implementation is faster than C implementation *is* badly >> written even it can be for maintainability or haste reason. C++ is >> another beast. Pun intended. I have similar experience to you but when >> I found that Erlang implementation is faster then I would look how is >> it possible and you are right, usually it is memory issue. Anyway >> every time you can do same tricks as Erlang does but you will end up >> with "half and error prone implementation of Erlang". You can also do >> tricks in C which you can't in Erlang. Can you write k/v store which >> is able do 200 millions of look up operations per second on one 2.4GHz >> i5 core? Anyway HiPE compiler does very good work here. If I count it >> correctly pythag3 and pythag4 does about hundred millions checks per >> seconds. Very good result I think. >> >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From michael.santos@REDACTED Tue Nov 16 13:59:15 2010 From: michael.santos@REDACTED (Michael Santos) Date: Tue, 16 Nov 2010 07:59:15 -0500 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <201011161114.01488.tom@diogunix.com> References: <201011161114.01488.tom@diogunix.com> Message-ID: <20101116125803.GA27248@ecn.lan> On Tue, Nov 16, 2010 at 11:14:01AM +0100, tom@REDACTED wrote: > Hello everybody in the Erlang universe, > > I got stuck with a problem getting ejabberd to run in a FreeBSD8 Jail. > Erlang and ejabberd were built using the Ports Collection (this week). > > As far as I could learn from googling all this, the issue goes back to the > erlang environment and abviously has something to do with "inet", TCP etc.. > Well, even I know a bit about Unix I can't help myself with the error > messages erlang spits out (no clue what these messages actually / precisely > want to tell me): > > ejabberd Start: > > # ejabberdctl status > {error_logger,{{2010,11,16},{9,56,20}},"Protocol: ~p: register error: ~p~n", > ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, When a distributed Erlang node starts up, it attempts to register itself with an epmd daemon listening on localhost on port 4369 and starts one, if an epmd is not running. Check epmd is running and if the node is allowed to contact it from the jail. You can get debugging info by running epmd manually: epmd -d -d -d From ebegumisa@REDACTED Tue Nov 16 14:52:54 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 17 Nov 2010 00:52:54 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: Thanks very much for your insight. You lost me at some point but I'm basically with you (I hope!) This thread has been very, very useful and educational for me. I'm trying some of the algo's presented in C for further study and comparision. It's clearer now that my assumptions about what should and should not be attempted in Erlang were completely wrong. Although that does introduce some confusion since I have to now think harder! The naive decisions were much easier to make :) There is a particular side project I wanted to do where I instinctively decided certain parts would be out-sourced to C/C++. As a result of this thread, I'm not so sure anymore. Venturing outside the emulator might not be worth it. I'll have to look much closer. One more question though! From real experiences, are there specific problems that you would just NEVER EVER attempt in Erlang? (I know of the list on the FAQ but even that seems shaky now.) - Edmond - PS: When this thread first appeared, I chose to ignore it, thinking "that's a flame thread with nothing worth reading." This mailing list is unique in resisting flame temptation and sticking to the issues. Other lists are not like that :) On Tue, 16 Nov 2010 12:55:01 +1100, Richard O'Keefe wrote: > > On 16/11/2010, at 4:02 AM, Edmond Begumisa wrote: > >> Hello Richard, >> >> I have a question... >> >>> Erlang is a fine tool for jobs like this one. >> >> Interesting... I would have never thought this. I would have >> instinctively reached for a language with mutable memory, thinking that >> this would make the most sense for combinatorial* work (I don't know >> this for fact, and I could be very wrong, but that's what instinct >> would tell me.) > >> * I've been calling this permutation but I don't know if that's >> accurate. > > It isn't. "Permutation" has a very specific meaning, which does not > apply here. > > Let me quibble slightly: *memoisation* can help a lot for some > combinatorial > algorithms, but mutable memory as such doesn't strike me as obviously > useful. > (Indeed, since mutable memory makes memoisation invalid, it can > seriously get > in your way.) Good support for large integers helps a lot. > > Different algorithms may well have different needs. >> >>> There is little point in optimising a bad algorithm. >> >> Well put. But say you have an 'ok' algorithm. Not the best, but not >> blatantly flawed either. > > I have a C program to compute various things from a data set which > consists of > a hundred million lines, each containing four non-negative integers. My > program had to read it several times. (The first pass determines the > sizes > of major areas; the second pass determines the size of minor areas within > major areas; the third pass loads the data.) It gets the effect of > sorting > without actually sorting (or more precisely, by count-sorting). The > first > version of the program took 68 seconds per pass on the fastest machine I > had access to. Considering that wc takes 20 seconds to _not_ process the > numbers in the file, that didn't seem too bad. But bypassing stdio and > using > my own code let me get the time down to 6 seconds per pass, which means > that > I can afford to run the program a lot more often. > > That was a case where there was a _good_ algorithm, but a horrible > library. > (Time is linear in the size of the input; results depend on all of the > input; > up to a constant factor the algorithm is as good as it gets.) > > But in the case we are discussing here, the initial algorithm was O(N**3) > when an O(N**2) algorithm -- in fact, several of them, there's even a > Wikipedia article on enumerating Pythagorean triples -- was not hard to > find. > >> >> So my question is: if version 1 isn't performing "reasonably" >> acceptably for Garcia's purpose, and version 1 isn't blatantly flawed. > > To me, version 1 *IS* blatantly flawed. > In a pythagorean triple (a,b,c), a and b uniquely determine c, > so it seems *obvious* that a quadratic algorithm should be sought. > (It is not obvious that one *exists*, just that it should be sought.) > > Make no mistake: I have used brute force algorithms quite happily > myself. > But I never blame the language for the performance I get when I do so! > > >> Isn't this a strong indicator that he's using the wrong tool? > > Ah, but what precisely is "the tool" in question? > - functional languages as such? > - Erlang? > - list comprehensions as such? > - Erlang's current implementation of list comprehensions? > > Here are some more times. Erlang R14A, MacBook Pro. > > 8> pyth:t(p3, 300). > {p3,300,{length,146},{msec,10}} > 9> pyth:t(p2, 300). > {p2,300,{length,146},{msec,210}} > 10> pyth:t(p1, 300). > {p1,300,{length,146},{msec,970}} > 11> pyth:t(p0, 300). > {p0,300,{length,146},{msec,420}} > > What's p0? > It's the utterly naive cubic loop nest, with the loops written > as function calls working on integers, with the only list in > existence being the one that's being built. > > The time for p0 was 420 milliseconds in Erlang. > On the same machine, the same function, the time was > 91 milliseconds in SML/NJ. > SML/NJ MLton Erlang > n=100 L= 34 s=0.004 m=0.004 e=0.020 > n=200 L= 86 s=0.028 m=0.027 e=0.130 > n=300 L=146 s=0.090 m=0.086 e=0.410 > n=400 L=210 s=0.213 m=0.202 e=0.980 > n=500 L=274 s=0.415 m=0.394 e=1.900 > > We see that Erlang is about 5 times slower than SML. > Here's the SML code. > > fun p0 n = > let fun fa 0 l = l > | fa a l = fb n a l > and fb 0 a l = fa (a-1) l > | fb b a l = fc n b a l > and fc 0 b a l = fb (b-1) a l > | fc c b a l = fc (c-1) b a ( > if a+b+c <= n andalso a*a+b*b = c*c > then (a,b,c)::l else l) > in fa n [] > end > > It's completely tail recursive, building the list from right to > left. *THIS* is the version that is strictly comparable to any > C or C# code one might plausibly write; it allocates no > storage not part of the result, all looping is done with "while" > loops that compare and increment integers. What's more, SML is > strongly typed, and the Dialyzer should be able to handle this, > so the compiler *knows* it is dealing with integers. > > I grant you that this code is not immediately clear. > Clear code in SML might look like this: > > fun cat_tab l u f = (* concatenate + tabulate *) > let fun loop i r = if i < l then r else loop (i-1) (f i @ r) > in loop u [] > end (* _happens_ not to be in List *) > > fun p1 n = > cat_tab 1 n (fn a => > cat_tab 1 n (fn b => > cat_tab 1 n (fn c => > if a+b+c <= n andalso a*a+b*b = c*c > then [(a,b,c)] else []))) > > What happens to the times? > (----------direct-----) (------cat_tab------) original! > SML/NJ MLton Erlang SML/NJ Mlton Erlang Erlang > n=100 L= 34 s=0.004 m=0.004 e=0.020 s=0.015 m=0.006 e=0.040 o=0.040 > n=200 L= 86 s=0.028 m=0.027 e=0.130 s=0.120 m=0.039 e=0.300 o=0.290 > n=300 L=146 s=0.090 m=0.086 e=0.410 s=0.403 m=0.128 e=0.950 o=0.980 > n=400 L=210 s=0.213 m=0.202 e=0.980 s=0.954 m=0.303 e=2.240 o=2.280 > n=500 L=274 s=0.415 m=0.394 e=1.900 s=1.857 m=0.591 e=4.300 o=4.400 > > The MLton compiler is a whole-program compiler, and I've a > strong suspicion that it is able to inline cat_tab, so it > may be able to avoid building the answer twice. I'm pretty > sure that SML/NJ isn't inlining cat_tab, and is building the > answer twice. While the Erlang compiler _could_ be taught > about cat_tab, it's not generally wonderful at inlining recursive > procedures, which is why the rightmost column shows list > comprehension not doing so well. > > There are two basic issues here: > (1) The Erlang compiler doesn't really handle list comprehension as > native loops; it compiles list comprehensions to recursive > functions. lists:seq(1, N) is not special syntax -- unlike Haskell > -- > and almost surely builds an actual list. > > This could be changed, and maybe some day it will change. > > (2) Erlang is compiled incrementally to support hot loading and > (again in order to support hot loading) does not do much if any > cross-module inlining. > MLton is a whole-program optimising compiler, it can SML _are_ > allowed to do cross-module inlining by the nature of the language. > > This *can't* be changed without making Erlang less useful for its > job. > > Ah, but there's one more wrinkle! > > SML/NJ Erlang SML/NJ Smalltalk > n=100 L= 34 s=0.004 e=0.020 s= 0.250 p1=0.016 > n=200 L= 86 s=0.028 e=0.130 s= 1.904 p1=0.109 > n=300 L=146 s=0.090 e=0.410 s= 6.567 p1=0.369 > n=400 L=210 s=0.213 e=0.980 s=15.581 p1=0.873 > n=500 L=274 s=0.415 e=1.900 s=30.218 p1=1.705 > > (3) Erlang integer arithmetic is *not* limited to some ridiculous > size like 32 bits, it is _semantically_ unbounded integer > arithmetic, with a space-efficient representation and fast > calculation path for small integers. To get something > comparable in SML, you have to use the type IntInf.int > instead of Int.int. And when you do *that*, Erlang really > really shines. > > The difference here is that if you use int in C or Java or C# > or SML, then you'll find a positive integer n such that n+1 < 0. > That kind of nonsense simply doesn't happen in Erlang. > > It comes with a price, but the price of *not* using a Lispy > (or Smalltalky, or Erlangy) approach to integers is the price > of getting wrong answers without knowing. > > The Smalltalk column was done using my Smalltalk-via-C compiler. > It might be possible to go a factor of 2 faster using machine- > specific assembly code, but what we seem to be looking at in > the Erlang column really does seem to be no more than the > price of arithmetic that gives correct results. > And by the way, no, I am not advocating writing code like p0 above > *UNTIL* it has been determined that the function is a performance > problem and that the algorithm is good. Like I said, you should > stick with the clean high level code until you understand the > problem space better. > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Tue Nov 16 14:53:00 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 17 Nov 2010 00:53:00 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: On Tue, 16 Nov 2010 08:28:22 +1100, Jesper Louis Andersen wrote: > My experience is that C, C++ or Java tend to require much more work > for these iterations I'm beginning to notice this. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From kostis@REDACTED Tue Nov 16 15:20:51 2010 From: kostis@REDACTED (Kostis Sagonas) Date: Tue, 16 Nov 2010 16:20:51 +0200 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> Message-ID: <4CE29343.30406@cs.ntua.gr> Richard O'Keefe wrote: > ... SNIP > > The time for p0 was 420 milliseconds in Erlang. > On the same machine, the same function, the time was > 91 milliseconds in SML/NJ. > SML/NJ MLton Erlang > n=100 L= 34 s=0.004 m=0.004 e=0.020 > n=200 L= 86 s=0.028 m=0.027 e=0.130 > n=300 L=146 s=0.090 m=0.086 e=0.410 > n=400 L=210 s=0.213 m=0.202 e=0.980 > n=500 L=274 s=0.415 m=0.394 e=1.900 > > We see that Erlang is about 5 times slower than SML. Is Erlang running native code here? (Because both SML/NJ and MLton are.) > Here's the SML code. > > ... What's more, SML is > strongly typed, and the Dialyzer should be able to handle this, > so the compiler *knows* it is dealing with integers. Let's leave dialyzer out of this discussion -- it is a type analyzer alright, but the types it infers are not necessarily usable for optimization purposes (I can possibly elaborate in some other mail). Can you post p0 to see whether the compiler can infer whether the program deals with integers or not? > There are two basic issues here: > (1) The Erlang compiler doesn't really handle list comprehension as > native loops; it compiles list comprehensions to recursive > functions. lists:seq(1, N) is not special syntax -- unlike Haskell -- > and almost surely builds an actual list. Yes. This is an issue. Actually, it's more general: in my experience most Erlang programmers do not understand the difference between list comprehensions and lists:foreach/2. Many of them find list comprehension notation so nice that they often use it in places where building the list is not only is not their intention but is completely unnecessary. The compiler is not sufficiently clever to perform deforestation and avoid construction of intermediate lists. > This could be changed, and maybe some day it will change. > > (2) Erlang is compiled incrementally to support hot loading and > (again in order to support hot loading) does not do much if any > cross-module inlining. > MLton is a whole-program optimising compiler, it can SML _are_ > allowed to do cross-module inlining by the nature of the language. > > This *can't* be changed without making Erlang less useful for its job. > > Ah, but there's one more wrinkle! Yes, there is that too in Erlang ;-) > SML/NJ Erlang SML/NJ Smalltalk > n=100 L= 34 s=0.004 e=0.020 s= 0.250 p1=0.016 > n=200 L= 86 s=0.028 e=0.130 s= 1.904 p1=0.109 > n=300 L=146 s=0.090 e=0.410 s= 6.567 p1=0.369 > n=400 L=210 s=0.213 e=0.980 s=15.581 p1=0.873 > n=500 L=274 s=0.415 e=1.900 s=30.218 p1=1.705 > > (3) Erlang integer arithmetic is *not* limited to some ridiculous > size like 32 bits, it is _semantically_ unbounded integer > arithmetic, with a space-efficient representation and fast > calculation path for small integers. To get something > comparable in SML, you have to use the type IntInf.int > instead of Int.int. And when you do *that*, Erlang really > really shines. > > The difference here is that if you use int in C or Java or C# > or SML, then you'll find a positive integer n such that n+1 < 0. > That kind of nonsense simply doesn't happen in Erlang. > > It comes with a price, but the price of *not* using a Lispy > (or Smalltalky, or Erlangy) approach to integers is the price > of getting wrong answers without knowing. > > The Smalltalk column was done using my Smalltalk-via-C compiler. > It might be possible to go a factor of 2 faster using machine- > specific assembly code, but what we seem to be looking at in > the Erlang column really does seem to be no more than the > price of arithmetic that gives correct results. Thanks for a nice post! Kostis From mevans@REDACTED Tue Nov 16 15:51:55 2010 From: mevans@REDACTED (Evans, Matthew) Date: Tue, 16 Nov 2010 09:51:55 -0500 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: Interesting thread...I think we can all take the following from this: 1) Write your solution in the "Erlang way", using Erlang's strengths such as spawning parallel processes, also HIPE and NIFS where appropriate. 2) Write your solution efficiently - true in all languages, but especially in Erlang where you could end up doing needless recursion. 3) Erlang isn't always the best language for all problems. Neither is C, C++, C#, Java or any other language. Choose the language, or a mix of languages, that are the best fit your problem. What I would like to add is that we are trying to get Erlang accepted in a very C/C++ centric development environment. Although time and time again Erlang has shown itself a superior language and development environment for our domain, there is still a big problem in people accepting it. One of the main stumbling blocks is it is perceived to be "slow". People always say "it won't be faster than a C/C++ solution", or "a language written in a VM will always be slow [rolleyes]". We can argue until we are blue in the face that it will be as good or better when you throw multi-cores in the mix. That there are other advantages including shorter development time, better debugging, redundancy, failure detection etc. But overcoming that initial "speed" bias is a tough sell. I'm not saying there is an answer to this, and I know that the development team is doing their best to make Erlang faster, but we mustn't forget that for many, the "perceived" slowness is one factor that prevents them developing in Erlang. /Rant over Matt -----Original Message----- From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Hynek Vychodil Sent: Tuesday, November 16, 2010 7:35 AM To: Edmond Begumisa Cc: Tony Rogvall; Willem de Jong; Gilberto Carmenate Garc?a; erlang-questions@REDACTED Subject: Re: [erlang-questions] Erlang shows its slow face! On Sun, Nov 14, 2010 at 5:50 PM, Edmond Begumisa wrote: > Hynek, > > A few comments... > >> When Erlang implementation is faster than C implementation *is* badly >> written even it can be for maintainability or haste reason. C++ is >> another beast > > I disagree. Quick example... > > 1. Write a "well-written" web-server in Erlang, the "Erlang way" > (lightweight-weight concurrency, message passing). > 2. Write a "well-written" web-server in C/C++, the "C/C++ way" > (mulit-threading, shared-memory). > > See which is faster. Or, you could save yourself some time and look at > Apache + Yaws. I wouldn't describe either of these as badly written. But the > performance difference is undeniable. > Why I should write C/C++ web-server in "C/C++ way" (mulit-threading, shared-memory)? Is dumbness included in using C? I can write C web-server using libevent for example. I can use kernel poll. I can do all this cool stuff as Erlang does and I can overperform Erlang using its own weapons. I can do it better and tune for this particular case and will be better. I can but if I should or would is another question. Keep in mind that Erlang VM itself is written in C. >> You can also do tricks in C which you can't in Erlang. > > And you can do tricks in Erlang that you can't do in typical C (e.g easy > parallelism with little change to code as evidenced on this thread.) Everything what I can do in Erlang I can do in C, principally. Opposite is not true. Keep in mind that Erlang VM itself *is* written in C. > >> Can you write k/v store which >> is able do 200 millions of look up operations per second on one 2.4GHz >> i5 core? > > Something is getting lost here. I go back to the promises a language makes. > In delivering on those promises, priorities have to be set and compromises > made. The language user has to weight these. > > C/C++ gives us pointers (amongst other things). The language makes great > sacrifices to give us pointers (like garbage collection and debugability) so > we can create insanely efficient data structures. When the benefits of > pointers far outweigh the trade-offs, the choice is clear. > > Erlang gives us high concurrency (amongst other things). The Erlang team > made great sacrifices to give us insanely efficient and scalable > concurrency. When the benefits of this far outweigh the trade-offs the > choice is clear. > > I see few areas where these benefits/trade-offs overlap. The choice is > usually very clear. In cases where overlapping does occur, there are various > ways of using both. > > - Edmond - > > I would not argue using only one language for everything. Did I? As I have written slight down in mine reply, I'm surprised by Erlang HiPE compiler work. It performs surprisingly well and I would not expect many orders of magnitude improvement by rewriting to C. I would not say it would be even ten times unless I would try, but I bet it would, if I pay enough effort. I would argue that Erlang with NIFs gives you ultimate tool. Erlang and C are not enemies, they don't fight themselves, they makes prefect pair for almost all purposes. > > On Sun, 14 Nov 2010 22:32:42 +1100, Hynek Vychodil > wrote: > >> On Sun, Nov 14, 2010 at 11:55 AM, Tony Rogvall wrote: >>> >>> On 14 nov 2010, at 10.17, Hynek Vychodil wrote: >>> >>> >>> >>> Good idea but it seems that native (HiPE) compiler does this >>> optimization for you when you keep staying in one module. (I also >>> keeps exporting only main function. It can possibly take effect here >>> also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not >>> right tool for this job. You should use NIF if performance matter. >>> >>> pythag4(N) when is_integer(N) -> pythag4(N,1). >>> >>> I have implemented several small Erlang programs that beat "native" C >>> code. >>> Most of the time the C programs where badly/hastily written, but that is >>> the >>> point ;-) >>> You must select the algorithm and the implementation wisely, and of >>> course >>> use the golden rule "Make it work, then make it fast". I would add, if it >>> is >>> needed. >>> This does not imply that you should write your program unnecessary slow >>> to >>> start with! >>> Any how. A couple of months ago I implemented the AKS algorithm in >>> Erlang. >>> The AKS algorithm is a deterministic primality test algorithm >>> (http://en.wikipedia.org/wiki/AKS_primality_test) >>> I benchmarked this implementation with an implementation in C++. >>> I was shocked: The Erlang version was about 5 times faster, NOT using >>> obvious parallelism. >>> In this case I would suspect that garbage collection is the major >>> contributor! The implementation use >>> a lot of temporary polynomials intermediate results. A copy collector >>> does >>> the trick. >>> /Tony >>> >>> >>> >>> "Have run Make so many times I dunno what's installed anymore" >>> >> >> When Erlang implementation is faster than C implementation *is* badly >> written even it can be for maintainability or haste reason. C++ is >> another beast. Pun intended. I have similar experience to you but when >> I found that Erlang implementation is faster then I would look how is >> it possible and you are right, usually it is memory issue. Anyway >> every time you can do same tricks as Erlang does but you will end up >> with "half and error prone implementation of Erlang". You can also do >> tricks in C which you can't in Erlang. Can you write k/v store which >> is able do 200 millions of look up operations per second on one 2.4GHz >> i5 core? Anyway HiPE compiler does very good work here. If I count it >> correctly pythag3 and pythag4 does about hundred millions checks per >> seconds. Very good result I think. >> >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From ebegumisa@REDACTED Tue Nov 16 15:59:27 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 17 Nov 2010 01:59:27 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: Hello Hynek, > I can do > all this cool stuff as Erlang does and I can overperform Erlang using > its own weapons I can't :) > Everything what I can do in Erlang I can do in C, principally. > Opposite is not true. Keep in mind that Erlang VM itself *is* written > in C. For me, the opposite is true. That settles it then! You're a gazillion times a better C programmer than I am :) And probably far better than any I've personally met! Light-weight concurrency, SMP parallelism, soft real-time properties, fault-tolerance, hot-code swapping, distribution, etc. My skill-set is nowhere near wide enough to implement those things. Actually, from *my* C code, those things are damn near impossible! It's quite magical the way Erlang brings them to life. And I'd need those things to be able to say anything I can do in Erlang I can do in C, principally or otherwise. My C is just not that good. My knowledge is just not that wide. I rely too heavily on the combined effort of others who are much smarter than me. But I do play a mean blues guitar though! Take that, Erlang team :) - Edmond - On Tue, 16 Nov 2010 23:34:51 +1100, Hynek Vychodil wrote: > On Sun, Nov 14, 2010 at 5:50 PM, Edmond Begumisa > wrote: >> Hynek, >> >> A few comments... >> >>> When Erlang implementation is faster than C implementation *is* badly >>> written even it can be for maintainability or haste reason. C++ is >>> another beast >> >> I disagree. Quick example... >> >> 1. Write a "well-written" web-server in Erlang, the "Erlang way" >> (lightweight-weight concurrency, message passing). >> 2. Write a "well-written" web-server in C/C++, the "C/C++ way" >> (mulit-threading, shared-memory). >> >> See which is faster. Or, you could save yourself some time and look at >> Apache + Yaws. I wouldn't describe either of these as badly written. >> But the >> performance difference is undeniable. >> > > Why I should write C/C++ web-server in "C/C++ way" (mulit-threading, > shared-memory)? Is dumbness included in using C? I can write C > web-server using libevent for example. I can use kernel poll. I can do > all this cool stuff as Erlang does and I can overperform Erlang using > its own weapons. I can do it better and tune for this particular case > and will be better. I can but if I should or would is another > question. Keep in mind that Erlang VM itself is written in C. > >>> You can also do tricks in C which you can't in Erlang. >> >> And you can do tricks in Erlang that you can't do in typical C (e.g easy >> parallelism with little change to code as evidenced on this thread.) > > Everything what I can do in Erlang I can do in C, principally. > Opposite is not true. Keep in mind that Erlang VM itself *is* written > in C. >> >>> Can you write k/v store which >>> is able do 200 millions of look up operations per second on one 2.4GHz >>> i5 core? >> >> Something is getting lost here. I go back to the promises a language >> makes. >> In delivering on those promises, priorities have to be set and >> compromises >> made. The language user has to weight these. >> >> C/C++ gives us pointers (amongst other things). The language makes great >> sacrifices to give us pointers (like garbage collection and >> debugability) so >> we can create insanely efficient data structures. When the benefits of >> pointers far outweigh the trade-offs, the choice is clear. >> >> Erlang gives us high concurrency (amongst other things). The Erlang team >> made great sacrifices to give us insanely efficient and scalable >> concurrency. When the benefits of this far outweigh the trade-offs the >> choice is clear. >> >> I see few areas where these benefits/trade-offs overlap. The choice is >> usually very clear. In cases where overlapping does occur, there are >> various >> ways of using both. >> >> - Edmond - >> >> > I would not argue using only one language for everything. Did I? As I > have written slight down in mine reply, I'm surprised by Erlang HiPE > compiler work. It performs surprisingly well and I would not expect > many orders of magnitude improvement by rewriting to C. I would not > say it would be even ten times unless I would try, but I bet it would, > if I pay enough effort. > > I would argue that Erlang with NIFs gives you ultimate tool. Erlang > and C are not enemies, they don't fight themselves, they makes prefect > pair for almost all purposes. > >> >> On Sun, 14 Nov 2010 22:32:42 +1100, Hynek Vychodil >> wrote: >> >>> On Sun, Nov 14, 2010 at 11:55 AM, Tony Rogvall wrote: >>>> >>>> On 14 nov 2010, at 10.17, Hynek Vychodil wrote: >>>> >>>> >>>> >>>> Good idea but it seems that native (HiPE) compiler does this >>>> optimization for you when you keep staying in one module. (I also >>>> keeps exporting only main function. It can possibly take effect here >>>> also.) In BEAM it gives you 50% performance gain. Anyway Erlang is not >>>> right tool for this job. You should use NIF if performance matter. >>>> >>>> pythag4(N) when is_integer(N) -> pythag4(N,1). >>>> >>>> I have implemented several small Erlang programs that beat "native" C >>>> code. >>>> Most of the time the C programs where badly/hastily written, but that >>>> is >>>> the >>>> point ;-) >>>> You must select the algorithm and the implementation wisely, and of >>>> course >>>> use the golden rule "Make it work, then make it fast". I would add, >>>> if it >>>> is >>>> needed. >>>> This does not imply that you should write your program unnecessary >>>> slow >>>> to >>>> start with! >>>> Any how. A couple of months ago I implemented the AKS algorithm in >>>> Erlang. >>>> The AKS algorithm is a deterministic primality test algorithm >>>> (http://en.wikipedia.org/wiki/AKS_primality_test) >>>> I benchmarked this implementation with an implementation in C++. >>>> I was shocked: The Erlang version was about 5 times faster, NOT using >>>> obvious parallelism. >>>> In this case I would suspect that garbage collection is the major >>>> contributor! The implementation use >>>> a lot of temporary polynomials intermediate results. A copy collector >>>> does >>>> the trick. >>>> /Tony >>>> >>>> >>>> >>>> "Have run Make so many times I dunno what's installed anymore" >>>> >>> >>> When Erlang implementation is faster than C implementation *is* badly >>> written even it can be for maintainability or haste reason. C++ is >>> another beast. Pun intended. I have similar experience to you but when >>> I found that Erlang implementation is faster then I would look how is >>> it possible and you are right, usually it is memory issue. Anyway >>> every time you can do same tricks as Erlang does but you will end up >>> with "half and error prone implementation of Erlang". You can also do >>> tricks in C which you can't in Erlang. Can you write k/v store which >>> is able do 200 millions of look up operations per second on one 2.4GHz >>> i5 core? Anyway HiPE compiler does very good work here. If I count it >>> correctly pythag3 and pythag4 does about hundred millions checks per >>> seconds. Very good result I think. >>> >>> >> >> >> -- >> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> > > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From fred.hebert@REDACTED Tue Nov 16 16:21:00 2010 From: fred.hebert@REDACTED (=?iso-8859-1?Q?Fr=E9d=E9ric_Trottier-H=E9bert?=) Date: Tue, 16 Nov 2010 10:21:00 -0500 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: <090F6B51-470F-4076-9DAC-586732A3C3B1@erlang-solutions.com> On 2010-11-16, at 09:51 AM, Evans, Matthew wrote: > Interesting thread...I think we can all take the following from this: > > 1) Write your solution in the "Erlang way", using Erlang's strengths such as spawning parallel processes, also HIPE and NIFS where appropriate. Spawning parallel processes is, in my opinion, a false ideal when it comes to algorithms like the ones presented here. In a micro-benchmark or in software that is oriented on doing only this task, multiprocessing might gain you points. In a loaded production system where all the processors are likely already busy, I wouldn't be surprised to see no big gain by using multiple processes, and if any, the gain would likely be very hard to predict. I think it's better to aim for the scaling kind of concurrency, not the speed-up kind. If you use processes for truly concurrent tasks (rather than attempting speedups of a single call), You'll benefit from concurrency on a higher level on a server than from each function individually. This has to be pretty application specific, though. > 2) Write your solution efficiently - true in all languages, but especially in Erlang where you could end up doing needless recursion. > > 3) Erlang isn't always the best language for all problems. Neither is C, C++, C#, Java or any other language. Choose the language, or a mix of languages, that are the best fit your problem. > > What I would like to add is that we are trying to get Erlang accepted in a very C/C++ centric development environment. Although time and time again Erlang has shown itself a superior language and development environment for our domain, there is still a big problem in people accepting it. One of the main stumbling blocks is it is perceived to be "slow". People always say "it won't be faster than a C/C++ solution", or "a language written in a VM will always be slow [rolleyes]". > > We can argue until we are blue in the face that it will be as good or better when you throw multi-cores in the mix. That there are other advantages including shorter development time, better debugging, redundancy, failure detection etc. But overcoming that initial "speed" bias is a tough sell. I'm not saying there is an answer to this, and I know that the development team is doing their best to make Erlang faster, but we mustn't forget that for many, the "perceived" slowness is one factor that prevents them developing in Erlang. > I think the speed obsession has to be remnants of an education always based on time efficiency, especially for programmers coming from the era where processors were really slow. Whether this explanation is right or not, I don't think it is easily possible to convince someone that 1. his priorities are not the right one and 2. the language he uses isn't the best choice for the priorities he has. We should consider that the burden of proof is ours, not theirs. Any unsubstantiated claim is usually met with skepticism and possibly violence. This is completely normal and natural, in my opinion. The best way to convince someone is with irrefutable proof, which might be why the most common suggested way to get a team to accept Erlang is to provide them with an implementation or success stories. And maybe Erlang is no better than what they're using right now, in which case it would be a waste of money to train everyone to work in a new environment with little benefit in the long run. If people still are not accepting it even after valid proofs, then all bets are off: maybe they don't feel like learning new technology, maybe they don't believe that *they* can write efficient Erlang code (and then you have to prove them they can), maybe they think it all comes down to the same anyway (project success isn't language dependent? which could be true in many cases), etc. I'm not sure much can be done except leading by example, in any case. > /Rant over > > Matt > -- Fred H?bert http://www.erlang-solutions.com From anders.nygren@REDACTED Tue Nov 16 16:23:10 2010 From: anders.nygren@REDACTED (Anders Nygren) Date: Tue, 16 Nov 2010 09:23:10 -0600 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <4CE29343.30406@cs.ntua.gr> References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE29343.30406@cs.ntua.gr> Message-ID: On Tue, Nov 16, 2010 at 8:20 AM, Kostis Sagonas wrote: > Richard O'Keefe wrote: >> >> ... SNIP >> >> The time for p0 was 420 milliseconds in Erlang. >> On the same machine, the same function, the time was >> 91 milliseconds in SML/NJ. >> ? ? ? ? ? ?SML/NJ ?MLton ? Erlang >> n=100 L= 34 s=0.004 m=0.004 e=0.020 >> n=200 L= 86 s=0.028 m=0.027 e=0.130 >> n=300 L=146 s=0.090 m=0.086 e=0.410 >> n=400 L=210 s=0.213 m=0.202 e=0.980 >> n=500 L=274 s=0.415 m=0.394 e=1.900 >> >> We see that Erlang is about 5 times slower than SML. > > Is Erlang running native code here? > (Because both SML/NJ and MLton are.) > >> Here's the SML code. >> >> ... ?What's more, SML is >> strongly typed, and the Dialyzer should be able to handle this, >> so the compiler *knows* it is dealing with integers. > > Let's leave dialyzer out of this discussion -- it is a type analyzer > alright, but the types it infers are not necessarily usable for optimization > purposes (I can possibly elaborate in some other mail). > > Can you post p0 to see whether the compiler can infer whether the program > deals with integers or not? > > >> There are two basic issues here: >> (1) The Erlang compiler doesn't really handle list comprehension as >> ? ?native loops; it compiles list comprehensions to recursive >> ? ?functions. ?lists:seq(1, N) is not special syntax -- unlike Haskell -- >> ? ?and almost surely builds an actual list. > > Yes. ?This is an issue. ?Actually, it's more general: in my experience most > Erlang programmers do not understand the difference between list > comprehensions and lists:foreach/2. Many of them find list comprehension > notation so nice that they often use it in places where building the list is > not only is not their intention but is completely unnecessary. > The compiler is not sufficiently clever to perform deforestation and avoid > construction of intermediate lists. Maybe You are talking about some special case here that I missed but the "Efficiency Guide" ch 5.2, in the documentation says "In R12B, if the result of the list comprehension will obviously not be used, a list will not be constructed." /Anders > >> ? ?This could be changed, and maybe some day it will change. >> >> (2) Erlang is compiled incrementally to support hot loading and >> ? ?(again in order to support hot loading) does not do much if any >> ? ?cross-module inlining. >> ? ?MLton is a whole-program optimising compiler, it can SML _are_ >> ? ?allowed to do cross-module inlining by the nature of the language. >> >> ? ?This *can't* be changed without making Erlang less useful for its job. >> >> Ah, but there's one more wrinkle! > > Yes, there is that too in Erlang ;-) > >> ? ? ? ? ? ?SML/NJ ?Erlang ?SML/NJ ? ? Smalltalk >> n=100 L= 34 s=0.004 e=0.020 s= 0.250 ? p1=0.016 >> n=200 L= 86 s=0.028 e=0.130 s= 1.904 ? p1=0.109 >> n=300 L=146 s=0.090 e=0.410 s= 6.567 ? p1=0.369 >> n=400 L=210 s=0.213 e=0.980 s=15.581 ? p1=0.873 >> n=500 L=274 s=0.415 e=1.900 s=30.218 ? p1=1.705 >> (3) Erlang integer arithmetic is *not* limited to some ridiculous >> ? ?size like 32 bits, it is _semantically_ unbounded integer >> ? ?arithmetic, with a space-efficient representation and fast >> ? ?calculation path for small integers. ?To get something >> ? ?comparable in SML, you have to use the type IntInf.int >> ? ?instead of Int.int. ?And when you do *that*, Erlang really >> ? ?really shines. >> >> ? ?The difference here is that if you use int in C or Java or C# >> ? ?or SML, then you'll find a positive integer n such that n+1 < 0. >> ? ?That kind of nonsense simply doesn't happen in Erlang. >> >> ? ?It comes with a price, but the price of *not* using a Lispy >> ? ?(or Smalltalky, or Erlangy) approach to integers is the price >> ? ?of getting wrong answers without knowing. >> >> ? ?The Smalltalk column was done using my Smalltalk-via-C compiler. >> ? ?It might be possible to go a factor of 2 faster using machine- >> ? ?specific assembly code, but what we seem to be looking at in >> ? ?the Erlang column really does seem to be no more than the >> ? ?price of arithmetic that gives correct results. > > Thanks for a nice post! > > Kostis > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From gregory.haskins@REDACTED Tue Nov 16 16:31:47 2010 From: gregory.haskins@REDACTED (Gregory Haskins) Date: Tue, 16 Nov 2010 10:31:47 -0500 Subject: Problems with gen_udp on multi-homed hosts Message-ID: <4CE2A3E3.7020803@gmail.com> Hi All, I am trying to develop a custom DHCP server against 14B on that is sitting on a multi-homed linux host. I only want the server to listen/respond on one of these interfaces, but I am having trouble getting gen_udp to behave in a way I would expect. My system has the following interfaces: 3> inet:getiflist(). {ok,["lo","br0","virbr0"]} 4> inet:ifget("virbr0", [addr]). {ok,[{addr,{192,168,122,1}}]} 5> inet:ifget("br0", [addr]). {ok,[{addr,{151,155,230,24}}]} and I am interested in packets only on 192.168.122.1/virbr0. So if I gen_udp:open() _without_ {ip, ?ADDR} set, I can see the broadcasts coming from 192.168.122.0/24 (as well as from the other interfaces) as I would expect. However, the {udp, Socket, ..} message I get has a inet:sockname() of {0.0.0.0}, and if I try to reply with a broadcast of {255.255.255.255} to the same socket, it goes out on the "br0" interface. So the obvious solution to me after googling was to use the {ip, {192.168.122.1}} option on the open. The open() call succeeds with this option set, but I never see any packets (even though I have sniffed the line and they are indeed being sent. As a sanity check, I also set a bogus IP in the open(), and it does indeed exception out, so it seems like it is taking the option I set. I am running out of ideas outside of abandoning gen_udp() in favor of a ports driver, and/or not writing the DHCP portion of my code in Erlang. Neither is particularly attractive, especially given that this seems like a fairly straight forward sockets app and I would think it could be made to work. Thoughts? -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 267 bytes Desc: OpenPGP digital signature URL: From ebegumisa@REDACTED Tue Nov 16 16:34:28 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 17 Nov 2010 02:34:28 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <4CE29343.30406@cs.ntua.gr> References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE29343.30406@cs.ntua.gr> Message-ID: On Wed, 17 Nov 2010 01:20:51 +1100, Kostis Sagonas wrote: > There are two basic issues here: > (1) The Erlang compiler doesn't really handle list comprehension as > native loops; it compiles list comprehensions to recursive > functions. lists:seq(1, N) is not special syntax -- unlike Haskell > -- > and almost surely builds an actual list. > Yes. This is an issue. Actually, it's more general: in my experience > most Erlang programmers do not understand the difference between list > comprehensions and lists:foreach/2. Many of them find list comprehension > notation so nice that they often use it in places where building the > list is not only is not their intention but is completely unnecessary. > The compiler is not sufficiently clever to perform deforestation and > avoid construction of intermediate lists. > First I'm hearing of this. It really ought to be in one of those green boxes in the documentation. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From gregory.haskins@REDACTED Tue Nov 16 16:47:57 2010 From: gregory.haskins@REDACTED (Gregory Haskins) Date: Tue, 16 Nov 2010 10:47:57 -0500 Subject: Problems with gen_udp on multi-homed hosts In-Reply-To: <4CE2A3E3.7020803@gmail.com> References: <4CE2A3E3.7020803@gmail.com> Message-ID: <4CE2A7AD.6010000@gmail.com> On 11/16/10 10:31 AM, Gregory Haskins wrote: > > So the obvious solution to me after googling was to use the {ip, > {192.168.122.1}} option on the open. The open() call succeeds with this > option set, but I never see any packets (even though I have sniffed the > line and they are indeed being sent. To be clear: The problem I am describing here is that, once {ip, ..} is set, I don't seem to _receive_ any packets on the socket, even though wireshark shows the packets are indeed floating around the L2 network. -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 267 bytes Desc: OpenPGP digital signature URL: From alessandro.sivieri@REDACTED Tue Nov 16 17:00:55 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Tue, 16 Nov 2010 17:00:55 +0100 Subject: Dynamic code loading and hotswap behaviour Message-ID: Hi all, I'm using the function code:load_binary in an application, for dynamically loading bytecode; I know that the hotswap mechanisms of Erlang support at most two versions of the same module, and if a third is loaded, the oldest is truncated and any process using its functions is terminated: now, the dynamic loading does some kind of version check of a module? I mean, if I load the same version three times, do the hotswap mechanisms apply anyway or does the implementation recognize that the code is the same, so there is no need to reload it? -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From rapsey@REDACTED Tue Nov 16 17:12:33 2010 From: rapsey@REDACTED (Rapsey) Date: Tue, 16 Nov 2010 17:12:33 +0100 Subject: [erlang-questions] Dynamic code loading and hotswap behaviour In-Reply-To: References: Message-ID: It reloads anyway and the mechanisms apply. Sergej On Tue, Nov 16, 2010 at 5:00 PM, Alessandro Sivieri < alessandro.sivieri@REDACTED> wrote: > Hi all, > > I'm using the function code:load_binary in an application, for dynamically > loading bytecode; I know that the hotswap mechanisms of Erlang support at > most two versions of the same module, and if a third is loaded, the oldest > is truncated and any process using its functions is terminated: now, the > dynamic loading does some kind of version check of a module? I mean, if I > load the same version three times, do the hotswap mechanisms apply anyway > or > does the implementation recognize that the code is the same, so there is no > need to reload it? > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://www.chimera-bellerofonte.eu/ > http://www.poul.org/ > From alessandro.sivieri@REDACTED Tue Nov 16 17:33:43 2010 From: alessandro.sivieri@REDACTED (Alessandro Sivieri) Date: Tue, 16 Nov 2010 17:33:43 +0100 Subject: [erlang-questions] Dynamic code loading and hotswap behaviour In-Reply-To: References: Message-ID: 2010/11/16 Rapsey > It reloads anyway and the mechanisms apply. > > And is there a motivation for this? I mean, it seems that reloading the same module more times can be seen as an error from a developer point of view, if this happens there is something wrong in my code, but anyway the runtime should understand that nothing has changed... -- Sivieri Alessandro alessandro.sivieri@REDACTED http://www.chimera-bellerofonte.eu/ http://www.poul.org/ From michael.santos@REDACTED Tue Nov 16 17:48:21 2010 From: michael.santos@REDACTED (Michael Santos) Date: Tue, 16 Nov 2010 11:48:21 -0500 Subject: [erlang-questions] Problems with gen_udp on multi-homed hosts In-Reply-To: <4CE2A3E3.7020803@gmail.com> References: <4CE2A3E3.7020803@gmail.com> Message-ID: <20101116164821.GA27596@ecn.lan> On Tue, Nov 16, 2010 at 10:31:47AM -0500, Gregory Haskins wrote: > Hi All, > > I am trying to develop a custom DHCP server against 14B on that is > sitting on a multi-homed linux host. I only want the server to > listen/respond on one of these interfaces, but I am having trouble > getting gen_udp to behave in a way I would expect. > > My system has the following interfaces: > > 3> inet:getiflist(). > {ok,["lo","br0","virbr0"]} > 4> inet:ifget("virbr0", [addr]). > {ok,[{addr,{192,168,122,1}}]} > 5> inet:ifget("br0", [addr]). > {ok,[{addr,{151,155,230,24}}]} > > and I am interested in packets only on 192.168.122.1/virbr0. > > So if I gen_udp:open() _without_ {ip, ?ADDR} set, I can see the > broadcasts coming from 192.168.122.0/24 (as well as from the other > interfaces) as I would expect. However, the {udp, Socket, ..} message I > get has a inet:sockname() of {0.0.0.0}, and if I try to reply with a > broadcast of {255.255.255.255} to the same socket, it goes out on the > "br0" interface. Does it work if you send the packet out using a more specific broadcast address? e.g., 192.168.122.255 > So the obvious solution to me after googling was to use the {ip, > {192.168.122.1}} option on the open. The open() call succeeds with this > option set, but I never see any packets (even though I have sniffed the > line and they are indeed being sent. As a sanity check, I also set a > bogus IP in the open(), and it does indeed exception out, so it seems > like it is taking the option I set. Not sure why this isn't working, maybe the packet is still going out your default interface. > I am running out of ideas outside of abandoning gen_udp() in favor of a > ports driver, and/or not writing the DHCP portion of my code in Erlang. > Neither is particularly attractive, especially given that this seems > like a fairly straight forward sockets app and I would think it could be > made to work. On Linux, you can try binding the socket to an interface [1]. klaar added support to do this for his quite interesting dhcp server, edhcpd [2]. [1] https://github.com/msantos/procket [2] https://github.com/klaar/edhcpd From lemenkov@REDACTED Tue Nov 16 18:51:15 2010 From: lemenkov@REDACTED (Peter Lemenkov) Date: Tue, 16 Nov 2010 20:51:15 +0300 Subject: Heads up, RHEL6 users. Erlang R14B in EPEL 6 Message-ID: Hello All. I plan to upgrade erlang in EPEL 6 (external repository for RHEL and derivatives) from R13B-04 to R14B in a couple of days. So if you have any objections, wishes or just a random thoughts regarding erlang in EPEL, then it's time to share them with us. -- With best regards, Peter Lemenkov. From gregory.haskins@REDACTED Tue Nov 16 19:03:18 2010 From: gregory.haskins@REDACTED (Gregory Haskins) Date: Tue, 16 Nov 2010 13:03:18 -0500 Subject: [erlang-questions] Problems with gen_udp on multi-homed hosts In-Reply-To: <20101116164821.GA27596@ecn.lan> References: <4CE2A3E3.7020803@gmail.com> <20101116164821.GA27596@ecn.lan> Message-ID: <4CE2C766.9@gmail.com> On 11/16/10 11:48 AM, Michael Santos wrote: > On Tue, Nov 16, 2010 at 10:31:47AM -0500, Gregory Haskins wrote: >> Hi All, >> >> I am trying to develop a custom DHCP server against 14B on that is >> sitting on a multi-homed linux host. I only want the server to >> listen/respond on one of these interfaces, but I am having trouble >> getting gen_udp to behave in a way I would expect. >> >> My system has the following interfaces: >> >> 3> inet:getiflist(). >> {ok,["lo","br0","virbr0"]} >> 4> inet:ifget("virbr0", [addr]). >> {ok,[{addr,{192,168,122,1}}]} >> 5> inet:ifget("br0", [addr]). >> {ok,[{addr,{151,155,230,24}}]} >> >> and I am interested in packets only on 192.168.122.1/virbr0. >> >> So if I gen_udp:open() _without_ {ip, ?ADDR} set, I can see the >> broadcasts coming from 192.168.122.0/24 (as well as from the other >> interfaces) as I would expect. However, the {udp, Socket, ..} message I >> get has a inet:sockname() of {0.0.0.0}, and if I try to reply with a >> broadcast of {255.255.255.255} to the same socket, it goes out on the >> "br0" interface. > > Does it work if you send the packet out using a more specific broadcast > address? e.g., 192.168.122.255 Hmm, good question. I could try this, though I think the procket reference you provide below is probably a better solution. > >> So the obvious solution to me after googling was to use the {ip, >> {192.168.122.1}} option on the open. The open() call succeeds with this >> option set, but I never see any packets (even though I have sniffed the >> line and they are indeed being sent. As a sanity check, I also set a >> bogus IP in the open(), and it does indeed exception out, so it seems >> like it is taking the option I set. > > Not sure why this isn't working, maybe the packet is still going out > your default interface. Well, the problem I was describing was on the RX side. I don't seem to get any packets once the {ip, ..} option is set (even though my client is basically making requests every few seconds regardless of the server-side mode, and I can confirm I see the packets on the wire via wireshark) > >> I am running out of ideas outside of abandoning gen_udp() in favor of a >> ports driver, and/or not writing the DHCP portion of my code in Erlang. >> Neither is particularly attractive, especially given that this seems >> like a fairly straight forward sockets app and I would think it could be >> made to work. > > On Linux, you can try binding the socket to an interface [1]. Very cool. I will take a look at procket. > klaar added > support to do this for his quite interesting dhcp server, edhcpd [2]. Cool, I wasnt aware of this project either. I was using code from http://code.google.com/p/erlang-dhcp-server as a base, but its fairly old now. Perhaps I should be looking at klaar's code instead, anyway. You mentioned he is doing some interesting things? Out of curiosity, can you elaborate on what makes it interesting to you? I noticed that he is using gproc and mnesia (vs dets in the project referenced above) so I assume at the very least this might be some kind of clustered DHCPd impl? Anything else interesting going on? Thanks Michael, I appreciate the feedback and pointers. Kind Regards, -Greg > > [1] https://github.com/msantos/procket > [2] https://github.com/klaar/edhcpd > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 267 bytes Desc: OpenPGP digital signature URL: From michael.santos@REDACTED Tue Nov 16 20:09:39 2010 From: michael.santos@REDACTED (Michael Santos) Date: Tue, 16 Nov 2010 14:09:39 -0500 Subject: [erlang-questions] Problems with gen_udp on multi-homed hosts In-Reply-To: <4CE2C766.9@gmail.com> References: <4CE2A3E3.7020803@gmail.com> <20101116164821.GA27596@ecn.lan> <4CE2C766.9@gmail.com> Message-ID: <20101116190939.GA28061@ecn.lan> On Tue, Nov 16, 2010 at 01:03:18PM -0500, Gregory Haskins wrote: > >> So the obvious solution to me after googling was to use the {ip, > >> {192.168.122.1}} option on the open. The open() call succeeds with this > >> option set, but I never see any packets (even though I have sniffed the > >> line and they are indeed being sent. As a sanity check, I also set a > >> bogus IP in the open(), and it does indeed exception out, so it seems > >> like it is taking the option I set. > > > > Not sure why this isn't working, maybe the packet is still going out > > your default interface. > > Well, the problem I was describing was on the RX side. I don't seem to > get any packets once the {ip, ..} option is set (even though my client > is basically making requests every few seconds regardless of the > server-side mode, and I can confirm I see the packets on the wire via > wireshark) Ok. Maybe confirm in wireshark that the MAC address for the packet being sent is the one you expect (assuming virbr0 isn't a physical interface). > >> I am running out of ideas outside of abandoning gen_udp() in favor of a > >> ports driver, and/or not writing the DHCP portion of my code in Erlang. > >> Neither is particularly attractive, especially given that this seems > >> like a fairly straight forward sockets app and I would think it could be > >> made to work. > > > > On Linux, you can try binding the socket to an interface [1]. > > Very cool. I will take a look at procket. > > > klaar added > > support to do this for his quite interesting dhcp server, edhcpd [2]. > > Cool, I wasnt aware of this project either. I was using code from > http://code.google.com/p/erlang-dhcp-server as a base, but its fairly > old now. Perhaps I should be looking at klaar's code instead, anyway. > > You mentioned he is doing some interesting things? Out of curiosity, > can you elaborate on what makes it interesting to you? I noticed that > he is using gproc and mnesia (vs dets in the project referenced above) > so I assume at the very least this might be some kind of clustered DHCPd > impl? Anything else interesting going on? To be honest, I haven't had a chance to look too closely at it, so mainly the idea of an Erlang dhcp server :) From tom@REDACTED Tue Nov 16 21:36:49 2010 From: tom@REDACTED (tom@REDACTED) Date: Tue, 16 Nov 2010 21:36:49 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101116125803.GA27248@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <20101116125803.GA27248@ecn.lan> Message-ID: <201011162136.50447.tom@diogunix.com> Hello Michael, thaniks for helping :-) > > ejabberd Start: > > > > # ejabberdctl status > > {error_logger,{{2010,11,16},{9,56,20}},"Protocol: ~p: register error: > > ~p~n", > > ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, > > When a distributed Erlang node starts up, it attempts to register itself > with an epmd daemon listening on localhost on port 4369 and starts one, > if an epmd is not running. I have a process running by unix user "ejabberd", yes: # ps axu ejabberd 2355 0.0 0.1 3448 1284 ?? SJ 5:54AM 0:00.14 /usr/local/lib/erlang/erts-5.8.1/bin/epmd -daemon I guess, this process was started by the ejabberdctl skript when it tried to launch ejabberd. EPMD tries to connect to "localhost" ? Then we are on the right track now I guess FreeBSD jails have only sort of a "half own localhost" and thus one has to configure the Jails IP address instead of localhost / 127.0.0.1 instead (what usually works for all other daemons). Where can I configure Erlang to use the Jails IP instead localhost/127.0.0.1 ? In that mysterious "inetrc" config file ? However, there must be a way as I earlier had it running already in another test host (half a year ago). Just cannot remember the details on how I did it. > Check epmd is running and if the node is allowed to contact it from the > jail. You can get debugging info by running epmd manually: epmd -d -d -d Ok, did that as # kill -9 2355 # /usr/local/lib/erlang/erts-5.8.1/bin/epmd -d -d -d epmd: Tue Nov 16 20:29:59 2010: epmd running - daemon = 0 epmd: Tue Nov 16 20:29:59 2010: try to initiate listening port 4369 epmd: Tue Nov 16 20:29:59 2010: starting epmd: Tue Nov 16 20:29:59 2010: entering the main select() loop epmd: Tue Nov 16 20:30:04 2010: time in seconds: 1289939404 epmd: Tue Nov 16 20:30:09 2010: time in seconds: 1289939409 epmd: Tue Nov 16 20:30:14 2010: time in seconds: 1289939414 epmd: Tue Nov 16 20:30:20 2010: time in seconds: 1289939420 epmd: Tue Nov 16 20:30:25 2010: time in seconds: 1289939425 epmd: Tue Nov 16 20:30:30 2010: time in seconds: 1289939430 epmd: Tue Nov 16 20:30:35 2010: time in seconds: 1289939435 epmd: Tue Nov 16 20:30:40 2010: time in seconds: 1289939440 ... ... ^C What could be the next step to test/check ? kind regards Tom From tom@REDACTED Tue Nov 16 22:08:47 2010 From: tom@REDACTED (tom@REDACTED) Date: Tue, 16 Nov 2010 22:08:47 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <201011162136.50447.tom@diogunix.com> References: <201011161114.01488.tom@diogunix.com> <20101116125803.GA27248@ecn.lan> <201011162136.50447.tom@diogunix.com> Message-ID: <201011162208.47981.tom@diogunix.com> Just an additional thought: > > > ejabberd Start: > > > > > > # ejabberdctl status > > > {error_logger,{{2010,11,16},{9,56,20}},"Protocol: ~p: register error: > > > ~p~n", > > > ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, is it sure the error message above means the connection between erlang and localhost:4369 ? I also built and configured ejabberd to use MySQL and may be the connection between erlangs mysql driver and mysql could be meant ? MySQL however is happily running and already in use by Postfix and Dovecot in the same jail ... kind regards Tom From michael.santos@REDACTED Wed Nov 17 00:22:02 2010 From: michael.santos@REDACTED (Michael Santos) Date: Tue, 16 Nov 2010 18:22:02 -0500 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <201011162136.50447.tom@diogunix.com> References: <201011161114.01488.tom@diogunix.com> <20101116125803.GA27248@ecn.lan> <201011162136.50447.tom@diogunix.com> Message-ID: <20101116232202.GB28061@ecn.lan> On Tue, Nov 16, 2010 at 09:36:49PM +0100, tom@REDACTED wrote: > I have a process running by unix user "ejabberd", yes: > # ps axu > ejabberd 2355 0.0 0.1 3448 1284 ?? SJ 5:54AM 0:00.14 > /usr/local/lib/erlang/erts-5.8.1/bin/epmd -daemon > > I guess, this process was started by the ejabberdctl skript when it tried to > launch ejabberd. > > EPMD tries to connect to "localhost" ? The Erlang node connects to epmd (on 127.0.0.1:4369). The TCP connection is working, but the socket is closed immmediately. > Then we are on the right track now I > guess FreeBSD jails have only sort of a "half own localhost" and thus one > has to configure the Jails IP address instead of localhost / 127.0.0.1 > instead (what usually works for all other daemons). Which version of Erlang are you using? R14B? epmd in R14B was changed to allow some messages (like name registrations) only from 127/8. > Where can I configure Erlang to use the Jails IP instead localhost/127.0.0.1 > ? In that mysterious "inetrc" config file ? inetrc is used for hostname resolution. See: http://www.erlang.org/doc/apps/erts/inet_cfg.html > However, there must be a way as I earlier had it running already in another > test host (half a year ago). Just cannot remember the details on how I did > it. > > > Check epmd is running and if the node is allowed to contact it from the > > jail. You can get debugging info by running epmd manually: epmd -d -d -d > > Ok, did that as > # kill -9 2355 > # /usr/local/lib/erlang/erts-5.8.1/bin/epmd -d -d -d > epmd: Tue Nov 16 20:29:59 2010: epmd running - daemon = 0 > epmd: Tue Nov 16 20:29:59 2010: try to initiate listening port 4369 > epmd: Tue Nov 16 20:29:59 2010: starting > epmd: Tue Nov 16 20:29:59 2010: entering the main select() loop > epmd: Tue Nov 16 20:30:04 2010: time in seconds: 1289939404 > epmd: Tue Nov 16 20:30:09 2010: time in seconds: 1289939409 > epmd: Tue Nov 16 20:30:14 2010: time in seconds: 1289939414 > epmd: Tue Nov 16 20:30:20 2010: time in seconds: 1289939420 > epmd: Tue Nov 16 20:30:25 2010: time in seconds: 1289939425 > epmd: Tue Nov 16 20:30:30 2010: time in seconds: 1289939430 > epmd: Tue Nov 16 20:30:35 2010: time in seconds: 1289939435 > epmd: Tue Nov 16 20:30:40 2010: time in seconds: 1289939440 > ... > ... > ^C Was epmd started up inside the jail? Did you bring up ejabberd as well? I don't see any registration attempts. > ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, > {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, > is it sure the error message above means the connection between erlang > and localhost:4369 ? Just a guess :) Here is the error message: {badmatch,{error,epmd_close}} When the ejabberd node starts up, it connects to 127.0.0.1:4369 and sends a EPMD_ALIVE2_REQ message to register its name and distribution port (an ephemeral port). The code that does this is in inet_tcp_dist:listen/1 which calls erl_epmd:register_node/2. {ok, Creation} = erl_epmd:register_node(Name, Port) epmd closes the connection immediately (possibly because the EPMD_ALIVE2_REQ message is not allowed) and register_node/2 returns {error, epmd_close}, causing the badmatch. You can get the same error by starting up 2 erlang nodes (kill epmd if it is running): $ erl Erlang R14B01 (erts-5.8.2) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.8.2 (abort with ^G) 1> {ok, L} = gen_tcp:listen(4369, [{active, false}]). 2> {ok, S} = gen_tcp:accept(L), ok = gen_tcp:close(S). And in another shell: $ erl -name test {error_logger,{{2010,11,16},{18,12,6}},"Protocol: ~p: register error: ~p~n",["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1},{net_kernel,start_protos,4},{net_kernel,start_protos,3},{net_kernel,init_node,2},{net_kernel,init,1},{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}]} From tom@REDACTED Wed Nov 17 00:45:34 2010 From: tom@REDACTED (tom@REDACTED) Date: Wed, 17 Nov 2010 00:45:34 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101116232202.GB28061@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011162136.50447.tom@diogunix.com> <20101116232202.GB28061@ecn.lan> Message-ID: <201011170045.34477.tom@diogunix.com> Hello, just as a quick feedback: > Which version of Erlang are you using? R14B? > > epmd in R14B was changed to allow some messages (like name registrations) > only from 127/8. as unix user "ejabberd": $ erl Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.8.1 (abort with ^G) So, does that mean, there's only a problem with this certain version ? Then we had the explanation why it worked with an older version half a year ago but not with the current version. > > Where can I configure Erlang to use the Jails IP instead > > localhost/127.0.0.1 > > > > ? In that mysterious "inetrc" config file ? > > inetrc is used for hostname resolution. See: > > http://www.erlang.org/doc/apps/erts/inet_cfg.html thank you very much, will immediately digg into this and then check back to you. > Was epmd started up inside the jail? Did you bring up ejabberd as well? > I don't see any registration attempts. Everything happened in this Jail only: - building erlang, ejabberd, erlang-mysql driver - starting ejabberd (trying to start it) - all other command lines and resuling output I sent to this list so far On the localhost pre-configuration of erlang/epmd: Do I have an option to change that behaviour and let erlang/epmd try to connect to the actual IP ? This wouldn't harm anything as anyway there's a tight firewall protecting access from outside the jail. May be the answer on this question will come out from reading the inetrc links you sent me but I did not already go through it. Will do that tonight. kind regards Tom From tom@REDACTED Wed Nov 17 02:12:25 2010 From: tom@REDACTED (tom@REDACTED) Date: Wed, 17 Nov 2010 02:12:25 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101116232202.GB28061@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011162136.50447.tom@diogunix.com> <20101116232202.GB28061@ecn.lan> Message-ID: <201011170212.25565.tom@diogunix.com> Hello Michael, another suggestion on what could have caused the issue and how to resolv it. The idea came into my mind while reading http://www.erlang.org/doc/apps/erts/inet_cfg.html and when remembering there was another person having a similar problem, which that person could resolve by adding a proper hostname to it's FreeBSD jail. Well, I of course HAVE a proper hostname configuration for this jail. But may be, epmd just cannot find the hostname as it - by default - does not use the regular methods ? This might be an explanation why the connection at first was established but then closed immediately. if true, the issue almost probably can be resolved by just editing the inetrc file and make it use the correct methods for finding the hostname. Ok, all this is wild speculation by a non erlang insider but however, here's the standard inetrc file as delivered by FreeBSD's Ports Collection: # cat inetrc.example {lookup,["file","native"]}. {host,{127,0,0,1}, ["localhost","hostalias"]}. {file, resolv, "/etc/resolv.conf"}. This example-inetrc might be working for a standard FreeBSD host but may be needs to get adapted for a FreeBSD jail. I'm just unsure on how to change the values there (as I do not understand one word of Erlang and am not sure I got it right from your link). kind regards Tom On Wednesday 17 November 2010 00:22:02 Michael Santos wrote: > On Tue, Nov 16, 2010 at 09:36:49PM +0100, tom@REDACTED wrote: > > I have a process running by unix user "ejabberd", yes: > > # ps axu > > ejabberd 2355 0.0 0.1 3448 1284 ?? SJ 5:54AM 0:00.14 > > /usr/local/lib/erlang/erts-5.8.1/bin/epmd -daemon > > > > I guess, this process was started by the ejabberdctl skript when it > > tried to launch ejabberd. > > > > EPMD tries to connect to "localhost" ? > > The Erlang node connects to epmd (on 127.0.0.1:4369). The TCP connection > is working, but the socket is closed immmediately. > > > Then we are on the right track now I > > guess FreeBSD jails have only sort of a "half own localhost" and thus > > one has to configure the Jails IP address instead of localhost / > > 127.0.0.1 instead (what usually works for all other daemons). > > Which version of Erlang are you using? R14B? > > epmd in R14B was changed to allow some messages (like name registrations) > only from 127/8. > > > Where can I configure Erlang to use the Jails IP instead > > localhost/127.0.0.1 > > > > ? In that mysterious "inetrc" config file ? > > inetrc is used for hostname resolution. See: > > http://www.erlang.org/doc/apps/erts/inet_cfg.html > > > However, there must be a way as I earlier had it running already in > > another test host (half a year ago). Just cannot remember the details > > on how I did it. > > > > > Check epmd is running and if the node is allowed to contact it from > > > the jail. You can get debugging info by running epmd manually: epmd > > > -d -d -d > > > > Ok, did that as > > # kill -9 2355 > > # /usr/local/lib/erlang/erts-5.8.1/bin/epmd -d -d -d > > epmd: Tue Nov 16 20:29:59 2010: epmd running - daemon = 0 > > epmd: Tue Nov 16 20:29:59 2010: try to initiate listening port 4369 > > epmd: Tue Nov 16 20:29:59 2010: starting > > epmd: Tue Nov 16 20:29:59 2010: entering the main select() loop > > epmd: Tue Nov 16 20:30:04 2010: time in seconds: 1289939404 > > epmd: Tue Nov 16 20:30:09 2010: time in seconds: 1289939409 > > epmd: Tue Nov 16 20:30:14 2010: time in seconds: 1289939414 > > epmd: Tue Nov 16 20:30:20 2010: time in seconds: 1289939420 > > epmd: Tue Nov 16 20:30:25 2010: time in seconds: 1289939425 > > epmd: Tue Nov 16 20:30:30 2010: time in seconds: 1289939430 > > epmd: Tue Nov 16 20:30:35 2010: time in seconds: 1289939435 > > epmd: Tue Nov 16 20:30:40 2010: time in seconds: 1289939440 > > ... > > ... > > ^C > > Was epmd started up inside the jail? Did you bring up ejabberd as well? > I don't see any registration attempts. > > > ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, > > {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, > > > > is it sure the error message above means the connection between erlang > > and localhost:4369 ? > > Just a guess :) > > Here is the error message: > > {badmatch,{error,epmd_close}} > > When the ejabberd node starts up, it connects to 127.0.0.1:4369 and sends > a EPMD_ALIVE2_REQ message to register its name and distribution port (an > ephemeral port). > > The code that does this is in inet_tcp_dist:listen/1 which calls > erl_epmd:register_node/2. > > {ok, Creation} = erl_epmd:register_node(Name, Port) > > epmd closes the connection immediately (possibly because the > EPMD_ALIVE2_REQ message is not allowed) and register_node/2 returns > {error, epmd_close}, causing the badmatch. > > You can get the same error by starting up 2 erlang nodes (kill epmd if > it is running): > > $ erl > Erlang R14B01 (erts-5.8.2) [source] [smp:2:2] [rq:2] [async-threads:0] > [hipe] [kernel-poll:false] > > Eshell V5.8.2 (abort with ^G) > 1> {ok, L} = gen_tcp:listen(4369, [{active, false}]). > 2> {ok, S} = gen_tcp:accept(L), ok = gen_tcp:close(S). > > And in another shell: > > $ erl -name test > {error_logger,{{2010,11,16},{18,12,6}},"Protocol: ~p: register error: > ~p~n",["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen, > 1},{net_kernel,start_protos,4},{net_kernel,start_protos,3},{net_kernel,in > it_node,2},{net_kernel,init,1},{gen_server,init_it,6},{proc_lib,init_p_do > _apply,3}]}]} > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From michael.santos@REDACTED Wed Nov 17 03:08:47 2010 From: michael.santos@REDACTED (Michael Santos) Date: Tue, 16 Nov 2010 21:08:47 -0500 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <201011170212.25565.tom@diogunix.com> References: <201011161114.01488.tom@diogunix.com> <201011162136.50447.tom@diogunix.com> <20101116232202.GB28061@ecn.lan> <201011170212.25565.tom@diogunix.com> Message-ID: <20101117020847.GA29358@ecn.lan> On Wed, Nov 17, 2010 at 02:12:25AM +0100, tom@REDACTED wrote: > Hello Michael, > > another suggestion on what could have caused the issue and how to resolv it. > The idea came into my mind while reading > http://www.erlang.org/doc/apps/erts/inet_cfg.html > and when remembering there was another person having a similar problem, > which that person could resolve by adding a proper hostname to it's FreeBSD > jail. It might help. I think the quickest way to debug this would be to: 1. Start up epmd manually in the jail with the debug switches epmd -d -d -d 2. Run tcpdump on port 4369 for the interfaces in the jail 3. Start up a distributed erlang node by hand: erl -name foo Report back on what you find! From ok@REDACTED Wed Nov 17 04:08:43 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 17 Nov 2010 16:08:43 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <4CE29343.30406@cs.ntua.gr> References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE29343.30406@cs.ntua.gr> Message-ID: On 17/11/2010, at 3:20 AM, Kostis Sagonas wrote: > Richard O'Keefe wrote: >> ... SNIP >> The time for p0 was 420 milliseconds in Erlang. >> On the same machine, the same function, the time was >> 91 milliseconds in SML/NJ. >> SML/NJ MLton Erlang >> n=100 L= 34 s=0.004 m=0.004 e=0.020 >> n=200 L= 86 s=0.028 m=0.027 e=0.130 >> n=300 L=146 s=0.090 m=0.086 e=0.410 >> n=400 L=210 s=0.213 m=0.202 e=0.980 >> n=500 L=274 s=0.415 m=0.394 e=1.900 >> We see that Erlang is about 5 times slower than SML. > > Is Erlang running native code here? > (Because both SML/NJ and MLton are.) Yes. > >> Here's the SML code. >> ... What's more, SML is >> strongly typed, and the Dialyzer should be able to handle this, >> so the compiler *knows* it is dealing with integers. > > Let's leave dialyzer out of this discussion -- it is a type analyzer alright, but the types it infers are not necessarily usable for optimization purposes (I can possibly elaborate in some other mail). Let's not leave dialyzer out just yet, because there's an important distinction. The dialyzer can discover that certain variables will have integer values, but that's not at all the same thing as discovering that they will have *small* integer values. But with arguments declared or inferred as 'int' in SML, small hardware-sized integers just the same as C is what you get. The best plausible case for Erlang here is code that still has to check arithmetic operations to see whether bignum arithmetic is needed, at run time. > > Can you post p0 to see whether the compiler can infer whether the program deals with integers or not? > It isn't pretty! p0(N) when is_integer(N), N >= 3 -> p0_A(N, N, []). p0_A(0, _, L) -> L; p0_A(A, N, L) -> p0_B(N, A, N, L). p0_B(0, A, N, L) -> p0_A(A-1, N, L); p0_B(B, A, N, L) -> p0_C(N, A, B, N, L). p0_C(0, A, B, N, L) -> p0_B(B-1, A, N, L); p0_C(C, A, B, N, L) when A+B+C =< N, A*A+B*B =:= C*C -> p0_C(C-1, A, B, N, [{A,B,C}|L]); p0_C(C, A, B, N, L) -> p0_C(C-1, A, B, N, L). It wouldn't take a *huge* amount of effort to make the existing compiler recognise "X <- lists:seq(Y, Z)" as a special case and generate code not entirely unlike this, except that I took the opportunity to count down to 0. Here's what p1 looks like in Smalltalk. p1 |r| r := OrderedCollection new. 1 to: self do: [:a | 1 to: self do: [:b | 1 to: self do: [:c | (a+b+c <= self and: [a squared + b squared = c squared]) ifTrue: [r addLast: {a. b. c}]]]]. ^r By tradition, Smalltalk compilers *do* treat _ to: _ do: _ as a special case, and mine is no exception, although the code is still sufficiently generic to work with any kind of number or even Dates, not just integers. I'll spare you all the generated C code, but here's the inner loop, cleaned up a bit. I'm including it to make a point. The lines flagged with ** are the lines that would are affected by (a) the lack of a static type system and (b) the possibility of bignums. The 'return is_not_boolean(...)' lines are there because you cannot rely on comparison operators like < always returning a Boolean. (The next big change to the compiler will be to deal with this another way.) The "i(...)" macros would be single hardware instructions in C, but the arguments might not be integers, and if integers, might be big, and if small, might lead to a big result. (Oh, #squared should have had a fast path too; I forgot.) #define ib_p(x, y) \ (AREINTS(x, y) ? BOXS(UNINT(x) + UNINT(y)) : b_p(GLOBALS_OUT x, y)) On a SPARC, BOXS(UNINT(x)+UNINT(y)) could be a single tadd, but I'm still trying to generate portable C, so there's a penalty there. The minimum at assembly level would be something like or x, y -> z and z, 3 -> z brnz z, L1 add x, y -> z boc L2 L1: call out_of_line_add L2: which is still a lot more than a simple add instruction. The thing I find astonishing is that it is fast enough to be *useful*. a4 /*c*/ = ASINT(1); /* c := 1 */ for (;;) { ** b4 = ib_l(self, a4); /* self < c */ if (b4 != FALSE) break; ** t4 = ib_p(a2 /*a*/, a3 /*b*/); /* t4 := a + b */ ** t3 = ib_p(t4, a4 /*c*/); /* t3 := (a+b) + c */ ** t2 = ib_le(t3, self); /* t2 := t3 <= self */ if (t2 == FALSE) { t1 = FALSE; } else if (t2 == TRUE) { ** t5 = u_squared(a2 /*a*/); ** t6 = u_squared(a3 /*b*/); ** t3 = ib_p(t5, t6); /* t3 := a**2 + b**2 */ ** t4 = u_squared(a4 /*c*/); ** t1 = ib_e(t3, t4); /* t1 := (a**2+b**2)=c**2 */ } else { ** return is_not_boolean(t2); } if (t1 == TRUE) { t3 = OBJECT_NEW(GCLASS(n_Array), ASINT(3)); ELEM(t3, 1) = a2 /*a*/; ELEM(t3, 2) = a3 /*b*/; ELEM(t3, 3) = a4 /*c*/; (void)k_addLast(l1 /*r*/, t3); } else ** if (t1 != FALSE) { ** return is_not_boolean(t1); } ** a4 = ib_p(a4, ASINT(1)); /* c := c + 1 */ } ** if (b4 != TRUE) return is_not_boolean(b4); H From tom@REDACTED Wed Nov 17 04:59:52 2010 From: tom@REDACTED (tom@REDACTED) Date: Wed, 17 Nov 2010 04:59:52 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101117020847.GA29358@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011170212.25565.tom@diogunix.com> <20101117020847.GA29358@ecn.lan> Message-ID: <201011170459.52991.tom@diogunix.com> > It might help. I think the quickest way to debug this would be to: donesults below ... > 1. Start up epmd manually in the jail with the debug switches > > epmd -d -d -d I did that already before but repeated it nonetheless. # epmd -d -d -d epmd: Wed Nov 17 03:27:57 2010: epmd running - daemon = 0 epmd: Wed Nov 17 03:27:57 2010: try to initiate listening port 4369 epmd: Wed Nov 17 03:27:57 2010: starting epmd: Wed Nov 17 03:27:57 2010: entering the main select() loop epmd: Wed Nov 17 03:28:02 2010: time in seconds: 1289964482 epmd: Wed Nov 17 03:28:07 2010: time in seconds: 1289964487 epmd: Wed Nov 17 03:28:12 2010: time in seconds: 1289964492 epmd: Wed Nov 17 03:28:17 2010: time in seconds: 1289964497 epmd: Wed Nov 17 03:28:22 2010: time in seconds: 1289964502 ^C You before mentioned, it did not even try to connect (can''t say anything about this). > 2. Run tcpdump on port 4369 for the interfaces in the jail # tcpdump -vv port 4369 tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes tcpdump doesn't get anything when listening on the em0 interface. So, I tried # tcpdump -i lo0 -vv port 4369 tcpdump: WARNING: lo0: no IPv4 address assigned tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 96 bytes > 3. Start up a distributed erlang node by hand: > > erl -name foo # erl -name foo {error_logger,{{2010,11,17},{3,33,15}},"Protocol: ~p: register error: ~p~n", ["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1}, {net_kernel,start_protos,4},{net_kernel,start_protos,3}, {net_kernel,init_node,2},{net_kernel,init,1},{gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}]} {error_logger,{{2010,11,17},{3,33,15}},crash_report,[[{initial_call, {net_kernel,init,['Argument__1']}},{pid,<0.19.0>},{registered_name,[]}, {error_info,{exit,{error,badarg},[{gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}},{ancestors,[net_sup,kernel_sup,<0.9.0>]}, {messages,[]},{links,[#Port<0.56>,<0.16.0>]},{dictionary, [{longnames,true}]},{trap_exit,true},{status,running},{heap_size,377}, {stack_size,24},{reductions,443}],[]]} {error_logger,{{2010,11,17},{3,33,15}},supervisor_report,[{supervisor, {local,net_sup}},{errorContext,start_error},{reason, {'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel}, {mfargs,{net_kernel,start_link,[[foo,longnames]]}},{restart_type,permanent}, {shutdown,2000},{child_type,worker}]}]} {error_logger,{{2010,11,17},{3,33,15}},supervisor_report,[{supervisor, {local,kernel_sup}},{errorContext,start_error},{reason,shutdown},{offender, [{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}}, {restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} {error_logger,{{2010,11,17},{3,33,15}},std_info,[{application,kernel}, {exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]} {"Kernel pid terminated",application_controller,"{application_start_failure,kernel, {shutdown,{kernel,start,[normal,[]]}}}"} Crash dump was written to: erl_crash.dump Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}}) and tcpdump output: 03:33:15.964680 IP (tos 0x0, ttl 64, id 24389, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->4f37)!) mail.kepos.org.10975 > mail.kepos.org.4369: Flags [S], cksum 0xd692 (correct), seq 3123827793, win 65535, options [mss 16344,nop,wscale 3,sackOK,TS val 77847542 ecr 0], length 0 03:33:15.964701 IP (tos 0x0, ttl 64, id 24390, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->4f36)!) mail.kepos.org.4369 > mail.kepos.org.10975: Flags [S.], cksum 0x776a (correct), seq 3252621940, ack 3123827794, win 65535, options [mss 16344,nop,wscale 3,sackOK,TS val 2304704868 ecr 77847542], length 0 03:33:15.964712 IP (tos 0x0, ttl 64, id 24391, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->4f3d)!) mail.kepos.org.10975 > mail.kepos.org.4369: Flags [.], cksum 0xbd56 (correct), seq 1, ack 1, win 8960, options [nop,nop,TS val 77847542 ecr 2304704868], length 0 03:33:15.964740 IP (tos 0x0, ttl 64, id 24392, offset 0, flags [DF], proto TCP (6), length 70, bad cksum 0 (->4f2a)!) mail.kepos.org.10975 > mail.kepos.org.4369: Flags [P.], cksum 0x6619 (correct), seq 1:19, ack 1, win 8960, options [nop,nop,TS val 77847542 ecr 2304704868], length 18 03:33:15.964896 IP (tos 0x0, ttl 64, id 24393, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->4f3b)!) mail.kepos.org.4369 > mail.kepos.org.10975: Flags [F.], cksum 0xbd43 (correct), seq 1, ack 19, win 8960, options [nop,nop,TS val 2304704868 ecr 77847542], length 0 03:33:15.964906 IP (tos 0x0, ttl 64, id 24394, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->4f3a)!) mail.kepos.org.10975 > mail.kepos.org.4369: Flags [.], cksum 0xbd43 (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 ecr 2304704868], length 0 03:33:15.964929 IP (tos 0x0, ttl 64, id 24395, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->4f39)!) mail.kepos.org.10975 > mail.kepos.org.4369: Flags [F.], cksum 0xbd42 (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 ecr 2304704868], length 0 03:33:15.964937 IP (tos 0x0, ttl 64, id 24396, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->4f38)!) mail.kepos.org.4369 > mail.kepos.org.10975: Flags [.], cksum 0xbd43 (correct), seq 2, ack 20, win 8959, options [nop,nop,TS val 2304704868 ecr 77847542], length 0 ^C 8 packets captured 41 packets received by filter 0 packets dropped by kernel Juust to exde any misconfiguation on the machine I allso tried itt again switching off the firewall for this test: same result. > Report back on what you find! I fear there's no real enlightenment from this - at least not for me. We already knew that we cannot make full use of the loopback interface regarding the virtualization situation in a FreeBSD jail. Also, I still do not know what actually it is, Erlang does not like with it's environment. Nonetheless there must be a way to get it up as there are people out there capable to do it. I'd like to again ask whether there was no way to make Erlang talk with the actual IP instead off talking to the localhost. And stll I'm unsure on how to deal with the inetrc file. What I read from your link (Michael) I find a bit unsual and confusing. kind regards Tom From matthias@REDACTED Wed Nov 17 05:44:21 2010 From: matthias@REDACTED (Matthias Lang) Date: Wed, 17 Nov 2010 05:44:21 +0100 Subject: [erlang-questions] EOF to external program In-Reply-To: References: Message-ID: <20101117044418.GA2173@corelatus.se> Hi, None of the people who know a definite answer have answered. One likely reason for ports being the way they are is that whoever first implemented them intended them for running external programs specifically written to work with Erlang. I.e. the general case of running Erlang-oblivious code wasn't considered, or was considered as something to be done through a wrapper, or was thrown on the "worry about that later" pile. And nobody since then has been both sufficiently annoyed and sufficiently skilled to remove that restriction. Using 'netcat' (my suggestion) is a pretty desperate workaround, i.e. it works, but it's not pretty. A proper fix would be to give ports a similar interface to gen_tcp in {active, false} mode, something like: {ok, Port} = os_process:open("wc /dev/zero", [binary, {active, false}]). {ok, Bin} = os_process:recv(Port, 0). {ok, Last_bin, Exit_value} = os_process:close(Port). it probably wouldn't hurt supporting {active, once} too. Matt ---------------------------------------------------------------------- On Friday, November 05, Timo Lindemann wrote: > Hello erlangers, > > I like to use external programs, so I thought I could use ports a lot. I > fail to understand one detail about ports: Why can one not close the write > end (such that the connected external program registers an EOF on its stdin > descriptor)? > > I dug around the web some, and found that this has stumped other people in > the past, with suggestions about using netcat to work around that problem, > to writing a wrapper that parses the data for some marker the process would > send to the external program signalling EOF and have the wrapper close the > write end. > > Why is there NO way in erlang to let a program know the input's data has > ended *and* fetching its response *and* getting its (normal) exit status? > port_close/1 will normally instantly kill the external program. > > Omitting this feature severely limits the things one can do with ports. awk > will not process its END clause, wc will never exit, ... if there's no EOF > to be found. There's a lot of programs waiting for stdin to close. > > So please, what's the reason for not allowing this? Will this be added > sometime later on? Or have I just not found a way to do it? > > All the best > -T. > > -- > Timo Lindemann > Software Systems Engineer From ok@REDACTED Wed Nov 17 06:12:29 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 17 Nov 2010 18:12:29 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: On 17/11/2010, at 3:51 AM, Evans, Matthew wrote: > Interesting thread...I think we can all take the following from this: > > 1) Write your solution in the "Erlang way", using Erlang's strengths such as spawning parallel processes, also HIPE and NIFS where appropriate. I think this is important for any language: you really do need to learn how to work *with* the language. We are *all* of us having to learn new ways of programming. I've got quite good at seeing how to make single-core programs go faster. Seeing the parallelism in a problem is still a challenge for me. Let's not forget that one of Erlang's strengths is distribution. Moving a computation where the data is has been a good idea (sometimes!) for a long time. Moving a computation is NOT necessarily the same thing as moving executable code; it could be some sort of data structure requesting a particular activity, which might or might not result in new code being generated, compiled, and loaded on a remote server. > 2) Write your solution efficiently - true in all languages, but especially in Erlang where you could end up doing needless recursion. "needless recursion"? There is nothing special about needless *recursion* to warrant distinguishing it from any other kind of needless work. I'm not sure that I go along with this. When I presented the A and B determine C version of the Pythagorean triple finder, I *knew* that there were more efficient ways to do it. The code was efficient *enough* to make my point. So let's say "Don't make your solution blatantly inefficient". People often focus on computational steps when thinking about efficiency. Memory is also something to think about. Let's take that 100 million element data set I mentioned earlier in this thread. It's basically just a rather large sparse matrix with 500,000 rows 20,000 columns 100,000,000 nonzero entries (they fit in a byte). Store it as a full array, and you need 10 GB of memory. The machine I'm typing on today has 3 GB. Oops. One way to represent a sparse matrix is as an array of pointers to pairs of (column number, value) arrays, which works out at about 303 MB of memory. The method I'm using takes about 210 MB of memory. It's not just the quantitive difference, taking >47 times less memory, it's the qualitative difference between being able to do the job in the memory I have or not. By the way, the same data structure could be adapted to Erlang, taking about 434 MB. I haven't tried it, and it would take a fair bit longer to load, but it should work. If done, this would provide an example of Erlang + efficient data structure : slowish but workable C + inefficient data structure : unusable In much the same way, I once had a statistical calculation where I wanted to use a published Fortran algorithm. I had trouble getting the published code to work, so I rewrote it in Prolog. When I had that going, I used it to help debug the Fortran version. And at the end of the day, the Prolog version was *faster*, because it used a better data structure. And the Prolog code wasn't even running as native instructions! (To this day, rewriting something tricky in another language is a debugging strategy I find helpful.) > 3) Erlang isn't always the best language for all problems. Neither is C, C++, C#, Java or any other language. Choose the language, or a mix of languages, that are the best fit your problem. It's also important to be clear about what's *problematic* about the problem. *Development* time and *execution* time are different. If you are trying to develop embedded software for something where the (moral) cost of failure is high, like a medical device, the SPARK subset of Ada, with an actual attempt to verify as much as you can of the software, would be a good choice. If you are trying to enumerate Pythagorean triples for fun, that would be a bad choice. In some projects, what you need to do is to move quickly from prototype to prototype as you learn more about the problem, the domain, the risks, the clients, &c. > What I would like to add is that we are trying to get Erlang accepted in a very C/C++ centric development environment. Although time and time again Erlang has shown itself a superior language and development environment for our domain, there is still a big problem in people accepting it. One of the main stumbling blocks is it is perceived to be "slow". People always say "it won't be faster than a C/C++ solution", or "a language written in a VM will always be slow [rolleyes]". Well, Erlang *is* slow at arithmetic. The question is whether that really matters. Let's put this in perspective. About 12 years ago I was an expert witness in a court case. Amongst other things, the contractor claimed that the reason the program was too slow was that the client's PC was too feeble. If memory serves me, the PC was one that the contractor had recommended when the project began. I pointed out that the whole of the banking in this country had been done 20 years earlier on a mainframe with far less memory, far slower, and with somewhat slower discs, so the machine really ought to be able to handle one shop! Today's machines are awesome. My laptop can sort 250 million 64-bit floats in under a minute (albeit not using qsort()). I was about to try a bigger benchmark when I realised that was 2GB. I used to be able to get times like that with a few thousand numbers. We now see people _happily_ developing entire applications in (sorry; I don't like to be offensive, but it's true) PHP. We see *major* natural language processing frameworks like GATE written in Java, and project management tools like Colibri. People will tell you that Java can be, or is about to be, or even is, as fast as C. Every benchmark I've ever tried, some of them quite recently, says NO. But it's fast _enough_. > > We can argue until we are blue in the face that it will be as good or better when you throw multi-cores in the mix. I'm sorry, but I don't believe that Erlang will *ever* run faster than C. What we need to remember is that *most* large projects fail. The question we should be looking at is not "will a [multicore] Erlang program be faster" "will a [multicore] C program be faster" but "will a [multicore] Erlang program be DELIVERABLE" "will a [multicore] C program be DELIVERABLE" An Erlang program that *exists* is infinitely faster than a C program that had to be abandoned. A slow Erlang program that customers can actually *use* is going to earn you more money (that you can spend on rewriting parts of it in C if you really want to) than a C program that isn't ready for release outside the Dilbert zone. You know, a lot of people think that their program is efficient *because it is written in C*. But there is a lot of inefficient code written in C. I'm sure we've all seen things like for (i = 0; i < strlen(s); i++) if (s[i] == ' ') s[i] = '_'; It might well be that the best way to overcome the "speed" bias would be to look at some recently written code and show how fast it *isn't*. A colleague here once challenged me to improve a back-propagation program he had written in C. Thinking like an APL programmer I speeded it up by a factor of 2.5. That code was actually a half-way house to using the BLAS, but since I'd made my point, I stopped there. > for many, the "perceived" slowness is one factor that prevents them developing in Erlang. Just like the way the "perceived" slowness of garbage collection stopped people using Lisp, until along came Java, and suddenly GC was respectable. Yet despite that, and despite experiments finding faster development in Lisp *and* better performance in the result, people still stayed away from Lisp. Wearing my complete cynic's hat here, I suspect that if someone developed an alternative syntax for Erlang that *looked* like C, it might be more attractive to the hard-of-thinking. From ok@REDACTED Wed Nov 17 06:23:07 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 17 Nov 2010 18:23:07 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE29343.30406@cs.ntua.gr> Message-ID: On 17/11/2010, at 4:23 AM, Anders Nygren wrote: > On Tue, Nov 16, 2010 at 8:20 AM, Kostis Sagonas wrote: >> Richard O'Keefe wrote: >>> >>> ... SNIP >>> >>> The time for p0 was 420 milliseconds in Erlang. >>> On the same machine, the same function, the time was >>> 91 milliseconds in SML/NJ. >>> SML/NJ MLton Erlang >>> n=100 L= 34 s=0.004 m=0.004 e=0.020 >>> n=200 L= 86 s=0.028 m=0.027 e=0.130 >>> n=300 L=146 s=0.090 m=0.086 e=0.410 >>> n=400 L=210 s=0.213 m=0.202 e=0.980 >>> n=500 L=274 s=0.415 m=0.394 e=1.900 >>> >>> We see that Erlang is about 5 times slower than SML. >> >> Is Erlang running native code here? >> (Because both SML/NJ and MLton are.) >> >>> Here's the SML code. >>> >>> ... What's more, SML is >>> strongly typed, and the Dialyzer should be able to handle this, >>> so the compiler *knows* it is dealing with integers. >> >> Let's leave dialyzer out of this discussion -- it is a type analyzer >> alright, but the types it infers are not necessarily usable for optimization >> purposes (I can possibly elaborate in some other mail). >> >> Can you post p0 to see whether the compiler can infer whether the program >> deals with integers or not? >> >> >>> There are two basic issues here: >>> (1) The Erlang compiler doesn't really handle list comprehension as >>> native loops; it compiles list comprehensions to recursive >>> functions. lists:seq(1, N) is not special syntax -- unlike Haskell -- >>> and almost surely builds an actual list. >> >> Yes. This is an issue. Actually, it's more general: in my experience most >> Erlang programmers do not understand the difference between list >> comprehensions and lists:foreach/2. Many of them find list comprehension >> notation so nice that they often use it in places where building the list is >> not only is not their intention but is completely unnecessary. >> The compiler is not sufficiently clever to perform deforestation and avoid >> construction of intermediate lists. > > > Maybe You are talking about some special case here that I missed but the > "Efficiency Guide" ch 5.2, in the documentation says > "In R12B, if the result of the list comprehension will obviously not > be used, a list will not be constructed." We are talking about completely different lists. What I'm talking about is that R = [f(X) || X <- lists:seq(1, N)] will construct lists:seq(1, N), even though it doesn't need to. You are talking about the fact that _ = [g(Y) || Y <- L] won't construct a list of g results. But that is not at all relevant to the problem at hand, where the list of triples obviously *WILL* be used, so the comprehension *MUST* construct it. That's fine, that's what we want. Compare this with Haskell, where for [(a,b,c) | a <- [1..n], b <- [1..n], c <- [1..n], a+b+c <= n, a^2 + b^2 == c^2] some compilers *won't* construct the lists [1..n]. From ok@REDACTED Wed Nov 17 06:45:09 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 17 Nov 2010 18:45:09 +1300 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE29343.30406@cs.ntua.gr> Message-ID: <8EDD1083-51A8-46BA-854E-D143520095A5@cs.otago.ac.nz> On 17/11/2010, at 4:34 AM, Edmond Begumisa wrote: > On Wed, 17 Nov 2010 01:20:51 +1100, Kostis Sagonas wrote: > >> There are two basic issues here: >> (1) The Erlang compiler doesn't really handle list comprehension as >> native loops; it compiles list comprehensions to recursive >> functions. lists:seq(1, N) is not special syntax -- unlike Haskell -- >> and almost surely builds an actual list. >> Yes. This is an issue. Actually, it's more general: in my experience most Erlang programmers do not understand the difference between list comprehensions and lists:foreach/2. Many of them find list comprehension notation so nice that they often use it in places where building the list is not only is not their intention but is completely unnecessary. >> The compiler is not sufficiently clever to perform deforestation and avoid construction of intermediate lists. >> > > First I'm hearing of this. It really ought to be in one of those green boxes in the documentation. Let's go around this one more time. The current Erlang compiler turns [f(X) || X <- lists:seq(1, N)] into (lists:seq(1, N), ...) where is a function generated by the compiler. The compiler *could* make a special case of lists:seq/[2,3] and generate (1, N, ...) But it doesn't. The actual definition of lists:seq/2 not only *could* change, in the not too distant past it *has* changed. The compiler can't inline the definition and use whether it is now, because it's a remote call (even if -imported), so it might change at run time. Haskell compilers and the Mercury compiler *do* perform deforestation because they are given an unchanging program to work with and can safely do cross-module inlining. Erlang can't. This would be a potentially serious problem if iterating over integer ranges were something Erlang spent a lot of time doing. I just checked an Erlang/OTP release I keep handy for such things. ~ 1,500,000 raw lines ~ 950,000 SLOW 34 list:seq iterations 18 in two EUNIT tests 3 in other test cases (at least) 13 list:seq not in an ?_assert ~ 3200 Pat <- Exp The unit tests don't matter for performance. We're left with about one SLOC in every 72,000 being an iteration of this kind, or about <- in every 250 being an integer iteration. The Erlang/OTP team probably feel they have other maddened grizzly bears to stun before they bother with something that seems to turn up mainly in toy benchmarks, and who could blame them? From toanhuyle.vn@REDACTED Wed Nov 17 08:47:58 2010 From: toanhuyle.vn@REDACTED (Toan H. Le) Date: Wed, 17 Nov 2010 14:47:58 +0700 Subject: Questions of configuring on SUSE Linux Enterprise 11 Message-ID: Hi, I try to compile Erlang R14B on SUSE 11 but I face errors of configuring. This is the output on screen ********************************************************************* ********************** APPLICATIONS DISABLED ********************** ********************************************************************* crypto : No usable OpenSSL found jinterface : No Java compiler found ssh : No usable OpenSSL found ssl : No usable OpenSSL found ********************************************************************* ********************************************************************* ********************** APPLICATIONS INFORMATION ******************* ********************************************************************* wx : wxWidgets not found, wx will NOT be usable ********************************************************************* ********************************************************************* ********************** DOCUMENTATION INFORMATION ****************** ********************************************************************* documentation : fop is missing. Using fakefop to generate placeholder PDF files. ********************************************************************* 1. APPLICATIONS DISABLED I don't install JDK, so I agree with error of jinterface. I trace upward and find that the following warning may affect the crypto, ssh, and ssl ---- checking for OpenSSL >= 0.9.7 in standard locations... no configure: WARNING: No (usable) OpenSSL found, skipping ssl, ssh and crypto applications ---- I did check with yast; two packages are install openssl (0.9.8h) and libopenssl0_9_8. I think those two are enough to use OpenSSL. Do I need something to bypass error of openssl? 2. APPLICATIONS INFORMATION I did check again with yast and package wxGTK is installed. I am not sure if it is enough or not. If I need to install any package, please tell me. 3. DOCUMENTATION INFORMATION Again, can anyone tell me what is the fop and do I need it? Thanks for all your help. -- Regards, Toan H. Le From ingela@REDACTED Wed Nov 17 09:14:10 2010 From: ingela@REDACTED (Ingela Andin) Date: Wed, 17 Nov 2010 09:14:10 +0100 Subject: [erlang-questions] Re: SSL and client authentication In-Reply-To: References: <1289750585.2064.ezmlm@erlang.org> Message-ID: Hi! It is possible and easy to upgrade an existing connection to a ssl-connection if the peers agreee to do so using the erlang ssl-API. Regards Ingela Erlang/OTP-team, Ericsson AB 2010/11/15 Saager Mhatre : > Shouldn't it be trivial to kick off another https process to do the ssl > handoff for various urls? > > Sent from my Motorola Milestone. > > On Nov 15, 2010 3:51 PM, "Alessandro Sivieri" > wrote: > > 2010/11/15 > > >> However, the SSL handshake *has* occured by the time the response is sent >> back, so it shouldn't... > This is what I thought, too; Apache and Nginx allow a user to configure > which Web paths have to be mutually authenticated and which not, even if > both use HTTPS... > > > >> Since the OP has decided to use nginx, there's not much point in going >> into the mechanics of h... > So there is a way to do this in (pure) Erlang? > > > -- > Sivieri Alessandro > alessandro.sivieri@REDACTED > http://www.chimera-bellerofonte.eu/ > http://www.... > From dkoch@REDACTED Wed Nov 17 09:41:20 2010 From: dkoch@REDACTED (dkoch@REDACTED) Date: Wed, 17 Nov 2010 09:41:20 +0100 (CET) Subject: Erlang pipermail or base retrieving ? Message-ID: <19429266.83511289983280092.JavaMail.www@wsfrf1136> Hello dear Erlang staff, I'd like to import the full KB that has grown in the mailing list, yet I cannot fetch mails before I suscribed to the lists. However the old mailing list database (up to May 2009) exists in a packed and archived .GZ form here : www.erlang.org\pipermail\eeps www.erlang.org\pipermail\erlang-announce www.erlang.org\pipermail\erlang-bugs www.erlang.org\pipermail\erlang-patches www.erlang.org\pipermail\erlang-questions Question : since the abandon of the previous mailing list model (pipermail) can the ezmlm-based mails be packed as previously on a regular basis so that I could fetch and include them manually into my mail client (Opera) to ease browsing them off-line ? Thanks, your dearly, David "Kochise" KOCH From attila.r.nohl@REDACTED Wed Nov 17 10:09:55 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Wed, 17 Nov 2010 10:09:55 +0100 Subject: [erlang-questions] Questions of configuring on SUSE Linux Enterprise 11 In-Reply-To: References: Message-ID: 2010/11/17, Toan H. Le : [...] > I trace upward and find that the following warning may affect the crypto, > ssh, and ssl > ---- > checking for OpenSSL >= 0.9.7 in standard locations... no > configure: WARNING: No (usable) OpenSSL found, skipping ssl, ssh and crypto > applications > ---- > I did check with yast; two packages are install openssl (0.9.8h) and > libopenssl0_9_8. I think those two are enough to use OpenSSL. > > Do I need something to bypass error of openssl? Probably the openssl-devel package. From raimo+erlang-questions@REDACTED Wed Nov 17 10:47:19 2010 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 17 Nov 2010 10:47:19 +0100 Subject: [erlang-questions] What's a solid way to have erlang automatically send email. In-Reply-To: <201011162137.39132.als@iinet.net.au> References: <201011162137.39132.als@iinet.net.au> Message-ID: <20101117094719.GA9948@erix.ericsson.se> On Tue, Nov 16, 2010 at 09:37:39PM +1100, Anthony Shipman wrote: > On Mon, 15 Nov 2010 10:31:09 am G.S. wrote: > > Hello, > > > > I have a website where the administrator needs to receive email alert (to > > login and fix a problem) when certain events occur. Is there an Erlang > > module that allows one to easily send out an email? Or something that can > > be used in the same manner? > > > > Thanks, > > -Gene > > On UNIX, you could feed the text into stdin of /usr/sbin/sendmail > or /usr/bin/mutt. erlang:open_port() will let you do this. Here is a part of a larger (escript) program that does that. I have made a perhaps overflexible cmd/2,3 that returns a lazy list which may look like overengineering in this case, but makes sense in the larger program. %% Send a completely formatted mail with all %% headers through the sendmail command. sendmail(Text) -> {_,Port,Cont} = cmd( ["sendmail","/usr/sbin/sendmail", "/usr/lib/sendmail","/usr/libexec/sendmail"], ["-t","-i"]), erlang:port_command(Port, Text), result(Cont). result([_|Cont]) -> result(Cont); result(Fun) when is_function(Fun) -> result(Fun()); result(Result) -> Result. %% Generic execute command returning a lazy list %% -> {[Command|Args],Port,Cont} %% %% Cont() -> [Line, ... | Cont] | [] %% Cont() -> ExitStatus cmd(Cmds, Args) -> cmd(Cmds, Args, []). %% cmd([Cmd|Cmds], Args, Opts) -> case os:find_executable(Cmd) of false -> case Cmds of [] -> erlang:error({not_found_executable,Cmd}); _ -> cmd(Cmds, Args, Opts) end; Command -> cmd_open(Command, Args, Opts) end. cmd_open(Command, Args, Opts) -> cmd_open_port(Command, {spawn_executable,Command}, [{args,Args}|Opts]). cmd_open_port(Command, PortName, Opts) -> Port = erlang:open_port(PortName, [{line,80},exit_status,eof,hide|Opts]), {Command,Port,cmd_cont(Port, [], undefined, false)}. cmd_cont(Port, Buf, Exit, Eof) -> fun () -> cmd_loop(Port, Buf, Exit, Eof) end. cmd_loop(Port, Buf, Exit, Eof) -> receive {Port,{data,{Flag,Line}}} -> case Flag of noeol -> cmd_loop(Port, lists:reverse(Line, Buf), Exit, Eof); eol -> Data = lists:reverse(Buf, Line), [Data|cmd_cont(Port, [], Exit, Eof)] end; {Port,{exit_status,Status}} when Exit =:= undefined -> case Eof of true -> cmd_close(Port, Buf, Status); false -> cmd_loop(Port, Buf, Status, Eof) end; {Port,eof} when Eof =:= false -> case Exit of undefined -> cmd_loop(Port, Buf, Exit, true); Status -> cmd_close(Port, Buf, Status) end; {Port,_} = Unexpected -> io:format("Unexpected from port: ~p~n~n", [Unexpected]), cmd_loop(Port, Buf, Exit, Eof) end. cmd_close(Port, Buf, Status) -> catch port_close(Port), case Buf of [] -> Status; Buf -> [lists:reverse(Buf)|Status] end. > > -- > Anthony Shipman Mamas don't let your babies > als@REDACTED grow up to be outsourced. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From tony@REDACTED Wed Nov 17 09:49:04 2010 From: tony@REDACTED (Tony Rogvall) Date: Wed, 17 Nov 2010 09:49:04 +0100 Subject: [erlang-questions] EOF to external program In-Reply-To: <20101117044418.GA2173@corelatus.se> References: <20101117044418.GA2173@corelatus.se> Message-ID: <118EC037-5F76-4C1A-8AC5-59AFDCD19803@rogvall.se> Hi Matthias! This should probably be turned into a EEP, "active mode support for spawn driver" Any takers ;-) Why not add packet_parser support while at it? I definitely vote for this (initial) suggestion. /Tony On 17 nov 2010, at 05.44, Matthias Lang wrote: > Hi, > > None of the people who know a definite answer have answered. > > One likely reason for ports being the way they are is that whoever > first implemented them intended them for running external programs > specifically written to work with Erlang. I.e. the general case of > running Erlang-oblivious code wasn't considered, or was considered as > something to be done through a wrapper, or was thrown on the "worry > about that later" pile. And nobody since then has been both > sufficiently annoyed and sufficiently skilled to remove that > restriction. > > Using 'netcat' (my suggestion) is a pretty desperate workaround, i.e. > it works, but it's not pretty. > > A proper fix would be to give ports a similar interface to gen_tcp in > {active, false} mode, something like: > > {ok, Port} = os_process:open("wc /dev/zero", [binary, {active, false}]). > {ok, Bin} = os_process:recv(Port, 0). > {ok, Last_bin, Exit_value} = os_process:close(Port). > > it probably wouldn't hurt supporting {active, once} too. > > Matt > > ---------------------------------------------------------------------- > > On Friday, November 05, Timo Lindemann wrote: >> Hello erlangers, >> >> I like to use external programs, so I thought I could use ports a lot. I >> fail to understand one detail about ports: Why can one not close the write >> end (such that the connected external program registers an EOF on its stdin >> descriptor)? >> >> I dug around the web some, and found that this has stumped other people in >> the past, with suggestions about using netcat to work around that problem, >> to writing a wrapper that parses the data for some marker the process would >> send to the external program signalling EOF and have the wrapper close the >> write end. >> >> Why is there NO way in erlang to let a program know the input's data has >> ended *and* fetching its response *and* getting its (normal) exit status? >> port_close/1 will normally instantly kill the external program. >> >> Omitting this feature severely limits the things one can do with ports. awk >> will not process its END clause, wc will never exit, ... if there's no EOF >> to be found. There's a lot of programs waiting for stdin to close. >> >> So please, what's the reason for not allowing this? Will this be added >> sometime later on? Or have I just not found a way to do it? >> >> All the best >> -T. >> >> -- >> Timo Lindemann >> Software Systems Engineer > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > "Have run Make so many times I dunno what's installed anymore" From raimo+erlang-questions@REDACTED Wed Nov 17 11:17:59 2010 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 17 Nov 2010 11:17:59 +0100 Subject: [erlang-questions] Problems with gen_udp on multi-homed hosts In-Reply-To: <20101116164821.GA27596@ecn.lan> References: <4CE2A3E3.7020803@gmail.com> <20101116164821.GA27596@ecn.lan> Message-ID: <20101117101759.GB9948@erix.ericsson.se> This seems to be a known property of the Linux TCP/IP stack, and I have checked FreeBSD 7.3 seems to behave the same meaning all BSDs probably do. Windows don't by the way, there you apparently get the broadcasts... http://www.developerweb.net/forum/showthread.php?t=5722 How network specific broadcasts work I have just tested briefly on Linux, and it seems not to make any difference. On Tue, Nov 16, 2010 at 11:48:21AM -0500, Michael Santos wrote: > On Tue, Nov 16, 2010 at 10:31:47AM -0500, Gregory Haskins wrote: > > Hi All, > > > > I am trying to develop a custom DHCP server against 14B on that is > > sitting on a multi-homed linux host. I only want the server to > > listen/respond on one of these interfaces, but I am having trouble > > getting gen_udp to behave in a way I would expect. > > > > My system has the following interfaces: > > > > 3> inet:getiflist(). > > {ok,["lo","br0","virbr0"]} > > 4> inet:ifget("virbr0", [addr]). > > {ok,[{addr,{192,168,122,1}}]} > > 5> inet:ifget("br0", [addr]). > > {ok,[{addr,{151,155,230,24}}]} > > > > and I am interested in packets only on 192.168.122.1/virbr0. > > > > So if I gen_udp:open() _without_ {ip, ?ADDR} set, I can see the > > broadcasts coming from 192.168.122.0/24 (as well as from the other > > interfaces) as I would expect. However, the {udp, Socket, ..} message I > > get has a inet:sockname() of {0.0.0.0}, and if I try to reply with a > > broadcast of {255.255.255.255} to the same socket, it goes out on the > > "br0" interface. > > Does it work if you send the packet out using a more specific broadcast > address? e.g., 192.168.122.255 > > > So the obvious solution to me after googling was to use the {ip, > > {192.168.122.1}} option on the open. The open() call succeeds with this > > option set, but I never see any packets (even though I have sniffed the > > line and they are indeed being sent. As a sanity check, I also set a > > bogus IP in the open(), and it does indeed exception out, so it seems > > like it is taking the option I set. > > Not sure why this isn't working, maybe the packet is still going out > your default interface. > > > I am running out of ideas outside of abandoning gen_udp() in favor of a > > ports driver, and/or not writing the DHCP portion of my code in Erlang. > > Neither is particularly attractive, especially given that this seems > > like a fairly straight forward sockets app and I would think it could be > > made to work. > > On Linux, you can try binding the socket to an interface [1]. klaar added > support to do this for his quite interesting dhcp server, edhcpd [2]. > > [1] https://github.com/msantos/procket > [2] https://github.com/klaar/edhcpd > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From raimo+erlang-questions@REDACTED Wed Nov 17 11:42:22 2010 From: raimo+erlang-questions@REDACTED (Raimo Niskanen) Date: Wed, 17 Nov 2010 11:42:22 +0100 Subject: [erlang-questions] Erlang pipermail or base retrieving ? In-Reply-To: <19429266.83511289983280092.JavaMail.www@wsfrf1136> References: <19429266.83511289983280092.JavaMail.www@wsfrf1136> Message-ID: <20101117104222.GC9948@erix.ericsson.se> On Wed, Nov 17, 2010 at 09:41:20AM +0100, dkoch@REDACTED wrote: > Hello dear Erlang staff, I'd like to import the full KB that has grown in the mailing list, yet I cannot fetch mails before I suscribed to the lists. However the old mailing list database (up to May 2009) exists in a packed and archived .GZ form here : > > > > www.erlang.org\pipermail\eeps > www.erlang.org\pipermail\erlang-announce > www.erlang.org\pipermail\erlang-bugs > www.erlang.org\pipermail\erlang-patches > www.erlang.org\pipermail\erlang-questions > > > Question : since the abandon of the previous mailing list model (pipermail) can the ezmlm-based mails be packed as previously on a regular basis so that I could fetch and include them manually into my mail client (Opera) to ease browsing them off-line ? It is a problem for when we upgrade the site in a not too distant future. We will then probably go back to Mailman (pipermail) and have to somehow transfer the ezmlm archives... > > > Thanks, your dearly, David "Kochise" KOCH > -- / Raimo Niskanen, Erlang/OTP, Ericsson AB From teiman.inna@REDACTED Wed Nov 17 12:19:35 2010 From: teiman.inna@REDACTED (Inna Teiman) Date: Wed, 17 Nov 2010 14:19:35 +0300 Subject: snmp - instrumentation functions Message-ID: Hello erlangers, My question is about the instrumentation functions for tables, e.g. table_access(get, RowIndex, Cols [, ExtraArgs]). How to process Oid = ..., that is, how to take into consideration? Is there any way to apply the instrumentation function to the table and to treat somehow the table entry, together with RowIndex and Cols? Thanks, Inna. From mbj@REDACTED Wed Nov 17 12:29:24 2010 From: mbj@REDACTED (Martin Bjorklund) Date: Wed, 17 Nov 2010 12:29:24 +0100 (CET) Subject: [erlang-questions] snmp - instrumentation functions In-Reply-To: References: Message-ID: <20101117.122924.146921425.mbj@tail-f.com> Hi, Inna Teiman wrote: > Hello erlangers, > My question is about the instrumentation functions for tables, e.g. > table_access(get, RowIndex, Cols [, ExtraArgs]). > How to process Oid =
..., that is, how to > take into consideration? Note that is always 1 (this is the way SMIv2 works). > Is there any way to apply the > instrumentation function to the table and to treat somehow the table > entry, together with RowIndex and Cols? The instrumentation function is called on a per-row basis. RowIndex is the part of the OID, and Cols is a list of all with the same in the PDU. I probably did not understand your question... /martin From ebegumisa@REDACTED Wed Nov 17 13:47:08 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Wed, 17 Nov 2010 23:47:08 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <090F6B51-470F-4076-9DAC-586732A3C3B1@erlang-solutions.com> References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> <090F6B51-470F-4076-9DAC-586732A3C3B1@erlang-solutions.com> Message-ID: Hello Fr?d?ric, > Spawning parallel processes is, in my opinion, a false ideal when it > comes to algorithms like the ones presented here. In a micro-benchmark > or in software that is oriented on doing only this task, multiprocessing > might gain you points. In a loaded production system where all the > processors are likely already busy, I wouldn't be surprised to see no > big gain by using multiple processes, and if any, the gain would likely > be very hard to predict. > > I think it's better to aim for the scaling kind of concurrency, not the > speed-up kind. If you use processes for truly concurrent tasks (rather > than attempting speedups of a single call), You'll benefit from > concurrency on a higher level on a server than from each function > individually. > > This has to be pretty application specific, though. Hmmm... I see what you mean. I look at it from an interaction perspective. If the application in question is an interactive one, I really hate to see it struggling with a problem for a long period of time with one of my cores saying "help!" while the other one just sits there, and more importantly, while *I* just sit there -- regardless of the algorithm it is working. (Any other Photoshop users been annoyed by this?) For a non-interactive/server application, I can see why you might not want to do this. But then again, if the muscle is there, and interactive clients are waiting... tough call. I guess it's very application specific as you said. One interesting thing I found from Garcia's results is that for this particular problem, crunching it using many processes on a single core was basically the same as doing it with one process on a single core. I was pleasantly surprised by that and didn't expect it. - Edmond - On Wed, 17 Nov 2010 02:21:00 +1100, Fr?d?ric Trottier-H?bert wrote: > > On 2010-11-16, at 09:51 AM, Evans, Matthew wrote: > >> Interesting thread...I think we can all take the following from this: >> >> 1) Write your solution in the "Erlang way", using Erlang's strengths >> such as spawning parallel processes, also HIPE and NIFS where >> appropriate. > Spawning parallel processes is, in my opinion, a false ideal when it > comes to algorithms like the ones presented here. In a micro-benchmark > or in software that is oriented on doing only this task, multiprocessing > might gain you points. In a loaded production system where all the > processors are likely already busy, I wouldn't be surprised to see no > big gain by using multiple processes, and if any, the gain would likely > be very hard to predict. > > I think it's better to aim for the scaling kind of concurrency, not the > speed-up kind. If you use processes for truly concurrent tasks (rather > than attempting speedups of a single call), You'll benefit from > concurrency on a higher level on a server than from each function > individually. > > This has to be pretty application specific, though. > >> 2) Write your solution efficiently - true in all languages, but >> especially in Erlang where you could end up doing needless recursion. >> >> 3) Erlang isn't always the best language for all problems. Neither is >> C, C++, C#, Java or any other language. Choose the language, or a mix >> of languages, that are the best fit your problem. >> >> What I would like to add is that we are trying to get Erlang accepted >> in a very C/C++ centric development environment. Although time and time >> again Erlang has shown itself a superior language and development >> environment for our domain, there is still a big problem in people >> accepting it. One of the main stumbling blocks is it is perceived to be >> "slow". People always say "it won't be faster than a C/C++ solution", >> or "a language written in a VM will always be slow [rolleyes]". >> >> We can argue until we are blue in the face that it will be as good or >> better when you throw multi-cores in the mix. That there are other >> advantages including shorter development time, better debugging, >> redundancy, failure detection etc. But overcoming that initial "speed" >> bias is a tough sell. I'm not saying there is an answer to this, and I >> know that the development team is doing their best to make Erlang >> faster, but we mustn't forget that for many, the "perceived" slowness >> is one factor that prevents them developing in Erlang. >> > > I think the speed obsession has to be remnants of an education always > based on time efficiency, especially for programmers coming from the era > where processors were really slow. Whether this explanation is right or > not, I don't think it is easily possible to convince someone that 1. his > priorities are not the right one and 2. the language he uses isn't the > best choice for the priorities he has. > > We should consider that the burden of proof is ours, not theirs. Any > unsubstantiated claim is usually met with skepticism and possibly > violence. This is completely normal and natural, in my opinion. The best > way to convince someone is with irrefutable proof, which might be why > the most common suggested way to get a team to accept Erlang is to > provide them with an implementation or success stories. And maybe Erlang > is no better than what they're using right now, in which case it would > be a waste of money to train everyone to work in a new environment with > little benefit in the long run. > > If people still are not accepting it even after valid proofs, then all > bets are off: maybe they don't feel like learning new technology, maybe > they don't believe that *they* can write efficient Erlang code (and then > you have to prove them they can), maybe they think it all comes down to > the same anyway (project success isn't language dependent? which could > be true in many cases), etc. I'm not sure much can be done except > leading by example, in any case. > >> /Rant over >> >> Matt >> > > > -- > Fred H?bert > http://www.erlang-solutions.com > > > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From rickard@REDACTED Wed Nov 17 13:57:01 2010 From: rickard@REDACTED (Rickard Green) Date: Wed, 17 Nov 2010 13:57:01 +0100 Subject: Bug in rwlock implementation Message-ID: <4CE3D11D.8040708@erlang.org> I've found a bug in the rwlock implementation that can cause a lock to end up in an inconsistent state. The bug is very seldom triggered, but can be. It is most likely to be triggered when using the read_concurrency option on an ets table that is frequently accessed from multiple processes doing lots of writes and reads. That is, a situation where you typically don't want to use the read_concurrency option in the first place. There is a fix for this bug currently under testing. The fix will appear in R14B01, but there might be changes to it since it is still under testing. You can help us out testing it, by using it. The fix can be fetched using: git fetch git://github.com/rickard-green/otp.git rickard/rwlock-euc2010/OTP-8925 Alternatively you can download the complete source including the fix by clicking at the download button at: https://github.com/rickard-green/otp/tree/rickard/rwlock-euc2010/OTP-8925 Note that you need to run "./otp_build autoconf" in the top directory before building. If you are worried that the bug might be triggered on your system, and you don't want to use a fix that isn't in its final state, it is also possible to enable pthread rwlocks as a workaround until R14B01. You do this by configuring the build as follows: ./configure force_pthread_rwlocks=yes Regards, Rickard Green, Erlang/OTP, Ericsson AB. From alain.odea@REDACTED Wed Nov 17 14:07:08 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Wed, 17 Nov 2010 09:37:08 -0330 Subject: [erlang-questions] Questions of configuring on SUSE Linux Enterprise 11 In-Reply-To: References: Message-ID: On Wed, Nov 17, 2010 at 5:39 AM, Attila Rajmund Nohl < attila.r.nohl@REDACTED> wrote: > 2010/11/17, Toan H. Le : > [...] > > I trace upward and find that the following warning may affect the crypto, > > ssh, and ssl > > ---- > > checking for OpenSSL >= 0.9.7 in standard locations... no > > configure: WARNING: No (usable) OpenSSL found, skipping ssl, ssh and > crypto > > applications > > ---- > > I did check with yast; two packages are install openssl (0.9.8h) and > > libopenssl0_9_8. I think those two are enough to use OpenSSL. > > > > Do I need something to bypass error of openssl? > > Probably the openssl-devel package. > And Java JDK. I usually install openjdk. And wxWidgets. You'll need wxWidgets 2.8 runtime and devel packages for that. It's worth having. From ebegumisa@REDACTED Wed Nov 17 14:21:21 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 18 Nov 2010 00:21:21 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: > But overcoming that initial "speed" bias is a tough sell. I'm not saying > there is an answer to this, and I know that the development team is > doing their best to make Erlang faster, but we mustn't forget that for > many, the "perceived" slowness is one factor that prevents them > developing in Erlang. I'm not sure that speed matters that much to people anymore (at least not for application developers, it obviously matters for system developers). Remember when Java and (classic) Visual Basic came out? There were many concerns about speed, but in a short period of time these languages dominated in application development (especially for business applications) because people had other higher priorities like ease-of-learning and time-to-market* as Richard pointed out. Personally, I think it's two other factors that are mainly causing Erlang's slower than expected penetration in general commercial application development (as opposed to it's well known niches)... 1. Erlang is perceived as "weird" A lot of people I've forwarded links to on Erlang just find it too strange! I'm not taking just syntax, concepts like variables that don't change are just too odd for them. And things they find familiar are done too differently. "How do I create an object-model?" they ask. 2. Not "main-stream" Realising that is a chicken-and-egg issue, one problem in application development is people feel comfort in numbers. If something isn't used by a good swath of software companies people are very hard to convince to use it. If it is wide-spread people graduate it to the mythical status of a "industry standard" and insist on using it even when there are far better tools for the problem space. JavaScript is a good example. I'm a huge fan of JavaScript for client-side development but I'm seeing it popup in the most unusual places, places which are inappropriate in my view. * Those two particular issues have a big impact on cost of development. Shorter development time obviously impacts the investment amount. Ease-of-learning means you can higher less-experienced, less-skilled programmers and more importantly, pay lower salaries. It wouldn't surprise me if the average cost to higher a competent Erlang programmer for 6 months is double that for a competent C# programmer! Sad, but these things matter. IMO, commercial application developers pay dearly in the end for choosing the wrong tools for the wrong reasons. - Edmond - -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ulf.wiger@REDACTED Wed Nov 17 14:26:42 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Wed, 17 Nov 2010 14:26:42 +0100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: <4CE1DAE9.6050500@infinite-interactive.com> References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> <4CE1DAE9.6050500@infinite-interactive.com> Message-ID: Looking closer at the schedule, the 4th may actually be a better date. Sorry about that. BR, Ulf W On 16 Nov 2010, at 02:14, Nikolas Bowe wrote: > Definitely! > > Id love to have a beer with Ulf and Justin and talk about mnesia and riak. > > > On 11/16/2010 7:03 AM, Edmond Begumisa wrote: >> Any one else in Melbourne up for an Erlounge Dec 2nd or 3rd? >> >> - Edmond - >> >> On Tue, 16 Nov 2010 03:58:44 +1100, Ulf Wiger wrote: >> >>> >>> Sure. My preference for Erlounges would be >>> >>> Melbourne: Dec 2 or 3 >>> Brisbane: Dec 9 >>> >>> BR, >>> Ulf >>> >>> On 15 Nov 2010, at 17:35, Edmond Begumisa wrote: >>> >>>> Hello Ulf, >>>> >>>> I'm in Melbourne! >>>> >>>> For those of us unfortunately unable to get time off to attend, is there a possibility for an ErlLounge? I personally would love to hear some war stories from your AXD days! >>>> >>>> - Edmond - >>>> >>>> >>>> On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger wrote: >>>> >>>>> >>>>> Just a short note for those of you Down Under. >>>>> >>>>> I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 >>>>> Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec 8-9). >>>>> >>>>> http://www.yowconference.com.au/index.html >>>>> >>>>> This should be a good opportunity for those interested in Erlang to get >>>>> together, ask questions and perhaps talk over beer afterwards. I'd love to hear >>>>> from some locals about how we can make the most of the opportunity. >>>>> >>>>> The conference itself will feature talks by me, Justin Sheehy and >>>>> Kresten Krab Thorup - as well as a load of distinguished speakers from >>>>> different camps. >>>>> >>>>> http://www.yowconference.com.au/melbourne/speakers/index.html >>>>> >>>>> It should be a hoot. Please sign up and recruit some friends too. >>>>> >>>>> I hope to see you there. >>>>> >>>>> BR, >>>>> Ulf W >>>>> >>>>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>>>> http://erlang-solutions.com >>>>> >>>>> >>>>> >>>>> >>>>> ________________________________________________________________ >>>>> erlang-questions (at) erlang.org mailing list. >>>>> See http://www.erlang.org/faq.html >>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>>> >>>> >>>> >>>> -- >>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >>> >>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>> http://erlang-solutions.com >>> >>> >>> >>> >>> ________________________________________________________________ >>> erlang-questions (at) erlang.org mailing list. >>> See http://www.erlang.org/faq.html >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>> >> >> > > > This message and its attachments may contain legally privileged or confidential information. This message is intended for the use of the individual or entity to which it is addressed. If you are not the addressee indicated in this message, or the employee or agent responsible for delivering the message to the intended recipient, you may not copy or deliver this message or its attachments to anyone. Rather, you should permanently delete this message and its attachments and kindly notify the sender by reply e-mail. Any content of this message and its attachments, which does not relate to the official business of the sending company must be taken not to have been sent or endorsed by the sending company or any of its related entities. No warranty is made that the e-mail or attachment(s) are free from computer virus or other defect. > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From tom@REDACTED Wed Nov 17 14:43:44 2010 From: tom@REDACTED (tom@REDACTED) Date: Wed, 17 Nov 2010 14:43:44 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101117020847.GA29358@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011170212.25565.tom@diogunix.com> <20101117020847.GA29358@ecn.lan> Message-ID: <201011171443.45115.tom@diogunix.com> I meanwhile also made some test with editing the intrc file as I have gone through the Erlang docs regarding inetrc in the mean time: No success unfortunately. Additional information from the FreeBSD questions mailing list on the Erlang 14B issue: > I believe we are facing the same issue. Our setup is ejabberd-2.1.5 in > a FreeBSD 7.3 amd64 jail and the workaround we found is running an > older version of erlang (erlang-r13b04_3,1). You can find the old > erlang port files in the FreeBSD CVS repository or here: So, even I cannot be already sure here, the issue might be caused through the changes in Erlang 14B as Michael said: > epmd in R14B was changed to allow some messages (like name registrations) > only from 127/8. If true, one might consider to make Erlang configurable regarding the IP address. On the short run, a second Port for the FreeBSD Ports Collection (using an older Erlang version) could ensure Erlang can be made available in FreeBSD Jails. Any other ideas / hints of course are still welcome. kind regards Tom From wiener.guy@REDACTED Wed Nov 17 15:04:18 2010 From: wiener.guy@REDACTED (Guy Wiener) Date: Wed, 17 Nov 2010 16:04:18 +0200 Subject: CouchDB tutorial presentation? Message-ID: Hello everyone, I am looking for a tutorial on CouchDB in presentation format. Preferably with Javascript examples. I want to show it in class, or at least to base on it, so it needs to be public or permitted by the author. Thanks a lot! :- Guy From michael.santos@REDACTED Wed Nov 17 15:48:54 2010 From: michael.santos@REDACTED (Michael Santos) Date: Wed, 17 Nov 2010 09:48:54 -0500 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <201011170459.52991.tom@diogunix.com> References: <201011161114.01488.tom@diogunix.com> <201011170212.25565.tom@diogunix.com> <20101117020847.GA29358@ecn.lan> <201011170459.52991.tom@diogunix.com> Message-ID: <20101117144854.GB29358@ecn.lan> On Wed, Nov 17, 2010 at 04:59:52AM +0100, tom@REDACTED wrote: > > It might help. I think the quickest way to debug this would be to: > > donesults below ... Did you run these steps concurrently or sequentially? e.g., when you brought up the erlang node, was the debug epmd running? epmd (with debug switches) needs to be running in one shell, the tcpdump's (with the "-n" switch) in another. Then start up the Erlang node. > # tcpdump -i lo0 -vv port 4369 > tcpdump: WARNING: lo0: no IPv4 address assigned > tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 96 > bytes Was the output below from the dump of the loopback? > and tcpdump output: > > 03:33:15.964680 IP (tos 0x0, ttl 64, id 24389, offset 0, flags [DF], proto > TCP (6), length 60, bad cksum 0 (->4f37)!) > mail.kepos.org.10975 > mail.kepos.org.4369: Flags [S], cksum 0xd692 > (correct), seq 3123827793, win 65535, options [mss 16344,nop,wscale > 3,sackOK,TS val 77847542 ecr 0], length 0 > 03:33:15.964701 IP (tos 0x0, ttl 64, id 24390, offset 0, flags [DF], proto > TCP (6), length 60, bad cksum 0 (->4f36)!) > mail.kepos.org.4369 > mail.kepos.org.10975: Flags [S.], cksum 0x776a > (correct), seq 3252621940, ack 3123827794, win 65535, options [mss > 16344,nop,wscale 3,sackOK,TS val 2304704868 ecr 77847542], length 0 > 03:33:15.964712 IP (tos 0x0, ttl 64, id 24391, offset 0, flags [DF], proto > TCP (6), length 52, bad cksum 0 (->4f3d)!) > mail.kepos.org.4369 > mail.kepos.org.10975: Flags [F.], cksum 0xbd43 > (correct), seq 1, ack 19, win 8960, options [nop,nop,TS val 2304704868 ecr > 77847542], length 0 > 03:33:15.964906 IP (tos 0x0, ttl 64, id 24394, offset 0, flags [DF], proto > TCP (6), length 52, bad cksum 0 (->4f3a)!) > mail.kepos.org.10975 > mail.kepos.org.4369: Flags [.], cksum 0xbd43 > (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 ecr > 2304704868], length 0 > 03:33:15.964929 IP (tos 0x0, ttl 64, id 24395, offset 0, flags [DF], proto > TCP (6), length 52, bad cksum 0 (->4f39)!) > mail.kepos.org.10975 > mail.kepos.org.4369: Flags [F.], cksum 0xbd42 > (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 ecr > 2304704868], length 0 > 03:33:15.964937 IP (tos 0x0, ttl 64, id 24396, offset 0, flags [DF], proto > TCP (6), length 52, bad cksum 0 (->4f38)!) > mail.kepos.org.4369 > mail.kepos.org.10975: Flags [.], cksum 0xbd43 > (correct), seq 2, ack 20, win 8959, options [nop,nop,TS val 2304704868 ecr > 77847542], length 0 > ^C Do not resolve IP addresses. We need to see the source and destination IP addresses. So something on port 4369 (on whatever "mail.kepos.org" is) is accepting and closing the connection. If it is not epmd (because there is nothing in your debug log), what is it? > Juust to exde any misconfiguation on the machine I allso tried itt again > switching off the firewall for this test: same result. Always a good idea. From ebegumisa@REDACTED Wed Nov 17 16:00:29 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 18 Nov 2010 02:00:29 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: <8EDD1083-51A8-46BA-854E-D143520095A5@cs.otago.ac.nz> References: <00C2FE7F-92D3-48B0-A5E6-7719FBB0D449@cs.otago.ac.nz> <4CE29343.30406@cs.ntua.gr> <8EDD1083-51A8-46BA-854E-D143520095A5@cs.otago.ac.nz> Message-ID: Aaaahhh, > The compiler *could* make a special case of lists:seq/[2,3] > and generate > > (1, N, ...) > > But it doesn't. OK. I erroneously thought that this *was* the case. > The compiler can't inline the definition and > use whether it is now, because it's a remote call (even > if -imported), so it might change at run time. I see it now :) - Edmond - On Wed, 17 Nov 2010 16:45:09 +1100, Richard O'Keefe wrote: > > On 17/11/2010, at 4:34 AM, Edmond Begumisa wrote: > >> On Wed, 17 Nov 2010 01:20:51 +1100, Kostis Sagonas >> wrote: >> >>> There are two basic issues here: >>> (1) The Erlang compiler doesn't really handle list comprehension as >>> native loops; it compiles list comprehensions to recursive >>> functions. lists:seq(1, N) is not special syntax -- unlike Haskell >>> -- >>> and almost surely builds an actual list. >>> Yes. This is an issue. Actually, it's more general: in my experience >>> most Erlang programmers do not understand the difference between list >>> comprehensions and lists:foreach/2. Many of them find list >>> comprehension notation so nice that they often use it in places where >>> building the list is not only is not their intention but is completely >>> unnecessary. >>> The compiler is not sufficiently clever to perform deforestation and >>> avoid construction of intermediate lists. >>> >> >> First I'm hearing of this. It really ought to be in one of those green >> boxes in the documentation. > > Let's go around this one more time. > > The current Erlang compiler turns > > [f(X) || X <- lists:seq(1, N)] > > into > (lists:seq(1, N), ...) > where > > is a function generated by the compiler. > > The compiler *could* make a special case of lists:seq/[2,3] > and generate > > (1, N, ...) > > But it doesn't. The actual definition of lists:seq/2 not > only *could* change, in the not too distant past it *has* > changed. The compiler can't inline the definition and > use whether it is now, because it's a remote call (even > if -imported), so it might change at run time. > > Haskell compilers and the Mercury compiler *do* perform > deforestation because they are given an unchanging program > to work with and can safely do cross-module inlining. > Erlang can't. > > This would be a potentially serious problem if iterating > over integer ranges were something Erlang spent a lot of > time doing. > > I just checked an Erlang/OTP release I keep handy for such > things. > ~ 1,500,000 raw lines > ~ 950,000 SLOW > 34 list:seq iterations > 18 in two EUNIT tests > 3 in other test cases (at least) > 13 list:seq not in an ?_assert > ~ 3200 Pat <- Exp > > The unit tests don't matter for performance. We're left > with about one SLOC in every 72,000 being an iteration of > this kind, or about <- in every 250 being an integer > iteration. > > The Erlang/OTP team probably feel they have other maddened > grizzly bears to stun before they bother with something that > seems to turn up mainly in toy benchmarks, and who could blame them? > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From ebegumisa@REDACTED Wed Nov 17 16:15:27 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Thu, 18 Nov 2010 02:15:27 +1100 Subject: [erlang-questions] Erlang shows its slow face! In-Reply-To: References: <61794868-036C-42E6-8D02-8931EE67F764@rogvall.se> Message-ID: That was supposed to be 'hire' not 'higher' in that last paragraph. My auto-spell-corrector goes insane at times :) - Edmond - On Thu, 18 Nov 2010 00:21:21 +1100, Edmond Begumisa wrote: >> But overcoming that initial "speed" bias is a tough sell. I'm not >> saying there is an answer to this, and I know that the development team >> is doing their best to make Erlang faster, but we mustn't forget that >> for many, the "perceived" slowness is one factor that prevents them >> developing in Erlang. > > I'm not sure that speed matters that much to people anymore (at least > not for application developers, it obviously matters for system > developers). Remember when Java and (classic) Visual Basic came out? > There were many concerns about speed, but in a short period of time > these languages dominated in application development (especially for > business applications) because people had other higher priorities like > ease-of-learning and time-to-market* as Richard pointed out. > > Personally, I think it's two other factors that are mainly causing > Erlang's slower than expected penetration in general commercial > application development (as opposed to it's well known niches)... > > 1. Erlang is perceived as "weird" > > A lot of people I've forwarded links to on Erlang just find it too > strange! I'm not taking just syntax, concepts like variables that don't > change are just too odd for them. And things they find familiar are done > too differently. "How do I create an object-model?" they ask. > > 2. Not "main-stream" > > Realising that is a chicken-and-egg issue, one problem in application > development is people feel comfort in numbers. If something isn't used > by a good swath of software companies people are very hard to convince > to use it. If it is wide-spread people graduate it to the mythical > status of a "industry standard" and insist on using it even when there > are far better tools for the problem space. JavaScript is a good > example. I'm a huge fan of JavaScript for client-side development but > I'm seeing it popup in the most unusual places, places which are > inappropriate in my view. > > * Those two particular issues have a big impact on cost of development. > Shorter development time obviously impacts the investment amount. > Ease-of-learning means you can higher less-experienced, less-skilled > programmers and more importantly, pay lower salaries. It wouldn't > surprise me if the average cost to higher a competent Erlang programmer > for 6 months is double that for a competent C# programmer! Sad, but > these things matter. IMO, commercial application developers pay dearly > in the end for choosing the wrong tools for the wrong reasons. > > - Edmond - > > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From comptekki@REDACTED Wed Nov 17 17:04:23 2010 From: comptekki@REDACTED (Wes James) Date: Wed, 17 Nov 2010 09:04:23 -0700 Subject: [erlang-questions] What's a solid way to have erlang automatically send email. In-Reply-To: References: Message-ID: On Sun, Nov 14, 2010 at 4:31 PM, G.S. wrote: > Hello, > > I have a website where the administrator needs to receive email alert (to > login and fix a problem) when certain events occur. Is there an Erlang > module that allows one to easily send out an email? Or something that can be > used in the same manner? > > Thanks, > -Gene > This might work for you: https://github.com/Vagabond/gen_smtp -wes From gregory.haskins@REDACTED Wed Nov 17 19:34:01 2010 From: gregory.haskins@REDACTED (Gregory Haskins) Date: Wed, 17 Nov 2010 13:34:01 -0500 Subject: [erlang-questions] Questions of configuring on SUSE Linux Enterprise 11 In-Reply-To: References: Message-ID: <4CE42019.200@gmail.com> On 11/17/10 2:47 AM, Toan H. Le wrote: > Hi, > > I try to compile Erlang R14B on SUSE 11 but I face errors of configuring. FYI: I have 14B already packaged for SUSE-11 (SP1), here: http://download.opensuse.org/repositories/home:/ghaskins:/erlang/SLE_11_SP1/ I could easily add SLE-11.0 to the build-service, if you were looking for supporting the older GMC. You can either just zypper ar my repo in, or at least take a look at the .src.rpm for hints as to how it was compiled. -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 267 bytes Desc: OpenPGP digital signature URL: From kaiduanx@REDACTED Wed Nov 17 19:42:30 2010 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Wed, 17 Nov 2010 13:42:30 -0500 Subject: Erlang SSL performance Message-ID: Hi, all, We all know Erlang can support thousands of simultaneous TCP/HTTP connections as reported by the following links, http://www.sics.se/~joe/apachevsyaws.html http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1/ I am wondering if there is any performance report on SSL with Erlang? For example, how many simultaneous SSL connections a box can support? What is the throughput? Any comments are welcome. Best regards, /Kaiduan From therevoltingx@REDACTED Wed Nov 17 22:56:49 2010 From: therevoltingx@REDACTED (Miguel Morales) Date: Wed, 17 Nov 2010 13:56:49 -0800 Subject: erlang_js - implement function in Erlang Message-ID: Hi all, I am interested in using the erlang_js module, however I am having problems understanding if it can do what I want to do with it. Basically, I'd like to implement javascript functions in Erlang, this is basically to create a scripting system for my application. Say I have a javascript string that looks like: "var x = MyFunction(1, 2);" I would then like to implement MyFunction in Erlang somewhat like this: -module(test). -compile(export_all). MyFunction(Arg1, Arg2) -> io:fwrite("MyFunction called args: ~p, ~p~n", [Arg1, Arg2]). I am using RabbitMQ and for the moment I simply send an MQ message to a Java node that executes the javascript code. The way Java does it is very nice, I just hava a MyClass class with all my custom functions which can be called from evaluated javascript code. The problem is keeping the code clean and the flow of things, I'd like to integrate the scripting closer to the game logic. Looking at the documentation http://www.erlware.org/lib/5.7.5/erlang_js-0.1/index.html, the closest thing I see is the very last example where ejsLog is called inside the javasript string. However, this is implemented in C in spidermonkey.c. Not sure if I'm just missing something. Any help would be appreciated. -- ~ Jeremiah:9:23-24 Android 2D MMORPG: http://developingthedream.blogspot.com/, http://www.youtube.com/user/revoltingx From anthonym@REDACTED Thu Nov 18 01:00:35 2010 From: anthonym@REDACTED (Anthony Molinaro) Date: Wed, 17 Nov 2010 16:00:35 -0800 Subject: xmerl simple out of CDATA blocks Message-ID: <20101118000035.GC7578@alumni.caltech.edu> Hi, So I noticed after some searching that while you can read CDATA blocks with xmerl you can't seem to write them out. So for instance Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false] Eshell V5.8.1 (abort with ^G) 1> I = "\n \n

Hello World

\n Bye\n \n\n]]>". "\n \n

Hello World

\n Bye\n \n\n]]>" 2> {X,_} = xmerl_scan:string (I). {{xmlElement,'HTMLResource','HTMLResource',[], {xmlNamespace,[],[]}, [],1,[], [{xmlText,[{'HTMLResource',1}], 1,[], "\n\n \n

Hello World

\n Bye\n \n\n", cdata}], [],"/home/molinaro/tmp",undeclared}, []} 3> O = lists:flatten (xmerl:export_simple_content ([X], xmerl_xml)). "\n<html>\n <body>\n <h1>Hello World</h1>\n <a href=\"http://www.example.com/?foo=bar&baz=bob\">Bye</a>\n </body>\n</html>\n" 4> I = O. ** exception error: no match of right hand side value "\n<html>\n <body>\n <h1>Hello World</h1>\n <a href=\"http://www.example.com/?foo=bar&baz=bob\">Bye</a>\n </body>\n</html>\n" 5> However, with the following patch --- a/xmerl.erl 2010-09-29 11:13:00.000000000 -0700 +++ b/xmerl.erl 2010-11-01 11:23:54.000000000 -0700 @@ -185,6 +185,8 @@ %% Content = [Element] %% Callback = [atom()] %% @doc Exports normal XML content directly, without further context. +export_content([#xmlText{value = Text, type = cdata} | Es], Callbacks) -> + [ "" | export_content(Es, Callbacks) ]; export_content([#xmlText{value = Text} | Es], Callbacks) -> [apply_text_cb(Callbacks, Text) | export_content(Es, Callbacks)]; export_content([#xmlPI{} | Es], Callbacks) -> I get the behavior I want. Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false] Eshell V5.8.1 (abort with ^G) 1> I = "\n \n

Hello World

\n Bye\n \n\n]]>". "\n \n

Hello World

\n Bye\n \n\n]]>" 2> {X,_} = xmerl_scan:string (I). {{xmlElement,'HTMLResource','HTMLResource',[], {xmlNamespace,[],[]}, [],1,[], [{xmlText,[{'HTMLResource',1}], 1,[], "\n\n \n

Hello World

\n Bye\n \n\n", cdata}], [],"/home/molinaro",undeclared}, []} 3> O = lists:flatten (xmerl:export_simple_content ([X], xmerl_xml)). "\n \n

Hello World

\n Bye\n \n\n]]>" 4> I = O. "\n \n

Hello World

\n Bye\n \n\n]]>" However, it seems like this sort of patch would be unacceptable since it may be invalid for certain formatters (ie, it assumes cdata is always rendered the same). However, I don't see anyway to do this with the current code as there is only a '#text#' callback which takes the text from an #xmlText element without the type. This means you can't have different rules based on the type. So I'm basically looking for a little guidance before I start hacking a larger patch. What sort of behavior would be acceptable? Maybe a new callback '#cdata#' with a default of using '#text#'? Or a '#text#'/2 function with takes the type? Other ideas? Or maybe this patch is fine? Thanks, -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro From nbowe@REDACTED Thu Nov 18 01:00:04 2010 From: nbowe@REDACTED (Nikolas Bowe) Date: Thu, 18 Nov 2010 11:00:04 +1100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> <4CE1DAE9.6050500@infinite-interactive.com> Message-ID: <4CE46C84.5040808@infinite-interactive.com> 4th it is. On 11/18/2010 12:26 AM, Ulf Wiger wrote: > Looking closer at the schedule, the 4th may actually be a better date. > > Sorry about that. > > BR, > Ulf W > > On 16 Nov 2010, at 02:14, Nikolas Bowe wrote: > >> Definitely! >> >> Id love to have a beer with Ulf and Justin and talk about mnesia and riak. >> >> >> On 11/16/2010 7:03 AM, Edmond Begumisa wrote: >>> Any one else in Melbourne up for an Erlounge Dec 2nd or 3rd? >>> >>> - Edmond - >>> >>> On Tue, 16 Nov 2010 03:58:44 +1100, Ulf Wiger wrote: >>> >>>> Sure. My preference for Erlounges would be >>>> >>>> Melbourne: Dec 2 or 3 >>>> Brisbane: Dec 9 >>>> >>>> BR, >>>> Ulf >>>> >>>> On 15 Nov 2010, at 17:35, Edmond Begumisa wrote: >>>> >>>>> Hello Ulf, >>>>> >>>>> I'm in Melbourne! >>>>> >>>>> For those of us unfortunately unable to get time off to attend, is there a possibility for an ErlLounge? I personally would love to hear some war stories from your AXD days! >>>>> >>>>> - Edmond - >>>>> >>>>> >>>>> On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger wrote: >>>>> >>>>>> Just a short note for those of you Down Under. >>>>>> >>>>>> I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 >>>>>> Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec 8-9). >>>>>> >>>>>> http://www.yowconference.com.au/index.html >>>>>> >>>>>> This should be a good opportunity for those interested in Erlang to get >>>>>> together, ask questions and perhaps talk over beer afterwards. I'd love to hear >>>>>> from some locals about how we can make the most of the opportunity. >>>>>> >>>>>> The conference itself will feature talks by me, Justin Sheehy and >>>>>> Kresten Krab Thorup - as well as a load of distinguished speakers from >>>>>> different camps. >>>>>> >>>>>> http://www.yowconference.com.au/melbourne/speakers/index.html >>>>>> >>>>>> It should be a hoot. Please sign up and recruit some friends too. >>>>>> >>>>>> I hope to see you there. >>>>>> >>>>>> BR, >>>>>> Ulf W >>>>>> >>>>>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>>>>> http://erlang-solutions.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ________________________________________________________________ >>>>>> erlang-questions (at) erlang.org mailing list. >>>>>> See http://www.erlang.org/faq.html >>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>>>> >>>>> >>>>> -- >>>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >>>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>>> http://erlang-solutions.com >>>> >>>> >>>> >>>> >>>> ________________________________________________________________ >>>> erlang-questions (at) erlang.org mailing list. >>>> See http://www.erlang.org/faq.html >>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>> >>> >> >> This message and its attachments may contain legally privileged or confidential information. This message is intended for the use of the individual or entity to which it is addressed. If you are not the addressee indicated in this message, or the employee or agent responsible for delivering the message to the intended recipient, you may not copy or deliver this message or its attachments to anyone. Rather, you should permanently delete this message and its attachments and kindly notify the sender by reply e-mail. Any content of this message and its attachments, which does not relate to the official business of the sending company must be taken not to have been sent or endorsed by the sending company or any of its related entities. No warranty is made that the e-mail or attachment(s) are free from computer virus or other defect. >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED This message and its attachments may contain legally privileged or confidential information. This message is intended for the use of the individual or entity to which it is addressed. If you are not the addressee indicated in this message, or the employee or agent responsible for delivering the message to the intended recipient, you may not copy or deliver this message or its attachments to anyone. Rather, you should permanently delete this message and its attachments and kindly notify the sender by reply e-mail. Any content of this message and its attachments, which does not relate to the official business of the sending company must be taken not to have been sent or endorsed by the sending company or any of its related entities. No warranty is made that the e-mail or attachment(s) are free from computer virus or other defect. From robert.virding@REDACTED Thu Nov 18 01:36:09 2010 From: robert.virding@REDACTED (Robert Virding) Date: Thu, 18 Nov 2010 00:36:09 +0000 (GMT) Subject: [erlang-questions] Dynamic code loading and hotswap behaviour In-Reply-To: <1223841212.96051290040267143.JavaMail.root@zimbra> Message-ID: <470652950.96071290040568985.JavaMail.root@zimbra> ----- "Alessandro Sivieri" wrote: > 2010/11/16 Rapsey > > > It reloads anyway and the mechanisms apply. > > > > > And is there a motivation for this? I mean, it seems that reloading > the same > module more times can be seen as an error from a developer point of > view, if > this happens there is something wrong in my code, but anyway the > runtime > should understand that nothing has changed... Well, the trivial answer is that it is doing exactly what you have told it to do. You told it to load a binary module, it loaded a binary module. How should the run-time be able to understand that "nothing" has happened? Given the semantics of code loading then loading the module and not reloading the module because it happens to be the same are very different operations and I would definitely not want the system to try and be smart in the background. It also a very useful way of killing processes when developing. Robert From info@REDACTED Thu Nov 18 16:46:44 2010 From: info@REDACTED (info) Date: Thu, 18 Nov 2010 16:46:44 +0100 Subject: end of process Message-ID: <201011181646430563447@its3.ch> Hello, Could you refresh my mind ... In an gen_server I have this: init() -> ... spawn_link(processA), ... processA()-> ... exit({error,processA_failed}) ... exit will kill the processA: correct ? who receives the error message ? In this example (erlang doc) what happens if loop(Socket)-> receive {tcp,Socket,Data}->process(Data),loop(Socket); {tcp_close,Socket}->ok end. ok terminates the loop process: correct ? Then what is the difference with my first example ? what is better: ok or exit or anything else ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From lukas@REDACTED Thu Nov 18 17:00:52 2010 From: lukas@REDACTED (Lukas Larsson) Date: Thu, 18 Nov 2010 17:00:52 +0100 Subject: [erlang-questions] end of process In-Reply-To: <201011181646430563447@its3.ch> References: <201011181646430563447@its3.ch> Message-ID: <1290096052.4890.6.camel@ancalagon.du.uab.ericsson.se> use exit/1 if the you want to manually terminate the process abnormally, just let the code finish with ok or whatever if you want to terminate it normally. See http://www.erlang.org/doc/reference_manual/processes.html#id78133 for more details. Lukas On Thu, 2010-11-18 at 16:46 +0100, info wrote: > Hello, > Could you refresh my mind ... > In an gen_server I have this: > > init() -> > ... > spawn_link(processA), > ... > > processA()-> > ... > exit({error,processA_failed}) > ... > > exit will kill the processA: correct ? > who receives the error message ? > > In this example (erlang doc) what happens if > > loop(Socket)-> > receive > {tcp,Socket,Data}->process(Data),loop(Socket); > {tcp_close,Socket}->ok > end. > > ok terminates the loop process: correct ? > > Then what is the difference with my first example ? what is better: ok or exit or anything else ? > > J-Ph. Constantin > ITS3 Genve > www.its3.ch From Grant_Li@REDACTED Thu Nov 18 17:21:16 2010 From: Grant_Li@REDACTED (Li, Grant) Date: Thu, 18 Nov 2010 10:21:16 -0600 Subject: Build Erlang on Solaris 10 In-Reply-To: <201011181646430563447@its3.ch> References: <201011181646430563447@its3.ch> Message-ID: <605635FE3E01874EB59477D4D01CA9DFD3609506@odcexch01.csg.csgsystems.com> Hello Experts, I'm new to Erlang... I built Erlang on Solaris 10, with the following configure command: $ ./configure --enable-threads --host=sparc --build=sparc-sun-solaris2.10 --without-ssl erl_xcomp_bigendian=yes erl_x comp_linux_nptl=yes erl_xcomp_linux_usable_sigusrx=yes erl_xcomp_poll=yes erl_xcomp_putenv_copy=yes erl_xcomp_gethrvtime_procfs_ioctl=yes erl_xcomp_clock_gettime_cpu_time=yes erl_xcomp_dlsym_brk_wrappers=yes When the make is completed, I observed the following issues: 1. Erlang was build in both sparc-sun-solaris2.10 and sparc-sun-sunos4.1.1. Does anyone know where sunos4.1.1 was coming from? The binaries in otp_src_R14B/bin/sparc-sun-solaris2.10 and otp_src_R14B/bin/sparc-sun-sunos4.1.1 have different sizes; 2. I wondered if erl was not built correctly. Two erl files were found: -rwxr-xr-x 1 ligr01 etsi 937 Nov 12 16:24 otp_src_R14B/bootstrap/bin/erl -rwxrwxr-x 1 ligr01 etsi 1448 Sep 13 12:00 otp_src_R14B/erts/etc/win32/cygwin_tools/erl For my understanding, erl should exist in otp_src_R14B/bin directory. Thank you, Grant Li CSG Systems, Inc From jameschurchman@REDACTED Thu Nov 18 18:00:24 2010 From: jameschurchman@REDACTED (James Churchman) Date: Thu, 18 Nov 2010 17:00:24 +0000 Subject: random entry from mensia Message-ID: Someone i know brought up an interesting question that is usually very easy to solve in sql but i could not see an obvious efficient solution in Mnesia If say you want to display on a webpage random entries. A bit like Facebook shows random friends when you view your profile, how do you go about doing it. You can obviously : 1) Retrieve or cache all the keys and pick a key at random 2) Traverse the entire table and stop at a random point (probably with a rough knowledge of the length of the table first) 3) Add an integer increment to every row, and pick a random number, index it and look that up (which is roughly what you would do in sql, though in sql having an auto increment is almost a prerequisite, unlike Mnesia) All of the above are fine solutions, but seem a bit wasteful in several different ways There seemed to be an obvious solution that is fast and requires no additions to the table, but unless i miss understand the function, it just does not work. There is the option in both ETS and Mnesia called dirty_slot, which i thought gets an item by its numeric order. So if i do mnesia:dirty_slot(table_name,50) as long as there are more than 50 items it will return item 50. I realise that it warns you that in some cases due to updates it will fail, but for me this and the ETS equivalent always return an empty list. I am using r13b02. Is this just a bug as its an under-used feature and never been fixed, or is there a specific table type it has to be used with? (it does not say it does in the docs) or is there a far better solution i missed? James From jesper.louis.andersen@REDACTED Thu Nov 18 19:11:26 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 18 Nov 2010 19:11:26 +0100 Subject: [erlang-questions] random entry from mensia In-Reply-To: References: Message-ID: On Thu, Nov 18, 2010 at 6:00 PM, James Churchman wrote: > Someone i know brought up an interesting question that is usually very easy to solve in sql but i could not see an obvious efficient solution in Mnesia The same problem is present in most other K/V stores where you can not rely on the fact that the K is an increasing value. My solution would probably be to have another table mapping {I, K} where I is a running index. The exercise left for the reader is to handle the case where K is deleted :) > > There is the option in both ETS and Mnesia called dirty_slot, which i thought gets an item by its numeric order. No, it doesn't. ETS is, for some ETS types, implemented as a hash-table. This means that the table contains buckets, some of which are full, and some of which are empty. the slot/2 calls retrieves the contents of the buckets in the hash-table so depending on load-factor of the table you will amost always get a result, almost never get a result or something in between. Leave slot for the debugging it was meant to do. For example, an empty table use a 256 bucket has table. If you insert a single element into this table, one of the bucket will be filled, but the remaining 255 buckets are not, yet, with any data. It does suggest a crude algorithm however: 1. Pick a number, i, randomly between 0 and BUCKET_SIZE (i.e., 0.256 above) 2. Search i, i+1, i+2, i+.., until you hit a non-empty bucket. Pick randomly from this bucket. 3. If you hit $end_of_table continue the search from 0. Depending on the randomness properties of the hash-table this is either really really bad or adequate. I'd hesitate to call the distribution good though as I am afraid something will go very wrong. Also, the run-time is potentially much greater than what you expect but you will hopefully hit something. -- J. From jameschurchman@REDACTED Thu Nov 18 19:53:41 2010 From: jameschurchman@REDACTED (James Churchman) Date: Thu, 18 Nov 2010 18:53:41 +0000 Subject: [erlang-questions] random entry from mensia In-Reply-To: References: Message-ID: Ok thanks jesper that explains a lot! I ran rp([{N,mnesia:dirty_slot(offline_msg,N)}||N<-lists:seq(0,255)]). on a table with only 103 items in it and now i understand what it does. Only about 3 of the items had anything in the at all & it seemed that item 199 had about 90 % of everything, so i can see it is nothing like what i wanted. I would assume that a heavily loaded table the distribution may be a bit more even but even so i could potentially end up with a few % of my table begin returned with that command. Out of interest would increasing the number of buckets say 100 fold have a strong detrimental effect on the overall performance of the table & is this a user controllable parameter? In conclusion however, just added an auto incremented indexed column is the most sensible solution to the problem i think James On 18 Nov 2010, at 18:11, Jesper Louis Andersen wrote: > On Thu, Nov 18, 2010 at 6:00 PM, James Churchman > wrote: >> Someone i know brought up an interesting question that is usually very easy to solve in sql but i could not see an obvious efficient solution in Mnesia > > The same problem is present in most other K/V stores where you can not > rely on the fact that the K is an increasing value. My solution would > probably be to have another table mapping {I, K} where I is a running > index. The exercise left for the reader is to handle the case where K > is deleted :) > >> >> There is the option in both ETS and Mnesia called dirty_slot, which i thought gets an item by its numeric order. > > No, it doesn't. > > ETS is, for some ETS types, implemented as a hash-table. This means > that the table contains buckets, some of which are full, and some of > which are empty. the slot/2 calls retrieves the contents of the > buckets in the hash-table so depending on load-factor of the table you > will amost always get a result, almost never get a result or something > in between. Leave slot for the debugging it was meant to do. For > example, an empty table use a 256 bucket has table. If you insert a > single element into this table, one of the bucket will be filled, but > the remaining 255 buckets are not, yet, with any data. It does suggest > a crude algorithm however: > > 1. Pick a number, i, randomly between 0 and BUCKET_SIZE (i.e., 0.256 above) > 2. Search i, i+1, i+2, i+.., until you hit a non-empty bucket. Pick > randomly from this bucket. > 3. If you hit $end_of_table continue the search from 0. > > Depending on the randomness properties of the hash-table this is > either really really bad or adequate. I'd hesitate to call the > distribution good though as I am afraid something will go very wrong. > Also, the run-time is potentially much greater than what you expect > but you will hopefully hit something. > > -- > J. From tom@REDACTED Thu Nov 18 20:55:40 2010 From: tom@REDACTED (tom@REDACTED) Date: Thu, 18 Nov 2010 20:55:40 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101117144854.GB29358@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011170459.52991.tom@diogunix.com> <20101117144854.GB29358@ecn.lan> Message-ID: <201011182055.40721.tom@diogunix.com> Hello Michael, sorry for the delay, I was off the office for one day. Ok, to be precisely I repeated the test: I first started tcpdum and let it run. I then started epmd -d -d -d in a second shell. I finally started erl -name foo in a third shell. epdm -d -d -d now spit out something more informative: # epmd -d -d -d epmd: Thu Nov 18 19:26:33 2010: epmd running - daemon = 0 epmd: Thu Nov 18 19:26:33 2010: try to initiate listening port 4369 epmd: Thu Nov 18 19:26:33 2010: starting epmd: Thu Nov 18 19:26:33 2010: entering the main select() loop epmd: Thu Nov 18 19:26:38 2010: time in seconds: 1290108398 epmd: Thu Nov 18 19:26:43 2010: time in seconds: 1290108403 epmd: Thu Nov 18 19:26:44 2010: Non-local peer connected epmd: Thu Nov 18 19:26:44 2010: time in seconds: 1290108404 epmd: Thu Nov 18 19:26:44 2010: opening connection on file descriptor 4 epmd: Thu Nov 18 19:26:44 2010: time in seconds: 1290108404 epmd: Thu Nov 18 19:26:44 2010: got 18 bytes ***** 00000000 00 10 78 7c e5 4d 00 00 05 00 05 00 03 66 6f 6f |..x|.M.......foo| ***** 00000010 00 00 |..| epmd: Thu Nov 18 19:26:44 2010: time in seconds: 1290108404 epmd: Thu Nov 18 19:26:44 2010: ** got ALIVE2_REQ epmd: Thu Nov 18 19:26:44 2010: ALIVE2_REQ from non local address epmd: Thu Nov 18 19:26:44 2010: closing connection on file descriptor 4 epmd: Thu Nov 18 19:26:49 2010: time in seconds: 1290108409 epmd: Thu Nov 18 19:26:54 2010: time in seconds: 1290108414 epmd: Thu Nov 18 19:26:59 2010: time in seconds: 1290108419 The inetrc file I was using was: {lookup,[file, dns]}. {host,{64,120,5,168}, ["mail.kepos.org"]}. {file, resolv, "/etc/resolv.conf"}. I then tried the same with the following inetrc version: {lookup,[file, dns]}. {host,{127,0,0,1}, ["localhost"]}. {file, resolv, "/etc/resolv.conf"}. # epmd -d -d -d epmd: Thu Nov 18 19:33:50 2010: epmd running - daemon = 0 epmd: Thu Nov 18 19:33:50 2010: try to initiate listening port 4369 epmd: Thu Nov 18 19:33:50 2010: starting epmd: Thu Nov 18 19:33:50 2010: entering the main select() loop epmd: Thu Nov 18 19:33:55 2010: time in seconds: 1290108835 epmd: Thu Nov 18 19:34:00 2010: time in seconds: 1290108840 epmd: Thu Nov 18 19:34:01 2010: Non-local peer connected epmd: Thu Nov 18 19:34:01 2010: time in seconds: 1290108841 epmd: Thu Nov 18 19:34:01 2010: opening connection on file descriptor 4 epmd: Thu Nov 18 19:34:01 2010: time in seconds: 1290108841 epmd: Thu Nov 18 19:34:01 2010: got 18 bytes ***** 00000000 00 10 78 3a f4 4d 00 00 05 00 05 00 03 66 6f 6f |..x:.M.......foo| ***** 00000010 00 00 |..| epmd: Thu Nov 18 19:34:01 2010: time in seconds: 1290108841 epmd: Thu Nov 18 19:34:01 2010: ** got ALIVE2_REQ epmd: Thu Nov 18 19:34:01 2010: ALIVE2_REQ from non local address epmd: Thu Nov 18 19:34:01 2010: closing connection on file descriptor 4 epmd: Thu Nov 18 19:34:06 2010: time in seconds: 1290108846 epmd: Thu Nov 18 19:34:11 2010: time in seconds: 1290108851 ^C Well, the expert are you, not me, but doesn't look this as epmd does not like to get the connection from a "non-local" address ? tcpdump in both cases did not put out anything helpful (see also my earlier posting). # tcpdump -i lo0 -vv port 4369 tcpdump: WARNING: lo0: no IPv4 address assigned tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 96 bytes 19:34:01.445154 IP (tos 0x0, ttl 64, id 42836, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->728)!) mail.58928 > mail.4369: Flags [S], cksum 0x1c18 (correct), seq 1795656002, win 65535, options [mss 16344,nop,wscale 3,sackOK,TS val 221277388 ecr 0], length 0 19:34:01.445174 IP (tos 0x0, ttl 64, id 42837, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->727)!) mail.4369 > mail.58928: Flags [S.], cksum 0xab70 (correct), seq 3890252666, ack 1795656003, win 65535, options [mss 16344,nop,wscale 3,sackOK,TS val 1967491061 ecr 221277388], length 0 19:34:01.445186 IP (tos 0x0, ttl 64, id 42838, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->72e)!) mail.58928 > mail.4369: Flags [.], cksum 0xf15c (correct), seq 1, ack 1, win 8960, options [nop,nop,TS val 221277388 ecr 1967491061], length 0 19:34:01.445214 IP (tos 0x0, ttl 64, id 42839, offset 0, flags [DF], proto TCP (6), length 70, bad cksum 0 (->71b)!) mail.58928 > mail.4369: Flags [P.], cksum 0x07d5 (correct), seq 1:19, ack 1, win 8960, options [nop,nop,TS val 221277388 ecr 1967491061], length 18 19:34:01.445357 IP (tos 0x0, ttl 64, id 42846, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->726)!) mail.4369 > mail.58928: Flags [F.], cksum 0xf149 (correct), seq 1, ack 19, win 8960, options [nop,nop,TS val 1967491061 ecr 221277388], length 0 19:34:01.445368 IP (tos 0x0, ttl 64, id 42847, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->725)!) mail.58928 > mail.4369: Flags [.], cksum 0xf149 (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 221277388 ecr 1967491061], length 0 19:34:01.445391 IP (tos 0x0, ttl 64, id 42850, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->722)!) mail.58928 > mail.4369: Flags [F.], cksum 0xf148 (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 221277388 ecr 1967491061], length 0 19:34:01.445400 IP (tos 0x0, ttl 64, id 42851, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->721)!) mail.4369 > mail.58928: Flags [.], cksum 0xf149 (correct), seq 2, ack 20, win 8959, options [nop,nop,TS val 1967491061 ecr 221277388], length 0 ^C 8 packets captured 8 packets received by filter 0 packets dropped by kernel So, I still guess, there might be a problem as Erlang somehow insists on using localhost solely while this isn't a good thing for FreeBSD Jails as Jails just have no fully functionable localhost (127.0.0.1 and locahost exist and answer for pings, yes, but there are limitations nonetheless). If there was a way to make Erlang use any configurable IP instead of localhost, the issue almost probably was resolved. I therefor tried to vary the content of the inetrc file but it seems, that's not enough to really point Erlang to the true IP address. As you asked for mail.kepos.org and 64.120.5.168: These are the Jails hostname and single IP address as also properly used by Postfix, Dovecot and MySQl in the same jail. I meanwile got a hint from the freebsd-questions Mailinglist: The person posting there mentioned, they would use Erlang 13B to run ejabberd 2.1.5 in reeBSD Jails without any issue as they were faced the same issue earlier as me now (with erlang 14B). Also this would match my earlier experience with an older Erlang half a year ago: No problems that time (but cannot remember which Erlang version I uses). Even I of course cannot be sure, the recent changes in Erlang 14B you mentioned might be the cause of all this. But as I have no clue about Erlang I only want to mention this as one possibility. What do you think ? kind regards Tom On Wednesday 17 November 2010 15:48:54 Michael Santos wrote: > On Wed, Nov 17, 2010 at 04:59:52AM +0100, tom@REDACTED wrote: > > > It might help. I think the quickest way to debug this would be to: > > donesults below ... > > Did you run these steps concurrently or sequentially? e.g., when you > brought up the erlang node, was the debug epmd running? > > epmd (with debug switches) needs to be running in one shell, the > tcpdump's (with the "-n" switch) in another. Then start up the Erlang > node. > > > # tcpdump -i lo0 -vv port 4369 > > tcpdump: WARNING: lo0: no IPv4 address assigned > > tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size > > 96 bytes > > Was the output below from the dump of the loopback? > > > and tcpdump output: > > > > 03:33:15.964680 IP (tos 0x0, ttl 64, id 24389, offset 0, flags [DF], > > proto TCP (6), length 60, bad cksum 0 (->4f37)!) > > > > mail.kepos.org.10975 > mail.kepos.org.4369: Flags [S], cksum 0xd692 > > > > (correct), seq 3123827793, win 65535, options [mss 16344,nop,wscale > > 3,sackOK,TS val 77847542 ecr 0], length 0 > > 03:33:15.964701 IP (tos 0x0, ttl 64, id 24390, offset 0, flags [DF], > > proto TCP (6), length 60, bad cksum 0 (->4f36)!) > > > > mail.kepos.org.4369 > mail.kepos.org.10975: Flags [S.], cksum > > 0x776a > > > > (correct), seq 3252621940, ack 3123827794, win 65535, options [mss > > 16344,nop,wscale 3,sackOK,TS val 2304704868 ecr 77847542], length 0 > > 03:33:15.964712 IP (tos 0x0, ttl 64, id 24391, offset 0, flags [DF], > > proto TCP (6), length 52, bad cksum 0 (->4f3d)!) > > > > mail.kepos.org.4369 > mail.kepos.org.10975: Flags [F.], cksum > > 0xbd43 > > > > (correct), seq 1, ack 19, win 8960, options [nop,nop,TS val 2304704868 > > ecr 77847542], length 0 > > 03:33:15.964906 IP (tos 0x0, ttl 64, id 24394, offset 0, flags [DF], > > proto TCP (6), length 52, bad cksum 0 (->4f3a)!) > > > > mail.kepos.org.10975 > mail.kepos.org.4369: Flags [.], cksum 0xbd43 > > > > (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 > > ecr 2304704868], length 0 > > 03:33:15.964929 IP (tos 0x0, ttl 64, id 24395, offset 0, flags [DF], > > proto TCP (6), length 52, bad cksum 0 (->4f39)!) > > > > mail.kepos.org.10975 > mail.kepos.org.4369: Flags [F.], cksum > > 0xbd42 > > > > (correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 > > ecr 2304704868], length 0 > > 03:33:15.964937 IP (tos 0x0, ttl 64, id 24396, offset 0, flags [DF], > > proto TCP (6), length 52, bad cksum 0 (->4f38)!) > > > > mail.kepos.org.4369 > mail.kepos.org.10975: Flags [.], cksum 0xbd43 > > > > (correct), seq 2, ack 20, win 8959, options [nop,nop,TS val 2304704868 > > ecr 77847542], length 0 > > ^C > > Do not resolve IP addresses. We need to see the source and destination > IP addresses. > > So something on port 4369 (on whatever "mail.kepos.org" is) is accepting > and closing the connection. If it is not epmd (because there is nothing > in your debug log), what is it? > > > Juust to exde any misconfiguation on the machine I allso tried itt > > again switching off the firewall for this test: same result. > > Always a good idea. > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From klacke@REDACTED Thu Nov 18 23:04:25 2010 From: klacke@REDACTED (Claes Wikstrom) Date: Thu, 18 Nov 2010 23:04:25 +0100 Subject: [erlang-questions] What's a solid way to have erlang automatically send email. In-Reply-To: References: Message-ID: <4CE5A2E9.8090806@hyber.org> On 11/17/2010 05:04 PM, Wes James wrote: > On Sun, Nov 14, 2010 at 4:31 PM, G.S. wrote: >> Hello, >> >> I have a website where the administrator needs to receive email alert (to >> login and fix a problem) when certain events occur. Is there an Erlang >> module that allows one to easily send out an email? Or something that can be >> used in the same manner? >> >> Thanks, >> -Gene >> > > This might work for you: > > https://github.com/Vagabond/gen_smtp And this, which is code I've used over and over again. https://github.com/klacke/yaws/blob/master/applications/mail/src/smtp.erl /klacke From michael.santos@REDACTED Fri Nov 19 00:03:55 2010 From: michael.santos@REDACTED (Michael Santos) Date: Thu, 18 Nov 2010 18:03:55 -0500 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <201011182055.40721.tom@diogunix.com> References: <201011161114.01488.tom@diogunix.com> <201011170459.52991.tom@diogunix.com> <20101117144854.GB29358@ecn.lan> <201011182055.40721.tom@diogunix.com> Message-ID: <20101118230355.GC32288@ecn.lan> On Thu, Nov 18, 2010 at 08:55:40PM +0100, tom@REDACTED wrote: > Ok, to be precisely I repeated the test: > > I first started tcpdum and let it run. > I then started epmd -d -d -d in a second shell. > I finally started erl -name foo in a third shell. Awesome, this is exactly what I needed to see! Thanks for being so patient with this. > epdm -d -d -d now spit out something more informative: > ***** 00000000 00 10 78 7c e5 4d 00 00 05 00 05 00 03 66 6f 6f > |..x|.M.......foo| > ***** 00000010 00 00 |..| > epmd: Thu Nov 18 19:26:44 2010: time in seconds: 1290108404 > epmd: Thu Nov 18 19:26:44 2010: ** got ALIVE2_REQ > epmd: Thu Nov 18 19:26:44 2010: ALIVE2_REQ from non local address > epmd: Thu Nov 18 19:26:44 2010: closing connection on file descriptor 4 So it's confirmed the problem was that the source address was not the one epmd expects. > The inetrc file I was using was: > > {lookup,[file, dns]}. > {host,{64,120,5,168}, ["mail.kepos.org"]}. > {file, resolv, "/etc/resolv.conf"}. inetrc is used for DNS resolution. It won't affect this particular issue. > tcpdump: WARNING: lo0: no IPv4 address assigned > tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 96 > bytes > 19:34:01.445154 IP (tos 0x0, ttl 64, id 42836, offset 0, flags [DF], proto > TCP (6), length 60, bad cksum 0 (->728)!) > mail.58928 > mail.4369: Flags [S], cksum 0x1c18 (correct), seq > 1795656002, win 65535, options [mss 16344,nop,wscale 3,sackOK,TS val > 221277388 ecr 0], length 0 > As you asked for mail.kepos.org and 64.120.5.168: > So, I still guess, there might be a problem as Erlang somehow insists on > using localhost solely while this isn't a good thing for FreeBSD Jails as > Jails just have no fully functionable localhost (127.0.0.1 and locahost > exist and answer for pings, yes, but there are limitations nonetheless). Thanks! I've never used FreeBSD jails, so I was confused about how they work. Erlang nodes are hard coded to connect to an epmd port on 127.0.0.1. The jail apparently re-writes connections to localhost with the IP address of the interface the jail is bound to. This behaviour is really sort of nasty ;) > If there was a way to make Erlang use any configurable IP instead of > localhost, the issue almost probably was resolved. I've attached a patch that disables the check. I'll put together a better patch later. I can see a few ways of doing this: 1. have a configurable source IP address as you suggested 2. checking the source IP is the same as the destination IP 3. checking the connection came over the loopback interface (probably no portable way to do this) 4. have an option to disable the check (the old behaviour) Aside from jails, I'm not sure anyone else would be affected by this. So maybe option 4 is the way to go. diff --git a/erts/epmd/src/epmd_srv.c b/erts/epmd/src/epmd_srv.c index ef471a4..e2cc2dc 100644 --- a/erts/epmd/src/epmd_srv.c +++ b/erts/epmd/src/epmd_srv.c @@ -766,6 +766,9 @@ static int conn_open(EpmdVars *g,int fd) dbg_tty_printf(g,2,(s->local_peer) ? "Local peer connected" : "Non-local peer connected"); + /* XXX allow local messages from all clients */ + s->local_peer = EPMD_TRUE; + s->want = 0; /* Currently unknown */ s->got = 0; s->mod_time = current_time(g); /* Note activity */ From tom@REDACTED Fri Nov 19 00:59:02 2010 From: tom@REDACTED (tom@REDACTED) Date: Fri, 19 Nov 2010 00:59:02 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101118230355.GC32288@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011182055.40721.tom@diogunix.com> <20101118230355.GC32288@ecn.lan> Message-ID: <201011190059.02797.tom@diogunix.com> Hello Michael, > I've attached a patch that disables the check. You're an angel, many thanks ! :-) > I'll put together a better patch later. I can see a few ways of doing > this: > > 1. have a configurable source IP address as you suggested > > 2. checking the source IP is the same as the destination IP > > 3. checking the connection came over the loopback interface (probably no > portable way to do this) > > 4. have an option to disable the check (the old behaviour) > > Aside from jails, I'm not sure anyone else would be affected by this. So > maybe option 4 is the way to go. Regarding FreeBSD's Jail virtualizatioin: Well, this is a grown sort of software virtualization which first appeared several years ago (probably long before the current Linux virtualization technologies popped up). Also, there's steady work to improve it's capabilities. However it's a pretty solid thing and worth supporting it as there are huge amounts of people and companies (such as Yahoo) outside there using it to propperly isolate different services running on a host. And for countless further purposes. May be you also just want to honor the good and old and true and "real" Unix: BSD ;-) I agree that your option #4 was good on the short run. As I don't know enough about other virtualization technologies outside the BSD world I just could guess that your option #1 or #2 might be best on the longer run. Also, I don't know for what purpose the Erlang community built in the check. There might have been good reasons to do so of course. Ok, I'll try to apply your patch now but also will contact the Port maintainer from the FreeBSD universe to keep him posted. Will CC you on this. Many thanks again ! kind regards Tom > diff --git a/erts/epmd/src/epmd_srv.c b/erts/epmd/src/epmd_srv.c > index ef471a4..e2cc2dc 100644 > --- a/erts/epmd/src/epmd_srv.c > +++ b/erts/epmd/src/epmd_srv.c > @@ -766,6 +766,9 @@ static int conn_open(EpmdVars *g,int fd) > dbg_tty_printf(g,2,(s->local_peer) ? "Local peer connected" : > "Non-local peer connected"); > > + /* XXX allow local messages from all clients */ > + s->local_peer = EPMD_TRUE; > + > s->want = 0; /* Currently unknown */ > s->got = 0; > s->mod_time = current_time(g); /* Note activity */ From toanhuyle.vn@REDACTED Fri Nov 19 02:50:34 2010 From: toanhuyle.vn@REDACTED (Toan H. Le) Date: Fri, 19 Nov 2010 08:50:34 +0700 Subject: [erlang-questions] Questions of configuring on SUSE Linux Enterprise 11 In-Reply-To: <4CE42019.200@gmail.com> References: <4CE42019.200@gmail.com> Message-ID: Thanks Greg. I can install from your repository. On Thu, Nov 18, 2010 at 1:34 AM, Gregory Haskins wrote: > On 11/17/10 2:47 AM, Toan H. Le wrote: > > Hi, > > > > I try to compile Erlang R14B on SUSE 11 but I face errors of configuring. > > FYI: I have 14B already packaged for SUSE-11 (SP1), here: > > > http://download.opensuse.org/repositories/home:/ghaskins:/erlang/SLE_11_SP1/ > > I could easily add SLE-11.0 to the build-service, if you were looking > for supporting the older GMC. > > You can either just zypper ar my repo in, or at least take a look at the > .src.rpm for hints as to how it was compiled. > > -Greg > > -- Regards, Toan H. Le From tom@REDACTED Fri Nov 19 02:57:59 2010 From: tom@REDACTED (tom@REDACTED) Date: Fri, 19 Nov 2010 02:57:59 +0100 Subject: [erlang-questions] crash dump at ejabberd startup In-Reply-To: <20101118230355.GC32288@ecn.lan> References: <201011161114.01488.tom@diogunix.com> <201011182055.40721.tom@diogunix.com> <20101118230355.GC32288@ecn.lan> Message-ID: <201011190258.00123.tom@diogunix.com> Hello Michael, your patch worked like a charm :-) Many thanks again ! Tom > diff --git a/erts/epmd/src/epmd_srv.c b/erts/epmd/src/epmd_srv.c > index ef471a4..e2cc2dc 100644 > --- a/erts/epmd/src/epmd_srv.c > +++ b/erts/epmd/src/epmd_srv.c > @@ -766,6 +766,9 @@ static int conn_open(EpmdVars *g,int fd) > dbg_tty_printf(g,2,(s->local_peer) ? "Local peer connected" : > "Non-local peer connected"); > > + /* XXX allow local messages from all clients */ > + s->local_peer = EPMD_TRUE; > + > s->want = 0; /* Currently unknown */ > s->got = 0; > s->mod_time = current_time(g); /* Note activity */ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From alexey.v.romanov@REDACTED Fri Nov 19 10:25:41 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Fri, 19 Nov 2010 12:25:41 +0300 Subject: Some ei questions Message-ID: 1. ei.h says about ERL_SMALL_INTEGER_EXT etc. "Defines used for ei_get_type_internal() output; we don't want to export these". Are they likely to go away in next release or two? Should some other #defines be used for output of `ei_get_type()` (not `ei_get_type_internal()`)? 2. ERL_SMALL_INTEGER_EXT should be decoded with ei_decode_long() and ERL_INTEGER_EXT with ei_decode_longlong()? Or something different? Yours, Alexey Romanov From alexey.v.romanov@REDACTED Fri Nov 19 12:26:39 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Fri, 19 Nov 2010 14:26:39 +0300 Subject: Some ei questions In-Reply-To: References: Message-ID: And also, while I am at it: why is there mismatch between arguments of ei_get_type (int *size) and ei_decode_binary (long *len), given that documentation states "The size required can be fetched by ei_get_type()"? How can the size of a binary be obtained in case it doesn't fit into int? Yours, Alexey Romanov On Fri, Nov 19, 2010 at 12:25 PM, Alexey Romanov wrote: > 1. ei.h says about ERL_SMALL_INTEGER_EXT etc. "Defines used for > ei_get_type_internal() output; we don't want to export these". Are > they likely to go away in next release or two? Should some other > #defines be used for output of `ei_get_type()` (not > `ei_get_type_internal()`)? > > 2. ERL_SMALL_INTEGER_EXT should be decoded with ei_decode_long() and > ERL_INTEGER_EXT with ei_decode_longlong()? Or something different? > > Yours, Alexey Romanov > From andy.kriger@REDACTED Fri Nov 19 13:21:14 2010 From: andy.kriger@REDACTED (Andy Kriger) Date: Fri, 19 Nov 2010 07:21:14 -0500 Subject: OTP and database initialization Message-ID: Trying to wrap my head around the OTP ecosystem. For a brand-new application, which would need to initialize a data store by creating tables (Mneisa, Riak, PostgreSQL, whatever), is there an OTP way to handle that? Should that be managed within the application with a check in the application supervisor init function? Or perhaps by breaking the data access into its own gen_server whose supervisor can init tables if necessary? Or is this usually handled outside the application w a one-time script? A lot to digest in OTP, so I'm looking to the experienced for best practices. -- Be well, andy Welcome to http://householder-yogi.net On family, NYC, and practicing yoga. From vincent.dephily@REDACTED Fri Nov 19 13:39:56 2010 From: vincent.dephily@REDACTED (Vincent de Phily) Date: Fri, 19 Nov 2010 13:39:56 +0100 Subject: [job] Erlang/OTP job at MobileDevices Message-ID: <201011191339.57003.vincent.dephily@mobile-devices.fr> Hello, my company is going for another round of hiring for Erlang developers (among other skills). We had good success adressing this mailing list directly in the past, and I hope many of you can take this new opportunity to join us. MobileDevices is a technology-based company that offers a lot of great job opportunities. We work on all kind of state of the art technical projects around the in-car connected portable devices ? including our core product which is an open OS specific to in-car environment (think Android for telematics but better.) We are looking for engineers as well as for interns for 4 up to 8 months paid internships with the prospect of long term contracts at the end of a successful period. We would be interested in receiving CVs and application from students interested in such opportunity. Obviously we are looking for Erlang / Ruby expertise but you will find a more global offer below, this will give you a window into our organisation and work. To give you a bit of background, the company is already quite international, everybody developing for us speaks English and we already have 6 different nationalities represented (Chinese, Italian, Algerian, Swedish, Polish and French). MobileDevices is an international company and sell to 47 different countries and already have sales offices in Singapore, London, US, France, China and technical ones in China, Ireland and France. We have rapidly evolved into one of the world leader in telematics systems for vehicles. We are known to be very competitive and very innovative (awarded with "Global Telematics & Navigation Product Differentiation Innovation Award" by Frost & Sullivan in 2009, also got innovation wise awards by Deloite Technology, Anvar and Eureka in 2007, 2008 and 2009). We are looking for computer science engineers from the technical project manager to the Phd in graph theory. System team : You will work on all our platform as well as customers one. Porting the boot- loader, developing advanced memory/energy management as well as other drivers, enhancing the VM... You will work in a team of 8 engineers. Core team : The core software is an event driven component based architecture. It involves meta-programming (code generation) in Ruby or XSLT, high level services development (UDP/TCP protocols, dead reckoning algorithms, navigation routing engine...), drivers logic (2G and 3G modems integration, bluetooth modules, ...) and many other things (graphical engine, ...) in mostly C++ and some Java. You will work in a team of 8 engineers. Server team: Bridge devices to the vast ecosystem of consumer services (think Twitter, Facebook...), fleet management and data analytics. The communication server is built to scale to millions of devices with impressive reliability and versatility (Heterogenous cloud in Erlang and Ruby / Rails, PostgreSQL and NoSQL databases, bi-directional push via web services and XMPP). The team also provides a Widget SDK to remotely communicate with the device (Javascript, HTML / Ruby). You will work in a team of 7 engineers. Services team : Directly working with the graphical designers, server team and core team, you will develop in Java and Ruby applications that will then be found in our products (navigation application, chat application, live trafic, movie reservation...) You will work in a team of 5 engineers. SDK team: Every great OS needs a development platform. From the emulation with QEMU to the graphical designer in EMF or GMF you will participate to the elaboration of our Eclipse based SDK called Morpheus. You will work in a team of 3 engineers. Test team: Validation is a major issue for all development teams that create a lot of code. Creating tools that can validate the code, check the regressions, the hardware compatibility. Mostly developing them in Ruby some drivers are in C. You will work in a team of 5 engineers. You can send applications to myself or to career@REDACTED Please let me know if you need more information to make this happen. -- Vincent de Phily Mobile Devices +33 (0) 142 119 325 +353 (0) 85 710 6320 Warning This message (and any associated files) is intended only for the use of its intended recipient and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Any views or opinions presented are solely those of the author vincent.dephily@REDACTED and do not necessarily represent those of the company. Although the company has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. From Grant_Li@REDACTED Fri Nov 19 15:37:08 2010 From: Grant_Li@REDACTED (Li, Grant) Date: Fri, 19 Nov 2010 08:37:08 -0600 Subject: `erl -boot start_sasl` Question In-Reply-To: <605635FE3E01874EB59477D4D01CA9DFD30F0294@odcexch01.csg.csgsystems.com> References: <201011181646430563447@its3.ch> <605635FE3E01874EB59477D4D01CA9DFD30F0294@odcexch01.csg.csgsystems.com> Message-ID: <605635FE3E01874EB59477D4D01CA9DFD36B048D@odcexch01.csg.csgsystems.com> All, I got the following error when I ran the command `erl -boot start_sasl`: $ erl -boot start_sasl {"init terminating in do_boot",{'cannot get bootfile','start_sasl.boot'}} Crash dump was written to: erl_crash.dump init terminating in do_boot () start_sasl.boot exits in $ROOT/bin directory, where $ROOT is the root of the Erlang distribution. I have attached erl_crash.dump file. I would appreciate very much for your help. Grant Li CSG Systems, Inc. -----Original Message----- From: Li, Grant Sent: Thursday, November 18, 2010 10:21 AM To: 'info'; 'erlang-questions' Subject: Build Erlang on Solaris 10 Hello Experts, I'm new to Erlang... I built Erlang on Solaris 10, with the following configure command: $ ./configure --enable-threads --host=sparc --build=sparc-sun-solaris2.10 --without-ssl erl_xcomp_bigendian=yes erl_x comp_linux_nptl=yes erl_xcomp_linux_usable_sigusrx=yes erl_xcomp_poll=yes erl_xcomp_putenv_copy=yes erl_xcomp_gethrvtime_procfs_ioctl=yes erl_xcomp_clock_gettime_cpu_time=yes erl_xcomp_dlsym_brk_wrappers=yes When the make is completed, I observed the following issues: 1. Erlang was build in both sparc-sun-solaris2.10 and sparc-sun-sunos4.1.1. Does anyone know where sunos4.1.1 was coming from? The binaries in otp_src_R14B/bin/sparc-sun-solaris2.10 and otp_src_R14B/bin/sparc-sun-sunos4.1.1 have different sizes; 2. I wondered if erl was not built correctly. Two erl files were found: -rwxr-xr-x 1 ligr01 etsi 937 Nov 12 16:24 otp_src_R14B/bootstrap/bin/erl -rwxrwxr-x 1 ligr01 etsi 1448 Sep 13 12:00 otp_src_R14B/erts/etc/win32/cygwin_tools/erl For my understanding, erl should exist in otp_src_R14B/bin directory. Thank you, Grant Li CSG Systems, Inc -------------- next part -------------- A non-text attachment was scrubbed... Name: erl_crash.dump Type: application/octet-stream Size: 37340 bytes Desc: erl_crash.dump URL: From info@REDACTED Fri Nov 19 16:15:58 2010 From: info@REDACTED (info) Date: Fri, 19 Nov 2010 16:15:58 +0100 Subject: [erlang-questions] end of process References: <201011181646430563447@its3.ch>, <4CE607A2.6070307@cmg.jovenclub.cu> Message-ID: <201011191615550934750@its3.ch> Thank you Ivan for this explanation. If I resume: exit for abnormal end and ok or nothing for normal end. trap to the parent level. J-Ph. Constantin ITS3 Gen?ve www.its3.ch From serge@REDACTED Fri Nov 19 16:44:45 2010 From: serge@REDACTED (Serge Aleynikov) Date: Fri, 19 Nov 2010 10:44:45 -0500 Subject: [erlang-questions] Re: Some ei questions In-Reply-To: References: Message-ID: <4CE69B6D.7040303@aleynikov.org> 1. External term encoding format treats small integers fitting in one byte differently from larger ones, and therefore has a special encoding type for them. From the definition of this encoding it's highly unlikely that that would change as such a change would not be backward compatible. 2. You can decode it with either one of the two functions. The difference between these functions is in the amount of space you give them to store the decoded result, but they are smart enough to handle different types of integers. 3. Even though ei_get_type() and ei_decode_binary() specify different types of length (on 64-bit platforms), a binary value is limited to 4G, and therefore in practice this discrepancy is irrelevant (though not very elegant). Serge On 11/19/2010 6:26 AM, Alexey Romanov wrote: > And also, while I am at it: why is there mismatch between arguments of > ei_get_type (int *size) and ei_decode_binary (long *len), given that > documentation states "The size required can be fetched by > ei_get_type()"? How can the size of a binary be obtained in case it > doesn't fit into int? > > Yours, Alexey Romanov > > > > On Fri, Nov 19, 2010 at 12:25 PM, Alexey Romanov > wrote: >> 1. ei.h says about ERL_SMALL_INTEGER_EXT etc. "Defines used for >> ei_get_type_internal() output; we don't want to export these". Are >> they likely to go away in next release or two? Should some other >> #defines be used for output of `ei_get_type()` (not >> `ei_get_type_internal()`)? >> >> 2. ERL_SMALL_INTEGER_EXT should be decoded with ei_decode_long() and >> ERL_INTEGER_EXT with ei_decode_longlong()? Or something different? >> >> Yours, Alexey Romanov >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From alexey.v.romanov@REDACTED Fri Nov 19 17:06:47 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Fri, 19 Nov 2010 19:06:47 +0300 Subject: [erlang-questions] Re: Some ei questions In-Reply-To: <4CE69B6D.7040303@aleynikov.org> References: <4CE69B6D.7040303@aleynikov.org> Message-ID: Thank you for http://www.trapexit.org/How_to_use_ei_to_marshal_binary_terms_in_port_programs which is what got me started with ei. On Fri, Nov 19, 2010 at 6:44 PM, Serge Aleynikov wrote: > 1. External term encoding format treats small integers fitting in one byte > differently from larger ones, and therefore has a special encoding type for > them. ?From the definition of this encoding it's highly unlikely that that > would change as such a change would not be backward compatible. > > 2. You can decode it with either one of the two functions. ?The difference > between these functions is in the amount of space you give them to store the > decoded result, but they are smart enough to handle different types of > integers. Got it. > 3. Even though ei_get_type() and ei_decode_binary() specify different types > of length (on 64-bit platforms), a binary value is limited to 4G, and > therefore in practice this discrepancy is irrelevant (though not very > elegant). This still leaves us with binaries over 2G. I believe the overflow works out correctly anyway. > Serge > > On 11/19/2010 6:26 AM, Alexey Romanov wrote: >> >> And also, while I am at it: why is there mismatch between arguments of >> ei_get_type (int *size) and ei_decode_binary (long *len), given that >> documentation states "The size required can be fetched by >> ei_get_type()"? How can the size of a binary be obtained in case it >> doesn't fit into int? >> >> Yours, Alexey Romanov >> >> >> >> On Fri, Nov 19, 2010 at 12:25 PM, Alexey Romanov >> ?wrote: >>> >>> 1. ei.h says about ERL_SMALL_INTEGER_EXT etc. "Defines used for >>> ei_get_type_internal() output; we don't want to export these". Are >>> they likely to go away in next release or two? Should some other >>> #defines be used for output of `ei_get_type()` (not >>> `ei_get_type_internal()`)? >>> >>> 2. ERL_SMALL_INTEGER_EXT should be decoded with ei_decode_long() and >>> ERL_INTEGER_EXT with ei_decode_longlong()? Or something different? >>> >>> Yours, Alexey Romanov >>> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From jeff.zellner@REDACTED Fri Nov 19 18:30:01 2010 From: jeff.zellner@REDACTED (Jeff Zellner) Date: Fri, 19 Nov 2010 12:30:01 -0500 Subject: Blocking NIFs Message-ID: While I have read the erl_nif documentation (that says "Avoid doing lengthy work in NIF calls as that may degrade the responsiveness of the VM. NIFs are called directly by the same scheduler thread that executed the calling Erlang code. The calling scheduler will thus be blocked from doing any other work until the NIF returns.") -- I'm trying to do keyboard input in a NIF. Is this absolutely a bad idea, or is there a way to make it work? I've found that simple example programs, running ~2 processes, one of them blocking on a getch() in a NIF works fine -- unfortunately trying to integrate that blocking NIF call into a larger program with upwards of 10 processes seems to not work at all, regardless of how many async threads I use. Can anyone give me some insight into this? Preferably something other than 'don't use NIFs to do that' -- I will eventually give up if it's just not workable, but I'm not ready to do that yet! :-) Cheers, Jeff From info@REDACTED Fri Nov 19 18:56:20 2010 From: info@REDACTED (info) Date: Fri, 19 Nov 2010 18:56:20 +0100 Subject: How to build a dynamic list of TCP clients ? Message-ID: <201011191856197030048@its3.ch> Hello, In a TCP/IP gen_server I do: Pid = spawn_link(... process_accept ...) My first idea was to create a record and to insert the Pid just after the spawn_link: Pid = spawn_link(... process_accept ...), C = #clients(pid=Pid,name=0,s=Socket), #name will be initialized by the process_accept [C | ListClients]; Now I have my list of clients but a client can disappear (tcp_closed) and the list must be updated. Is it a good approach or do you have better ! My objective is to give the possibility to send info to one client from another module by sending to the gen_server (handle_cast) the following command: send_to_client(Name,Message) Next gen_server shall find the corresponding process and send the message by gen_tcp:send(S,Message) J-Ph. Constantin ITS3 Gen?ve www.its3.ch From rapsey@REDACTED Fri Nov 19 19:05:49 2010 From: rapsey@REDACTED (Rapsey) Date: Fri, 19 Nov 2010 19:05:49 +0100 Subject: [erlang-questions] Blocking NIFs In-Reply-To: References: Message-ID: I don't understand exactly what you mean by "async threads". I presume you don't mean blocking within a NIF thread (created with enif_thread_create), because that would work and it is how you should be doing it. Create a NIF thread, block there. Once you get an event, send a message from that thread to an erlang process. Sergej On Fri, Nov 19, 2010 at 6:30 PM, Jeff Zellner wrote: > While I have read the erl_nif documentation (that says "Avoid doing lengthy > work in NIF calls as that may degrade the responsiveness of the VM. NIFs > are > called directly by the same scheduler thread that executed the calling > Erlang code. The calling scheduler will thus be blocked from doing any > other > work until the NIF returns.") -- I'm trying to do keyboard input in a NIF. > > Is this absolutely a bad idea, or is there a way to make it work? I've > found > that simple example programs, running ~2 processes, one of them blocking on > a getch() in a NIF works fine -- unfortunately trying to integrate that > blocking NIF call into a larger program with upwards of 10 processes seems > to not work at all, regardless of how many async threads I use. > > Can anyone give me some insight into this? Preferably something other than > 'don't use NIFs to do that' -- I will eventually give up if it's just not > workable, but I'm not ready to do that yet! :-) > > Cheers, > Jeff > From jeff.zellner@REDACTED Fri Nov 19 19:05:45 2010 From: jeff.zellner@REDACTED (Jeff Zellner) Date: Fri, 19 Nov 2010 13:05:45 -0500 Subject: Blocking NIFs In-Reply-To: References: Message-ID: To clarify a bit: I'm writing an ncurses interface in NIFs, and using it running with -noinput/-noshell. I'm looking to capture actual key presses, not prompted input. I'd like to avoid dealing with a port program or a driver just for the single blocking operation I want to do. On Fri, Nov 19, 2010 at 12:30 PM, Jeff Zellner wrote: > While I have read the erl_nif documentation (that says "Avoid doing lengthy > work in NIF calls as that may degrade the responsiveness of the VM. NIFs are > called directly by the same scheduler thread that executed the calling > Erlang code. The calling scheduler will thus be blocked from doing any other > work until the NIF returns.") -- I'm trying to do keyboard input in a NIF. > > Is this absolutely a bad idea, or is there a way to make it work? I've > found that simple example programs, running ~2 processes, one of them > blocking on a getch() in a NIF works fine -- unfortunately trying to > integrate that blocking NIF call into a larger program with upwards of 10 > processes seems to not work at all, regardless of how many async threads I > use. > > Can anyone give me some insight into this? Preferably something other than > 'don't use NIFs to do that' -- I will eventually give up if it's just not > workable, but I'm not ready to do that yet! :-) > > Cheers, > Jeff > From jeff.zellner@REDACTED Fri Nov 19 19:10:04 2010 From: jeff.zellner@REDACTED (Jeff Zellner) Date: Fri, 19 Nov 2010 13:10:04 -0500 Subject: [erlang-questions] Blocking NIFs In-Reply-To: References: Message-ID: enif_thread_create sounds like what I'm looking for. I will give it a shot, Thanks! -Jeff On Fri, Nov 19, 2010 at 1:05 PM, Rapsey wrote: > I don't understand exactly what you mean by "async threads". I presume you > don't mean blocking within a NIF thread (created with enif_thread_create), > because that would work and it is how you should be doing it. > Create a NIF thread, block there. Once you get an event, send a message > from that thread to an erlang process. > > > Sergej > > > On Fri, Nov 19, 2010 at 6:30 PM, Jeff Zellner wrote: > >> While I have read the erl_nif documentation (that says "Avoid doing >> lengthy >> work in NIF calls as that may degrade the responsiveness of the VM. NIFs >> are >> called directly by the same scheduler thread that executed the calling >> Erlang code. The calling scheduler will thus be blocked from doing any >> other >> work until the NIF returns.") -- I'm trying to do keyboard input in a NIF. >> >> Is this absolutely a bad idea, or is there a way to make it work? I've >> found >> that simple example programs, running ~2 processes, one of them blocking >> on >> a getch() in a NIF works fine -- unfortunately trying to integrate that >> blocking NIF call into a larger program with upwards of 10 processes seems >> to not work at all, regardless of how many async threads I use. >> >> Can anyone give me some insight into this? Preferably something other >> than >> 'don't use NIFs to do that' -- I will eventually give up if it's just not >> workable, but I'm not ready to do that yet! :-) >> >> Cheers, >> Jeff >> > > From jesper.louis.andersen@REDACTED Fri Nov 19 19:35:15 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Fri, 19 Nov 2010 19:35:15 +0100 Subject: [erlang-questions] How to build a dynamic list of TCP clients ? In-Reply-To: <201011191856197030048@its3.ch> References: <201011191856197030048@its3.ch> Message-ID: On Fri, Nov 19, 2010 at 6:56 PM, info wrote: > In a TCP/IP gen_server I do: > Pid = spawn_link(... process_accept ...) I will assume that your gen server is a *listener* in the sense that is has called gen_tcp:listen/2 on a socket. You then spawn_link other processes to call gen_tcp:accept/1 on this ListenSocket and accept connections. If I am wrong, please correct me. > My first idea was to create a record and to insert the Pid just after the spawn_link: > Pid = spawn_link(... process_accept ...), > C = #clients(pid=Pid,name=0,s=Socket), ? #name will be initialized by the process_accept > [C | ListClients]; > > Now I have my list of clients but a client can disappear (tcp_closed) and the list must be updated. > Is it a good approach or do you have better ! Yes, you will need to do some bookkeeping when clients disappear. Your current setup is that the cliens are spawn-linked to the listener. By default, if you do not set the trap_exit process flag, it means that if one of the clients errs, then your whole tree dies. I generally do not recommend this approach since it isn't very robust. I don't recommend you trap_exits as well, but rather add a client_pool supervisor process running as simple_one_for_one to your supervisor tree. This means that the death of a client won't affect your, but the pool supervisor -- and supervisors are built to handle crashes of processes. The next part is that you call erlang:monitor/2 on the Pid you get back from supervisor:start_child/2. In effect, if you lose a client you will get a 'DOWN' message you can grab in ?MODULE:handle_info/2 of your listening server and thus remove the client from your client list. In the long run, you want to eliminate the serialization on having to go through the listen server on each client lookup. Let the listen server maintain ListClients as a protected ETS table so other processes can look up pid information without having to explicitly call into the listener gen_server. A slightly more general variant use Ulf Wigers Gproc process table for storing the processes. > My objective is to give the possibility to send info to one client from another module by sending to the gen_server (handle_cast) the following command: send_to_client(Name,Message) > Next gen_server shall find the corresponding process and send the message by gen_tcp:send(S,Message) My mantra is to do this without involving the listener at all. Take the ETS variant from above (if you really go down this path, GProc is way easier in the long run). Then in the listener module (freely written, not tested): -module(listener). -behaviour(gen_server). [..] send_to_client(Name, Msg) -> P = ets:lookup_element(?MODULE, Name, #client.pid), % Crash if Name isn't there client:send_msg(P, Msg). and then in module client: -module(client). -behaviour(gen_server). [..] send_msg(P, Msg) -> gen_server:cast(Pid, {msg, Msg}). handle_cast({msg, M}, #state { socket = S } = State) -> gen_tcp:send(S, encode(M)), {noreply, State}; [..] handle_cast(Unknown, State) -> error_logger:info_report({client_unknown_msg, Unknown}), {noreply, State}. Ie, the client governs the socket completely and the server is never in on the Socket. Notice that you can give away a socket to another process through gen_tcp:controlling_process/2. This setup tend to be vastly simpler than having a single server "demuxing" amongst several connections. And it is faster too, since there is no lookup and demux necessary anymore: Data goes straight to where it belongs. If you have further questions, feel free to ask. -- J. From mazen.harake@REDACTED Fri Nov 19 21:55:12 2010 From: mazen.harake@REDACTED (Mazen Harake) Date: Fri, 19 Nov 2010 22:55:12 +0200 Subject: [erlang-questions] Re: Blocking NIFs In-Reply-To: References: Message-ID: <4CE6E430.4040508@erlang-solutions.com> Just curious; are you aware of cecho? https://github.com/mazenharake/cecho BR, /Mazen On 19/11/2010 20:05, Jeff Zellner wrote: > To clarify a bit: > > I'm writing an ncurses interface in NIFs, and using it running with > -noinput/-noshell. I'm looking to capture actual key presses, not prompted > input. > > I'd like to avoid dealing with a port program or a driver just for the > single blocking operation I want to do. > > On Fri, Nov 19, 2010 at 12:30 PM, Jeff Zellnerwrote: > >> While I have read the erl_nif documentation (that says "Avoid doing lengthy >> work in NIF calls as that may degrade the responsiveness of the VM. NIFs are >> called directly by the same scheduler thread that executed the calling >> Erlang code. The calling scheduler will thus be blocked from doing any other >> work until the NIF returns.") -- I'm trying to do keyboard input in a NIF. >> >> Is this absolutely a bad idea, or is there a way to make it work? I've >> found that simple example programs, running ~2 processes, one of them >> blocking on a getch() in a NIF works fine -- unfortunately trying to >> integrate that blocking NIF call into a larger program with upwards of 10 >> processes seems to not work at all, regardless of how many async threads I >> use. >> >> Can anyone give me some insight into this? Preferably something other than >> 'don't use NIFs to do that' -- I will eventually give up if it's just not >> workable, but I'm not ready to do that yet! :-) >> >> Cheers, >> Jeff >> From pablo.platt@REDACTED Sat Nov 20 01:18:52 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Fri, 19 Nov 2010 16:18:52 -0800 (PST) Subject: pagination (skip, limit) ets items (gproc pids) Message-ID: <804797.46106.qm@web112604.mail.gq1.yahoo.com> Hi I'm using gproc to save user sessions. I want to present a list of online users in an admin html page. I have a large number of sessions. Is it possible to paginate the list similar to how SQL use skip and limit? The admin will see a list of 1-10 online users and have links previous and next to see previous and next group of users. I can't use ets continuation because I'll have to save it and reuse it which is hard in a website. The only way I see to solve it is to use ets:match(Tab, Pattern, Limit) several times. For example, if the admin requests for users 50-60 I'll do ets:match 5 times when using the returned continuation. Is there a better way? Is there a reason why Skip isn't implemented even if it'll be implemented like a suggested above? Thanks From robert.virding@REDACTED Sat Nov 20 01:52:02 2010 From: robert.virding@REDACTED (Robert Virding) Date: Sat, 20 Nov 2010 00:52:02 +0000 (GMT) Subject: [erlang-questions] end of process In-Reply-To: <2010280338.105921290214061404.JavaMail.root@zimbra> Message-ID: <100849688.105941290214322716.JavaMail.root@zimbra> Whether calling exit/1 results in a normal or abnormal process termination depends solely on the exit value. Calling exit(normal) results in a normal termination and is exactly equivalent to the spawned function running to its end. The return value from the spawned function is completely ignored and is lost. A process does not "return a value" and if you want to get some value from it you need to send it in a message.* Calling exit/1 with any other value results in an abnormal process termination which is equivalent to an exception occurring in the process. The exit value is not returned anywhere but passed as an exit signal to processes which are linked to the exiting process. Robert * Or by cunning use of exit values and signals. :-) ----- "info" wrote: > Thank you Ivan for this explanation. > If I resume: exit for abnormal end and ok or nothing for normal end. > trap to the parent level. > > J-Ph. Constantin > ITS3 Gen?ve > www.its3.ch -- Robert Virding, Erlang Solutions Ltd. From hynek@REDACTED Sat Nov 20 14:27:48 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Sat, 20 Nov 2010 14:27:48 +0100 Subject: [erlang-questions] LAST RESUME of Pytagoriam Numbers!!! In-Reply-To: References: <43C9CB7E.7080606@cmg.jovenclub.cu> Message-ID: (I'm just retuning from Stockholm and sitting at the Vienna airport and hacking for the fun, so ...) This version should be little bit faster than py3R (~30% in bytecode and ~20% in native for N=1000): pythag4a(N) when is_integer(N) -> pythag4a(N,1). pythag4a(N, A) when A+2 > N -> []; pythag4a(N, A) when A rem 2 =:= 0 -> pythag4a(N, A, A*A, 1, 1); pythag4a(N, A) -> pythag4a(N, A, A*A, 2, 2). pythag4a(N, A, _A2, B, _S) when A+B+1 > N -> pythag4a(N, A+1); pythag4a(N, A, A2, B, S) -> B2 = B*B, C = round(math:sqrt(A2+B2)), if A+B+C =< N, A2 + B2 =:= C*C -> [{A, B, C}|pythag4a(N, A, A2, B+S, S)]; true -> pythag4a(N, A, A2, B+S, S) end. On Mon, Nov 15, 2010 at 9:53 PM, Edmond Begumisa wrote: > Below is a parallelised version of Richard/H?kan's blazing and yet > easy-to-read algo using a max of 2000 processes (find lpmap function on > previous thread.) > > On my dual-core it's about 20% faster for when N is small (300) and gets up > to 40% faster when N is large (3,000). Evens out from there. I'm getting > about 4.5s for N = 5,000! > > I must say, I'm floored. I need to learn much, much, more about Erlang. I > just wasn't convinced it would be any good at this! And their algo retains > Garcia's original beauty too! It reads much like the high-school math > problems. > > py3R_p(N)-> > ? ?lists:flatten(lpmap(fun(A) -> > ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{A, B, C} || > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?B <- lists:seq(1, N - A), > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?C <- [trunc(math:sqrt(A * A + B * B))], > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?A + B + C =< N, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?A*A + B*B =:= C*C] > ? ? ? ? ? ? ? ? ? ? ? ?end, lists:seq(1, N div 2), 2000, ordered)). > > > - Edmond - > > > > H?kan Huss" > > > On Sun, 15 Jan 2006 15:11:42 +1100, Ivan Carmenates Garc?a > wrote: > >> >> Compiled using c(py) normal in the shell. >> >> Feel free to make a new Resume on a DualCore CPU!!! >> >> >> Now we are re-talking about!!! >> >> Tested on Intel(R) Celeron CPU 2.13 GHz, 192 of Ram >> >> *** Pytagoriam's Numbers REPORT! *** >> >> Using N = 300 >> >> *** THE ORIGINAL IMPLEMENTATIONS (ME!)**** >> -py1: ? ? ? ? 14860000 mcs = 14.86 s >> -py2: ? ? ? ? 14843999 mcs = 14.843999 s >> -py3: ? ? ? ? 12749999 mcs = 12.749999 s >> >> *** Tony's implementations **** >> -pythag1: ? ? 11702999 mcs = 11.702999 s >> -pythag2: ? ? 1983999 mcs = 1.983999 s >> >> *** Hynek's implementation **** >> Simpler and about 5% faster version >> -pythag3: ? ? 1952999 mcs = 1.952999 s >> >> *** Edmond's implementation, using parallelism**** >> -py2E: ? ? ? ?14843999 mcs = 14.843999 s >> >> *** Willem's implementation**** >> -wpythag2: ? ?1749999 mcs = 1.749999 s >> >> *** Hynek's new implementation**** >> -pythag4: ? ? 1140999 mcs = 1.140999 s >> >> *** Willem's new implementation in parallel by Edmond!!!**** >> -wpythag2P: ? 1795999 mcs = 1.795999 s >> >> *** Morten's implementation**** >> -pythag5: ? ? 46999 mcs = 0.046999 s >> >> *** Richard's improvement**** >> -py3R: ? ? ? ?46999 mcs = 0.046999 s ? *** NEW *** >> >> *** Joe's improvement**** >> -py3a: ? ? ? ?1577999 mcs = 1.577999 s ? ?*** NEW *** >> >> Comparisons in results agains 'pythag1' Tony's >> function >> py1 returns the same than 'pythag1' >> py2 returns the same than 'pythag1' >> py3 returns the same than 'pythag1' >> pythag1 returns the same than 'pythag1' >> pythag2 returns the same than 'pythag1' >> pythag3 returns the same than 'pythag1' >> py2E returns the same than 'pythag1' >> wpythag2 returns the same than 'pythag1' >> pythag4 returns the same than 'pythag1' >> wpythag2P returns the same than 'pythag1' >> pythag5 returns the same than 'pythag1' >> py3R returns the same than 'pythag1' <- now are we talking about!!! (is >> the best!!!) >> py3a returns the same than 'pythag1' >> >> NOTE: Time took using timer:tc/3 function. >> >> >> %******************* ? START CODE ********************* >> >> -module(py). >> -compile([export_all]). >> >> >> start(N)-> >> ? ? %% My implementations. >> ? ? {T1,R1} = timer:tc(py,py1, [N]), >> ? ? {T2,R2} = timer:tc(py,py2,[N]), >> ? ? {T3,R3} = timer:tc(py,py3,[N]), >> >> ? ? %% Tony's improvement of the original form 3. >> ? ? {T4,R4} = timer:tc(py,pythag1,[N]), >> ? ? %% Tony's implementation. >> ? ? {T5,R5} = timer:tc(py,pythag2,[N]), >> >> ? ? %% Hynek's implementation. >> ? ? %% Simpler and about 5% faster version: >> ? ? {T6,R6} = timer:tc(py,pythag3,[N]), >> >> ? ? %% Edmond's implementation using parallelism. >> ? ? {T7,R7} = timer:tc(py,py2E,[N]), >> >> ? ? %% Willem's implementation. >> ? ? {T8,R8} = timer:tc(py,wpythag2,[N]), >> >> ? ? %% Hynek's new version >> ? ? {T9,R9} = timer:tc(py,pythag4,[N]), >> >> ? ? %% Willem's implementation in parallel by Edmond. >> ? ? {T10,R10} = timer:tc(py,wpythag2P,[N]), >> >> ? ? %% Morten's implementation. >> ? ? {T11,R11} = timer:tc(py,pythag5,[N]), >> >> ? ? %% Richard's improvement. >> ? ? {T12,R12} = timer:tc(py,py3R,[N]), >> >> ? ? %% Joe's improvement. >> ? ? {T13,R13} = timer:tc(py,py3a,[N]), >> >> ? ? io:format("~n *** Pytagoriam's Numbers REPORT! ***~n~n"), >> ? ? io:format("Using N = ~p~n", [N]), >> >> ? ? io:format("~n*** THE ORIGINAL IMPLEMENTATIONS (ME!)****~n"), >> ? ? io:format("-py1: ? ? ? ? ~p mcs = ~p s~n", [T1,T1/1000000]), >> ? ? io:format("-py2: ? ? ? ? ~p mcs = ~p s~n", [T2,T2/1000000]), >> ? ? io:format("-py3: ? ? ? ? ~p mcs = ~p s~n", [T3,T3/1000000]), >> >> ? ? io:format("~n*** Tony's implementations ****~n"), >> ? ? io:format("-pythag1: ? ? ~p mcs = ~p s~n", [T4,T4/1000000]), >> ? ? io:format("-pythag2: ? ? ~p mcs = ~p s~n", [T5,T5/1000000]), >> >> ? ? io:format("~n*** Hynek's implementation ****~n"), >> ? ? io:format("Simpler and about 5% faster version~n"), >> ? ? io:format("-pythag3: ? ? ~p mcs = ~p s~n", [T6,T6/1000000]), >> >> ? ? io:format("~n*** Edmond's implementation, using parallelism****~n"), >> ? ? io:format("-py2E: ? ? ? ?~p mcs = ~p s~n", [T7,T7/1000000]), >> >> ? ? io:format("~n*** Willem's implementation****~n"), >> ? ? io:format("-wpythag2: ? ?~p mcs = ~p s~n", [T8,T8/1000000]), >> >> ? ? io:format("~n*** Hynek's new implementation****~n"), >> ? ? io:format("-pythag4: ? ? ~p mcs = ~p s~n", [T9,T9/1000000]), >> >> ? ? io:format("~n*** Willem's new implementation in parallel by >> Edmond!!!****~n"), >> ? ? io:format("-wpythag2P: ? ~p mcs = ~p s~n", [T10,T10/1000000]), >> >> ? ? io:format("~n*** Morten's implementation****~n"), >> ? ? io:format("-pythag5: ? ? ~p mcs = ~p s~n", [T11,T11/1000000]), >> >> ? ? io:format("~n*** Richard's improvement****~n"), >> ? ? io:format("-py3R: ? ? ? ?~p mcs = ~p s~n", [T12,T12/1000000]), >> >> ? ? io:format("~n*** Joe's improvement****~n"), >> ? ? io:format("-py3a: ? ? ? ?~p mcs = ~p s~n", [T13,T13/1000000]), >> >> ? ? io:format("~nComparisons in results agains 'pythag1' Tony's >> function~n"), >> ? ? Rs = [{py1,R1}, {py2,R2}, {py3,R3}, {pythag1,R4}, {pythag2,R5}, >> {pythag3,R6}, >> ? ? ? ? ? {py2E,R7}, {wpythag2,R8}, {pythag4,R9}, {wpythag2P,R10}, >> {pythag5,R11}, {py3R,R12}, >> ? ? ? ? ? {py3a,R13}], >> ? ? lists:foreach(fun({Name, R})-> >> ? ? ? ? if >> ? ? ? ? ? ? (R=:=R4)-> >> ? ? ? ? ? ? ? ? io:format("~p returns the same than 'pythag1'~n", [Name]); >> ? ? ? ? ? ? true-> >> ? ? ? ? ? ? ? ? io:format("~p NO returns the same than 'pythag1'~n", >> [Name]) >> ? ? ? ? end >> ? ? end, Rs), >> ? ? io:format("~nNOTE: Time took using timer:tc/3 function.~n"). >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% The original form 1. >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> py1(Max)-> >> ? ? L = lists:seq(1, Max), >> ? ? lists:foldr( >> ? ? ? ? fun(A, Acc3)-> >> ? ? ? ? ? ? lists:foldr( >> ? ? ? ? ? ? ? ? fun(B, Acc2)-> >> ? ? ? ? ? ? ? ? ? ? lists:foldr( >> ? ? ? ? ? ? ? ? ? ? ? ? fun(C, Acc)-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? case ((A*A + B*B =:= C*C) andalso (A+B+C =< >> Max)) of >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{A,B,C}|Acc]; >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? false-> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Acc >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? end >> ? ? ? ? ? ? ? ? ? ? ? ? end >> ? ? ? ? ? ? ? ? ? ? , Acc2, L) >> ? ? ? ? ? ? ? ? end >> ? ? ? ? ? ? , Acc3, L) >> ? ? ? ? end >> ? ? , [], L). >> >> >> >> %% The original form 2. >> py2(Max)-> >> ? ? lists:reverse(fora(1, [], Max)). >> >> fora(A, Acc, Max)-> >> ? ? Acc1 = forb(A,1, Acc, Max), >> ? ? case A< ?Max of >> ? ? ? ? true-> >> ? ? ? ? ? ? fora(A+1, Acc1, Max); >> ? ? ? ? false-> >> ? ? ? ? ? ? Acc1 >> ? ? end. >> >> forb(A,B, Acc, Max)-> >> ? ? Acc1 = forc(A,B,1, Acc, Max), >> ? ? case B< ?Max of >> ? ? ? ? true-> >> ? ? ? ? ? ? forb(A,B+1, Acc1, Max); >> ? ? ? ? false-> >> ? ? ? ? ? ? Acc1 >> ? ? end. >> >> forc(A,B,C, Acc, Max)-> >> ? ? Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< ?Max) of >> ? ? ? ? true-> >> ? ? ? ? ? ? [{A,B,C}|Acc]; >> ? ? ? ? _-> >> ? ? ? ? ? ? Acc >> ? ? end, >> ? ? case C< ?Max of >> ? ? ? ? true-> >> ? ? ? ? ? ? forc(A,B,C+1, Acc1, Max); >> ? ? ? ? false-> >> ? ? ? ? ? ? Acc1 >> ? ? end. >> >> %% The original form 3. >> py3(Max)-> >> ? ? [{A,B,C} || >> ? ? ? ? A<-lists:seq(1, Max), >> ? ? ? ? B<-lists:seq(1, Max), >> ? ? ? ? C<-lists:seq(1, Max), >> ? ? ? ? A*A + B*B =:= C*C, >> ? ? ? ? A+B+C =< ?Max]. >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Tony's improvement of the original form 3. >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> pythag1(N) -> >> ? ? L = lists:seq(1,N), >> ? ? [ {A,B,C} || >> ? ? ? ? A<- L, >> ? ? ? ? B<- L, >> ? ? ? ? C<- L, >> ? ? ? ? A+B+C =< ?N, >> ? ? ? ? A*A+B*B =:= C*C]. >> >> %% Tony's implementation. >> pythag2(N) -> >> ? ? lists:reverse(pythan2_A(1, N, [])). >> >> pythan2_A(A, N, Acc) when A> ?N -> ?Acc; >> pythan2_A(A, N, Acc) -> ?pythan2_A(A+1,N,pythan2_B(A, 1, N, Acc)). >> >> pythan2_B(A, B, N, Acc) when A+B> ?N -> ?Acc; >> pythan2_B(A, B, N, Acc) -> ?pythan2_B(A,B+1,N,pythan2_C(A, B, 1, N, Acc)). >> >> pythan2_C(A, B, C, N, Acc) when A+B+C> ?N -> ?Acc; >> pythan2_C(A, B, C, N, Acc) -> >> ? ? if A*A+B*B =:= C*C -> >> ? ? ? ? pythan2_C(A, B, C+1, N, [{A,B,C}|Acc]); >> ? ? true -> >> ? ? ? ? pythan2_C(A, B, C+1, N, Acc) >> ? ? end. >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Hynek's implementation. >> %% Simpler and about 5% faster version: >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> pythag3(N) when is_integer(N) -> ?pythag3(N,1). >> >> pythag3(N, A) when A+2> ?N -> ?[]; >> pythag3(N, A) -> ?pythag3(N, A, 1). >> >> pythag3(N, A, B) when A+B+1> ?N -> ?pythag3(N, A+1); >> pythag3(N, A, B) -> ?pythag3(N, A, B, 1). >> >> pythag3(N, A, B, C) when A+B+C> ?N -> ?pythag3(N, A, B+1); >> pythag3(N, A, B, C) when A*A + B*B =:= C*C -> ?[{A, B, C}|pythag3(N, A, >> B, C+1)]; >> pythag3(N, A, B, C) -> ?pythag3(N, A, B, C+1). >> >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Edmond's implementation using parallelism. >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> >> %%---- START CODE ---- >> >> py2E(Max)-> >> ? ? lists:flatten(lpmap(fun(A) -> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? forbE(A, 1, [], Max) >> ? ? ? ? ? ? ? ? ? ? ? ? end, lists:seq(1, Max), ordered)). >> >> forbE(A, B, Acc, Max) -> >> ? ? Acc1 = forcE(A, B, 1, Acc, Max), >> ? ? case B< ?Max of >> ? ? ? ? true -> ?forbE(A, B+1, Acc1, Max); >> ? ? ? ? false -> ?Acc1 >> ? ? end. >> >> forcE(A, B, C, Acc, Max) -> >> ? ? Acc1 = case (A*A + B*B =:= C*C) andalso (A+B+C =< ?Max) of >> ? ? ? ? ? ? ? ? true -> ?[{A,B,C}|Acc]; >> ? ? ? ? ? ? ? ? _ -> ?Acc >> ? ? ? ? ? ?end, >> ? ? case C< ?Max of >> ? ? ? ? true-> ?forcE(A, B, C+1, Acc1, Max); >> ? ? ? ? false-> ?Acc1 >> ? ? end. >> >> >> pythag2E(N)-> >> ? ? lists:flatten(lpmap(fun(A) -> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? pythan2_BE(A, 1, N, []) >> ? ? ? ? ? ? ? ? ? ? ? ? end, lists:seq(1, N), ordered)). >> >> pythan2_AE(A, N, Acc) when A> ?N -> ?Acc; >> pythan2_AE(A, N, Acc) -> ?pythan2_AE(A+1,N,pythan2_BE(A, 1, N, Acc)). >> >> pythan2_BE(A, B, N, Acc) when A+B> ?N -> ?Acc; >> pythan2_BE(A, B, N, Acc) -> ?pythan2_BE(A,B+1,N,pythan2_CE(A, B, 1, N, >> Acc)). >> >> pythan2_CE(A, B, C, N, Acc) when A+B+C> ?N -> ?Acc; >> pythan2_CE(A, B, C, N, Acc) -> >> ? ? if A*A+B*B =:= C*C -> >> ? ? ? ? pythan2_CE(A, B, C+1, N, [{A,B,C}|Acc]); >> ? ? ? ?true -> >> ? ? ? ? pythan2_CE(A, B, C+1, N, Acc) >> ? ? end. >> >> %% @spec ? ?lpmap(fun(), list(), (atom() = ordered|unordered)) -> ?list() >> %% @doc ? ? Spawns a process for each element in list L, performs >> specified >> %% ? ? ? ? ?function F against each in parallel and then returns results >> either >> %% ? ? ? ? ?same order as L (ordered) or in any order (unordered). >> %% ? ? ? ? ?NB: See also lpmap/4. >> >> lpmap(F, L, ordered) -> >> ? ? Ref = erlang:make_ref(), >> ? ? Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- L], >> ? ? lpmap_gather_ordered(Pids, Ref, [], 0, void); >> lpmap(F, L, unordered) -> >> ? ? Ref = erlang:make_ref(), >> ? ? lists:foreach(fun(I) -> >> ? ? ? ? ? ? ? ? ? ? lpmap_spawn_link(self(), Ref, F, I) >> ? ? ? ? ? ? ? ? ? end, L), >> ? ? lpmap_gather_unordered(length(L), Ref, [], 0, void). >> >> %% @spec ? ?lpmap(fun(), integer(), list(), (atom() = >> ordered|unordered)) -> >> ?list() >> %% @doc ? ? Same as lpmap/3 except ensures only a maximum of MaxPs >> parallel >> %% ? ? ? ? ?processes execute function F at any one time (i.e. first takes >> MaxPs >> %% ? ? ? ? ?items from list, executes F in parallel against each, then as >> each >> %% ? ? ? ? ?process returns, spawns another process on next item in L as >> long as >> %% ? ? ? ? ?active processes are less than MaxPs). >> %% ? ? ? ? ?NB: See also lpmap/4. >> >> lpmap(F, L, MaxPs, ordered) when MaxPs>0 -> >> ? ? Ref = erlang:make_ref(), >> ? ? {HPids, TPids} = if >> ? ? ? ? ? ? ? ? ? ? ? ? length(L)> ?MaxPs -> ?lists:split(MaxPs, L); >> ? ? ? ? ? ? ? ? ? ? ? ? true -> ?{L, []} >> ? ? ? ? ? ? ? ? ? ? ?end, >> ? ? Pids = [lpmap_spawn_link(self(), Ref, F, I) || I<- HPids], >> ? ? lpmap_gather_ordered(Pids, Ref, TPids, MaxPs, F); >> lpmap(F, L, MaxPs, unordered) when MaxPs>0 -> >> ? ? Ref = erlang:make_ref(), >> ? ? {HPids, TPids} = if >> ? ? ? ? ? ? ? ? ? ? ? ? length(L)> ?MaxPs -> ?lists:split(MaxPs, L); >> ? ? ? ? ? ? ? ? ? ? ? ? true -> ?{L, []} >> ? ? ? ? ? ? ? ? ? ? ?end, >> ? ? lists:foreach(fun(I) -> >> ? ? ? ? ? ? ? ? ? ? lpmap_spawn_link(self(), Ref, F, I) >> ? ? ? ? ? ? ? ? ? end, HPids), >> ? ? lpmap_gather_unordered(length(HPids), Ref, TPids, MaxPs, F). >> >> %% lpmap internal functions >> >> lpmap_spawn_link(Parent, Ref, F, I) -> >> ? ? spawn_link(fun() -> >> ? ? ? ? ? ? ? ? ? ? Parent ! {self(), Ref, F(I)} >> ? ? ? ? ? ? ? ?end). >> >> lpmap_gather_ordered([], _Ref, [], _MaxPs, _F) -> >> ? ? []; >> lpmap_gather_ordered([HPid|TPids], Ref, L, MaxPs, F) -> >> ? ? receive >> ? ? ? ? {HPid, Ref, Ret} when length(TPids) >> ? ? ? ? ? ? [H | T] = L, >> ? ? ? ? ? ? [Ret | lpmap_gather_ordered( >> ? ? ? ? ? ? ? ? lists:append(TPids, [lpmap_spawn_link(self(), Ref, F, >> H)]), >> ? ? ? ? ? ? ? ? Ref, T, MaxPs, F)]; >> ? ? ? ? {HPid, Ref, Ret} -> >> ? ? ? ? ? ? [Ret | lpmap_gather_ordered(TPids, Ref, L, MaxPs, F)] >> ? ? end. >> >> lpmap_gather_unordered(0, _Ref, [], _MaxPs, _F) -> >> ? ? []; >> lpmap_gather_unordered(NPs, Ref, L, MaxPs, F) -> >> ? ? receive >> ? ? ? ? {_Pid, Ref, Ret} when NPs-1 >> ? ? ? ? ? ? [H | T] = L, >> ? ? ? ? ? ? lpmap_spawn_link(self(), Ref, F, H), >> ? ? ? ? ? ? [Ret | lpmap_gather_unordered(NPs, Ref, T, MaxPs, F)]; >> ? ? ? ? {_Pid, Ref, Ret} -> >> ? ? ? ? ? ? [Ret | lpmap_gather_unordered(NPs-1, Ref, L, MaxPs, F)] >> ? ? end. >> >> >> %%---- END CODE ----- >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Willem's implementation. >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> >> wpythag2(N) -> >> ? ?L = [{A, A*A} || A<- lists:seq(1,N)], >> ? ?lists:flatten([forAllBs(A, A2, L, N) || {A, A2}<- L]). >> >> forAllBs(A, A2, L, N) -> >> ? [forAllCs(A, B, A + B, A2 + B2, L, N) || {B, B2}<- L, A + B< ?N]. >> >> forAllCs(A, B, AB, A2B2, L, N) -> >> ? [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< ?N]. >> >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Hynek's new version >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> >> pythag4(N) when is_integer(N) -> ?pythag4(N,1). >> >> pythag4(N, A) when A+2> ?N -> ?[]; >> pythag4(N, A) -> ?pythag4(N, A, A*A, 1). >> >> pythag4(N, A, _A2, B) when A+B+1> ?N -> ?pythag4(N, A+1); >> pythag4(N, A, A2, B) -> ?pythag4(N, A, A2, B, B*B, 1). >> >> pythag4(N, A, A2, B, _B2, C) when A+B+C> ?N -> ?pythag4(N, A, A2, B+1); >> pythag4(N, A, A2, B, B2, C) when A2 + B2 =:= C*C -> >> ? [{A, B, C}|pythag4(N, A, A2, B, B2, C+1)]; >> pythag4(N, A, A2, B, B2, C) -> ?pythag4(N, A, A2, B, B2, C+1). >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Willem's implementation in parallel by Hynek >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> >> wpythag2P(N) -> >> ? ? L = [{A, A*A} || A<- lists:seq(1,N)], % For all A's >> ? ? lists:flatten(lpmap(fun({A, A2}) -> ? ? % For all B's in parallel >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? [forAllCsWH(A, B, A + B, A2 + B2, L, N) >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? || {B, B2}<- L, A + B< ?N] >> ? ? ? ? ? ? ? ? ? ? ? ? end, L, 2000, ordered)). >> >> forAllCsWH(A, B, AB, A2B2, L, N) -> >> ? ? [{A, B, C} || {C, C2}<- L, A2B2 =:= C2, AB + C =< ?N]. >> >> >> >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> %% Morten's implementation. >> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> >> pythag5(N) when is_integer(N) -> >> ? Primes = sieve(N div 2), >> ? M1M2s = incorporate_primes([{1,1}], N, Primes), >> ? lists:usort(lists:flatten([ [{A,B,C}, {B,A,C}] || {M1, M2}<- M1M2s, >> M1> ?M2, A<- [M1-M2], B<- [2*round(math:sqrt(M1*M2))], C<- [M1+M2], >> A+B+C =< ?N])). >> >> sieve(N) when is_integer(N) -> >> ? erase(), >> ? sieve(N,2). >> >> sieve(N, K) when K>= N -> >> ?[X || X<- lists:seq(2, N), erase(X) == undefined]; >> sieve(N, K) -> >> ? cross_off(K, K, N div K - 1), >> ? sieve(N, find_next_in_sieve(K + 1)). >> >> cross_off(_K, _Current, 0) -> >> ? ok; >> cross_off(K, Current, Left) -> >> ? Next = Current + K, >> ? put(Next, out), >> ? cross_off(K, Next, Left - 1). >> >> find_next_in_sieve(K) -> >> ? case get(K) of >> ? ? undefined -> >> ? ? ? K; >> ? ? _ -> >> ? ? ? find_next_in_sieve(K+1) >> ? end. >> >> incorporate_prime(M1M2s, N, P) -> >> ? lists:flatten([incorporate_prime_single({M1,M2}, N, P)|| {M1, M2}<- >> M1M2s]). >> >> incorporate_prime_single({M1,M2}, N, P) -> >> ? Evens = [{X, Y} || X<- incorporate_prime_even(M1, N, P), Y<- >> incorporate_prime_even(M2, N, P)], >> ? Odds = [{X, Y} || X<- incorporate_prime_odd(M1, N, P), Y<- >> incorporate_prime_odd(M2, N, P)], >> ? Evens ++ Odds. >> >> incorporate_prime_even(M, N, P) -> >> ? incorporate_prime(M, N, P, []). >> >> incorporate_prime_odd(M, N, P) -> >> ? incorporate_prime(M * P, N, P, []). >> >> incorporate_prime(M, N, _P, Acc) when M> ?N/2 -> >> ? Acc; >> incorporate_prime(M, N, P, Acc) -> >> ? incorporate_prime(M * P * P, N, P, [M|Acc]). >> >> incorporate_primes(M1M2s, _N, []) -> >> ? M1M2s; >> incorporate_primes(M1M2s, N, [P|Rest]) -> >> ? M1M2s_new = incorporate_prime(M1M2s, N, P), >> ? incorporate_primes(M1M2s_new, N, Rest). >> >> >> ?%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> ?%% Richard's improvement. >> ?%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> >> ?py3R(N)-> >> ? ? [{A,B,C} || >> ? ? ? ? A <- lists:seq(1, N div 2), >> ? ? ? ? B <- lists:seq(1, N - A), >> ? ? ? ? C <- [trunc(math:sqrt(A * A + B * B))], >> ? ? ? ? A + B + C =< N, >> ? ? ? ? A*A + B*B =:= C*C]. >> >> ?%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> ?%% Joe's improvement. >> ?%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% >> py3a(Max) -> >> ? ? N = Max div 2, >> ? ? [{A,B,C} || >> ? ?A <- lists:seq(1,N+1), >> ? ?B <- lists:seq(1,Max-A), >> ? ?C <- lists:seq(1,Max-A-B), >> ? ?A*A + B*B =:= C*C]. >> >> %************************* END CODE************************* >> >> Cheers, >> >> Ivan. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico >> que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para >> respaldar el cumplimiento de los objetivos de la organizacion y su >> politica informativa. La persona que envia este correo asume el >> compromiso de ?usar el servicio a tales fines y cumplir con las >> regulaciones establecidas. >> >> >> ======================================================================= >> Este mensaje ha sido enviado mediante el servicio de correo electronico >> que ofrece la Federacion de Radioaficionados de Cuba a sus miembros para >> respaldar el cumplimiento de los objetivos de la organizacion y su >> politica informativa. La persona que envia este correo asume el >> compromiso de ?usar el servicio a tales fines y cumplir con las >> regulaciones establecidas. >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From andy.kriger@REDACTED Sat Nov 20 15:18:03 2010 From: andy.kriger@REDACTED (Andy Kriger) Date: Sat, 20 Nov 2010 09:18:03 -0500 Subject: checking for unbound? Message-ID: Is there a way to test if a variable is unbound? I thought I could do something clever like... [E|EL] = mnesia:dirty_index_read(myTable, value, indexField), if -> do something; % nothing was returned length(EL) > 0 -> do something; % more than 1 item was returned -> do something; % a match was returned true -> do something % whatever is left end But I don't see a way to make the first test. From jesper.louis.andersen@REDACTED Sat Nov 20 16:57:38 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 20 Nov 2010 16:57:38 +0100 Subject: [erlang-questions] checking for unbound? In-Reply-To: References: Message-ID: On Sat, Nov 20, 2010 at 3:18 PM, Andy Kriger wrote: > Is there a way to test if a variable is unbound? I am a bit hesitant to guess at what you mean by "unbound variable". Variables which are free (In the lambda-calculus terminology of "free") lead to compilation errors since terms ultimately have to be closed. Thus referral to a variable not existing in the scope or failing to use a variable in a scope results in an error. > I thought I could do something clever like... > [E|EL] = mnesia:dirty_index_read(myTable, value, indexField), > if -> do something; % nothing was returned > ?length(EL) > 0 -> do something; % more than 1 item was returned > ? -> do something; % a match was returned > ?true -> do something % whatever is left We can look up mnesia:dirty_index_read/3 via 'erl -man mnesia', and see it returns a RecordList. Thus, pattern matching to the rescue: case mnesia:dirty_index_read(T, v, i) of [] -> do something; % Nothing was returned [Obj] -> do something ; % Exactly one object was returned RL when is_list(RL) -> do something; % We got a whole list of things end, notice how the case match is exhaustive. It matches all possible outcomes (and strictly speaking we could have been exhaustive without the [Obj] match). A rule of Erlang is that any expression will *always* return a value. There is no equivalent to C's "void" and no "null" as well. Any identifier is always bound to a value or the program is wrong. This concept is very powerful: you don't have to fear the null pointer exception and you don't have to check for null unless the program returns an atom playing the role of the concept "null". It is quite often the case that functional languages choose their expressions in this way. In fact, it would be hard to do otherwise, since it would remove you from the lambda-calculus basis. > But I don't see a way to make the first test. Pattern matching is the key to the puzzle. You let a specific value play the role of being "unbound". In the above case, the role is played by the empty list, [], in other cases it is played by an atom -- usually 'undefined', 'none', 'aborted' -- and in some cases, it is played by a complete term: '{error, Reason}', '{parse_error, PartialParse}' and so on. An example of an atom, 'undefined', playing the role is when asking the application module for configuration options: case application:get_env(MyModule, profiling) of undefined -> false {ok, Bool} when is_boolean(Bool) -> Bool end, Many imperative languages would have chosen to write a program along the lines (somewhat Go-ish syntax), V := application.GetEnvBool(MYMODULE, "profiling") if (V == nil) { return false } else { return V } because any identifier in these languages referencing an object is essentially a pointer which can be nil/null. Personally, I find this ugly, and cumbersome to write and I much prefer the functional variant, of which Erlang is an example. -- J. From alain.odea@REDACTED Sat Nov 20 17:28:28 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Sat, 20 Nov 2010 12:58:28 -0330 Subject: [erlang-questions] checking for unbound? In-Reply-To: References: Message-ID: On Saturday, November 20, 2010, Jesper Louis Andersen wrote: > On Sat, Nov 20, 2010 at 3:18 PM, Andy Kriger wrote: >> Is there a way to test if a variable is unbound? > > I am a bit hesitant to guess at what you mean by "unbound variable". > Variables which are free (In the lambda-calculus terminology of > "free") lead to compilation errors since terms ultimately have to be > closed. Thus referral to a variable not existing in the scope or > failing to use a variable in a scope results in an error. > >> I thought I could do something clever like... >> [E|EL] = mnesia:dirty_index_read(myTable, value, indexField), >> if -> do something; % nothing was returned >> ?length(EL) > 0 -> do something; % more than 1 item was returned >> ? -> do something; % a match was returned >> ?true -> do something % whatever is left > > We can look up mnesia:dirty_index_read/3 via 'erl -man mnesia', and > see it returns a RecordList. Thus, pattern matching to the rescue: > > case mnesia:dirty_index_read(T, v, i) of > ?[] -> do something; % Nothing was returned > ?[Obj] -> do something ; % Exactly one object was returned > ?RL when is_list(RL) -> do something; % We got a whole list of things > end, > > notice how the case match is exhaustive. It matches all possible > outcomes (and strictly speaking we could have been exhaustive without > the [Obj] match). A rule of Erlang is that any expression will > *always* return a value. There is no equivalent to C's "void" and no > "null" as well. Any identifier is always bound to a value or the > program is wrong. This concept is very powerful: you don't have to > fear the null pointer exception and you don't have to check for null > unless the program returns an atom playing the role of the concept > "null". > > It is quite often the case that functional languages choose their > expressions in this way. In fact, it would be hard to do otherwise, > since it would remove you from the lambda-calculus basis. > >> But I don't see a way to make the first test. > > Pattern matching is the key to the puzzle. You let a specific value > play the role of being "unbound". In the above case, the role is > played by the empty list, [], in other cases it is played by an atom > -- usually 'undefined', 'none', 'aborted' -- and in some cases, it is > played by a complete term: '{error, Reason}', '{parse_error, > PartialParse}' and so on. An example of an atom, 'undefined', playing > the role is when asking the application module for configuration > options: > > case application:get_env(MyModule, profiling) of > ?undefined -> false > ?{ok, Bool} when is_boolean(Bool) -> Bool > end, > > Many imperative languages would have chosen to write a program along > the lines (somewhat Go-ish syntax), > > V := application.GetEnvBool(MYMODULE, "profiling") > if (V == nil) { > ?return false > } else { > ?return V > } > > because any identifier in these languages referencing an object is > essentially a pointer which can be nil/null. Personally, I find this > ugly, and cumbersome to write and I much prefer the functional > variant, of which Erlang is an example. > Short version: [E|EL] = mnesia:dirty_index_read(myTable, value, indexField) is a pattern match expression. All expressions in Erlang have values (dirty_index_read must have some value, unbound is not possible). If you get something other than a list with one or more elements you'll get a crash with {error, bad_match} as the cause. Jesper's case expression is a possible solution. My style preference is to use multiple functions and pattern-matching: %... handle_read(mnesia:dirty_index_read(myTable, value, indexField)), %... handle_read([]) -> % handle no items handle_read([E]) -> % handle one item handle_read([_|_] = EL) -> % handle many items From andy.kriger@REDACTED Sat Nov 20 18:01:24 2010 From: andy.kriger@REDACTED (Andy Kriger) Date: Sat, 20 Nov 2010 12:01:24 -0500 Subject: application startup failure? Message-ID: In my gen_server:init/1 function, I return {stop, Reason} if some initialization step goes wrong (like creating a new mnesia schema or tables)... http://www.erlang.org/doc/man/gen_server.html#Module:init-1 I don't want to just return what mnesia returns because in some cases the error is ok (like if the schema or table already exists) However, when I start the application, I only see... =INFO REPORT==== 20-Nov-2010::11:49:47 === application: yaksha exited: {shutdown,{yaksha_app,start,[normal,[]]}} type: temporary {error,{shutdown,{yaksha_app,start,[normal,[]]}}} Where can I actually see the Reason returned by init which would tell me why application startup is failing I've looked through erlang.log.1 and don't see any details Seems odd that the real reason would get swallowed by the gen_server/application structure Or maybe there's a better way to handle this situation? From ebegumisa@REDACTED Sun Nov 21 11:30:29 2010 From: ebegumisa@REDACTED (Edmond Begumisa) Date: Sun, 21 Nov 2010 21:30:29 +1100 Subject: [erlang-questions] checking for unbound? In-Reply-To: References: Message-ID: From your example, I think what you're looking for is pattern matching... case mnesia:dirty_index_read(myTable, value, indexField) of [] -> do something; % nothing was returned [E] -> do something with E; % 1 item was returned [H|T] -> do something with H and T % more than 1 item was returned end. - Edmond - On Sun, 21 Nov 2010 01:18:03 +1100, Andy Kriger wrote: > Is there a way to test if a variable is unbound? > > I thought I could do something clever like... > [E|EL] = mnesia:dirty_index_read(myTable, value, indexField), > if -> do something; % nothing was returned > length(EL) > 0 -> do something; % more than 1 item was returned > -> do something; % a match was returned > true -> do something % whatever is left > end > > But I don't see a way to make the first test. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From roberto@REDACTED Sun Nov 21 11:52:45 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 21 Nov 2010 11:52:45 +0100 Subject: fast file sending - erlang & nginx Message-ID: dear list, i've just made a basic test to measure the sending of a small file (5080 bytes) via http, using erlang and nginx. code for erlang's file sending is appended here below. the results from ab benchmarks on an ubuntu box show that the throughput of nginx is around 5 times the one i can reach with erlang. i've developed a fastcgi backend in erlang [soon to be released on github], and using it i can reach comparable [if not faster] throughtputs on it than the ones i can reach with nginx. so, the fact that erlang is slower in sending static files cannot be due to the http handling itself. i decided to investigate if this was due to I/O reading speed. i've seen that joel has done some research on this using NIF to replace the file reading module on couchdb with the fd.erl module and it's related NIFs [https://github.com/wagerlabs/couchdb/tree/23527eb8165f81e63d47b230f3297d3072c88d83/src/couchdb], however if i'm not wrong i've seen that he is not using this module anymore, so i decided not test it. instead, i tried using the sendfile system call developed by steve vinosky sendfile_drv, packaged by tuncer [https://github.com/tuncer/sendfile], since in this way the file reading is anyway done by a fast C system call and i would be able to see if there are any differences. the results are that raw throughput remains around the same. therefore, i believe that the difference in throughput between nginx and this erlang code is not due to I/O issues. i'm wandering if nginx is nmapping files into memory, which would allow it to outperform in this way, or if i am missing a point here. i'd be glad to have pointers on how to improve fast file sending over http. cheers, r. ========================================= -module(send_file). -include_lib("kernel/include/file.hrl"). -compile(export_all). % sending of a file file_send(Socket, FilePath, Headers) -> % get file size case file:read_file_info(FilePath) of {ok, FileInfo} -> % get filesize FileSize = FileInfo#file_info.size, % do the gradual sending file_open_and_send(Socket, FilePath); {error, Reason} -> {error, Reason} end. file_open_and_send(Socket, FilePath) -> case file:open(FilePath, [read, binary]) of {error, Reason} -> {error, Reason}; {ok, IoDevice} -> % read portions case file_read_and_send(Socket, IoDevice, 0) of {error, Reason} -> file:close(IoDevice), {error, Reason}; ok -> file:close(IoDevice), ok end end. file_read_and_send(Socket, IoDevice, Position) -> % read buffer case file:pread(IoDevice, Position, ?FILE_READ_BUFFER) of {ok, Data} -> % file read, send gen_tcp:send(Socket, Data), % loop file_read_and_send(Socket, IoDevice, Position + ?FILE_READ_BUFFER); eof -> % finished ok; {error, Reason} -> {error, Reason} end. From mk@REDACTED Sun Nov 21 12:43:04 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 21 Nov 2010 12:43:04 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: Message-ID: <4CE905C8.3030505@amberbio.com> Robert, what throughput do you get by sending a preloaded binary instead? In other words, read the file into a binary once, and then just serve the replies from that binary. Morten. On 11/21/10 11:52 AM, Roberto Ostinelli wrote: > dear list, > > i've just made a basic test to measure the sending of a small file > (5080 bytes) via http, using erlang and nginx. code for erlang's file > sending is appended here below. the results from ab benchmarks on an > ubuntu box show that the throughput of nginx is around 5 times the one > i can reach with erlang. > > i've developed a fastcgi backend in erlang [soon to be released on > github], and using it i can reach comparable [if not faster] > throughtputs on it than the ones i can reach with nginx. so, the fact > that erlang is slower in sending static files cannot be due to the > http handling itself. > > i decided to investigate if this was due to I/O reading speed. > > i've seen that joel has done some research on this using NIF to > replace the file reading module on couchdb with the fd.erl module and > it's related NIFs > [https://github.com/wagerlabs/couchdb/tree/23527eb8165f81e63d47b230f3297d3072c88d83/src/couchdb], > however if i'm not wrong i've seen that he is not using this module > anymore, so i decided not test it. > > instead, i tried using the sendfile system call developed by steve > vinosky sendfile_drv, packaged by tuncer > [https://github.com/tuncer/sendfile], since in this way the file > reading is anyway done by a fast C system call and i would be able to > see if there are any differences. the results are that raw throughput > remains around the same. > > therefore, i believe that the difference in throughput between nginx > and this erlang code is not due to I/O issues. i'm wandering if nginx > is nmapping files into memory, which would allow it to outperform in > this way, or if i am missing a point here. > > i'd be glad to have pointers on how to improve fast file sending over http. > > cheers, > > r. > > ========================================= > -module(send_file). > -include_lib("kernel/include/file.hrl"). > -compile(export_all). > > % sending of a file > file_send(Socket, FilePath, Headers) -> > % get file size > case file:read_file_info(FilePath) of > {ok, FileInfo} -> > % get filesize > FileSize = FileInfo#file_info.size, > % do the gradual sending > file_open_and_send(Socket, FilePath); > {error, Reason} -> > {error, Reason} > end. > file_open_and_send(Socket, FilePath) -> > case file:open(FilePath, [read, binary]) of > {error, Reason} -> > {error, Reason}; > {ok, IoDevice} -> > % read portions > case file_read_and_send(Socket, IoDevice, 0) of > {error, Reason} -> > file:close(IoDevice), > {error, Reason}; > ok -> > file:close(IoDevice), > ok > end > end. > file_read_and_send(Socket, IoDevice, Position) -> > % read buffer > case file:pread(IoDevice, Position, ?FILE_READ_BUFFER) of > {ok, Data} -> > % file read, send > gen_tcp:send(Socket, Data), > % loop > file_read_and_send(Socket, IoDevice, Position + ?FILE_READ_BUFFER); > eof -> > % finished > ok; > {error, Reason} -> > {error, Reason} > end. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From rapsey@REDACTED Sun Nov 21 13:15:17 2010 From: rapsey@REDACTED (Rapsey) Date: Sun, 21 Nov 2010 13:15:17 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: Message-ID: I've done quite a bit of testing on this stuff and the end conclusion is that gen_tcp:send is simply slow. It uses way to much CPU than it should and I don't know why. I tried looking at the inet_drv.c stuff, but there is just to much code in there. Since my work involves writing a streaming server and throughput is king. I gave up on gen_tcp and just implemented my own socket driver and later turned into a NIF (more flexible and general use). It's fast as hell now. Sergej On Sun, Nov 21, 2010 at 11:52 AM, Roberto Ostinelli wrote: > dear list, > > i've just made a basic test to measure the sending of a small file > (5080 bytes) via http, using erlang and nginx. code for erlang's file > sending is appended here below. the results from ab benchmarks on an > ubuntu box show that the throughput of nginx is around 5 times the one > i can reach with erlang. > > i've developed a fastcgi backend in erlang [soon to be released on > github], and using it i can reach comparable [if not faster] > throughtputs on it than the ones i can reach with nginx. so, the fact > that erlang is slower in sending static files cannot be due to the > http handling itself. > > i decided to investigate if this was due to I/O reading speed. > > i've seen that joel has done some research on this using NIF to > replace the file reading module on couchdb with the fd.erl module and > it's related NIFs > [ > https://github.com/wagerlabs/couchdb/tree/23527eb8165f81e63d47b230f3297d3072c88d83/src/couchdb > ], > however if i'm not wrong i've seen that he is not using this module > anymore, so i decided not test it. > > instead, i tried using the sendfile system call developed by steve > vinosky sendfile_drv, packaged by tuncer > [https://github.com/tuncer/sendfile], since in this way the file > reading is anyway done by a fast C system call and i would be able to > see if there are any differences. the results are that raw throughput > remains around the same. > > therefore, i believe that the difference in throughput between nginx > and this erlang code is not due to I/O issues. i'm wandering if nginx > is nmapping files into memory, which would allow it to outperform in > this way, or if i am missing a point here. > > i'd be glad to have pointers on how to improve fast file sending over http. > > cheers, > > r. > > ========================================= > -module(send_file). > -include_lib("kernel/include/file.hrl"). > -compile(export_all). > > % sending of a file > file_send(Socket, FilePath, Headers) -> > % get file size > case file:read_file_info(FilePath) of > {ok, FileInfo} -> > % get filesize > FileSize = FileInfo#file_info.size, > % do the gradual sending > file_open_and_send(Socket, FilePath); > {error, Reason} -> > {error, Reason} > end. > file_open_and_send(Socket, FilePath) -> > case file:open(FilePath, [read, binary]) of > {error, Reason} -> > {error, Reason}; > {ok, IoDevice} -> > % read portions > case file_read_and_send(Socket, IoDevice, 0) of > {error, Reason} -> > file:close(IoDevice), > {error, Reason}; > ok -> > file:close(IoDevice), > ok > end > end. > file_read_and_send(Socket, IoDevice, Position) -> > % read buffer > case file:pread(IoDevice, Position, ?FILE_READ_BUFFER) of > {ok, Data} -> > % file read, send > gen_tcp:send(Socket, Data), > % loop > file_read_and_send(Socket, IoDevice, Position + > ?FILE_READ_BUFFER); > eof -> > % finished > ok; > {error, Reason} -> > {error, Reason} > end. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From kenrobinsonster@REDACTED Sun Nov 21 14:58:44 2010 From: kenrobinsonster@REDACTED (Ken Robinson) Date: Sun, 21 Nov 2010 23:58:44 +1000 Subject: Preferred way for phased startup for app Message-ID: Hi All, At the moment I am running an application where I am using a supervisor with several gen_servers and gen_fsms. Certain servers require other servers to start and initialize successfully. Is the order in which I start them the order in which they fact start? What is the best way to ensure a particular server has initialized correctly? At the moment I am running the following code at the end of this post. However I saw in doco for app the concept of start_phases (http://www.erlang.org/doc/man/app.html). If this is the preferred way is there any example code anyone can point me to? Many thanks. %% I need this things to be up before I can proceed {ok, RequiredProcesses} = jaus.util.config:get(requires, Args), UpdatedRequiredProcesses = lists:append(RequiredProcesses, [jaus_core_componentserver]), ProcessPidList = [{Elem, erlang:whereis(Elem)} || Elem <- UpdatedRequiredProcesses ], ProcessAliveList = [{ProcessName, erlang:is_process_alive(ProcessId)} || {ProcessName, ProcessId} <- ProcessPidList], ProcessNotUpList = [ ProcessName || {ProcessName, ProcessUp} <- ProcessAliveList, ProcessUp == false], case erlang:length(ProcessNotUpList) of 0 -> %% do nothing log4erl:info("[Src ~p Line ~p] msg - ~p~n", [?FILE, ?LINE, ProcessNotUpList]); _Default -> %% terminate by throwing error log4erl:info("[Src ~p Line ~p] msg Process(s) not up - ~p~n", [?FILE, ?LINE, ProcessNotUpList]), erlang:error({stop, {process_not_up, ProcessNotUpList}}) end, -- regards, Ken Robinson Mob +61438681120 Home +61738523767 Work +61733063137 From roberto@REDACTED Sun Nov 21 17:10:43 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 21 Nov 2010 17:10:43 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: <4CE905C8.3030505@amberbio.com> References: <4CE905C8.3030505@amberbio.com> Message-ID: 2010/11/21 Morten Krogh : > Robert, what throughput do you get by sending a preloaded binary instead? > > In other words, read the file into a binary once, and then just serve the > replies from that binary. > > Morten. here are the details, running ab -n 50000 -c 5 http://ubuntu.loc/image.jpeg with Document Length = 4163 bytes NGINX: Time per request: 0.410 [ms] (mean) Time per request: 0.082 [ms] (mean, across all concurrent requests) Transfer rate: 52161.96 [Kbytes/sec] received Percentage of the requests served within a certain time (ms) 50% 0 66% 0 75% 0 80% 0 90% 1 95% 1 98% 1 99% 1 100% 6 (longest request) MISULTIN: Time per request: 1.428 [ms] (mean) Time per request: 0.286 [ms] (mean, across all concurrent requests) Transfer rate: 14525.79 [Kbytes/sec] received Percentage of the requests served within a certain time (ms) 50% 1 66% 2 75% 2 80% 2 90% 2 95% 2 98% 2 99% 2 100% 6 (longest request) MISULTIN WITH FILE READ IN MEMORY: Time per request: 0.795 [ms] (mean) Time per request: 0.159 [ms] (mean, across all concurrent requests) Transfer rate: 26088.81 [Kbytes/sec] received Percentage of the requests served within a certain time (ms) 50% 1 66% 1 75% 1 80% 1 90% 1 95% 1 98% 1 99% 1 100% 2 (longest request) ratios: nginx : misultin = 1 : 3.6 nginx : misultin with cache = 1 : 2 code to perform these tests is hereby provided. r. ============== normal file sending ============================== -module(misultin_file). -export([start/1, stop/0]). % start misultin http server start(Port) -> misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]). % stop misultin stop() -> misultin:stop(). % callback on request received handle_http(Req) -> Req:file("image.jpeg"). =========================================================== ============== cached file sending ============================= -module(misultin_file2). -export([start/1, stop/0]). -include_lib("kernel/include/file.hrl"). % start misultin http server start(Port) -> % load file FilePath = "roberto2.jpeg", {ok, Binary} = file:read_file(FilePath), {ok, FileInfo} = file:read_file_info(FilePath), FileSize = FileInfo#file_info.size, HeadersFull = [{'Content-Type', misultin_utility:get_content_type(FilePath)}, {'Content-Length', FileSize}], misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req, Binary, HeadersFull) end}]). % stop misultin stop() -> misultin:stop(). % callback on request received handle_http(Req, Binary, HeadersFull) -> Req:stream(head, HeadersFull), Req:stream(Binary). =========================================================== From roberto@REDACTED Sun Nov 21 17:11:54 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 21 Nov 2010 17:11:54 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: Message-ID: 2010/11/21 Rapsey : > I've done quite a bit of testing on this stuff and the end conclusion is > that gen_tcp:send is simply slow.?It uses way to much CPU than it should and > I don't know why. I tried looking at the inet_drv.c stuff, but there is just > to much code in there. > Since my work involves writing a streaming server and throughput is king. I > gave up on gen_tcp and just implemented my own socket driver and later > turned into a NIF (more flexible and general use). It's fast as hell now. > > > Sergej hi sergej, is this closed-source? if not, where can it be found? thank you, r. From rapsey@REDACTED Sun Nov 21 17:32:57 2010 From: rapsey@REDACTED (Rapsey) Date: Sun, 21 Nov 2010 17:32:57 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: Message-ID: On Sun, Nov 21, 2010 at 5:11 PM, Roberto Ostinelli wrote: > 2010/11/21 Rapsey : > > I've done quite a bit of testing on this stuff and the end conclusion is > > that gen_tcp:send is simply slow. It uses way to much CPU than it should > and > > I don't know why. I tried looking at the inet_drv.c stuff, but there is > just > > to much code in there. > > Since my work involves writing a streaming server and throughput is king. > I > > gave up on gen_tcp and just implemented my own socket driver and later > > turned into a NIF (more flexible and general use). It's fast as hell now. > > > > > > Sergej > > hi sergej, > > is this closed-source? if not, where can it be found? > > thank you, > > r. > Closed. But it really does not take that much effort to implement it. The basic building blocks are an epoll/kqueue thread and a pipe to communicate with it. Once it's all created, send an erlang PID to the thread over the pipe so that the thread can communicate with erlang. Sergej From mk@REDACTED Sun Nov 21 18:11:45 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 21 Nov 2010 18:11:45 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: <4CE905C8.3030505@amberbio.com> Message-ID: <4CE952D1.503@amberbio.com> Hi Robert That was a big speedup. So what you can do, is to read all static files into memory at server startup, and have some notification whenever the directory changes. The you can ignore the performance of erlang file reading, and sendfile as well. Actually, your server should perform better than any sendfile based server, I would guess. It is not completely clear, but I don't think sendfile, even from disk cache, can outperform a memory buffer. I don't know if erlang has a way to send you a notification whenever a directory changes. But you could do it with epoll or kqueue. Or just implement some simple timed poller from erlang. To get further improvements, you could look at header parsing. What happens to your numbers if you drop header parsing, and just look for the url and \r\n\r\n, or maybe just \r\n\r\n for this simple test. About gen_tcp, that is a serious problem and something the otp team should rectify. If you have to implement a nif for kqueue or epoll, why is the server written in erlang to start with? Just make a full server, written in c, with a callback to erlang handlers. Morten. On 11/21/10 5:10 PM, Roberto Ostinelli wrote: > 2010/11/21 Morten Krogh: >> Robert, what throughput do you get by sending a preloaded binary instead? >> >> In other words, read the file into a binary once, and then just serve the >> replies from that binary. >> >> Morten. > here are the details, running ab -n 50000 -c 5 > http://ubuntu.loc/image.jpeg with Document Length = 4163 bytes > > NGINX: > Time per request: 0.410 [ms] (mean) > Time per request: 0.082 [ms] (mean, across all concurrent requests) > Transfer rate: 52161.96 [Kbytes/sec] received > Percentage of the requests served within a certain time (ms) > 50% 0 > 66% 0 > 75% 0 > 80% 0 > 90% 1 > 95% 1 > 98% 1 > 99% 1 > 100% 6 (longest request) > > MISULTIN: > Time per request: 1.428 [ms] (mean) > Time per request: 0.286 [ms] (mean, across all concurrent requests) > Transfer rate: 14525.79 [Kbytes/sec] received > Percentage of the requests served within a certain time (ms) > 50% 1 > 66% 2 > 75% 2 > 80% 2 > 90% 2 > 95% 2 > 98% 2 > 99% 2 > 100% 6 (longest request) > > MISULTIN WITH FILE READ IN MEMORY: > Time per request: 0.795 [ms] (mean) > Time per request: 0.159 [ms] (mean, across all concurrent requests) > Transfer rate: 26088.81 [Kbytes/sec] received > Percentage of the requests served within a certain time (ms) > 50% 1 > 66% 1 > 75% 1 > 80% 1 > 90% 1 > 95% 1 > 98% 1 > 99% 1 > 100% 2 (longest request) > > ratios: > > nginx : misultin = 1 : 3.6 > nginx : misultin with cache = 1 : 2 > > code to perform these tests is hereby provided. > > r. > > ============== normal file sending ============================== > > -module(misultin_file). > -export([start/1, stop/0]). > > % start misultin http server > start(Port) -> > misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]). > > % stop misultin > stop() -> > misultin:stop(). > > % callback on request received > handle_http(Req) -> > Req:file("image.jpeg"). > > =========================================================== > > ============== cached file sending ============================= > > -module(misultin_file2). > -export([start/1, stop/0]). > -include_lib("kernel/include/file.hrl"). > > % start misultin http server > start(Port) -> > % load file > FilePath = "roberto2.jpeg", > {ok, Binary} = file:read_file(FilePath), > {ok, FileInfo} = file:read_file_info(FilePath), > FileSize = FileInfo#file_info.size, > HeadersFull = [{'Content-Type', > misultin_utility:get_content_type(FilePath)}, {'Content-Length', > FileSize}], > misultin:start_link([{port, Port}, {loop, fun(Req) -> > handle_http(Req, Binary, HeadersFull) end}]). > > % stop misultin > stop() -> > misultin:stop(). > > % callback on request received > handle_http(Req, Binary, HeadersFull) -> > Req:stream(head, HeadersFull), > Req:stream(Binary). > > =========================================================== > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From roberto@REDACTED Sun Nov 21 21:00:04 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Sun, 21 Nov 2010 21:00:04 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: <4CE952D1.503@amberbio.com> References: <4CE905C8.3030505@amberbio.com> <4CE952D1.503@amberbio.com> Message-ID: 2010/11/21 Morten Krogh : > Hi Robert > > That was a big speedup. So what you can do, is to read all static files into > memory at server startup, and have some notification whenever the > directory changes. morten, thank you for your suggestion, but this is basically setting up a cache server, which is not what i want to do here. thus, i was trying to understand if the difference with nginx is because it uses nmap, or some other reason. cheers, r. From mk@REDACTED Sun Nov 21 21:25:49 2010 From: mk@REDACTED (Morten Krogh) Date: Sun, 21 Nov 2010 21:25:49 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: <4CE905C8.3030505@amberbio.com> <4CE952D1.503@amberbio.com> Message-ID: <4CE9804D.1050807@amberbio.com> Hi Robert You mean mmap. I don't see how that would help. Either you have the file in a memory buffer when the request arrives, like you did in the example or you don't. If you don't, sendfile should be the best call. If mmaping and sending on the socket, after getting the request, outperforms sendfile, then sendfile doesn't have a reason to exist. Of course, you could also mmap in advance which is similar to reading the file in advance. You still had a factor of 2 up to nginx, and that then cannot have anything to do with how the files are read, but must have to do with the epoll/kqueue of the sockets, or with the speed of the erlang code, including the parsing of the request header. Do you have an idea of the speed of your header parsing? If you spend 100 microseconds on that, it will explain part of the remaining difference. If you spend 1 microsecond, it is irrelevant. Morten. On 11/21/10 9:00 PM, Roberto Ostinelli wrote: > 2010/11/21 Morten Krogh: >> Hi Robert >> >> That was a big speedup. So what you can do, is to read all static files into >> memory at server startup, and have some notification whenever the >> directory changes. > morten, > > thank you for your suggestion, but this is basically setting up a > cache server, which is not what i want to do here. > > thus, i was trying to understand if the difference with nginx is > because it uses nmap, or some other reason. > > cheers, > > r. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From vinoski@REDACTED Sun Nov 21 22:35:43 2010 From: vinoski@REDACTED (Steve Vinoski) Date: Sun, 21 Nov 2010 16:35:43 -0500 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: Message-ID: On Sun, Nov 21, 2010 at 7:15 AM, Rapsey wrote: > I've done quite a bit of testing on this stuff and the end conclusion is > that gen_tcp:send is simply slow. It uses way to much CPU than it should and > I don't know why. I tried looking at the inet_drv.c stuff, but there is just > to much code in there. My experiences and measurements show pretty much the same thing. I don't think gen_tcp necessarily targets top performance, but rather is designed to be portable and tie cleanly into the whole inet framework, but by doing so it suffers from unnecessary overhead in a variety of places. Some of the problem areas we've observed are too many {set,get}sockopt calls made that don't seem to add value, inet_db ets lookups on each gen_tcp operation to find the socket driver, and some handling of socket options/parameters in prim_inet that seems overly expensive. gen_tcp2, anyone? Regarding the sendfile driver: > On Sun, Nov 21, 2010 at 11:52 AM, Roberto Ostinelli wrote: >> instead, i tried using the sendfile system call developed by steve >> vinosky sendfile_drv, packaged by tuncer Vinoski, not Vinosky. :-) >> [https://github.com/tuncer/sendfile], since in this way the file >> reading is anyway done by a fast C system call and i would be able to >> see if there are any differences. the results are that raw throughput >> remains around the same. That surprises me, but at the very least you should see much lower CPU with the sendfile driver. --steve From josh@REDACTED Mon Nov 22 00:34:04 2010 From: josh@REDACTED (Josh Johnston) Date: Mon, 22 Nov 2010 10:34:04 +1100 Subject: [erlang-questions] Erlang and OTP tutorials at YOW!, Melbourne & Brisbane, Dec 2010 In-Reply-To: References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> Message-ID: Hi - I'm in Melbourne and would definitely be interested. Not sure about my availability on these dates though... but we'll see! :) On 16/11/2010, at 7:03 AM, Edmond Begumisa wrote: > Any one else in Melbourne up for an Erlounge Dec 2nd or 3rd? > > - Edmond - > > On Tue, 16 Nov 2010 03:58:44 +1100, Ulf Wiger wrote: > >> >> Sure. My preference for Erlounges would be >> >> Melbourne: Dec 2 or 3 >> Brisbane: Dec 9 >> >> BR, >> Ulf >> >> On 15 Nov 2010, at 17:35, Edmond Begumisa wrote: >> >>> Hello Ulf, >>> >>> I'm in Melbourne! >>> >>> For those of us unfortunately unable to get time off to attend, is there a possibility for an ErlLounge? I personally would love to hear some war stories from your AXD days! >>> >>> - Edmond - >>> >>> >>> On Mon, 15 Nov 2010 19:23:42 +1100, Ulf Wiger wrote: >>> >>>> >>>> Just a short note for those of you Down Under. >>>> >>>> I will be giving 1-day tutorials on Erlang and OTP during the YOW! 2010 >>>> Developer Conference in Melbourne (Nov 30-Dec 1) and Brisbane (Dec 8-9). >>>> >>>> http://www.yowconference.com.au/index.html >>>> >>>> This should be a good opportunity for those interested in Erlang to get >>>> together, ask questions and perhaps talk over beer afterwards. I'd love to hear >>>> from some locals about how we can make the most of the opportunity. >>>> >>>> The conference itself will feature talks by me, Justin Sheehy and >>>> Kresten Krab Thorup - as well as a load of distinguished speakers from >>>> different camps. >>>> >>>> http://www.yowconference.com.au/melbourne/speakers/index.html >>>> >>>> It should be a hoot. Please sign up and recruit some friends too. >>>> >>>> I hope to see you there. >>>> >>>> BR, >>>> Ulf W >>>> >>>> Ulf Wiger, CTO, Erlang Solutions, Ltd. >>>> http://erlang-solutions.com >>>> >>>> >>>> >>>> >>>> ________________________________________________________________ >>>> erlang-questions (at) erlang.org mailing list. >>>> See http://www.erlang.org/faq.html >>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >>>> >>> >>> >>> -- >>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ >> >> Ulf Wiger, CTO, Erlang Solutions, Ltd. >> http://erlang-solutions.com >> >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > -- > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > From ulf.wiger@REDACTED Mon Nov 22 07:42:46 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Mon, 22 Nov 2010 07:42:46 +0100 Subject: [erlang-questions] pagination (skip, limit) ets items (gproc pids) In-Reply-To: <804797.46106.qm@web112604.mail.gq1.yahoo.com> References: <804797.46106.qm@web112604.mail.gq1.yahoo.com> Message-ID: <5A75D035-5104-4466-97D4-5B4476336E4F@erlang-solutions.com> Hi Pablo, Paging can be done using QLC cursors, and if you want, you can create a special pager process that repeatedly calls qlc:next_answers/2. Using gproc, you could register the pager process using a request cookie or other identifier that makes it convenient for you to find it. The following shell dialogue might illustrate: 2> [gproc:reg({n,l,K}) || K <- lists:seq(1,30)]. [true,true,true,true,true,true,true,true,true,true,true, true,true,true,true,true,true,true,true,true,true,true,true, true,true,true,true,true,true|...] 3> QH = qlc:q([X || X <- gproc:table(all)]). {qlc_handle,{qlc_lc,#Fun, {qlc_opt,false,false,-1,any,[],any,524288,allowed}}} 4> Cu = qlc:cursor(QH). {qlc_cursor,{<0.74.0>,<0.67.0>}} 5> qlc:next_answers(Cu, 3). [{{n,l,1},<0.67.0>,undefined}, {{n,l,2},<0.67.0>,undefined}, {{n,l,3},<0.67.0>,undefined}] 6> qlc:next_answers(Cu, 3). [{{n,l,4},<0.67.0>,undefined}, {{n,l,5},<0.67.0>,undefined}, {{n,l,6},<0.67.0>,undefined}] 7> qlc:next_answers(Cu, 3). [{{n,l,7},<0.67.0>,undefined}, {{n,l,8},<0.67.0>,undefined}, {{n,l,9},<0.67.0>,undefined}] BR, Ulf W On 20 Nov 2010, at 01:18, Pablo Platt wrote: > Hi > > I'm using gproc to save user sessions. > I want to present a list of online users in an admin html page. > I have a large number of sessions. > Is it possible to paginate the list similar to how SQL use skip and limit? > The admin will see a list of 1-10 online users and have links previous and next > to see previous and next group of users. > I can't use ets continuation because I'll have to save it and reuse it which is > hard in a website. > > The only way I see to solve it is to use ets:match(Tab, Pattern, Limit) several > times. > For example, if the admin requests for users 50-60 I'll do ets:match 5 times > when using the returned continuation. > > Is there a better way? > Is there a reason why Skip isn't implemented even if it'll be implemented like a > suggested above? > > Thanks > > Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From alexey.v.romanov@REDACTED Mon Nov 22 14:14:51 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Mon, 22 Nov 2010 16:14:51 +0300 Subject: More natural interface for SQLite wrapper Message-ID: I am working on my fork of the Erlang wrapper for SQLite 3 (https://github.com/alexeyr/erlang-sqlite3) and one this question arose: which Erlang values should correspond to SQLite types (http://www.sqlite.org/datatype3.html) TEXT and BLOB? Currently TEXT is any iodata (i.e. binary or iolist), and BLOB is represented by tuple {blob, Binary}. The alternative would be to allow only iolists as TEXT, and represent blobs as binaries. Which would be easier to use, in your opinion? Yours, Alexey Romanov From rj@REDACTED Mon Nov 22 14:45:08 2010 From: rj@REDACTED (Richard Jones) Date: Mon, 22 Nov 2010 13:45:08 +0000 Subject: Websockets Message-ID: Is anyone using websockets off an erlang backend in production? I couldn't find any complete implementations, so here's my patch to mochiweb to support websockets: http://groups.google.com/group/mochiweb/browse_thread/thread/3ae55bf6792eb851 I'm planning to start using it on irccloud.com in the next week or two, as an alternative to the awkward comet hack in use currently. I'll report back once it's had some production use. In the meantime, I'm curious if anyone else has a working implementation or is using websockets in anger. RJ From bob@REDACTED Mon Nov 22 15:00:32 2010 From: bob@REDACTED (Bob Ippolito) Date: Mon, 22 Nov 2010 22:00:32 +0800 Subject: [erlang-questions] More natural interface for SQLite wrapper In-Reply-To: References: Message-ID: Does there need to be a difference? Can you use binary for either without a tuple tag? Maybe blob if it's not strictly valid UTF8? On Monday, November 22, 2010, Alexey Romanov wrote: > I am working on my fork of the Erlang wrapper for SQLite 3 > (https://github.com/alexeyr/erlang-sqlite3) and one this question > arose: which Erlang values should correspond to SQLite types > (http://www.sqlite.org/datatype3.html) TEXT and BLOB? Currently TEXT > is any iodata (i.e. binary or iolist), and BLOB is represented by > tuple {blob, Binary}. The alternative would be to allow only iolists > as TEXT, and represent blobs as binaries. Which would be easier to > use, in your opinion? > > Yours, Alexey Romanov > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From alexey.v.romanov@REDACTED Mon Nov 22 15:19:56 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Mon, 22 Nov 2010 17:19:56 +0300 Subject: [erlang-questions] More natural interface for SQLite wrapper In-Reply-To: References: Message-ID: On Mon, Nov 22, 2010 at 5:00 PM, Bob Ippolito wrote: > Does there need to be a difference? Can you use binary for either > without a tuple tag? Probably not. I have to call different functions on C side for blobs and text. > Maybe blob if it's not strictly valid UTF8? That's possible, but there still needs to be a way to pass a blob which would be valid UTF8. Plus having to check validity for every binary would hurt performance... Yours, Alexey Romanov > On Monday, November 22, 2010, Alexey Romanov wrote: >> I am working on my fork of the Erlang wrapper for SQLite 3 >> (https://github.com/alexeyr/erlang-sqlite3) and one this question >> arose: which Erlang values should correspond to SQLite types >> (http://www.sqlite.org/datatype3.html) TEXT and BLOB? Currently TEXT >> is any iodata (i.e. binary or iolist), and BLOB is represented by >> tuple {blob, Binary}. The alternative would be to allow only iolists >> as TEXT, and represent blobs as binaries. Which would be easier to >> use, in your opinion? >> >> Yours, Alexey Romanov >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > From pablo.platt@REDACTED Mon Nov 22 15:47:54 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Mon, 22 Nov 2010 06:47:54 -0800 (PST) Subject: [erlang-questions] Websockets In-Reply-To: References: Message-ID: <923932.43728.qm@web112620.mail.gq1.yahoo.com> Are you using a proxy in front of mochiweb for your websockets implementation or just port forwarding with iptables? nginx doesn't support http 1.1 so I think it's not suitable for websockets. ________________________________ From: Richard Jones To: erlang-questions@REDACTED Sent: Mon, November 22, 2010 3:45:08 PM Subject: [erlang-questions] Websockets Is anyone using websockets off an erlang backend in production? I couldn't find any complete implementations, so here's my patch to mochiweb to support websockets: http://groups.google.com/group/mochiweb/browse_thread/thread/3ae55bf6792eb851 I'm planning to start using it on irccloud.com in the next week or two, as an alternative to the awkward comet hack in use currently. I'll report back once it's had some production use. In the meantime, I'm curious if anyone else has a working implementation or is using websockets in anger. RJ ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From alexey.v.romanov@REDACTED Mon Nov 22 16:19:42 2010 From: alexey.v.romanov@REDACTED (Alexey Romanov) Date: Mon, 22 Nov 2010 18:19:42 +0300 Subject: [erlang-questions] Separate messages from stdout and stderr of a port In-Reply-To: References: Message-ID: On Mon, Nov 22, 2010 at 5:55 PM, Samuel Rivas wrote: >> I can get the output of a spawned port on its stderr stream by using >> stderr_to_stdout. However, >> is there a way to get messages from the port's stdout and stderr >> separately? I don't see one in >> http://www.erlang.org/doc/man/erlang.html >> In fact, it would be enough for my purposes to only get stderr and >> ignore stdout. > > If you just want to do that you can use standard shell redirection, > since port commands are evaluated in a shell process. > > the-port-command 2>&1 > /dev/null > > Should be enough to trash all stdout and redirect stderr to stdout Right, simple and nice. Yours, Alexey Romanov > Cheers > -- > Samuel > From rj@REDACTED Mon Nov 22 16:23:32 2010 From: rj@REDACTED (Richard Jones) Date: Mon, 22 Nov 2010 15:23:32 +0000 Subject: [erlang-questions] Websockets In-Reply-To: <923932.43728.qm@web112620.mail.gq1.yahoo.com> References: <923932.43728.qm@web112620.mail.gq1.yahoo.com> Message-ID: Thanks for the zotonic tip, I'll have a look at that. I don't need a full CMS, but it'll be useful to compare websockets implementations. I'm currently using nginx in front of mochiweb for comet and static serving etc. nginx is not suitable for websockets as mentioned.. HAProxy supposedly is, but I'm probably just going to listen directly with mochiweb for websockets on another IP using a subdomain. This should work fine, since I can validate the origin and permit traffic from the main domain to the subdomain. I'll be listening on the https port for websockets using either an iptables forwarding rule, or setcap on the vm as detailed here: http://erlanganswers.com/web/mcedemo/PrivilegedPort.html RJ On 22 November 2010 14:47, Pablo Platt wrote: > Are you using a proxy in front of mochiweb for your websockets > implementation > or just port forwarding with iptables? > nginx doesn't support http 1.1 so I think it's not suitable for websockets. > > > > ________________________________ > From: Richard Jones > To: erlang-questions@REDACTED > Sent: Mon, November 22, 2010 3:45:08 PM > Subject: [erlang-questions] Websockets > > Is anyone using websockets off an erlang backend in production? > I couldn't find any complete implementations, so here's my patch to > mochiweb to support websockets: > > http://groups.google.com/group/mochiweb/browse_thread/thread/3ae55bf6792eb851 > > I'm planning to start using it on irccloud.com in the next week or > two, as an alternative to the awkward comet hack in use currently. > I'll report back once it's had some production use. In the meantime, > I'm curious if anyone else has a working implementation or is using > websockets in anger. > > RJ > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > From roberto@REDACTED Mon Nov 22 16:30:19 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 22 Nov 2010 16:30:19 +0100 Subject: [erlang-questions] Websockets In-Reply-To: References: Message-ID: 2010/11/22 Richard Jones : > Is anyone using websockets off an erlang backend in production? > I couldn't find any complete implementations not sure if this is what you are looking for, but misultin does support websockets: http://code.google.com/p/misultin/source/browse/trunk/src/misultin_websocket.erl we use it on private client's backends. cheers, r. From mevans@REDACTED Mon Nov 22 16:47:03 2010 From: mevans@REDACTED (Evans, Matthew) Date: Mon, 22 Nov 2010 10:47:03 -0500 Subject: Mnesia Transactions Message-ID: Hi Team, I have a quick question WRT mnesia and transactions, and can't seem to find the answer anywhere. Let us say I have a distributed table that is active on several nodes, and wish to apply a transaction to insert multiple records in that table. Is each operation inside that transaction applied individually on each node, or is the whole transaction applied on each node in turn? For example ListOfOneThousandRecords = create_records(), mnesia:transaction(fun() -> [mnesia:write(Rec)||Rec<-ListOfOneThousandRecords] end). So this transaction will insert 1000 records into the table. Will mnesia: 1) Take the record in turn and insert it into node1, node2...node X , take the second record and insert into node1, node2...node X until complete. Or 2) Apply the whole transaction to node1 (insert all 1000 records locally) and then apply the fun (again locally) to node 2 and so on? Thanks Matt From hm@REDACTED Mon Nov 22 17:28:37 2010 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Mon, 22 Nov 2010 17:28:37 +0100 Subject: [erlang-questions] Mnesia Transactions In-Reply-To: References: Message-ID: On Mon, Nov 22, 2010 at 4:47 PM, Evans, Matthew wrote: > Hi Team, > > I have a quick question WRT mnesia and transactions, and can't seem to find the answer anywhere. > > Let us say I have a distributed table that is active on several nodes, and wish to apply a transaction to insert multiple records in that table. Is each operation inside that transaction applied individually on each node, or is the whole transaction applied on each node in turn? > > For example > > ListOfOneThousandRecords = create_records(), > mnesia:transaction(fun() -> > ? ? ? ? ? ?[mnesia:write(Rec)||Rec<-ListOfOneThousandRecords] > end). > > So this transaction will insert 1000 records into the table. > > Will mnesia: > > 1) Take the record in turn and insert it into node1, node2...node X , take > the second record and insert into node1, node2...node X until complete. > > Or > > 2) Apply the whole transaction to node1 (insert all 1000 records locally) > and then apply the fun (again locally) to node 2 and so on? 3) Mnesia sends all records(*) to the involved nodes so they can perform their commit work in parallel. Depending on the type of transaction, Mnesia will then either wait for the local node to commit (transaction) or all nodes to commit (sync_transaction) before the function returns. *) If several tables are involved in the transaction, different sets of records may be sent to different nodes. /H?kan From info@REDACTED Mon Nov 22 18:23:38 2010 From: info@REDACTED (info) Date: Mon, 22 Nov 2010 18:23:38 +0100 Subject: The optimal value of backlog Message-ID: <201011221823376710661@its3.ch> Hello, I know that the backlog value depend on the OS and on the application. How can I determine the optimal value according to my OS (windows 2003) and my application ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From jesper.louis.andersen@REDACTED Mon Nov 22 18:26:38 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 22 Nov 2010 18:26:38 +0100 Subject: [erlang-questions] The optimal value of backlog In-Reply-To: <201011221823376710661@its3.ch> References: <201011221823376710661@its3.ch> Message-ID: On Mon, Nov 22, 2010 at 6:23 PM, info wrote: > Hello, > I know that the backlog value depend on the OS and on the application. How can I determine the optimal value according to my OS (windows 2003) and my application ? We need more information. Are you thinking of gen_tcp:listen/2 and its {backlog, B} option? -- J. From rvg@REDACTED Mon Nov 22 18:47:05 2010 From: rvg@REDACTED (Rudolph van Graan) Date: Mon, 22 Nov 2010 17:47:05 +0000 Subject: [erlang-questions] Mnesia Transactions In-Reply-To: References: Message-ID: <1D932473-2164-4955-A4D5-ECBBE92D959C@patternmatched.com> Hi, > Let us say I have a distributed table that is active on several nodes, and wish to apply a transaction to insert multiple records in that table. Is each operation inside that transaction applied individually on each node, or is the whole transaction applied on each node in turn? The answer to your question is that the entire transaction is applied in its entirety to all nodes at the same time and it is ACID compliant. > 1) Take the record in turn and insert it into node1, node2...node X , take the second record and insert into node1, node2...node X until complete. > > Or > > 2) Apply the whole transaction to node1 (insert all 1000 records locally) and then apply the fun (again locally) to node 2 and so on? Neither. The fact that the 1000 writes are all encapsulated in one mnesia:transaction(...) means the following: 1. Your process starts a transaction, because it called mnesia:transaction(...) - this is written to the LOG(*) on all nodes 2. Your process will acquire 1000 write locks and write 1000 records on the entire database (i.e. all the nodes where the table that you write to reside) - this implies network communication between your process's node and the lock managers on all the nodes holding a copy of the same table. A write lock will be required before you can write the record. 3. If your process reaches the commit decision point (i.e. it was granted all 1000 write logs and when all nodes received the entries in their LOG), this decision is recorded in the LOG (on all nodes) and each node makes the changes to the local data. If a node fails at this stage, the other nodes will continue to apply the LOG. Once the commit operation is done, this is recorded in the LOG. The failed node will recover from this when restarting. * with LOG I mean the write-ahead log (WAL) Rudolph van Graan On Nov 22, 2010, at 3:47 PM, Evans, Matthew wrote: > Hi Team, > > I have a quick question WRT mnesia and transactions, and can't seem to find the answer anywhere. > > Let us say I have a distributed table that is active on several nodes, and wish to apply a transaction to insert multiple records in that table. Is each operation inside that transaction applied individually on each node, or is the whole transaction applied on each node in turn? > > For example > > ListOfOneThousandRecords = create_records(), > mnesia:transaction(fun() -> > [mnesia:write(Rec)||Rec<-ListOfOneThousandRecords] > end). > > So this transaction will insert 1000 records into the table. > > Will mnesia: > > 1) Take the record in turn and insert it into node1, node2...node X , take the second record and insert into node1, node2...node X until complete. > > Or > > 2) Apply the whole transaction to node1 (insert all 1000 records locally) and then apply the fun (again locally) to node 2 and so on? > > Thanks > > Matt -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3822 bytes Desc: not available URL: From info@REDACTED Mon Nov 22 19:19:06 2010 From: info@REDACTED (=?utf-8?B?aW5mbw==?=) Date: Mon, 22 Nov 2010 19:19:06 +0100 Subject: =?utf-8?B?UmU6IFJlOiBbZXJsYW5nLXF1ZXN0aW9uc10gVGhlIG9wdGltYWwgdmFsdWUgb2YgYmFja2xvZw==?= References: <201011221823376710661@its3.ch>, Message-ID: <201011221919055004269@its3.ch> Yes. J-Ph. Constantin ITS3 Gen?ve www.its3.ch From jesper.louis.andersen@REDACTED Mon Nov 22 19:39:04 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 22 Nov 2010 19:39:04 +0100 Subject: [erlang-questions] The optimal value of backlog In-Reply-To: <201011221919055004269@its3.ch> References: <201011221823376710661@its3.ch> <201011221919055004269@its3.ch> Message-ID: On Mon, Nov 22, 2010 at 7:19 PM, info wrote: > Yes. The problem with backlog settings are that they depend a lot on the nature of your application. The general rule of thumb is SOMAXCONN, see http://stackoverflow.com/questions/114874/socket-listen-backlog-parameter-how-to-determine-this-value Also note that there are two queues. The listen backlog and the SYN-queue backlog (inside the OS kernel). Both might need tuning - but I am no Windows tuning expert. The difference is that the latter is the limit on the SYN queue, while the former is the limit on fully established connections (i.e., SYN -> SYN+ACK -> ACK) For rather small amounts of incoming connections, it is possible that the default of 5 is adequate. For the linux users out there, read the latter half of the listen(2) man page because it is slightly different than just throwing SOMAXCONN out. Also read tcp(7). -- J. From roberto@REDACTED Mon Nov 22 22:21:39 2010 From: roberto@REDACTED (Roberto Ostinelli) Date: Mon, 22 Nov 2010 22:21:39 +0100 Subject: [erlang-questions] fast file sending - erlang & nginx In-Reply-To: References: Message-ID: 2010/11/21 Steve Vinoski : > My experiences and measurements show pretty much the same thing. I > don't think gen_tcp necessarily targets top performance, but rather is > designed to be portable and tie cleanly into the whole inet framework, > but by doing so it suffers from unnecessary overhead in a variety of > places. Some of the problem areas we've observed are too many > {set,get}sockopt calls made that don't seem to add value, inet_db ets > lookups on each gen_tcp operation to find the socket driver, and some > handling of socket options/parameters in prim_inet that seems overly > expensive. > > gen_tcp2, anyone? that could be an interesting idea. >> On Sun, Nov 21, 2010 at 11:52 AM, Roberto Ostinelli wrote: >>> instead, i tried using the sendfile system call developed by steve >>> vinosky sendfile_drv, packaged by tuncer > > Vinoski, not Vinosky. :-) oops. sorry for that :) >>> [https://github.com/tuncer/sendfile], since in this way the file >>> reading is anyway done by a fast C system call and i would be able to >>> see if there are any differences. the results are that raw throughput >>> remains around the same. > > That surprises me, but at the very least you should see much lower CPU > with the sendfile driver. > > --steve yes, i confirm. however mine haven't been exhaustive tests at all, just some benches on a local osx with snow leopard. r. From pablo.platt@REDACTED Mon Nov 22 22:11:37 2010 From: pablo.platt@REDACTED (Pablo Platt) Date: Mon, 22 Nov 2010 13:11:37 -0800 (PST) Subject: [erlang-questions] pagination (skip, limit) ets items (gproc pids) In-Reply-To: <5A75D035-5104-4466-97D4-5B4476336E4F@erlang-solutions.com> References: <804797.46106.qm@web112604.mail.gq1.yahoo.com> <5A75D035-5104-4466-97D4-5B4476336E4F@erlang-solutions.com> Message-ID: <964633.50233.qm@web112605.mail.gq1.yahoo.com> I'll put the cursor in a process as you suggest and kill it after 10 minutes. If a user request a page and the cursor process is already dead I'll either automatically paginate until I reach the page or just send an error. Thanks ________________________________ From: Ulf Wiger To: Pablo Platt Cc: erlang-questions@REDACTED Sent: Mon, November 22, 2010 8:42:46 AM Subject: Re: [erlang-questions] pagination (skip, limit) ets items (gproc pids) Hi Pablo, Paging can be done using QLC cursors, and if you want, you can create a special pager process that repeatedly calls qlc:next_answers/2. Using gproc, you could register the pager process using a request cookie or other identifier that makes it convenient for you to find it. The following shell dialogue might illustrate: 2> [gproc:reg({n,l,K}) || K <- lists:seq(1,30)]. [true,true,true,true,true,true,true,true,true,true,true, true,true,true,true,true,true,true,true,true,true,true,true, true,true,true,true,true,true|...] 3> QH = qlc:q([X || X <- gproc:table(all)]). {qlc_handle,{qlc_lc,#Fun, {qlc_opt,false,false,-1,any,[],any,524288,allowed}}} 4> Cu = qlc:cursor(QH). {qlc_cursor,{<0.74.0>,<0.67.0>}} 5> qlc:next_answers(Cu, 3). [{{n,l,1},<0.67.0>,undefined}, {{n,l,2},<0.67.0>,undefined}, {{n,l,3},<0.67.0>,undefined}] 6> qlc:next_answers(Cu, 3). [{{n,l,4},<0.67.0>,undefined}, {{n,l,5},<0.67.0>,undefined}, {{n,l,6},<0.67.0>,undefined}] 7> qlc:next_answers(Cu, 3). [{{n,l,7},<0.67.0>,undefined}, {{n,l,8},<0.67.0>,undefined}, {{n,l,9},<0.67.0>,undefined}] BR, Ulf W On 20 Nov 2010, at 01:18, Pablo Platt wrote: > Hi > > I'm using gproc to save user sessions. > I want to present a list of online users in an admin html page. > I have a large number of sessions. > Is it possible to paginate the list similar to how SQL use skip and limit? > The admin will see a list of 1-10 online users and have links previous and next > > to see previous and next group of users. > I can't use ets continuation because I'll have to save it and reuse it which is > > hard in a website. > > The only way I see to solve it is to use ets:match(Tab, Pattern, Limit) several > > times. > For example, if the admin requests for users 50-60 I'll do ets:match 5 times > when using the returned continuation. > > Is there a better way? > Is there a reason why Skip isn't implemented even if it'll be implemented like >a > > suggested above? > > Thanks > > Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From fritchie@REDACTED Tue Nov 23 05:11:06 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Mon, 22 Nov 2010 22:11:06 -0600 Subject: [erlang-questions] application startup failure? In-Reply-To: Message of "Sat, 20 Nov 2010 12:01:24 EST." Message-ID: <37105.1290485466@snookles.snookles.com> Andy Kriger wrote: ak> Or maybe there's a better way to handle this situation? Andy, would it be awful to simply call error_logger:error_msg() or similar in your app's init() function before it returns {error, Reason}? If you have the sasl application already running, then you should also get both a crasher report and an info report that contain the returned reason. Code follows the messages. Comparing info reports, it looks like your app's Reason term really is 'shutdown'? {shrug} -Scott 1> application:start(sasl). ok 2> application:start(foo). {error,{yofoo,{foo,start,[normal,[]]}}} =ERROR REPORT==== 22-Nov-2010::22:05:20 === foo:start type normal args [] =CRASH REPORT==== 22-Nov-2010::22:05:20 === crasher: initial call: application_master:init/4 pid: <0.47.0> registered_name: [] exception exit: {yofoo,{foo,start,[normal,[]]}} in function application_master:init/4 ancestors: [<0.46.0>] messages: [{'EXIT',<0.48.0>,normal}] links: [<0.46.0>,<0.6.0>] dictionary: [] trap_exit: true status: running heap_size: 233 stack_size: 24 reductions: 102 neighbours: =INFO REPORT==== 22-Nov-2010::22:05:21 === application: foo exited: {yofoo,{foo,start,[normal,[]]}} type: temporary --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- %% foo.erl -module(foo). -author('fritchie@REDACTED'). -behaviour(application). -export([start/2, stop/1]). start(Type, StartArgs) -> error_logger:error_msg("~s:start type ~p args ~p\n", [?MODULE, Type, StartArgs]), {error, yofoo}. stop(State) -> ok. --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- %% foo.app {application, foo, [{description, "foo app"}, {vsn, "99"}, {modules, [ foo ]}, {registered, []}, {applications, [kernel, stdlib, sasl]}, {mod, { foo, []}}, {env, []} ]}. From matthias@REDACTED Tue Nov 23 08:09:09 2010 From: matthias@REDACTED (Matthias Lang) Date: Tue, 23 Nov 2010 08:09:09 +0100 Subject: how do I do the equivalent of ets:tab2list(timer_tab) for BIF timers Message-ID: <20101123070909.GA2263@corelatus.se> Hi, How can I see a list of timers started by erlang:send_after/3 and 'receive ... after' constructs? The timer module "lets" me peek at timers fairly easily, but that only works for timers started through the timer module, i.e.: 1> ets:tab2list(timer_tab). [{{1290461339356320,#Ref<0.0.0.30>}, timeout, {timer,send,[<0.31.0>,hello]}}] Is there an easier way than hacking erl_bif_timer.c? Matt From info@REDACTED Tue Nov 23 10:03:57 2010 From: info@REDACTED (info) Date: Tue, 23 Nov 2010 10:03:57 +0100 Subject: [erlang-questions] How to build a dynamic list of TCP clients? References: <201011191856197030048@its3.ch>, <4CE6B77F.6020403@frcuba.co.cu> Message-ID: <201011231003567818751@its3.ch> Good idea but this module is experimental, isn't it ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From kenji.rikitake@REDACTED Tue Nov 23 09:34:07 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 23 Nov 2010 17:34:07 +0900 Subject: How long a NIF can block the Erlang BEAM? Message-ID: <20101123083407.GA1414@k2r.org> Kenneth and all: I wonder how long a NIF can block the Erlang BEAM. To put it another way, I want to know the maximum time allowed a NIF can run in the BEAM. One millisecond? Or 100 microseconds? Or even shorter? I am asking this to find out if I can run a fairly large operation of initialized a random number table in a NIF. I haven't really benchmarked the program, but the code is already there at https://github.com/jj1bdx/sfmt-erlang/blob/master/c_src/sfmt_nif.c https://github.com/jj1bdx/sfmt-erlang What I want to do is to manipulate bits and bytes. No external I/O included. Any suggestions welcome. Kenji Rikitake (I've been catching up the mailing list so late, so I decided to put this into another mailing list thread.) In the message dated Wed, Oct 27, 2010 at 09:13:02PM +0200, Kenneth Lundin writes: > My and the OTP teams view in this matter is: > > - Reimplementing standard functionality already written in C (but as a > driver and with asynch thread support) as NIFs is generally a bad > idea) > - Implementing potentially blocking function calls with NIFs is a bad idea. > - You should have VERY strong reasons for writing NIFs at all. It is a > big point in not writing anything in C if it can be avoided. > - The implementation of NIFs is more modern than the driver concept > and among other things the passing of data between Erlang and C-code > is more efficient for NIFs than for drivers. The driver concept does > still have its place and advantages especially for > handling external asynchronous input to Erlang processes. We plan to > improve the driver mechanisms and make solutions > from the NIFs to be available when writing drivers as well. > > If it is correct that NIFs for file operations is 2 times faster than > the current file operations in raw mode we will do something about it > because that difference is not justified. But first we must > investigate if that really is the case and where the performance is > lost. > > /Kenneth Erlang/OTP Ericsson > > On Wed, Oct 27, 2010 at 9:46 AM, Edmond Begumisa > wrote: > > Hi, > > > > I hope the Couch team isn't planning on doing this by default -- something > > about it makes me nervous... > > > > When CouchDB is on it's own, it might not be alarming/noticeable, but I'm > > using CouchDB "embedded" in a wider Erlang/OTP application stack (i.e. where > > Couch is just one of many OTP apps running in the *SAME* VM -- I have a few > > hacks for avoiding socket communication.) I too worry about the potential > > for NIF-endowed couch io disturbing the balance of Erlang's scheduling. > > > > It would be good to see similar benchmarking with the VM concurrently doing > > things other than handling couch-related requests (which are implicitly > > synchronised in your case.) > > > > - Edmond - > > > > On Mon, 25 Oct 2010 07:59:11 +1100, Joel Reymont wrote: > > > >> > >> On Oct 24, 2010, at 7:16 PM, Kenneth Lundin wrote: > >> > >>> I wonder how responsive the system is to other events when running the > >>> benchmark. > >> > >> The benchmark simulates several hundred clients hitting a (mochiweb) web > >> server to read and write couchdb (json) documents. The system seems to stay > >> -highly- responsive, 10x so compared to the same system not using NIFs at > >> all. > >> > >> If low write response time is taken as a measure of system responsiveness > >> then the first graph shows that the responsiveness of the system has > >> increased dramatically. The write response here is the take taken to process > >> a web request to write a couch document. > >> > >> --- > >> http://twitter.com/wagerlabs > >> > >> > >> ________________________________________________________________ > >> erlang-questions (at) erlang.org mailing list. > >> See http://www.erlang.org/faq.html > >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From kenji.rikitake@REDACTED Tue Nov 23 10:42:32 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 23 Nov 2010 18:42:32 +0900 Subject: [erlang-questions] Why Erlang/OTP binaries are available only for Win32 ? In-Reply-To: References: <20101107195134.GA12011@corelatus.com> Message-ID: <20101123094232.GB1414@k2r.org> On FreeBSD you can have the latest packages (pre-built binaries) available at the FreeBSD ftp sites. "Jimmy" Giacomo Olgeni has been an active maintainer of the port (set of makefile and other directives to automatically fetch and build a software package). Kenji Rikitake From jesper.louis.andersen@REDACTED Tue Nov 23 11:08:21 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 23 Nov 2010 11:08:21 +0100 Subject: [erlang-questions] How to build a dynamic list of TCP clients? In-Reply-To: <201011231003567818751@its3.ch> References: <201011191856197030048@its3.ch> <4CE6B77F.6020403@frcuba.co.cu> <201011231003567818751@its3.ch> Message-ID: Perhaps, but if nobody are first movers it will keep being experimental. And redoing the work in gproc yourself is what will otherwise happen in most cases. On Nov 23, 2010 10:04 AM, "info" wrote: > Good idea but this module is experimental, isn't it ? > > J-Ph. Constantin > ITS3 Gen?ve > www.its3.ch From alceste@REDACTED Tue Nov 23 11:10:02 2010 From: alceste@REDACTED (Alceste Scalas) Date: Tue, 23 Nov 2010 11:10:02 +0100 Subject: [erlang-questions] How long a NIF can block the Erlang =?UTF-8?Q?BEAM=3F?= In-Reply-To: <20101123083407.GA1414@k2r.org> References: <20101123083407.GA1414@k2r.org> Message-ID: On Tue, 23 Nov 2010 17:34:07 +0900, Kenji Rikitake wrote: > I wonder how long a NIF can block the Erlang BEAM. To put it another > way, I want to know the maximum time allowed a NIF can run in the > BEAM. > One millisecond? Or 100 microseconds? Or even shorter? > > I am asking this to find out if I can run a fairly large operation of > initialized a random number table in a NIF. AFAIK, there are no hard limits: a NIF could block its Erlang VM thread as long as needed --- but long-running NIFs will disrupt the scheduling of other Erlang processes, and reduce the responsiveness of unrelated applications running in the same VM. If you need to perform a long operation, and latency is not an issue, you could spawn a dedicated NIF thread (using enif_thread_create()), and let it send its return value to the calling Erlang process (using enif_send()). Thus, in your sfmt.erl file, you could do something like this: init_gen_rand() -> unexported_nif_init_gen_rand(self()), receive InitGenRandResult -> InitGenRandResult end. This way, init_gen_rand/0 will still block until the PRNG is ready, but its Erlang process will be scheduled away. BTW: if you use this strategy, you may also need to perform the PRNG initialization within the "on_load" function (otherwise, you may need to protect the PRNG table with locks...). Regards, -- Alceste Scalas From ulf.wiger@REDACTED Tue Nov 23 11:23:50 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Tue, 23 Nov 2010 11:23:50 +0100 Subject: [erlang-questions] How to build a dynamic list of TCP clients? In-Reply-To: <201011231003567818751@its3.ch> References: <201011191856197030048@its3.ch>, <4CE6B77F.6020403@frcuba.co.cu> <201011231003567818751@its3.ch> Message-ID: On 23 Nov 2010, at 10:03, info wrote: > Good idea but this module is experimental, isn't it ? If you mean gproc, then no, it isn't experimental. I believe quite a few people use it in actual systems. The distributed part of gproc still has experimental status, but that implementation is separate from the local registry, and doesn't run by default. BR, Ulf W Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From matthias@REDACTED Tue Nov 23 11:30:40 2010 From: matthias@REDACTED (Matthias Lang) Date: Tue, 23 Nov 2010 11:30:40 +0100 Subject: [erlang-questions] How long a NIF can block the Erlang BEAM? In-Reply-To: References: <20101123083407.GA1414@k2r.org> Message-ID: <20101123103040.GA3635@corelatus.se> Kenji> >I wonder how long a NIF can block the Erlang BEAM. To put it another Kenji> >way, I want to know the maximum time allowed a NIF can run in the Kenji> >BEAM. Kenji> >One millisecond? Or 100 microseconds? Or even shorter? Alceste> AFAIK, there are no hard limits: a NIF could block its Erlang Alceste> VM thread as long as needed --- but long-running NIFs will Alceste> disrupt the scheduling of other Erlang processes, and reduce Alceste> the responsiveness of unrelated applications running in the Alceste> same VM. Exactly. The limit is whatever the tolerable latency in your application is. If it's fine for your application to freeze for 500ms, then a NIF which takes 500ms to run is ok. The rest of Erlang is made so that an application can react to things within a few milliseconds, so 100 microseconds is unlikely to give anyone a nasty surprise. 100 milliseconds will. Matt From cbenac@REDACTED Tue Nov 23 11:45:05 2010 From: cbenac@REDACTED (Clara Benac Earle) Date: Tue, 23 Nov 2010 11:45:05 +0100 Subject: [ANN] Madrid Erlounge 2nd December Message-ID: <4CEB9B31.7010200@fi.upm.es> Dear all, The Madrid Erlang user group will be meeting on Thursday 2nd of December at 19:30 at the Escuela Universitaria de Ingenier?a T?cnica Industrial. Ronda de Valencia 3 (metro/cercanias Embajadores). Jordi Chac?n from Klarna will be giving a talk. More details (abstract, etc.) soon to follow. Around 9pm we will go somewhere in the neighbourhood for an Erlounge (beer, food and talk). Everybody is welcome to join us. Please drop me an email if you are planning to attend so I know roughly how many people are coming. Looking forward to seeing you in Madrid! Clara From kenji.rikitake@REDACTED Tue Nov 23 12:00:13 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 23 Nov 2010 20:00:13 +0900 Subject: [erlang-questions] How long a NIF can block the Erlang BEAM? In-Reply-To: <20101123103040.GA3635@corelatus.se> References: <20101123083407.GA1414@k2r.org> <20101123103040.GA3635@corelatus.se> Message-ID: <20101123110013.GA22113@k2r.org> Thanks Alceste and Mattias. I was thinking about using SFMT instead of the random module, and was worrying about the scheduling disruption issue. Functions in the random module is found at many places among the Erlang/OTP libraries, so I thought I had to be conservative. The enif_thread_create() and enif_send() pair Alceste wrote looks very interesting, though I also guess it's a bit too complicated for a random number generator. Regards, Kenji Rikitake In the message <20101123103040.GA3635@REDACTED> dated Tue, Nov 23, 2010 at 11:30:16AM +0100, Matthias Lang writes: > Kenji> >I wonder how long a NIF can block the Erlang BEAM. To put it another > Kenji> >way, I want to know the maximum time allowed a NIF can run in the > Kenji> >BEAM. > Kenji> >One millisecond? Or 100 microseconds? Or even shorter? > > Alceste> AFAIK, there are no hard limits: a NIF could block its Erlang > Alceste> VM thread as long as needed --- but long-running NIFs will > Alceste> disrupt the scheduling of other Erlang processes, and reduce > Alceste> the responsiveness of unrelated applications running in the > Alceste> same VM. > > Exactly. The limit is whatever the tolerable latency in your > application is. If it's fine for your application to freeze for 500ms, > then a NIF which takes 500ms to run is ok. > > The rest of Erlang is made so that an application can react to things > within a few milliseconds, so 100 microseconds is unlikely to give > anyone a nasty surprise. 100 milliseconds will. > > Matt From kenji.rikitake@REDACTED Tue Nov 23 12:15:42 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 23 Nov 2010 20:15:42 +0900 Subject: Implementation of a 2006 version of Wichmann-Hull random number generator for Erlang/OTP Message-ID: <20101123111542.GB22113@k2r.org> The random module of Erlang/OTP library uses the 1982(!)[1] version of Wichmann-Hull random number generator (RNG), which has been known to have a limitation of short cycle length (~ 7 x (10^12)). The authors have published a new version of 2006[2], which has a much longer cycle length (~ 2^120 (~ 10^36)) at the expense of adding one more variable to the seed/state (from three 16-bit integers to four 32-bit integers). I've briefly tested the implementation, available at: https://gist.github.com/709614 under the name of random_wh06.erl [1] B. A. Wichmann, I. D. Hill, Algorithm AS 183: An efficient and portable pseudo-random number generator, Journal of Applied Statistics, 31, 188-190 (1982). [2] B. A. Wichmann, I. D. Hill, Generating good pseudo-random numbers, Computational Statistics & Data Analysis, 51, 1614-1622 (2006). Reference implementation in C and Ada with the drafts of the paper PDFs available at: http://resource.npl.co.uk/docs/science_technology/scientific_computing/ssfm/documents/wh_rng_version096.zip Suggestions and comments appreciated. Regards, Kenji Rikitake From jesper.louis.andersen@REDACTED Tue Nov 23 13:20:33 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Tue, 23 Nov 2010 13:20:33 +0100 Subject: Erlang overload/bandwidth limitation (JOBS?) Message-ID: Hi List, The etorrent application, a bittorrent client for erlang, is in need of some load balancing in two ways, which I describe below, and I need a tool to do some load balancing in the two cases. Are there any suggestions besides using the Erlang Solutions JOBS application for the task? Case 1: The client can handle multiple torrents at once. When you start off on a torrent and you find a file existing in the download directory matching the name and size of what you want, you have to check it. This check is running SHA1 checksums over the whole file and thus puts quite a burden on the disk IO subsystem of the computer. In the lucky case that we already did some checking and the file passes a sanity check, we can look up a lot of the checking result in a persistent table - but in general we have to go over the file. To not trash the disk we only want to allow a single torrent to be checked at a time. The current solution is ugly. A token is passed among the torrents who wants checking and he who has the token may check. There are some monitoring going on to ensure that if we loose the token, we reinject a new one in the system. As such, there is some reminiscence of a token ring configuration. JOBS seems like a good fit for this case: Add a job queue with a severe limitation of 1. Fire off a job to the queue (which you make large enough that it can hold the pending jobs) which is the actual checking code. Done. Case 2: Upstream bandwidth limitation. TCP will by default eat all of a lines upstream bandwidth, which unfortunately includes ACK-packets needed for the downstream. The usual trick is to put some limitation to how much you send out from the client to crudely steer the connection. A very big problem here is that most internet connections from homes have cheap routers in front of them and these have a small buffer of, say, 8 packets. Any more packets arriving are simply dropped from the router. This makes TCP sad and erratic such that it cannot really do its job too well. There is an UDP based protocol extension, ?TP, which attempt to battle this problem directly but that protocol is a bit out in the future, hence the bandwidth limitation. The question is, can JOBS or some other overload framework be easily adapted to this scenario? Discussion: Basically, you have a lot of willing senders contending for a global upstream bandwidth resource. You allow them access to the resource but you don't want one sender to claim all of the resource. Hence you can round-robin schedule the bandwidth in smaller chunks or use RED - random early detection; in which the greatest hoarder runs a larger risk of being denied bandwidth. I don't know if the JOBS framework can do this in full or in part. -- J. From cristian@REDACTED Tue Nov 23 13:39:52 2010 From: cristian@REDACTED (Cristian Greco) Date: Tue, 23 Nov 2010 13:39:52 +0100 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hull random number generator for Erlang/OTP In-Reply-To: <20101123111542.GB22113@k2r.org> References: <20101123111542.GB22113@k2r.org> Message-ID: <20101123133952.1909c4c7@regolo> On Tue, 23 Nov 2010 20:15:42 +0900 Kenji Rikitake wrote: > The random module of Erlang/OTP library uses the 1982(!)[1] version of > Wichmann-Hull random number generator (RNG), which has been known to > have a limitation of short cycle length (~ 7 x (10^12)). > [...] Hi Kenji, the current implementation of the WH prng is Erlang is also slightly broken. I sent a patch some weeks ago, it is still cooking in pu: http://www.erlang.org/cgi-bin/ezmlm-cgi?3:1556 http://www.erlang.org/cgi-bin/ezmlm-cgi?3:1557 Thanks, -- Cristian Greco GPG key ID: 0xCF4D32E4 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From kenji.rikitake@REDACTED Tue Nov 23 13:59:19 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Tue, 23 Nov 2010 21:59:19 +0900 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hull random number generator for Erlang/OTP In-Reply-To: <20101123133952.1909c4c7@regolo> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> Message-ID: <20101123125919.GA24633@k2r.org> Greco: Yes - I remembered your code pieces and bug reports. I should add that giving zero to any of the seed tuple elements will render the random number generator useless because the zeros will stay forever, and that this should be checked at least in the (re)seeding functions. (This is a known limitation of the algorithm based on multiplicative congluency generators.) Regards, Kenji Rikitake In the message <20101123133952.1909c4c7@REDACTED> dated Tue, Nov 23, 2010 at 01:39:28PM +0100, Cristian Greco writes: > On Tue, 23 Nov 2010 20:15:42 +0900 > Kenji Rikitake wrote: > > > The random module of Erlang/OTP library uses the 1982(!)[1] version of > > Wichmann-Hull random number generator (RNG), which has been known to > > have a limitation of short cycle length (~ 7 x (10^12)). > > [...] > > Hi Kenji, > > the current implementation of the WH prng is Erlang is also slightly > broken. I sent a patch some weeks ago, it is still cooking in pu: > > http://www.erlang.org/cgi-bin/ezmlm-cgi?3:1556 > http://www.erlang.org/cgi-bin/ezmlm-cgi?3:1557 > > Thanks, > -- > Cristian Greco > GPG key ID: 0xCF4D32E4 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From paul.joseph.davis@REDACTED Tue Nov 23 14:57:25 2010 From: paul.joseph.davis@REDACTED (Paul Davis) Date: Tue, 23 Nov 2010 08:57:25 -0500 Subject: [erlang-questions] How long a NIF can block the Erlang BEAM? In-Reply-To: <20101123110013.GA22113@k2r.org> References: <20101123083407.GA1414@k2r.org> <20101123103040.GA3635@corelatus.se> <20101123110013.GA22113@k2r.org> Message-ID: On Tue, Nov 23, 2010 at 6:00 AM, Kenji Rikitake wrote: > Thanks Alceste and Mattias. > > I was thinking about using SFMT instead of the random module, and was > worrying about the scheduling disruption issue. ?Functions in the random > module is found at many places among the Erlang/OTP libraries, so I > thought I had to be conservative. > > The enif_thread_create() and enif_send() pair Alceste wrote looks very > interesting, though I also guess it's a bit too complicated for a random > number generator. > > Regards, > Kenji Rikitake > > In the message <20101123103040.GA3635@REDACTED> > dated Tue, Nov 23, 2010 at 11:30:16AM +0100, > Matthias Lang writes: >> Kenji> >I wonder how long a NIF can block the Erlang BEAM. ?To put it another >> Kenji> >way, I want to know the maximum time allowed a NIF can run in the >> Kenji> >BEAM. >> Kenji> >One millisecond? Or 100 microseconds? Or even shorter? >> >> Alceste> AFAIK, there are no hard limits: a NIF could block its Erlang >> Alceste> VM thread as long as needed --- but long-running NIFs will >> Alceste> disrupt the scheduling of other Erlang processes, and reduce >> Alceste> the responsiveness of unrelated applications running in the >> Alceste> same VM. >> >> Exactly. The limit is whatever the tolerable latency in your >> application is. If it's fine for your application to freeze for 500ms, >> then a NIF which takes 500ms to run is ok. >> >> The rest of Erlang is made so that an application can react to things >> within a few milliseconds, so 100 microseconds is unlikely to give >> anyone a nasty surprise. 100 milliseconds will. >> >> Matt > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > Quite randomly I just implemented this yesterday. I haven't tested this code other than to check that send_to_self works in the Erlang shell and it survives an init:restart/0 which can wreak havoc with thread maintenance. https://github.com/davisp/nif-examples/blob/master/termsend_threaded/c_src/termsend_threaded.c Its pretty much just what's being discussed. A thread is spawned in the load function and waits for requests from Erlang for a random number. HTH, Paul Davis From netch@REDACTED Tue Nov 23 15:10:24 2010 From: netch@REDACTED (Valentin Nechayev) Date: Tue, 23 Nov 2010 16:10:24 +0200 Subject: systematic global registration discrepancies Message-ID: <20101123141024.GA12822@netch.kiev.ua> Hi, we are using Erlang cluster for 20-25 nodes which all resides on different hosts. We are experiencing systematic problems with global registration of following kinds: 1. Attempt of register via global:register_name() hangs for an unlimited time (we could see it for a few hours until our patience is expired). 2. A name which is successfully reported as registered disappears from registered name lists at all nodes (including the registering one!) We had to add monitoring of global functionality which stops the node where registration hangs. It periodically detects registration failure and stops nodes, usually this is group of 7-10 nodes per one such failure. But it can't detect second case (silent disappearing). We use R12B5; it's planned to upgrade but is impossible for the closest next release. Did anybody seen this? Please suggest how to debug such problem. -netch- From cristian@REDACTED Tue Nov 23 15:58:47 2010 From: cristian@REDACTED (Cristian Greco) Date: Tue, 23 Nov 2010 15:58:47 +0100 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hull random number generator for Erlang/OTP In-Reply-To: <20101123125919.GA24633@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> Message-ID: <20101123155847.77783a5f@regolo> On Tue, 23 Nov 2010 21:59:19 +0900 Kenji Rikitake wrote: > Greco: Yes - I remembered your code pieces and bug reports. > > I should add that giving zero to any of the seed tuple elements will > render the random number generator useless because the zeros will stay > forever, and that this should be checked at least in the (re)seeding > functions. (This is a known limitation of the algorithm based on > multiplicative congluency generators.) Sure, you're right. This happens for seeds which are multiples of the modules used by the generator: 1> S = [ {N*30269,N*30307,N*30323} || N <- lists:seq(0,4) ]. [{0,0,0}, {30269,30307,30323}, {60538,60614,60646}, {90807,90921,90969}, {121076,121228,121292}] 2> lists:map(fun({A,B,C}) -> random:seed(A,B,C), io:format("~p~n", [[random:uniform() || _ <- lists:seq(1,10)]]) end, S). [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] [ok,ok,ok,ok,ok] The actual prng in Erlang is intrinsecally limited in many ways. IMHO you should rework your MT nif implementation and submit it as a patch for stdlib. A bonus would be the code to provide access to OS-dependant randomness sources (e.g. /dev/urandom in *nix, CryptoApi in windows), in order to automatically initialize the generator with a "non-hardcoded" seed. Thanks, -- Cristian Greco GPG key ID: 0xCF4D32E4 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From fritchie@REDACTED Tue Nov 23 18:37:14 2010 From: fritchie@REDACTED (Scott Lystig Fritchie) Date: Tue, 23 Nov 2010 11:37:14 -0600 Subject: [erlang-questions] How long a NIF can block the Erlang BEAM? In-Reply-To: Message of "Tue, 23 Nov 2010 20:00:13 +0900." <20101123110013.GA22113@k2r.org> Message-ID: <83998.1290533834@snookles.snookles.com> Kenji Rikitake wrote: kr> The enif_thread_create() and enif_send() pair Alceste wrote looks kr> very interesting, though I also guess it's a bit too complicated for kr> a random number generator. Kenji, if creating and joining new Pthreads is overkill, there's another "you must be joking" way: just increase the number of scheduler threads. The "erl +A" flag is just waiting to be (ab)used this way. :-) If you needed to avoid having too many of these running at one time, and each one is going to run for 10s or 100s of milliseconds (or more), then writing a simple gen_server to provide a rate-limiting/admission-control mechanism would have relatively zero overhead. -Scott From armando.dicianno@REDACTED Tue Nov 23 19:12:18 2010 From: armando.dicianno@REDACTED (Armando Di Cianno) Date: Tue, 23 Nov 2010 13:12:18 -0500 Subject: Adding a second supervisor / gen_server to an application? Message-ID: I have an application (behavior) implemented that starts a supervisor, which takes care of starting the server/s (gen_server). This server accepts many connections, and I believe I generally have the supervisor <-> gen_server interaction laid out correctly. I would like to add a second server to the application, and for it to have it's own supervisor. The second server will be cleaning up garbage the first server/s creates, so there only needs to be one. Is a call to a second supervisor:start_link() normal or "ok" to add to applicaiton:start()? I'm guessing the answer is no, since the return value from application start is the {ok, Pid} from the first supervisor. Thanks for any input, __armando From armando.dicianno@REDACTED Tue Nov 23 19:18:19 2010 From: armando.dicianno@REDACTED (Armando Di Cianno) Date: Tue, 23 Nov 2010 13:18:19 -0500 Subject: [erlang-questions] Adding a second supervisor / gen_server to an application? In-Reply-To: References: Message-ID: On Tue, Nov 23, 2010 at 1:15 PM, Jesper Louis Andersen wrote: > Build a tree of supervisors. /me facepalm at my own question The obviousness of this completely alluded me. Thanks! __armando > > On Tue, Nov 23, 2010 at 7:12 PM, Armando Di Cianno > wrote: >> I have an application (behavior) implemented that starts a supervisor, >> which takes care of starting the server/s (gen_server). This server >> accepts many connections, and I believe I generally have the >> supervisor <-> gen_server interaction laid out correctly. >> >> I would like to add a second server to the application, and for it to >> have it's own supervisor. The second server will be cleaning up >> garbage the first server/s creates, so there only needs to be one. >> >> Is a call to a second supervisor:start_link() normal or "ok" to add to >> applicaiton:start()? I'm guessing the answer is no, since the return >> value from application start is the {ok, Pid} from the first >> supervisor. >> >> Thanks for any input, >> __armando >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > > > -- > J. > From hd2010@REDACTED Tue Nov 23 23:59:48 2010 From: hd2010@REDACTED (Henning Diedrich) Date: Tue, 23 Nov 2010 23:59:48 +0100 Subject: Lua Driver Message-ID: <4CEC4764.2020905@eonblast.com> Hi folks, I posted some details on Lua embedding here: http://www.eonblast.com/blog/optimizing-erlualib-calls/ It's about an Erlang embedded Lua driver, discussing in detail how to go beyond single step stack manipulation port calls towards directly accessing the Lua C API in one go. Feedback and critique very welcome. Regards, Henning From ok@REDACTED Wed Nov 24 00:49:01 2010 From: ok@REDACTED (Richard O'Keefe) Date: Wed, 24 Nov 2010 12:49:01 +1300 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hull random number generator for Erlang/OTP In-Reply-To: <20101123111542.GB22113@k2r.org> References: <20101123111542.GB22113@k2r.org> Message-ID: <3837006F-9632-4C25-A14C-EB54E8385BB7@cs.otago.ac.nz> I made the mistake of downloading the reference implementation of the new Wichmann-Hill RNG. Unfortunately, it is *licenced* software, with provisions like The User may only make one copy of the Software and any associated written material, as a back up. Arguably, this means that having downloaded it on my desktop machine, for which backups are *automatically* made, I do not have the right to copy it to my laptop. Another restriction: The User will not rent, lease, sublicense, transfer, resell the Software (either in whole or in part) or use it for commercial purposes without NPL?s prior written permission. This pretty clearly means you can't put it in open source software. It's not completely clear just what NPL intend here. I wrote to Wichmann about it and he said that kind of stuff was outside his expertise, but if there was any doubt, change one of the primes. It's probably worth asking NPL formally if they have any problem with free implementations *not* derived from their code. A thing that neither the 'small.pdf' nor the 'long.pdf' version of the paper that comes with the reference implementation discusses is how to initialise the generator. They *do* describe how to set up multiple instances of the generator, *assuming* that you know how to set up a single instance. The NPL package sets it from the time() function, which clearly provides at most 32 bits. The generator has 16 bytes worth of state. In C, it might be advisable to use gettimeofday() and get a few more bits. To get the full range of possible RNG states, you're looking at using /dev/random. From kenji.rikitake@REDACTED Wed Nov 24 06:11:55 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Wed, 24 Nov 2010 14:11:55 +0900 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hill random number generator for Erlang/OTP In-Reply-To: <20101123111542.GB22113@k2r.org> References: <20101123111542.GB22113@k2r.org> Message-ID: <20101124051155.GA38476@k2r.org> The 2006 Wichmann-Hill RNG for Erlang moved to https://gist.github.com/713144 under the name of random_wh06.erl (The gist number changed due to possible licensing issues. Sorry for your inconvenience.) Kenji Rikitake In the message <20101123111542.GB22113@REDACTED> dated Tue, Nov 23, 2010 at 08:15:18PM +0900, Kenji Rikitake writes: > The random module of Erlang/OTP library uses the 1982(!)[1] version of > Wichmann-Hull random number generator (RNG), which has been known to > have a limitation of short cycle length (~ 7 x (10^12)). The authors > have published a new version of 2006[2], which has a much longer cycle > length (~ 2^120 (~ 10^36)) at the expense of adding one more variable to > the seed/state (from three 16-bit integers to four 32-bit integers). > [1] > B. A. Wichmann, I. D. Hill, > Algorithm AS 183: An efficient and portable pseudo-random number generator, > Journal of Applied Statistics, 31, 188-190 (1982). > > [2] > B. A. Wichmann, I. D. Hill, > Generating good pseudo-random numbers, > Computational Statistics & Data Analysis, 51, 1614-1622 (2006). From kenji.rikitake@REDACTED Wed Nov 24 06:39:27 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Wed, 24 Nov 2010 14:39:27 +0900 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hull random number generator for Erlang/OTP In-Reply-To: <3837006F-9632-4C25-A14C-EB54E8385BB7@cs.otago.ac.nz> References: <20101123111542.GB22113@k2r.org> <3837006F-9632-4C25-A14C-EB54E8385BB7@cs.otago.ac.nz> Message-ID: <20101124053927.GA39018@k2r.org> Richard and all: Thanks for reminding us about the license. I understand that the paper has already been published from Elsevier (though it's not a public domain and behind the paywall), so the algorithm (including the multiplier values) are open to the public. The problem is that seeding values are in the coverage of NPL license. I've already changed the initialization values on seed/0 in my source ASAP. (tested it on my laptop) Regards, Kenji Rikitake In the message <3837006F-9632-4C25-A14C-EB54E8385BB7@REDACTED> dated Wed, Nov 24, 2010 at 12:48:37PM +1300, Richard O'Keefe writes: > I made the mistake of downloading the reference implementation of > the new Wichmann-Hill RNG. Unfortunately, it is *licenced* software, > with provisions like > > The User may only make one copy of the Software and any > associated written material, as a back up. > > Arguably, this means that having downloaded it on my desktop machine, > for which backups are *automatically* made, I do not have the right > to copy it to my laptop. Another restriction: > > The User will not rent, lease, sublicense, transfer, resell > the Software (either in whole or in part) or use it for > commercial purposes without NPL?s prior written permission. > > This pretty clearly means you can't put it in open source software. > > It's not completely clear just what NPL intend here. I wrote to > Wichmann about it and he said that kind of stuff was outside his > expertise, but if there was any doubt, change one of the primes. > It's probably worth asking NPL formally if they have any problem > with free implementations *not* derived from their code. > > A thing that neither the 'small.pdf' nor the 'long.pdf' version of > the paper that comes with the reference implementation discusses > is how to initialise the generator. They *do* describe how to set > up multiple instances of the generator, *assuming* that you know > how to set up a single instance. > > The NPL package sets it from the time() function, which clearly > provides at most 32 bits. The generator has 16 bytes worth of > state. In C, it might be advisable to use gettimeofday() and > get a few more bits. To get the full range of possible RNG > states, you're looking at using /dev/random. > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From kenji.rikitake@REDACTED Wed Nov 24 09:00:25 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Wed, 24 Nov 2010 17:00:25 +0900 Subject: New random module? In-Reply-To: <20101123155847.77783a5f@regolo> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> Message-ID: <20101124080025.GA40466@k2r.org> In the message <20101123155847.77783a5f@REDACTED> dated Tue, Nov 23, 2010 at 03:58:23PM +0100, Cristian Greco writes: > The actual prng in Erlang is intrinsecally limited in many ways. Sure it is. > IMHO you should rework your MT nif implementation and submit it as a > patch for stdlib. A bonus would be the code to provide access to > OS-dependant randomness sources (e.g. /dev/urandom in *nix, CryptoApi > in windows), in order to automatically initialize the generator with a > "non-hardcoded" seed. I've been planning to evaluate the following five PRNGs: * Current random module (Wichmann-Hill 1982, state: 6 bytes: period: ~ 10^13) * SFMT (period 2^19937-1, state: 2496 bytes) * Wichmann-Hill 2006, state: 16 bytes, period: ~ 2^120 * XORshift 7 stage (Panneton-L'Ecuyer, 2005[1], state: 8 bytes, period: 2^256-1) * SFMT with smaller state (period 2^607-1, state: 66 bytes) Obtaining entropy from /dev/urandom will be surely a good feature. (I have no Windows dev environment available at hand) Regards, Kenji Rikitake From dkoch@REDACTED Wed Nov 24 08:31:07 2010 From: dkoch@REDACTED (dkoch@REDACTED) Date: Wed, 24 Nov 2010 08:31:07 +0100 (CET) Subject: [erlang-questions] Lua Driver Message-ID: <13448848.82521290583867809.JavaMail.www@wsfrf1116> Nice, but what have Lua to offer to Erlang ? Scripting ? Concurrency ? Fault tolerancy ? Lua is pretty, fast and small (less than 5 MiB) ths you wouldn't install 100 MiB of Erlang to control Lua... David KOCH ======================================== Message du : 24/11/2010 De : "Henning Diedrich " A : "Erlang Questions" Copie ? : Sujet : [erlang-questions] Lua Driver Hi folks, I posted some details on Lua embedding here: http://www.eonblast.com/blog/optimizing-erlualib-calls/ It's about an Erlang embedded Lua driver, discussing in detail how to go beyond single step stack manipulation port calls towards directly accessing the Lua C API in one go. Feedback and critique very welcome. Regards, Henning From kenji.rikitake@REDACTED Wed Nov 24 09:41:30 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Wed, 24 Nov 2010 17:41:30 +0900 Subject: [erlang-questions] New random module? In-Reply-To: <20101124080025.GA40466@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> <20101124080025.GA40466@k2r.org> Message-ID: <20101124084130.GA41191@k2r.org> In the message <20101124080025.GA40466@REDACTED> dated Wed, Nov 24, 2010 at 05:00:01PM +0900, Kenji Rikitake writes: > * XORshift 7 stage (Panneton-L'Ecuyer, 2005[1], state: 8 bytes, period: 2^256-1) [1] Francis Panneton and Pierre L'ecuyer. 2005. On the xorshift random number generators. ACM Trans. Model. Comput. Simul. 15, 4 (October 2005), 346-361. DOI=10.1145/1113316.1113319 http://doi.acm.org/10.1145/1113316.1113319 FYI Kenji Rikitake From kenji.rikitake@REDACTED Wed Nov 24 12:03:09 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Wed, 24 Nov 2010 20:03:09 +0900 Subject: [erlang-questions] New random module? In-Reply-To: <20101124084130.GA41191@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> <20101124080025.GA40466@k2r.org> <20101124084130.GA41191@k2r.org> Message-ID: <20101124110309.GA43137@k2r.org> It's not Francis, but Francois (which the c is French cedille). Kenji In the message <20101124084130.GA41191@REDACTED> dated Wed, Nov 24, 2010 at 05:41:06PM +0900, Kenji Rikitake writes: > In the message <20101124080025.GA40466@REDACTED> > dated Wed, Nov 24, 2010 at 05:00:01PM +0900, > Kenji Rikitake writes: > > * XORshift 7 stage (Panneton-L'Ecuyer, 2005[1], state: 8 bytes, period: 2^256-1) > > [1] Francois Panneton and Pierre L'ecuyer. 2005. On the xorshift random > number generators. ACM Trans. Model. Comput. Simul. 15, 4 (October > 2005), 346-361. DOI=10.1145/1113316.1113319 http://doi.acm.org/10.1145/1113316.1113319 From pguyot@REDACTED Wed Nov 24 15:04:36 2010 From: pguyot@REDACTED (Paul Guyot) Date: Wed, 24 Nov 2010 15:04:36 +0100 Subject: [erlang-questions] How long a NIF can block the Erlang BEAM? In-Reply-To: <1290585630.27001.ezmlm@erlang.org> References: <1290585630.27001.ezmlm@erlang.org> Message-ID: > Thanks Alceste and Mattias. > > I was thinking about using SFMT instead of the random module, and was > worrying about the scheduling disruption issue. Functions in the random > module is found at many places among the Erlang/OTP libraries, so I > thought I had to be conservative. > > The enif_thread_create() and enif_send() pair Alceste wrote looks very > interesting, though I also guess it's a bit too complicated for a random > number generator. Please note that using enif_send() for this purpose only works with SMP emulators. Paul -- Semiocast http://semiocast.com/ +33.175000290 - 62 bis rue Gay-Lussac, 75005 Paris From alceste@REDACTED Wed Nov 24 16:50:18 2010 From: alceste@REDACTED (Alceste Scalas) Date: Wed, 24 Nov 2010 16:50:18 +0100 Subject: =?UTF-8?Q?enif=5Fsend=28=29=20thread=20safety?= In-Reply-To: References: <1290585630.27001.ezmlm@erlang.org> Message-ID: On Wed, 24 Nov 2010 15:04:36 +0100, Paul Guyot wrote: >> The enif_thread_create() and enif_send() pair Alceste wrote looks >> very >> interesting, though I also guess it's a bit too complicated for a >> random >> number generator. > > Please note that using enif_send() for this purpose only works with > SMP emulators. Uh, you're right. I must confess that I completely missed the following paragraph (from the docs): http://erlang.org/doc/man/erl_nif.html unsigned enif_send(...) This function is only thread-safe when the emulator with SMP support is used. It can only be used in a non-SMP emulator from a NIF-calling thread. Maybe it's due to the fact that, last time I checked, the non-SMP emulator does not lock its internal data structures, since just one thread is expected to access them. From previous discussions in this ML, I've seen that other developers are using enif_send() from separate NIF threads. See, for example: http://comments.gmane.org/gmane.comp.lang.erlang.general/48788 If people start using this (very convenient) feature, we could see a whole new class of Erlang modules which depend on the type of emulator being used... And it may be quite confusing, IMVHO. Regards, -- Alceste Scalas From co7eb@REDACTED Wed Nov 24 17:43:01 2010 From: co7eb@REDACTED (=?iso-8859-1?Q?Gilberio_Carmenates_Garc=EDa?=) Date: Wed, 24 Nov 2010 13:43:01 -0300 Subject: The most better Erlang web server! Message-ID: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Hi what is the best framework and web server for web develop, like yaws, nitrogen, etc. I need of your consideration and knowledge an advice of what is the best web server of all. Cheers, Ivan. ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From demeshchuk@REDACTED Wed Nov 24 19:44:06 2010 From: demeshchuk@REDACTED (Dmitry Demeshchuk) Date: Wed, 24 Nov 2010 21:44:06 +0300 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: All the servers have their advantages and disadvantages, each of them has its specials making it better for some tasks. If you aim for great performance, have a look at mochiweb or misultin. If you need a lot of features or have a web server from the box, without having to write any extra code, try yaws. The same concerns web frameworks. Have a look at nitrogen and zotonic if you want something powerful. Finally, sometimes it's all about your taste. You'd better play with them all and then choose your favorite one. 2010/11/24 Gilberio Carmenates Garc?a : > Hi what is the best framework and web server for web develop, like yaws, > nitrogen, etc. I need of your consideration and knowledge an advice of what > is the best web server of all. > > > > > > > > Cheers, > > > > Ivan. > > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de ?usar el servicio a tales fines y cumplir con las regulaciones establecidas. > -- Best regards, Dmitry Demeshchuk From max.lapshin@REDACTED Wed Nov 24 21:16:35 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Wed, 24 Nov 2010 23:16:35 +0300 Subject: Send message to thousands of clients Message-ID: I remember, that there was such a discussion, but can't find it. So, I have a problem: about 40 times per second process ems_media gets a frame, performs some actions and then retransmit it up to 1500 clients and more. Problem is in bad responsiveness of this process. Clients cannot wait when it reply for some tasks such as unsubscribe, etc. Simple benchmarks shows that [Pid ! Message || Pid <- List] can take upto 13 milliseconds when List length is 2000. Question is: what should I do? Current idea is to create 5-10 retransmit processes, that will store about 100-300 of pids each and ems_media will send frames only to them. After that, these processes, each on separate core, will send messages to clients. Or there is a better idea? From fred.hebert@REDACTED Wed Nov 24 21:27:41 2010 From: fred.hebert@REDACTED (=?iso-8859-1?Q?Fr=E9d=E9ric_Trottier-H=E9bert?=) Date: Wed, 24 Nov 2010 15:27:41 -0500 Subject: [erlang-questions] Send message to thousands of clients In-Reply-To: References: Message-ID: <318AFB3F-6AA6-4194-8955-18B9260D3BB9@erlang-solutions.com> I remember some benchmark case where the idea was to raise the sender's priority with process_flag(priority, low | normal | high | max). Raise it as much as needed when sending (max should usually be kept for erts processes), and then turn it back to normal as soon as you're done; raising the priority to high should ideally be for short periods of time. On 2010-11-24, at 15:16 PM, Max Lapshin wrote: > I remember, that there was such a discussion, but can't find it. > > So, I have a problem: about 40 times per second process ems_media gets > a frame, performs some actions and then retransmit it up to 1500 > clients and more. > > Problem is in bad responsiveness of this process. Clients cannot wait > when it reply for some tasks such as unsubscribe, etc. > > Simple benchmarks shows that > [Pid ! Message || Pid <- List] > > can take upto 13 milliseconds when List length is 2000. > > Question is: what should I do? > > Current idea is to create 5-10 retransmit processes, that will store > about 100-300 of pids each and ems_media will send frames only to > them. After that, these processes, each on separate core, will send > messages to clients. > > Or there is a better idea? > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > -- Fred H?bert http://www.erlang-solutions.com From andy.kriger@REDACTED Wed Nov 24 23:22:14 2010 From: andy.kriger@REDACTED (Andy Kriger) Date: Wed, 24 Nov 2010 17:22:14 -0500 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: It really depends on what you need - there are several choices for HTTP servers: yaws, mochiweb, and mislutin. Easy enough for you to run some simple flood tests using a tool like Tsung (http://tsung.erlang-projects.org/) to see which meets your needs. I suspect, to get up-and-running, it really doesn't matter which one you go with. Of course there are differences - I don't mean to suggest there aren't, but in the early stages of building an app, it shouldn't become a sticking point. There are also different frameworks built on top of these servers. Nitrogen seems aimed at building webapps out-of-the-box and letting Nitrogen do the heavy lifting (like providing APIs for Ajax, Comet, form submission, etc). Webmachine's architecture is built around delivering RESTful services. Zotonic is a content-management system (CMS), so that's an entirely different domain. I'm sure there are others I've missed, but speaking as a newcomer to Erlang myself, I've found it very easy to get up-and-running on any of these servers and frameworks, using rebar and their quick-start guides. Play around and see what works for you. 2010/11/24 Gilberio Carmenates Garc?a : > Hi what is the best framework and web server for web develop, like yaws, > nitrogen, etc. I need of your consideration and knowledge an advice of what > is the best web server of all. > > > > > > > > Cheers, > > > > Ivan. > > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de ?usar el servicio a tales fines y cumplir con las regulaciones establecidas. > -- Be well, andy Welcome to http://householder-yogi.net On family, NYC, and practicing yoga. From ok@REDACTED Wed Nov 24 23:35:12 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 25 Nov 2010 11:35:12 +1300 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hill random number generator for Erlang/OTP In-Reply-To: <20101124051155.GA38476@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101124051155.GA38476@k2r.org> Message-ID: On 24/11/2010, at 6:11 PM, Kenji Rikitake wrote: > The 2006 Wichmann-Hill RNG for Erlang moved to > https://gist.github.com/713144 > under the name of > random_wh06.erl Didn't we just have a discussion about not allowing zeros? Shouldn't seed/4 be seed(A1, A2, A3, A4) -> put(random_wh06_seed, {abs(A1) rem 2147483578 + 1, abs(A2) rem 2147483542 + 1, abs(A3) rem 2147483422 + 1, abs(A4) rem 2147483122 + 1}). Or did I misunderstand? From jan@REDACTED Wed Nov 24 23:59:25 2010 From: jan@REDACTED (Jan Koum) Date: Wed, 24 Nov 2010 14:59:25 -0800 Subject: FreeBSD 8 + R14B problems Message-ID: hi there, a few months ago we used to run FreeBSD 7.3 and R13B4. our cluster started having major problems, so we upgraded to FreeBSD 8 and R14B. ever since upgrading we have two nagging problems which caused us problems and user visible downtime: 1. every few days we get "corrupted external term" from erts/emulator/beam/external.c:bad_dist_ext() which in turns calls erts_kill_dist_connection(). the atom cache string that follow is: ATOM_CACHE_REF translations: 0='ejabberd@REDACTED', 1='', 2=xmlelement, 3=xmlcdata, 4=jid, 5=never, 6=offline_msg 2. our front-end servers run on single proc Xeon-Westmere 5670 Hex core with hyperthreading, in essence getting 12 CPUs: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads every few days one of our front-end nodes goes into a weird state where only four or two threads are running any doing work instead of 12. this eventually causes each one of them to hit CPU limit and node starts getting back log, brining entire cluster down with it. it almost feels like a bug in the erlang migration logic in how it handles schedulers and runqueues. here is a compare of the top(1) output: healthy machine: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 28414 whatsapp 64 0 23073M 13302M CPU0 0 25:55 27.49% {beam.smp} 28414 whatsapp 65 0 23073M 13302M ucond 6 25:52 26.56% {beam.smp} 28414 whatsapp 64 0 23073M 13302M ucond 5 25:49 26.56% {beam.smp} 28414 whatsapp 67 0 23073M 13302M ucond 6 24:51 26.37% {beam.smp} 28414 whatsapp 64 0 23073M 13302M ucond 3 25:46 26.27% {beam.smp} 28414 whatsapp 63 0 23073M 13302M ucond 6 26:10 26.17% {beam.smp} 28414 whatsapp 64 0 23073M 13302M ucond 8 25:41 25.98% {beam.smp} 28414 whatsapp 64 0 23073M 13302M ucond 11 25:49 25.88% {beam.smp} 28414 whatsapp 64 0 23073M 13302M ucond 2 25:47 25.88% {beam.smp} 28414 whatsapp 62 0 23073M 13302M ucond 1 25:46 25.88% {beam.smp} 28414 whatsapp 62 0 23073M 13302M ucond 9 25:52 25.59% {beam.smp} 28414 whatsapp 63 0 23073M 13302M ucond 10 25:56 25.49% {beam.smp} 28414 whatsapp 44 0 23073M 13302M ucond 4 0:12 0.00% {beam.smp} 28414 whatsapp 44 0 23073M 13302M ucond 9 0:00 0.00% {beam.smp} 28414 whatsapp 44 0 23073M 13302M ucond 8 0:00 0.00% {beam.smp} problematic machine: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 33329 whatsapp 109 0 31021M 17250M CPU11 11 17.3H 62.79% {beam.smp} 33329 whatsapp 76 0 31021M 17250M ucond 1 17.3H 62.70% {beam.smp} 33329 whatsapp 76 0 31021M 17250M kqread 0 17.4H 61.57% {beam.smp} 33329 whatsapp 109 0 31021M 17250M CPU2 2 17.5H 60.06% {beam.smp} 33329 whatsapp 71 0 31021M 17250M ucond 1 557:40 0.00% {beam.smp} 33329 whatsapp 70 0 31021M 17250M ucond 1 517:10 0.00% {beam.smp} 33329 whatsapp 76 0 31021M 17250M ucond 4 435:15 0.00% {beam.smp} 33329 whatsapp 76 0 31021M 17250M ucond 11 259:11 0.00% {beam.smp} 33329 whatsapp 66 0 31021M 17250M ucond 1 178:32 0.00% {beam.smp} 33329 whatsapp 59 0 31021M 17250M ucond 6 137:56 0.00% {beam.smp} 33329 whatsapp 54 0 31021M 17250M ucond 0 71:03 0.00% {beam.smp} 33329 whatsapp 54 0 31021M 17250M ucond 3 65:09 0.00% {beam.smp} sorry about possible text alignment issues in advance, but you can clearly see that in problematic case only four beam.smp threads are doing work and CPU time for each thread time is quite high and 2x the normal machine. from the TIME column you can also see threads did work but at some point just stopped doing work. we don't use any +S or +s options. procstat(1) output was identical on both machines. is there anything we should be looking at in the future when this bug happens again? we appreciate any help or suggestions regarding 1) or 2) thanks, -- jan From ok@REDACTED Thu Nov 25 00:14:21 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 25 Nov 2010 12:14:21 +1300 Subject: [erlang-questions] New random module? In-Reply-To: <20101124080025.GA40466@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> <20101124080025.GA40466@k2r.org> Message-ID: <94222D68-A26C-4FC4-8346-2178BC616381@cs.otago.ac.nz> > * Current random module (Wichmann-Hill 1982, state: 6 bytes: period: ~ 10^13) I should provide a bit of background here. More years ago than I care to remember, I needed a random number generator for Prolog. I wanted something tolerably portable. This meant that - the stored integers had to fit into 18 bits (DEC-10 Prolog) - the intermediate integers had to fit into 28 bits - it had to be possible to generate random integers, random lists, and so on without floating point. At the time I was aware of fancier RNGS, like the lagged Fibonacci ones discussed in Knuth vol. 2. But they didn't meet these criteria. Algorithm AS183 was *new* and fitted these criteria perfectly. When at Quintus I revised library(random) to use C code for the actual numeric calculations. This got the speed up to a blazing 1ms per random number on a Sun-3/50. (The figure is written in the source code!) At that speed, a period of 10^13 takes 10^10 seconds to exhaust, or about 317 years, so the period looked pretty good. Erlang probably got it from Prolog. Now, the SC-40 ran at 12 MIPS and was 8 times as fast as a KL10, so that's 1.5 MIPS for a KL-10, which we thought was fast back in the days of DEC-10 Prolog. The Sun-3/50, pretty much the latest and greatest in the early days of Quintus, was also 1.5MIPs (and the best keyboard I've ever used). The 3/50 didn't have floating point hardware. On a 500 MHz UltraSPARC II, market value today probably 0, the same AS183 code takes 0.3 usec. That's 333 times faster. The period would be exhausted in a year. An Intel Core 2 Duo running at 2.66 GHz does it another 10 times faster still, which would exhaust the period in a little over 36 days. We do have a 16-core machine that I have an account on a quarter of (it's a long story), so that could use up a full period of AS183 in a little over 2 days. So what *used* to be an excellent choice is so no longer. The 2006 Wichmann-Hill algorithm not only has a longer period, but has the relevant-to-Erlang property of letting you set up *multiple* streams quite straightforwardly, as discussed in the paper. It also has a small state, which is nice for keeping Erlang processes that use random numbers small. It really is time that the Erlang library used something other than AS183, and this looks like a good choice. From ok@REDACTED Thu Nov 25 00:31:52 2010 From: ok@REDACTED (Richard O'Keefe) Date: Thu, 25 Nov 2010 12:31:52 +1300 Subject: [erlang-questions] New random module? In-Reply-To: <20101124084130.GA41191@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> <20101124080025.GA40466@k2r.org> <20101124084130.GA41191@k2r.org> Message-ID: <83F504CE-A233-4A8C-8525-7030268A5681@cs.otago.ac.nz> On 24/11/2010, at 9:41 PM, Kenji Rikitake wrote: > In the message <20101124080025.GA40466@REDACTED> > dated Wed, Nov 24, 2010 at 05:00:01PM +0900, > Kenji Rikitake writes: >> * XORshift 7 stage (Panneton-L'Ecuyer, 2005[1], state: 8 bytes, period: 2^256-1) > > [1] Francis Panneton and Pierre L'ecuyer. 2005. On the xorshift random > number generators. ACM Trans. Model. Comput. Simul. 15, 4 (October > 2005), 346-361. DOI=10.1145/1113316.1113319 http://doi.acm.org/10.1145/1113316.1113319 The C code for the 7 stage Xorshift generator displayed in figure 1 of that paper has 32 bytes of state, not 8. The 'doubles' you get from it have only 32 bits worth of randomness; the bottom 21 bits are 0. The paper concludes by saying "These generators are fast, but not reliable, according to our analysis." They don't report the results of tests on the 7 stage generator. From matthew@REDACTED Thu Nov 25 01:16:42 2010 From: matthew@REDACTED (Matthew Sackman) Date: Thu, 25 Nov 2010 00:16:42 +0000 Subject: [erlang-questions] Send message to thousands of clients In-Reply-To: References: Message-ID: <20101125001640.GB24622@wellquite.org> On Wed, Nov 24, 2010 at 11:16:35PM +0300, Max Lapshin wrote: > I remember, that there was such a discussion, but can't find it. > > So, I have a problem: about 40 times per second process ems_media gets > a frame, performs some actions and then retransmit it up to 1500 > clients and more. > > Problem is in bad responsiveness of this process. Clients cannot wait > when it reply for some tasks such as unsubscribe, etc. > > Simple benchmarks shows that > [Pid ! Message || Pid <- List] > > can take upto 13 milliseconds when List length is 2000. > > Question is: what should I do? This is _highly_ experimental, but you could try my gm module which does guaranteed multicast. It's rapidly changing and indeed may still contain some bugs. You need: http://hg.rabbitmq.com/rabbitmq-ha/file/default/src/gm.erl and will want to read http://hg.rabbitmq.com/rabbitmq-ha/file/default/src/gm_test.erl to figure out how it works. (You can more than ignore all the other files in that repo!) Overall latency will be _higher_ (i.e. end-to-end latency), but the "sending" process does vastly less work and thus will return much faster. It may or may not do quite what you need, but it might be worth a look. Matthew From andy.kriger@REDACTED Thu Nov 25 01:25:34 2010 From: andy.kriger@REDACTED (Andy Kriger) Date: Wed, 24 Nov 2010 19:25:34 -0500 Subject: good form? Message-ID: I'm playing around with Webmachine and what I'm finding is that in my resources (the handlers for RESTful dispatch targets), there's duplicated functionality. Fine, no problem there - three approaches I can take: 1) a library module that those Webmachine functions can call out to 2) have my resource modules extend a base resource module that implements common functionality 3) setup a delegate module that those Webmachine functions can call out (one benefit of this over #1 is that a change to which implementation is needed is handled either in the delegate or by changing delegates, not in each resource calling a library module) Combinations of the approaches are possible too. I'm curious to hear from more experienced Erlangers on if extends is an accepted practice, if delegation is a common Erlang pattern (or just a pattern that I'm carrying over from OO/Java), if there's a pattern I'm missing here (I'm suspecting something with messaging is the real solution, but I honestly don't know enough Erlang to know if I'm barking up the wrong tree there). thx From michael.santos@REDACTED Thu Nov 25 01:29:00 2010 From: michael.santos@REDACTED (Michael Santos) Date: Wed, 24 Nov 2010 19:29:00 -0500 Subject: [erlang-questions] FreeBSD 8 + R14B problems In-Reply-To: References: Message-ID: <20101125002900.GA25672@ecn.lan> On Wed, Nov 24, 2010 at 02:59:25PM -0800, Jan Koum wrote: > hi there, > > a few months ago we used to run FreeBSD 7.3 and R13B4. our cluster started > having major problems, so we upgraded to FreeBSD 8 and R14B. ever since > upgrading we have two nagging problems which caused us problems and user > visible downtime: > > 1. every few days we get "corrupted external term" from > erts/emulator/beam/external.c:bad_dist_ext() which in turns calls > erts_kill_dist_connection(). the atom cache string that follow is: > ATOM_CACHE_REF translations: 0='ejabberd@REDACTED', 1='', > 2=xmlelement, 3=xmlcdata, 4=jid, 5=never, 6=offline_msg What was the term that caused the error? Looking at the code, there should be dump of the term in the external binary format. From kenji.rikitake@REDACTED Thu Nov 25 04:21:23 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Thu, 25 Nov 2010 12:21:23 +0900 Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hill random number generator for Erlang/OTP In-Reply-To: References: <20101123111542.GB22113@k2r.org> <20101124051155.GA38476@k2r.org> Message-ID: <20101125032123.GA53650@k2r.org> The fix looks much better. I'll put it in the gist. (BTW gist is a git archive itself so you can fork by yourself) Kenji Rikitake In the message dated Thu, Nov 25, 2010 at 11:34:48AM +1300, Richard O'Keefe writes: > On 24/11/2010, at 6:11 PM, Kenji Rikitake wrote: > > > The 2006 Wichmann-Hill RNG for Erlang moved to > > https://gist.github.com/713144 > > under the name of > > random_wh06.erl > > Didn't we just have a discussion about not allowing zeros? > Shouldn't seed/4 be > > seed(A1, A2, A3, A4) -> > put(random_wh06_seed, > {abs(A1) rem 2147483578 + 1, > abs(A2) rem 2147483542 + 1, > abs(A3) rem 2147483422 + 1, > abs(A4) rem 2147483122 + 1}). > > Or did I misunderstand? From kenji.rikitake@REDACTED Thu Nov 25 04:28:47 2010 From: kenji.rikitake@REDACTED (Kenji Rikitake) Date: Thu, 25 Nov 2010 12:28:47 +0900 Subject: [erlang-questions] New random module? In-Reply-To: <83F504CE-A233-4A8C-8525-7030268A5681@cs.otago.ac.nz> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> <20101124080025.GA40466@k2r.org> <20101124084130.GA41191@k2r.org> <83F504CE-A233-4A8C-8525-7030268A5681@cs.otago.ac.nz> Message-ID: <20101125032847.GB53650@k2r.org> The state for XORshift is indeed 32 bytes or eight 32-bit words. My mistake. The mantissa part of IEEE floats has 53 bits, so two calls of 32-bit integer RNGs will be sufficient (this is not only the XORshift issue but also for other 32-bit integer-based RNGs). So far as I know XORshift 7-stage version RNG passed the Dieharder tests as well as the 2006 Wichmann-Hill RNG (converted to 32-bit unsigned integers) did. (Both didn't pass all the tests for one key, but giving an independent seed made them pass the tests with "WEAK" or "FAILED" results.) In the message <83F504CE-A233-4A8C-8525-7030268A5681@REDACTED> dated Thu, Nov 25, 2010 at 12:31:28PM +1300, Richard O'Keefe writes: > On 24/11/2010, at 9:41 PM, Kenji Rikitake wrote: > > > In the message <20101124080025.GA40466@REDACTED> > > dated Wed, Nov 24, 2010 at 05:00:01PM +0900, > > Kenji Rikitake writes: > >> * XORshift 7 stage (Panneton-L'Ecuyer, 2005[1], state: 8 bytes, period: 2^256-1) > > > > [1] Francis Panneton and Pierre L'ecuyer. 2005. On the xorshift random > > number generators. ACM Trans. Model. Comput. Simul. 15, 4 (October > > 2005), 346-361. DOI=10.1145/1113316.1113319 http://doi.acm.org/10.1145/1113316.1113319 > > The C code for the 7 stage Xorshift generator displayed in figure 1 > of that paper has 32 bytes of state, not 8. The 'doubles' you get > from it have only 32 bits worth of randomness; the bottom 21 bits > are 0. The paper concludes by saying "These generators are fast, > but not reliable, according to our analysis." They don't report the > results of tests on the 7 stage generator. From vinoski@REDACTED Thu Nov 25 05:15:05 2010 From: vinoski@REDACTED (Steve Vinoski) Date: Wed, 24 Nov 2010 23:15:05 -0500 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: 2010/11/24 Dmitry Demeshchuk : > > If you aim for great performance, have a look at mochiweb or misultin. > If you need a lot of features or have a web server from the box, > without having to write any extra code, try yaws. This seems an unusual way to describe the three web servers. If I were a new Erlang user, I'd read this and think Yaws must be super slow! For the record, that's definitely not the case. --steve From demeshchuk@REDACTED Thu Nov 25 05:28:57 2010 From: demeshchuk@REDACTED (Dmitry Demeshchuk) Date: Thu, 25 Nov 2010 07:28:57 +0300 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: Well, yaws needs to be tuned to show that impressive performance. From the box, with all default settings, it's not very fast: http://lionet.livejournal.com/42016.html Sorry that my previous comment was unclear a bit. My point was not that yaws is slow, but that you need to know how to cook it in order to make it really fast. On Thu, Nov 25, 2010 at 7:15 AM, Steve Vinoski wrote: > 2010/11/24 Dmitry Demeshchuk : >> >> If you aim for great performance, have a look at mochiweb or misultin. >> If you need a lot of features or have a web server from the box, >> without having to write any extra code, try yaws. > > This seems an unusual way to describe the three web servers. If I were > a new Erlang user, I'd read this and think Yaws must be super slow! > For the record, that's definitely not the case. > > --steve > -- Best regards, Dmitry Demeshchuk From max.lapshin@REDACTED Thu Nov 25 07:18:02 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Thu, 25 Nov 2010 09:18:02 +0300 Subject: [erlang-questions] Send message to thousands of clients In-Reply-To: <20101125001640.GB24622@wellquite.org> References: <20101125001640.GB24622@wellquite.org> Message-ID: I have tried to use process_flag priority, but it doesn't change time, required to send 2000 messages. I'll take a look at rabbitmq code, thanks. From qoocku@REDACTED Thu Nov 25 07:46:45 2010 From: qoocku@REDACTED (=?UTF-8?B?RGFtaWFuIERvYnJvY3p5xYRza2k=?=) Date: Thu, 25 Nov 2010 07:46:45 +0100 Subject: Vlad Dumitrescu has looked at porting to QNX. If you're interested, ask on the erlang mailing list. Message-ID: <4CEE0655.1060406@gmail.com> Hi list, hi Vlad, Following the Erlang FAQ tip I'm asking a question about Erlang-QNX world. Does anybody have some more fresher news about Erlang @ QNX? -- D. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0x0CAE3AEB.asc Type: application/pgp-keys Size: 4563 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From vladdu55@REDACTED Thu Nov 25 08:36:26 2010 From: vladdu55@REDACTED (Vlad Dumitrescu) Date: Thu, 25 Nov 2010 08:36:26 +0100 Subject: [erlang-questions] Vlad Dumitrescu has looked at porting to QNX. If you're interested, ask on the erlang mailing list. In-Reply-To: <4CEE0655.1060406@gmail.com> References: <4CEE0655.1060406@gmail.com> Message-ID: Hi, 2010/11/25 Damian Dobroczy?ski : > Following the Erlang FAQ tip I'm asking a question about Erlang-QNX > world. Does anybody have some more fresher news about Erlang @ QNX? I don't -- it's been a long time since I was looking into that and there was no immediate business case for me to continue... regards, Vlad From jesper.louis.andersen@REDACTED Thu Nov 25 08:50:03 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Thu, 25 Nov 2010 08:50:03 +0100 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: On Thu, Nov 25, 2010 at 5:28 AM, Dmitry Demeshchuk wrote: > Well, yaws needs to be tuned to show that impressive performance. From > the box, with all default settings, it's not very fast: > http://lionet.livejournal.com/42016.html > > Sorry that my previous comment was unclear a bit. My point was not > that yaws is slow, but that you need to know how to cook it in order > to make it really fast. > I actually use inets httpd webserver. It cannot scale by any means defined by the other webservers. But it is built-in, it is fairly simply to get working and I hardly need it to serve anything else than a couple of ajax-requests for jQuery. It is also worth a look if your demands are rather small. -- J. From torben.lehoff@REDACTED Thu Nov 25 08:52:15 2010 From: torben.lehoff@REDACTED (Torben Hoffmann) Date: Thu, 25 Nov 2010 08:52:15 +0100 Subject: [erlang-questions] New random module? In-Reply-To: <20101125032847.GB53650@k2r.org> References: <20101123111542.GB22113@k2r.org> <20101123133952.1909c4c7@regolo> <20101123125919.GA24633@k2r.org> <20101123155847.77783a5f@regolo> <20101124080025.GA40466@k2r.org> <20101124084130.GA41191@k2r.org> <83F504CE-A233-4A8C-8525-7030268A5681@cs.otago.ac.nz> <20101125032847.GB53650@k2r.org> Message-ID: Hi, Maybe this could be of interest to you... Some time ago I created a fork of the cryptographic application by Martin Logan where I have created a module called hash_random that follows the guidelines for implementing a DRBG based on a hash function as per NIST Special Publication 800-90. There is half-baked code for accessing a good entropy source written in C that should be easy to port to different platforms - this needs more work though. The hash_random should, however, be good to go - just pass in a good entropy function and rock'n'roll! The fork is at https://github.com/lehoff/cryptographic I have been short for time so this has not gotten the attention that it deserves. Background info on the reason for the fork to answer the question "Why Erlang?" up front: I wanted to create an encryption library that works on all major platforms without the tie to OpenSSL since the intention was to create an application that could easily be distributed to non-Erlangers without asking them to install Erlang and OpenSSL. Cheers, Torben On Thu, Nov 25, 2010 at 04:28, Kenji Rikitake wrote: > The state for XORshift is indeed 32 bytes or eight 32-bit words. My > mistake. > > The mantissa part of IEEE floats has 53 bits, so two calls of > 32-bit integer RNGs will be sufficient (this is not only the XORshift > issue but also for other 32-bit integer-based RNGs). > > So far as I know > XORshift 7-stage version RNG passed the Dieharder tests > as well as the 2006 Wichmann-Hill RNG (converted to 32-bit > unsigned integers) did. > (Both didn't pass all the tests for one key, > but giving an independent seed made them pass the tests > with "WEAK" or "FAILED" results.) > > In the message <83F504CE-A233-4A8C-8525-7030268A5681@REDACTED> > dated Thu, Nov 25, 2010 at 12:31:28PM +1300, > Richard O'Keefe writes: > > On 24/11/2010, at 9:41 PM, Kenji Rikitake wrote: > > > > > In the message <20101124080025.GA40466@REDACTED> > > > dated Wed, Nov 24, 2010 at 05:00:01PM +0900, > > > Kenji Rikitake writes: > > >> * XORshift 7 stage (Panneton-L'Ecuyer, 2005[1], state: 8 bytes, > period: 2^256-1) > > > > > > [1] Francis Panneton and Pierre L'ecuyer. 2005. On the xorshift random > > > number generators. ACM Trans. Model. Comput. Simul. 15, 4 (October > > > 2005), 346-361. DOI=10.1145/1113316.1113319 > http://doi.acm.org/10.1145/1113316.1113319 > > > > The C code for the 7 stage Xorshift generator displayed in figure 1 > > of that paper has 32 bytes of state, not 8. The 'doubles' you get > > from it have only 32 bits worth of randomness; the bottom 21 bits > > are 0. The paper concludes by saying "These generators are fast, > > but not reliable, according to our analysis." They don't report the > > results of tests on the 7 stage generator. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- http://www.linkedin.com/in/torbenhoffmann From timo.lindemann@REDACTED Thu Nov 25 09:27:11 2010 From: timo.lindemann@REDACTED (Timo Lindemann) Date: Thu, 25 Nov 2010 09:27:11 +0100 Subject: [erlang-questions] Lua Driver In-Reply-To: <13448848.82521290583867809.JavaMail.www@wsfrf1116> References: <13448848.82521290583867809.JavaMail.www@wsfrf1116> Message-ID: Hi, lua has qt bindings, for instance, which erlang doesn't offer. So I'd say yeah, having lua is definitely worth it. -T. 2010/11/24 > Nice, but what have Lua to offer to Erlang ? Scripting ? Concurrency ? > Fault tolerancy ? Lua is pretty, fast and small (less than 5 MiB) ths you > wouldn't install 100 MiB of Erlang to control Lua... > > David KOCH > > ======================================== > > Message du : 24/11/2010 > De : "Henning Diedrich " > A : "Erlang Questions" > Copie ? : > Sujet : [erlang-questions] Lua Driver > > > Hi folks, > > I posted some details on Lua embedding here: > http://www.eonblast.com/blog/optimizing-erlualib-calls/ > > It's about an Erlang embedded Lua driver, discussing in detail how to go > beyond single step stack manipulation port calls towards directly > accessing the Lua C API in one go. > > Feedback and critique very welcome. > > Regards, > > Henning > > -- Timo Lindemann Software Systems Engineer +49 (0) 160 99 13 06 23 Mon-Thu 0900 - 1700 (GMT +1) SOL VENETUS Software GmbH Seb.-Kneipp-Str. 2, 86482 Aystetten Telefon: +49 (0) 170 77 62 48 0 Telefax: +49 (0) 821 48 65 78 6 Gesch?ftsf?hrer: Benjamin Hilbig Amtsgericht Augsburg, HRB 24559 USt.-IdNr.: DE267014985 From glacjay@REDACTED Thu Nov 25 09:39:06 2010 From: glacjay@REDACTED (Jay True) Date: Thu, 25 Nov 2010 16:39:06 +0800 Subject: How can I use PKCS8 private key file with ssl application in R14B? Message-ID: It seems that it only support RSA and DSA private key file. From sigmastar@REDACTED Thu Nov 25 09:54:02 2010 From: sigmastar@REDACTED (Jesse Gumm) Date: Thu, 25 Nov 2010 02:54:02 -0600 Subject: [erlang-questions] Lua Driver In-Reply-To: References: <13448848.82521290583867809.JavaMail.www@wsfrf1116> Message-ID: Given that Lua is so commonly used as scripting for other projects, it makes sense to incorporate lua support - it can serve as a slightly more convenient bridge from Erlang to other systems, or as Timo suggested, to take advantage of it's bindings where Erlang lacks them. Improved interoperability is always good. -Jesse On Thu, Nov 25, 2010 at 2:27 AM, Timo Lindemann wrote: > Hi, > > lua has qt bindings, for instance, which erlang doesn't offer. So I'd say > yeah, having lua is definitely worth it. > > -T. > > 2010/11/24 > >> Nice, but what have Lua to offer to Erlang ? Scripting ? Concurrency ? >> Fault tolerancy ? Lua is pretty, fast and small (less than 5 MiB) ths you >> wouldn't install 100 MiB of Erlang to control Lua... >> >> David KOCH >> >> ======================================== >> >> Message du : 24/11/2010 >> De : "Henning Diedrich " >> A : "Erlang Questions" >> Copie ? : >> Sujet : [erlang-questions] Lua Driver >> >> >> ?Hi folks, >> >> I posted some details on Lua embedding here: >> http://www.eonblast.com/blog/optimizing-erlualib-calls/ >> >> It's about an Erlang embedded Lua driver, discussing in detail how to go >> beyond single step stack manipulation port calls towards directly >> accessing the Lua C API in one go. >> >> Feedback and critique very welcome. >> >> Regards, >> >> Henning >> >> > > > -- > Timo Lindemann > Software Systems Engineer > +49 (0) 160 99 13 06 23 > Mon-Thu 0900 - 1700 (GMT +1) > > SOL VENETUS Software GmbH > Seb.-Kneipp-Str. 2, 86482 Aystetten > Telefon: +49 (0) 170 77 62 48 0 > Telefax: +49 (0) 821 48 65 78 6 > Gesch?ftsf?hrer: Benjamin Hilbig > Amtsgericht Augsburg, HRB 24559 > USt.-IdNr.: DE267014985 > -- Jesse Gumm Sigma Star Systems 414.940.4866 gumm@REDACTED http://www.sigma-star.com From dkoch@REDACTED Thu Nov 25 12:24:52 2010 From: dkoch@REDACTED (dkoch@REDACTED) Date: Thu, 25 Nov 2010 12:24:52 +0100 (CET) Subject: [erlang-questions] Lua Driver In-Reply-To: References: <13448848.82521290583867809.JavaMail.www@wsfrf1116> Message-ID: <19678273.25121290684292415.JavaMail.www@wsfrf1110> And true, it's not like Lua weigth that much either, especially face to Erlang and its libraries :p David KOCH ======================================== Message du : 25/11/2010 De : "Jesse Gumm " A : "Erlang Questions" Copie ? : Sujet : Re: [erlang-questions] Lua Driver Given that Lua is so commonly used as scripting for other projects, it makes sense to incorporate lua support - it can serve as a slightly more convenient bridge from Erlang to other systems, or as Timo suggested, to take advantage of it's bindings where Erlang lacks them. Improved interoperability is always good. -Jesse On Thu, Nov 25, 2010 at 2:27 AM, Timo Lindemann wrote: > Hi, > > lua has qt bindings, for instance, which erlang doesn't offer. So I'd say > yeah, having lua is definitely worth it. > > -T. > > 2010/11/24 > >> Nice, but what have Lua to offer to Erlang ? Scripting ? Concurrency ? >> Fault tolerancy ? Lua is pretty, fast and small (less than 5 MiB) ths you >> wouldn't install 100 MiB of Erlang to control Lua... >> >> David KOCH >> >> ======================================== >> >> Message du : 24/11/2010 >> De : "Henning Diedrich " >> A : "Erlang Questions" >> Copie ? : >> Sujet : [erlang-questions] Lua Driver >> >> >> Hi folks, >> >> I posted some details on Lua embedding here: >> http://www.eonblast.com/blog/optimizing-erlualib-calls/ >> >> It's about an Erlang embedded Lua driver, discussing in detail how to go >> beyond single step stack manipulation port calls towards directly >> accessing the Lua C API in one go. >> >> Feedback and critique very welcome. >> >> Regards, >> >> Henning >> >> > > > -- > Timo Lindemann > Software Systems Engineer > +49 (0) 160 99 13 06 23 > Mon-Thu 0900 - 1700 (GMT +1) > > SOL VENETUS Software GmbH > Seb.-Kneipp-Str. 2, 86482 Aystetten > Telefon: +49 (0) 170 77 62 48 0 > Telefax: +49 (0) 821 48 65 78 6 > Gesch?ftsf?hrer: Benjamin Hilbig > Amtsgericht Augsburg, HRB 24559 > USt.-IdNr.: DE267014985 > -- Jesse Gumm Sigma Star Systems 414.940.4866 gumm@REDACTED http://www.sigma-star.com ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED From dkoch@REDACTED Thu Nov 25 12:23:46 2010 From: dkoch@REDACTED (dkoch@REDACTED) Date: Thu, 25 Nov 2010 12:23:46 +0100 (CET) Subject: [erlang-questions] Lua Driver In-Reply-To: <13448848.82521290583867809.JavaMail.www@wsfrf1116> References: <13448848.82521290583867809.JavaMail.www@wsfrf1116> Message-ID: <474952.24371290684226341.JavaMail.www@wsfrf1110> That answers my questions, thanks :) However, a native Qt-binding driver for Erlang wouldn't be smarter ? David KOCH ======================================== Message du : 25/11/2010 De : "Timo Lindemann " A : dkoch@REDACTED Copie ? : erlang-questions@REDACTED Sujet : Re: [erlang-questions] Lua Driver Hi, lua has qt bindings, for instance, which erlang doesn't offer. So I'd say yeah, having lua is definitely worth it. -T. 2010/11/24 > Nice, but what have Lua to offer to Erlang ? Scripting ? Concurrency ? > Fault tolerancy ? Lua is pretty, fast and small (less than 5 MiB) ths you > wouldn't install 100 MiB of Erlang to control Lua... > > David KOCH From hynek@REDACTED Thu Nov 25 13:45:43 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Thu, 25 Nov 2010 13:45:43 +0100 Subject: [erlang-questions] application startup failure? In-Reply-To: <37105.1290485466@snookles.snookles.com> References: <37105.1290485466@snookles.snookles.com> Message-ID: Why not just do pattern match to return value from mnesia call? I think it is The Idiomatic Erlang Way. Let's it crash. start(Type, StartArgs) -> {atomic, ok} = {error, ohmygod}, %% return value from any call {ok, []}. 1> application:start(foo). =CRASH REPORT==== 25-Nov-2010::13:41:51 === crasher: initial call: application_master:init/4 pid: <0.58.0> registered_name: [] exception exit: {bad_return, {{foo,start,[normal,[]]}, {'EXIT', {{badmatch,{error,ohmygod}}, [{foo,start,2}, {application_master,start_it_old,4}]}}}} in function application_master:init/4 ancestors: [<0.57.0>] messages: [{'EXIT',<0.59.0>,normal}] links: [<0.57.0>,<0.6.0>] dictionary: [] trap_exit: true status: running heap_size: 377 stack_size: 24 reductions: 105 neighbours: =INFO REPORT==== 25-Nov-2010::13:41:51 === application: foo exited: {bad_return,{{foo,start,[normal,[]]}, {'EXIT',{{badmatch,{error,ohmygod}}, [{foo,start,2}, {application_master,start_it_old,4}]}}}} type: temporary {error,{bad_return,{{foo,start,[normal,[]]}, {'EXIT',{{badmatch,{error,ohmygod}}, [{foo,start,2},{application_master,start_it_old,4}]}}}}} On Tue, Nov 23, 2010 at 5:11 AM, Scott Lystig Fritchie wrote: > Andy Kriger wrote: > > ak> Or maybe there's a better way to handle this situation? > > Andy, would it be awful to simply call error_logger:error_msg() or > similar in your app's init() function before it returns {error, Reason}? > > If you have the sasl application already running, then you should also > get both a crasher report and an info report that contain the returned > reason. ?Code follows the messages. ?Comparing info reports, it looks > like your app's Reason term really is 'shutdown'? ?{shrug} > > -Scott > > 1> application:start(sasl). > ok > 2> application:start(foo). > {error,{yofoo,{foo,start,[normal,[]]}}} > > =ERROR REPORT==== 22-Nov-2010::22:05:20 === > foo:start type normal args [] > > =CRASH REPORT==== 22-Nov-2010::22:05:20 === > ?crasher: > ? ?initial call: application_master:init/4 > ? ?pid: <0.47.0> > ? ?registered_name: [] > ? ?exception exit: {yofoo,{foo,start,[normal,[]]}} > ? ? ?in function ?application_master:init/4 > ? ?ancestors: [<0.46.0>] > ? ?messages: [{'EXIT',<0.48.0>,normal}] > ? ?links: [<0.46.0>,<0.6.0>] > ? ?dictionary: [] > ? ?trap_exit: true > ? ?status: running > ? ?heap_size: 233 > ? ?stack_size: 24 > ? ?reductions: 102 > ?neighbours: > > =INFO REPORT==== 22-Nov-2010::22:05:21 === > ? ?application: foo > ? ?exited: {yofoo,{foo,start,[normal,[]]}} > ? ?type: temporary > > --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- > > %% foo.erl > > -module(foo). > -author('fritchie@REDACTED'). > > -behaviour(application). > > -export([start/2, stop/1]). > > start(Type, StartArgs) -> > ? ?error_logger:error_msg("~s:start type ~p args ~p\n", [?MODULE, Type, > ? ?StartArgs]), > ? ?{error, yofoo}. > > stop(State) -> > ? ?ok. > > --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- > > %% foo.app > > {application, foo, > ?[{description, "foo app"}, > ?{vsn, "99"}, > ?{modules, [ foo > ? ? ? ? ? ? ?]}, > ?{registered, []}, > ?{applications, [kernel, > ? ? ? ? ? ? ? ? ?stdlib, > ? ? ? ? ? ? ? ? ?sasl]}, > ?{mod, { foo, []}}, > ?{env, []} > ]}. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From zabrane3@REDACTED Thu Nov 25 13:49:59 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Thu, 25 Nov 2010 13:49:59 +0100 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: For Erlang Web development: - MochiWeb (very stable, for production): http://code.google.com/p/mochiweb/ - BeepBeep (based on MochiWeb): https://github.com/davebryson/beepbeep - Nitrogen: http://nitrogenproject.com/ - ElangWeb: http://www.erlang-web.org/ - Misultin (speedy): http://code.google.com/p/misultin/ - Chicago Boss (a newcomer): http://www.chicagoboss.org/ -- Regards Zabrane 2010/11/25 Jesper Louis Andersen : > On Thu, Nov 25, 2010 at 5:28 AM, Dmitry Demeshchuk wrote: >> Well, yaws needs to be tuned to show that impressive performance. From >> the box, with all default settings, it's not very fast: >> http://lionet.livejournal.com/42016.html >> >> Sorry that my previous comment was unclear a bit. My point was not >> that yaws is slow, but that you need to know how to cook it in order >> to make it really fast. >> > > I actually use inets httpd webserver. It cannot scale by any means > defined by the other webservers. But it is built-in, it is fairly > simply to get working and I hardly need it to serve anything else than > a couple of ajax-requests for jQuery. It is also worth a look if your > demands are rather small. > > -- > J. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From garazdawi@REDACTED Thu Nov 25 14:49:55 2010 From: garazdawi@REDACTED (Lukas Larsson) Date: Thu, 25 Nov 2010 14:49:55 +0100 Subject: [erlang-questions] good form? In-Reply-To: References: Message-ID: Hi, I've had precisely the same problem with webmachine and went with solution number two. It does however present some problems as webmachine (last I checked) uses the exports of the module to determine which responses to give and you could end up replicating some of the functionality of webmachine with in the base module. Besides that it worked very well and I've used the same aproach in a lot of places with great success. The library module aproach is probably easier to understand and maintain by someone else, but in my opinion a base module is the better solution. Lukas On Thu, Nov 25, 2010 at 1:25 AM, Andy Kriger wrote: > I'm playing around with Webmachine and what I'm finding is that in my > resources (the handlers for RESTful dispatch targets), there's > duplicated functionality. Fine, no problem there - three approaches I > can take: > 1) a library module that those Webmachine functions can call out to > 2) have my resource modules extend a base resource module that > implements common functionality > 3) setup a delegate module that those Webmachine functions can call > out (one benefit of this over #1 is that a change to which > implementation is needed is handled either in the delegate or by > changing delegates, not in each resource calling a library module) > > Combinations of the approaches are possible too. I'm curious to hear > from more experienced Erlangers on if extends is an accepted practice, > if delegation is a common Erlang pattern (or just a pattern that I'm > carrying over from OO/Java), if there's a pattern I'm missing here > (I'm suspecting something with messaging is the real solution, but I > honestly don't know enough Erlang to know if I'm barking up the wrong > tree there). > > thx > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From rtrlists@REDACTED Thu Nov 25 14:56:42 2010 From: rtrlists@REDACTED (Robert Raschke) Date: Thu, 25 Nov 2010 13:56:42 +0000 Subject: [erlang-questions] Lua Driver In-Reply-To: <474952.24371290684226341.JavaMail.www@wsfrf1110> References: <13448848.82521290583867809.JavaMail.www@wsfrf1116> <474952.24371290684226341.JavaMail.www@wsfrf1110> Message-ID: I use Erlang to drive Lua business logic chunks (which tend to implement a state machine) from AMPQ messages (i.e., RabbitMQ). Infinitely more fun than Tibco or Biztalk :-)) Ended up writing my own Lua Erlang node code, which unfortunately is tied to a rather bespoke Lua env. But maybe one day I can get a spare week and convert that to wrap around normal Lua. Robby From ingela.andin@REDACTED Thu Nov 25 15:44:32 2010 From: ingela.andin@REDACTED (Ingela Andin) Date: Thu, 25 Nov 2010 15:44:32 +0100 Subject: [erlang-questions] How can I use PKCS8 private key file with ssl application in R14B? In-Reply-To: References: Message-ID: Hi! At the moment RSA and DSA are the supported options. To be able to support PKCS8 we need to implement support for PKS8 in the public_key application. You are welcome to contribute if this important to you. Regards Ingela Erlang/OTP team - Ericsson AB 2010/11/25 Jay True : > It seems that it only support RSA and DSA private key file. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From ngreco@REDACTED Thu Nov 25 15:52:39 2010 From: ngreco@REDACTED (Nahuel Greco) Date: Thu, 25 Nov 2010 11:52:39 -0300 Subject: [erlang-questions] Lua Driver In-Reply-To: <474952.24371290684226341.JavaMail.www@wsfrf1110> References: <13448848.82521290583867809.JavaMail.www@wsfrf1116> <474952.24371290684226341.JavaMail.www@wsfrf1110> Message-ID: On Thu, Nov 25, 2010 at 8:23 AM, wrote: > That answers my questions, thanks :) However, a native Qt-binding driver for Erlang wouldn't be smarter ? > A possibility is to make a thin bridge to Qt using this class: http://doc.qt.nokia.com/4.6/qscriptengine.html And then code the entire GUI using QtScript (a Javascript variant) without needing to bind anything more. Saludos, Nahuel Greco. From ingela@REDACTED Thu Nov 25 18:02:42 2010 From: ingela@REDACTED (Ingela Andin) Date: Thu, 25 Nov 2010 18:02:42 +0100 Subject: [erlang-questions] The most better Erlang web server! In-Reply-To: References: <000601cb8bf6$aa7d1190$ff7734b0$@co.cu> Message-ID: Hi! The Inets web-server was originally designed to be a lightweight web-server for embeded systems that is easy to customize. So it does not come with a lot of bells and whistles! It may be enough for Gilberios needs, it may be that with a little customization it will be just what he wants or depending on what he wants it maybe that one of the other servers suggested is just what he wants. So I think Andys advice is good, playaround with them and see which one fits your needs the best. Regards Ingela Erlang/OTP team - Ericsson AB 2010/11/25 Jesper Louis Andersen : > On Thu, Nov 25, 2010 at 5:28 AM, Dmitry Demeshchuk wrote: >> Well, yaws needs to be tuned to show that impressive performance. From >> the box, with all default settings, it's not very fast: >> http://lionet.livejournal.com/42016.html >> >> Sorry that my previous comment was unclear a bit. My point was not >> that yaws is slow, but that you need to know how to cook it in order >> to make it really fast. >> > > I actually use inets httpd webserver. It cannot scale by any means > defined by the other webservers. But it is built-in, it is fairly > simply to get working and I hardly need it to serve anything else than > a couple of ajax-requests for jQuery. It is also worth a look if your > demands are rather small. > > -- > J. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From co7eb@REDACTED Thu Nov 25 19:15:52 2010 From: co7eb@REDACTED (=?iso-8859-1?Q?Gilberio_Carmenates_Garc=EDa?=) Date: Thu, 25 Nov 2010 15:15:52 -0300 Subject: How to autoload a module Message-ID: <000001cb8ccc$cd086e90$67194bb0$@co.cu> Hi all! I have a little problem , for example if I want to load a module made by me using the shell e.g. erl -sname test What can I put to complete that line before making me able to do something like this Once in the Erlang shell get_args(). Instead of myapp:get_args(). NOTE: I don?t wanna use boot files. Best, Ivan. ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From fernando.benavides@REDACTED Thu Nov 25 21:44:00 2010 From: fernando.benavides@REDACTED (Fernando Benavides) Date: Thu, 25 Nov 2010 17:44:00 -0300 Subject: [erlang-questions] How to autoload a module In-Reply-To: <000001cb8ccc$cd086e90$67194bb0$@co.cu> References: <000001cb8ccc$cd086e90$67194bb0$@co.cu> Message-ID: <1290717840.1901.14.camel@army-desktop> I think you'll find the user_default module useful :) On Thu, 2010-11-25 at 15:15 -0300, Gilberio Carmenates Garc?a wrote: > Hi all! > > > > I have a little problem , for example if I want to load a module made by me > using the shell > > e.g. > > > > erl -sname test > > > > What can I put to complete that line before making me able to do something > like this > > > > Once in the Erlang shell > > > > get_args(). > > > > Instead of > > > > myapp:get_args(). > > > > NOTE: I don?t wanna use boot files. > > > > Best, > > > > Ivan. > > > > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. ________________________________________________________________________ Fernando Benavides fernando@REDACTED From erlang@REDACTED Thu Nov 25 21:44:31 2010 From: erlang@REDACTED (Joe Armstrong) Date: Thu, 25 Nov 2010 21:44:31 +0100 Subject: [erlang-questions] How to autoload a module In-Reply-To: <000001cb8ccc$cd086e90$67194bb0$@co.cu> References: <000001cb8ccc$cd086e90$67194bb0$@co.cu> Message-ID: There is a special module called user_default which was designed to extend the shell. Any functions in user_default can be called from the shell without a module prefix. -module(user_default). -export([get_args/0]). get_args() -> io:format("get_args was called~n"). Compile this and make sure it is your path. > erlc user_default.erl joe$ erl Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.8 (abort with ^G) 1> get_args(). get_args was called ok /Joe 2010/11/25 Gilberio Carmenates Garc?a : > Hi all! > > > > I have a little problem , for example if I want to load a module made by me > using the shell > > e.g. > > > > erl ?-sname test > > > > What can I put to complete that line before making me able to do something > like this > > > > Once in the Erlang shell > > > > get_args(). > > > > Instead of > > > > myapp:get_args(). > > > > NOTE: I don?t wanna use boot files. > > > > Best, > > > > Ivan. > > > > > > ======================================================================= > Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de ?usar el servicio a tales fines y cumplir con las regulaciones establecidas. > From cbenac@REDACTED Thu Nov 25 23:25:05 2010 From: cbenac@REDACTED (Clara Benac Earle) Date: Thu, 25 Nov 2010 23:25:05 +0100 Subject: Convincing industry to use Erlang Message-ID: <4CEEE241.2010400@fi.upm.es> Hi all, I am currently involved in a proposal for a project with some companies in Spain. They want to develop an intelligent system for controlling street lighting. Has anybody experience in this or in a similar kind of system? We would like to convince them to use Erlang. Any successful industrial stories with data (good performance, rapid development, etc.) compared to traditional programming languages that we could use? Thanks in advance Clara From taavi@REDACTED Fri Nov 26 01:18:36 2010 From: taavi@REDACTED (Taavi Talvik) Date: Fri, 26 Nov 2010 02:18:36 +0200 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: <4CEEE241.2010400@fi.upm.es> References: <4CEEE241.2010400@fi.upm.es> Message-ID: <1571C1D3-1296-4E08-8F63-76C7ACC7AE3D@uninet.ee> On Nov 26, 2010, at 12:25 AM, Clara Benac Earle wrote: > Hi all, > > I am currently involved in a proposal for a project with some companies in Spain. They want to develop an intelligent system for controlling street lighting. > > Has anybody experience in this or in a similar kind of system? We would like to convince them to use Erlang. Any successful industrial stories with data (good performance, rapid development, etc.) compared to traditional programming languages that we could use? One success story from this list. Lyons metro. http://www.trapexit.org/forum/viewtopic.php?t=6229&sid=328def1e11f2c484a23a5c3f01e19c0a Hopefully author is still reachable. best regards, taavi From co7eb@REDACTED Fri Nov 26 02:30:25 2010 From: co7eb@REDACTED (=?iso-8859-1?Q?Gilberio_Carmenates_Garc=EDa?=) Date: Thu, 25 Nov 2010 22:30:25 -0300 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: <4CEEE241.2010400@fi.upm.es> References: <4CEEE241.2010400@fi.upm.es> Message-ID: <000001cb8d09$83ba5540$8b2effc0$@co.cu> Hi Clara, I know that this is not enough what I about to tell you but I hope you get some help and orientation about it. Having in consideration a test made maybe a year or two or maybe more ago, a test of Erlang yaws web server against apache for php. Where 4 000 concurrent clients made the apache web server goes down, And Erlang yaws web server with more than 80 000 of concurrent connections kept online and about to decrease de performance with that number, that because the testers stopped it. So, using ports in Erlang to handle the hardware that handles the lights and the advantages for concurrency handle and the ability of Erlang to communicate with others languages, they can do the intelligence part perfectly in other language like Mozar/Oz and being able to program any server application using Erlang without having nothing to do with the client applications, a total uncouple policy. The database Mnesia is perfectly right for that kind of applications since that seems to a telephony applications in concurrence and fast and practical data store using transactions like semaphores. Also the name Erlang!, makes the honors to the Erlang algorithm to calculate the traffic bottle locks. That all and the destiny makes Erlang one of the best choices for that project. -----Mensaje original----- De: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] En nombre de Clara Benac Earle Enviado el: Jueves, 25 de Noviembre de 2010 07:25 p.m. Para: erlang-questions@REDACTED Asunto: [erlang-questions] Convincing industry to use Erlang Hi all, I am currently involved in a proposal for a project with some companies in Spain. They want to develop an intelligent system for controlling street lighting. Has anybody experience in this or in a similar kind of system? We would like to convince them to use Erlang. Any successful industrial stories with data (good performance, rapid development, etc.) compared to traditional programming languages that we could use? Thanks in advance Clara ________________________________________________________________ erlang-questions (at) erlang.org mailing list. See http://www.erlang.org/faq.html To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED ======================================================================= Este mensaje ha sido enviado mediante el servicio de correo electr?nico que ofrece la Federaci?n de Radioaficionados de Cuba a sus miembros para respaldar el cumplimiento de los objetivos de la organizaci?n y su pol?tica informativa. La persona que env?a este correo asume el compromiso de usar el servicio a tales fines y cumplir con las regulaciones establecidas. From alain.odea@REDACTED Fri Nov 26 13:29:39 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Fri, 26 Nov 2010 08:59:39 -0330 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: <000001cb8d09$83ba5540$8b2effc0$@co.cu> References: <4CEEE241.2010400@fi.upm.es> <000001cb8d09$83ba5540$8b2effc0$@co.cu> Message-ID: On Thursday, November 25, 2010, Gilberio Carmenates Garc?a wrote: > Hi Clara, > > I know that this is not enough what I about to tell you but I hope you get > some help and orientation about it. > Having in consideration a test made maybe a year or two or maybe more ago, a > test of Erlang yaws web server against apache for php. > Where 4 000 concurrent clients made the apache web server goes down, And > Erlang yaws web server with more than 80 000 of concurrent > connections kept online and about to decrease de performance with that > number, that because the testers stopped it. > > So, using ports in Erlang to handle the hardware that handles the lights and > the advantages for concurrency handle and the ability of > Erlang to communicate with others languages, they can do the intelligence > part perfectly in other language like Mozar/Oz and being able to program any > server application using Erlang without having nothing to do with the client > applications, a total uncouple policy. The database Mnesia is perfectly > right for that kind of applications since that seems to a telephony > applications in concurrence and fast and practical data store using > transactions like semaphores. Also the name Erlang!, makes the honors to the > Erlang algorithm to calculate the traffic bottle locks. That all and the > destiny makes Erlang one of the best choices for that project. > > > > > -----Mensaje original----- > De: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] En > nombre de Clara Benac Earle > Enviado el: Jueves, 25 de Noviembre de 2010 07:25 p.m. > Para: erlang-questions@REDACTED > Asunto: [erlang-questions] Convincing industry to use Erlang > > Hi all, > > I am currently involved in a proposal for a project with some companies > in Spain. They want to develop an intelligent system for controlling > street lighting. > > Has anybody experience in this or in a similar kind of system? We would > like to convince them to use Erlang. Any successful industrial stories > with data (good performance, rapid development, etc.) compared to > traditional programming languages that we could use? > > Thanks in advance > Clara > Some case study material: http://www.erlang-factory.com/conference/SFBayAreaErlangFactory2009/tracks/ErlangCaseStudies http://www.erlang.se/euc/06/proceedings/1600Nystrom.ppt http://www.rabbitmq.com/faq.html#what-is-otp From info@REDACTED Fri Nov 26 16:31:32 2010 From: info@REDACTED (info) Date: Fri, 26 Nov 2010 16:31:32 +0100 Subject: How to find an object in an ETS table ? Message-ID: <201011261631321570072@its3.ch> Hello, Probably an error of beginner ... E=ets:new('my_table',[]), ets:insert(E,{123,1}), ets:insert(E,{456,2}), The following command find the key 123 and return the object 1 ets:lookup(E,123), The following command returns empty ! R1=ets:match(E,{'_',2,'$1'}), How to find the object 2 ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From hynek@REDACTED Fri Nov 26 16:47:31 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Fri, 26 Nov 2010 16:47:31 +0100 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: <4CEEE241.2010400@fi.upm.es> References: <4CEEE241.2010400@fi.upm.es> Message-ID: I think Thomas Arts's great talk about Testing automotive software with Erlang at EUC 2010 in Stockholm http://www.erlang-factory.com/conference/ErlangUserConference2010/speakers/ThomasArts is very close to your domain. On Thu, Nov 25, 2010 at 11:25 PM, Clara Benac Earle wrote: > Hi all, > > I am currently involved in a proposal for a project with some companies in > Spain. They want to develop an intelligent system for controlling street > lighting. > > Has anybody experience in this or in a similar kind of system? We would like > to convince them to use Erlang. Any successful industrial stories with data > (good performance, rapid development, etc.) compared to traditional > programming languages that we could use? > > Thanks in advance > Clara > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From hynek@REDACTED Fri Nov 26 17:07:06 2010 From: hynek@REDACTED (Hynek Vychodil) Date: Fri, 26 Nov 2010 17:07:06 +0100 Subject: [erlang-questions] How to find an object in an ETS table ? In-Reply-To: <201011261631321570072@its3.ch> References: <201011261631321570072@its3.ch> Message-ID: On Fri, Nov 26, 2010 at 4:31 PM, info wrote: > Hello, > > Probably an error of beginner ... > E=ets:new('my_table',[]), > ets:insert(E,{123,1}), > ets:insert(E,{456,2}), > The following command find the key 123 and return the object 1 > ets:lookup(E,123), > The following command returns empty ! > R1=ets:match(E,{'_',2,'$1'}), > How to find the object 2 ? > > J-Ph. Constantin > ITS3 Gen?ve > www.its3.ch > You may be want to do ets:match(E,{'$1',2}). or ets:match_object(E,{'$1',2}). or may be ets:select(E, [{{'_',2}, [], ['$_']}]). -- --Hynek (Pichi) Vychodil Analyze your data in minutes. Share your insights instantly. Thrill your boss.? Be a data hero! Try GoodData now for free: www.gooddata.com From info@REDACTED Fri Nov 26 17:34:46 2010 From: info@REDACTED (info) Date: Fri, 26 Nov 2010 17:34:46 +0100 Subject: [erlang-questions] How to find an object in an ETS table ? References: <201011261631321570072@its3.ch> Message-ID: <201011261734459630111@its3.ch> Yes ! My syntax was bad. Suppose that the key is unique (should be defined) but not the value: ets:insert(E,{123,2}), ets:insert(E,{456,2}), ets:match(E,{'$1',2}), gives me a strange result: ["{",[456]] I was expected a list of keys or a list of {key,value} J-Ph. Constantin ITS3 Gen?ve www.its3.ch From rtrlists@REDACTED Fri Nov 26 17:47:55 2010 From: rtrlists@REDACTED (Robert Raschke) Date: Fri, 26 Nov 2010 16:47:55 +0000 Subject: [erlang-questions] How to find an object in an ETS table ? In-Reply-To: <201011261734459630111@its3.ch> References: <201011261631321570072@its3.ch> <201011261734459630111@its3.ch> Message-ID: On Fri, Nov 26, 2010 at 4:34 PM, info wrote: > Yes ! > My syntax was bad. > Suppose that the key is unique (should be defined) but not the value: > > ets:insert(E,{123,2}), > ets:insert(E,{456,2}), > > ets:match(E,{'$1',2}), > gives me a strange result: > ["{",[456]] > I was expected a list of keys or a list of {key,value} > > > That's an artifact of the shell trying to be clever when printing lists that look like strings: 58> [[123],[456]]. ["{",[456]] Robby From ariel.gomez@REDACTED Fri Nov 26 18:33:02 2010 From: ariel.gomez@REDACTED (=?iso-8859-1?Q?Ariel_G=F3mez?=) Date: Fri, 26 Nov 2010 14:33:02 -0300 Subject: [erlang-questions] How to find an object in an ETS table ? In-Reply-To: <201011261631321570072@its3.ch> References: <201011261631321570072@its3.ch> Message-ID: <010201cb8d8f$fc4ebf10$f4ec3d30$@gomez@inswitch.us> This is the right command: 18> ets:match_object(E,{'_',2}). [{456,2}] Ariel G?mez -----Mensaje original----- De: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] En nombre de info Enviado el: viernes, 26 de noviembre de 2010 12:32 Para: erlang-questions Asunto: [erlang-questions] How to find an object in an ETS table ? Hello, Probably an error of beginner ... E=ets:new('my_table',[]), ets:insert(E,{123,1}), ets:insert(E,{456,2}), The following command find the key 123 and return the object 1 ets:lookup(E,123), The following command returns empty ! R1=ets:match(E,{'_',2,'$1'}), How to find the object 2 ? J-Ph. Constantin ITS3 Genhve www.its3.ch From paf@REDACTED Fri Nov 26 20:59:59 2010 From: paf@REDACTED (Paulo Alexandre Ferreira) Date: Fri, 26 Nov 2010 19:59:59 +0000 Subject: [erlang-questions] Convincing industry to use Erlang Message-ID: <20101126195838.M17562@dei.isep.ipp.pt> You better not try to sell Erlang, or any language to managers. You should try selling a reliable, robust, fault-tolerant, code verified system. You will do code upgrades without stopping the system. You can offer proofs the system is correct. They don't want Erlang. They don't care about Erlang. You will use Erlang to build the system they want. The advantages of language X,Y or Z are the stuff programmers care about. Managers care about different things like costs, down-time, reliability. Focus on that. My apologies if I sound a little bit grumpy. Paulo Ferreira From info@REDACTED Fri Nov 26 23:25:38 2010 From: info@REDACTED (=?utf-8?B?aW5mbw==?=) Date: Fri, 26 Nov 2010 23:25:38 +0100 Subject: =?utf-8?B?UmU6IFJlOiBSZTogW2VybGFuZy1xdWVzdGlvbnNdIEhvdyB0byBmaW5kIGFuIG9iamVjdCBpbiBhbiBFVFMgdGFibGUgPw==?= References: <201011261631321570072@its3.ch>, <201011261734459630111@its3.ch>, Message-ID: <201011261932540128406@its3.ch> How to evict this artifact ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From erlang@REDACTED Sat Nov 27 10:51:55 2010 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 27 Nov 2010 10:51:55 +0100 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: <20101126195838.M17562@dei.isep.ipp.pt> References: <20101126195838.M17562@dei.isep.ipp.pt> Message-ID: As soon as you know *anything* about the requirements of the system start building a prototype - show the customer the prototype. At the next meeting say "I don't really know what your problem is but I've built a little prototype to test my understanding ..." Do not tell them "your problem is easy" (never, never, never (even if it is), this is always perceived as an insult) Offer the customer a fixed price, fixed delivery time, pre-development prototype. The cost of this should be less than the cost of making a requirements specification. Do not make an offer for the final product - say "I cannot I do not know enough" - Try to deliver a working prototype within three weeks - even if they don't pay - do it anyway. Assume you will land the project and start working on it today. Don't sell the final product - but the next step - which should be small understandable, cheap and quickly delivered. Tell them this is "agile" and you are a "scrum master" - they will love this you don't have to know what the words mean - these are just magic words that need to be said. Do not use words like "functional" "proof" etc. these have bad karma. Start building the prototype as soon as you read this mail. This approach has worked many times. You're selling time-to-market and low-development costs - they will be skeptical - you will do this with 10% of the effort needed for Java/C++ - but they will not believe you. So go do it ... Cheers, and congratulations on landing the project :-) /Joe On Fri, Nov 26, 2010 at 8:59 PM, Paulo Alexandre Ferreira wrote: > > > You better not try to sell ?Erlang, or any language to managers. > > > You should try ?selling a reliable, robust, fault-tolerant, code verified ?system. > You will do code upgrades without stopping the system. > You can offer proofs the system is correct. > > > They don't want Erlang. ?They don't care about Erlang. > > You will use Erlang to build the system they want. > > The advantages of language X,Y or Z are the stuff programmers care about. > Managers care about ?different things like ?costs, down-time, reliability. > Focus on that. > > > My apologies if I sound ?a little bit grumpy. > > Paulo Ferreira > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From zabrane3@REDACTED Sat Nov 27 12:30:50 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Sat, 27 Nov 2010 12:30:50 +0100 Subject: "sendfile" call as NIF. Good or bad idea? Message-ID: Hi guys, There exists a "sendfile" linked-in driver extracted (Tuncer Ayaz) from "yaws" (Steve Vinoski) at: https://github.com/tuncer/sendfile The code is a bit cryptic to me (no offense) and I was wondering if it's good (or not) to re-implement it as a NIF? I'm asking that question after seeing Joel experiment on CouchDB speedup: http://www.trapexit.org/forum/viewtopic.php?p=62191&sid=67e9d1167a4d0d6c1fb6f9f4b81ff92b Thanks in advance. -- Regards Zabrane From alain.odea@REDACTED Sat Nov 27 14:34:44 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Sat, 27 Nov 2010 10:04:44 -0330 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: References: <20101126195838.M17562@dei.isep.ipp.pt> Message-ID: Joe, I love how you can condense whole books on contracting down to a few paragraphs :) I took a similar approach with http://verafin.com/. Though it was an internal project, there was already a VisualBasic CMS in place that was "perfectly serviceable". I had to prototype a solution to make the problem with the status quo visible. I installed Zotonic, loaded the content from the old CMS and implemented some key interactions the other CMS was making hard in a single day. I showed it to the team: as soon as they saw the radical speed difference (5-10x faster page loads), the CMS interface, the new interactions and realized that all this was done in a day they were sold. On Saturday, November 27, 2010, Joe Armstrong wrote: > As soon as you know *anything* about the requirements of the system > start building > a prototype - show the customer the prototype. > > At the next meeting say "I don't really know what your problem is but > I've built a little > prototype to test my understanding ..." > > Do not tell them "your problem is easy" (never, never, never (even if > it is), this is > always perceived as an insult) > > Offer the customer a fixed price, fixed delivery time, pre-development > prototype. > The cost of this should be less than the cost of making a requirements > specification. > > Do not make an offer for the final product - say "I cannot I do not > know enough" - > > Try to deliver a working prototype within three weeks - even if they > don't pay - do it anyway. > > Assume you will land the project and start working on it today. > > Don't sell the final product - but the next step - which should be > small understandable, cheap > and quickly delivered. Tell them this is "agile" and you are a "scrum > master" - they will love this > you don't have to know what the words mean - these are just magic > words that need to be said. > > Do not use words like "functional" "proof" etc. these have bad karma. > > Start building the prototype as soon as you read this mail. > > This approach has worked many times. > > You're selling time-to-market and low-development costs - they will be > skeptical - you > will do this with 10% of the effort needed for Java/C++ - but they > will not believe you. So > go do it ... > > Cheers, and congratulations on landing the project :-) > > /Joe > > > On Fri, Nov 26, 2010 at 8:59 PM, Paulo Alexandre Ferreira > wrote: >> >> >> You better not try to sell ?Erlang, or any language to managers. >> >> >> You should try ?selling a reliable, robust, fault-tolerant, code verified ?system. >> You will do code upgrades without stopping the system. >> You can offer proofs the system is correct. >> >> >> They don't want Erlang. ?They don't care about Erlang. >> >> You will use Erlang to build the system they want. >> >> The advantages of language X,Y or Z are the stuff programmers care about. >> Managers care about ?different things like ?costs, down-time, reliability. >> Focus on that. >> >> >> My apologies if I sound ?a little bit grumpy. >> >> Paulo Ferreira >> >> >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From wiener.guy@REDACTED Sat Nov 27 15:00:46 2010 From: wiener.guy@REDACTED (Guy Wiener) Date: Sat, 27 Nov 2010 16:00:46 +0200 Subject: How to configure Inets so that ESI would work? Message-ID: Hello everyone, I am trying to make Erlang's Inets mode_esi work and run some function. So, far, I did not succeed. Can someone post a minimal example of how to run Inets so that a url like http:localhost:8099/esi/my_mod:foo will invoke the method my_mod:foo/3? (Due disclosure: I also posted this question at stackoverflow) Thanks, Guy From ulf.wiger@REDACTED Sat Nov 27 15:52:05 2010 From: ulf.wiger@REDACTED (Ulf Wiger) Date: Sat, 27 Nov 2010 15:52:05 +0100 Subject: [erlang-questions] How to configure Inets so that ESI would work? In-Reply-To: References: Message-ID: <30260CE4-4C84-43F4-B22D-9F3B0A6A02BC@erlang-solutions.com> So, the inets/examples/server_root/conf/8080.conf and inets/src/http_server/httpd_example.erl didn't help? (Perhaps you didn't find them? They are not that well advertised, and the examples themselves are not exactly minimal either.) The user guide could definitely use a Getting Started section that walks through the existing examples, at least. BR, Ulf W On 27 Nov 2010, at 15:00, Guy Wiener wrote: > Hello everyone, > I am trying to make Erlang's Inets mode_esi work and run some function. So, > far, I did not succeed. Can someone post a minimal example of how to run > Inets so that a url like http:localhost:8099/esi/my_mod:foo will invoke the > method my_mod:foo/3? > > (Due disclosure: I also posted this question at stackoverflow) > > Thanks, > Guy Ulf Wiger, CTO, Erlang Solutions, Ltd. http://erlang-solutions.com From alain.odea@REDACTED Sat Nov 27 17:14:07 2010 From: alain.odea@REDACTED (Alain O'Dea) Date: Sat, 27 Nov 2010 12:44:07 -0330 Subject: [erlang-questions] How to configure Inets so that ESI would work? In-Reply-To: References: Message-ID: Can you post the StackOverflow link? :) On Saturday, November 27, 2010, Guy Wiener wrote: > Hello everyone, > I am trying to make Erlang's Inets mode_esi work and run some function. So, > far, I did not succeed. Can someone post a minimal example of how to run > Inets so that a url like http:localhost:8099/esi/my_mod:foo will invoke the > method my_mod:foo/3? > > (Due disclosure: I also posted this question at stackoverflow) > > Thanks, > ?Guy > From jesper.louis.andersen@REDACTED Sat Nov 27 17:20:24 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Sat, 27 Nov 2010 17:20:24 +0100 Subject: [erlang-questions] How to configure Inets so that ESI would work? In-Reply-To: References: Message-ID: http://stackoverflow.com/questions/4291973/working-example-for-erlang-server-interface :) On Sat, Nov 27, 2010 at 5:14 PM, Alain O'Dea wrote: > Can you post the StackOverflow link? :) > > On Saturday, November 27, 2010, Guy Wiener wrote: >> Hello everyone, >> I am trying to make Erlang's Inets mode_esi work and run some function. So, >> far, I did not succeed. Can someone post a minimal example of how to run >> Inets so that a url like http:localhost:8099/esi/my_mod:foo will invoke the >> method my_mod:foo/3? >> >> (Due disclosure: I also posted this question at stackoverflow) >> >> Thanks, >> ??Guy >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- J. From banibrata.dutta@REDACTED Sat Nov 27 18:51:33 2010 From: banibrata.dutta@REDACTED (Banibrata Dutta) Date: Sat, 27 Nov 2010 23:21:33 +0530 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: References: <20101126195838.M17562@dei.isep.ipp.pt> Message-ID: As Alain rightly said, Joe you just managed to condense couple of books, and I'd say a semester's worth of study on Solution Selling and Agile development. Essentially that's all you need to know. Rest you'd probably figure out on the way. All of this applies to other languages as well, and had this been a bulletin-board style discussion forum, I'd have voted for this to be a sticky !! On Sat, Nov 27, 2010 at 3:21 PM, Joe Armstrong wrote: > As soon as you know *anything* about the requirements of the system > start building > a prototype - show the customer the prototype. > > At the next meeting say "I don't really know what your problem is but > I've built a little > prototype to test my understanding ..." > > Do not tell them "your problem is easy" (never, never, never (even if > it is), this is > always perceived as an insult) > > Offer the customer a fixed price, fixed delivery time, pre-development > prototype. > The cost of this should be less than the cost of making a requirements > specification. > > Do not make an offer for the final product - say "I cannot I do not > know enough" - > > Try to deliver a working prototype within three weeks - even if they > don't pay - do it anyway. > > Assume you will land the project and start working on it today. > > Don't sell the final product - but the next step - which should be > small understandable, cheap > and quickly delivered. Tell them this is "agile" and you are a "scrum > master" - they will love this > you don't have to know what the words mean - these are just magic > words that need to be said. > > Do not use words like "functional" "proof" etc. these have bad karma. > > Start building the prototype as soon as you read this mail. > > This approach has worked many times. > > You're selling time-to-market and low-development costs - they will be > skeptical - you > will do this with 10% of the effort needed for Java/C++ - but they > will not believe you. So > go do it ... > > Cheers, and congratulations on landing the project :-) > > /Joe > > > On Fri, Nov 26, 2010 at 8:59 PM, Paulo Alexandre Ferreira > wrote: > > > > > > You better not try to sell Erlang, or any language to managers. > > > > > > You should try selling a reliable, robust, fault-tolerant, code verified > system. > > You will do code upgrades without stopping the system. > > You can offer proofs the system is correct. > > > > > > They don't want Erlang. They don't care about Erlang. > > > > You will use Erlang to build the system they want. > > > > The advantages of language X,Y or Z are the stuff programmers care about. > > Managers care about different things like costs, down-time, > reliability. > > Focus on that. > > > > > > My apologies if I sound a little bit grumpy. > > > > Paulo Ferreira > > > > > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > -- regards, Banibrata http://www.linkedin.com/in/bdutta http://twitter.com/edgeliving From dmitriy.kargapolov@REDACTED Sat Nov 27 18:52:19 2010 From: dmitriy.kargapolov@REDACTED (Dmitriy Kargapolov) Date: Sat, 27 Nov 2010 12:52:19 -0500 Subject: SNMP AGENT-CAPABILITIES Message-ID: Hello, Trying to improve SNMP agent for some app written in Erlang, I found that Erlang MIB compiler does not support AGENT-CAPABILITIES statement which is actually a macro from SNMPv2-CONF. Does anybody know what was the reason for that? SNMPv2-CONF introduced 4 macros, but only 3 of them are supported by the compiler. It looks for me like a sad oversight. If there was a serious reason for that, such incompatibility should be at least documented. Thanks. From vinoski@REDACTED Sat Nov 27 18:53:57 2010 From: vinoski@REDACTED (Steve Vinoski) Date: Sat, 27 Nov 2010 12:53:57 -0500 Subject: [erlang-questions] "sendfile" call as NIF. Good or bad idea? In-Reply-To: References: Message-ID: On Sat, Nov 27, 2010 at 6:30 AM, zabrane Mikael wrote: > Hi guys, > > There exists a "sendfile" linked-in driver extracted (Tuncer Ayaz) > from "yaws" (Steve Vinoski) at: > https://github.com/tuncer/sendfile > > The code is a bit cryptic to me (no offense) and I was wondering if > it's good (or not) to re-implement it as a NIF? > I'm asking that question after seeing Joel experiment on CouchDB speedup: > http://www.trapexit.org/forum/viewtopic.php?p=62191&sid=67e9d1167a4d0d6c1fb6f9f4b81ff92b Not sure why you find it cryptic, since as drivers go it's pretty simple. Can you provide further details? The benefit of a driver is that we can just register each socket FD with erts and it lets us know when each socket is writeable. The driver mainly just handles tracking how much was written by each sendfile call, incrementing offsets, etc. If you implemented it as a NIF, you'd need to perform your own polling on the socket FDs to know when sendfile can write to them. This alone might mean a portability issue, since you'd have to know to use the right polling mechanism for the given platform. You'd also need to do all this in your own thread, whereas the driver doesn't have to deal with threads. --steve From erlang@REDACTED Sat Nov 27 19:47:01 2010 From: erlang@REDACTED (Joe Armstrong) Date: Sat, 27 Nov 2010 19:47:01 +0100 Subject: [erlang-questions] Convincing industry to use Erlang In-Reply-To: References: <20101126195838.M17562@dei.isep.ipp.pt> Message-ID: On Sat, Nov 27, 2010 at 6:51 PM, Banibrata Dutta wrote: > As Alain rightly said, Joe you just managed to condense couple of books, and > I'd say a semester's worth of study on Solution Selling and Agile > development. Funny thing is that I haven't got a clue what all this agile and scrum stuff is. Somebody did describe agile to me and I thought "that's what we've being doing for years" - I *do* know that the word "agile" has magic powers when used against management. And "the value proposition is ..." - this is also a magic expression. Say: "the value proposition is that instead of spending time and money on a paper specification, why not use less money to build a working prototype and use the code as the specification ..." On several occasions I've seen people just start work on a project without prior approval doing the work in their spare time, they just assumed that they would get the job and that a contract was something that would just "happen" if they did the right thing. Actually basic psychology is involved - if you work on their project for free and give them the results this is a "present" that they did not expect nor ask for. Basic human nature expects some kind of response - this is what Cialdini calls "reciprocity" (read "The psychology of persuasion") there is a wikipedia reference here http://en.wikipedia.org/wiki/Robert_Cialdini#Six_.22Weapons_of_Influence.22 If you do a lot of work for free they reciprocate with a contract ... /Joe > Essentially that's all you need to know. Rest you'd probably > figure out on the way. All of this applies to other languages as well, and > had this been a bulletin-board style discussion forum, I'd have voted for > this to be a sticky !! > > On Sat, Nov 27, 2010 at 3:21 PM, Joe Armstrong wrote: >> >> As soon as you know *anything* about the requirements of the system >> start building >> a prototype - show the customer the prototype. >> >> At the next meeting say "I don't really know what your problem is but >> I've built a little >> prototype to test my understanding ..." >> >> Do not tell them "your problem is easy" (never, never, never (even if >> it is), this is >> always perceived as an insult) >> >> Offer the customer a fixed price, fixed delivery time, pre-development >> prototype. >> The cost of this should be less than the cost of making a requirements >> specification. >> >> Do not make an offer for the final product - say "I cannot I do not >> know enough" - >> >> Try to deliver a working prototype within three weeks - even if they >> don't pay - do it anyway. >> >> Assume you will land the project and start working on it today. >> >> Don't sell the final product - but the next step - which should be >> small understandable, cheap >> and quickly delivered. Tell them this is "agile" and you are a "scrum >> master" - they will love this >> you don't have to know what the words mean - these are just magic >> words that need to be said. >> >> Do not use words like "functional" "proof" etc. these have bad karma. >> >> Start building the prototype as soon as you read this mail. >> >> This approach has worked many times. >> >> You're selling time-to-market and low-development costs - they will be >> skeptical - you >> will do this with 10% of the effort needed for Java/C++ - but they >> will not believe you. So >> go do it ... >> >> Cheers, and congratulations on landing the project :-) >> >> /Joe >> >> >> On Fri, Nov 26, 2010 at 8:59 PM, Paulo Alexandre Ferreira >> wrote: >> > >> > >> > You better not try to sell ?Erlang, or any language to managers. >> > >> > >> > You should try ?selling a reliable, robust, fault-tolerant, code >> > verified ?system. >> > You will do code upgrades without stopping the system. >> > You can offer proofs the system is correct. >> > >> > >> > They don't want Erlang. ?They don't care about Erlang. >> > >> > You will use Erlang to build the system they want. >> > >> > The advantages of language X,Y or Z are the stuff programmers care >> > about. >> > Managers care about ?different things like ?costs, down-time, >> > reliability. >> > Focus on that. >> > >> > >> > My apologies if I sound ?a little bit grumpy. >> > >> > Paulo Ferreira >> > >> > >> > >> > ________________________________________________________________ >> > erlang-questions (at) erlang.org mailing list. >> > See http://www.erlang.org/faq.html >> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > >> > >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > > > -- > regards, > Banibrata > http://www.linkedin.com/in/bdutta > http://twitter.com/edgeliving > From zabrane3@REDACTED Sat Nov 27 20:06:43 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Sat, 27 Nov 2010 20:06:43 +0100 Subject: [erlang-questions] "sendfile" call as NIF. Good or bad idea? In-Reply-To: References: Message-ID: Hi Steve, > Not sure why you find it cryptic, since as drivers go it's pretty > simple. Can you provide further details? Maybe it was because NIF are very simple to read/write. Again, no offense ... forget about that comment. > The benefit of a driver is that we can just register each socket FD > with erts and it lets us know when each socket is writeable. The > driver mainly just handles tracking how much was written by each > sendfile call, incrementing offsets, etc. Thanks for this. I didn't consider this problem at all. > If you implemented it as a NIF, you'd need to perform your own polling > on the socket FDs to know when sendfile can write to them. This alone > might mean a portability issue, since you'd have to know to use the > right polling mechanism for the given platform. You'd also need to do > all this in your own thread, whereas the driver doesn't have to deal > with threads. So, I'll stick with your driver. Thanks Steve. -- Regards Zabrane From banibrata.dutta@REDACTED Sun Nov 28 06:53:35 2010 From: banibrata.dutta@REDACTED (Banibrata Dutta) Date: Sun, 28 Nov 2010 11:23:35 +0530 Subject: Any 'Complex Event Processing' framework on Erlang ? Message-ID: Hi, Did my search, and only found a blog-post on how this would be a great combination, and a mailing list post on similar lines, with a suggestion for using the Actor pattern. Is anyone aware of a CEP implementation in Erlang, or any suggestions. My main fundamental need is real-time processing of Alarm pipeline, and their correlation. Esper (in Java and .NET) seem like near perfect choices, to put things in context. Any pointers, advice etc. would be very useful. -- regards, Banibrata Dutta From wiener.guy@REDACTED Sun Nov 28 11:48:26 2010 From: wiener.guy@REDACTED (Guy Wiener) Date: Sun, 28 Nov 2010 12:48:26 +0200 Subject: [erlang-questions] How to configure Inets so that ESI would work? In-Reply-To: <30260CE4-4C84-43F4-B22D-9F3B0A6A02BC@erlang-solutions.com> References: <30260CE4-4C84-43F4-B22D-9F3B0A6A02BC@erlang-solutions.com> Message-ID: Thanks Ulf! These files are indeed unadvertised. I copied and modified the conf file, and it worked. I don't know which property exactly my configuration was missing, but starting with a working example seemed to do the trick. And yes, the user guide could certainly use a Getting Started section that walks through the existing example, at least... :- Guy On Sat, Nov 27, 2010 at 4:52 PM, Ulf Wiger wrote: > > So, the inets/examples/server_root/conf/8080.conf > and inets/src/http_server/httpd_example.erl didn't help? > > (Perhaps you didn't find them? They are not that well advertised, and the > examples themselves are not exactly minimal either.) > > The user guide could definitely use a Getting Started section that walks > through the existing examples, at least. > > BR, > Ulf W > > > On 27 Nov 2010, at 15:00, Guy Wiener wrote: > > > Hello everyone, > > I am trying to make Erlang's Inets mode_esi work and run some function. > So, > > far, I did not succeed. Can someone post a minimal example of how to run > > Inets so that a url like http:localhost:8099/esi/my_mod:foo will invoke > the > > method my_mod:foo/3? > > > > (Due disclosure: I also posted this question at stackoverflow) > > > > Thanks, > > Guy > > Ulf Wiger, CTO, Erlang Solutions, Ltd. > http://erlang-solutions.com > > > > From rtrlists@REDACTED Sun Nov 28 19:29:15 2010 From: rtrlists@REDACTED (Robert Raschke) Date: Sun, 28 Nov 2010 18:29:15 +0000 Subject: [erlang-questions] How to find an object in an ETS table ? In-Reply-To: <201011261932540128406@its3.ch> References: <201011261631321570072@its3.ch> <201011261734459630111@its3.ch> <201011261932540128406@its3.ch> Message-ID: On Fri, Nov 26, 2010 at 10:25 PM, info wrote: > How to evict this artifact ? > > You do not have to do anything special at all. This: [[123],[456]] is a list with two elements, each of wich is a list with one integer element. Only when you try and print it using the default Erlang term printing rules, does the _formatting_ turn the first list into its equivalent string representation. It does that, because it looks like a string (i.e., a list of integers < 128). 1> [[V1],[V2]]=[[123],[456]]. ["{",[456]] 2> V1. 123 3> Robby From uaforum1@REDACTED Mon Nov 29 06:26:05 2010 From: uaforum1@REDACTED (u a) Date: Sun, 28 Nov 2010 21:26:05 -0800 (PST) Subject: ets question Message-ID: Hello, i build a small service where i store a huge lit of devices in a single ets table. When i query this list, i have to make a full table scan, because i have to look at each device. For optimization i want to classify the devices, so i can strore them in different ets tables. For instance, all symbian devices will be stored in the symbian ets table and in the all_devices table. My question is the following: When i create the device and store them in both tables, symbian and all_devices, how is it managed. Is there only one object, which is shared across these two table or are there copies ? What, if a delete one device in the symbian table? Is it still in the all_devices table? If there are copies, i have to think about another solution, perhaps storing the devices into mnesia. Thanks for your help, Ulf From kuleshovmail@REDACTED Mon Nov 29 08:08:15 2010 From: kuleshovmail@REDACTED (shk) Date: Sun, 28 Nov 2010 23:08:15 -0800 (PST) Subject: Client server in Erlang Message-ID: <1291014495228-3063129.post@n4.nabble.com> Hello, I recently started to learn Erlang. I read book - Erlang Programming. In Client-Server chapter i try to run example. Code of this example: https://gist.github.com/719677 This is simple client/sever example. In erlang shell i see: 1> f:start(). true 2> f:allocate() When i call allocate erlang shell, emacs and other hangs. What's wrong? Thank you. -- View this message in context: http://erlang.2086793.n4.nabble.com/Client-server-in-Erlang-tp3063129p3063129.html Sent from the Erlang Questions mailing list archive at Nabble.com. From kuleshovmail@REDACTED Mon Nov 29 08:12:26 2010 From: kuleshovmail@REDACTED (Alexander Kuleshov) Date: Mon, 29 Nov 2010 07:12:26 +0000 Subject: [erlang-questions] Client server in Erlang In-Reply-To: <1291014495228-3063129.post@n4.nabble.com> References: <1291014495228-3063129.post@n4.nabble.com> Message-ID: On Mon, Nov 29, 2010 at 7:08 AM, shk wrote: > > Hello, > > I recently started to learn Erlang. I read book - Erlang Programming. In > Client-Server chapter i try to run example. > > Code of this example: https://gist.github.com/719677 > > This is simple client/sever example. > > In erlang shell i see: > > 1> f:start(). > true > 2> f:allocate() > > When i call allocate erlang shell, emacs and other hangs. > > What's wrong? > > Thank you. > -- > View this message in context: http://erlang.2086793.n4.nabble.com/Client-server-in-Erlang-tp3063129p3063129.html > Sent from the Erlang Questions mailing list archive at Nabble.com. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > I find error: function reply was wrong: Must be: reply(Pid, Reply) -> Pid ! {reply, Reply}. From Mike.French@REDACTED Mon Nov 29 08:49:43 2010 From: Mike.French@REDACTED (French, Mike) Date: Mon, 29 Nov 2010 07:49:43 -0000 Subject: [erlang-questions] Any 'Complex Event Processing' framework o n Erlang ? Message-ID: <3F8EEA01CF53D74DB4A9EC314D82B4F3024E2AC7@wells154924> Here is a talk about a CEP prototype at SAP research labs: http://www.erlang-factory.com/upload/presentations/57/SumeetBajaj_ErlangatSA P.pdf Mike > -----Original Message----- > From: erlang-questions@REDACTED > [mailto:erlang-questions@REDACTED]On > Behalf Of Banibrata Dutta > Sent: 28 November 2010 05:54 > To: erlang-questions@REDACTED > Subject: [erlang-questions] Any 'Complex Event Processing' > framework on > Erlang ? > > > Hi, > > Did my search, and only found a blog-post on how this would be a great > combination, and a mailing list post on similar lines, with a > suggestion for using the Actor pattern. > Is anyone aware of a CEP implementation in Erlang, or any > suggestions. My main fundamental need is real-time processing of Alarm > pipeline, and their correlation. > Esper (in Java and .NET) seem like near perfect choices, to > put things in context. Any pointers, advice etc. would be very useful. > > -- > regards, > Banibrata Dutta > Thales UK Ltd (Wells) DISCLAIMER: The information contained in this e-mail is confidential. It may also be legally privileged. It is intended only for the stated addressee(s) and access to it by any other person is unauthorised. If you are not an addressee, you must not disclose, copy, circulate or in any other way use or rely on the information contained in this e-mail. Such unauthorised use may be unlawful. We may monitor all e-mail communications through our networks. If you have received this e-mail in error, please inform us immediately on sender's telephone number above and delete it and all copies from your system. We accept no responsibility for changes to any e-mail which occur after it has been sent. Attachments to this e-mail may contain software viruses which could damage your system. We therefore recommend you virus-check all attachments before opening. Thales UK Ltd. Registered Office: 2 Dashwood Lang Road, The Bourne Business Park, Addlestone, Weybridge, Surrey KT15 2NX Registered in England No. 868273 From banibrata.dutta@REDACTED Mon Nov 29 09:36:03 2010 From: banibrata.dutta@REDACTED (Banibrata Dutta) Date: Mon, 29 Nov 2010 14:06:03 +0530 Subject: [erlang-questions] Any 'Complex Event Processing' framework o n Erlang ? In-Reply-To: <3F8EEA01CF53D74DB4A9EC314D82B4F3024E2AC7@wells154924> References: <3F8EEA01CF53D74DB4A9EC314D82B4F3024E2AC7@wells154924> Message-ID: Thanks Mike. The paper does have some interesting information, but apparently they are researching some implementation of RETE networks, and not a comprehensive CEP framework. I wonder, why they didn't consider ERSEYE for RETE based rule-processing. Also, anyone aware of a Bayesian Classifier implementation in Erlang ? Anyone has experience in using the CRM114 lib, integrated with Erlang ? regards, BDutta On Mon, Nov 29, 2010 at 1:19 PM, French, Mike < Mike.French@REDACTED> wrote: > > Here is a talk about a CEP prototype at SAP research labs: > > > > http://www.erlang-factory.com/upload/presentations/57/SumeetBajaj_ErlangatSA > P.pdf > > Mike > > > -----Original Message----- > > From: erlang-questions@REDACTED > > [mailto:erlang-questions@REDACTED]On > > Behalf Of Banibrata Dutta > > Sent: 28 November 2010 05:54 > > To: erlang-questions@REDACTED > > Subject: [erlang-questions] Any 'Complex Event Processing' > > framework on > > Erlang ? > > > > > > Hi, > > > > Did my search, and only found a blog-post on how this would be a great > > combination, and a mailing list post on similar lines, with a > > suggestion for using the Actor pattern. > > Is anyone aware of a CEP implementation in Erlang, or any > > suggestions. My main fundamental need is real-time processing of Alarm > > pipeline, and their correlation. > > Esper (in Java and .NET) seem like near perfect choices, to > > put things in context. Any pointers, advice etc. would be very useful. > > > From dkoch@REDACTED Mon Nov 29 14:07:20 2010 From: dkoch@REDACTED (dkoch@REDACTED) Date: Mon, 29 Nov 2010 14:07:20 +0100 (CET) Subject: [erlang-questions] Vlad Dumitrescu has looked at porting to QNX. If you're interested, ask on the erlang mailing list. Message-ID: <14198425.36961291036040685.JavaMail.www@wsfrf1112> Maybe you already know that the latest Erlang VM compiled against QNX is R9C that you may found here : http://www.openqnx.com/index.php?name=News&file=article&sid=229 I think that with a little refresh of the source code, some diff and patches, you might be compiling the latest R14B1 as well under QNX 6.5.0 :) See ya... David KOCH ======================================== Message du : 25/11/2010 De : "Damian Dobroczy?ski " A : erlang-questions@REDACTED Copie ? : Sujet : [erlang-questions] Vlad Dumitrescu has looked at porting to QNX. If you're interested, ask on the erlang mailing list. From info@REDACTED Mon Nov 29 16:30:55 2010 From: info@REDACTED (info) Date: Mon, 29 Nov 2010 16:30:55 +0100 Subject: erlang and the GUI Message-ID: <201011291630544682713@its3.ch> Hello, Triple question: 1) I would like to display in a large multiline textbox data received from tcp socket but the GS package has only the one-line entry object ! What can I use ? the editor object ? do you know an example ? 2) Several GUI packages are available: gs,wxErlang, ... Which one do you advice to use taking into account several criteria: support (doc,examples), standard, stability, reliability 3) And what about GUI builder ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From garazdawi@REDACTED Mon Nov 29 16:49:39 2010 From: garazdawi@REDACTED (Lukas Larsson) Date: Mon, 29 Nov 2010 16:49:39 +0100 Subject: [erlang-questions] erlang and the GUI In-Reply-To: <201011291630544682713@its3.ch> References: <201011291630544682713@its3.ch> Message-ID: Hello! You probably want to use wxErlang to build any GUI as gs is depricated and not supported any more. See the big warning on the top of http://www.erlang.org/doc/man/gs.html As for GUI builder you might try wxGlade and then export to XRC and use the wxXmlResource module to run the GUI. I haven't tried it but hopefully it will work :) Lukas On Mon, Nov 29, 2010 at 4:30 PM, info wrote: > Hello, > Triple question: > 1) I would like to display in a large multiline textbox data received from > tcp socket but the GS package has only the one-line entry object ! What can > I use ? the editor object ? do you know an example ? > 2) Several GUI packages are available: gs,wxErlang, ... > Which one do you advice to use taking into account several criteria: > support (doc,examples), standard, stability, reliability > 3) And what about GUI builder ? > > J-Ph. Constantin > ITS3 Gen?ve > www.its3.ch > From qoocku@REDACTED Mon Nov 29 17:08:52 2010 From: qoocku@REDACTED (=?UTF-8?B?RGFtaWFuIERvYnJvY3p5xYRza2k=?=) Date: Mon, 29 Nov 2010 17:08:52 +0100 Subject: [erlang-questions] Vlad Dumitrescu has looked at porting to QNX. If you're interested, ask on the erlang mailing list. In-Reply-To: <14198425.36961291036040685.JavaMail.www@wsfrf1112> References: <14198425.36961291036040685.JavaMail.www@wsfrf1112> Message-ID: <4CF3D014.6080200@gmail.com> W dniu 29.11.2010 14:07, dkoch@REDACTED pisze: > Maybe you already know that the latest Erlang VM compiled against QNX is R9C that you may found here : > > > http://www.openqnx.com/index.php?name=News&file=article&sid=229 > > Yes, I have found it. And so my question about some fresh news ;) Do you know if these "missing" bits (SA_ONSTACK & SIGSTKSZ) are stil missing? -- D. > I think that with a little refresh of the source code, some diff and patches, you might be compiling the latest R14B1 as well under QNX 6.5.0 :) > > > See ya... > > > David KOCH -------------- next part -------------- A non-text attachment was scrubbed... Name: 0x0CAE3AEB.asc Type: application/pgp-keys Size: 4563 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From extern.4plus.zerfass.p@REDACTED Mon Nov 29 17:51:36 2010 From: extern.4plus.zerfass.p@REDACTED (Peter Zerfass) Date: Mon, 29 Nov 2010 17:51:36 +0100 Subject: Silent install of Erlang on Windows / Mac Message-ID: <4CF3E828.2115.00EA.1@duerr.de> Hello, I'm trying to find out if there is a way to install Erlang silently on Mac (OSX) and Windows (XP, 7). We're developing a software that may use RabbitMQ as a message queue system (which requires Erlang to be installed on the server side). However the technicians would like to minimize the number of installation steps. How would one go about installing Erlang without the Installer (i.e. how would we integrate the installation procedure of Erlang into our own installer)? THX Peter Zerfass --------------------------------------------------------------- D?RR DENTAL AG H?pfigheimer Strasse 17 74321 Bietigheim-Bissingen Deutschland / Germany Tel. + 49 (0) 7142 / 705-0 Fax + 49 (0) 7142 / 705-500 http://www.duerr.de Firmensitz / Registered Office: H?pfigheimer Str. 17, 74321 Bietigheim-Bissingen Handelsregister / Register of Commerce: Amtsgericht Stuttgart HRB 727006 Vorstand / Executive Board: Martin D?rrstein (Vorsitzender / CEO), Joachim Eppinger, Axel Schramm Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Walter D?rr From mikko.hamalainen@REDACTED Mon Nov 29 18:56:28 2010 From: mikko.hamalainen@REDACTED (=?iso-8859-1?Q?Mikko_H=E4m=E4l=E4inen?=) Date: Mon, 29 Nov 2010 17:56:28 +0000 Subject: [Job] Location Based MMORPG server engineers - Grey Area & Shadow Cities Message-ID: <45A0CCFE-CB0B-41F9-BC2A-999DEF6EF164@greyarealabs.com> Grey Area is an incredible company building next generation location aware multiplayer worlds. We are turning every city on the planet, including yours, into a game that can be accessed any time through your mobile phone. Think World of Warcraft on city streets. The first game world, Shadow Cities (http://www.shadowcities.com) is currently in Beta in Finland. We are going to launch it globally in coming months are are looking for talented individuals to join our server team. The whole game server is implemented in Erlang. This is very exciting time as we are building something truly new. We are looking for a highly motivated individuals who are able to work as part of a team but are also able to take ownership of parts of the system and improve the product as a whole. Erlang Server Engineers: Building the game world server architecture is a challenging task. We use Erlang as our main tool to implement an actor based and highly scalable server that is able handle our concurrent workload. We use MongoDB as our datastore. The succesful candidate will have strong Erlang knowledge and experience in designing, implementing and continuing support of large scale server applications. Skills and attributes that we?re looking for include: - A BSc in Computer Science or Software Engineering or similar experience gained from work. - Professional experience in writing server side applications. Preferably in Erlang. - Architectual expertise in designing distributed systems running in a cluster. - Expertise in Linux operating system. - Skill and patience for finding and fixing bugs in distributed multimachine systems. - Experience tuning real-time server systems for performance and reliability - Interest in building next generation multiplayer game worlds - And most important of all, you are a nice person to work with and ready to be part of an amazing team Please check out: - http://www.greyarealabs.com/jobs/ - http://www.shadowcities.com You can contact us at jobs@REDACTED Best regards, Mikko H?m?l?inen Co-Founder, Grey Area From boris.muehmer@REDACTED Mon Nov 29 19:58:50 2010 From: boris.muehmer@REDACTED (Boris =?ISO-8859-1?Q?M=FChmer?=) Date: Mon, 29 Nov 2010 19:58:50 +0100 Subject: [erlang-questions] erlang and the GUI In-Reply-To: References: <201011291630544682713@its3.ch> Message-ID: <1291057130.2497.17.camel@yang> Am Montag, den 29.11.2010, 16:49 +0100 schrieb Lukas Larsson: > As for GUI builder you might try wxGlade and then export to XRC and use the > wxXmlResource module to run the GUI. I haven't tried it but hopefully it > will work :) I did some tests with XRC (using wxFormBuilder) and it works, but it takes some time to get used to it. I'm not sure if it is because of Erlang (or that I am still a rookie), the wxWidgets framework, or the lack of documentation. Also I think it is better to use XRC than building a UI from code; maybe with better libraries and a more native (in terms of Erlang support) UI builder, it would also be much easier to use wxWidgets/XRC. One more thing: have a look at the examples (under lib/wx/examples), especially at "xrc" and "demo". In "demo" there is still the best "documentation" for the "wx_object" behaviour. Well, just my 2 cents... - boris From francesco@REDACTED Mon Nov 29 21:16:35 2010 From: francesco@REDACTED (Francesco Cesarini) Date: Mon, 29 Nov 2010 20:16:35 +0000 Subject: Free Erlang/OTP Tutorial Melbourne, OZ 4/12 In-Reply-To: References: <8EE1FC7C-C219-452D-AAE2-9E549A7D54D6@erlang-solutions.com> <7CCBF5F0-1FBA-4B1C-B1B9-2089F36D8FD0@erlang-solutions.com> Message-ID: <4CF40A23.6050008@erlang-solutions.com> Hi all, I will be giving a tutorial covering Basic & Sequential Erlang, Concurrency and Gen Servers as a replacement to Ulf Wiger's YOW! tutorial which had to be canceled at short notice. Lonely Planet (The guide book publisher!) has kindly volunteered one of their meeting rooms, seating anything from 5 to 100 people. The location is in Footscray, 5km from the city center. (A short walk from the station). The tutorial will start at 9.00 - 14.30, on Satuday December 4th. When done, we should all head to downtown Footscray to grab a bite. This hands on tutorial will give you an introduction to the Erlang programming language. You will learn the basics of how to read, write and structure Erlang programs. We start with an insight into the theory and concepts behind sequential and concurrent Erlang, allowing you to get acquainted with the Erlang syntax and semantics. We provide an overview of the error handling mechanisms used to build fault tolerant systems with five nines availability, and wrap up by introducing the gen_server, the most commonly used behaviour of the OTP middleware. This is a hands on tutorial, so install Erlang on your laptop and bring it along with lots of questions. Last catch.. While registration is free, we need to know who is attending. Drop me an email with your name, phone number and affiliation and I will forward all of the details. Please mark the subject Erlang Tutorial. Really looking forward to visiting Australia again! It's been a while. Francesco -- Erlang Solutions Ltd. http://www.erlang-solutions.com From andrey.paramonov@REDACTED Mon Nov 29 21:41:07 2010 From: andrey.paramonov@REDACTED (Andrey Paramonov) Date: Mon, 29 Nov 2010 12:41:07 -0800 (PST) Subject: Is Erlang dynamic language? Message-ID: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> Greetings. Erlang is definitely a dynamically typed language. But is it a dynamic language? I see that the term itself is not what all developers agree on - there is a whole discussion on Wikipedia about that (http:// en.wikipedia.org/wiki/Talk:Dynamic_programming_language). Many people associate dynamic languages with metaprogramming, runtime method injection, AST transformations and the like. As far as I can tell, there is no such things in Erlang. But on the other hand, Erlang has runtime code update and you can change runtime behaviour by passing functions to the methods. So it seems to be dynamic. What do you think guys? Would you consider Erlang dynamic? and why? Andrey From max.lapshin@REDACTED Mon Nov 29 21:49:12 2010 From: max.lapshin@REDACTED (Max Lapshin) Date: Mon, 29 Nov 2010 23:49:12 +0300 Subject: [erlang-questions] Is Erlang dynamic language? In-Reply-To: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> References: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> Message-ID: It is used not so often as, for example in Ruby, but there are some libraries that use metaprogramming: erlydtl created template modules or log4erl dynamically create log reducer. From tony.arcieri@REDACTED Mon Nov 29 21:49:37 2010 From: tony.arcieri@REDACTED (Tony Arcieri) Date: Mon, 29 Nov 2010 13:49:37 -0700 Subject: [erlang-questions] Is Erlang dynamic language? In-Reply-To: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> References: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> Message-ID: On Mon, Nov 29, 2010 at 1:41 PM, Andrey Paramonov < andrey.paramonov@REDACTED> wrote: > Erlang is definitely a dynamically typed language. But is it a dynamic > language? I see that the term itself is not what all developers agree > on - there is a whole discussion on Wikipedia about that (http:// > en.wikipedia.org/wiki/Talk:Dynamic_programming_language). > I think generally "dynamic language" means there are no type signatures associated with functions and therefore no automatically enforced type checking on arguments. Type checking is performed by a late binding mechanism, and if you try to do something inappropriate with a particular data type an exception is thrown. By that definition, Erlang is most certainly a dynamic language. And I think that's a good thing, because trying to support runtime code change on a system with static types sounds extremely difficult, but for the runtime implementer and the end user. > Many people associate dynamic languages with metaprogramming, runtime > method injection, AST transformations and the like. As far as I can > tell, there is no such things in Erlang. You can do all of that in Erlang. AST transformations have first class support via -parse_transform() and the rest can be done with a library like Smerl. -- Tony Arcieri Medioh! A Kudelski Brand From jesper.louis.andersen@REDACTED Mon Nov 29 22:23:19 2010 From: jesper.louis.andersen@REDACTED (Jesper Louis Andersen) Date: Mon, 29 Nov 2010 22:23:19 +0100 Subject: [erlang-questions] Is Erlang dynamic language? In-Reply-To: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> References: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> Message-ID: On Mon, Nov 29, 2010 at 9:41 PM, Andrey Paramonov wrote: > > Many people associate dynamic languages with metaprogramming, runtime > method injection, AST transformations and the like. As far as I can > tell, there is no such things in Erlang. But on the other hand, Erlang > has runtime code update and you can change runtime behaviour by > passing functions to the methods. So it seems to be dynamic. Erlang actually has all of the things you associate with a dynamic language as Tony Arcieri hinted. As for the dynamically/statically typing question, one way to see it (which is in the first few chapters of SICP if I remember correctly) is that a dynamically typed language puts type on values whereas a statically ditto puts them on variables/identifiers. It seem to make sense: Foo can be either an integer(), an atom() or a string(), so no type is associated with the Foo identifier. On the other hand, 37 is an integer(), trap_exit an atom() and "Hello Lambda!" a string(). In a statically typed language, Foo would have had a specific type declaring, e.g., "Foo :: integer()" so "Foo can only contain integers!!!". Type theorists tend to call dynamically typed languages for *untyped* languages. The correct term is perhaps trivially typed, for we can invent a "magical" type ? and proceed to give every expression in the language this type ?. What is ? you ask? Well, it is any primitive or compound value you can validly form in the programming language. We have thus just added a type system to the language - albeit a trivial one and in this sense, all programming languages are typed. Our magical ? has a big shortcoming however: If a program is well-typed, we can't say anything about it at all! In contrast, with statically typed programs we tend to have certain properties (called meta-theoretic properties) of programs that *passes* a type check. Most often it has something to do with the idea that you can't accidentally form certain invalid statements, like for instance the expression '37 + trap_exit' which has no meaning. We tend to like the term 'untyped' but really we mean 'programming language with no type system guarantees'. In general, I don't think it is interesting to classify a language as being dynamic by claiming a broad group of features a dynamic language must have to be called such. Rather I'd recommend that you look at each of those features you mentioned, metaprogramming, code injection, ..., in isolation and look for languages that have each individual component. -- J. From dave@REDACTED Mon Nov 29 22:33:25 2010 From: dave@REDACTED (Dave Cottlehuber) Date: Tue, 30 Nov 2010 10:33:25 +1300 Subject: [erlang-questions] Silent install of Erlang on Windows / Mac In-Reply-To: <4CF3E828.2115.00EA.1@duerr.de> References: <4CF3E828.2115.00EA.1@duerr.de> Message-ID: On 30 November 2010 05:51, Peter Zerfass wrote: > Hello, > > I'm trying to find out if there is a way to install Erlang silently on > Mac (OSX) and Windows (XP, 7). We're developing a software that may > use > RabbitMQ as a message queue system (which requires Erlang to be > installed on the server side). However the technicians would like to > minimize the number of installation steps. > > How would one go about installing Erlang without the Installer (i.e. > how would we integrate the installation procedure of Erlang into our > own > installer)? > > THX > ? Peter Zerfass For windows here's some options. Include the released erlang win32 precompiled binary, and run this monolithic silent install stage within your custom app installer. You can also set the install directory with this approach. More info http://nsis.sourceforge.net/Docs/Chapter3.html#3.2.1 as Erlang is packaged using the NSIS tool. Alternatively build your erlang from scratch & then include your app's extra bits under the created file structure "./release/win32/" in your custom app installer. When you install your app, you'll need first to do "Install.exe -s" to set erlang up before doing anything further. This is the tidiest but involves a reasonable amount of mess to get the installer working. A quicker but possibly not clean approach would be to install erlang, removing the two erl.ini files which to my knowledge are the only changes "install.exe -s" actually makes, as a way of avoiding building erlang from scratch. Cheers Dave From ok@REDACTED Tue Nov 30 00:50:19 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 30 Nov 2010 12:50:19 +1300 Subject: [erlang-questions] Client server in Erlang In-Reply-To: <1291014495228-3063129.post@n4.nabble.com> References: <1291014495228-3063129.post@n4.nabble.com> Message-ID: <56978044-8483-4B14-9E50-47776B90E614@cs.otago.ac.nz> In the server process, reply(Client, {ok,10}) sends Client!{ok,10} but the client is waiting at receive {reply,Reply} -> Reply They don't match. From ok@REDACTED Tue Nov 30 01:55:07 2010 From: ok@REDACTED (Richard O'Keefe) Date: Tue, 30 Nov 2010 13:55:07 +1300 Subject: [erlang-questions] Is Erlang dynamic language? In-Reply-To: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> References: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> Message-ID: On 30/11/2010, at 9:41 AM, Andrey Paramonov wrote: > Greetings. > > Erlang is definitely a dynamically typed language. But is it a dynamic > language? I see that the term itself is not what all developers agree > on - there is a whole discussion on Wikipedia about that (http:// > en.wikipedia.org/wiki/Talk:Dynamic_programming_language). Look at the list of criteria: Interpreted? Yes, maybe, no. The shell interprets, normal loading compiles to byte codes, native code is available. Dynamically typed? Yes. But we're seeing increasing use of types for documentation and checking. Check as much as possible at run time? No. By that criterion, the only dynamic language I have on any machine would be APL (where symbols can change syntax between nullary, unary, and binary so that you can't in general tell whether F G H means F(G(H)) or G(F,H) until a statement is actually executed, and on the next execution it might be different). No need for type annotations? Yes. Modifying properties of the language itself at run time? Yes, in principle. Nothing stops you writing a modified compiler and loading it. (Which is what you would have to do in Smalltalk too.) Run-time read-only introspection? Yes. (Even Java does that.) Run-time read-write introspection? In a language with immutable data structures, can you tell the difference between a facility that isn't there and one that would have nothing to work on? Arguably the tracing and profiling tools *are* this sort of thing. Dynamically linked? Yes. New and shiny? Shiny but not new. I think it makes more sense to ask whether a language has a particular feature or allows a particular effect than to ask whether it is a "dynamic language" as such. > > Many people associate dynamic languages with metaprogramming, runtime > method injection, AST transformations and the like. As far as I can > tell, there is no such things in Erlang. As for AST transformation, what did you think parse transforms were? There's a difference between a language in which something cannot be done at all and a language in which something is awkward. If you want *simple* metaprogramming, LFE is a front end for Erlang that provides Lispy syntax, including macro I'm not perfectly familiar with the term "method injection"; it seems to be some form of compiling new functions at run time, possibly from some sort of template. It's quite straightforward to generate new modules (as ASTs) at run time, compile them, and load them. It's not a thing people seem to do much, possibly because people don't design code that needs it. Erlang is dynamic *enough* for its intended uses. From bob@REDACTED Tue Nov 30 03:04:40 2010 From: bob@REDACTED (Bob Ippolito) Date: Tue, 30 Nov 2010 09:04:40 +0700 Subject: [erlang-questions] Erlang overload/bandwidth limitation (JOBS?) In-Reply-To: References: Message-ID: I'm not familiar with JOBS but your first case is easily solved with just serializing that work to a single process, e.g. a gen_server. The mailbox of that process is your queue. Without seeing the code I don't have anything very specific to suggest for the second case, but you'll probably want to look at using ets and ets:update_counter/3. On Tue, Nov 23, 2010 at 7:20 PM, Jesper Louis Andersen wrote: > Hi List, > > The etorrent application, a bittorrent client for erlang, is in need > of some load balancing in two ways, which I describe below, and I need > a tool to do some load balancing in the two cases. Are there any > suggestions besides using the Erlang Solutions JOBS application for > the task? > > Case 1: The client can handle multiple torrents at once. When you > start off on a torrent and you find a file existing in the download > directory matching the name and size of what you want, you have to > check it. This check is running SHA1 checksums over the whole file and > thus puts quite a burden on the disk IO subsystem of the computer. In > the lucky case that we already did some checking and the file passes a > sanity check, we can look up a lot of the checking result in a > persistent table - but in general we have to go over the file. To not > trash the disk we only want to allow a single torrent to be checked at > a time. > > The current solution is ugly. A token is passed among the torrents who > wants checking and he who has the token may check. There are some > monitoring going on to ensure that if we loose the token, we reinject > a new one in the system. As such, there is some reminiscence of a > token ring configuration. > > JOBS seems like a good fit for this case: Add a job queue with a > severe limitation of 1. Fire off a job to the queue (which you make > large enough that it can hold the pending jobs) which is the actual > checking code. Done. > > Case 2: Upstream bandwidth limitation. TCP will by default eat all of > a lines upstream bandwidth, which unfortunately includes ACK-packets > needed for the downstream. The usual trick is to put some limitation > to how much you send out from the client to crudely steer the > connection. A very big problem here is that most internet connections > from homes have cheap routers in front of them and these have a small > buffer of, say, 8 packets. Any more packets arriving are simply > dropped from the router. This makes TCP sad and erratic such that it > cannot really do its job too well. There is an UDP based protocol > extension, ?TP, which attempt to battle this problem directly but that > protocol is a bit out in the future, hence the bandwidth limitation. > > The question is, can JOBS or some other overload framework be easily > adapted to this scenario? > > Discussion: Basically, you have a lot of willing senders contending > for a global upstream bandwidth resource. You allow them access to the > resource but you don't want one sender to claim all of the resource. > Hence you can round-robin schedule the bandwidth in smaller chunks or > use RED - random early detection; in which the greatest hoarder runs a > larger risk of being denied bandwidth. I don't know if the JOBS > framework can do this in full or in part. > > > -- > J. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From andrey.paramonov@REDACTED Tue Nov 30 03:30:13 2010 From: andrey.paramonov@REDACTED (Andrey Paramonov) Date: Mon, 29 Nov 2010 18:30:13 -0800 (PST) Subject: Is Erlang dynamic language? In-Reply-To: References: <075c682f-0426-4ab5-8f75-7f006431c1f5@r19g2000prm.googlegroups.com> Message-ID: <94d7c843-8415-4dcc-9163-34f1dc383fb9@s5g2000yqm.googlegroups.com> Thank you guys. I didn't know about smerl, parse transforms and all metaprogramming goodies. Erlang has always been dynamic *enough* for me; now I see that it's *quite* dynamic for any dynamic purist. And I totally agree with you that dynamic/static classification doesn't really matter as soon as the language allows me to solve the problem elegantly. Thanks again. From kaiduanx@REDACTED Tue Nov 30 05:18:21 2010 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Mon, 29 Nov 2010 23:18:21 -0500 Subject: {error, emfile} on windows Message-ID: Hi, all, I tried to create 1K TCP/TLS client connections on a Windows box, and got {error, emfile} error, any idea to solve it? I have no problem on Linux after raising the ulimit. Thanks, /Kaiduan From nbowe@REDACTED Tue Nov 30 06:42:07 2010 From: nbowe@REDACTED (Nikolas Bowe) Date: Tue, 30 Nov 2010 16:42:07 +1100 Subject: Erlounge Melbourne - Sat 4/12 Message-ID: <4CF48EAF.5070304@infinite-interactive.com> Hi everyone. There will be an Erlounge at Beer DeLuxe from 7pm on Saturday 4th Dec. This is just a social event. No presentations this time. Maybe next time depending on what people want. This is an opportunity to mingle with other people that are using or are interested in Erlang, and generally geek out. Hopefully a few of the YOW conference speakers can come and sample some of our fine Australian beers. The weather should be good, so we may as well meet in their beer garden so we can enjoy the last of the day's sun. You dont have to be using Erlang. Anyone that's interested is welcome. Just a reminder: Francesco is giving a tutorial in the day (from 9). See his email for details. If you work at a company that is thinking of using Erlang for a product or prototype I highly recommend you send some people along. This message and its attachments may contain legally privileged or confidential information. This message is intended for the use of the individual or entity to which it is addressed. If you are not the addressee indicated in this message, or the employee or agent responsible for delivering the message to the intended recipient, you may not copy or deliver this message or its attachments to anyone. Rather, you should permanently delete this message and its attachments and kindly notify the sender by reply e-mail. Any content of this message and its attachments, which does not relate to the official business of the sending company must be taken not to have been sent or endorsed by the sending company or any of its related entities. No warranty is made that the e-mail or attachment(s) are free from computer virus or other defect. From als@REDACTED Tue Nov 30 10:00:33 2010 From: als@REDACTED (Anthony Shipman) Date: Tue, 30 Nov 2010 20:00:33 +1100 Subject: [erlang-questions] Erlounge Melbourne - Sat 4/12 In-Reply-To: <4CF48EAF.5070304@infinite-interactive.com> References: <4CF48EAF.5070304@infinite-interactive.com> Message-ID: <201011302000.33706.als@iinet.net.au> On Tue, 30 Nov 2010 04:42:07 pm Nikolas Bowe wrote: > Hi everyone. > > There will be an Erlounge at Beer DeLuxe from 7pm on Saturday 4th Dec. > This is just a social event. No presentations this time. Maybe next time > depending on what people want. How will we recognise you? -- Anthony Shipman Mamas don't let your babies als@REDACTED grow up to be outsourced. From info@REDACTED Tue Nov 30 10:49:52 2010 From: info@REDACTED (info) Date: Tue, 30 Nov 2010 10:49:52 +0100 Subject: emacs tool and the -smp erlang option Message-ID: <201011301049518598807@its3.ch> Hello, In order to use wxErlang, the -smp option shall be defined for erl.exe I am using on windows the emacs tool for the compilation. Is it possible to define the -smp option in the .emacs file ? If yes, how ? J-Ph. Constantin ITS3 Gen?ve www.its3.ch From garazdawi@REDACTED Tue Nov 30 11:08:00 2010 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 30 Nov 2010 11:08:00 +0100 Subject: [erlang-questions] emacs tool and the -smp erlang option In-Reply-To: <201011301049518598807@its3.ch> References: <201011301049518598807@its3.ch> Message-ID: First hit on google gives you this link, http://www.trapexit.org/forum/viewtopic.php?p=62741&sid=8bc60ffa55629c2b84a45ad04856ef47 Lukas On Tue, Nov 30, 2010 at 10:49 AM, info wrote: > Hello, > In order to use wxErlang, the -smp option shall be defined for erl.exe > I am using on windows the emacs tool for the compilation. Is it possible to > define the -smp option in the .emacs file ? > If yes, how ? > > J-Ph. Constantin > ITS3 Gen?ve > www.its3.ch > From info@REDACTED Tue Nov 30 11:34:06 2010 From: info@REDACTED (info) Date: Tue, 30 Nov 2010 11:34:06 +0100 Subject: [erlang-questions] emacs tool and the -smp erlang option References: <201011301049518598807@its3.ch>, Message-ID: <201011301134059536446@its3.ch> Ok but the following modification: (defvar inferior-erlang-machine-options '("-smp") has no impact for me ! J-Ph. Constantin ITS3 Gen?ve www.its3.ch From zabrane3@REDACTED Tue Nov 30 11:58:40 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 11:58:40 +0100 Subject: "simple_one_for_one" supervisor childs termination problem Message-ID: Hi everybody, I'm testing a "simple_one_for_one" supervisor (foo_sup.erl) to supervise a bunch of workers (foo_srv.erl implemented as gen_server). On the top of that, there's an application process (foo_app.erl) which let me start/stop the entire application easily. Everything went fine except the termination. When calling "application:stop(foo_app)", the workers never get their "terminate/2" function called. Here's my supervisor (foo_sup.erl) init/1 function: init([]) -> AChild = {foo_srv, {foo_srv, start_link, []}, transient, 180000, worker, [foo_srv]}, Strategy = {simple_one_for_one, 10, 100}, {ok, {Strategy, [AChild]}}. Any help will be very appreciated! N.B: it seems to me that the Erlang doc is also missing the fact the atom 'infinity' couldn't be use with simple_one_for_one. Am I right? So in this example, I'm using a kill timeout of 3 minutes (18000). -- Regards Zabrane From garazdawi@REDACTED Tue Nov 30 12:09:56 2010 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 30 Nov 2010 12:09:56 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: Message-ID: When terminating a child the supervisor calls exit(Pid,shutdown). When a process receives an exit signal it will automatically die unless it is trapping exits. So in order for terminate to be called before the process dies you have to trap exits within the child process. This is described in the gen_server Design Principles documentation: http://www.erlang.org/doc/design_principles/gen_server_concepts.html#id62546 Lukas On Tue, Nov 30, 2010 at 11:58 AM, zabrane Mikael wrote: > Hi everybody, > > I'm testing a "simple_one_for_one" supervisor (foo_sup.erl) to > supervise a bunch of workers (foo_srv.erl implemented as gen_server). > On the top of that, there's an application process (foo_app.erl) which > let me start/stop the entire application easily. > Everything went fine except the termination. > > When calling "application:stop(foo_app)", the workers never get their > "terminate/2" function called. > > Here's my supervisor (foo_sup.erl) init/1 function: > init([]) -> > AChild = {foo_srv, {foo_srv, start_link, []}, > transient, 180000, worker, [foo_srv]}, > Strategy = {simple_one_for_one, 10, 100}, > {ok, {Strategy, [AChild]}}. > > > Any help will be very appreciated! > > > N.B: it seems to me that the Erlang doc is also missing the fact the > atom 'infinity' > couldn't be use with simple_one_for_one. Am I right? So in this > example, I'm using a kill timeout of 3 minutes (18000). > > -- > Regards > Zabrane > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From matthew@REDACTED Tue Nov 30 12:20:44 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 11:20:44 +0000 Subject: [erlang-questions] {error, emfile} on windows In-Reply-To: References: Message-ID: <20101130112044.GA21723@rabbitmq.com> On Mon, Nov 29, 2010 at 11:18:21PM -0500, Kaiduan Xie wrote: > I tried to create 1K TCP/TLS client connections on a Windows box, and > got {error, emfile} error, any idea to solve it? I have no problem on > Linux after raising the ulimit. Whilst in theory windows has a limit of around 16M fds per process, the Erlang windows port seems to use bits of the MS C runtime. This seems to limit severely wrt number of available sockets. Whilst erlang:system_info(check_io) reports 2048 under Windows, that's clearly wrong. But hey, a few years ago, I found that fsync under the Windows port was implemented as "return 1;". Such is the quality of the Windows port. Matthew From matthew@REDACTED Tue Nov 30 12:26:01 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 11:26:01 +0000 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: Message-ID: <20101130112601.GB21723@rabbitmq.com> On Tue, Nov 30, 2010 at 12:09:56PM +0100, Lukas Larsson wrote: > When terminating a child the supervisor calls exit(Pid,shutdown). When a > process receives an exit signal it will automatically die unless it is > trapping exits. So in order for terminate to be called before the process > dies you have to trap exits within the child process. Yeah, but trapping exits is not normally a good idea. Zabrane probably wants to use Rabbit's supervisor2 [0] which adds simple_one_for_one_terminate which is the same as simple_one_for_one except children are killed off as per the indicated shutdown spec (see man supervisor). [0] http://hg.rabbitmq.com/rabbitmq-server/file/default/src/supervisor2.erl Matthew From cbenac@REDACTED Tue Nov 30 13:18:21 2010 From: cbenac@REDACTED (Clara Benac Earle) Date: Tue, 30 Nov 2010 13:18:21 +0100 Subject: [ANN] Madrid Erlounge 2nd December Message-ID: <4CF4EB8D.9000504@fi.upm.es> The Madrid Erlang user group will be meeting on Thursday 2nd of December at 19:30 at the Escuela Universitaria de Ingenier?a T?cnica Industrial. Ronda de Valencia 3 (metro/cercanias Embajadores). Jordi Chac?n from Klarna will be giving a talk. Abstract: Klarna AB is a successful fast-growing company that provides payment solutions for online shopping. Klarna's system is 100% built in Erlang and this is without question, one of the main keys for this success. In this talk, I will describe how Klarna benefits from working with Erlang and how we make use of all the nice features that this language provides such as concurrency, fault-tolerance and hot code swapping. Furthermore, I will also talk about the tools we use to perform day to day tasks like debugging, testing and code edition. Around 9pm we will go somewhere in the neighbourhood for an Erlounge (beer, food and talk). Everybody is welcome to join us. Looking forward to seeing you in Madrid! Clara From andy.kriger@REDACTED Tue Nov 30 13:18:43 2010 From: andy.kriger@REDACTED (Andy Kriger) Date: Tue, 30 Nov 2010 07:18:43 -0500 Subject: mnesia and lazy loading? Message-ID: In SQL, it's possible to request specific fields on a table. Is it possible to do something similar in mnesia? Which is to say, only return the values in certain fields of the record? Yes or no determines a design decision on which data belongs in the record and which data requires an secondary record associated to the first by id. thx From zabrane3@REDACTED Tue Nov 30 13:40:09 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 13:40:09 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: Message-ID: Hi Lucas, 2010/11/30 Lukas Larsson : > When terminating a child the supervisor calls exit(Pid,shutdown). When a > process receives an exit signal it will automatically die unless it is > trapping exits. So in order for terminate to be called before the process > dies you have to trap exits within the child process. I forgot to mention that I'm already traping exit in "foo_srv.erl": %% --- gen server callbacks init(Args) -> process_flag(trap_exit, true), {ok, Args}. Even with that, the worker process never get the "shutdown" EXIT message. -- Regards Zabrane From zabrane3@REDACTED Tue Nov 30 13:42:17 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 13:42:17 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: <20101130112601.GB21723@rabbitmq.com> References: <20101130112601.GB21723@rabbitmq.com> Message-ID: Hi Matthew, > Yeah, but trapping exits is not normally a good idea. Zabrane probably > wants to use Rabbit's supervisor2 [0] which adds > simple_one_for_one_terminate which is the same as simple_one_for_one > except children are killed off as per the indicated shutdown spec (see > man supervisor). > > [0] http://hg.rabbitmq.com/rabbitmq-server/file/default/src/supervisor2.erl I'll give this a try as I'm already using "gen_server2" (from RabbitMQ). Thanks for the pointer. -- Regards Zabrane From matthew@REDACTED Tue Nov 30 13:59:31 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 12:59:31 +0000 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: Message-ID: <20101130125930.GC21723@rabbitmq.com> On Tue, Nov 30, 2010 at 01:40:09PM +0100, zabrane Mikael wrote: > I forgot to mention that I'm already traping exit in "foo_srv.erl": > %% --- gen server callbacks > init(Args) -> > process_flag(trap_exit, true), > {ok, Args}. > > Even with that, the worker process never get the "shutdown" EXIT message. That's odd then. That should only happen if there's a brutal_kill/exit(kill) going on. Matthew From zabrane3@REDACTED Tue Nov 30 14:02:01 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 14:02:01 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: <20101130125930.GC21723@rabbitmq.com> References: <20101130125930.GC21723@rabbitmq.com> Message-ID: Yep > That's odd then. That should only happen if there's a > brutal_kill/exit(kill) going on. > > Matthew From hm@REDACTED Tue Nov 30 14:02:07 2010 From: hm@REDACTED (=?ISO-8859-1?Q?H=E5kan_Mattsson?=) Date: Tue, 30 Nov 2010 14:02:07 +0100 Subject: [erlang-questions] mnesia and lazy loading? In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 1:18 PM, Andy Kriger wrote: > In SQL, it's possible to request specific fields on a table. > Is it possible to do something similar in mnesia? No, not if the purpose is to gain performance. Otherwise you can use mnesia:select/2 or qlc. /H?kan From zabrane3@REDACTED Tue Nov 30 14:05:44 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 14:05:44 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: <20101130125930.GC21723@rabbitmq.com> References: <20101130125930.GC21723@rabbitmq.com> Message-ID: Hi Matthew, It works like a charm with supervisor2. You save my day ;-) Thousand thanks. -- Regards Zabrane 2010/11/30 Matthew Sackman : > On Tue, Nov 30, 2010 at 01:40:09PM +0100, zabrane Mikael wrote: >> I forgot to mention that I'm already traping exit in "foo_srv.erl": >> %% --- gen server callbacks >> init(Args) -> >> ? ? process_flag(trap_exit, true), >> ? ? {ok, Args}. >> >> Even with that, the worker process never get the "shutdown" EXIT message. > > That's odd then. That should only happen if there's a > brutal_kill/exit(kill) going on. > > Matthew > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From garazdawi@REDACTED Tue Nov 30 14:14:24 2010 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 30 Nov 2010 14:14:24 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: <20101130125930.GC21723@rabbitmq.com> Message-ID: might be gen_server2 which does not play nice with supervisor, using OTP's supervisor and gen_server the terminate call works as it should. Lukas On Tue, Nov 30, 2010 at 2:05 PM, zabrane Mikael wrote: > Hi Matthew, > > It works like a charm with supervisor2. > You save my day ;-) Thousand thanks. > > -- > Regards > Zabrane > > 2010/11/30 Matthew Sackman : > > On Tue, Nov 30, 2010 at 01:40:09PM +0100, zabrane Mikael wrote: > >> I forgot to mention that I'm already traping exit in "foo_srv.erl": > >> %% --- gen server callbacks > >> init(Args) -> > >> process_flag(trap_exit, true), > >> {ok, Args}. > >> > >> Even with that, the worker process never get the "shutdown" EXIT > message. > > > > That's odd then. That should only happen if there's a > > brutal_kill/exit(kill) going on. > > > > Matthew > > > > ________________________________________________________________ > > erlang-questions (at) erlang.org mailing list. > > See http://www.erlang.org/faq.html > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > > > > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From matthew@REDACTED Tue Nov 30 14:14:18 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 13:14:18 +0000 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: <20101130125930.GC21723@rabbitmq.com> Message-ID: <20101130131418.GE21723@rabbitmq.com> Hi Zabrane, On Tue, Nov 30, 2010 at 02:05:44PM +0100, zabrane Mikael wrote: > It works like a charm with supervisor2. > You save my day ;-) Thousand thanks. Glad it works. I'm sure over time we'll likely end up replacing/rewriting most of the stdlib and OTP framework ;) Matthew From matthew@REDACTED Tue Nov 30 14:16:41 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 13:16:41 +0000 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: <20101130125930.GC21723@rabbitmq.com> Message-ID: <20101130131640.GF21723@rabbitmq.com> On Tue, Nov 30, 2010 at 02:14:24PM +0100, Lukas Larsson wrote: > might be gen_server2 which does not play nice with supervisor, using OTP's > supervisor and gen_server the terminate call works as it should. Not impossible - we've found bugs in gs2 before. If Zabrane can post the code he's using I can certainly take a look at that. Matthew From zabrane3@REDACTED Tue Nov 30 14:20:07 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 14:20:07 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: <20101130125930.GC21723@rabbitmq.com> Message-ID: Maybe that's the reason. Anyway, I'm pretty happy with gen_server2 as we're using it for a while in production (we moved all our code from gen_server togen_server2). Thanks Lukas -- Regards Zabrane 2010/11/30 Lukas Larsson : > might be gen_server2 which does not play nice with supervisor, using OTP's > supervisor and gen_server the terminate call works as it should. > > Lukas > > On Tue, Nov 30, 2010 at 2:05 PM, zabrane Mikael wrote: >> >> Hi Matthew, >> >> It works like a charm with supervisor2. >> You save my day ;-) Thousand thanks. >> >> -- >> Regards >> Zabrane >> >> 2010/11/30 Matthew Sackman : >> > On Tue, Nov 30, 2010 at 01:40:09PM +0100, zabrane Mikael wrote: >> >> I forgot to mention that I'm already traping exit in "foo_srv.erl": >> >> %% --- gen server callbacks >> >> init(Args) -> >> >> ? ? process_flag(trap_exit, true), >> >> ? ? {ok, Args}. >> >> >> >> Even with that, the worker process never get the "shutdown" EXIT >> >> message. >> > >> > That's odd then. That should only happen if there's a >> > brutal_kill/exit(kill) going on. >> > >> > Matthew >> > >> > ________________________________________________________________ >> > erlang-questions (at) erlang.org mailing list. >> > See http://www.erlang.org/faq.html >> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > >> > >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> > > From zabrane3@REDACTED Tue Nov 30 14:20:44 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 14:20:44 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: <20101130131418.GE21723@rabbitmq.com> References: <20101130125930.GC21723@rabbitmq.com> <20101130131418.GE21723@rabbitmq.com> Message-ID: ;-) 2010/11/30 Matthew Sackman : > Hi Zabrane, > > On Tue, Nov 30, 2010 at 02:05:44PM +0100, zabrane Mikael wrote: >> It works like a charm with supervisor2. >> You save my day ;-) Thousand thanks. > > Glad it works. I'm sure over time we'll likely end up > replacing/rewriting most of the stdlib and OTP framework ;) > > Matthew > -- Regards Zabrane From zabrane3@REDACTED Tue Nov 30 14:38:48 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 14:38:48 +0100 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: <20101130131640.GF21723@rabbitmq.com> References: <20101130125930.GC21723@rabbitmq.com> <20101130131640.GF21723@rabbitmq.com> Message-ID: Sorry, that's a closed source code! -- Regards Zabrane 2010/11/30 Matthew Sackman : > On Tue, Nov 30, 2010 at 02:14:24PM +0100, Lukas Larsson wrote: >> might be gen_server2 which does not play nice with supervisor, using OTP's >> supervisor and gen_server the terminate call works as it should. > > Not impossible - we've found bugs in gs2 before. If Zabrane can post the > code he's using I can certainly take a look at that. > > Matthew From mazen.harake@REDACTED Tue Nov 30 14:48:02 2010 From: mazen.harake@REDACTED (Mazen Harake) Date: Tue, 30 Nov 2010 15:48:02 +0200 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: References: <20101130125930.GC21723@rabbitmq.com> Message-ID: <4CF50092.3020905@erlang-solutions.com> Just out of curiousity, what does gen_server2 bring to the table? /M On 30/11/2010 15:20, zabrane Mikael wrote: > Maybe that's the reason. Anyway, I'm pretty happy with gen_server2 as > we're using it for a while in production (we moved all our code from gen_server > togen_server2). > > Thanks Lukas > From matthew@REDACTED Tue Nov 30 14:47:26 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 13:47:26 +0000 Subject: [erlang-questions] "simple_one_for_one" supervisor childs termination problem In-Reply-To: <4CF50092.3020905@erlang-solutions.com> References: <20101130125930.GC21723@rabbitmq.com> <4CF50092.3020905@erlang-solutions.com> Message-ID: <20101130134726.GH21723@rabbitmq.com> On Tue, Nov 30, 2010 at 03:48:02PM +0200, Mazen Harake wrote: > Just out of curiousity, what does gen_server2 bring to the table? Please read the documentation at the top of it: http://hg.rabbitmq.com/rabbitmq-server/file/default/src/gen_server2.erl Some of the points (eg point 2) are now taken care of (since R14) directly in Erlang. However, we still support Rabbit back to R12B5 so we can't rely on that just yet. Matthew From kenneth.lundin@REDACTED Tue Nov 30 15:08:33 2010 From: kenneth.lundin@REDACTED (Kenneth Lundin) Date: Tue, 30 Nov 2010 15:08:33 +0100 Subject: [erlang-questions] {error, emfile} on windows In-Reply-To: <20101130112044.GA21723@rabbitmq.com> References: <20101130112044.GA21723@rabbitmq.com> Message-ID: On Windows there is no limit to set in the shell like in UNIX/Linux that the Erlang VM can read and adjust according to. Therefore you can use the environment variable ERL_MAX_PORTS to set a higher limit thant the default (1024). But bevare that the number you set will imply an allocation of memory to hold all these port data structures. So don't set a higher value than you really need. You can actually set this on the command line when you start erlang, like this example: erl -env ERL_MAX_PORTS 4096 See the documentation at: http://www.erlang.org/doc/efficiency_guide/advanced.html#id64904 (I admit that it is not easy to find this, but the Efficiency Guide is important to know about and to read through for all system builders) We are thinking of changing this to a totally dynamic number of ports only limited by the OS setup, but that work is not scheduled for a specific release yet. On Tue, Nov 30, 2010 at 12:20 PM, Matthew Sackman wrote: > On Mon, Nov 29, 2010 at 11:18:21PM -0500, Kaiduan Xie wrote: >> I tried to create 1K TCP/TLS client connections on a Windows box, and >> got {error, emfile} error, any idea to solve it? I have no problem on >> Linux after raising the ulimit. > > Whilst in theory windows has a limit of around 16M fds per process, the > Erlang windows port seems to use bits of the MS C runtime. This seems to > limit severely wrt number of available sockets. Well this is not a correct statement as already described. > > Whilst erlang:system_info(check_io) reports 2048 under Windows, that's > clearly wrong. But hey, a few years ago, I found that fsync under the > Windows port was implemented as "return 1;". Such is the quality of the > Windows port. This is not the case now, fsync is definitely implemented. Windows has not been our main target platform for Erlang but I still think the quality is good even on Windows. With an increasing number of developers doing serious applications with Erlang on Windows the functionality and quality will become even better. /Kenneth Erlang/OTP, Ericsson > > Matthew > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From armando.dicianno@REDACTED Tue Nov 30 15:10:06 2010 From: armando.dicianno@REDACTED (Armando Di Cianno) Date: Tue, 30 Nov 2010 09:10:06 -0500 Subject: [erlang-questions] mnesia and lazy loading? In-Reply-To: References: Message-ID: 2010/11/30 H?kan Mattsson : > On Tue, Nov 30, 2010 at 1:18 PM, Andy Kriger wrote: >> In SQL, it's possible to request specific fields on a table. >> Is it possible to do something similar in mnesia? > > No, not if the purpose is to gain performance. > Otherwise you can use ?mnesia:select/2 or qlc. Agreed. I would go with qlc, in this case, as the list comprehension used makes it make more sense in my mind. E.g. of qlc with some search constraints. > {MegaSecs, Secs, _MicroSecs} = now(). > SecondsAtNow = MegaSecs * 1000000 + Secs. > Q = qlc:q([{X.id, X.other_column} || X <- mnesia:table(x), (SecondsAtNow - X#x.timestamp) > 60]). > F = fun() -> qlc:e(Q) end. > {atomic, List} = mnesia:transaction(F), List now contains tuples of X.id and X.other_column, where X.timestamp is older than a minute. __armando > > /H?kan > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From kaiduanx@REDACTED Tue Nov 30 17:00:39 2010 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Tue, 30 Nov 2010 11:00:39 -0500 Subject: [erlang-questions] {error, emfile} on windows In-Reply-To: References: <20101130112044.GA21723@rabbitmq.com> Message-ID: Kenneth, Setting ERL_MAX_PORT still does not resolve the problem, it can only open 31 TLS connections. C:\erl5.7.1\usr>..\bin\erl.exe -env ERL_MAX_PORTS 4096 Eshell V5.7.1 (abort with ^G) 1> application:start(ssl). ok 2> Socks = test_tls_sip:connect("10.9.22.85", 5061, 512, 1). ** exception error: no match of right hand side value {error,emfile} in function test_tls_sip:'-connect/4-fun-0-'/5 in call from lists:foldl/3 connect(Addr, Port, N, TimerValue) -> lists:foldl(fun(_I, Acc) -> {ok, Socket} = ssl:connect(Addr, Port, [{active, true}]), {ok, Pid} = gen_server:start(erlang_tls_connection, {Socket, TimerValue}, []), ssl:controlling_process(Socket, Pid), [Pid|Acc] end, [], lists:seq(1, N)). Thanks, /Kaiduan On Tue, Nov 30, 2010 at 9:08 AM, Kenneth Lundin wrote: > On Windows there is no limit to set in the shell like in UNIX/Linux > that the Erlang VM > can read and adjust according to. > Therefore you can use the environment variable ERL_MAX_PORTS to set a > higher limit > thant the default (1024). But bevare that the number you set will > imply an allocation > of ?memory to hold all these port data structures. So don't set a > higher value than you really need. > You can actually set this on the command line when you start erlang, > like this example: > erl -env ERL_MAX_PORTS 4096 > > See the documentation at: > http://www.erlang.org/doc/efficiency_guide/advanced.html#id64904 > (I admit that it is not easy to find this, but the Efficiency Guide is > important to know > about and to read through for all system builders) > > We are thinking of changing this to a totally dynamic number of ports > only limited by the OS setup, but that work is not scheduled for a > specific release yet. > > > On Tue, Nov 30, 2010 at 12:20 PM, Matthew Sackman wrote: >> On Mon, Nov 29, 2010 at 11:18:21PM -0500, Kaiduan Xie wrote: >>> I tried to create 1K TCP/TLS client connections on a Windows box, and >>> got {error, emfile} error, any idea to solve it? I have no problem on >>> Linux after raising the ulimit. >> >> Whilst in theory windows has a limit of around 16M fds per process, the >> Erlang windows port seems to use bits of the MS C runtime. This seems to >> limit severely wrt number of available sockets. > > Well this is not a correct statement as already described. >> >> Whilst erlang:system_info(check_io) reports 2048 under Windows, that's >> clearly wrong. But hey, a few years ago, I found that fsync under the >> Windows port was implemented as "return 1;". Such is the quality of the >> Windows port. > This is not the case now, fsync is definitely implemented. > > Windows has not been our main target platform for Erlang but I still > think the quality is good even on Windows. > > With an increasing number of developers doing serious applications > with Erlang on > Windows the functionality and quality will become even better. > > > /Kenneth Erlang/OTP, Ericsson > >> >> Matthew >> >> ________________________________________________________________ >> erlang-questions (at) erlang.org mailing list. >> See http://www.erlang.org/faq.html >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED >> >> > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From matthew@REDACTED Tue Nov 30 17:06:48 2010 From: matthew@REDACTED (Matthew Sackman) Date: Tue, 30 Nov 2010 16:06:48 +0000 Subject: [erlang-questions] {error, emfile} on windows In-Reply-To: References: <20101130112044.GA21723@rabbitmq.com> Message-ID: <20101130160648.GI21723@rabbitmq.com> On Tue, Nov 30, 2010 at 03:08:33PM +0100, Kenneth Lundin wrote: > See the documentation at: > http://www.erlang.org/doc/efficiency_guide/advanced.html#id64904 > (I admit that it is not easy to find this, but the Efficiency Guide is > important to know about and to read through for all system builders) Interesting - I've read that several times before ... except clearly not the bits about ports and sockets. Thanks for the pointer. Could you clarify the documentation a bit though - for example, on my Linux machine, I have a ulimit -n of 100000 and I don't seem to need to configure anything further (certainly I've never touched ERL_MAX_PORTS) for Erlang to be able to use many sockets and fds. Thus for platforms which have an Erlang port and ulimit, which of the ulimit -n and ERL_MAX_PORTS is used if both are set? > We are thinking of changing this to a totally dynamic number of ports > only limited by the OS setup, but that work is not scheduled for a > specific release yet. So under Windows, various sources eg http://blogs.technet.com/markrussinovich/archive/2009/09/29/3283844.aspx suggest that the limit is 16M. You've suggested that unnecessarily high numbers are likely to waste memory, so would the limit be detected and set dynamically at startup or would memory only be allocated as needed? Also, under unices, file descriptors and sockets come from the same pool, but under Windows, my understanding is that this is not the case. Could you confirm that in the Windows port, file descriptors are win32 file handles, and sockets are winsock socket handles? Is there going to be a supported+documented API to be able to inspect the limits of each and the number of each allocated that works on both Windows and unices? > > Whilst in theory windows has a limit of around 16M fds per process, the > > Erlang windows port seems to use bits of the MS C runtime. This seems to > > limit severely wrt number of available sockets. > > Well this is not a correct statement as already described. Indeed not. Thanks for the illumination. Matthew From kaiduanx@REDACTED Tue Nov 30 17:34:42 2010 From: kaiduanx@REDACTED (Kaiduan Xie) Date: Tue, 30 Nov 2010 11:34:42 -0500 Subject: Improve the SSL accept performance Message-ID: Hi all, I just found the SSL accept in erlang is slower than C version. The load test opens 10K persistent SSL connection at the same time, it takes longer for Erlang version to finish. In Erlang version, for each TLS connection, a gen_fsm is started to waiting for packet. The code snippet to accept SSL connection is listed as below, {ok, Socket} = ssl:transport_accept(State#state.lsocket), gen_fsm:start(erlsip_transport_tls_connection, {server, State#state.lsocket, State#state.parent}, []), ok = ssl:ssl_accept(Socket), {ok, {PeerAddr, PeerPort}} = ssl:peername(Socket), ?DEBUG("Accepted peer:~p:~p", [PeerAddr, PeerPort]), State#state.parent ! {add_tls_connection, PeerAddr, PeerPort, self()}, {next_state, wait_for_crlf_crlf, State#state{socket = Socket, peer_address = PeerAddr, peer_port = PeerPort}}. Can anyone point out if I am doing things wrong? Can we start multiple erlsip_transport_tls_connection gen_fsm to wait for connection? The documentation is not clear on this point. Any comments are welcome. I am building an Erlang prototype to persuade the management to go for Erlang instead of legacy C version. BTW, OTP-12B-5 is used on Linux. Thanks, /Kaiduan From zabrane3@REDACTED Tue Nov 30 17:38:22 2010 From: zabrane3@REDACTED (zabrane Mikael) Date: Tue, 30 Nov 2010 17:38:22 +0100 Subject: [erlang-questions] "sendfile" call as NIF. Good or bad idea? In-Reply-To: References: Message-ID: Hi everybody, Michale Santos pointed me that there's some work (by Bj?rn Gustavsson) to integrate "sendfile" natively in Erlang: https://github.com/erlang/otp/commit/34042f49c7b5f0ac1b91463344db663eddc2e76d#diff-3 Could someone from OTP team tell me if this will happen soon? In which release if possible? -- Regards Zabrane 2010/11/27 zabrane Mikael : > Hi Steve, > >> Not sure why you find it cryptic, since as drivers go it's pretty >> simple. Can you provide further details? > > Maybe it was because NIF are very simple to read/write. > Again, no offense ... forget about that comment. > >> The benefit of a driver is that we can just register each socket FD >> with erts and it lets us know when each socket is writeable. The >> driver mainly just handles tracking how much was written by each >> sendfile call, incrementing offsets, etc. > > Thanks for this. I didn't consider this problem at all. > >> If you implemented it as a NIF, you'd need to perform your own polling >> on the socket FDs to know when sendfile can write to them. This alone >> might mean a portability issue, since you'd have to know to use the >> right polling mechanism for the given platform. You'd also need to do >> all this in your own thread, whereas the driver doesn't have to deal >> with threads. > > So, I'll stick with your driver. > Thanks Steve. > > -- > Regards > Zabrane > From attila.r.nohl@REDACTED Tue Nov 30 18:15:42 2010 From: attila.r.nohl@REDACTED (Attila Rajmund Nohl) Date: Tue, 30 Nov 2010 18:15:42 +0100 Subject: [erlang-questions] Improve the SSL accept performance In-Reply-To: References: Message-ID: Hello! I suggest testing with OTP R14B, the SSL implementation in Erlang has been rewritten since R12B. 2010/11/30, Kaiduan Xie : > Hi all, > > I just found the SSL accept in erlang is slower than C version. The > load test opens 10K persistent SSL connection at the same time, it > takes longer for Erlang version to finish. In Erlang version, for each > TLS connection, a gen_fsm is started to waiting for packet. The code > snippet to accept SSL connection is listed as below, > > {ok, Socket} = ssl:transport_accept(State#state.lsocket), > gen_fsm:start(erlsip_transport_tls_connection, > {server, State#state.lsocket, State#state.parent}, > []), > ok = ssl:ssl_accept(Socket), > {ok, {PeerAddr, PeerPort}} = ssl:peername(Socket), > ?DEBUG("Accepted peer:~p:~p", [PeerAddr, PeerPort]), > State#state.parent ! {add_tls_connection, PeerAddr, PeerPort, self()}, > {next_state, wait_for_crlf_crlf, State#state{socket = Socket, > peer_address = PeerAddr, peer_port = PeerPort}}. > > > Can anyone point out if I am doing things wrong? Can we start multiple > erlsip_transport_tls_connection gen_fsm to wait for connection? The > documentation is not clear on this point. > > Any comments are welcome. I am building an Erlang prototype to > persuade the management to go for Erlang instead of legacy C version. > > BTW, OTP-12B-5 is used on Linux. > > Thanks, > > /Kaiduan > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From tomas.abrahamsson@REDACTED Tue Nov 30 19:27:10 2010 From: tomas.abrahamsson@REDACTED (Tomas Abrahamsson) Date: Tue, 30 Nov 2010 19:27:10 +0100 Subject: [erlang-questions] emacs tool and the -smp erlang option In-Reply-To: <201011301134059536446@its3.ch> References: <201011301049518598807@its3.ch> <201011301134059536446@its3.ch> Message-ID: > Ok but the following modification: > (defvar inferior-erlang-machine-options '("-smp") > has no impact for me ! Hi, try (setq inferior-erlang-machine-options '("-smp")) instead! The defvar special form sets a variable only if it is undefined, while setq sets the variable regardless. Thus, defvar is useful if you want to define a default value and you can override it using setq in e.g your .emacs. BRs Tomas From marcel.meyer@REDACTED Tue Nov 30 19:36:14 2010 From: marcel.meyer@REDACTED (Marcel Meyer) Date: Tue, 30 Nov 2010 13:36:14 -0500 Subject: spawning funs on other nodes that does not contain my module Message-ID: Hi there, Is it possible to spawn a function (which is pretty straight forward, essentially a receive loop that grabs the message and file:write_file's it to the file system) on another node, which does not have a copy of my module? I thought I would be able to make a fun that returns the mentioned fun above and then spawn that message sink on another node. I am investigating writing a file propagation system and it would be nice if there is one process that gets notified of new files, which spawns a fun on each recipient node, which tells that node how to handle it. Any insights? Or should each node be running a file sink that just listens for these {save, Data} msgs? I am trying to write something where I dont have to manage the child nodes and their code. Perhaps a file sink gen_server on each node that supports upgrades? Regards, Marcel From garazdawi@REDACTED Tue Nov 30 21:31:04 2010 From: garazdawi@REDACTED (Lukas Larsson) Date: Tue, 30 Nov 2010 21:31:04 +0100 Subject: [erlang-questions] spawning funs on other nodes that does not contain my module In-Reply-To: References: Message-ID: It is not possible directly, you would have to get the code to the other side somehow. code:load_binary is made specifically for this purpose, http://erlang.org/doc/man/code.html#load_binary-3 Just use rpc:call(MyNode, code,load_binary,[mymod,"mymod.erl",BinaryModuleCode]) and off you go! Lukas On Tue, Nov 30, 2010 at 7:36 PM, Marcel Meyer wrote: > Hi there, > > Is it possible to spawn a function (which is pretty straight > forward, essentially a receive loop that grabs the message and > file:write_file's it to the file system) on another node, which does not > have a copy of my module? > > I thought I would be able to make a fun that returns the mentioned fun > above > and then spawn that message sink on another node. > > I am investigating writing a file propagation system and it would be nice > if > there is one process that gets notified of new files, which spawns a fun on > each recipient node, which tells that node how to handle it. > > Any insights? Or should each node be running a file sink that just listens > for these {save, Data} msgs? I am trying to write something where I dont > have to manage the child nodes and their code. Perhaps a file sink > gen_server on each node that supports upgrades? > > Regards, > Marcel > From krab@REDACTED Tue Nov 30 20:19:42 2010 From: krab@REDACTED (Kresten Krab Thorup) Date: Tue, 30 Nov 2010 20:19:42 +0100 Subject: [erlang-questions] spawning funs on other nodes that does not contain my module In-Reply-To: References: Message-ID: Marcel, I discuss these issues at some length in this blog post: http://bit.ly/fQoKlu; and towards the end you'll see some code that lets you construct an interpreted fun, like this: 1> FunStr = "fun (A) -> A+B end.". 2> {ok, Tokens, _} = erl_scan:string(FunStr). 3> {ok, [Form]} = erl_parse:parse_exprs(Tokens). 4> Bindings = erl_eval:add_binding('B', 2, erl_eval:new_bindings()). 5> {value, Fun, _} = erl_eval:expr(Form, Bindings). 6> Fun(1). 3 Such a fun should be passable between nodes, if only the target node has erl_eval (and related modules). Which should be very likely, but it needs to be the same version of erl_eval. It is not very fast though, but that may not be important to you; especially if what you want to do is something simple. Kresten On Nov 30, 2010, at 19:36 , Marcel Meyer wrote: > Hi there, > > Is it possible to spawn a function (which is pretty straight > forward, essentially a receive loop that grabs the message and > file:write_file's it to the file system) on another node, which does not > have a copy of my module? > > I thought I would be able to make a fun that returns the mentioned fun above > and then spawn that message sink on another node. > > I am investigating writing a file propagation system and it would be nice if > there is one process that gets notified of new files, which spawns a fun on > each recipient node, which tells that node how to handle it. > > Any insights? Or should each node be running a file sink that just listens > for these {save, Data} msgs? I am trying to write something where I dont > have to manage the child nodes and their code. Perhaps a file sink > gen_server on each node that supports upgrades? > > Regards, > Marcel Kresten Krab Thorup, CTO, Trifork From mazen.harake@REDACTED Tue Nov 30 23:00:05 2010 From: mazen.harake@REDACTED (Mazen Harake) Date: Wed, 01 Dec 2010 00:00:05 +0200 Subject: [erlang-questions] spawning funs on other nodes that does not contain my module In-Reply-To: References: Message-ID: <4CF573E5.8040804@erlang-solutions.com> This is how I do it (taken from: https://github.com/mazenharake/entop/blob/master/src/entop_view.erl#L59) remote_load_code(Module, Node) -> {_, Binary, Filename} = code:get_object_code(Module), rpc:call(Node, code, load_binary, [Module, Filename, Binary]). /M On 30/11/2010 22:31, Lukas Larsson wrote: > It is not possible directly, you would have to get the code to the other > side somehow. code:load_binary is made specifically for this purpose, > > http://erlang.org/doc/man/code.html#load_binary-3 > > Just use rpc:call(MyNode, > code,load_binary,[mymod,"mymod.erl",BinaryModuleCode]) and off you go! > > Lukas > > On Tue, Nov 30, 2010 at 7:36 PM, Marcel Meyerwrote: > >> Hi there, >> >> Is it possible to spawn a function (which is pretty straight >> forward, essentially a receive loop that grabs the message and >> file:write_file's it to the file system) on another node, which does not >> have a copy of my module? >> >> I thought I would be able to make a fun that returns the mentioned fun >> above >> and then spawn that message sink on another node. >> >> I am investigating writing a file propagation system and it would be nice >> if >> there is one process that gets notified of new files, which spawns a fun on >> each recipient node, which tells that node how to handle it. >> >> Any insights? Or should each node be running a file sink that just listens >> for these {save, Data} msgs? I am trying to write something where I dont >> have to manage the child nodes and their code. Perhaps a file sink >> gen_server on each node that supports upgrades? >> >> Regards, >> Marcel >> From marcel.meyer@REDACTED Tue Nov 30 23:10:36 2010 From: marcel.meyer@REDACTED (Marcel Meyer) Date: Tue, 30 Nov 2010 17:10:36 -0500 Subject: [erlang-questions] spawning funs on other nodes that does not contain my module In-Reply-To: References: Message-ID: Thanks Kresten, The bottom part of your article where you rpc:multicall the code and force load it on each node ended up working perfectly. You know, this stuff really is magic. Thank you again, Marcel On Tue, Nov 30, 2010 at 2:19 PM, Kresten Krab Thorup wrote: > Marcel, > > I discuss these issues at some length in this blog post: > http://bit.ly/fQoKlu; and towards the end you'll see some code that lets > you construct an interpreted fun, like this: > > 1> FunStr = "fun (A) -> A+B end.". > 2> {ok, Tokens, _} = erl_scan:string(FunStr). > 3> {ok, [Form]} = erl_parse:parse_exprs(Tokens). > 4> Bindings = erl_eval:add_binding('B', 2, erl_eval:new_bindings()). > 5> {value, Fun, _} = erl_eval:expr(Form, Bindings). > 6> Fun(1). > 3 > > Such a fun should be passable between nodes, if only the target node has > erl_eval (and related modules). Which should be very likely, but it needs > to be the same version of erl_eval. It is not very fast though, but that > may not be important to you; especially if what you want to do is something > simple. > > Kresten > > > On Nov 30, 2010, at 19:36 , Marcel Meyer wrote: > > > Hi there, > > > > Is it possible to spawn a function (which is pretty straight > > forward, essentially a receive loop that grabs the message and > > file:write_file's it to the file system) on another node, which does not > > have a copy of my module? > > > > I thought I would be able to make a fun that returns the mentioned fun > above > > and then spawn that message sink on another node. > > > > I am investigating writing a file propagation system and it would be nice > if > > there is one process that gets notified of new files, which spawns a fun > on > > each recipient node, which tells that node how to handle it. > > > > Any insights? Or should each node be running a file sink that just > listens > > for these {save, Data} msgs? I am trying to write something where I dont > > have to manage the child nodes and their code. Perhaps a file sink > > gen_server on each node that supports upgrades? > > > > Regards, > > Marcel > > Kresten Krab Thorup, CTO, Trifork > > From julian.doherty.ml@REDACTED Tue Nov 30 23:26:07 2010 From: julian.doherty.ml@REDACTED (Julian Doherty) Date: Wed, 1 Dec 2010 09:26:07 +1100 Subject: [erlang-questions] Erlounge Melbourne - Sat 4/12 In-Reply-To: <201011302000.33706.als@iinet.net.au> References: <4CF48EAF.5070304@infinite-interactive.com> <201011302000.33706.als@iinet.net.au> Message-ID: So many tech events happening in Melbourne around YOW this week all at once! Tricky to fit them all in. Would love to come along on Saturday night, but not sure if I'll make it or not. I'm hosting Francesco's talk at Lonely Planet on Saturday, so will see a few of you there. Am definitely keen for future events and getting regular meetups happening with the Erlang crowd in Melbourne. Cheers Julian Doherty On 30 November 2010 20:00, Anthony Shipman wrote: > On Tue, 30 Nov 2010 04:42:07 pm Nikolas Bowe wrote: > > Hi everyone. > > > > There will be an Erlounge at Beer DeLuxe from 7pm on Saturday 4th Dec. > > This is just a social event. No presentations this time. Maybe next time > > depending on what people want. > > How will we recognise you? > -- > Anthony Shipman Mamas don't let your babies > als@REDACTED grow up to be outsourced. > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED > > From robert.virding@REDACTED Tue Nov 30 23:50:35 2010 From: robert.virding@REDACTED (Robert Virding) Date: Tue, 30 Nov 2010 22:50:35 +0000 (GMT) Subject: [erlang-questions] Implementation of a 2006 version of Wichmann-Hill random number generator for Erlang/OTP In-Reply-To: <611195571.146781291157291062.JavaMail.root@zimbra> Message-ID: <832946219.146801291157435038.JavaMail.root@zimbra> What would be nice is if we could just drop the new algorithm into the old module so we avoid having to have 2 random modules. As I see it the main problem is that the old random has 3 seeds while the new has 4, which makes the seed functions incompatible. Couldn't you just have one seed be two? It would mean that the "new" random would return better values, but they would be different from what the "old" random returned. I don't see this as a problem but others might. Robert ----- "Kenji Rikitake" wrote: > The fix looks much better. I'll put it in the gist. > (BTW gist is a git archive itself so you can fork by yourself) > Kenji Rikitake > > In the message > dated Thu, Nov 25, 2010 at 11:34:48AM +1300, > Richard O'Keefe writes: > > On 24/11/2010, at 6:11 PM, Kenji Rikitake wrote: > > > > > The 2006 Wichmann-Hill RNG for Erlang moved to > > > https://gist.github.com/713144 > > > under the name of > > > random_wh06.erl > > > > Didn't we just have a discussion about not allowing zeros? > > Shouldn't seed/4 be > > > > seed(A1, A2, A3, A4) -> > > put(random_wh06_seed, > > {abs(A1) rem 2147483578 + 1, > > abs(A2) rem 2147483542 + 1, > > abs(A3) rem 2147483422 + 1, > > abs(A4) rem 2147483122 + 1}). > > > > Or did I misunderstand? > > ________________________________________________________________ > erlang-questions (at) erlang.org mailing list. > See http://www.erlang.org/faq.html > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED